Inject routes with multiple next-hops in Junos
Recently I’ve been working on solutions to scale out terminating VNFs on multiple compute nodes attached to a router via multiple 10GE ports. ECMP is used on the router to spread traffic across these VNFs equally.
The routes with multiple next hops (one per operational VNF instance) shall be injected into a router and shared via BGP with neighbors. How hard can it be to achieve this? I immediately had various idea’s on how to solve this challenge: either use netconf/yang to inject static routes into the ephemeral DB in Junos or inject the same via the RIB Services JET API, followed by some route policies that export these static routes with BGP.
Well, turns out, neither of these options actually work, because static routes exported to BGP will only keep a single next hop, despite having the device configured with multiple. Giving up right here isn’t an option, so I expanded my search for other options and found one in using the BGP Route Service JET API directly.
This blog describes a simple proof of concept Python JET app that successfully injects 2 routes with multiple next hops into Junos thru the BGP Route Service API, starting from the downloadable JET IDL files, compiling them for Python and execute the sample application in a Docker container. While one can run all this also directly in Linux, packaging it in a Docker Container makes it usable also on OS/X and Windows.
The sample Python script and Dockerfile to build the supporting libraries can be found in the github repo
https://github.com/mwiget/jet-bgp-static-routes.
Junos configuration
Enable gRPC API in clear-text with basic authentication (username/password) via:
set system services extension-service request-response grpc clear-text port 50051
set routing-options programmable-rpd purge-timeout 120
The purge-timeout defines the purge timeout for all programmable-rpd clients in seconds, when expired, will purge all routes added form a given gRPC client after its connection to the router terminated.
In addition to enabling gRPC on the router, a basic BGP configuration is required:
set routing-options autonomous-system 64512
set protocols bgp group internal type internal
set protocols bgp group internal family inet unicast add-path send path-count 6
set protocols bgp group internal allow 0.0.0.0/0
Build the Container
Download the latest JET client IDL library from Juniper Networks at https://www.juniper.net/support/downloads/?p=jet#sw and place the file in the cloned repo directory:
$ clone https://github.com/mwiget/jet-bgp-static-routes.git
$ cd jet-bgp-static-routes
$ cp ~/Downloads/jet-idl-17.4R1.16.tar.gz .
Now run ‘make’ to build the container:
$ make
docker build -t jet .
Sending build context to Docker daemon 2.226MB
. . .
Successfully built a27b6c83bcd4
Successfully tagged jet:latest
Verify the container image. It contains the python modules built from the JET IDL files, ready to be used. The container isn’t optimised for size, allowing additional Python modules to be added via ‘pip’.
$ docker images |head
REPOSITORY TAG IMAGE ID CREATED SIZE
jet latest a27b6c83bcd4 1 min ago 245MB
Run the Container and routes_bgp.py script
‘make shell’ launches the container jet with the current directory mounted into the container, allowing you direct access to the jroutes_bgp.py script to execute with parameters for the target router, port, username and password.
The script will add routes to 10.123.0.0/24 and 10.123.1.0/24 via the next-hops 10.0.0.1, 10.0.0.2, 10.0.0.3, 10.0.0.4. These are defined via the arrays ‘routes’ and ‘nexthops’:
$ make shell
docker run -ti --rm -v /Users/mwiget/git/jet-bgp-static-routes:/root jet
root@89b73fd45577:~# ./jroutes_bgp.py --help
usage: jroutes_bgp.py [-h] [-t TARGET] [-P PORT] [-u USERNAME] [-p PASSWORD]
optional arguments:
-h, --help show this help message and exit
-t TARGET, --target TARGET
-P PORT, --port PORT
-u USERNAME, --user USERNAME
-p PASSWORD, --pass PASSWORD
root@89b73fd45577:~#
root@89b73fd45577:~# ./jroutes_bgp.py --t 192.168.1.33 --port 50051 --user lab --pass lab123
./jroutes_bgp.py
Trying to Login to 192.168.1.33 port 50051 as user lab ... Login successful
Successfully connected to BGP Route Service
routes successfully added
root@89b73fd45577:~#
If you run the app again within less than 120 seconds (or whatever the devices purge-timeout is set to), you’ll will get error ROUTE_EXISTS(17) back, which means the request contained routes that are already present in the table:
root@89b73fd45577:~# ./jroutes_bgp.py --t 192.168.1.33 --port 50051 --user lab --pass lab123
./jroutes_bgp.py
Trying to Login to 192.168.1.33 port 50051 as user lab ... Login successful
Successfully connected to BGP Route Service
BgpRouteAdd failed with code 17
Check https://www.juniper.net/documentation/en_US/jet17.4/information-products/pathway-pages/bgp_route_service.html
for BgpRouteOperReply.BgpRouteOperStatus description
root@89b73fd45577:~#
A more useful jroutes_bgp.py application will keep the gRPC session alive and add/update/remove routes as required.
Check the route in Junos:
mwiget@vmxdockerlight_vmx1_1> show route 10/8
inet.0: 9 destinations, 15 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
10.123.0.0/24 *[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
10.123.1.0/24 *[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
[BGP-Static/10/-201] 00:00:03, metric2 0
> to 172.21.0.1 via fxp0.0
You might wonder, why the next hops all show 172.21.0.1. This is a result of next hop resolution done by Junos and this system only has one default route via fxp0 (yes, that wouldn’t actually work). Looking at the route details will expose the actual programmed next hops, which will be shared with BGP neighbors:
mwiget@vmxdockerlight_vmx1_1> show route 10/8 detail
inet.0: 9 destinations, 15 routes (9 active, 0 holddown, 0 hidden)
10.123.0.0/24 (4 entries, 1 announced)
*BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xb676590
Next-hop reference count: 5
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.1
Indirect next hop: 0xbdb6980 344 INH Session ID: 0x0
State: <Active Int Ext AlwaysFlash NSR-incapable Programmed>
Age: 3 Metric2: 0
Validation State: unverified
Announcement bits (2): 0-KRT 2-Resolve tree 1
AS path: I
BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xb6757b0
Next-hop reference count: 3
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.2
Indirect next hop: 0xbdb6b00 - INH Session ID: 0x0
State: <Int Ext AlwaysFlash NSR-incapable Programmed>
Inactive reason: Nexthop address
Age: 3 Metric2: 0
Validation State: unverified
AS path: I
BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xcde6370
Next-hop reference count: 3
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.3
Indirect next hop: 0xbdb6500 - INH Session ID: 0x0
State: <Int Ext AlwaysFlash NSR-incapable Programmed>
Inactive reason: Nexthop address
Age: 3 Metric2: 0
Validation State: unverified
AS path: I
BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xcde6190
Next-hop reference count: 3
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.4
Indirect next hop: 0xbdb6680 - INH Session ID: 0x0
State: <Int Ext AlwaysFlash NSR-incapable Programmed>
Inactive reason: Nexthop address
Age: 3 Metric2: 0
Validation State: unverified
AS path: I
10.123.1.0/24 (4 entries, 1 announced)
*BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xb676590
Next-hop reference count: 5
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.1
Indirect next hop: 0xbdb6980 344 INH Session ID: 0x0
State: <Active Int Ext AlwaysFlash NSR-incapable Programmed>
Age: 3 Metric2: 0
Validation State: unverified
Announcement bits (2): 0-KRT 2-Resolve tree 1
AS path: I
BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xb6757b0
Next-hop reference count: 3
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.2
Indirect next hop: 0xbdb6b00 - INH Session ID: 0x0
State: <Int Ext AlwaysFlash NSR-incapable Programmed>
Inactive reason: Nexthop address
Age: 3 Metric2: 0
Validation State: unverified
AS path: I
BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xcde6370
Next-hop reference count: 3
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.3
Indirect next hop: 0xbdb6500 - INH Session ID: 0x0
State: <Int Ext AlwaysFlash NSR-incapable Programmed>
Inactive reason: Nexthop address
Age: 3 Metric2: 0
Validation State: unverified
AS path: I
BGP-Static Preference: 10/-201
Next hop type: Indirect, Next hop index: 0
Address: 0xcde6190
Next-hop reference count: 3
Next hop type: Router, Next hop index: 343
Next hop: 172.21.0.1 via fxp0.0, selected
Session Id: 0x0
Protocol next hop: 10.0.0.4
Indirect next hop: 0xbdb6680 - INH Session ID: 0x0
State: <Int Ext AlwaysFlash NSR-incapable Programmed>
Inactive reason: Nexthop address
Age: 3 Metric2: 0
Validation State: unverified
AS path: I
The BGP Services API also contains functions to Update and Delete routes. Check out https://www.juniper.net/documentation/en_US/jet17.4/information-products/pathway-pages/bgp_route_service.html for more information.