Providing disjoint paths in an IP/MPLS network has always been a pain, especially when the disjointness must be achieved between pairs of routers. Cisco XR Traffic Controller is a PCE server allowing path disjointness computation. This post presents how it works and how it must be configured.
Static Path disjointness setup
In the diagram below, we want to provide a pair of disjoint paths to carry the traffic between the two customer sites. We consider LSP1 to be the blue LSP from PE1 to PE3 and LSP2 the red LSP from PE2 to PE4.
The usual shortest path (SPF) computed by the IGP cannot provide the required disjointness due to the metrics that have been configured.
In order to move one of the MPLS LSP out of the shortest path, we need to implement MPLS traffic engineering (through RSVP-TE or Segment Routing TE).
By using the usual distributed CSPF (Constrained SPF) coming with MPLS traffic engineering, there is no way to ensure a real time disjointness (mainly because one PE is not aware of the path used by the other PE). Disjointness may be achieved at LSP setup time by, for instance, adding a constraint on LSP2 which avoids the links PE1-P1, P1-P3 and P3-PE3 (if link disjointness is desired).
The main concern with this network design is that if the topology changes (due to network failures, or any network redesign), the disjointness may not be achieved anymore.
As an example, if the operator wants to change the link metric between P1 and P3 to an higher value like 1000 because he wants to favor the link P1-P4, the LSP1 will be rerouted onto a new path which will not be disjoint anymore from LSP2.
A manual action must be triggered to reconfigure the constraints on LSP2 to avoid the new path of LSP1.
This manual/static disjointness setup is crazy to maintain over time and requires a lot of operational resources. A more dynamic approach is suitable to provide a path disjointness service.
Dynamic disjointness using a Path Computation Element (PCE)
The Path Computation Element brings a central computation unit in the network. By using a stateful PCE, the PCE will keep track of the paths of all LSPs in the network. This LSP database, in addition to the traditional TED (Traffic Engineering Database), will allow the PCE to compute a path for an LSP by taking into account the path of another LSP. The TED provides the topology information and can be populated by routing protocols like ISIS-TE, OSPF-TE or BGP-LS. The LSP database can be populated mainly by PCEP today (BGP-LS is also a solution).
This is exactly what we need for our disjointness use case: the path of LSP2 depends on the path of LSP1.
The stateful PCE will be in charge of updating the LSPs each time the topology will change. This will be done by using PCEP messages.
Using XTC as a simple solution for paths disjointness
Enabling XTC
XTC (XR Traffic Controller) is a Cisco IOS XR PCE server feature (Cisco's blog on XTC) which embeds a disjointness computation algorithm.
Enabling XTC feature is really easy and requires a single CLI command:
RP/0/RP0/CPU0:XR_PCE1#conf t
Fri May 12 07:39:31.136 UTC
RP/0/RP0/CPU0:XR_PCE1(config)#pce
RP/0/RP0/CPU0:XR_PCE1(config-pce)#address ipv4 99.0.0.97
RP/0/RP0/CPU0:XR_PCE1(config-pce)#commit
The IP referenced in the configuration should be a local address of the router on which it will listen for PCEP sessions.
PCEP sessions state can be checked using the following command:
RP/0/RP0/CPU0:XR_PCE1#show pce ipv4 peer
Fri May 12 07:42:07.961 UTC
PCE's peer database:
--------------------
Peer address: 192.168.217.26
State: Up
Capabilities: Stateful, Update
Peer address: 192.168.217.32
State: Up
Capabilities: Stateful, Update
Peer address: 10.200.2.77
State: Up
Capabilities: Stateful, Segment-Routing, Update
The capabilities of each peer is recorded. XTC supports PCEP extensions for Segment Routing.
Topology acquisition on XTC
XTC retrieves the network topology through BGP-LS.
The first way of learning the topology is to peer with a BGP-LS speaker using the following sample configuration:
router bgp 65000
address-family link-state link-state
!
neighbor 99.0.0.1
remote-as 65000
update-source Loopback1
address-family link-state link-state
route-policy ACCEPT in
route-policy ACCEPT out
!
!
!
RP/0/RP0/CPU0:XR_PCE1#show bgp link-state link-state summary
Wed May 17 07:26:49.122 UTC
BGP router identifier 99.0.0.97, local AS number 65000
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 160089
BGP main routing table version 160089
BGP NSR Initial initsync version 84 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 160089 160089 160089 160089 160089 0
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
99.0.0.1 0 65000 207017 238245 160089 0 0 1w0d 12
The second way is for XTC to have an IGP adjacency with a network node and let the IGP distribute its topology in BGP-LS locally. In that case, there is no need to configure a "router bgp", the distribution to BGP-LS in the IGP is enough:
RP/0/RP0/CPU0:XR_PCE1#sh run router isis
Wed May 17 07:28:42.451 UTC
router isis RAEI
is-type level-2-only
net 49.0001.0000.0000.9997.00
distribute link-state throttle 1
The "throttle" parameters allows to batch IGP updates into BGP-LS. In the configuration displayed above, the updates are batched every 1 second.
The command "show pce ipv4 topology" allows to display the topology as seen by XTC:
RP/0/RP0/CPU0:XR_PCE1#show pce ipv4 topology
Wed May 17 07:30:18.522 UTC
PCE's topology database:
------------------------
Node 1
TE router ID: 99.99.99.6
Host name: SLI6
ISIS system ID: 0000.0000.0006 level-2
Prefix SID:
Prefix 99.99.99.6, label 800050 (regular), flags: N
SRGB INFO:
ISIS system ID: 0000.0000.0006 level-2 SRGB Start: 800000 Size: 1024
Link[0]: local address 10.98.0.2, remote address 10.98.0.1
Local node:
ISIS system ID: 0000.0000.0006 level-2
Remote node:
TE router ID: 192.168.217.32
Host name: SLI1
ISIS system ID: 0000.c237.0967 level-2
Metric: IGP 10000, TE 10000
Bandwidth: Total link 0, Reservable 0
Adj SID: 299856 (unprotected)
Node 2
TE router ID: 192.168.217.32
Host name: SLI1
ISIS system ID: 0000.c237.0967 level-2
Prefix SID:
Prefix 192.168.217.32, label 800005 (regular), flags: N
SRGB INFO:
ISIS system ID: 0000.c237.0967 level-2 SRGB Start: 800000 Size: 10240
Link[0]: local address 10.98.0.1, remote address 10.98.0.2
Local node:
ISIS system ID: 0000.c237.0967 level-2
Remote node:
TE router ID: 99.99.99.6
Host name: SLI6
ISIS system ID: 0000.0000.0006 level-2
Metric: IGP 10000, TE 10000
Bandwidth: Total link 0, Reservable 0
Adj SID: 655827 (unprotected)
Link[1]: local address 54.0.0.6, remote address 54.0.0.5
Local node:
ISIS system ID: 0000.c237.0967 level-2
Remote node:
TE router ID: 99.99.99.102
Host name: P2
ISIS system ID: 0000.0000.0102 level-2
Metric: IGP 1200, TE 1200
Bandwidth: Total link 125000000, Reservable 125000000
Adj SID: 760739 (unprotected) 760755 (unprotected)
Configuring the disjointness constraint on XTC
PCEP does not have yet a standard object to carry the disjointness constraint. An IETF draft is in progress but is not finished yet draft-ietf-pce-association-diversity.
As a consequence, the only way we have today to add the disjointness constraint is to update the LSP attributes in XTC. This could be achieved by creating a disjoint-path group-id in XTC.
XR_PCE1(config)#pce
XR_PCE1(config-pce)#disjoint-path group-id 400 type link
XR_PCE1(config-pce-path-group)#lsp 1 pcc ipv4 10.200.2.77 lsp-name PE1_PE3
XR_PCE1(config-pce-path-group)#lsp 2 pcc ipv4 192.168.217.32 lsp-name PE2_PE4
XR_PCE1(config-pce-path-group)#commit
The configuration statements are the following:
- The group-id which is a unique identifier of the disjoint-group.
- The disjointness type: XTC supports link, node, srlg and srlg-node.
- A reference to the first LSP by setting the LSP name as well as the PCC address.
- A reference to the second LSP by setting the LSP name as well as the PCC address.
The disjointness status of the group can be checked using the following command:
RP/0/RP0/CPU0:XR_PCE1#show pce association group-id 400
Fri May 12 07:53:41.041 UTC
PCE's association database:
----------------------
Association: Type Link-Disjoint, Group 400, Not Strict
Associated LSPs:
LSP[0]:
PCC 10.200.2.77, tunnel name PE1_PE3, PLSP ID 81, tunnel ID 1261, LSP ID 84, Configured on PCE
LSP[1]:
PCC 192.168.217.32, tunnel name PE2_PE4, PLSP ID 2, tunnel ID 4640, LSP ID 712, Configured on PCE
Status: Satisfied
In the example above, XTC displays that the disjointness requirement has been satisfied.
The detail of the path can be checked using the following command:
RP/0/RP0/CPU0:XR_PCE1#show pce lsp name PE2_PE4 detail
Fri May 12 07:55:02.044 UTC
PCE's tunnel database:
----------------------
PCC 192.168.217.32:
Tunnel Name: PE2_PE4
LSPs:
LSP[0]:
source 192.168.217.32, destination 192.168.217.15, tunnel ID 4640, LSP ID 712
State: Admin up, Operation active
Binding SID: 0
Bandwidth: signaled 0
Affinity: exclude-any 0x0 include-any 0x0 include-all 0x0
Priority: setup 7 hold 0
Local Protection: no
PCEP information:
plsp-id 2, flags: D:1 S:0 R:0 A:1 O:2
LSP Role: Disjoint LSP
State-sync PCE: None
PCC: 192.168.217.32
LSP is subdelegated to: None
Reported path:
Metric type: TE, Accumulated Metric 1780
54.0.0.5
192.168.218.185
192.168.218.194
Computed path: (Local PCE)
Computed Time: Fri May 12 07:53:02 2017 (00:02:00 ago)
Metric type: TE, Accumulated Metric 1780
54.0.0.5
192.168.218.185
192.168.218.194
Recorded path:
IPv4 54.0.0.5, flags 0x0
IPv4 192.168.218.185, flags 0x0
IPv4 192.168.218.194, flags 0x0
Disjoint Group Information:
Type Link-Disjoint, Group 400
This command is really informative and displays the content of the LSP database. It contains:
- The characteristics of the LSP (head end, tail end, bandwidth...).
- The delegation status (D flag).
- The admin and operation status (A and O flag).
- The ERO (Explicit Route Object) reported by the PCC (Reported path).
- The path computed by the PCE and when it was computed (Computed path).
- The path currently used by the PCC (Recorded path).
XTC provides a buffer which contains the events of a particular LSP. It can be obtained using the "private" keyword of the "show pce lsp" command instead of "detail".
Event history (latest first):
Time Event
May 12 2017 07:53:02 Path Computation (Disjoint LSP)
Symbolic-name: PE2_PE4, LSP-ID: 712, D:1
Source: 192.168.217.32 Destination: 192.168.217.15
Status: Disjoint Path Success
Computed Path: (Metric 1780)
54.0.0.5
192.168.218.185
192.168.218.194
May 12 2017 07:53:00 Report from 192.168.217.32 (LSP owner)
Symbolic-name: PE2_PE4, LSP-ID: 712,
Source: 192.168.217.32 Destination: 192.168.217.15,
D:1, R:0, A:1 O:2, Sig.BW: 0, Act.BW: 0
Reported Path: (Metric 1780)
54.0.0.5
192.168.218.185
192.168.218.194
Fine tuning the disjointness
The disjointness computation can be tuned using two parameters in addition to the disjointness type we already discussed.
- Shortest-path
By default, XTC will provide a pair of disjointness path and will ensure that the sum of the two paths will be minimum. This kind of computation may not be optimal in some situation where a customer absolutely wants the shortest available path for its primary LSP (providing the lowest latency).
This behavior can be tuned by setting the "shortest-path" keyword on the "lsp 1" which is considered as the primary LSP.
PCE(config)#pce
PCE(config-pce)# disjoint-path
PCE(config-pce-path)# group-id 400 type link
PCE(config-pce-path-group)#lsp 1 pcc ipv4 10.200.2.77 lsp-name PE1_PE3 shortest-path
PCE(config-pce-path-group)#commit
By using this configuration, XTC will compute the shortest path for LSP 1 and will ensure that LSP 2 is disjoint.
This "shortest-path" keyword adds a new constraint on the disjointness computation which may prevent the possibility to find a disjoint path if the topology does not allow for it.
- Strictness
- SRLG-NODE to NODE to LINK to SPT
- NODE to LINK to SPT
- SRLG to LINK to SPT
- LINK to SPT
This means that if "srlg-node" disjointness is requested but XTC cannot find an "srlg-node" disjoint path, it will try to find a "node" disjoint path, and if it fails it will try "link" disjointness and finally will place the LSP on the shortest path.
The command "show pce association" provides the status of the disjointness provided in case of fallback:
RP/0/RP0/CPU0:XR_PCE1#show pce association
Fri May 12 08:16:53.556 UTC
PCE's association database:
----------------------
Association: Type SRLG-NODE-Disjoint, Group 400, Not Strict
Associated LSPs:
LSP[0]:
PCC 10.200.2.77, tunnel name PE1_PE3, PLSP ID 81, tunnel ID 1261, LSP ID 84, Configured on PCE
LSP[1]:
PCC 192.168.217.32, tunnel name PE2_PE4, PLSP ID 2, tunnel ID 4640, LSP ID 714, Configured on PCE
Status: Fallback (SRLG-NODE -> LINK)
XTC can be configured to generate a log when no path can be found for an LSP or when a fallback has been done:
RP/0/RP0/CPU0:XR_PCE1#conf t
Fri May 12 08:18:57.357 UTC
RP/0/RP0/CPU0:XR_PCE1(config)#pce
RP/0/RP0/CPU0:XR_PCE1(config-pce)#logging
RP/0/RP0/CPU0:XR_PCE1(config-pce-logging)#no-path
RP/0/RP0/CPU0:XR_PCE1(config-pce-logging)#fallback
RP/0/RP0/CPU0:XR_PCE1(config-pce-logging)#commit
RP/0/RP0/CPU0:May 12 08:16:52.315 : pce_server[1106]: %OS-PCE-7-CSPF_FALLBACK : PCC: 192.168.217.32, LSP-NAME: PE2_PE4, SRC: 192.168.217.32, DST:192.168.217.15, SRLG-Node-disjoint and Node-disjoint paths could not be found. Fallback to link-disjoint .
To ensure a strict disjointness at the price of having no path if the topology does not provide a disjoint path, the keyword "strict" could be added to the disjoint group:
RP/0/RP0/CPU0:XR_PCE1(config-pce)#disjoint-path group-id 400 type srlg-node
RP/0/RP0/CPU0:XR_PCE1(config-pce-path-group)#strict
RP/0/RP0/CPU0:XR_PCE1(config-pce-path-group)#commit
RP/0/RP0/CPU0:XR_PCE1#show pce association
Fri May 12 08:21:10.124 UTC
PCE's association database:
----------------------
Association: Type SRLG-NODE-Disjoint, Group 400, Strict
Associated LSPs:
LSP[0]:
PCC 10.200.2.77, tunnel name PE1_PE3, PLSP ID 81, tunnel ID 1261, LSP ID 84, Configured on PCE
LSP[1]:
PCC 192.168.217.32, tunnel name PE2_PE4, PLSP ID 2, tunnel ID 4640, LSP ID 714, Configured on PCE
Status: No path
XTC redundancy
Deploying a single PCE in a network is dangerous as if the PCE fails no more central computation can be achieved.
A resilient design can be done by deploying two XTC instances. Each PCC will have a PCEP session towards both XTC instances: a single XTC instance will be owner of the LSP as described in the stateful PCEP extensions. Each XTC instance will maintain a TED by using BGP-LS or the IGP.
To ensure a synchronization between the two XTC instances, a special PCEP session called "state-sync" must be configured between the two PCEs. This session will ensure an LSP database synchronization between the PCEs. And in addition, it will ensure a synchronization of the path computation between the instances: one PCE will be master for the computation while the other will be a slave.
The state-sync session must be configured on each XTC instance:
RP/0/RP0/CPU0:XR_PCE1(config)#pce state-sync ipv4 99.0.0.96
RP/0/RP0/CPU0:XR_PCE2(config)#pce state-sync ipv4 99.0.0.97
The IP address referenced in the command is the address of the remote PCE.
The "show pce lsp detail" command provides informations how the state-sync affects the path computation:
RP/0/RP0/CPU0:XR_PCE1#show pce lsp name PE1_PE3 detail
Fri May 12 08:36:25.500 UTC
PCE's tunnel database:
----------------------
PCC 10.200.2.77:
Tunnel Name: PE1_PE3
LSPs:
LSP[0]:
source 10.200.2.77, destination 192.168.217.14, tunnel ID 1261, LSP ID 84
State: Admin up, Operation active
Binding SID: 0
Bandwidth: signaled 0
Affinity: exclude-any 0x0 include-any 0x0 include-all 0x0
Priority: setup 7 hold 0
Local Protection: no
PCEP information:
plsp-id 81, flags: D:1 S:0 R:0 A:1 O:2
LSP Role: Disjoint LSP
State-sync PCE: None
PCC: 10.200.2.77
LSP is subdelegated to: 99.0.0.96
Reported path:
Metric type: TE, Accumulated Metric 1060
54.0.0.1
54.0.0.9
10.168.218.249
10.0.99.41
42.42.42.1
Computed path: (Slave PCE)
None
Computed Time: Fri May 12 08:32:07 2017 (00:04:18 ago)
Recorded path:
IPv4 54.0.0.1, flags 0x0
IPv4 54.0.0.9, flags 0x0
IPv4 10.168.218.249, flags 0x0
IPv4 10.0.99.41, flags 0x0
IPv4 42.42.42.1, flags 0x0
Disjoint Group Information:
Type SRLG-NODE-Disjoint, Group 400
RP/0/RP0/CPU0:XR_PCE2#show pce lsp name PE1_PE3 detail
Fri May 12 08:39:33.643 UTC
PCE's tunnel database:
----------------------
PCC 10.200.2.77:
Tunnel Name: PE1_PE3
LSPs:
LSP[0]:
source 10.200.2.77, destination 192.168.217.14, tunnel ID 1261, LSP ID 84
State: Admin up, Operation active
Binding SID: 0
Bandwidth: signaled 0
Affinity: exclude-any 0x0 include-any 0x0 include-all 0x0
Priority: setup 7 hold 0
Local Protection: no
PCEP information:
plsp-id 81, flags: D:0 S:1 R:0 A:1 O:2
LSP Role: Exclude LSP
State-sync PCE: 99.0.0.97
PCC: 10.200.2.77
LSP is subdelegated to: None
Reported path:
Metric type: TE, Accumulated Metric 1060
54.0.0.1
54.0.0.9
10.168.218.249
10.0.99.41
42.42.42.1
Computed path: (Subdelegated Master PCE)
Computed Time: Fri May 12 08:39:17 2017 (00:00:16 ago)
Metric type: TE, Accumulated Metric 1060
54.0.0.1
54.0.0.9
10.168.218.249
10.0.99.41
42.42.42.1
Recorded path:
IPv4 54.0.0.1, flags 0x0
IPv4 54.0.0.9, flags 0x0
IPv4 10.168.218.249, flags 0x0
IPv4 10.0.99.41, flags 0x0
IPv4 42.42.42.1, flags 0x0
Disjoint Group Information:
Type SRLG-NODE-Disjoint, Group 400
In the example above, PE1 has delegated its LSP to PCE1 but the PCE1 is considered as the slave PCE as per the state-sync election mechanism. PCE1 subdelegates the LSP to the PCE2 which is considered as the master PCE. Thus PCE2 becomes responsible of the path computation for all the LSPs in the disjoint group.
In the output above, we can see that the LSP PE1_PE3 has been learned by PCE2 from the PCC (10.200.2.77) as well as from the state-sync session with 99.0.0.97.
The state-sync mechanism is under standardization through the IETF draft draft-litkowski-pce-state-sync.
As mentioned previously, the disjointness constraint is not carried in PCEP yet. As a consequence both XTC instances must have the same disjoint group configuration to ensure the right computation.
This may change in future when the PCEP extension will be standardized and available in XTC.
Deploying XTC
As XTC is an IOS XR feature, it can run on any XR node already available in the network. To prevent a mix of functions, it may be desirable to deploy separate nodes as PCEs.
This can be achieved by deploying new hardware platforms like ASR9k, or by using IOS-XRv running on an x86 server.
Running XTC as a VM allows for a very fast deployment and is really suitable as it is a pure control plane function: a XEON based x86 server will provide more computation power than any Route Processor available in routers hardware today.
