Remote Leaf

1. ACI remote leaf switch deployment allows extension of ACI fabric to remote data centers without a local spine switch or APIC.

   
   

2. Remote leaf switches are connected to an existing pod in the fabric via the Wide Area Network (WAN).

   
   

3. Policies set in the main data center are applied to remote switches, which function like local leaf switches within the fabric.

   
   

4. Unicast traffic is sent through VXLAN over Layer 3.

   
   

5. Layer 2 Broadcast, Unknown unicast, and Multicast (BUM) traffic use Head End Replication (HER) tunnels, without needing Multicast.

   
   

6. Local traffic between endpoints at the remote site is switched directly, whether endpoints are physical or virtual.

   
   

7. Traffic that requires the spine proxy is sent to the main fabric.

   
   

8. Remote leaf switches can connect to virtual servers, physical servers, and containers.

   
   

9. Traffic to endpoints connected to the remote leaf is handled locally by the remote leaf switches.

   
   

10. You need second-generation spines and leafs, such as EX or FX, to use the remote leaf solution.

An ACI Remote Leaf (RL) extends the main ACI fabric to a remote location using Nexus 9300 switches that are fully managed by the primary APIC cluster. Despite being located off-site and connecting over an IP-based WAN (through an IPN or ISN), the remote leaf is logically treated as part of the same fabric POD in the main data center.

Evolution by ACI Version

• Pre-4.x: Required VLAN-4 sub-interfaces for DHCP relay and VLAN-5 sub-interfaces for multi-pod, causing complexity.

• 4.1(2): Introduced support for RL with Multi-Site, which eliminates the need for VLAN-5, and allows direct forwarding between two RL nodes within the same site (traffic no longer must hairpin through a spine).

• 4.2: Adds direct data-plane VXLAN tunnels between RLs (and also between RLs and main-site leaves if endpoints are known).

• 4.2(3): Further enhances Remote Leaf capabilities, enabling headend replication instead of multicast in the WAN, plus routable TEP-based DHCP discovery and simpler RL-to-RL or RL-to-main-leaf tunneling. You can have up to 128 remote leaf switches (64 sites, two RLs per site).

HIGH-LEVEL ARCHITECTURE

Main DC Components

• Spines: Provide a “proxy” function for endpoints not locally known by the remote leaf.

• APIC Cluster: Physically located at the main data center, managing the entire fabric (including remote leaves).

  
  

Remote Leaf Site

• Typically one or two leaf switches (often in a vPC pair) handle local endpoints.

• Connects back to main spines over the WAN or IPN.

• Fully controlled by the same APIC cluster, just in a different physical location.

  
  

Connectivity

• Uses VXLAN data plane over the IPN/WAN.

• No WAN multicast required because ACI uses headend replication for BUM traffic.

REMOTE LEAF DISCOVERY & DHCP PROCESS

1. Out-of-Box

   • The RL interfaces typically use a DHCP-based bootstrap over VLAN-4 (or a routed sub-interface) for connectivity to APIC.

   • The RL sends a DHCP Discover, which the IPN relays to the APIC’s DHCP server in the main data center.

2. DHCP Offer

   • APIC replies with an IP for the RL uplink, plus bootstrap details (e.g., next-server IP, node ID, TEP addresses).

3. Final Steps

   • The RL downloads its bootstrap file, forms an IS-IS adjacency with the spines, and joins the main fabric.

4. Enhancements in ACI 4.x

   • RL can discover APIC via a “routable TEP pool,” so VLAN-4 and NAT are no longer mandatory.

   • The RL is assigned addresses from that routable TEP pool, simplifying WAN connectivity.

KEY CONCEPTS & COMPONENTS

1. Spine Proxy

   • Main DC spines proxy unknown or unflooded traffic for the remote leaf. If the RL has no local entry for an endpoint, it sends the traffic to a spine for resolution.

2. HREP (Headend Replication)

   • Used for Broadcast, Unknown unicast, and Multicast (BUM) traffic.

   • RLs replicate frames to local vPC peers or spines; spines handle replication to other parts of the fabric. No multicast is required on the WAN.

3. COOP Sessions

   • The RL establishes COOP sessions with spines to register endpoint information, ensuring the entire fabric is aware of remote endpoints.

ROUTABLE TEP POOL

Definition: A separate IP address pool from which APIC assigns “routable TEP” addresses to spines, border leaves, or remote leaf nodes. These addresses are publicly (or externally) routable over the WAN/IPN, allowing direct node-to-node VXLAN tunnels.

Benefits

• Reduces or eliminates sub-interfaces and NAT complexity.

• Allows RL pairs to route traffic directly without relying on spines for every path.

• Simplifies multi-site deployments.

Implementation

• Configure a Pod External Routable Subnet on APIC (e.g., 192.154.x.0/24).

• Spines may NAT private TEPs (10.x.x.x) to these routable TEPs (192.154.x.x) if needed.

• The IPN or WAN must carry routes for this routable TEP subnet.

TRAFFIC FLOWS

1. Local RL-to-RL

   • Two RL switches in the same site can forward traffic directly via VXLAN (no spine hairpin).

2. Remote Leaf to Main DC Leaf

   • If the RL knows a destination leaf, it can establish a direct tunnel.

   • For unknown or silent endpoints, the RL forwards traffic to the spine for ARP glean or further proxying.

3. BUM Traffic

   • RL replicates frames toward the spine or vPC peer.

   • The spine replicates traffic back to RL for inbound BUM.

   • No multicast needed in the WAN.

4. Failure Cases

   • ACI 4.x introduces a t_glean process on RLs, letting them initiate ARP glean locally if spines are unreachable, reducing downtime for new endpoint discovery.

CONFIGURATION & VERIFICATION

• Wizard for Remote Leaf (4.1+): APIC provides a setup wizard for specifying remote leaf sites, seed POD, TEP pools, etc.

• Verification

  • show ip interface vrf overlay-1 on RL: Checks assigned IP addresses on L3 interfaces.

  • show dhcp internal event-history traces on RL: Ensures DHCP offers are received.

  • APIC/Spine moquery for natEntry, fabricExtRoutablePodSubnet, etc., to confirm NAT settings and routes.

  • acidiag avread on APIC: Shows assigned addresses, including routableAddress.

Commands :

rleaf# show ip interface vrf overlay-1

IP Interface Status for VRF "overlay-1"

eth1/49.1, Interface status: protocol-up/link-up/admin-up, iod: 59, mode: external

  IP address: 10.10.32.30, IP subnet: 10.10.32.28/30

  IP broadcast address: 255.255.255.255

  IP primary address route-preference: 0, tag: 0

eth1/50.2, Interface status: protocol-up/link-up/admin-up, iod: 60, mode: external

  IP address: 10.10.32.90, IP subnet: 10.10.32.88/30

  IP broadcast address: 255.255.255.255

  IP primary address route-preference: 0, tag: 0

lo0, Interface status: protocol-up/link-up/admin-up, iod: 4, mode: ptep

  IP address: 10.4.2.96, IP subnet: 10.4.2.96/32

  IP broadcast address: 255.255.255.255

  IP primary address route-preference: 0, tag: 0

lo1, Interface status: protocol-up/link-up/admin-up, iod: 65, mode: unspecified

  IP address: 10.10.32.108, IP subnet: 10.10.32.108/32

  IP broadcast address: 255.255.255.255

  IP primary address route-preference: 0, tag: 0

lo2, Interface status: protocol-up/link-up/admin-up, iod: 66, mode: dp-ptep

  IP address: 10.4.2.97, IP subnet: 10.4.2.97/32

  IP broadcast address: 255.255.255.255

  IP primary address route-preference: 0, tag: 0

lo1023, Interface status: protocol-up/link-up/admin-up, iod: 70, mode: ftep

  IP address: 10.4.0.32, IP subnet: 10.4.0.32/32

  IP broadcast address: 255.255.255.255

  IP primary address route-preference: 0, tag: 0

RL# show dhcp internal event-history traces

RL# pwd

/var/run/mgmt

bdsol-aci32-rleaf11# ls -al bootstrap.xml

-rw-rw-rw- 1 root root 42630 Mar  5 15:41 bootstrap.xml

spine# show nattable

-----NAT TABLE---------

Private Ip  Routable Ip

----------  -----------

10.0.0.3    192.154.2.227

10.0.0.2    192.154.0.228

10.0.0.1    192.154.0.227

apic1# avread

APICs:

-------------------------------------------------------------------------

                    APIC 1                  APIC 2                  APIC 3

version           4.2(2f)                 4.2(2f)                 4.2(2f)

address           10.0.0.1                10.0.0.2                10.0.0.3

oobAddress        10.48.25.60/25          10.48.25.61/25          10.48.25.62/25

routableAddress   192.154.0.227           192.154.0.228           192.154.2.227

tepAddress        10.0.0.0/16             10.0.0.0/16             10.0.0.0/16

podId             1                       1                       2

admin@apic1:~> moquery -c fabricExtRoutablePodSubnet | egrep '^pool|^dn|^reserveAddressCount|^state'

pool                 : 192.154.2.0/24

dn                   : uni/controller/setuppol/setupp-2/extrtpodsubnet-[192.154.2.0/24]

reserveAddressCount  : 2

state                : active

pool                 : 192.154.1.0/24

dn                   : uni/controller/setuppol/setupp-3/extrtpodsubnet-[192.154.1.0/24]

reserveAddressCount  : 0

state                : inactive

pool                 : 192.154.0.0/24

dn                   : uni/controller/setuppol/setupp-1/extrtpodsubnet-[192.154.0.0/24]

reserveAddressCount  : 2

state                : active

spine# moquery -c natEntry

Total Objects shown: 3

# nat.Entry

pvtIp        : 10.0.0.3

childAction  :

dn           : sys/nat/entry-10.0.0.3

lcOwn        : local

modTs        : 2020-02-07T02:37:53.816+00:00

rn           : entry-10.0.0.3

routableIp   : 192.154.2.227

status       :

# nat.Entry

pvtIp        : 10.0.0.2

childAction  :

dn           : sys/nat/entry-10.0.0.2

lcOwn        : local

modTs        : 2020-02-07T06:45:24.124+00:00

rn           : entry-10.0.0.2

routableIp   : 192.154.0.228

status       :

# nat.Entry

pvtIp        : 10.0.0.1

childAction  :

dn           : sys/nat/entry-10.0.0.1

lcOwn        : local

modTs        : 2020-02-07T06:51:13.156+00:00

rn           : entry-10.0.0.1

routableIp   : 192.154.0.227

leaf/Spine# show ip route 192.154.0.227 vrf overlay-1

IP Route Table for VRF "overlay-1"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

192.154.0.227/32, ubest/mbest: 1/0

    *via 10.0.0.1, vlan80, [1/0], 01w00d, static

Routing & NAT

• Spines maintain static routes for APIC’s routable IP or the routable TEP subnet, redistributing them into IS-IS or OSPF for the IPN.

• On the RL side, a static route to the APIC’s routable IP may be created to complete discovery if spines are not directly on the path.

MULTI-SITE & MIGRATION

Multi-Site

• RL nodes in 4.1(2)+ can integrate more easily with an MSO-managed multi-site fabric. If traffic needs to reach another site, it flows via the spines of the RL’s “seed” POD.

Migration from Pre-4.x

1. Upgrade APIC/nodes to 4.1(2)+.

2. Define a Pod External Routable Subnet if you want RL Direct mode.

3. Update your WAN (IPN) to carry routes for the new TEP subnet.

4. Enable “Remote Leaf Direct” in fabric-wide settings (irreversible).

5. (Optional) Re-discover the RL so it uses the routable TEP addresses instead of older VLAN-4 sub-interfaces.

A phased approach is possible: keep older sub-interfaces until the WAN is ready, then switch the DHCP relay to APIC’s routable IP.

Adding Routable Subnet's to the POD

Data plane TEP may exist already for Mpod

Go to Infra Tenant ---> Go to Fabric Ext Connection Policy.

Once RL direct is enabled then it can't be disabled again.

As a last step in IPN need to add "ip dhcp relay" as APIC Routable IP & also decommission & recommission followed by remove from controller each Remote Leaf to get it discovered on APIC's Routable IP.

TROUBLESHOOTING TIPS

• Missing DHCP Offer:

  • Verify ip dhcp relay address <APIC-routable-IP> on IPN devices.

  • Check routing toward the APIC’s routable IP.

• RL Discovery Failure:

  • Confirm the RL is placed in the correct VRF (often overlay-1) for WAN connectivity.

  • Use show dhcp internal event-history traces to see if the RL receives any DHCP responses.

• Routing Issues:

  • Inspect spine route tables (show ip route vrf overlay-1) for RL TEP addresses.

  • Make sure OSPF/ISIS is redistributing the subnets to the IPN.

• Silent Host Traffic:

  • If a destination is silent, the RL sends traffic to the spine for ARP glean. In ACI 4.0+, t_glean on the RL can handle certain ARP glean tasks locally if spines are down.

    
    

Lleaf# show ip arp internal event-history event | egrep -B 1 "10.200.2.104"

1795) Event:E_DEBUG_DSF, length:149, at 2020-03-19T10:47:57.701737000+00:00

    [116] TID 23450:arp_handle_inband_glean:3101: log_collect_arp_glean;sip = 10.200.1.111;dip = 10.200.2.104;info = Received glean packet is an IP packet

Failure Handling in Remote Leaf

If all Spines connected to the POD, for which the Remote Leaf (RLeaf) is being configured, go down, traffic from the RLeaf to other Pods and within/across the POD will continue to function. The traffic will be rerouted through an alternate available POD.

Note: This scenario is expected to be rare, as each POD is designed with at least two Spines.

System → Remote Leaf POD Redundancy Policy

• Enable Remote Leaf POD Redundancy Policy

• Enable Remote Leaf POD Redundancy Pre-emption

  
  

Note: Navigate to System → BGP Route Reflector and ensure that the external route reflector node includes a Spine from each POD.

Before Spine failure for POD in which Remote leaf is being configured

bdsol-aci32-rleaf1# show bgp vpnv4 unicast summary vrf overlay-1

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

192.154.0.231   4   132   26631   24335    14860    0    0 00:06:54 197

192.154.0.233   4   132   26808   24364    14860    0    0 00:06:23 196

192.154.0.234   4   132   26952   24362    14860    0    0 00:06:35 197

192.154.1.227   4   132     199      13    14860    0    0 00:01:42 172

After the failover is enabled

bdsol-aci32-rleaf1# show bgp vpnv4 unicast summary  vrf overlay-1

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

192.154.0.231   4   132   26736   24367        0    0    0 00:04:15 Idle

192.154.0.233   4   132   26913   24396        0    0    0 00:04:16 Idle

192.154.0.234   4   132   27078   24396        0    0    0 00:04:15 Idle

192.154.1.227   4   132     402      39    16759    0    0 00:27:59 14

 Coop entry exists on Remote Leaf as well

rleaf# show coop internal info  ip-db key 3014656 10.200.2.102

IP address : 10.200.2.102

Vrf : 3014656

Flags : 0

EP bd vnid : 15990734

EP mac :  00:00:00:01:10:02

Record timestamp : 03 16 2020 08:47:13 552182599

Publish timestamp : 03 16 2020 08:47:13 552452809

Remote publish timestamp: 01 01 1970 00:00:00 0

rleaf# show coop  internal  info repo ep key 15990734 0000.0001.1002

MTS RX OK

Next repo refresh: 2168 seconds 492 ms

Repo Hdr Checksum : 0

..

Repo Hdr flags : IN_OBJ EXPORT ACTIVE

EP bd vnid : 15990734

EP mac :  00:00:00:01:10:02

flags : 0x80

repo flags : 0x122

Vrf vnid : 3014656

Epg vnid : 0

EVPN Seq no : 0

Remote publish timestamp: 01 01 1970 00:00:00 0

Snapshot timestamp: 01 01 1970 00:00:00 0

num of active ipv4 addresses : 1

num of ipv4 addresses : 1

..

Current citizen (publisher_id): 10.0.80.94       Spine that did send it to us

Publisher Oracle (Oracle_id): 10.0.88.64

Tunnel nh : 10.0.80.94

RL Tunnel nh : 192.154.0.2                      Tunnel NH (here anycast Spine RTEP)

Dirty : No

Leaf 0 Info :

..

IPv4 Repo Hdr flags : IN_OBJ EXPORT

Real IPv4 EP : 10.200.2.102

-------------------------------------------------------------------------------------------------------------------------------

The remote leaf solution is supported from the ACI 3.1(1) release.

Topology

These are the configurations used in the IPN device connected to the ACI Spine(s) in the main fabric:

vrf context RLEAF

description VRF created for remote-leaf lab

router ospf 1

vrf RLEAF

router-id 172.16.191.191

area 0.0.0.1 nssa

# In this example same IPN router is used to connect to RLEAF and SPINE

interface loopback191

vrf member RLEAF

ip address 172.16.191.191/32

Interface specific configurations on the IPN that connects to the Spine.

Remote WAN Configuration (RLEAF side)

vrf context RLEAF

description VRF created for remote-leaf lab

router ospf 1

vrf RLEAF

router-id 172.16.191.191

area 0.0.0.1 nssa

# In this example same IPN router is used to connect to RLEAF and SPINE

interface loopback191

vrf member RLEAF

ip address 172.16.191.191/32

Interface specific configurations on the IPN that connects to the RLEAF:

Note: Ensure the dhcp-relay IP is configured with the APIC fabric IP address under the interface connected to the remote-leaf.

This is required for the remote leaf to obtain the bootstrap files from APIC.

Note :

All inter-VRF traffic (pre-release 4.0(1)) goes to the spine switch before being forwarded.

For releases prior to Release 4.1(2), before decommissioning a remote leaf switch, you must first delete the vPC.

ACI Configuration

Step 1. Configure Pod Fabric Setup Policy

1. Navigate to Fabric > Inventory > Pod Fabric Setup Policy.

2. Double click to open Fabric Setup Policy for existing Pod.

3. Add (+) Remote Pool, provide a Remote ID (in this example: 11) and Remote Pool (in this example: 11.0.0.0/20) and click Submit.

Step 2. Configure Routed Outside from Spine to IPN

1. Navigate to Tenant > Infra > External Routed Networks.

2. Right-click and create Routed Outside.

3. Configure OSPF Routed Outside for Spine to IPN.

4. Use OSPF as a routing protocol.

5. Use overlay-1 as VRF.

If you use remote leaf with a multipod fabric, this "Enable remote leaf with Multipod" option must be checked.

Step 3 : Configure the Node profile for each spine connected to IPN

Step 4 : Configure interface profile for Node

Note : Ensure to use encap vlan-4 for remote leaf integration with a single pod.

Step 5 : Configure L3Out Network(External EPG) for IPN

Step 6 : Verification

Now that you've configured OSPF L3Out from Spine to the IPN device.

spine# show ip ospf neighbors vrf overlay-1

IPN# show ip ospf neighbors vrf RLEAF

IPN# show ip show ip route vrf RLEAF

Step 7 : Discover the Remote Leaf(s)

At this stage, the fabric is ready to discover a remote leaf connected to IPN across the WAN. Ensure that the IPN connected to the RLEAF has the route to the ACI pod infra network over the WAN network

RLEAF-IPN# show lldp neighborsCapability codes:  (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device  (W) WLAN Access Point, (P) Repeater, (S) Station, (O) OtherDevice ID            Local Intf      Hold-time  Capability  Port ID

switch               Eth3/34         120        BR           Eth1/54

switch               Eth3/35         120        BR           Eth1/54

RLEAF-IPN# show ip route vrf RLEAF

10.0.0.0/16, ubest/mbest: 2/0   

via 10.10.19.11, Eth3/38.4, [110/20], 00:01:21, ospf-1, nssa type-2   

via 10.10.20.11, Eth3/39.4, [110/20], 00:01:21, ospf-1, nssa type-2

Step 8 : Confirm if IPN got RLEAF is acting as DHCP - Relay

RLEAF-IPN# show ip dhcp relay

Helper addresses are configured on the following interfaces: 

Interface        Relay Address     VRF Name -------------    -------------     -------- 

Ethernet3/34.4    10.0.0.1 

Ethernet3/34.4    10.0.0.2 

Ethernet3/34.4    10.0.0.3 

Ethernet3/35.4    10.0.0.1 

Ethernet3/35.4    10.0.0.2 

Ethernet3/35.4    10.0.0.3

Step 9 : At this stage the RLEAF switches must be discovered in fabric

Inventory > Fabric Membership 

Step 10 : Register RLEAF switches

1. Identify the new leaf based on the serial number.

2. Right-click on the newly discovered leaf and click Register.

3. Provide the right Pod ID and Node ID.

4. Select the RL TEP POOL.

5. Provide a Node Name.

6. Check and Confirm the Role is selected as remote leaf.

7. Click Update.

Note: Ensure to select the correct RL TEP Pool you configured in .Also, check and confirm the Role is selected as a remote leaf automatically when you select the RL TEP POOL from the dropdown.

Now you can see the node type is identified as "remote leaf" and status as "Discovering". The node hasn't got a fabric IP address yet.

Step 11 : Configure Routed OutSide from RLEAF to IPN

Navigate to Tenant > Infra > External Routed Networks and create Routed Outside (L3 Out)

Step 12 : Create RLEAF node profiles for rleaf-203 (Node-203) and rleaf-204(Node-204)

Note: You can not see the Noderleaf-203 (Node-203)or rleaf-204 (Node-204) from theNode dropdown list as the RLEAF203 or RLEAF204 is not registered. So, manually enter the path in Node & Path fields as shown in the image.

Create the interface profile for node-203. Manually enter Node and Path fields as shown.

Node: topology/pod-1/node-203

Path: topology/pod-1/paths-203/pathep-[eth1/54]

Step 13 : Create Fabric External Connection Policy

Navigate to Tenant > Infra > Policies > Protocol > Fabric Ext Connection Policy > Fabric External Connection Policy and create Intrasite/Intersite Profile.

Add Fabric External Routing Profile with an external network of RLEAF203 and RLEAF204 connected to the WAN router (IPN).

In this case, those are 10.10.22.0/24 and 10.10.21.0/24 respectively.

Step 14 : Verify remote leaf obtains the fabric IP address from the APIC TEP pool.

Step 15 : QoS Configuration for Remote Leaf

It is required to classify ACI fabric classes (QoS Levels) to a DSCP value within IPN. To achieve this requirement, ACI Fabric should be enabled with DSCP class-cos translation policy for L3 traffic.

Navigate to Tenant > Infra > Policies > DSCP class-cos translation policy for L3 traffic