Floating L3Out Version Support
Floating L3Out Introduction:
Started in ACI Version 4.2(1): The floating L3Out feature became available starting with Cisco ACI version 4.2(1) for VMware vSphere Distributed Switch (VDS) in Virtual Machine Manager (VMM) domains.
Expanded in ACI Version 5.0(1): From version 5.0(1), the support for floating L3Out was extended to physical domains. This allows using floating L3Out with virtual routers without needing VMM domain integration, or with physical routers without complex interface path configurations.
Configuration Simplification:
Previous Setup Requirements: Before version 4.2(1), when connecting to an external virtual router, you had to configure the L3Out logical interface path from each border leaf switch to the hypervisor's uplink where the virtual device was running. If the virtual machines moved to different hosts within a hypervisor cluster, you had to manually configure the L3Out on all potential leaf switches where the virtual machine could run.
Problem Example: For instance, if a hypervisor cluster has 12 leaf switches, you had to create L3Out configurations for every leaf switch to ensure connectivity as virtual machines moved across the cluster.
Floating L3Out Benefits:
Simplified Configuration: With floating L3Out, the process is much easier. After setting it up, you no longer need to manually configure the L3Out logical interfaces on all the leaf switches in the cluster.
Automatic Handling: This feature automatically manages the connectivity, even as virtual machines move between different leaf switches, removing the need for complex and time-consuming manual configurations.
Example to use the Floating Layer 3 Outside Network Connection (Floating L3Out) feature in Cisco ACI.
The example includes a Virtual Machine Manager (VMM) domain and a Virtual Port Channel (vPC) setup. However, Floating L3Out also works with physical domains, and using vPC is optional.
Key Components
Virtual Routers:
Can be a regular router, a virtual firewall, or any virtual device.
Acts as the next hop for static routes in the Cisco ACI fabric.
Establishes a routing connection with the ACI fabric.
Anchor Leaf Nodes:
Example Setup: Two leaf switches named Leaf1 and Leaf2.
Role:
Connect directly with external routers.
Establish Layer 3 (L3) routing connections with these external routers.
Scalability:
Up to 6 anchor leaf nodes are supported per L3Out in Cisco ACI release 6.0(1).
IP Addresses:
Use primary and floating IP addresses.
Can also have secondary and floating secondary IP addresses if needed.
Non-Anchor Leaf Nodes:
Example Setup: Two leaf switches named Leaf3 and Leaf4.
Role:
Do not connect directly with external routers.
Act as "pass-through" nodes for traffic between external routers and anchor leaf nodes.
Scalability:
Up to 32 non-anchor leaf nodes are supported per L3Out in Cisco ACI release 6.0(1).
IP Addresses:
Use a common floating IP address.
Can have floating secondary IP addresses if needed.
In VMM domains with VMware vDS:
Floating IP is used only when the virtual router is connected to the leaf node.
In physical domains:
Floating IP is used if the leaf port uses an Attachment Endpoint Profile (AEP) associated with the Floating L3Out.
How Floating L3Out Works
Creating L3Out Bridge Domain:
When you set up an L3Out, an L3Out bridge domain is created on the anchor leaf nodes.
This bridge domain is often called the “L3Out’s SVI subnet.”
Floating L3Out with VMM Domain:
If a virtual router moves to a host connected to a non-anchor leaf switch:
Cisco APIC automatically deploys the L3Out bridge domain on the non-anchor leaf switch.
Installs the floating IP address (and floating secondary IPs if needed) on the non-anchor leaf switch.
If an external Endpoint Group (EPG) under the L3Out has a contract with another EPG:
Routes to the EPG and policy rules for the contract are also installed on the non-anchor leaf switch.
Maintaining Routing Adjacencies:
Even if the virtual router moves, it maintains routing connections with the SVI interfaces on the anchor leaf nodes.
This is possible because ACI extends connectivity for the L3Out bridge domain across both anchor and non-anchor leaf nodes.
Example Scenario
Setup:
A hypervisor cluster is connected across 12 leaf switches.
Virtual machines (VMs) can move between any of these 12 leaf switches.
Without Floating L3Out:
You would need to configure L3Out on all 12 leaf switches.
This ensures connectivity as VMs move, but it’s time-consuming and complex.
With Floating L3Out:
Configure Floating L3Out once.
No need to set up L3Out on each of the 12 leaf switches.
APIC automatically manages connectivity as VMs move between leaf switches.
Simplifies the setup and reduces configuration effort.
Benefits of Floating L3Out
Simplified Configuration:
Reduces the need to manually configure multiple L3Outs on various leaf switches.
Scalability:
Easily supports large numbers of leaf switches without additional configuration.
Flexibility:
Works with both virtual and physical domains.
Supports dynamic movement of virtual routers without disrupting connectivity.
Efficient Resource Use:
Shares IP addresses among non-anchor leaf nodes, optimising IP usage.
Types of IP Addresses Used
Primary IP Address:
What It Is: A unique IP address assigned to the Switched Virtual Interface (SVI) on each leaf node that's part of the L3Out.
Purpose: On anchor leaf nodes, it's required to establish routing adjacencies with external routers.
Leaf1 (Anchor Leaf Node): 172.16.1.251
Leaf2 (Anchor Leaf Node): 172.16.1.252
These are unique to each anchor leaf node and used for establishing routing adjacencies with external routers.
Secondary IP Address (Optional):
What It Is: An additional IP address assigned to the SVI interface on anchor leaf nodes.
Uses:
Shared Virtual IP:
Acts as a common IP shared by anchor leaf nodes, functioning as a virtual IP.
External devices using static routing set their next-hop gateway to this IP.
Additional Subnets:
Provides unique IP addresses per anchor leaf node for an extra IP subnet beyond the primary one.
Shared Secondary IP for Static Routing:
A common IP for the secondary subnet, used similarly to the primary subnet in static routing setups.
Shared by Leaf1 and Leaf2: 172.16.1.254
Acts as a virtual IP shared by anchor leaf nodes.
External devices use this IP as the next-hop gateway in static routes to reach the internal subnet 192.168.1.0/24 inside the ACI fabric.
Floating (Primary) IP Address:
What It Is: An IP address programmed on both anchor and non-anchor leaf nodes.
Purpose:
Configures a Layer 3 interface on the floating SVI.
Programs the same MAC address on all anchor and non-anchor switches.
Allows switches to directly forward traffic from external routers into the fabric.
Used for ARP (Address Resolution Protocol) resolution from an anchor leaf node.
Floating (Primary) IP Address:
Used by Both Anchor and Non-Anchor Leaf Nodes: 172.16.1.250/24
Used for ARP resolution and allows direct forwarding of traffic into the fabric.
Adding Additional Subnets (If Needed):
Secondary IP Address: 172.16.2.254
Floating Secondary IP Address (Optional):
What It Is: A shared IP address on anchor and non-anchor leaf nodes.
Purpose:
Used when multiple subnets exist within the same external bridge domain (SVI).
Not intended for external communication.
Floating Secondary IP Address: 172.16.2.250
These can be added under the same floating SVI if another subnet is required using the same SVI VLAN encapsulation.
Non-anchor leaf nodes use the floating (primary) IP address and share the same MAC address as the next-hop IP address used in static routes (which is 172.16.1.254 in the example).
What This Means:
Because they have the same IP and MAC address setup, non-anchor leaf nodes can directly route traffic without needing to pass it through anchor leaf nodes.
Even If the External Router Moves:
If the external router connects to a non-anchor leaf node instead of an anchor leaf node, the non-anchor leaf node can still handle the traffic directly.
Result:
Traffic flows efficiently because non-anchor leaf nodes can route it themselves, thanks to the shared floating IP and MAC address configuration.
1. Traffic Flow Before the Virtual Router Moves
Connection Setup:
External Routers are connected to Anchor Leaf Nodes (e.g., Leaf1 and Leaf2).
Virtual Routers are initially connected to these anchor leaf nodes.
External to Internal Traffic (L3Out-to-Web):
Entry Point: Traffic from external routers enters the network through an Anchor Leaf Node.
Spine Switch: The traffic is sent to the Spine Switch.
Destination: From the spine switch, traffic reaches the Web Endpoint located on Host 4.
Return Traffic (Web-to-L3Out):
Origin: Traffic originates from the Web Endpoint on Host 4.
Spine Switch: It travels back through the Spine Switch.
Anchor Leaf Node: The traffic reaches the Anchor Leaf Node.
External Router: Finally, it is sent back to the Virtual Router and then to the External Routers.
2. Traffic Flow After the Virtual Router Moves to a Non-Anchor Leaf Node
Virtual Router Relocation:
The Virtual Router moves from an Anchor Leaf Node to Host 3, which is connected to a Non-Anchor Leaf Node (Leaf3).
External to Internal Traffic (L3Out-to-Web):
Entry Point: Traffic from external routers now enters the network through Leaf3 (a Non-Anchor Leaf Node).
Spine Switch: The traffic is sent to the Spine Switch.
Destination: From the spine switch, traffic reaches the Web Endpoint on Host 4.
Return Traffic (Web-to-L3Out):
Origin: Traffic originates from the Web Endpoint on Host 4.
Spine Switch: It travels back through the Spine Switch.
Anchor Leaf Node: The traffic reaches an Anchor Leaf Node.
Non-Anchor Leaf Node: The anchor leaf node forwards the traffic to Leaf3 (the Non-Anchor Leaf Node).
External Router: Finally, it is sent back to the Virtual Router and then to the External Routers.
3. Why This Happens
Route Learning and Redistribution:
Anchor Leaf Nodes learn the external routes through the Virtual Router.
They redistribute these routes to all other leaf nodes in the network.
Seamless Traffic Routing:
Even after the Virtual Router moves to a Non-Anchor Leaf Node, the Anchor Leaf Nodes ensure that traffic knows how to reach the new location.
This allows traffic to flow correctly through the network without manual reconfiguration.
Points to Remember
Anchor Leaf Nodes:
Initially handle both incoming and outgoing traffic.
Learn and distribute external routes to the entire network.
Non-Anchor Leaf Nodes:
Handle traffic when the Virtual Router is moved to them.
Use floating IP addresses to route traffic efficiently.
Floating L3Out Feature:
Ensures that traffic flows seamlessly, even when the Virtual Router moves between different leaf nodes.
Automatically updates routing information to maintain connectivity.
Before the Move:
Traffic flows through Anchor Leaf Nodes to reach internal hosts.
After the Move:
Traffic enters through a Non-Anchor Leaf Node where the Virtual Router resides.
Anchor Leaf Nodes still play a role in routing by learning and redistributing routes.
Overall Benefit:
The Floating L3Out feature provides flexibility and resilience, allowing the network to adapt dynamically to changes without disrupting traffic flow.
Steps to Configure Floating L3Out in Cisco ACI:
1. Create Two L3Outs:
You need two separate L3Out configurations, each connected to a different set of leaf switches.
L3Out 1 (Primary)
Go to Tenants -> Tenant_name -> Networking -> External Routed Networks -> Create L3Out.
Name it, for example, Primary_L3Out.
Select the External Routed Domain.
L3Out 2 (Secondary)
Repeat the same process to create the second L3Out, name it, for example, Secondary_L3Out.
Select the External Routed Domain.
2. Configure Interface Profiles for Both L3Outs:
Each L3Out should have its own interface profiles.
Primary L3Out Interface Configuration:
In the Primary_L3Out -> Logical Node Profiles, select a leaf switch (e.g., Leaf 101).
Under Logical Interface Profiles, configure a Layer 3 interface for the primary path.
Set the IP address and subnet mask for the interface.
Secondary L3Out Interface Configuration:
In the Secondary_L3Out -> Logical Node Profiles, select another leaf switch (e.g., Leaf 102).
Under Logical Interface Profiles, configure a Layer 3 interface for the secondary path.
Set a different IP address and subnet mask for this secondary interface.
3. Configure Routing Peering for Both L3Outs:
You need to configure routing peers for both L3Outs.
BGP (or OSPF) Peering for Primary L3Out:
In Primary_L3Out -> Logical Node Profile -> BGP Peer Connectivity, configure BGP with your primary external router (Neighbor IP, ASN, etc.).
BGP (or OSPF) Peering for Secondary L3Out:
In Secondary_L3Out -> Logical Node Profile -> BGP Peer Connectivity, configure BGP with your secondary external router.
Alternatively, configure OSPF in both L3Outs if using dynamic routing.
4. Configure VRF Policies:
Both L3Outs must share the same VRF (Virtual Routing and Forwarding).
Go to Tenants -> VRF and ensure both Primary_L3Out and Secondary_L3Out are associated with the same VRF.
This allows both L3Outs to handle traffic for the same tenant/network.
5. Create a Contract and Apply to L3Out EPGs:
Create a contract to control traffic between the external network and the internal ACI network.
Go to Tenant -> Primary_L3Out -> External EPG and apply the contract.
Do the same for Secondary_L3Out -> External EPG.
6. Configure Floating IP for Redundancy (Optional):
You can use a floating IP address if you want traffic to failover between the two L3Outs.
Configure the same IP address on both the primary and secondary external routers and apply tracking or any first-hop redundancy protocol (e.g., VRRP or HSRP) to make it "floating" between the two L3Outs.
7. Advertise Subnets:
Advertise the same subnet from both L3Outs for reachability.
Advertise Subnet in Primary L3Out:
Go to Primary_L3Out -> Subnets, add the subnet you want to advertise, and set it to Advertised External.
Advertise Subnet in Secondary L3Out:
Go to Secondary_L3Out -> Subnets, add the same subnet you want to advertise and set it to Advertised External.
8. Configure Floating Preference (Optional):
In scenarios where you want the Primary_L3Out to be preferred over the Secondary_L3Out, you can configure a preference or cost metric in your routing protocol (BGP, OSPF, etc.).
For example:
In Primary_L3Out, configure the BGP AS path prepend to be shorter or the OSPF cost to be lower.
In Secondary_L3Out, configure a longer AS path or a higher OSPF cost, making it a less preferred path.
9. Verification:
Check the routes in the ACI fabric using show ip route.
Verify BGP or OSPF neighbors using:
show bgp neighbors
show ospf neighbor
Diagram for Floating L3Out Setup:
| External Router 1 |
+------------------------+
|
(Primary L3Out)
|
+-----------------+
| Leaf Switch 101 |
+-----------------+
|
(ACI Fabric)
|
+-----------------+
| Leaf Switch 102 |
+-----------------+
|
(Secondary L3Out)
|
+------------------------+
| External Router 2 |
+------------------------+
In this configuration:
Traffic will normally flow through the Primary L3Out, and in case of failure, traffic will automatically shift to the Secondary L3Out.
Both L3Outs are part of the same VRF, ensuring seamless failover.
Comments