Topology
Spines can connect to routers for the Inter-pod/inter-spine network for both Multi-Pod and Multi-site. Spines can also directly connect to another Site's spines in the case of Multisite.
Only 3 APICs can participate in a cluster at a time. You may configure more than three but they will be on standby.
vPC Peering requires an identical switch model.
The Nexus 9300 FX platform supports FC and FCoE and EX do not support it. Besides that, EX does not support MACsec.
Fex requires a license in case required to use in ACI Fabric.
Hardware
APIC Controller
Leaf Switch
Spine Switch
Portchannel & vPC should be assigned to separate interface policy group.
Bridge Domain is not Vlan as it may contain more than one subnet.
Vlan is only locally significant.
Vlan tag is only used for classifying traffic into EPG.
Vlan does not dictate communication between two endpoints.
Application Profile is a group of EPGs working together for the same application.
Example: If there is web application which fetches data from database server then both these servers will be in same Application Profile.
ACI Underlay & Overlay
ACI Physical Topology
Physical Layer Construct
Packet Flow
Let's imagine simple topology of 2leaf/2spine with Host A attached to Leaf1 and with HostB attached to Leaf2.
Assume we chose infrastructure vlan 3999 and associated subnet 10.0.1.0/24
Leaf1 has a VTEP address of 10.0.1.101
Leaf2 has a VTEP address of 10.0.1.102
Spine1 has a VTEP address of 10.0.1.201
Spine2 has a VTEP address of 10.0.1.202
IS-IS
The leaves and spines will exchange IS-IS routing updates with each other so that Leaf1 sees that it has two equally good paths to reach Leaf2 and Leaf2 sees that it has two equally good paths to reach Leaf1.
IS-IS is the routing protocol used by the VTEPs to learn how to reach the other VTEPs.
Let's say
Host A has a MAC address of A and an IP address of 192.168.1.1 and is attached to port 1/5 on Leaf1
HostB has a MAC address of B and an IP address of 192.168.1.2 and is attached to port 1/6 on Leaf2
Let's say Host A sent an ARP request seeking the MAC address of 192.168.1.2
Now Leaf1 learns about Host A mac address say MAC A is present on port 1/5 & also learn about IP address of 192.168.1.1 associated with it & records all this information in its Local Station Table.
Council Of Oracles Protocol (COOP)
Leaf1 then reports this information to one of the spine switches (chosen at random) using the Council of Oracles Protocol (COOP).
The spine switch that was chosen then relays this information to all the other spines so that every spine has a complete record of every end point in the system.
The spines record the information learned via the COOP in the Global Proxy Table, and this information is used to resolve unknown destination MAC/IP addresses when traffic is sent to the Proxy address.
COOP is used solely for the purpose of distributing endpoint information to Spine switches & not to distribute end host information to leaf switches.
MP-BGP
BGP is used to exchange routing information between ISP's, Business Partner, field office etc. which means now an external router connected to ACI fabric comes in picture.
Say Leaf2 is also a border leaf & has a router connected and has learned some routes from that external router for a particular VRF for a particular Tenant.
How can Leaf2 pass this information on to Leaf1 where Host A is trying to send packets to one of these external networks?
Note: Border Leaf shares external routes with the rest of Leafs. Hence, Leaf will have three routing tables i.e. LST, GST & External Routing Table.
For Leaf2 to be able to pass routing information on to Leaf1 and keep that information exclusive to the same VRF, we need a routing protocol that is capable of exchanging routing information for multiple VRFs across an underlay network.
In the case of ACI, BGP is configured by choosing an Autonomous System number and nominating one or more spine switches to be a route reflector.
Why do we need to make Spine switch Route Reflector and leaf as RR clients?
Spine is connected to all Leaf switches & also leaf is connected to all Spine switches but there isn't any connectivity between leaf & leaf as well as Spine & Spine.
iBGP peers do not pass information to their iBGP peers which they have learned to iBGP peers. This helps to avoid loop between iBGP peers.
ACI is a two-tier hierarchical design where the core and distribution layers are collapsed into one layer.
Hence mesh topology won't work in ACI fabric and hence we do require to configure RR.
MP-BGP is self-configuring, you don't need to do anything to make it work!
Fabric - Inventory
Need to add all Spine & Leaf Switches by clicking "+"
When Auto Firmware Update on Switch Discovery is enabled, APIC automatically updates the switch firmware for the following scenarios:
A new switch discovery with a new node ID.
A switch replacement with an existing node ID.
An initialization and rediscovering of an existing node.
If the new switch's node ID is already part of a firmware update group under Admin > Firmware, such as a replacement scenario, the new switch is updated to the target version specified by the update group. Otherwise, it is updated to Default Firmware Version specified by Auto Firmware Update on Switch Discovery.
Displays information related to the F3083 faults, which regard duplicate IP addresses in the fabric. Each row in the table contains information for one duplicate IP address. You can use the information to locate easily where the conflicts are and to know the corresponding entities that have the issue. You should resolve the duplicate IP address issues as soon as possible. In addition to the information on this screen, you can see the F3083 faults by clicking the bell button, which displays the alert list.
Fabric Policies are configured for Fabric Connectivity i.e. Spine & Leaves and also connectivity of spine switch to other POD or multi-tenant.
Access Policies
Physical Connectivity of Access Policies Port Configuration
Step 1: Create VLAN Pool
Step 2: Attach Vlan Pool with Physical Domain
Step 3: Create AAEP Profile and attach the domain to it
Step 4: Configure Interface Policies.
Fabric --> Access Policies --> Policies --> Interface
CDP--Enable/Disable
MCP--Enable/Disable
LLDP--Enable/Disable
Port-Channel-->Active/Passive
Link_Level--Speed/Auto-Negotiation
Step 5: Now create a Interface Policy Group for various interfaces i.e. Access Port, VPC, etc. & attach AAEP profile to it.
Let's take example of access port
Step 6 : Create a Interface Profile which is a combination of interface selector + Interface Policy Group.
Step 7: Create a Switch Profile and add a switch to it.
Now add Switch Interface Profile To it.
Logical Connectivity of Physical Configuration to Tenant.
Step 1 : Create Tenant & VRF
Step 2 : Create Bridge Domain
Step 3 : Create Application Profile
Step 4 : Create EPG
Step 5 : Add Domain to EPG
Step 6 : Create Static Port Binding
Trunk: The default deployment mode. Choose this mode if the traffic from the host is tagged with a VLAN ID.
Access (802.1P): Choose this mode if the traffic from the host is tagged with a 802.1P tag.
Access (Untagged): Choose this mode if the traffic from the host is untagged (without a VLAN ID).
Micro Segment EPG
Make Intra EFP Isolation --> Enforce
Contracts, Subject & Filter
We can use a common tenant for this.
Tenant --> Common --> Contracts
Step 1: Create a Filter
Step 2: Create Contract
Step 3 : Create Contract Subject & call filter in it.
Step 4: Now once the contract is created then it has to be added as provided or consumed to EPG.
Taboo contracts can be used to deny specific traffic that is otherwise allowed by contracts. The traffic to be dropped matches a pattern (such as any EPG, a specific EPG, or traffic matching a filter). Taboo rules are unidirectional, denying any matching traffic coming toward an EPG that provides the contract.
Note: Unicast, multicast & unknown broadcast traffic are allowed by default.
Out Of Band Management IP Address
Apic Controller & switches can be connected to OOB IP address i.e. non fabric IP address which could be accessed from the legacy network.
This will be configured in the management tenant.
How to Restrict access to particular Subnets and applications to the OOB address of fabric (APIC, Leaf & Spine)?
Step 1: Define the Filter to be applied to the OOB Contract (Here we only allowed https)
Step 2: Apply the filter to OOB Contract
Step 3: This OOB Contract will be consumed by OOB EPG
Step 4 : The Provider of this OOB Contract will be External Management Network Instance Profile where we also define subnets that could access OOB Addresses.
NTP Configuration
Step 1: This NTP clock will work only for devices configured for OOB ip address.
Step 2: Configure Date & Time in POD group i.e. basically, provide ip address/hostname of NTP server.
Step 3: Create a POD Policy Group and call the NTP server Defined above
Note: SNMP Policy will also be pushed in the same manner.
Step 4: Create a POD Profile to link PODs to various POD Policies.
Step 5 : Go to step where you defined ip/hostname of NTP server and if all is good than NTP will sync
VPC in ACI Fabric
vPC works on Active-Active mode.
In ACI, running vPC is pretty straight forward, we do not need to prepare vPC Peerlink or vPC keepalive connections.
vPC Peer-Link information will be learned via Spines by vPC Peers.
vPC domain is equal to vPC explicit protection group.
Step 1: Create vPC domain
Step 2 : Create vPC protection Group
Now you can see VTEP address
Step 3: Create vPC Interface Policy Group
Step 4 : Now bind the policy group created above to interface profile
Step 5 : Now add this interface selector to switch profile
L2 OUT
Two Methods
First: By extending the EPG out to external layer 2
Step 1: Create a Vlan pool
Step 2: Create a physical Domain
Step 3 : Create AAEP Profile
Step 4: Create Interface Policies for CDP, LDP & Port channel
Step 5: Create VPC domain & VPC Explicit Protection Group add switches who will participate in VPC
Step 6: Create Leaf Access & VPC Interface Policy and call interface policies, port channel & AAEP
Step 7: Now go to the interface profile & link the interface group policy with the physical interface being used for VPC or access port.
Step 8: Now create switch profiles & link interface profiles for both access port & vPC.
Step 9
Now comes the main part
Create Tenant: Mukesh
Create Bridge Domain: BD1
Create Application Group AP1
Create EPG: EPG1
Now inside EPG map it with the physical domain and do static port binding.
Similarly, do mapping for vPC as well
So, traffic coming from the legacy network via vPC is being mapped with EPG.
Here we just define one VLAN 21 in pool but there can be multiple VLANs.
Second Method: Create separate EPG for each external VLAN.
Also, because separate EPG is defined for external VLAN. Hence, a contract is required for communication.
Here, the number of EGPs required to be created will depend on the number of VLANs whose traffic is allowed.
Step 1: Define Vlan (single)
Step 2: Define External Bridge Domain
Step 3: Create AAEP profile for external Bridge Domain
Step 4 : Create vPC
Step 5: Create a Policy Interface Group & access port
Step 6: Create interface profile for both access port and vPC
Step 7: Create L2Out on Tenant
Step 8: Map Domain and do static port binding for port 21 in EPG1
Step 9: Create Contract
L3 OUT
Step 1: Create vlan for OSPF routes
Step 2: Create L3 External Domain
Step 3: Create AAEP profile for OSPF routes
Step 4: Create Interface Policy
Step 5: As we are going to use here physical interface for connecting the a router so need to create Leaf Access Port Policy Group.
Step 6: Bind the interface policy to interface eth1/21 with the help of interface profile.
Step 7: Connect he interface profile with Switch i.e. leaf1
Now it's very important to note that in order to Redistribute routes from OSPF to EIGRP which are using ACI fabric as a transit network, the routes have to be distributed in BGP.
Step 8: Systems --> Create Route Reflector.
We just need to define AS for BGP and add Spine to it as Route Reflector.
Step 9: Call the RR Policy to POD Policy Group
Step 10 : Call above POD Policy Group to POD Profiles
Note you must be able to see BGP Neighbours with Spine & Leaf
Step 11 : Go to tenant and create OSPF interface Policy. Click Advertise Subnet.
Step 12: Now we need to call L3 Out created on fabric to the tenant L3 Out. Also, need to define the scope ( we chose vrf here ), area ID & area type.
Step 13: Next we need to choose a physical interface and we chose Leaf int1/21 so call that here. Also define ip (192.168.1.1/24).
Similarly Configure EIGRP
Step 14: Define vlan 42 for EIGRP
Step 15 : Define Domain
Step 16 : Define AAEP
Step 17 : Define Access Port Policy Group
Step 18: Map port eth1/42 with above defined IPG
Step 19: Add the above Interface Profile to Switch2/Leaf2
Step 20 : Go to tenant Policy Protocol & Eigrp interface attributes.
Step 21: Create L3 Out for the tenant and call here L3 OUT defined in Fabric.
Step 22: Call the physical port which we defined already Leaf2 port eth1/42
Step 23: Call Eigrp Policy
Step 24: Create EPG
Step 25 : Create Contract for OSPF & EIGRP
Step 26: Add contract as both provider & consumer
Step 27 : Add contract for OSPF L3 OUt EPG as well
Step 28 : TheSubnet in BD will be associated with L3 OUT Profile.
Why is it considered a "Best Practice" to deploy a VLANsingle subnet & VLAN in the bridge domain?
Treat as a virtual IP address
Indicates whether the subnet is a virtual IP address configured for the associated bridge domain.
This is typically used for the Common Pervasive Gateway use case. For more information, see Common Pervasive Gateway in Cisco APIC Layer 3 Configuration Guide.
Make this IP Address Primary
Indicates if the subnet is preferred (primary) over the available alternatives. Only one preferred subnet is allowed.
Choosing this IP address as primary affects DHCP relay only.
Scope
The network visibility of the subnet. The scope can be:
Private to VRF—The subnet applies only to its tenant.
Advertised Externally—The subnet can be exported to a routed connection.
Shared between VRFs—The subnet can be shared with and exported to multiple contexts (VRFs) in the same tenant or across tenants as part of a shared service. An example of a shared service is a routed connection to an EPG present in another context (VRF) in a different tenant.
The subnet control state. The control can be specific protocols applied to the subnet such as IGMP Snooping. The control can be:
ND RA Prefix—Enables Neighbor Discovery on the subnet.
No Default SVI Gateway—If you enable this, Pervasive SVI will not be configured for this subnet. It is used to leak more specific prefix routes to other VRFs.
IP Data-plane Learning
Choose whether to enable or disable IP address learning for this subnet. The possible values are:
Disabled: Disables IP address learning for this subnet.
Enabled: Enables IP address learning for this subnet. This is the default value.
If a BD has multiple VLANs and subnets then even if the broadcast is intended for a single subnet or network then it will flood all subnets available in BD.
Please make a note here that if BPDs are intended for a particular VLAN then it will flow towards its respective VLAN only even if BD has multiple subnets.
Flooding Mode in ACI
Consider that you are Extending the Bridge Domain out of the ACI Fabric & the default Gateway resides in legacy network.
By default, ARP traffic is not flooded but sent to the destination endpoint.
in order to make communication work between host in legacy network and ACI fabric we need to allow Arp flooding in ACI confined to specific EPG.
In Bridge Domain we need to enable ARP flooding & doing so in its associated EPG ARP traffic is also flooded.
What are the three components of ACI architecture?
Application Network Profile (ANP)– a collection of end-point groups (EPG), their connections, and the policies that define those connections.
Application Policy Infrastructure Controller (APIC)– a centralized controller that manages downstream switches.
ACI fabric– This is connection of Spine and Leaf switches. In the ACI world Spine and Leaf are the Cisco Nexus 9000 Series Switches (N9k).
Difference between Common tenant, management tenant and infrastructure tenant?
Management Tenant is used for infrastructure discovery and also used for all communication/integration with virtual machine controllers.
It has separate Out of Band (OOB) address space for APIC to Fabric communication, it is used to connect all fabric management interfaces.
Infrastructure Tenant governs operation of fabric resources like allocating VXLAN overlays and allows fabric administrator to deploy selective shared services to tenants.
Common Tenant contains all the shared resources which can be used by all tenants i.e. Firewall, Load Balancer or LDAP Server etc.
Note: It is possible that BD can be created in Common Tenant and EGP in any of tenants and can be mapped but doing so will have two limitations.
First: The subnets in BD will be visible to all tenants.
Secondly: Same ip address can't be assigned to hosts in EPG1 & EPG2.
How ARP and broadcast handled by ACI?
L2 Unknown Unicast can be Flood or Hardware Proxy.
By default, ACI will convert ARP broadcast traffic into unicast traffic and send it to the correct leaf node or simply Arp flooding is disable in BD.
ARP FLOODING, if enabled, Arp packets sent as broadcast else it would be sent as unicast.
When the BD has L2 Unknown Unicast set to Flood and Clear Remote MAC Entries is selected, if an endpoint is deleted the system deletes it from both the local leaf switches as well as the remote leaf switches where the BD is deployed else remote leaf continues to have this endpoint learned until the timer expires.
Hardware Proxy: If selected then switch will send unicast packet only in case, if it doesn't have information of destination mac address then it will send Arp unicast packet to spine. If Spine Switch does not know the destination, then drops the packet.
Now for Unicast ARP to work IP routing (BD is Gw) has to be enabled, because the mapping database must be populated with the IP addresses of the endpoints. Hardware proxy must be enabled too.
If the hardware proxy is turned off, then ARP flooding is on and cannot be turned off.
If the hardware proxy is turned on but IP routing is turned off, ARP flooding is still on and cannot be turned off.
And it makes perfect sense as now Gw is outside ACI i.e. not on BD, so Arp request has to be flooded in BD if destination mac is unknown.
END POINT MOVEMENT AND BOUNCE ENTERIES
This is the story of Host1 who resides in Leaf1 And his friend Host2 who resides in Leaf2.
One Day Host1 wrote a letter to Host2 .... with address from Host1 who resides on Leaf1 to Host 2 who resides on Leaf2.
Host 2 was very happy to receive a letter from Host1 & replied with an address from Host2 who resides on Leaf2 to Host1 who resides on Leaf1.
Now due to some reason Host2 had to move to Leaf3 and Host3 came to live at Leaf2.
Host2 was very sad as his friend Host1 didn't know that he had now moved to Leaf3 so he will no longer receive letter from Host1.
But Host3 promised that he will give (forward) a letter to Host2 whenever he receives it from Host1.
Now Host1 again wrote a letter to Host2 & letter went to Host3 but as per promise Host3 forwarded it to Host2 on leaf3.
Host2 was delighted to see a letter from Host1 & thanked Host3.
Host 2 then wrote a letter back to Host1 with the address from "Host2 who resides on Leaf3"
Host1 now came to know that Host2 now resides on Leaf3 so now he sends letter with address Leaf3 so Host3 don't have to forward it to Host1 again.
ARP Timers
Timers are configurable in two different configuration locations:
As part of the bridge domain configuration: Tenant > Networking > BD > Policy > General > Endpoint Retention Policy
As part of the VRF configuration: Tenant > Networking > VRF > Policy > Endpoint Retention Policy
The same options appear in both configuration locations.
Local Endpoint Aging Interval: This is the timeout for locally learned endpoints. The endpoint retention timer in Cisco ACI by default is 900 seconds, so Cisco ACI will re-ARP for endpoints every 675 seconds (75% of 900).
Note: Common timers used by various servers' implementation to keep the ARP tables updated are normally a few minutes, such as 1 or 2 minutes, or less. A server that ARPs the default gateway (the bridge domain subnet) automatically also updates the endpoint database in Cisco ACI.
Remote Endpoint Aging Interval: This is the timeout for entries on the leaf switch that point to a different leaf switch (remote entries). The default interval is 300 seconds.
Bounce Entry Aging Interval: This is the timeout for bounce entries, which is the entry that is installed when an endpoint moves to a different leaf switch. Its default interval is 630 seconds.
Hold Interval: This entry refers to the Endpoint Move Dampening feature and the Endpoint Loop Protection feature, is the amount of time that data plane learning is disabled if a loop is observed. The default interval is 300 seconds.
BD Move Frequency or Endpoint Move Dampening: Whether it is a single endpoint flap or a simultaneous move of multiple endpoints, or a combination of both.
If the number of movements per second is exceeded the Move Frequency, the Hold Interval (described above) is triggered, and learning the new endpoint in the BD is disabled until the Hold Interval expires.
The feature is called BD Move Frequency or Endpoint Move Dampening. The default is 256.
How to perform unicast routing on ACI?
Unicast Routing: If this setting is enabled and a subnet address is configured, the fabric provides the default gateway function and routes the traffic.
Enabling unicast routing also instructs the mapping database to learn the endpoint IP-to-VTEP mapping for this bridge domain. IP learning is not dependent upon having a subnet configured under the bridge domain.
What is ARP gleaning?
If hardware proxy is turned on and IP routing is turned on (ARP flooding is disabled)
If the endpoint had been silent Cisco ACI can resolve the endpoint IP address by sending ARP messages from the subnet IP address (By looking at VXLAN header) of the bridge domain & it requires the bridge domain to be configured with a subnet IP address i.e. Local Endpoint Aging Interval.
Now consider a scenario when the host is just connected and request for its mac address has been received from Leaf by Spine switch as obviously Spine, at his point of time, will not have database mapping of requested Arp.
Now what will Spine do?
Spine will discard Arp received from leaf and create a new Arp on behalf of existing Arp and send it Gw ip of VXLAN subnet (configured on BD).
This process of creating "new ARP" is called ARP gleaning
Pervasive Gateway
Common Pervasive Gateway is an older feature that was used to connect multiple ACI fabrics together via an L2 connection prior to the availability of ACI MultiPod and ACI Multisite. While most customers will use a newer feature such as ACI MultiPod, or ACI Multisite.
This feature addresses the requirement to extend Bridge Domain (BD) across multiple ACI Fabrics to provide the same default gateway to servers in each ACI Fabrics.
This cool thing about this feature is that it enables moving one or more virtual machines (VM) or conventional hosts across different ACI Fabrics seamlessly without any configuration changes or operations. It’s as if VMs are moving within the same ACI Fabric.
Common pervasive gateway allows us to have a virtual MAC and virtual IP which is common (i.e., the same) across multiple ACI Fabrics.
All devices in the Bridge Domain with the Common Pervasive Gateway feature are supposed to point at the Common Pervasive Gateway (virtual IP) as its default gateway.
However, it is still required to have a non-virtual IP on the BD in each of the ACI Fabrics on top of virtual IP. This non-virtual IP should be in the same subnet as virtual IP and should be unique for each of the ACI Fabrics. This is similar to HSRP physical IP and virtual IP configuration.
Contract
Contracts are used to permit or deny traffic flows within the ACI fabric. They control traffic between EPGs.
Compared with ACLs we won’t find here source and destination IP definitions.
The ACI contract is composed of three elements:
Label: It is just an optional identifier.
Subject: It contains a Filter (TELNET, SSH, and HTTP), Action, and optional Label.
Contract: It contains one or more Subjects.
Contract is configured between EGPs i.e. provider contract & consumer contract.
Taboo Contracts
Taboo filters don't work "between" EPGs, but are applied to an entire EPG.
Example: Let's say you want to prevent an EPG from ever using cleartext communications, you could apply a taboo contract with filters for port 80 and 23.
Micro EPGs (uEPG)
It is similar to a private isolation vlan. By default, communication within EPG is permitted & for inter EPG communication contracts are required.
To control traffic in between servers inside same EPG.
Base EPG & IP uEPG must associate with same BD and the BD MUST have an IP subnet configured.
Infrastructure VLAN
During fabric setup, ACI requires a VLAN to be used as the infrastructure VLAN. This VLAN is used for controlling traffic between devices that make up the fabric
(i.e. Leafs, spines, and APICs).
Tunnel End Point (TEP) IP address:
The TEP range defines the Overlay-1 VRF. Overlay-1 VRF contains /32 routes to each VTEP, VPC Virtual IP, APIC, and Spine Proxy IP.
Physical Tunnel Endpoint (PTEP): - This is the IP address provided by APIC from Infrastructure Subnets to Leaf switches as a loopback interface. This address is used for communication with APIC, other Leafs, MP-BGP peering, traceroute or ping.
Proxy TEP IP address: - This is an anycast IP address (a single IP address to all Spines) that is present across all spines and is used for forwarding lookups into the mapping database.
Fabric loopback TEP (FTEP) IP address: - This address is used when VMM domain (ESXI environment) is present. A fabric loopback TEP (FTEP) is used to encapsulate traffic in VXLAN to a vSwitch VTEP. It is a unique FTEP address that is identical on all leaf nodes to allow the mobility of downstream VTEP devices.
vPC loopback VTEP address: - This IP address is used when the two leaf nodes forward traffic that enters through a vPC port. Traffic is forwarded by the leaf using the VXLAN encapsulation. This address is shared with the vPC peer.
Reset APIC & Switch
Reset APIC
apic1# acidiag touch setup
This command will reset the device configuration, Proceed? [y/N] N
Exited from the command
Reset Leaf
admin@leaf1:~> acidiag touch clean
This command will wipe out this device, Proceed? [y/N] N
Exited from the command
Service Graph PBR
ACI technology provides the capability to insert Layer 4 through Layer 7 (L4-L7) functions using a service graph approach.
One of the main features of the service graph is Policy-Based Redirect (PBR).
With PBR, the Cisco ACI fabric can redirect traffic between security zones to L4-L7 devices, such as a firewall, Intrusion-Prevention System (IPS), or load balancer, without the need for the L4-L7 device to be the default gateway for the servers.
Cisco ACI can selectively send traffic to L4-L7 devices based, for instance, on the protocol and the Layer 4 port.
Firewall inspection can be transparently inserted in a Layer 2 domain with almost no modification to existing routing and switching configurations.
Troubleshooting
Tools for monitoring traffic, debugging, and detecting issues such as traffic drops, misrouting, blocked paths, and uplink failures.
ACL Contract Permit and Deny Logs — It Enables the logging of packets or flows that were allowed to be sent because of contract permit rules and the logging of packets or flows dropped because of taboo contract deny rules.
Atomic Counters — It Enables you to gather statistics about traffic between flows for detecting drops and misrouting in the fabric and for enabling quick debugging and isolation of application connectivity issues.
Digital Optical Monitoring — It Enables you to view digital optical monitoring (DOM) statistics about a physical interface.
Health Scores — It Enables you to isolate performance issues by drilling down through the network hierarchy to isolate faults to specific managed objects (MOs).
Port Tracking — It Enables you to monitor the status of links between leaf switches and spine switches for detecting uplink failure.
SNMP — Simple Network Management Protocol (SNMP) - It enables you to remotely monitor individual hosts (APIC or another host) and find out the state of any particular node.
SPAN — Switchport Analyzer (SPAN) enables you to perform detailed troubleshooting or to take a sample of traffic from a particular application host for proactive monitoring and analysis.
Statistics — It Provides real-time measures of observed objects. Viewing statistics enables you to perform trend analysis and troubleshooting.
Syslog — It Enables you to specify the minimum severity level of messages to be sent, the items to be included in the syslog messages, and the syslog destination. The format can also be displayed in NX-OS CLI format.
Traceroute—Enables you to find the routes that packets actually take when traveling to their destination.
Troubleshooting Wizard — It Enables administrators to troubleshoot issues that occur during specific time frames, which can be designated by selecting two endpoints.
Configuration Sync Issues—This enables you to see if any transactions in Cisco APIC have not yet synced.
CMD
apic1# acidiag fnvread
ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
101 1 leaf-1 TEP-1-101 10.0.232.64/32 leaf active 0
102 1 leaf-2 TEP-1-102 10.0.232.66/32 leaf active 0
201 1 spine-1 TEP-1-103 10.0.232.65/32 spine active 0
Total 3 nodes
From VTEP address we can login in switch and spine
apic1# ssh 10.0.232.64
Warning: Permanently added '10.0.232.64' (RSA) to the list of known hosts.
admin@10.0.232.64's password:
admin@leaf1:~> exit
logout
Connection to 10.0.232.64 closed.
apic1# cat /proc/net/bonding/bond0 ( VTEP address of fabric )
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: apic1-eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: apic1-eth3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 6a:73:e1:71:cb:c5
Slave queue ID: 0
apic1# cat /proc/net/bonding/bond1 (out of band management)
DNS, NTP & MP-BGP has to be configured for entire fabric
Fabric Policies --> Policies --> Monitoring
DOMAIN IN ACI
Domains are classified as types of networks or environment we have in cisco ACI.
Four Types of domains are available in Cisco ACI
1) Physical: BareMetal Server or endpoints Connected such as Storage, Esxi Host, Citrix Host etc.
2) L2Out or External Bridged Domain: Gateway Resides outside ACI.
3) L3Out or External Routed Domain: ACI with external router for learning and advertising networks. We will use routing protocols such as OSPF & BGP
4) VMM is for Virtualization
Only one pool can be associated with each domain but all domains can have the same pool.
Bridge Domain:
BD refers to a VXLAN and is represented by VNI number.
It is a container which carries multiple subnets with bridging functionality.
Traffic between the subnets within BD will be bridged, i.e. No routing is required.
Traffic between the subnet of different BD will require routing.
Every host is represented as /32 in ACI.
L2 Flooding is disabled by default in BD however it can be enabled.
IPING
iping <destination_ip> -V <Tenant>:<VRF> -S <source Ip> -c <count> (optional -i 0 for flooding).
If you don't know which VRFs are deployed on the switch, use "show vrf" to find out.
If you don't know which IPs are in which VRF, use "show ip int brief" to find out, which will also tell you the VRFs.
Always remember to include VRF, otherwise it will use management interface for pings. If you don't specify source IP, it will randomly select an IP for you in the VRF that you specified.
TO DO
How VLAN working in Cisco ACI?
How can you configure trunk and access port on ACI?
You tie a vlan pool to a Physical Domain and that physical domain is then tied to an AEP profile. That is the "object" that you then associate to your interfaces. Specifically, you tie it to your Port Policy Group which is where you associate all the behaviors you want for that port or interface like 10G, CDP disabled, LLD enabled, and AEP (what vlans can be associated with this set of interface behaviors)). That is tied to an Interface Profile that you associate with a Switch Profile. You now have an interface configured on a particular leaf or leafs.
You go to your Tenant > ANP > EPG > and define a Static Port where you select the interface you configured above and configure it for that EPG with the vlan encapsulation you want for the EPG (which should be one of the valid vlans in your vlan pool).
What is micro segmentation and how do you configure it?
How to configure access and trunk vlan in ACI ?
How to configure inter-VRF and Inter-tenant communication?
How can you integrate Cisco ACI with VMware?
Explain about ACI fabric discovery process?
Explain about traffic flow lookup on ACI fabric?
In Fabric, which switch will act as default gateway for particular subnet?