In-Band and Out-of-Band Management
Two Management Options
Out-of-Band (OOB): Uses a dedicated physical management port on the back of the device.
In-Band (INB): Uses a special EPG/BD/VRF inside the management tenant. You can configure various parameters for it.
Default OOB EPG
The management tenant already has an OOB EPG by default.
You cannot modify this default OOB EPG except to configure its Provided OOB Contracts.
Interfaces on the APIC
The OOB interface appears as oobmgmt in the output of the ifconfig command.
The In-Band interface appears as bond.x, where x is the VLAN ID used for the in-band EPG.
apic1# ifconfig oobmgmt
oobmgmt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.4.20 netmask 255.255.255.0 broadcast 192.168.4.255
inet6 fe80::7269:5aff:feca:2986 prefixlen 64 scopeid 0x20<link>
ether 70:69:5a:ca:29:86 txqueuelen 1000 (Ethernet)
RX packets 495815 bytes 852703636 (813.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 432927 bytes 110333594 (105.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
apic1# ifconfig bond0.300
bond0.300: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1496
inet 10.30.30.254 netmask 255.255.255.0 broadcast 10.30.30.255
inet6 fe80::25d:73ff:fec1:8d9e prefixlen 64 scopeid 0x20<link>
ether 00:5d:73:c1:8d:9e txqueuelen 1000 (Ethernet)
RX packets 545 bytes 25298 (24.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6996 bytes 535314 (522.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Interfaces on Leaf Switches
The OOB interface appears as eth0 in the ifconfig command.
The In-Band interface is a dedicated SVI (Switch Virtual Interface).
You can see the In-Band interface by using:
ifconfig
show ip interface vrf mgmt:<vrf> (where <vrf> is the name of the In-Band VRF).
leaf101# show interface mgmt 0
mgmt0 is up
admin state is up,
Hardware: GigabitEthernet, address: 00fc.baa8.2760 (bia 00fc.baa8.2760)
Internet Address is 192.168.4.23/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is routed
full-duplex, 1000 Mb/s
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
EtherType is 0x0000
30 seconds input rate 3664 bits/sec, 4 packets/sec
30 seconds output rate 4192 bits/sec, 4 packets/sec
Rx
14114 input packets 8580 unicast packets 5058 multicast packets
476 broadcast packets 2494768 bytes
Tx
9701 output packets 9686 unicast packets 8 multicast packets
7 broadcast packets 1648081 bytes
In-Band on Leaf Switches (Secondary IP)
The command show ip interface vrf mgmt:<vrf> will display the in-band management BD (Bridge Domain) subnet IP as a secondary IP address.
This is the expected behavior.
leaf# show ip interface vrf mgmt:inb
IP Interface Status for VRF "mgmt:inb-vrf"
vlan16, Interface status: protocol-up/link-up/admin-up, iod: 4, mode: pervasive
IP address: 10.30.30.1, IP subnet: 10.30.30.0/24
secondary IP address: 10.30.30.3, IP subnet: 10.30.30.0/24
IP broadcast address: 255.255.255.255
IP primary address route-preference: 0, tag: 0
Interfaces on Spine Switches
The In-Band management IP is added as a dedicated loopback interface in the mgmt:<vrf> VRF.
This is different from how leaf switches handle In-Band management.
spine201# show ip interface vrf mgmt:inb
IP Interface Status for VRF "mgmt:inb"
lo10, Interface status: protocol-up/link-up/admin-up, iod: 98, mode: pervasive
IP address: 10.30.30.12, IP subnet: 10.30.30.12/32
IP broadcast address: 255.255.255.255
IP primary address route-preference: 0, tag: 0
APIC Connectivity Preference
Under System Settings, you can select In-Band or Out-of-Band as the preferred connectivity method for the APICs.
Only traffic sent from the APIC uses the selected management preference.
The APIC can still receive traffic on both In-Band and Out-of-Band interfaces if they are configured.
APIC Routing Table (Simplified)
When OOB (Out-of-Band) is Selected:
The metric value for oobmgmt (OOB interface) is 16.
The metric value for bond0.300 (In-Band interface) is 32 (higher than 16).
Since lower metric values are preferred, the OOB interface (oobmgmt) is used for outgoing management traffic.
apic1# bash
admin@apic1:~> route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.4.1 0.0.0.0 UG 16 0 0 oobmgmt
0.0.0.0 10.30.30.1 0.0.0.0 UG 32 0 0 bond0.300
When In-Band is Selected:
The metric value for bond0.300 (In-Band interface) is 8.
The metric value for oobmgmt (OOB interface) is 16 (higher than 8).
Since lower metric values are preferred, the In-Band interface (bond0.300) is used for outgoing management traffic.
admin@apic1:~> route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.30.30.1 0.0.0.0 UG 8 0 0 bond0.300
0.0.0.0 192.168.4.1 0.0.0.0 UG 16 0 0 oobmgmt
Note : The leaf and spine node management preferences are not affected by this setting.
Verification of Out-of-band Configuration
To verify out-of-band configuration, check four folders within a specific tenant named 'mgmt':
• Node Management Addresses.
Node Management Addresses can either be assigned statically or from a pool.
• Node Management EPGs.
The out-of-band EPG should be present under the Node Management EPGs folder.
• Out-of-band Contracts (located under Contracts).
The contracts that determine which management services the out-of-band EPG provides are unique contracts set up in the out-of-band contracts folder.
• External Network Instance Profiles.
Then, ensure that the External Management Network Instance Profile has been created and that the appropriate out-of-band contract is set up as the 'Consumed Out-Of-Band Contract'.
Network Connectivity (through OOB)
To verify network connectivity via the OOB, use ping to test the path
of the packet through the out-of-band network.
apic# ping 192.168.4.1
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_seq=1 ttl=255 time=0.409 ms
64 bytes from 192.168.4.1: icmp_seq=2 ttl=255 time=0.393 ms
64 bytes from 192.168.4.1: icmp_seq=3 ttl=255 time=0.354 ms
apic# bash
admin@apic1:~> traceroute 10.55.0.16
traceroute to 10.55.0.16 (10.55.0.16), 30 hops max, 60 byte packets
1 192.168.4.1 (192.168.4.1) 0.368 ms 0.355 ms 0.396 ms
2 *
3 *
4 10.0.255.221 (10.0.255.221) 6.419 ms 10.0.255.225 (10.0.255.225) 6.447 ms *
5 *
6 *
7 10.55.0.16 (10.55.0.16) 8.652 ms 8.676 ms 8.694 ms
Mgmt Tenant
Make sure the management tenant includes a BD with a subnet for assigning in-band node management addresses to the fabric nodes for in-band connectivity, and verify that the L3Out is associated with the in-band management BD.
Like out-of-band, fabric node in-band management IP addresses can be assigned either statically or dynamically.
Note :
The spine nodes won't reply to ping on the in-band because they utilize loopback
interfaces for connectivity, which do not respond to ARP.
Comments