Oracle Infrastructure Series, part 4. Previous: part 3, preparing Oracle Linux 9. Next up, part 5: HAIP vs bonding for the private interconnect.
A RAC node that loses its only public NIC takes its VIP, its SCAN listener and every client session with it, even though the database is fine. Bonding two physical ports into one logical interface removes that single point of failure. This post builds the public network bond (bond0) from the part 2, planning the RAC cluster on Oracle Linux 9 with NetworkManager’s nmcli, compares the two bonding modes that make sense for Oracle (active-backup and 802.3ad/LACP), and shows how to test failover before Grid Infrastructure is installed. Everything applies to 19c, 23ai and 26ai alike: Clusterware does not care how the bond is built, only that the interface name and subnet are identical on every node.
What we are building
switch A switch B
(MLAG/vPC pair needed only for cross-switch LACP)
| |
rac1 ens192 -+-----. .-------+- ens193
bond0 10.10.10.11/24 (public, VIP .21, SCAN .31-.33)
rac2 ens192 -+-----' '-------+- ens193
bond0 10.10.10.12/24 (public, VIP .22)
Interconnect stays on two separate NICs (ens224 / ens256) for HAIP, see part 5.Active-backup vs 802.3ad (LACP)
| mode=active-backup (1) | mode=802.3ad (4, LACP) | |
|---|---|---|
| Traffic | One port carries everything; the other waits | Both ports carry traffic, flows hashed across ports |
| Switch configuration | None (plain access ports in the same VLAN) | LACP port-channel on the switch; across two switches needs MLAG/vPC/stacking |
| Failover time | Link detection (miimon) plus a gratuitous ARP, typically under a second | Port leaves the aggregate; remaining port keeps flowing |
| Works in VMs | Yes (VMware, KVM, OCI) | Usually not; the hypervisor does the teaming |
| Single flow bandwidth | One port | Still one port per flow (hashing is per flow) |
| Best for | Most RAC public networks; the safe default | High client throughput where the network team already runs LACP |
For the public network of a typical RAC cluster, active-backup is the right choice: it needs nothing from the switch team, it survives a whole switch failing when each port goes to a different switch, and the public network rarely needs more than one port of bandwidth. Choose LACP when you have MLAG-capable switches and real throughput needs (large data loads over SQL*Net, application servers pulling big result sets). Avoid the other modes (balance-rr, balance-alb, balance-tlb) for RAC; they reorder packets or rewrite MAC addresses in ways that confuse ARP for the floating VIP and SCAN addresses.
Prerequisites
- Oracle Linux 9 nodes prepared as in part 3, preparing Oracle Linux 9.
- Two physical ports per node for the public network (here
ens192andens193), ideally on different NIC cards and cabled to different switches. - The IP plan:
rac110.10.10.11/24,rac210.10.10.12/24, gateway 10.10.10.1, DNS servers that resolve the SCAN name. - For LACP only: a port-channel configured by the network team with LACP active or passive on both switch ports, and the hash policy agreed.
- Console access (iLO, iDRAC, hypervisor console) in case you cut yourself off while moving the IP address onto the bond.
Step 1: Identify the ports and clean up old profiles
# nmcli device status
DEVICE TYPE STATE CONNECTION
ens192 ethernet connected ens192
ens193 ethernet disconnected --
ens224 ethernet connected ens224
ens256 ethernet connected ens256
lo loopback connected (externally) lo
# ethtool ens192 | grep -E 'Speed|Link detected'
Speed: 10000Mb/s
Link detected: yes
# ethtool ens193 | grep -E 'Speed|Link detected'
Speed: 10000Mb/s
Link detected: yesBoth ports must have link and the same speed. Note the existing profile that holds the public IP (ens192 above); we delete it after the bond is defined, so do this from the console, not an SSH session over that IP.
Step 2a: Create an active-backup bond
# nmcli connection add type bond con-name bond0 ifname bond0 \
bond.options "mode=active-backup,miimon=100,primary=ens192" \
ipv4.method manual ipv4.addresses 10.10.10.11/24 ipv4.gateway 10.10.10.1 \
ipv4.dns "10.10.10.5,10.10.10.6" ipv4.dns-search example.com \
ipv6.method disabled connection.autoconnect yes
# nmcli connection add type ethernet con-name bond0-port1 ifname ens192 \
master bond0 slave-type bond
# nmcli connection add type ethernet con-name bond0-port2 ifname ens193 \
master bond0 slave-type bondmiimon=100 checks link state every 100 ms. primary=ens192 makes that port active whenever it is healthy, so traffic always takes a predictable path. On newer NetworkManager releases the same options are also spelled controller bond0 port-type bond; master/slave-type works on every Oracle Linux 9 update. Use ipv6.method disabled only if you do not run IPv6 on the public network, and keep the choice identical on all nodes.
Step 2b: Or create an 802.3ad (LACP) bond
# nmcli connection add type bond con-name bond0 ifname bond0 \
bond.options "mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer3+4" \
ipv4.method manual ipv4.addresses 10.10.10.11/24 ipv4.gateway 10.10.10.1 \
ipv4.dns "10.10.10.5,10.10.10.6" ipv4.dns-search example.com \
ipv6.method disabled
# nmcli connection add type ethernet con-name bond0-port1 ifname ens192 master bond0 slave-type bond
# nmcli connection add type ethernet con-name bond0-port2 ifname ens193 master bond0 slave-type bondlacp_rate=fast sends LACPDUs every second instead of every 30, so a dead partner is noticed in about three seconds. xmit_hash_policy=layer3+4 spreads flows by IP and port, which balances many client connections well; layer2+3 is the more conservative, fully standards-compliant choice. Match whatever the switch side uses for its own hashing as closely as the platform allows.
Step 3: Move the IP address to the bond
# nmcli connection delete ens192 # old profile that held 10.10.10.11
# nmcli connection up bond0
# nmcli connection up bond0-port1
# nmcli connection up bond0-port2
# nmcli device status
DEVICE TYPE STATE CONNECTION
bond0 bond connected bond0
ens192 ethernet connected bond0-port1
ens193 ethernet connected bond0-port2
ens224 ethernet connected ens224
ens256 ethernet connected ens256NetworkManager stores the profiles as keyfiles in /etc/NetworkManager/system-connections/ (bond0.nmconnection, bond0-port1.nmconnection and so on); the old ifcfg-* files from Oracle Linux 6 and 7 are not used for new profiles on Oracle Linux 9. Repeat steps 1 to 3 on rac2 with 10.10.10.12, using the same bond name, mode and options.
Verification
Bond state
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.15.0-209.161.7.2.el9uek.x86_64
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: ens192 (primary_reselect always)
Currently Active Slave: ens192
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
Slave Interface: ens192
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:50:56:8a:11:01
Slave queue ID: 0
Slave Interface: ens193
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:50:56:8a:11:02
Slave queue ID: 0For an LACP bond, the same file shows Bonding Mode: IEEE 802.3ad Dynamic link aggregation, the transmit hash policy, and an “Active Aggregator Info” block. Check that both ports report the same Aggregator ID and that Partner Mac Address is the switch’s MAC, not 00:00:00:00:00:00:
# grep -E 'Bonding Mode|Hash Policy|LACP rate|Aggregator ID|Partner Mac' /proc/net/bonding/bond0
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
LACP rate: fast
Aggregator ID: 1
Partner Mac Address: 2c:3a:fd:41:7e:00
Aggregator ID: 1
Aggregator ID: 1Addressing and routing
# ip -br addr show bond0
bond0 UP 10.10.10.11/24
# ip route
default via 10.10.10.1 dev bond0 proto static metric 300
10.10.10.0/24 dev bond0 proto kernel scope link src 10.10.10.11 metric 300
192.168.10.0/24 dev ens224 proto kernel scope link src 192.168.10.11 metric 101
192.168.11.0/24 dev ens256 proto kernel scope link src 192.168.11.11 metric 102
# ping -c3 rac2 && ping -c3 10.10.10.1Step 4: Test failover before installing Grid Infrastructure
Run a continuous ping from another host to rac1, then fail the active port. Pulling the cable or shutting the switch port is the real test; from the node you can simulate it by taking the port down:
(from an admin host) $ ping 10.10.10.11
(on rac1)
# ip link set dev ens192 down
# grep 'Currently Active' /proc/net/bonding/bond0
Currently Active Slave: ens193
# journalctl -k --since "-2min" | grep bond0
kernel: bond0: (slave ens192): link status definitely down, disabling slave
kernel: bond0: (slave ens193): making interface the new active one
# ip link set dev ens192 up
# grep 'Currently Active' /proc/net/bonding/bond0
Currently Active Slave: ens192Expect at most one or two lost pings. The kernel log shows the failed port being disabled and the surviving port becoming active. Because of primary=ens192 the bond returns to ens192 when it comes back. Repeat with the other port, and repeat on rac2. Once the cluster is running, do the same test again while watching the VIP: it must stay on its node, because the bond, not Clusterware, handles a single port failure.
How Clusterware sees the bond
During the Grid Infrastructure installation you classify bond0 as public and the two interconnect NICs as ASM and private. Afterwards oifcfg records it:
$ oifcfg getif
bond0 10.10.10.0 global public
ens224 192.168.10.0 global cluster_interconnect,asm
ens256 192.168.11.0 global cluster_interconnect,asm
$ srvctl config nodeapps -viponly
VIP exists: network number 1, hosting node rac1
VIP Name: rac1-vip.example.com
VIP IPv4 Address: 10.10.10.21
...The VIPs and SCAN VIPs appear as secondary addresses on bond0 (ip addr show bond0 lists them with labels like bond0:1). The global keyword is why the interface name must be the same on every node: a bond called bond0 on one node and team0 on another breaks the network resource.
Troubleshooting and gotchas
- LACP bond up but only one port in the aggregate: the switch side is not an LACP port-channel, or the two switch ports land in different aggregators.
Partner Mac Address: 00:00:00:00:00:00means no LACPDUs are arriving from the switch. - LACP across two separate switches without MLAG/vPC: the ports end up in different aggregators and only one carries traffic. Use active-backup in that case.
- Failover takes 30 seconds or more: spanning tree on the switch port. Ask for PortFast (or the vendor equivalent edge-port setting) on server-facing ports, or add
updelayso the bond waits for the port to really forward. - LACP inside a virtual machine: the virtual switch does not speak LACP to the guest. Give the VM two vNICs on separate uplinks and use active-backup, or a single vNIC and let the hypervisor team the uplinks.
- VIP unreachable from other subnets after a failover: routers or firewalls with long ARP caches or ARP inspection. Clusterware sends gratuitous ARPs when the VIP moves; for a bond port failover the bond sends them (
num_grat_arp, default 1, can be raised). - Mixing miimon and arp_interval: use one link monitor.
arp_interval/arp_ip_targetis only valid for active-backup and adds checking of the path beyond the switch, but a wrong target makes the bond flap. - Renaming the interface after installation: Clusterware stores the name. Add the new name with
oifcfg setif -globaland remove the old one withoifcfg delif, then update the network resource withsrvctl modify network, during a maintenance window and following Oracle’s documented procedure. - firewalld zone:
bond0belongs in the zone that allows the listener ports (1521/tcp for the SCAN and node listeners). The firewalld cheat sheet covers zone assignment.
If you still maintain older systems, the same idea with ifcfg files is in configuring a bond interface on Oracle Linux 6 and redundant NIC configuration with bonding on CentOS; on Oracle Linux 9, use NetworkManager as shown here.
Oracle Infrastructure Series: this is part 4. Previous: part 3, preparing Oracle Linux 9. Earlier: part 2, planning the RAC cluster and part 1, the overview. Next is part 5, Redundant Interconnect Usage (HAIP) vs OS bonding for the RAC private interconnect.
References: Oracle’s Grid Infrastructure Installation and Upgrade Guide 19c for Linux (network interface configuration requirements), the Oracle Linux 9 networking documentation on network bonding, and the Linux kernel bonding driver documentation for every bond option.