Oracle Infrastructure Series, part 5. Previous: part 4, bonding the public network. Next up, part 6: DNS and GNS for SCAN.
The private interconnect is the most important network in a RAC cluster. It carries the Clusterware heartbeat, Cache Fusion block transfers between instances, and (with Flex ASM) ASM traffic. If it fails on a node, Clusterware evicts that node to protect the data. So it needs redundancy, and there are two ways to get it: Oracle’s own Redundant Interconnect Usage (also called HAIP, Highly Available IP), or an operating system bond like the one built in part 4, bonding the public network. This post explains how each works, which one to pick, how to set up and verify HAIP on Oracle Linux 9 with Grid Infrastructure 19c, and how to switch an existing cluster between the two. It applies unchanged to 23ai and 26ai, where HAIP is still the documented default for multiple private interfaces.
How HAIP works
When you classify more than one interface as private during the Grid Infrastructure installation, Clusterware starts the ora.cluster_interconnect.haip resource. It plumbs one to four link-local addresses from the 169.254.0.0/16 range on top of your private interfaces. The ASM and database instances send interconnect traffic to those HAIP addresses, not to your 192.168.x.x addresses, and spread it across all of them. If an interface dies, Clusterware moves its HAIP address to a surviving interface; when the interface returns, the address moves back.
rac1 rac2
ens224 192.168.10.11 ---- switch P1 ---- ens224 192.168.10.12
+-- HAIP 169.254.a.x +-- HAIP 169.254.a.y
ens256 192.168.11.11 ---- switch P2 ---- ens256 192.168.11.12
+-- HAIP 169.254.b.x +-- HAIP 169.254.b.y
ens224 fails on rac1: 169.254.a.x moves onto ens256 (both HAIPs now on ens256)
Clusterware heartbeat (ocssd) keeps using whichever private network still worksThis is the layout from part 2, planning the RAC cluster: two interconnect NICs on two separate subnets and two separate switches, with no bond. Oracle’s documentation is explicit that each private interface should be on a different subnet, and every node must have an interface on every private subnet.
HAIP vs an OS bond for the interconnect
| Redundant Interconnect Usage (HAIP) | OS bond (bond1, active-backup or LACP) | |
|---|---|---|
| Who handles failover | Clusterware (GIPC and the HAIP resource) | The Linux bonding driver |
| Interfaces Clusterware sees | Two to four, each its own subnet | One logical interface, one subnet |
| Load balancing | Yes, across all HAIP addresses | Only with LACP, and per flow |
| Switch configuration | None; plain ports, separate VLANs or switches | None for active-backup; port-channel/MLAG for LACP |
| Failover time | Seconds; the HAIP move is visible in the GI alert log | Sub-second link failover |
| Fits virtualized or cloud RAC | Yes, with two vNICs | Often the hypervisor already teams the uplinks |
| Gotchas | Needs 169.254.0.0/16 free on the host; rp_filter | Single subnet, so a misbehaving switch can hit both ports |
Practical guidance: on bare metal with two dedicated interconnect ports, HAIP on two subnets is the simplest choice and is what Oracle’s installer sets up by default. Pick a bond when the site standard already bonds everything, when you need to share the ports for other private traffic, or when the platform is a VM with a single teamed vNIC. Do not stack both on the same ports (a bond and two HAIP interfaces built from the same physical NICs): you gain nothing and debugging gets harder. If you use a single bonded interconnect, Clusterware still starts HAIP and puts one 169.254 address on the bond; that is normal.
Prerequisites
- Two nodes prepared as in part 3, preparing Oracle Linux 9, with the private interfaces
ens224(192.168.10.0/24) andens256(192.168.11.0/24) up and in the trusted firewalld zone. - Reverse path filtering set to loose (2) or off (0) on the private interfaces. Strict mode (1) drops HAIP packets that arrive on the “wrong” interface after a failover. Part 3 set
rp_filter=2. - No other use of
169.254.0.0/16on the nodes: no link-local addresses from NetworkManager, no avahi, and no static routes into that range. - MTU identical on every interconnect port and every switch port in the path. If you use jumbo frames (9000), set them everywhere.
- Each private subnet reachable from every node, tested before installation.
Step 1: Configure and check the private interfaces
Each interconnect NIC gets a plain static address with no gateway and no DNS. On rac1:
# nmcli con add type ethernet ifname ens224 con-name priv1 \
ipv4.method manual ipv4.addresses 192.168.10.11/24 \
ipv4.never-default yes ipv6.method disabled 802-3-ethernet.mtu 9000
# nmcli con add type ethernet ifname ens256 con-name priv2 \
ipv4.method manual ipv4.addresses 192.168.11.11/24 \
ipv4.never-default yes ipv6.method disabled 802-3-ethernet.mtu 9000
# nmcli con up priv1 && nmcli con up priv2
# nmcli -g ipv4.link-local con show priv1
defaultipv4.link-local of default means disabled for a manually addressed profile, so NetworkManager will not add its own 169.254 address. Repeat on rac2 with .12. Then check reachability on both subnets and that jumbo frames really pass (8972 bytes of payload plus 28 bytes of headers is 9000):
# sysctl net.ipv4.conf.ens224.rp_filter net.ipv4.conf.ens256.rp_filter
net.ipv4.conf.ens224.rp_filter = 2
net.ipv4.conf.ens256.rp_filter = 2
# ip route | grep 169.254
(no output)
# ping -c3 -I ens224 192.168.10.12
# ping -c3 -I ens256 192.168.11.12
# ping -c3 -M do -s 8972 -I ens224 192.168.10.12
3 packets transmitted, 3 received, 0% packet lossA ping: local error: message too long or 100% loss on the large ping means an MTU mismatch somewhere in the path. Fix that before installing; MTU problems on the interconnect show up later as lost blocks and “gc” wait events, not as clean errors.
Step 2: Classify the interfaces during installation
In the Grid Infrastructure installer’s Specify Network Interface Usage screen, set bond0 to Public and both ens224 and ens256 to ASM & Private. For a silent install, the response file line is:
oracle.install.crs.config.networkInterfaceList=bond0:10.10.10.0:1,ens224:192.168.10.0:5,ens256:192.168.11.0:5Interface type 1 is public, 2 private, 3 do not use, 4 ASM and 5 ASM & private. The cluster verification utility checks the same thing before you install:
$ ./runcluvfy.sh comp nodecon -n rac1,rac2 -networks ens224,ens256 -verbose
...
Verification of node connectivity was successful.Step 3: Verify HAIP after installation
The HAIP resource
$ crsctl stat res ora.cluster_interconnect.haip -init
NAME=ora.cluster_interconnect.haip
TYPE=ora.haip.type
TARGET=ONLINE
STATE=ONLINE on rac1HAIP is a lower-stack resource, so it only shows with -init. The registered interfaces come from oifcfg:
$ oifcfg getif
bond0 10.10.10.0 global public
ens224 192.168.10.0 global cluster_interconnect,asm
ens256 192.168.11.0 global cluster_interconnect,asm
$ oifcfg iflist -p -n
bond0 10.10.10.0 PRIVATE 255.255.255.0
ens224 192.168.10.0 PRIVATE 255.255.255.0
ens224 169.254.0.0 UNKNOWN 255.255.224.0
ens256 192.168.11.0 PRIVATE 255.255.255.0
ens256 169.254.32.0 UNKNOWN 255.255.224.0The PRIVATE/PUBLIC column in iflist -p only reports whether the address is in an RFC 1918 range, not how Clusterware uses it; getif is the source of truth. The exact 169.254 subnets and masks vary by release and by how many interfaces are in use, so treat the values above as an example.
The addresses on the interfaces
# ip -4 addr show ens224
4: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 ...
inet 192.168.10.11/24 brd 192.168.10.255 scope global noprefixroute ens224
inet 169.254.13.207/19 brd 169.254.31.255 scope global ens224:1What the instances use
SQL> select inst_id, name, ip_address, source
from gv$cluster_interconnects order by 1,2;
INST_ID NAME IP_ADDRESS SOURCE
---------- --------------- ---------------- -------------------------------
1 ens224:1 169.254.13.207
1 ens256:1 169.254.44.91
2 ens224:1 169.254.7.18
2 ens256:1 169.254.60.140An empty SOURCE column is what you want: the addresses came from Clusterware. If it says cluster_interconnects parameter, someone set the CLUSTER_INTERCONNECTS initialization parameter, which overrides HAIP and removes its failover for that instance. Oracle recommends leaving that parameter unset and using Redundant Interconnect Usage instead.
Step 4: Test an interconnect failure
Do this in a test window with the cluster running. Watch the Grid Infrastructure alert log on rac1 in one window, then take one private port down in another (pulling the cable or disabling the switch port is the more realistic test):
$ tail -f /u01/app/grid/diag/crs/rac1/crs/trace/alert.log
# ip link set dev ens224 down
# ip -4 addr show ens256 | grep 169.254
inet 169.254.44.91/19 brd 169.254.63.255 scope global ens256:1
inet 169.254.13.207/19 brd 169.254.31.255 scope global ens256:2
# ip link set dev ens224 up
# sleep 30; ip -4 addr show ens224 | grep 169.254
inet 169.254.13.207/19 brd 169.254.31.255 scope global ens224:1Both HAIP addresses sit on ens256 while ens224 is down, and the address returns once the link is back. The node must not be evicted and the database must stay open. crsctl stat res -t should look exactly the same before and after. Losing both private links on one node is a different test: that node is evicted after the CSS misscount (30 seconds on Linux by default), which is the correct behaviour.
Using a bond instead
If you choose an OS bond for the interconnect, build bond1 exactly as in part 4, bonding the public network (active-backup with miimon=100 is the safe choice) on the two private ports, give it one private subnet, and classify only bond1 as ASM & Private in the installer:
# nmcli con add type bond ifname bond1 con-name bond1 \
bond.options "mode=active-backup,miimon=100,primary=ens224" \
ipv4.method manual ipv4.addresses 192.168.10.11/24 \
ipv4.never-default yes ipv6.method disabled 802-3-ethernet.mtu 9000
# nmcli con add type ethernet ifname ens224 master bond1 con-name bond1-ens224
# nmcli con add type ethernet ifname ens256 master bond1 con-name bond1-ens256
# nmcli con up bond1
$ oifcfg getif
bond0 10.10.10.0 global public
bond1 192.168.10.0 global cluster_interconnect,asmWith a bond on two switches, those switches must carry the same VLAN and be linked to each other, otherwise a node whose active port is on switch P1 cannot reach a node whose active port is on switch P2. That cross-switch requirement does not exist with HAIP on separate subnets.
Changing an existing cluster between HAIP and a bond
Interconnect changes are recorded in the OCR and the GPnP profile, so plan a maintenance window, take an OCR backup first, and follow Oracle’s procedure for changing the private network. The outline, run as the grid owner from one node, for moving from two HAIP interfaces to bond1:
# /u01/app/19.0.0/grid/bin/ocrconfig -manualbackup (as root)
$ oifcfg setif -global bond1/192.168.10.0:cluster_interconnect,asm
$ oifcfg getif (bond1 now listed)
-- stop the clusterware stack on every node (as root), then build bond1
# crsctl stop crs
# crsctl disable crs (optional, so a reboot does not start a half-configured node)
... rebuild the private networking as bond1 on every node ...
# crsctl enable crs
# crsctl start crs
$ oifcfg delif -global ens224/192.168.10.0
$ oifcfg delif -global ens256/192.168.11.0
$ oifcfg getifThe rule is to add the new interface before stopping the stack and remove the old ones only once the cluster is running on the new interface: if the GPnP profile lists no working private interface when CRS starts, the stack will not come up. Moving the other way (bond to HAIP) is the same pattern with setif for ens224 and ens256 and delif for bond1. Check the current My Oracle Support note on modifying the private network for your exact release before you start.
Troubleshooting and gotchas
- HAIP resource OFFLINE or failing to start: look in
$ORACLE_BASE/diag/crs/<node>/crs/trace/ohasd_orarootagent_root.trc. Common causes are an existing route or address in 169.254.0.0/16, or the private interface being down when the stack starts. - Cloud and virtual platforms that use 169.254.169.254 for instance metadata: HAIP addresses can clash with the metadata route. Oracle’s cloud images handle this; on your own VMs, check
ip routefor 169.254 routes before installing and read the platform’s RAC guidance. - Both private interfaces on the same subnet: it may appear to work, but replies leave by the wrong interface, strict rp_filter drops them, and failover misbehaves. Use one subnet per interface.
- rp_filter=1 on the private NICs: after a HAIP move, packets arrive on an interface that does not own the route and the kernel drops them. Set
net.ipv4.conf.<if>.rp_filter = 2in/etc/sysctl.d/. - MTU mismatch: small pings succeed and Cache Fusion still suffers. Test with
ping -M do -s 8972across every path, and look forgc cr block lostorgc current block lostin AWR. - CLUSTER_INTERCONNECTS set in the spfile: instances bypass HAIP. Reset it with
alter system reset cluster_interconnects scope=spfile sid='*';and restart the instances in a rolling fashion. - Firewall on the interconnect: the private interfaces belong in a zone that allows all traffic between the nodes (part 3 used the trusted zone). Blocking UDP between nodes causes evictions that look like network failures.
- Network diagnostics:
oifcfg iflist -p -n,gv$cluster_interconnects, the GI alert log, andcluvfy comp nodecontogether usually pinpoint the problem.
Oracle Infrastructure Series: this is part 5. Previous: part 4, bonding the public network. Earlier: part 3, preparing Oracle Linux 9, part 2, planning the RAC cluster and part 1, the overview. Next is part 6, configuring DNS and GNS for SCAN.
References: Oracle’s Grid Infrastructure Installation and Upgrade Guide 19c for Linux and the 26ai private IP interface configuration requirements (Redundant Interconnect Usage), the Clusterware Administration and Deployment Guide 19c for oifcfg, and My Oracle Support notes 1210883.1 and 1664291.1 on ora.cluster_interconnect.haip.