Oracle Infrastructure Series, part 2. Previous: part 1, the Oracle infrastructure overview. Next up, part 3: preparing Oracle Linux 9 for Grid Infrastructure and RAC.
Almost every painful Oracle RAC installation I have seen went wrong before anyone ran gridSetup.sh. The SCAN name was in /etc/hosts instead of DNS, the interconnect shared a switch with backup traffic, one node called its private NIC ens224 and the other ens256, or the voting disk group had two disks when it needed three. None of that can be fixed with a clever installer flag. It has to be right on paper first.
This post is the planning worksheet for a two-node Oracle RAC cluster on Oracle Linux 9 with Grid Infrastructure and Database 19c. It covers servers, shared storage and ASM disk groups, the three networks (public, private interconnect and VIPs), SCAN, and the DNS records you need. Notes for Oracle Database 23ai and Oracle AI Database 26ai are called out where they differ. By the end you will have an IP plan, a storage plan and a DNS zone you can hand to the network and storage teams.
Prerequisites and assumptions
- Two (or more) x86-64 servers that will run Oracle Linux 9 with UEK 7 or the Red Hat compatible kernel. Grid Infrastructure 19c is supported on Oracle Linux 9 starting with Release Update 19.19, so plan to install with a current RU applied at install time (
gridSetup.sh -applyRU). - Shared block storage (SAN over Fibre Channel or iSCSI, or NVMe over fabrics) that can present the same LUNs to every node.
- Access to the DNS server for your domain, or a DNS team that will add records for you.
- At least two physical network paths per node: one for the public network and one or more for the private interconnect.
- The architecture picture from part 1, the Oracle infrastructure overview.
The reference architecture
Here is the cluster this series builds. Two nodes, one SAN, a public network that clients use, and a private network that only the cluster uses:
Clients / application servers
|
DNS: rac-scan.example.com -> 3 SCAN IPs
|
========================= PUBLIC NETWORK 10.10.10.0/24 (bond0) ======
| rac1 10.10.10.11 | rac2 10.10.10.12
| rac1-vip 10.10.10.21 | rac2-vip 10.10.10.22
| SCAN VIPs 10.10.10.31-33 float across both nodes
+-----+--------------+ +-----+--------------+
| rac1 | | rac2 |
| GI 19c + DB 19c | | GI 19c + DB 19c |
| ASM instance +ASM1 | | ASM instance +ASM2 |
+--+-----------+------+ +--+-----------+------+
| | | |
======+== PRIVATE INTERCONNECT 192.168.10.0/24 (ens224) ==+=====
======+== PRIVATE INTERCONNECT 192.168.11.0/24 (ens256) ==+=====
| |
+------- dual FC/iSCSI paths ---------+
|
SAN: +CRS, +DATA, +RECO LUNs (DM-Multipath)The rest of the post fills in each box in that picture.
Step 1: Size the servers
Oracle’s documented minimums for Grid Infrastructure 19c on Linux are modest, and you should plan well above them. The table shows the minimums next to what is sensible for a small production cluster:
| Resource | Oracle minimum (19c) | Practical starting point |
|---|---|---|
| RAM | 8 GB for Grid Infrastructure | 64 GB or more; the SGA and PGA of every instance on the node come on top of Clusterware and ASM |
| Swap | Equal to RAM for 4 to 16 GB of RAM; 16 GB above that (exclude HugePages from the calculation) | 16 GB |
| /tmp | 1 GB free | 5 GB or more |
| Grid home | 12 GB | 50 GB or more, so out-of-place patching can hold two homes |
| Oracle base (logs, diag) | About 10 GB | 50 GB or more; trace and audit files grow |
| Database home | About 7 to 8 GB | 50 GB or more, for the same patching reason |
| CPU | x86-64 | Identical CPU model and count on every node |
Two rules matter more than the numbers. First, make the nodes identical: same CPU, same RAM, same kernel, same patch level, same interface names. Clusterware tolerates small differences, but services, workload balancing and capacity planning all assume any node can carry the load of any other. Second, size for N-1. If one node of a two-node cluster dies, the survivor gets every session. If it can’t carry that load, RAC gives you availability on paper only.
Put /u01 (the Oracle software) on its own local file system, separate from /, so a runaway trace file can’t fill the root file system and hang the node.
Step 2: Plan the networks
A RAC node needs at least two networks, and every node must use the same interface name for the same role. If bond0 is public on rac1, it must be public on rac2. The installer and Clusterware store the network definition by interface name and subnet, not per node.
Public network
This is the network clients reach. Each node has a static public IP, and Clusterware adds a node VIP and the SCAN VIPs on the same subnet. Bond two NICs (active-backup is the safe default) for the public interface so a single cable or switch port failure doesn’t make the node look dead to clients. Bonding on Oracle Linux 9 gets its own post (part 4); the older bond interface guide for Oracle Linux 6 shows the concept if you want a head start.
Private interconnect
The interconnect carries Cache Fusion (blocks shipped between instances’ buffer caches) and the Clusterware network heartbeat. Latency here becomes latency for every RAC query that touches a block mastered on another node, and missed heartbeats cause node evictions. Plan it like this:
- Use a dedicated, non-routable private subnet on its own VLAN or physical switches. Don’t share it with backup, storage replication or vMotion traffic.
- Use 10 GbE or faster. Oracle’s minimum is 1 GbE, but that is a lab number.
- Crossover cables between nodes are not supported, even for two nodes. Go through a switch.
- If you enable jumbo frames (MTU 9000), set the same MTU on every node NIC and every switch port in the path. A mismatch causes intermittent hangs that are hard to diagnose.
- For redundancy, either give Clusterware up to four separate private interfaces and let Redundant Interconnect Usage (HAIP) spread traffic across them, or bond the interfaces at the OS level. Oracle recommends each HAIP interface be on its own subnet, as in the diagram above.
- Don’t use 169.254.0.0/16 anywhere. HAIP assigns link-local addresses from that range on top of your private interfaces.
Node VIPs
Each node gets one virtual IP on the public subnet. When a node fails, Clusterware moves its VIP to a surviving node, where the listener answers connection attempts with an immediate error instead of letting clients wait for a TCP timeout. The VIP address must be unused before installation: it should resolve in DNS but must not answer a ping.
SCAN (Single Client Access Name)
SCAN is one host name for the whole cluster. Clients connect to rac-scan.example.com:1521/service_name and never need to know which nodes exist. Plan it with these rules:
- One name that resolves to three IP addresses in DNS, returned round robin. Oracle recommends three regardless of cluster size.
- The SCAN IPs are on the public subnet and, like VIPs, must be unused before installation.
- Don’t put SCAN in
/etc/hosts. A hosts file can only return one address, so you lose load balancing and failover across SCAN listeners, andcluvfyflags it. - The alternative is Grid Naming Service (GNS), where Clusterware hands out the addresses through DHCP and a delegated subdomain. Static DNS is simpler and is what this series uses.
- The SCAN name must be unique and shouldn’t start with a digit. Keep it short, since the cluster name has a 15-character limit and many sites derive one from the other.
The IP plan
Here is the complete address plan for the two-node example. Fill in the same table for your own cluster before you talk to anyone else:
| Name | Type | Address | Resolved by | Interface |
|---|---|---|---|---|
| rac1.example.com | Public, static | 10.10.10.11 | DNS and /etc/hosts | bond0 |
| rac2.example.com | Public, static | 10.10.10.12 | DNS and /etc/hosts | bond0 |
| rac1-vip.example.com | Node VIP | 10.10.10.21 | DNS and /etc/hosts | bond0 (managed by Clusterware) |
| rac2-vip.example.com | Node VIP | 10.10.10.22 | DNS and /etc/hosts | bond0 (managed by Clusterware) |
| rac-scan.example.com | SCAN VIPs | 10.10.10.31, .32, .33 | DNS only | bond0 (managed by Clusterware) |
| rac1-priv1 / rac2-priv1 | Interconnect 1 | 192.168.10.11 / .12 | /etc/hosts | ens224 |
| rac1-priv2 / rac2-priv2 | Interconnect 2 | 192.168.11.11 / .12 | /etc/hosts | ens256 |
That is nine public addresses for two nodes (two hosts, two VIPs, three SCAN, plus room to grow) and four private ones. Each additional node needs one more public IP, one more VIP and one private IP per interconnect.
Step 3: Create the DNS records
If you run BIND (the site has an older guide to installing BIND), the forward zone entries look like this. Note the three A records with the same name for SCAN:
; example.com forward zone - Oracle RAC cluster "rac"
rac1 IN A 10.10.10.11
rac2 IN A 10.10.10.12
rac1-vip IN A 10.10.10.21
rac2-vip IN A 10.10.10.22
rac-scan IN A 10.10.10.31
rac-scan IN A 10.10.10.32
rac-scan IN A 10.10.10.33Add matching PTR records in the reverse zone (10.10.10.in-addr.arpa). Reverse lookups are not strictly required for SCAN to work, but missing PTR records cause slow logins and confusing warnings from tools that do reverse lookups. If your DNS is Active Directory, create the same records as static A and PTR records and make sure scavenging can’t delete them.
Also put the public, VIP and private names in /etc/hosts on every node so the cluster keeps resolving its own names if DNS is slow. Leave the SCAN out of it:
# /etc/hosts on rac1 and rac2
127.0.0.1 localhost localhost.localdomain
10.10.10.11 rac1.example.com rac1
10.10.10.12 rac2.example.com rac2
10.10.10.21 rac1-vip.example.com rac1-vip
10.10.10.22 rac2-vip.example.com rac2-vip
192.168.10.11 rac1-priv1
192.168.10.12 rac2-priv1
192.168.11.11 rac1-priv2
192.168.11.12 rac2-priv2Don’t map the host name to 127.0.0.1 or ::1. Clusterware needs the node name to resolve to its public address.
Verify DNS before installation
From each node, check that the SCAN returns all three addresses and that the order rotates between queries:
[root@rac1 ~]# dig +short rac-scan.example.com
10.10.10.32
10.10.10.33
10.10.10.31
[root@rac1 ~]# dig +short rac-scan.example.com
10.10.10.33
10.10.10.31
10.10.10.32
[root@rac1 ~]# dig +short -x 10.10.10.21
rac1-vip.example.com.
[root@rac1 ~]# getent hosts rac2-vip
10.10.10.22 rac2-vip.example.com rac2-vipThen confirm the VIPs and SCAN addresses are not in use yet. Every one of these pings should fail before installation:
[root@rac1 ~]# for ip in 10.10.10.21 10.10.10.22 10.10.10.31 10.10.10.32 10.10.10.33; do
> ping -c1 -W1 $ip >/dev/null && echo "$ip IN USE - fix before install" || echo "$ip free"
> done
10.10.10.21 free
10.10.10.22 free
10.10.10.31 free
10.10.10.32 free
10.10.10.33 freedig comes from the bind-utils package (dnf install -y bind-utils).
Step 4: Plan the shared storage and ASM disk groups
Every node must see the same shared LUNs, and every LUN must reach each node over at least two paths combined by DM-Multipath (covered in part 7). On top of multipath, ASM needs persistent, correctly owned device names. As part 1 explained, on Oracle Linux 9 use udev rules or ASMLib v3 for that, because the ASM Filter Driver’s filtering is disabled on kernel 5.14 and later and AFD is desupported in 23ai and 26ai.
Disk group layout
A layout that works for most clusters is three disk groups:
| Disk group | Holds | Typical redundancy | Example LUNs |
|---|---|---|---|
| +CRS (or +OCR) | OCR, voting files, ASM password file, ASM SPFILE | NORMAL (3 failure groups) or EXTERNAL | 3 x 10 GB |
| +DATA | Data files, control files, online redo, SPFILE | EXTERNAL when the array protects data; NORMAL for host mirroring | Same-size LUNs, for example 4 x 500 GB |
| +RECO | Fast recovery area: archived logs, flashback logs, RMAN backups, control file copy | Same as +DATA | Sized for your backup retention, often 1 to 2 times +DATA |
Keeping the Clusterware files in their own small disk group means you can rebuild or migrate +DATA without touching the files that keep the cluster alive.
Voting file math
The voting file count depends on the redundancy of the disk group that holds it, and this is the planning mistake I see most often:
| Disk group redundancy | Voting files | Failure groups (disks) needed |
|---|---|---|
| EXTERNAL | 1 | 1 |
| NORMAL | 3 | 3 (a two-disk NORMAL group can hold data, but not voting files) |
| HIGH | 5 | 5 |
| FLEX | 3 or 5 | 3 or 5 |
A node must be able to access a majority of the voting files to stay in the cluster. With a stretched cluster across two sites, place the third voting file on a quorum failure group at a third location (an NFS share works); that design comes later in the series.
LUN rules
- Make every LUN in a disk group the same size and from the same tier of storage. ASM spreads extents evenly, so one slow or small disk slows or limits the whole group.
- Present whole LUNs; don’t partition them unless your storage team or naming method needs it.
- Use the same LUN IDs (WWIDs) on every node, then map them to friendly names such as
/dev/mapper/crs01or ASMLib labels. - Ask for LUNs you can grow. Adding a disk and letting ASM rebalance online is routine; that procedure gets its own post.
Checking what a node sees
Once the storage team has presented the LUNs, run these on each node and compare. The WWIDs must match across nodes:
[root@rac1 ~]# multipath -ll | grep -E '^[a-z0-9]'
crs01 (36000d31000e44200000000000000001a) dm-2 COMPELNT,Compellent Vol
crs02 (36000d31000e44200000000000000001b) dm-3 COMPELNT,Compellent Vol
crs03 (36000d31000e44200000000000000001c) dm-4 COMPELNT,Compellent Vol
data01 (36000d31000e44200000000000000002a) dm-5 COMPELNT,Compellent Vol
reco01 (36000d31000e44200000000000000003a) dm-6 COMPELNT,Compellent Vol
[root@rac1 ~]# lsblk -o NAME,SIZE,TYPE /dev/mapper/crs01
NAME SIZE TYPE
crs01 10G mpathThe vendor strings and WWIDs above are sample output; yours will reflect your array.
Step 5: Time, names and the rest of the checklist
- Time synchronization. Run chrony against the same time sources on every node. If no NTP service is configured, Clusterware’s Cluster Time Synchronization Service (CTSS) runs in active mode and adjusts time itself; with chrony running, CTSS stays in observer mode, which is what you want.
- Cluster name. Up to 15 characters, letters, digits and hyphens, unique across your network. Example:
rac-prod1. - Host names. Lowercase, no underscores, and resolvable both short and fully qualified.
- Users and groups. Decide UIDs and GIDs now (for example
gridandoracle, withoinstall,asmadmin,asmdba,dbaand friends) and keep them identical on every node. Part 3 creates them. - SSH. The installer sets up passwordless SSH between nodes for the
gridandoracleusers, so SSH must be allowed between the public addresses. - Firewall and SELinux. Oracle’s installation guide expects the cluster networks to be open between nodes; most sites disable firewalld on the interconnect or trust the private interfaces. Decide your policy before installing rather than during troubleshooting.
- Grid Infrastructure Management Repository (GIMR). Optional in 19c and desupported in 23ai, so most new 19c clusters skip it. If you configure it, plan extra space in its disk group.
Version notes: 23ai and 26ai
- SCAN, node VIPs and the public and private network model are unchanged in Grid Infrastructure 23ai and 26ai. A plan made for 19c carries over.
- ASM Filter Driver is desupported in 23ai and later, so plan udev or ASMLib v3 naming from the start if an upgrade is on your roadmap.
- GIMR is desupported in 23ai, so there is nothing to size for it.
- Check the certification matrix on My Oracle Support for the exact Oracle Linux 9 kernel and release combinations before you order hardware.
Verification: the pre-install checklist
Before moving on, you should be able to answer yes to every line:
- All nodes run the same Oracle Linux 9 release and kernel, with identical CPU and RAM.
- Interface names for public and private networks are identical on every node.
- The SCAN name resolves to three addresses from every node, and none of the VIP or SCAN addresses answer ping.
- Public, VIP and private names are in
/etc/hostson every node; SCAN is not. - The interconnect is on a dedicated non-routable subnet through a switch, with consistent MTU.
- Every node sees the same shared LUNs with the same WWIDs over at least two paths.
- The voting file disk group has enough failure groups for its redundancy level.
- chrony is configured and synchronized on every node.
Part 10 of the series runs Oracle’s Cluster Verification Utility (cluvfy stage -pre crsinst), which checks most of this automatically. Doing the checklist by hand first means cluvfy confirms a good plan instead of discovering a bad one.
Troubleshooting and gotchas
- “PRVF-4664: Found inconsistent name resolution entries for SCAN name”: the SCAN resolves differently on different nodes, usually because one node has it in
/etc/hostsor points at a different DNS server. Remove the hosts entry and check/etc/resolv.confon every node. - Only one SCAN IP comes back: check for a hosts-file entry, a caching resolver returning a single record, or a DNS server with round robin disabled.
- The installer says a VIP is already in use: another host has the address. Find it with
arping -I bond0 10.10.10.21and get a new address rather than “borrowing” it. - Interface names differ between nodes: rename them with NetworkManager connection profiles or udev link files before installation. Changing the network definition after installation is possible with
oifcfgbut is a maintenance window. - Intermittent node evictions under load: almost always the interconnect. Look for MTU mismatches, a shared or oversubscribed switch, or a bond with mismatched settings on the two nodes.
- Voting files won’t go into a NORMAL redundancy disk group: the group has fewer than three failure groups. Add a third disk (or a quorum failure group) and try again.
Oracle Infrastructure Series: this is part 2. Previous: part 1, the Oracle infrastructure overview. Next is part 3, OS preparation for Oracle Grid and RAC on Oracle Linux 9: the preinstall package, users and groups, limits, HugePages and chrony.
References: Oracle’s Grid Infrastructure Installation and Upgrade Guide 19c for Linux (network, storage and server checklists), Clusterware Administration and Deployment Guide 19c, and Automatic Storage Management Administrator’s Guide 19c.