Oracle Infrastructure Series, part 3. Previous: part 2, planning the RAC cluster. Next up, part 4: bonded network interfaces for the public network and interconnect.
With the cluster planned, every node needs the same operating system baseline before Grid Infrastructure goes on: kernel parameters, the Oracle software owners and OS groups, shell limits, HugePages, time synchronization and a few systemd settings that quietly break clusters if you leave them at the defaults. This post builds that baseline on Oracle Linux 9 for Oracle Grid Infrastructure and Database 19c, with notes where 23ai/26ai differ. Run every step on every node (rac1 and rac2 in our example) and keep them identical.
What we are building
rac1 / rac2 (Oracle Linux 9, identical)
+-----------------------------------------------------------+
| oracle-database-preinstall-19c -> sysctl, limits, oracle |
| extra groups: asmadmin asmdba asmoper + user: grid |
| /u01/app/19.0.0/grid (grid home) /u01/app/grid (base) |
| /u01/app/oracle (db base) /u01/app/oraInventory |
| vm.nr_hugepages sized for all SGAs, THP disabled |
| chronyd synced (CTSS runs in observer mode) |
| RemoveIPC=no, avahi off, interconnect trusted in firewalld|
+-----------------------------------------------------------+Prerequisites
- Oracle Linux 9 (UEK 7 or RHCK) installed on each node with the “Server” or minimal environment, registered to ULN or using the public
yum.oracle.comrepositories (ol9_baseos_latestandol9_appstreamenabled). - The network and storage plan from part 2, planning the RAC cluster: host names, public IPs, VIPs, SCAN in DNS and interconnect subnets.
- Root access on every node, and the same RAM size on every node (HugePages and limits are sized from it).
- Remember the 19c-on-OL9 rule from part 1: install with Release Update 19.19 or later (
gridSetup.sh -applyRU). The OS prep below is the same either way.
Step 1: Update the OS and install the preinstall package
Oracle ships an RPM that installs the required packages, creates the oracle user and database groups, and writes the kernel parameters and shell limits. Use it instead of hand-typing values from the install guide.
# dnf -y update
# dnf -y install oracle-database-preinstall-19c
# rpm -q oracle-database-preinstall-19c
oracle-database-preinstall-19c-1.0-1.el9.x86_64For 23ai install oracle-database-preinstall-23ai instead; 26ai has its own preinstall package for that release, so check the name with dnf search preinstall. A few extra packages are worth adding for Grid Infrastructure and troubleshooting:
# dnf -y install ksh libnsl net-tools nfs-utils bind-utils sysstat unzip chronyThe package records what it changed. These are the files to review:
# ls /etc/sysctl.d/ /etc/security/limits.d/
/etc/sysctl.d/:
99-oracle-database-preinstall-19c-sysctl.conf 99-sysctl.conf
/etc/security/limits.d/:
oracle-database-preinstall-19c.conf
# ls /var/log/oracle-database-preinstall-19c/
backup resultsStep 2: Review the kernel parameters
The preinstall values are the documented minimums. Check that they are live:
# sysctl fs.file-max fs.aio-max-nr kernel.sem kernel.shmmni kernel.shmmax kernel.shmall \
kernel.panic_on_oops net.ipv4.ip_local_port_range net.core.rmem_max net.core.wmem_max
fs.file-max = 6815744
fs.aio-max-nr = 1048576
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmmax = 4398046511104
kernel.shmall = 1073741824
kernel.panic_on_oops = 1
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_max = 4194304
net.core.wmem_max = 1048576RAC adds one setting the preinstall package does not know about: reverse path filtering on the interconnect interfaces. With two private interfaces (HAIP, part 5) strict filtering (1) drops packets, so set them to loose mode (2). Put your own settings in a file that sorts after the preinstall file:
# cat > /etc/sysctl.d/99-zz-oracle-rac.conf <<'EOF'
# RAC private interconnect: loose reverse-path filtering
net.ipv4.conf.ens224.rp_filter = 2
net.ipv4.conf.ens256.rp_filter = 2
EOF
# sysctl --system | tail -3
* Applying /etc/sysctl.d/99-zz-oracle-rac.conf ...
net.ipv4.conf.ens224.rp_filter = 2
net.ipv4.conf.ens256.rp_filter = 2Step 3: Create the ASM groups and the grid user
The preinstall package creates oinstall, dba, oper, backupdba, dgdba, kmdba and racdba plus the oracle user. With role separation (recommended for RAC) Grid Infrastructure gets its own owner, grid, and the ASM groups. UIDs and GIDs must be identical on every node, so use fixed numbers:
# groupadd -g 54327 asmdba
# groupadd -g 54328 asmoper
# groupadd -g 54329 asmadmin
# useradd -u 54331 -g oinstall -G dba,asmadmin,asmdba,asmoper,racdba grid
# usermod -a -G asmdba oracle
# passwd grid
# passwd oracle
# id grid
uid=54331(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54329(asmadmin),54327(asmdba),54328(asmoper),54330(racdba)
# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba),54327(asmdba)Compare id grid and id oracle on all nodes; a single mismatched number causes ownership chaos on shared storage. If your users come from LDAP or FreeIPA, the IDs still have to match and the accounts must resolve locally when the directory is down, so most sites keep these two accounts in /etc/passwd.
Step 4: Shell limits for grid and oracle
The preinstall file only covers oracle. Add the same limits for grid. memlock must be at least the HugePages you allocate in step 6; Oracle suggests 90% of RAM in KB. For a 64 GB node that is about 60397977 KB:
# cat > /etc/security/limits.d/99-grid-oracle.conf <<'EOF'
grid soft nofile 1024
grid hard nofile 65536
grid soft nproc 16384
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid soft memlock 60397977
grid hard memlock 60397977
oracle soft memlock 60397977
oracle hard memlock 60397977
EOF
# su - grid -c 'ulimit -Sn; ulimit -Hn; ulimit -Su; ulimit -l'
1024
65536
16384
60397977Step 5: Directories
The Grid home must not sit under the grid user’s Oracle base, and after installation root.sh changes the Grid home’s parent path to root ownership. Create the standard layout:
# mkdir -p /u01/app/19.0.0/grid /u01/app/grid /u01/app/oracle /u01/app/oraInventory
# chown -R grid:oinstall /u01
# chown oracle:oinstall /u01/app/oracle
# chmod -R 775 /u01
# ls -ld /u01/app/*
drwxrwxr-x. 3 grid oinstall 18 Oct 31 10:02 /u01/app/19.0.0
drwxrwxr-x. 2 grid oinstall 6 Oct 31 10:02 /u01/app/grid
drwxrwxr-x. 2 oracle oinstall 6 Oct 31 10:02 /u01/app/oracle
drwxrwxr-x. 2 grid oinstall 6 Oct 31 10:02 /u01/app/oraInventoryGive /u01 its own file system with enough room for the Grid home, the database home, logs and staged patches; 100 GB per node is comfortable.
Step 6: HugePages and Transparent HugePages
An SGA in HugePages is not swappable, uses far less page-table memory and is supported by Oracle for production; Transparent HugePages (THP) is the opposite and Oracle recommends disabling it because it causes latency spikes and node evictions. First disable THP on the kernel command line:
# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
# grubby --update-kernel=ALL --args="transparent_hugepage=never"
# reboot
...
# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]Then size HugePages to cover every SGA on the node: the database instances plus the ASM instance. On x86_64 a huge page is 2 MB. Example: a 24 GB database SGA plus ASM (allow 3 GB) is 27 GB = 27648 MB / 2 = 13824 pages; add a small margin:
# grep Hugepagesize /proc/meminfo
Hugepagesize: 2048 kB
# echo "vm.nr_hugepages = 13900" > /etc/sysctl.d/99-zz-hugepages.conf
# sysctl --system >/dev/null
# grep -E 'HugePages_(Total|Free)' /proc/meminfo
HugePages_Total: 13900
HugePages_Free: 13900If the kernel cannot find enough contiguous memory at runtime, HugePages_Total comes back lower than requested; a reboot allocates them at boot. Once the database exists, set use_large_pages=ONLY so an instance refuses to start rather than silently running without HugePages, and recompute with the script from My Oracle Support note 401749.1 (hugepages_settings.sh) against the running instances. Do not use MEMORY_TARGET (AMM); it is incompatible with HugePages, so use SGA_TARGET and PGA_AGGREGATE_TARGET.
Step 7: Time synchronization with chrony
Cluster nodes must agree on time. If an NTP service (chronyd) is configured, Oracle’s Cluster Time Synchronization Service (CTSS) runs in observer mode and leaves the clock alone; if none is found, CTSS goes active and adjusts time itself. Use chrony with your site’s time servers, and allow stepping only at startup so the clock is never jumped under a running cluster:
# vi /etc/chrony.conf
server ntp1.example.com iburst
server ntp2.example.com iburst
makestep 1.0 3
rtcsync
# systemctl enable --now chronyd
# chronyc tracking | grep -E 'Reference|System time|Leap'
Reference ID : 0A0A0A05 (ntp1.example.com)
System time : 0.000012841 seconds fast of NTP time
Leap status : Normal
# chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* ntp1.example.com 2 6 377 34 +18us[ +22us] +/- 1.2ms
^+ ntp2.example.com 2 6 377 35 -41us[ -37us] +/- 1.6msAfter Grid is installed you will see the observer mode confirmed:
$ crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.Step 8: systemd, avahi, firewall and SELinux
RemoveIPC: systemd-logind removes a user’s shared memory and semaphores when that user’s last session ends. If the grid or oracle account logs out, ASM and the database can crash. Oracle documents setting it to no (MOS note 2081410.1):
# sed -i 's/^#\?RemoveIPC=.*/RemoveIPC=no/' /etc/systemd/logind.conf
# grep ^RemoveIPC /etc/systemd/logind.conf
RemoveIPC=no
# systemctl restart systemd-logindavahi-daemon uses multicast and can interfere with the interconnect; disable it on cluster nodes:
# systemctl disable --now avahi-daemon.socket avahi-daemon.serviceFirewall: the private interconnect must be completely open between nodes. Put the interconnect interfaces in the trusted zone and open the listener on the public side (see the firewalld cheat sheet for more):
# firewall-cmd --permanent --zone=trusted --add-interface=ens224
# firewall-cmd --permanent --zone=trusted --add-interface=ens256
# firewall-cmd --permanent --zone=public --add-port=1521/tcp
# firewall-cmd --reload
# firewall-cmd --get-active-zones
public
interfaces: bond0
trusted
interfaces: ens224 ens256Many sites simply stop firewalld on RAC nodes that sit behind a network firewall; either way, nothing may block traffic between the nodes on the interconnect. SELinux can stay enforcing for 19c on Oracle Linux; the preinstall package does not change it. If you set it to permissive for the install, record that decision rather than leaving it by accident.
Step 9: Environment for grid and oracle
# su - grid
$ cat >> ~/.bash_profile <<'EOF'
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/19.0.0/grid
export ORACLE_SID=+ASM1 # +ASM2 on rac2
export PATH=$ORACLE_HOME/bin:$PATH
umask 022
EOFThe oracle user gets ORACLE_BASE=/u01/app/oracle and its database home (created in a later part). Do not set ORACLE_HOME in the profile while running the installers if it points at a home that does not exist yet.
Verification checklist
Run this on each node and diff the output between nodes; it should be identical apart from host names:
# for u in grid oracle; do id $u; su - $u -c 'ulimit -n -u -l -s' ; done
# sysctl kernel.sem fs.file-max vm.nr_hugepages net.ipv4.conf.ens224.rp_filter
# grep -E 'HugePages_Total|Hugepagesize' /proc/meminfo
# cat /sys/kernel/mm/transparent_hugepage/enabled
# chronyc tracking | grep 'Leap status'
# grep ^RemoveIPC /etc/systemd/logind.conf
# systemctl is-enabled avahi-daemon.service chronyd
# ls -ld /u01/app/* /u01/app/19.0.0/gridCluster Verification Utility (runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose) checks all of this again before installation; we run it in a later part once storage and SSH equivalence are ready.
Troubleshooting and gotchas
- ASM or database instances die when someone logs out:
RemoveIPCis stillyes. Check/var/log/messagesfor the session end at the crash time and fix logind.conf on every node. - “PRVG-0449 / user equivalence” or ownership errors only on one node: UID or GID mismatch.
id gridmust print identical numbers everywhere. - HugePages_Total is lower than vm.nr_hugepages: memory was fragmented when you applied it. Reboot, or lower the value and confirm.
- Instance starts but HugePages_Free barely moves:
memlockis too small for the user that starts the instance (for databases started by Clusterware, the limits come from the Clusterware environment), orMEMORY_TARGETis set. The alert log prints a “Large Pages Information” block at startup showing how much SGA landed in HugePages. - cluvfy complains about the preinstall-set values on OL9 for 19c: older 19c base images do not recognize OL9. Use RU 19.19 or later with
-applyRU; as a temporary workaround for cluvfy only,export CV_ASSUME_DISTID=OL8. - Node evictions with “time drift” messages: chrony is not synchronized or was stepping the clock. Check
chronyc trackingand make suremakestepis limited to startup. - Interconnect pings work but cluster formation fails: firewalld is filtering the private interfaces, or
rp_filter=1is dropping HAIP traffic.
Oracle Infrastructure Series: this is part 3. Previous: part 2, planning the RAC cluster. Earlier: part 1, the Oracle infrastructure overview. Next is part 4, network bonding for Oracle RAC on Oracle Linux 9 with nmcli (active-backup vs LACP).
Related: if your DBA accounts authenticate with Kerberos, see configuring a Kerberos client; the steps carry over to Oracle Linux. Ship the nodes’ logs off-box with central rsyslog logging so an eviction leaves evidence.
References: Oracle’s Grid Infrastructure Installation and Upgrade Guide 19c for Linux (operating system, user and group, and HugePages configuration), Database Installation Guide 19c for Linux, and the Oracle Linux 9 documentation.