Oracle Infrastructure Series, part 1 of 60. Next up, part 2: planning an Oracle RAC cluster, covering hardware, shared storage, the public and private networks, VIPs, SCAN and DNS.
Most Oracle outages that turn into long nights aren’t caused by the database itself. They come from the layers around it: a cluster node that gets evicted, a SAN path that disappears, a disaster recovery site nobody has tested, or a replication link that quietly stopped. Oracle sells a separate answer to each of those problems: Real Application Clusters (RAC), Grid Infrastructure, Automatic Storage Management (ASM), Data Guard and GoldenGate. The names get thrown around together so often that it’s easy to lose track of which one does what.
This first post in the series maps out the whole stack. It explains what each component does, which failure it protects you from, how the pieces connect, and which commands you use to see each layer on a running system. Everything else in the series builds on this picture. The platform throughout is Oracle Linux 9 with Oracle Database 19c, with notes where Oracle Database 23ai and its successor, Oracle AI Database 26ai, behave differently.
The big picture: one database, five layers of protection
Think of an Oracle high availability deployment as layers. Each layer protects against a bigger failure than the one below it:
| Layer | Component | Protects against | Scope |
|---|---|---|---|
| Storage | ASM (with DM-Multipath underneath) | Disk failure, path failure, storage hot spots | One cluster |
| Cluster | Grid Infrastructure (Oracle Clusterware) | Server crash, NIC or process failure, OS hangs | One site |
| Database instances | Oracle RAC | Instance or node loss with the database still open on the surviving nodes | One site |
| Site | Data Guard | Loss of the whole data center, corruption, human error (with Flashback) | Two or more sites |
| Logical replication | GoldenGate | Zero-downtime migrations and upgrades, active-active multi-site, feeding other systems | Any number of sites |
Here is how a typical two-site deployment fits together. The primary site runs a two-node RAC database on Grid Infrastructure and ASM. The DR site runs a physical standby, also on RAC. GoldenGate is optional and sits beside Data Guard, not instead of it.
Clients / application servers
(connect to SCAN + service name, receive FAN events)
|
+---------------------------+-------------------------------+
| |
PRIMARY SITE (dc1) DR SITE (dc2)
+-------------------------------------+ +-------------------------------------+
| SCAN: rac-scan.example.com (3 IPs) | | SCAN: stby-scan.example.com |
| | | |
| rac1 (orcl1) rac2 (orcl2) | | stby1 (orcl1) stby2 (orcl2) |
| VIP rac1-vip VIP rac2-vip | | VIP stby1-vip VIP stby2-vip |
| \ private interconnect / | | \ private interconnect / |
| \ (Cache Fusion, CSS) / | | \ / |
| Grid Infrastructure: CRS, CSS, | redo | Grid Infrastructure |
| OCR, voting files, listeners |======>| Physical standby (Redo Apply) |
| ASM: +DATA +RECO (+OCR/vote) | Data | ASM: +DATA +RECO |
| DM-Multipath -> SAN array A | Guard | DM-Multipath -> SAN array B |
+-------------------------------------+ +-------------------------------------+
| |
+---- GoldenGate Extract --> trail --> Distribution path ---+--> Replicat
(optional: migrations, active-active, other targets)The rest of this post walks up that diagram from the bottom.
Layer 1: shared storage, multipath and ASM
Multipath underneath
A RAC cluster needs storage that every node can see at the same time. That’s usually Fibre Channel or iSCSI LUNs from a SAN, with at least two paths from each server. On Oracle Linux 9, the Device Mapper Multipath (dm-multipath) driver combines those paths into one device such as /dev/mapper/asm_data01, so losing a cable, HBA or switch port doesn’t take the disk away. Older posts here cover formatting a LUN on Oracle Linux. Part 7 of this series covers multipath.conf, WWIDs and aliases for Oracle in detail.
ASM: Oracle’s volume manager and file system
Automatic Storage Management sits on top of those devices. ASM groups disks into disk groups and stripes every file across all disks in the group. It can also mirror across failure groups: normal redundancy keeps two copies and high redundancy keeps three. When you add or drop a disk, ASM rebalances data online. A typical layout looks like this:
+DATAholds datafiles, control files, online redo logs and the server parameter file (spfile).+RECOis the Fast Recovery Area: archived logs, RMAN backups, flashback logs and a second copy of the control file and redo logs.+OCRor+CRS(optional) holds the Clusterware OCR and voting files, kept apart from the database files so they’re easier to manage.
ASM runs as its own instance (+ASM1, +ASM2) and belongs to Grid Infrastructure, not to the database home. Since 12.2, Flex ASM is the default: database instances can reach an ASM instance on another node if their local one goes down.
Device naming on Oracle Linux 9: ASMLib v3, not AFD
ASM needs disk names that stay the same across reboots, with the right ownership. You have three options: udev rules, Oracle ASMLib, or the ASM Filter Driver (AFD). AFD used to be the recommended choice, but the version picture has changed:
- Oracle’s 19c documentation states that ASM filtering is disabled starting with Linux kernel 5.14. Oracle Linux 9’s Red Hat Compatible Kernel is 5.14, and UEK 7 is 5.15. So on OL9 you don’t get AFD’s main benefit, which is blocking stray non-Oracle writes to ASM disks.
- ASM Filter Driver is desupported starting with 23ai. If you plan to upgrade Grid Infrastructure from 19c to 23ai or 26ai, you have to move off AFD as part of the upgrade.
- Oracle’s recommended replacement is ASMLib v3 (
oracleasmlibandoracleasm-support3.x), which Oracle Linux 8 and 9 require if you use ASMLib at all. It uses the kernel’s built-inio_uringinterface and adds its own I/O filter on OL9.
The practical advice for a new OL9 build: use ASMLib v3 or udev rules, and don’t start new clusters on AFD. Part 8 covers all three methods, including how to move an existing cluster from AFD to ASMLib.
# List ASM disk groups (run as the grid user with the +ASM environment set)
[grid@rac1 ~]$ asmcmd lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 204800 151230 0 151230 0 N DATA/
MOUNTED NORMAL N 512 512 4096 4194304 30720 29640 10240 9700 0 Y OCR/
MOUNTED EXTERN N 512 512 4096 4194304 409600 371880 0 371880 0 N RECO/
# ASMLib v3 view of the same disks
[root@rac1 ~]# oracleasm listdisks
DATA01
DATA02
OCR01
OCR02
OCR03
RECO01Layer 2: Grid Infrastructure and Oracle Clusterware
Grid Infrastructure is the software you install first, in its own Oracle home and owned by its own user, usually grid. It contains two things: Oracle Clusterware and ASM. Clusterware turns separate servers into a cluster. It tracks which nodes are members, starts and restarts resources, and fences a node that stops responding by evicting it.
The main pieces, which part 12 covers in depth:
- OHASD (Oracle High Availability Services daemon) is started by systemd and brings up the rest of the stack.
- CSS (Cluster Synchronization Services) handles membership and heartbeats over the private network and the voting files. It’s the component that evicts a node.
- CRS (Cluster Ready Services) manages resources: databases, instances, listeners, VIPs, services and disk groups.
- OCR (Oracle Cluster Registry) holds the cluster configuration, and the OLR is each node’s local copy. The voting files decide who survives a split brain. In a new 19c install, both the OCR and the voting files live in an ASM disk group.
- Networks: a public network carries the node IPs and node VIPs. A private interconnect, which should be redundant, carries heartbeats and Cache Fusion traffic. The SCAN (Single Client Access Name) is one DNS name that resolves to three IPs, each with a SCAN listener, so clients never need to know which nodes exist.
Bonded interfaces and Oracle’s own Redundant Interconnect (HAIP) both protect the interconnect. Parts 4 and 5 compare them. If you’ve bonded NICs on older releases, configuring a bond interface on Oracle Linux 6 shows the older ifcfg approach. OL9 uses NetworkManager and nmcli instead.
[grid@rac1 ~]$ crsctl check cluster -all
**************************************************************
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[grid@rac1 ~]$ olsnodes -n -i -s
rac1 1 rac1-vip Active
rac2 2 rac2-vip Active
[grid@rac1 ~]$ srvctl config scan
SCAN name: rac-scan.example.com, Network: 1
Subnet IPv4: 192.168.10.0/255.255.255.0/ens192, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 192.168.10.31
SCAN VIP is enabled.
SCAN 2 IPv4 VIP: 192.168.10.32
SCAN VIP is enabled.
SCAN 3 IPv4 VIP: 192.168.10.33
SCAN VIP is enabled.crsctl stat res -t is the command you’ll run most often. It lists every resource Clusterware manages and where it’s running:
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
ora.orcl_prim.db
1 ONLINE ONLINE rac1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE rac2 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------A single server can run a cut-down version of Grid Infrastructure called Oracle Restart. You get ASM and automatic restarts of the database and listener without a cluster. It’s a common choice for Data Guard standbys and smaller sites.
Layer 3: Oracle RAC
Real Application Clusters runs one database on several instances at once. Every node mounts the same datafiles in ASM, and each instance has its own memory (SGA), background processes, redo thread and undo tablespace. When one instance needs a block that another instance has in memory, Cache Fusion ships the block over the private interconnect instead of writing it to disk first. That’s why the interconnect needs low latency and redundancy.
If a node fails, the surviving instances recover the failed instance’s redo and keep the database open. Sessions on the failed node have to reconnect. Whether users notice depends on the client setup: services, FAN, Transparent Application Failover or Application Continuity, which parts 24 to 29 cover. RAC protects against server failure. It doesn’t protect against losing the storage array or the building, because every instance shares one copy of the data.
[oracle@rac1 ~]$ srvctl status database -db orcl_prim -verbose
Instance orcl1 is running on node rac1. Instance status: Open.
Instance orcl2 is running on node rac2. Instance status: Open.
[oracle@rac1 ~]$ srvctl status service -db orcl_prim
Service sales_rw is running on instance(s) orcl1,orcl2
SQL> select inst_id, instance_name, host_name, status from gv$instance order by 1;
INST_ID INSTANCE_NAME HOST_NAME STATUS
---------- ---------------- -------------------- ------------
1 orcl1 rac1.example.com OPEN
2 orcl2 rac2.example.com OPENApplications should always connect through the SCAN and a service name, never a node name or SID. Services let you choose which instances serve which workload and move that work during maintenance. Clients also connect the same way after a Data Guard role change. Here’s a preview of the kind of TNS entry part 25 builds:
SALES_RW =
(DESCRIPTION =
(CONNECT_TIMEOUT=90)(RETRY_COUNT=20)(RETRY_DELAY=3)(TRANSPORT_CONNECT_TIMEOUT=3)
(ADDRESS_LIST =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.example.com)(PORT = 1521)))
(ADDRESS_LIST =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST = stby-scan.example.com)(PORT = 1521)))
(CONNECT_DATA = (SERVICE_NAME = sales_rw)))Clients try the primary SCAN first and then the standby SCAN. Only the site whose database is currently primary runs the sales_rw service, because the service is tied to the PRIMARY database role. That way the same connect string keeps working after a switchover.
Licensing note: RAC is a separately licensed option for Enterprise Edition. Oracle desupported RAC on Standard Edition 2 in 19c.
Layer 4: Data Guard for disaster recovery
Data Guard keeps one or more standby databases synchronized with the primary by shipping redo, the database’s change records, to them. The primary’s log writer sends redo to the standby, which writes it into standby redo logs and applies it. Part 35 covers the architecture in depth. The main concepts are:
- Physical standby (Redo Apply): a block-for-block copy of the primary that applies redo with media recovery. This is what most DR designs use.
- Logical standby (SQL Apply): rebuilds SQL from redo. It’s rarely used now that GoldenGate exists.
- Snapshot standby: a physical standby opened read-write for testing, then flashed back and resynchronized.
- Protection modes: Maximum Performance (asynchronous, the default), Maximum Availability (synchronous unless the standby becomes unreachable) and Maximum Protection (synchronous, and the primary stops rather than lose data).
- Data Guard Broker (
dgmgrl) manages the whole configuration as a unit. It handles switchover, which is a planned role change and how you change the primary, and failover, which is the unplanned one. Fast-Start Failover lets an observer fail over automatically. - Active Data Guard is a separately licensed option. It lets you open a physical standby read-only while redo apply keeps running, so you can send reports and backups to the standby. Plain Data Guard is included with Enterprise Edition.
[oracle@rac1 ~]$ dgmgrl /
DGMGRL> show configuration;
Configuration - orcl_dg
Protection Mode: MaxAvailability
Members:
orcl_prim - Primary database
orcl_stby - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 37 seconds ago)
DGMGRL> show database orcl_stby;
Database - orcl_stby
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 2.10 MByte/s
Real Time Query: OFF
Instance(s):
orcl1 (apply instance)
orcl2
Database Status:
SUCCESSRAC and Data Guard work well together. The standby can be a RAC cluster, a single instance or an Oracle Restart server, and only one standby instance applies redo at a time. Oracle’s Maximum Availability Architecture (MAA) labels the combinations as tiers: Bronze, Silver, Gold and Platinum. Part 47 explains them.
Layer 5: GoldenGate for logical, multi-site replication
Data Guard copies the whole database physically to an identical standby. GoldenGate replicates logical changes, such as row inserts, updates and deletes, for whichever schemas or tables you choose. The target can be a different database version, a different platform, or even a different database product. It’s also the tool for active-active multi-site setups where more than one site accepts writes.
- Extract captures changes from the source. For Oracle databases, integrated Extract works with the database’s log mining server.
- Trail files are GoldenGate’s own change records on disk.
- Distribution Service and paths send trails to the remote site, and the Receiver Service writes them there.
- Replicat applies the changes to the target. Parallel Replicat is the usual choice for throughput.
- Microservices Architecture manages everything through the Service Manager, the Administration Service, a web UI, the
adminclientcommand line and a REST API.
[ogg@gg1 ~]$ adminclient
OGG (not connected) 1> CONNECT https://gg1.example.com:9001 DEPLOYMENT dc1 AS oggadmin
OGG (https://gg1.example.com:9001 dc1) 2> INFO ALL
Program Status Group Type Lag at Chkpt Time Since Chkpt
ADMINSRVR RUNNING
DISTSRVR RUNNING
PMSRVR RUNNING
RECVSRVR RUNNING
EXTRACT RUNNING EXTDC1 INTEGRATED 00:00:02 00:00:05
REPLICAT RUNNING REPDC2 PARALLEL NONINT 00:00:00 00:00:03Active-active replication brings problems that Data Guard never has. Changes have to be kept from looping back to the site that made them, and two sites can update the same row at the same moment, so conflicts need detecting and resolving. Part 54 covers both. GoldenGate is licensed separately from the database.
Which tool solves which problem?
| Scenario | Use | Why |
|---|---|---|
| A server crashes or needs a kernel patch | RAC + Clusterware | Other instances keep the database open. Services move and patching is rolling. |
| A disk or SAN path fails | ASM redundancy + DM-Multipath | Mirrored extents and path failover. The database never notices. |
| The data center is lost | Data Guard (physical standby) | A complete, consistent copy at another site. Failover takes minutes, with zero data loss in sync modes. |
| Someone drops a table or corrupts data | Flashback, plus Data Guard with an apply delay | Redo Apply would copy the mistake, so you need a way back in time. |
| Upgrade with near-zero downtime | GoldenGate, or a Data Guard transient logical rolling upgrade | Run the old and new versions side by side and switch over. |
| Several sites writing at once | GoldenGate active-active | Data Guard allows only one primary. |
| Offload reporting | Active Data Guard, or GoldenGate to a reporting database | A read-only standby, or a reshaped copy of the data. |
Oracle Linux 9 with 19c, and what changes in 23ai and 26ai
- 19c on Oracle Linux 9 is certified starting with Release Update 19.19, on UEK 7 (5.15.0-1.43.4.2 or later) or the Red Hat Compatible Kernel (5.14.0-70.22.1.0.2 or later). The base 19.3 installer predates OL9, so you install with
gridSetup.sh -applyRUandrunInstaller -applyRUpointing at a current RU. Part 11 shows the exact steps. - Support: 19c is the long-term release, with Premier Support through December 2029 and Extended Support through December 2032. Oracle AI Database 26ai (October 2025) is the next long-term release and replaces 23ai, with Premier Support through December 2031.
- Storage: ASM Filter Driver is desupported starting with 23ai. Use ASMLib v3 or udev rules, as described above.
- Multitenant only: since 21c, every database is a container database (CDB) with pluggable databases (PDBs), so non-CDB architecture is gone in 23ai and 26ai. If you build 19c as a CDB today, upgrades later are simpler.
- GoldenGate: GoldenGate 23ai supports only the Microservices Architecture. The old Classic Architecture (GGSCI with a manager process) is gone, so this series uses Microservices throughout.
- The concepts in this post don’t change across these releases: Clusterware, SCAN, services, ASM disk groups, Data Guard Broker and the MAA tiers all work the same way.
Quick health check of the whole stack
Here’s a short checklist you can run as the right user on any node. Later posts explain each output line by line:
# As grid
crsctl check cluster -all # Clusterware daemons on every node
crsctl stat res -t # every resource, where it runs
olsnodes -n -i -s # cluster members and VIPs
srvctl status scan_listener # SCAN listeners
asmcmd lsdg # disk groups, free space, offline disks
ocrcheck # OCR integrity (run as root for the full check)
crsctl query css votedisk # voting files
# As oracle
srvctl status database -db orcl_prim -verbose
srvctl status service -db orcl_prim
dgmgrl / "show configuration" # Data Guard Broker status
dgmgrl / "show database orcl_stby" # transport and apply lagWhat’s coming in this series
The series follows the stack in the order you’d build it:
- Parts 2 to 11: planning, OS preparation on Oracle Linux 9, network bonding and HAIP, DNS for SCAN, multipath and ASM device naming,
cluvfy, and installing Grid Infrastructure 19c. - Parts 12 to 22: Clusterware internals and everyday commands, OCR and voting files, node evictions, diagnostics, and ASM disk group work, including moving to a new SAN online.
- Parts 23 to 34: creating the RAC database, services, tnsnames for RAC, TAF, FAN, Application Continuity, adding and removing nodes, rolling patching, and changing the network.
- Parts 35 to 50: Data Guard from the ground up: building standbys, Broker, switchover and failover, Fast-Start Failover, Active Data Guard, RAC-to-RAC standbys, Far Sync, MAA tiers, two-site DR design and DR drills.
- Parts 51 to 60: GoldenGate 23ai Microservices, one-way and multi-site active-active replication, GoldenGate on RAC, upgrades with GoldenGate, maintenance checklists, standby-first patching, and upgrading to 23ai and 26ai.
Securing database logins with Kerberos comes up later as well. If you want to get ready now, the site already covers setting up an MIT Kerberos KDC.
Oracle Infrastructure Series: this is part 1. Next is part 2, Planning an Oracle RAC cluster: hardware, shared storage, networks (public, private interconnect, VIPs, SCAN) and DNS.
References: Oracle’s Grid Infrastructure Installation and Upgrade Guide 19c for Linux, Data Guard Concepts and Administration 19c, Oracle ASMLIB documentation and the Oracle Lifetime Support Policy.