Oracle Infrastructure Series, part 7. Previous: part 6, DNS and GNS for SCAN. Next up, part 8: persistent ASM device naming with ASMLib v3 and udev.
Every RAC node has to see the same shared LUNs, and each LUN has to survive the loss of a cable, an HBA, a switch or an array controller. On Linux that job belongs to DM-Multipath: it notices that /dev/sdb, /dev/sdh, /dev/sdn and /dev/sdt are really four paths to one LUN, and gives you a single device, /dev/mapper/crs01, that keeps working while paths come and go. This post configures DM-Multipath on Oracle Linux 9 for the Grid Infrastructure 19c cluster planned in part 2, planning the RAC cluster: WWIDs, multipath.conf defaults, blacklist and aliases, the queueing settings that matter for RAC, ownership for the grid user, and how to prove that failover works before ASM ever touches the disks. Nothing here changes for 23ai or 26ai; the differences are in the naming layer on top, which is the subject of part 8.
If you only need the basics on an older release, my earlier post Configure Multipath on CentOS, Oracle, Red Hat Linux covers the CentOS 6 era mpathconf workflow. This one is the RAC-specific version for Oracle Linux 9.
Architecture
rac1 (10.10.10.11) rac2 (10.10.10.12)
HBA host2 HBA host3 HBA host2 HBA host3
| | | |
| +-------------+ | |
| +----------------------|-------+ |
v v v v
+-----------+ +-----------+
| Fabric A | | Fabric B |
+-----------+ +-----------+
| | | |
v v v v
ctrl A ctrl B ctrl A ctrl B (array front-end ports)
+------+------ array ------+------+
|
LUNs: crs01 crs02 crs03 (10 GB), data01, reco01
Per node: host2 -> fabric A, host3 -> fabric B, each fabric reaches both
controllers = 4 SCSI paths per LUN, combined into one dm-multipath device:
sdb sdh sdn sdt -> /dev/mapper/crs01 -> ASM disk in +CRS
Each node has two HBAs (SCSI hosts 2 and 3), each cabled to a different fabric, and each fabric reaches both array controllers. That gives four paths per LUN per node. The same design works for iSCSI with two NICs on two storage VLANs; the only differences are the transport timeouts, covered below. NVMe over Fabrics is different: Oracle Linux 9 uses native NVMe multipathing for those namespaces, not DM-Multipath, so this post applies to SCSI LUNs (FC and iSCSI).
Prerequisites
- Oracle Linux 9 prepared as in part 3, preparing Oracle Linux 9, with the
griduser and theasmadmingroup (gid 54329) present on every node. - The storage team has zoned and presented the same LUNs to every node in the cluster, ideally with the same LUN numbers. For this series: three 10 GB LUNs for +CRS, plus +DATA and +RECO LUNs.
- At least two working paths per LUN on every node. Check this before you configure anything.
- The LUNs are blank. Do not create partitions, file systems or LVM physical volumes on them.
- Root access on all nodes and a maintenance window if any node already runs multipath for its boot disk.
Step 1: Install the packages and enable multipath
On Oracle Linux 9 the package is device-mapper-multipath. sg3_utils provides rescan-scsi-bus.sh, which is handy when LUNs are added later. Run this on every node:
[root@rac1 ~]# dnf install -y device-mapper-multipath sg3_utils
[root@rac1 ~]# mpathconf --enable --with_multipathd y
[root@rac1 ~]# systemctl is-enabled multipathd; systemctl is-active multipathd
enabled
active
mpathconf --enable creates /etc/multipath.conf if it does not exist, and --with_multipathd y enables and starts the daemon. The generated file on Oracle Linux 9 is short, along these lines:
defaults {
user_friendly_names yes
find_multipaths yes
enable_foreign "^$"
}
blacklist_exceptions {
property "(SCSI_IDENT_|ID_WWN)"
}
blacklist {
}
Everything else comes from built-in defaults and the built-in hardware table, which already contains settings for most arrays. multipathd show config prints the full merged configuration, which is the only reliable way to see what is actually in effect.
Step 2: Find the WWIDs and check the paths
A LUN’s WWID is its identity. It is the same on every path and on every node, and it is what multipath uses to group paths. Read it from any one path with scsi_id:
[root@rac1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb
36000d31000e44200000000000000001a
[root@rac1 ~]# for d in sdb sdh sdn sdt; do echo -n "$d "; /usr/lib/udev/scsi_id -g -u -d /dev/$d; done
sdb 36000d31000e44200000000000000001a
sdh 36000d31000e44200000000000000001a
sdn 36000d31000e44200000000000000001a
sdt 36000d31000e44200000000000000001a
Four paths, one WWID. Once multipathd is running it reports the same information for every path in one command, which is quicker than looping over devices:
[root@rac1 ~]# multipathd show paths format "%d %w %i %t %T"
dev uuid hcil dm_st chk_st
sda 3600508b1001c5e0d8a4f2b1c9e7d3a10 0:1:0:0 undef ready
sdb 36000d31000e44200000000000000001a 2:0:0:1 active ready
sdh 36000d31000e44200000000000000001a 2:0:1:1 active ready
sdn 36000d31000e44200000000000000001a 3:0:0:1 active ready
sdt 36000d31000e44200000000000000001a 3:0:1:1 active ready
sdc 36000d31000e44200000000000000001b 2:0:0:2 active ready
...
The HCIL column (host:channel:target:LUN) shows the path layout: hosts 2 and 3 are the two HBAs, targets 0 and 1 are the two controller ports each HBA can see. If a LUN shows fewer than four paths here, fix zoning or cabling first; multipath cannot add paths the kernel does not see. Run the same command on rac2 and compare the WWIDs. They must match exactly, even though the sdX names usually will not.
Step 3: Write multipath.conf with aliases
With user_friendly_names yes, multipath names devices mpatha, mpathb and so on, and records the mapping in /etc/multipath/bindings. That file is local to each node, so the same LUN can be mpathc on rac1 and mpathe on rac2. That is harmless for a single server but confusing in a cluster. Explicit aliases in a multipaths section fix the problem: the alias wins over the user-friendly name, and copying one multipath.conf to every node gives identical names everywhere.
Back up the generated file and replace it with this (WWIDs are the sample values from part 2; use your own):
[root@rac1 ~]# cp -p /etc/multipath.conf /etc/multipath.conf.orig
[root@rac1 ~]# vi /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths yes
enable_foreign "^$"
polling_interval 5
}
blacklist_exceptions {
property "(SCSI_IDENT_|ID_WWN)"
}
# Local boot disk: never multipath it
blacklist {
wwid "3600508b1001c5e0d8a4f2b1c9e7d3a10"
}
# Array settings that matter for RAC (merged with the built-in entry)
devices {
device {
vendor "COMPELNT"
product "Compellent Vol"
path_grouping_policy multibus
no_path_retry 12
}
}
multipaths {
multipath {
wwid 36000d31000e44200000000000000001a
alias crs01
}
multipath {
wwid 36000d31000e44200000000000000001b
alias crs02
}
multipath {
wwid 36000d31000e44200000000000000001c
alias crs03
}
multipath {
wwid 36000d31000e44200000000000000002a
alias data01
}
multipath {
wwid 36000d31000e44200000000000000003a
alias reco01
}
}
What each setting does
- user_friendly_names: only affects LUNs without an alias. Leaving it on keeps any unplanned LUN readable as
mpathX; the aliased ASM LUNs ignore it. - find_multipaths yes: multipath only builds a map for a device that has at least two paths with the same WWID, or whose WWID is already in
/etc/multipath/wwids. That is why the single-path local disk is left alone even without the blacklist entry. The catch: if a node boots with only one path to a LUN, no map is created and something could open the raw/dev/sdX. Registering the shared WWIDs (step 4) prevents that. The other values arestrict(only WWIDs already in the wwids file),greedy(everything not blacklisted) andsmart. - blacklist: blacklist by WWID, never by
devnode "sda", becausesdXnames can change at boot. Get the local disk’s WWID from themultipathd show pathsoutput above or withscsi_id. - path_grouping_policy:
multibusputs all paths in one group and spreads I/O over them;failoveruses one path at a time;group_by_priogroups paths by ALUA priority so I/O goes to the optimized controller first. Use what your array vendor documents. Many current arrays are ALUA and wantgroup_by_prio, which the built-in hardware table usually already sets. - no_path_retry: what happens when all paths are down. A number means “queue I/O for that many checker intervals, then fail it”;
queuemeans queue forever;failmeans fail immediately. Withpolling_interval 5,12queues for about 60 seconds. The next section explains why this matters for RAC.
Why no_path_retry matters for RAC
Queueing forever (no_path_retry queue, or the deprecated features "1 queue_if_no_path" without a no_path_retry value) is a common default in array hardware table entries, including the built-in Compellent entry used here. It is fine for a single server: if the SAN blips, I/O pauses and then carries on. In a cluster it is dangerous. If one node loses every path, its processes hang in uninterruptible I/O instead of getting an error. ASM cannot take the failed disks offline and switch to the mirror copy in a NORMAL redundancy disk group, and Clusterware sees a node that is alive on the network but cannot write the voting files. The node is evicted after the CSS disk timeout, but only after a long, messy hang.
So use a finite value: long enough to ride through a controller failover or a fabric reboot (your array vendor documents this, typically 30 to 60 seconds), and well below the CSS disk timeout. Check the Clusterware timeouts once Grid Infrastructure is installed:
[grid@rac1 ~]$ crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
[grid@rac1 ~]$ crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
Two details catch people out. First, precedence: settings in multipaths beat overrides, which beat devices (your entries merged over the built-in hardware table), which beat defaults. Putting no_path_retry 12 only in defaults has no effect on an array whose built-in entry says queue. That is why it goes in the devices entry above (an overrides section would also work). Second, vendor matching: the vendor and product strings are regular expressions matched against what the array reports, which you can see in multipath -ll output. Check your array’s Oracle RAC best practices guide for its recommended values; some vendors want no_path_retry lower, and some want fast_io_fail_tmo and dev_loss_tmo set as well.
Transport timeouts (FC and iSCSI)
For Fibre Channel the defaults are reasonable: fast_io_fail_tmo 5 fails I/O on a dead remote port after 5 seconds so multipath can switch paths. For iSCSI, the initiator’s own timeout is the one to tune. The default node.session.timeo.replacement_timeout = 120 in /etc/iscsi/iscsid.conf holds I/O for two minutes before multipath even sees a failure; with multipath, set it low (5 to 15 seconds) and let no_path_retry control the queueing.
Step 4: Apply the configuration and register the WWIDs
Check the file for syntax errors, reload the daemon and look at the result:
[root@rac1 ~]# multipath -t > /dev/null && echo OK
OK
[root@rac1 ~]# systemctl reload multipathd
[root@rac1 ~]# multipath -a 36000d31000e44200000000000000001a
wwid '36000d31000e44200000000000000001a' added
[root@rac1 ~]# for w in 1b 1c 2a 3a; do multipath -a 36000d31000e4420000000000000000$w; done
wwid '36000d31000e44200000000000000001b' added
wwid '36000d31000e44200000000000000001c' added
wwid '36000d31000e44200000000000000002a' added
wwid '36000d31000e44200000000000000003a' added
[root@rac1 ~]# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
...
/36000d31000e44200000000000000001a/
/36000d31000e44200000000000000001b/
...
multipath -a adds a WWID to /etc/multipath/wwids (multipathd also does this automatically when it first builds a map). With the WWIDs registered, find_multipaths yes builds the map even if only one path is present at boot. If old mpathX maps are still around from before the aliases, multipath -r reloads all maps with the new names. Renaming a map that is already open fails, so do this before Grid Infrastructure is installed.
Rebuild the initramfs
If multipath is part of the initramfs (always the case when the boot disk is multipathed, and often the case once mpathconf has run), the copy of multipath.conf inside it must match the one on disk. Otherwise the early boot environment builds maps with old names or settings. Check, and rebuild after every change to the file:
[root@rac1 ~]# lsinitrd -m | grep -i multipath
multipath
[root@rac1 ~]# dracut -f
[root@rac1 ~]# lsinitrd /boot/initramfs-$(uname -r).img | grep -E 'multipath.conf|wwids'
-rw-r--r-- 1 root root 845 Nov 15 16:02 etc/multipath.conf
-rw------- 1 root root 312 Nov 15 16:02 etc/multipath/wwids
If you want multipath in the initramfs even though the boot disk is not multipathed, Red Hat’s documented command is dracut --force --add multipath.
Step 5: Verify the maps
[root@rac1 ~]# multipath -ll crs01
crs01 (36000d31000e44200000000000000001a) dm-2 COMPELNT,Compellent Vol
size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 2:0:0:1 sdb 8:16 active ready running
|- 2:0:1:1 sdh 8:112 active ready running
|- 3:0:0:1 sdn 8:208 active ready running
`- 3:0:1:1 sdt 65:48 active ready running
Read it line by line: the alias and WWID; features='1 queue_if_no_path', which appears for a finite no_path_retry as well as for queue (multipathd switches queueing off itself once the retries run out, so check the real setting with multipathd show maps below); one path group because of multibus, using the service-time path selector; and four paths that are all active ready running. With group_by_prio on an ALUA array you would see two groups instead, one with prio=50 status=active and one with prio=10 status=enabled.
For the whole node at once, multipathd gives compact output:
[root@rac1 ~]# multipathd show maps
name sysfs uuid
crs01 dm-2 36000d31000e44200000000000000001a
crs02 dm-3 36000d31000e44200000000000000001b
crs03 dm-4 36000d31000e44200000000000000001c
data01 dm-5 36000d31000e44200000000000000002a
reco01 dm-6 36000d31000e44200000000000000003a
[root@rac1 ~]# multipathd show maps format "%n %N %Q %t"
name paths queueing dm-st
crs01 4 12 chk active
crs02 4 12 chk active
crs03 4 12 chk active
data01 4 12 chk active
reco01 4 12 chk active
Then copy the configuration to the other node, register the WWIDs and repeat the checks there. The dm-N numbers may differ between nodes, which does not matter, because nothing should ever refer to /dev/dm-N. The aliases and WWIDs must be identical:
[root@rac1 ~]# scp -p /etc/multipath.conf rac2:/etc/multipath.conf
[root@rac2 ~]# systemctl reload multipathd
[root@rac2 ~]# for w in 1a 1b 1c 2a 3a; do multipath -a 36000d31000e4420000000000000000$w; done
[root@rac2 ~]# dracut -f
[root@rac1 ~]# diff <(multipathd show maps format "%n %w") <(ssh rac2 multipathd show maps format "%n %w") && echo identical
identical
Step 6: Test path failover
Do this before Grid Infrastructure is installed, while a mistake costs nothing. Start I/O against one map in one terminal (reading only, so the LUN stays blank):
[root@rac1 ~]# dd if=/dev/mapper/data01 of=/dev/null bs=1M iflag=direct status=progress
In a second terminal, take down every path through HBA host2, as if a fabric had failed. Setting the SCSI device state to offline is a clean way to simulate it without touching cables:
[root@rac1 ~]# for d in $(multipathd show paths format "%d %i" | awk '$2 ~ /^2:/ {print $1}'); do echo offline > /sys/block/$d/device/state; done
[root@rac1 ~]# multipath -ll data01
data01 (36000d31000e44200000000000000002a) dm-5 COMPELNT,Compellent Vol
size=500G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 2:0:0:5 sdf 8:80 failed faulty offline
|- 2:0:1:5 sdl 8:176 failed faulty offline
|- 3:0:0:5 sdr 65:16 active ready running
`- 3:0:1:5 sdx 65:112 active ready running
[root@rac1 ~]# journalctl -u multipathd --since "-5 min" | tail -4
Nov 15 16:10:02 rac1 multipathd[1123]: checker failed path 8:80 in map data01
Nov 15 16:10:02 rac1 multipathd[1123]: data01: remaining active paths: 3
Nov 15 16:10:02 rac1 multipathd[1123]: checker failed path 8:176 in map data01
Nov 15 16:10:02 rac1 multipathd[1123]: data01: remaining active paths: 2
The dd keeps running. Bring the paths back and confirm they are reinstated within a checker interval or two:
[root@rac1 ~]# for d in $(multipathd show paths format "%d %i" | awk '$2 ~ /^2:/ {print $1}'); do echo running > /sys/block/$d/device/state; done
[root@rac1 ~]# multipathd show paths format "%d %i %t %T" | grep -c 'active ready'
20
Twenty is five LUNs times four paths. For a real test, also have the storage team fail over an array controller and have the network team disable a fabric switch port, one at a time, and watch the same output. multipathd fail path sdf and multipathd reinstate path sdf exist too, but the path checker reinstates a healthy path almost immediately, so they are better for quick checks than for a sustained test.
Step 7: Ownership for ASM and the discovery string
ASM opens the disks as the grid user, so the multipath devices must be owned by grid:asmadmin with mode 0660, and that has to survive reboots and map reloads. Part 8 covers persistent naming properly (udev rules versus ASMLib v3). For a multipath-only setup, a minimal udev rule keyed on the device-mapper name is enough:
[root@rac1 ~]# cat /etc/udev/rules.d/99-oracle-asm-mpath.rules
ACTION=="add|change", ENV{DM_NAME}=="crs0*|data0*|reco0*", OWNER="grid", GROUP="asmadmin", MODE="0660"
[root@rac1 ~]# udevadm control --reload-rules
[root@rac1 ~]# udevadm trigger --type=devices --action=change --subsystem-match=block
[root@rac1 ~]# ls -lL /dev/mapper/crs0* /dev/mapper/data01 /dev/mapper/reco01
brw-rw----. 1 grid asmadmin 253, 2 Nov 15 16:20 /dev/mapper/crs01
brw-rw----. 1 grid asmadmin 253, 3 Nov 15 16:20 /dev/mapper/crs02
brw-rw----. 1 grid asmadmin 253, 4 Nov 15 16:20 /dev/mapper/crs03
brw-rw----. 1 grid asmadmin 253, 5 Nov 15 16:20 /dev/mapper/data01
brw-rw----. 1 grid asmadmin 253, 6 Nov 15 16:20 /dev/mapper/reco01
On Oracle Linux 9 the names under /dev/mapper are symlinks to /dev/dm-N; ls -lL follows them and shows the ownership that matters, on the dm-N node. Never put the underlying /dev/sdX paths in ASM; they would bypass multipath, and ASM would also see the same disk four times.
During the Grid Infrastructure installation, set the disk discovery path so ASM only scans the multipath devices. /dev/mapper/* works, because ASM skips devices it cannot open or that have no ASM header, but a narrower string is cleaner and keeps the LVM volumes of the boot disk out of the scan. Before installing, kfod from the unzipped Grid home shows exactly what ASM will discover:
[grid@rac1 ~]$ /u01/app/19.0.0/grid/bin/kfod disks=all asm_diskstring='/dev/mapper/crs*,/dev/mapper/data*,/dev/mapper/reco*'
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 10240 MB /dev/mapper/crs01 grid asmadmin
2: 10240 MB /dev/mapper/crs02 grid asmadmin
3: 10240 MB /dev/mapper/crs03 grid asmadmin
4: 512000 MB /dev/mapper/data01 grid asmadmin
5: 512000 MB /dev/mapper/reco01 grid asmadmin
--------------------------------------------------------------------------------
After installation, asmcmd dsget shows the discovery string in use and asmcmd lsdsk -k lists the disks with their paths. If you choose ASMLib v3 in part 8 instead of plain udev, ASMLib labels the multipath devices and scans dm devices only (ORACLEASM_SCANORDER="dm" and ORACLEASM_SCANEXCLUDE="sd" in /etc/sysconfig/oracleasm), so the discovery string becomes ORCL:* or /dev/oracleasm/disks/* instead.
19c, 23ai and 26ai differences
- DM-Multipath itself is an operating system layer and is configured the same way for every database release on Oracle Linux 9.
- 19c on Oracle Linux 9 needs Release Update 19.19 or later (install with
gridSetup.sh -applyRU), as covered in part 1, the overview. - The ASM Filter Driver’s filtering does not work on kernel 5.14 and later, and AFD is desupported in 23ai and 26ai. Plan on udev rules or ASMLib v3 on top of the multipath devices from the start, so an upgrade does not force a naming change.
Adding a LUN later
- Have the LUN presented to every node, then rescan on each node:
rescan-scsi-bus.sh -a(orecho "- - -" > /sys/class/scsi_host/host2/scanfor each HBA). - Check the WWID and path count with
multipathd show paths format "%d %w %i"on every node. - Add a
multipathentry with its alias tomultipath.confon every node, thenmultipath -athe WWID,systemctl reload multipathdanddracut -f. - Confirm ownership with
ls -lL, then add the disk to the disk group from ASM. That procedure gets its own post later in the series.
Troubleshooting and gotchas
- No map for a LUN:
multipath -v3explains every decision. The usual causes are a single visible path withfind_multipaths yesand an unregistered WWID, or a blacklist entry that matches more than you intended. - Settings not taking effect: check
multipathd show config, not your file. A value indefaultsloses to the built-in hardware table entry for the array; put it indevicesoroverrides. - Different names on each node: user-friendly names from separate
bindingsfiles. Use aliases and keep onemultipath.conffor the cluster. - Alias does not change after editing: the map is open (ASM, a
dd, a stale process). Find the holder withdmsetup info -c crs01(Open count) andlsof; rename before GI is installed. - Wrong names or settings after a reboot: the initramfs still holds the old
multipath.conf. Rebuild it withdracut -f. - Node hangs instead of being evicted when storage disappears:
multipathd show maps format "%n %Q"showson(queue forever) for the shared LUNs. Set a finiteno_path_retryindevicesoroverridesso it shows12 chk. - Paths flapping between failed and active: a bad cable, SFP or switch port. Watch
journalctl -u multipathd -fand the HBA counters in/sys/class/fc_host/host*/statistics/, and fix the hardware; do not tune it away. - LVM sees the ASM LUNs: Oracle Linux 9 uses the LVM devices file (
/etc/lvm/devices/system.devices) by default, so LVM only uses devices listed there. Never runpvcreateorvgimportdevices -aagainst the shared LUNs. - Ownership resets to root:disk: the udev rule did not match. Check the name with
udevadm info --query=property --name=/dev/mapper/crs01 | grep DM_NAMEand test the rule withudevadm test.
Oracle Infrastructure Series: this is part 7. Previous: part 6, DNS and GNS for SCAN. Earlier: part 5, HAIP vs bonding for the private interconnect, part 4, bonding the public network, part 3, preparing Oracle Linux 9, part 2, planning the RAC cluster and part 1, the overview. Next is part 8, persistent device naming for ASM with ASMLib v3 and udev rules.
References: Red Hat Enterprise Linux 9 Configuring device mapper multipath (Oracle Linux 9 uses the same device-mapper-multipath package), the Oracle Linux 9 documentation, the multipath.conf(5) and multipathd(8) man pages, and Oracle’s Grid Infrastructure Installation and Upgrade Guide 19c for Linux (configuring storage device path persistence).
