Solaris path_to_inst, /devices and /dev Device Paths Explained

Solaris Admin Series, part 4. Previous: Solaris Filesystem Layout Tour. Related: Solaris on SPARC vs x86: sun4u, sun4v, i86pc and What Differs and part 1, Solaris 10 and 11.4 Releases, Support Lifecycle and Which to Run. Next up: essential first-day commands (uname, showrev, /etc/release, isainfo).

Sooner or later every Solaris admin has to answer questions like “which physical disk is sd3?”, “why did my new HBA show up as c5 instead of c2?” or “where did the device node for this LUN go?”. The answers live in three places: the physical device tree under /devices, the friendly logical names under /dev, and the kernel’s instance number database in /etc/path_to_inst. This post explains how they fit together, how to map one name to another, and how to rebuild device links safely.

Applies to: both Solaris 10 and Solaris 11 (11.4), on SPARC and x86. Differences, such as network link names and the man page sections, are called out where they matter.

Prerequisites

A Solaris 10 or Solaris 11.4 host. Reading the device tree works as a normal user; devfsadm, cfgadm changes and reconfiguration boots need root (or the root role on Solaris 11)
Part 3 of the series, for where /devices, /dev and /kernel/drv sit in the overall layout
A console connection (ILOM, LOM or serial) before you try any reconfiguration boot on a remote system

The three layers in one picture

Layer	Where	Example	Who manages it
Physical device path	`/devices` (the `devfs` filesystem)	`/devices/pci@0,0/pci15ad,1976@10/sd@0,0:a`	The kernel, from the firmware device tree and the drivers
Instance name	`/etc/path_to_inst`	`sd0`, `e1000g1`, `mpt_sas2`	The kernel; assigned the first time a device is seen
Logical name	`/dev`	`/dev/dsk/c1t0d0s0`, `/dev/rmt/0n`, `/dev/net/net0`	`devfsadm` (and on Solaris 11 the `/dev` filesystem itself)

Admins type logical names, the kernel and error messages often use instance names, and the physical path is the stable truth that connects the two.

/devices: the physical device tree

/devices mirrors the hardware tree the firmware handed to the kernel: buses, bridges, controllers and the leaf devices hanging off them. Each node name is name@unit-address, and the part after the colon is the minor node, for example a disk slice or a raw versus block device.

$ df -n /devices
/devices           : devfs
$ ls -l /devices/pci@0,0/pci15ad,1976@10/
total 0
drwxr-xr-x   2 root     sys            0 Oct 29 08:02 sd@0,0
brw-r-----   1 root     sys      203,  0 Oct 29 08:02 sd@0,0:a
crw-r-----   1 root     sys      203,  0 Oct 29 08:02 sd@0,0:a,raw
brw-r-----   1 root     sys      203,  1 Oct 29 08:02 sd@0,0:b
crw-r-----   1 root     sys      203,  1 Oct 29 08:02 sd@0,0:b,raw
...

This listing is from an x86 VMware guest, so your controller names and major numbers will differ. The major number (here 203) identifies the driver and comes from /etc/name_to_major; the minor number identifies the slice or channel within that driver.

prtconf prints the same tree in a readable form. The -D option adds the driver bound to each node, which is the quickest way to see which hardware has no driver attached.

$ prtconf -D | sed -n '1,40p'
System Configuration:  Oracle Corporation  i86pc
Memory size: 8192 Megabytes
System Peripherals (Software Nodes):

i86pc (driver name: rootnex)
    ...
    pci, instance #0 (driver name: npe)
        ...
        pci15ad,1976, instance #0 (driver name: mpt)
            sd, instance #0 (driver name: sd)
            sd, instance #1 (driver name: sd)
        pci15ad,7b0, instance #0 (driver name: vmxnet3s)
        pci15ad,740 (driver not attached)

Useful variants: prtconf -v dumps every property (vendor IDs, serial numbers, inquiry data), prtconf -P also shows nodes that are not attached, and on SPARC prtconf -pv prints the OpenBoot PROM device tree properties. Use prtdiag -v for a hardware summary by slot.

On SPARC the paths look different but follow the same rules. A disk on a sun4v server behind an LSI SAS controller is addressed by its SAS WWN:

/devices/pci@300/pci@1/pci@0/pci@4/pci@0/pci@c/scsi@0/disk@w5000cca0251b2d39,0:a

That is the same path you would use, minus /devices and the minor name, as an OBP devalias or as a boot-device value at the ok prompt, which is why it pays to learn to read it.

/dev: logical names that point into /devices

Almost everything in /dev is a symbolic link into /devices. Disks follow the cXtYdZsN convention: controller, target, LUN (disk) and slice. On x86, pN names (p0 to p4) refer to the fdisk partitions instead of slices.

$ ls -l /dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0
lrwxrwxrwx   1 root     root          50 Sep 14 10:02 /dev/dsk/c1t0d0s0 -> ../../devices/pci@0,0/pci15ad,1976@10/sd@0,0:a
lrwxrwxrwx   1 root     root          54 Sep 14 10:02 /dev/rdsk/c1t0d0s0 -> ../../devices/pci@0,0/pci15ad,1976@10/sd@0,0:a,raw

Slices map to minor letters: s0 is :a, s1 is :b, and so on. /dev/dsk holds block devices (used by mount), and /dev/rdsk holds raw character devices (used by format, newfs, prtvtoc, fsck and dd).

With MPxIO (Solaris multipathing, scsi_vhci) enabled, disks get long names built from the LUN’s WWN, such as c0t5000CCA0251B2D38d0, and their physical path sits under /scsi_vhci instead of under one HBA. Multipathing gets its own post later in the series.

Other common /dev directories

Path	What it holds
`/dev/dsk`, `/dev/rdsk`	Block and raw disk slices and partitions
`/dev/rmt`	Tape drives (`0`, `0n` for no-rewind, `0cbn` and so on)
`/dev/cfg`	Attachment points used by `cfgadm`, such as `c2` or a USB port
`/dev/net`	Solaris 11 network datalinks (`net0`, `net1`, VNICs)
`/dev/zvol/dsk`, `/dev/zvol/rdsk`	ZFS volumes, as `pool/volume`
`/dev/md`	Solaris Volume Manager metadevices (`d10` and so on)
`/dev/term`, `/dev/cua`	Serial ports
`/dev/null`, `/dev/zero`, `/dev/random`	Pseudo devices, also links into `/devices/pseudo`

On Solaris 11, /dev is itself a filesystem (df -n /dev reports dev), which builds some entries on demand; this is also how each non-global zone gets its own limited /dev. On Solaris 10 the global zone’s /dev is an ordinary directory on the root filesystem filled in by devfsadm.

/etc/path_to_inst: instance numbers

The kernel gives every device that a driver attaches an instance number per driver: the first disk the sd driver sees is sd0, the next is sd1, and so on. To keep those numbers stable across reboots, it records each physical path with its instance number and driver name in /etc/path_to_inst.

$ grep '"sd"' /etc/path_to_inst
"/pci@0,0/pci15ad,1976@10/sd@0,0" 0 "sd"
"/pci@0,0/pci15ad,1976@10/sd@1,0" 1 "sd"
"/pci@0,0/pci-ide@7,1/ide@1/sd@0,0" 2 "sd"
$ grep '"vmxnet3s"' /etc/path_to_inst
"/pci@0,0/pci15ad,7a0@15/pci15ad,7b0@0" 0 "vmxnet3s"

Instance names appear in many places: iostat -x and iostat -E, /var/adm/messages warnings like WARNING: /pci@0,0/pci15ad,1976@10/sd@1,0 (sd1), prtconf -D, and kstat module names. The format is documented in path_to_inst(4) on Solaris 10 and path_to_inst(5) on Solaris 11.4.

Two rules matter in practice:

Numbers are not reused. If you pull a NIC or HBA and put a new one in a different slot, the new hardware gets the next free instance, so e1000g0 can become e1000g2. Old entries stay in the file and do no harm.
Do not hand-edit it on a running system. The kernel owns the file and rewrites it. Make a copy before any hardware change (cp -p /etc/path_to_inst /etc/path_to_inst.YYYYMMDD) so you can see later what changed. Renumbering instances by editing it is a recovery-only operation; on Solaris 11 the usual answer to unfriendly names is the vanity link names described below rather than touching instance numbers.

Mapping between names: practical recipes

Which cXtYdZ is sdN?

iostat prints instance names by default and logical names with -n. Running both side by side on the same interval lines them up; the simpler route is to resolve the link and look the path up in path_to_inst.

$ ls -l /dev/dsk/c1t1d0s0 | awk '{print $NF}'
../../devices/pci@0,0/pci15ad,1976@10/sd@1,0:a
$ grep '"/pci@0,0/pci15ad,1976@10/sd@1,0"' /etc/path_to_inst
"/pci@0,0/pci15ad,1976@10/sd@1,0" 1 "sd"

So c1t1d0 is sd1. A small loop does it for every disk:

#!/bin/ksh
# map logical disk names to sd/ssd instance numbers
for l in /dev/rdsk/c*s2 /dev/rdsk/c*d0; do
    [ -h "$l" ] || continue
    p=$(ls -l "$l" | awk '{print $NF}' | sed -e 's#^.*/devices##' -e 's#:.*$##')
    i=$(grep "\"$p\"" /etc/path_to_inst | awk '{print $3 $2}' | tr -d '"')
    echo "$(basename $l)  $i  $p"
done | sort -u

c1t0d0s2  sd0  /pci@0,0/pci15ad,1976@10/sd@0,0
c1t1d0s2  sd1  /pci@0,0/pci15ad,1976@10/sd@1,0

EFI-labelled disks (the default for ZFS whole-disk pools) have no s2 backup slice, which is why the loop also checks the bare d0 names.

Which physical disk is this?

format shows each logical name with its physical path, and iostat -En adds vendor, product and serial number:

# echo | format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <VMware-Virtual disk-2.0 cyl 4093 alt 2 hd 128 sec 32>
          /pci@0,0/pci15ad,1976@10/sd@0,0
       1. c1t1d0 <VMware-Virtual disk-2.0-20.00GB>
          /pci@0,0/pci15ad,1976@10/sd@1,0
Specify disk (enter its number):

$ iostat -En c1t1d0
c1t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: VMware   Product: Virtual disk     Revision: 2.0  Serial No:
Size: 21.47GB <21474836480 bytes>

On Solaris 11.4 servers with supported chassis, diskinfo and croinfo map disks to labelled drive bays (for example /SYS/HDD3), which is what you want when someone has to pull the right drive.

Network devices: instances vs link names

On Solaris 10 the network interface name is the driver plus instance, such as e1000g0 or ixgbe1, and those names appear in /etc/hostname.e1000g0. Solaris 11 adds generic link names (net0, net1) assigned by physical location, so a replaced card in the same slot keeps its name. dladm show-phys shows the mapping, and -L adds the physical location:

$ dladm show-phys
LINK            MEDIA         STATE      SPEED  DUPLEX    DEVICE
net0            Ethernet      up         1000   full      vmxnet3s0
net1            Ethernet      unknown    0      unknown   vmxnet3s1
$ dladm show-phys -L
LINK            DEVICE       LOC
net0            vmxnet3s0    MB
net1            vmxnet3s1    MB

devfsadm: rebuilding /dev

devfsadm creates and removes the /dev links and makes sure drivers are loaded for new hardware. You run it after adding storage, a tape drive or a card that the system did not pick up on its own.

# devfsadm -v                 # scan everything, show what changes
devfsadm[1832]: verbose: symlink /dev/dsk/c1t2d0s0 -> ../../devices/pci@0,0/pci15ad,1976@10/sd@2,0:a
...
# devfsadm -c disk            # only the disk class (also: tape, port, audio, pseudo)
# devfsadm -i e1000g          # only nodes for one driver
# devfsadm -Cv                # also remove dangling links for hardware that is gone

devfsadm -C is the right fix for stale /dev/dsk entries left behind by removed LUNs, which otherwise make format slow and confuse scripts. Read the verbose output before you trust it on a production host with SAN storage that might be temporarily offline. The man page is devfsadm(1M) on Solaris 10 and devfsadm(8) on Solaris 11.4.

cfgadm for hot-pluggable devices

For SAS, SATA, Fibre Channel and USB attachment points, cfgadm configures and unconfigures devices without a reboot:

# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             scsi-sas     connected    configured   unknown
c1::dsk/c1t0d0                 disk         connected    configured   unknown
c3                             fc-fabric    connected    unconfigured unknown
# cfgadm -c configure c3
# devfsadm -c disk

Reconfiguration boot

When hardware was added while the system was down, or a driver needs a full rescan, do a reconfiguration boot. It rebuilds the device tree and links during boot.

# touch /reconfigure && init 6       # Solaris 10 and 11
# reboot -- -r                       # same result
ok boot -r                           # from the OBP prompt on SPARC

On x86 Solaris 10 you can also add -r to the kernel line in the GRUB menu for a single boot. In practice, devfsadm plus cfgadm handles most cases without a reboot on both releases.

Driver configuration files that touch device naming

/etc/name_to_major: driver name to major number. Maintained by add_drv and rem_drv; never edit by hand.
/etc/driver_aliases: which hardware IDs bind to which driver (for example e1000g "pci8086,100e"). Change with update_drv -a -i.
/etc/minor_perm: owner and mode for device nodes. Change with update_drv -m.
/kernel/drv/*.conf on Solaris 10, and /etc/driver/drv/*.conf for local overrides on Solaris 11.4, for driver properties such as sd.conf or mpt_sas.conf.
modinfo | grep -w sd shows whether a driver module is loaded.

Solaris 10 vs Solaris 11.4 at a glance

Item	Solaris 10	Solaris 11.4
`/dev` in the global zone	Directory on root, managed by `devfsadm`	`dev` filesystem (sdev) plus `devfsadm`
Network names	Driver plus instance (`e1000g0`)	Generic link names (`net0`) mapped to instances
Local driver config	Edit `/kernel/drv/*.conf`	`/etc/driver/drv/*.conf`
Disk to bay mapping	`prtdiag`, vendor tools	`diskinfo`, `croinfo`
Man page sections	`devfsadm(1M)`, `path_to_inst(4)`	`devfsadm(8)`, `path_to_inst(5)`
Boot disk type	UFS slices or ZFS root	ZFS root on whole disk or slice

Verification checklist

prtconf -D shows your new device with a driver attached (no “driver not attached”)
grep for its physical path in /etc/path_to_inst returns an instance number
ls -l /dev/dsk (or dladm show-phys for NICs) shows a logical name pointing at that path
echo | format or iostat -En lists the disk with the expected size and serial number

Troubleshooting and gotchas

New LUN not visible. Check cfgadm -al for an unconfigured attachment point, configure it, then run devfsadm -c disk. For FC, confirm zoning and LUN masking first; Solaris cannot see what the array does not present.
Device shows “driver not attached” in prtconf -D. Either no driver is installed for that hardware ID, or the driver failed to attach. Check /var/adm/messages and the pci vendor and device IDs from prtconf -v against the hardware compatibility list.
NIC changed name after a card swap on Solaris 10. The new card got a new instance number. Rename /etc/hostname.e1000g0 to the new instance (for example hostname.e1000g2) rather than editing path_to_inst. On Solaris 11, check dladm show-phys -L; use dladm rename-link if the generic name moved.
format is very slow or lists dead disks. Stale links from removed SAN LUNs. Run devfsadm -Cv after confirming those LUNs are really gone.
System will not boot after moving a disk or cloning a root disk to new hardware. The boot path changed. Boot from the correct device (on SPARC, find it with show-disks or probe-scsi-all at the ok prompt and set boot-device), then do a reconfiguration boot so /dev and path_to_inst pick up the new paths.
Corrupt or empty /etc/path_to_inst. Restore it from a backup or from a boot environment snapshot. Booting with boot -a lets you point the kernel at alternate system files during recovery. This is why a dated copy before hardware work is worth keeping.
Scripts that hard-code c0t0d0. Controller numbers depend on discovery order and differ between hosts, SPARC and x86, and with MPxIO on or off. Discover names at run time with format, zpool status or iostat -En.

What’s next in the Solaris Admin Series

Next comes a set of essential first-day commands for identifying a Solaris system: uname -a, showrev, /etc/release, isainfo and isalist. Disk labels, format, MPxIO and ZFS pools all build on the naming covered here and get their own posts later. Catch up with part 3, Solaris Filesystem Layout Tour, and part 2, Solaris on SPARC vs x86: sun4u, sun4v, i86pc and What Differs.

Official references: Oracle Solaris 11.4 documentation library (see “Managing Devices in Oracle Solaris 11.4”) and Oracle Solaris 10 1/13 documentation library (see “System Administration Guide: Devices and File Systems”).

Solaris /devices, /dev and /etc/path_to_inst: Device Paths and Instance Numbers