Solaris Process Management: ps, pgrep, pkill, kill, nohup

Solaris Admin Series, part 10. Previous: Solaris Text Editors on a Bare System: vi Essentials and ed When Nothing Else Works. Related: part 8, The Solaris PATH Problem, and part 6, Shells on Solaris 10 and 11. Next up: run levels and init states (init 0/S/1/2/3/5/6) and shutdown -i vs init vs reboot.

Finding a process, working out what it is doing, and stopping it cleanly is daily work on any Unix box, and Solaris looks familiar enough that Linux habits mostly work. Mostly. The signal numbers are different, killall does something very different, the global zone can see (and kill) every process in every zone, and SMF will restart a daemon you just killed. This post covers ps, pgrep, pkill, kill and nohup the Solaris way, plus the /proc tools (ptree, pargs, pfiles, pstop, prun, preap) that make Solaris process work so much easier.

Applies to: both Solaris 10 and Solaris 11 (11.4), SPARC and x86. The commands and options shown are the same on both releases unless noted. Differences called out below: Solaris 10 has /usr/ucb/ps installed as part of the core OS for BSD-style ps auxww; on 11.4 it is in the optional compatibility/ucb package. On Solaris 11, root is a role by default, so you run privileged kills with su, pfexec or sudo. On both releases, SMF-managed daemons should be stopped and restarted with svcadm, not kill.

Prerequisites

A shell on a Solaris 10 or Solaris 11.4 system (console, serial console or SSH).
Root, a root role, or an RBAC profile for signalling processes you do not own. Without privileges you can only signal your own processes; signalling other users’ processes needs the proc_owner privilege (root has it, and on Solaris 11 the Process Management rights profile grants it).
A sensible PATH (part 8, The Solaris PATH Problem): /usr/bin/ps is the System V ps; /usr/ucb/ps is the BSD one.

Step 1: List processes with ps -ef

The System V form is the one to learn. -e selects every process, -f gives the full listing:

$ ps -ef | head
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root     0     0   0   Nov 03 ?           0:02 sched
    root     5     0   0   Nov 03 ?          12:47 zpool-rpool
    root     1     0   0   Nov 03 ?           0:01 /usr/sbin/init
    root     2     0   0   Nov 03 ?           0:00 pageout
    root     3     0   0   Nov 03 ?          41:15 fsflush
    root    11     1   0   Nov 03 ?           0:09 /lib/svc/bin/svc.startd
    root    13     1   0   Nov 03 ?           0:31 /lib/svc/bin/svc.configd
  daemon   112     1   0   Nov 03 ?           0:00 /usr/lib/crypto/kcfd
    root   642     1   0   Nov 03 ?           0:00 /usr/lib/ssh/sshd

Column	Meaning
`UID`, `PID`, `PPID`	Owner, process ID, parent process ID
`C`	Recent processor usage used by the scheduler (not a percentage)
`STIME`	Start time today, or the date if it started on an earlier day
`TTY`	Controlling terminal; `?` means none (daemons)
`TIME`	Cumulative CPU time consumed, not wall-clock time
`CMD`	Command and arguments, truncated (see the 80-character gotcha below)

Useful variations that work on both releases:

$ ps -fu oracle                      # processes owned by a user
$ ps -fp 642                         # one PID (comma list allowed: -p 642,700)
$ ps -ft pts/2                       # processes on a terminal
$ ps -ef | awk '$3 == 1'             # children of PID 1 (or: pgrep -l -P 1)
$ ps -eLf                            # one line per LWP (thread): adds LWP and NLWP columns
$ ps -efZ                            # adds a ZONE column (global zone sees all zones)
$ ps -fz appzone1                    # only processes in zone appzone1

For scripts, pick your own columns with -o. This is far more reliable than parsing ps -ef output with awk:

$ ps -eo pid,ppid,user,s,etime,time,vsz,rss,pcpu,comm | head -5
  PID  PPID     USER S     ELAPSED        TIME  VSZ  RSS %CPU COMMAND
    0     0     root T  5-02:14:33       00:02    0    0  0.0 sched
    5     0     root S  5-02:14:30       12:47    0    0  0.1 zpool-rpool
    1     0     root S  5-02:14:30       00:01 3312 1920  0.0 /usr/sbin/init
    2     0     root S  5-02:14:30       00:00    0    0  0.0 pageout

# Top 5 memory users (RSS in KB), header suppressed with "rss=" style names
$ ps -eo rss=,pid=,user=,comm= | sort -rn | head -5

ps -o field	What it shows
`s`	State: `O` on CPU, `S` sleeping, `R` runnable, `T` stopped, `Z` zombie, `W` waiting for CPU cap
`etime`	Elapsed wall-clock time since start, `[[dd-]hh:]mm:ss`
`time`	Cumulative CPU time
`vsz`, `rss`	Virtual size and resident set size in kilobytes
`pcpu`, `pmem`	Recent CPU and memory percentage
`comm`, `args`	Command name only, or command with arguments
`zone`, `project`, `taskid`, `ctid`	Zone name, project, task ID and SMF contract ID
`pri`, `nice`, `class`	Priority, nice value and scheduling class (TS, IA, FX, RT, FSS, SYS)

For a live view use prstat rather than top, which is not in the base OS on Solaris 10. prstat -s cpu -n 10 sorts by CPU, prstat -a adds a per-user summary, prstat -Z a per-zone summary and prstat -mL microstate accounting per thread. prstat gets its own post in the performance section of this series.

Step 2: See the process tree and the full command line

The /proc tools in /usr/bin are one of the best things about Solaris. They take a PID (and many accept /proc/PID or a core file):

$ ptree 642
642   /usr/lib/ssh/sshd
  20811 /usr/lib/ssh/sshd
    20814 /usr/lib/ssh/sshd
      20815 -bash
        20990 ptree 642

$ pargs 20990            # full argv, no 80-character limit
20990:  ptree 642
argv[0]: ptree
argv[1]: 642

$ pargs -e 4120          # environment of a running process
$ pwdx 4120              # current working directory
$ pfiles 4120            # open files, sockets and their addresses
$ pcred 4120             # effective/real/saved UIDs and GIDs
$ pstack 4120            # stack trace of each thread
$ pldd 4120              # shared libraries loaded
$ pmap -x 4120           # address space map with RSS and anon memory

ptree -a includes the system processes, ptree user shows the trees for that user, and ptree -c shows the process contract (and therefore the SMF service) each process belongs to. pfiles briefly stops the target while it inspects it, so be gentle with it on latency-sensitive production processes.

Step 3: Find processes with pgrep

pgrep searches the process table and prints matching PIDs. It never matches itself, so there is no more ps -ef | grep foo | grep -v grep:

$ pgrep sshd
642
20811
20814

$ pgrep -l sshd                 # PID and name
642 sshd
20811 sshd
20814 sshd

$ pgrep -fl 'java.*weblogic'    # -f matches the argument string, not just the name
4120 /u01/jdk/bin/java -Xms4g -Xmx4g -Dweblogic.Name=AdminServer -Djava.security.po

$ pgrep -x -u oracle ora_pmon_ORCL   # exact name match, owned by oracle
$ pgrep -n java                 # newest matching process only
$ pgrep -o java                 # oldest matching process only
$ pgrep -P 642                  # children of PID 642
$ pgrep -z appzone1 -l httpd    # only in zone appzone1 (run from the global zone)
$ pgrep -d, sshd                # comma-separated, handy for ps -p or prstat -p
642,20811,20814
$ prstat -p $(pgrep -d, java)

Option	Selects by
`-u` / `-U`	Effective / real user (name or UID, comma list)
`-G`, `-g`	Real group / process group ID
`-P`	Parent PID
`-t`	Terminal, for example `-t pts/2`
`-s`	Session ID
`-z`	Zone name or ID
`-J`, `-T`	Project ID or name, task ID
`-c`	Contract ID (useful for SMF services)
`-x`	Pattern must match the whole name (or whole argument string with `-f`)
`-v`	Invert the match
`-n`, `-o`	Newest or oldest match only

The pattern is an extended regular expression (see regex(7) on 11.4, regex(5) on Solaris 10). Without -f it is matched against the executable name, which the kernel stores truncated to 15 characters, so a long program name may need -f or a shorter pattern. pgrep exits 0 if something matched, 1 if nothing did and 2 on a syntax error, which makes it ideal in scripts:

if pgrep -x -u oracle tnslsnr >/dev/null; then
    echo "listener is running"
else
    echo "listener is down"
fi

Step 4: Understand Solaris signals

List the signals with kill -l. The names are standard but several numbers differ from Linux, so always use names in scripts and in muscle memory:

$ kill -l
HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM USR1 USR2 CLD PWR WINCH URG POLL STOP TSTP CONT TTIN TTOU VTALRM PROF XCPU XFSZ WAITING LWP FREEZE THAW CANCEL LOST XRES JVM1 JVM2 INFO RTMIN RTMIN+1 ... RTMAX

The exact list depends on the shell (ksh93 and bash builtins format it differently from /usr/bin/kill), but the numbering comes from /usr/include/sys/iso/signal_iso.h and is the same on both releases:

Signal	Solaris number	Linux number	Default action	Typical admin use
`SIGHUP`	1	1	Terminate	Many daemons re-read their config; also sent when a terminal hangs up
`SIGINT`	2	2	Terminate	Ctrl-C
`SIGQUIT`	3	3	Terminate with core	Ctrl-\; a JVM prints a thread dump to stdout instead
`SIGKILL`	9	9	Terminate, cannot be caught	Last resort
`SIGTERM`	15	15	Terminate	Default for `kill`; polite stop request
`SIGUSR1` / `SIGUSR2`	16 / 17	10 / 12	Terminate	Application defined (log rotation, debug toggles)
`SIGCHLD` (`CLD`)	18	17	Ignore	Child exited
`SIGSTOP`	23	19	Stop, cannot be caught	Freeze a process
`SIGTSTP`	24	20	Stop	Ctrl-Z
`SIGCONT`	25	18	Continue	Resume a stopped process

So kill -19 on Solaris sends SIGPWR, not SIGSTOP, and kill -10 sends SIGBUS, not SIGUSR1. Scripts ported from Linux with numeric signals are a classic source of surprises.

Step 5: Send signals with kill and pkill

$ kill 4120                 # SIGTERM (15), ask nicely
$ kill -s TERM 4120         # same thing, POSIX form
$ kill -HUP 1234            # reload, if the program supports it
$ kill -0 4120 && echo alive    # signal 0: test that the PID exists and you may signal it
$ kill -9 4120              # SIGKILL, only after TERM failed

$ pkill -HUP -x syslogd     # by name (but see the SMF section: prefer svcadm refresh)
$ pkill -u alice            # SIGTERM every process owned by alice
$ pkill -TERM -t pts/7      # everything on an abandoned terminal
$ pkill -f "java .*-Dweblogic.Name=ManagedServer1"

A safe stop procedure for a misbehaving non-SMF process:

Identify it precisely: pgrep -fl pattern, then confirm with ptree PID and pargs PID that it is the right one.
Collect evidence before you kill it if you will need a root cause: pstack PID > /var/tmp/PID.stack, pfiles PID, and if needed a core with gcore -o /var/tmp/core PID (the process keeps running).
Send SIGTERM and wait a few seconds: kill PID; sleep 10; kill -0 PID 2>/dev/null && echo still running.
Only then kill -9 PID. SIGKILL gives the program no chance to flush buffers, remove lock or PID files, or release shared memory and semaphores (check with ipcs -a afterwards).

Before running any pkill, run the same arguments through pgrep -l (or pgrep -fl) and read the list. pkill has no confirmation and no dry-run.

Freezing and resuming without signals: pstop and prun

$ pstop 4120               # stop all LWPs through /proc
$ ps -o pid,s,comm -p 4120
  PID S COMMAND
 4120 T /u01/jdk/bin/java
$ prun 4120                # set it running again

pstop/prun act through /proc rather than SIGSTOP/SIGCONT, so the target cannot tell via a signal handler, and they are handy for pausing a runaway batch job while you free up disk space or CPU.

Step 6: Keep jobs running after logout with nohup

When your SSH session drops, the shell sends SIGHUP to its jobs. nohup starts a command with SIGHUP ignored and, if standard output is a terminal, appends output to nohup.out in the current directory (or $HOME/nohup.out if that is not writable):

$ nohup /u01/scripts/long_export.sh > /var/tmp/export.log 2>&1 &
[1]     7733

$ nohup ./rebuild_index.ksh &
Sending output to nohup.out
[1]     7741

Solaris’ nohup has an extra trick the GNU one lacks: it can make an already running process immune to hangups, which is a lifesaver when you started a long job without nohup and need to log off:

$ nohup -p 7802            # one running process (stdout/stderr redirected to nohup.out if they were the terminal)
$ nohup -g 7800            # every process in process group 7800 (a whole pipeline)
$ nohup -Fp 7802           # force, even if the process is being controlled by another tool
$ nohup -ap 7802           # always change the signal disposition, even if the process installed its own SIGHUP handler

In ksh93 and bash you can also disown a background job, and screen or tmux are available as optional packages on Solaris 11 (terminal/screen, terminal/tmux). For anything that should run permanently, write an SMF service instead; that is covered in the SMF section of this series.

Step 7: SMF-managed processes: use svcadm, not kill

On both Solaris 10 and 11 most daemons run under the SMF restarter inside a process contract. If you kill one, svc.startd notices the contract emptied and starts it again, and repeated deaths push the service into maintenance. Find the service first:

$ ps -o ctid= -p 642
   87
$ svcs -o fmri,ctid | awk '$2 == 87'
svc:/network/ssh:default    87

$ svcs -p ssh              # the processes belonging to the service
STATE          STIME    FMRI
online         Nov_03   svc:/network/ssh:default
               Nov_03        642 sshd

# svcadm restart network/ssh       # restart properly
# svcadm refresh system/system-log # re-read config (sends the service's refresh method, often HUP)
# svcadm disable -t network/ssh    # stop until next boot (-t = temporary)

pkill -c CTID and pgrep -c CTID select by contract, and ctstat -v -i CTID shows the contract details, which is useful when a service has many child processes.

Step 8: Zombies and processes that will not die

$ ps -eo pid,ppid,s,comm | awk '$3 == "Z"'
 9120  9001 Z <defunct>
$ ptree 9001
$ preap 9120               # reap the zombie on the parent's behalf

A zombie (<defunct>, state Z) has already exited; kill -9 does nothing. The parent has not collected its exit status. Fix or restart the parent, or use preap PID, which waits on the zombie through /proc. Do not use preap -F on a parent that you expect to handle the child itself.
A process that survives kill -9 is usually blocked in the kernel, typically on a hung NFS server or a dead device. ps -o pid,s,wchan,comm -p PID and pstack PID show where it is waiting. It will die once the I/O completes or times out; for a hard NFS mount, fixing the server (or umount -f) releases it.
To find who is holding a filesystem busy before an unmount: fuser -cu /export/home lists PIDs and users; fuser -ck /export/home kills them (dangerous on / or /usr).

Verification

$ pgrep -fl long_export        # still running after reconnecting?
7733 /bin/ksh /u01/scripts/long_export.sh
$ kill -0 4120; echo $?         # 0 = exists, nonzero = gone (or not yours)
1
$ svcs -x                       # no services left in maintenance by an over-eager kill
$ ps -eo s | sort | uniq -c     # quick state census: look for growing Z or T counts

Troubleshooting and gotchas

killall is not the Linux killall. On Solaris, /usr/sbin/killall sends a signal to every active process except those directly involved in shutdown. It exists for shutdown scripts. Typing killall java as root on a production box is effectively a crash. Use pkill.
Numeric signals from Linux scripts. STOP is 23, CONT is 25, USR1 is 16 on Solaris. Use names (kill -USR1, kill -s STOP).
The global zone sees every zone. pkill httpd as root in the global zone kills the web servers in all non-global zones too. Add -z global (or the zone name) to scope it, and use ps -efZ to see where a process lives.
Arguments truncated at 80 characters. ps -ef and pgrep -f use the 80-character argument summary the kernel keeps in psinfo. Long Java or Oracle command lines are cut off, so a pgrep -f pattern on the tail of the line will not match. Use pargs PID for the full argv, or /usr/ucb/ps -auxww (core on Solaris 10, compatibility/ucb package on 11.4).
Process name limited to 15 characters. Plain pgrep/pkill match pr_fname, which is truncated; pgrep -x some_long_program_name may silently match nothing.
“Not owner” or “permission denied” from kill. You need to own the process or hold proc_owner. On Solaris 11 assume the root role or use pfexec with a suitable profile; ppriv -v $$ shows your current privileges.
Killing a daemon that comes straight back. It is SMF doing its job. Use svcs -p to find the FMRI and svcadm disable or restart it.
kill -9 on databases. Killing Oracle background processes or a PostgreSQL postmaster with SIGKILL can leave shared memory segments and semaphores behind and force crash recovery. Stop them with their own tools first.
nohup.out filling a filesystem. If you do not redirect output, it grows forever in the current directory, often / when run as root. Always redirect to a log file you control.
Signals to process 1. init ignores most signals, and kill -HUP 1 is not how you change run levels; use init N or shutdown (next post).

What’s next in the Solaris Admin Series

Next up are run levels and init states on Solaris 10 and 11 (init 0/S/1/2/3/5/6) and when to use shutdown -i, init or reboot. Catch up with part 9, Solaris Text Editors on a Bare System, part 8, The Solaris PATH Problem, part 6, Shells on Solaris 10 and 11, and part 3, Solaris Filesystem Layout Tour.

Official references: ps(1), pgrep(1), pkill(1), kill(1), nohup(1), proc(1) (ptree, pargs, pfiles, pstop, prun, preap and friends), signal.h(3HEAD), killall(8), fuser(8), svcs(1) and svcadm(8) in the Oracle Solaris 11.4 Reference Library, and the same pages (with section 1M in place of 8) in the Oracle Solaris 10 1/13 Information Library.

Solaris Process Management: ps -ef, pgrep, pkill, kill Signals and nohup