Troubleshoot Stuck and Hung NFS Mounts on AlmaLinux 10

October 28, 2026 · by Zwiegnet Consulting · Linux, Storage

A hung NFS mount is one of the most frustrating problems on a Linux server: df never returns, ls freezes your shell, processes pile up in the uninterruptible D state, and sometimes the whole box looks dead. This guide shows how to troubleshoot stuck and hung NFS mounts on AlmaLinux 10 step by step, from the client side (mount options, D-state processes, stale file handles, forced and lazy unmounts) to the network (rpcinfo, nfsstat, tcpdump) and the NFS server itself.

The commands work the same on RHEL 10, Rocky Linux 10 and Oracle Linux 10, and almost all of them apply to EL9 and EL8 as well. If you are chasing a specific mount error instead of a hang, see the older posts on requested NFS version or transport protocol is not supported and input/output error from an NFS client.

Lab layout

  • nfs1.example.com 192.168.1.20, NFS server exporting /srv/nfs/data
  • app1.example.com 192.168.1.30, NFS client mounting it at /mnt/data
  • Both running AlmaLinux 10 with nfs-utils installed, NFSv4.2 over TCP (the default)

Prerequisites

  • Root or sudo access on the client, and ideally on the server too
  • The nfs-utils, psmisc (for fuser), tcpdump and nmap-ncat packages on the client
  • A second terminal or SSH session that is not sitting inside the hung mount point, so your diagnostic shell does not freeze too
sudo dnf install -y nfs-utils psmisc tcpdump nmap-ncat

Golden rule while troubleshooting: run cd / first, and never run a plain df, ls /mnt or tab completion on the broken path. Any command that calls stat() on a dead hard mount will hang. Wrap risky checks in timeout, as shown below.

Step 1: Understand hard vs soft mounts

How a mount behaves when the server disappears depends on its options, so check them first:

  • hard (the default): the client retries forever. Applications block until the server comes back, then carry on with no data loss. This is why a hard mount “hangs”.
  • soft: after retrans retries the call fails with EIO. Applications get an error instead of hanging, but writes can be silently lost or files corrupted. Only use it for read-only or throwaway data.
  • softerr: like soft but returns ETIMEDOUT, so applications can tell a timeout from a real I/O error.
  • timeo is in tenths of a second (default 600, so 60 seconds, for TCP) and retrans is the number of retries before a “server not responding” message or a soft failure (default 2).
  • The old intr/nointr options are ignored by modern kernels. NFS waits are killable, so kill -9 can usually end a stuck process on a hard mount.

Show the options each NFS mount is really using, as negotiated with the server:

nfsstat -m
findmnt -t nfs,nfs4 -o TARGET,SOURCE,FSTYPE,OPTIONS
/mnt/data from nfs1.example.com:/srv/nfs/data
 Flags:	rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.30,local_lock=none,addr=192.168.1.20

Both commands read the kernel mount table and do not touch the server, so they are safe on a hung mount.

Step 2: Confirm the mount is actually hung

Test the mount point with a short timeout instead of a bare ls:

cd /
timeout 5 stat -f /mnt/data && echo "mount responds" || echo "mount is hung or erroring (exit $?)"

Exit code 124 means stat was still waiting after 5 seconds. To check disk space on local file systems without touching NFS, exclude it:

df -h -x nfs -x nfs4

Then look at the kernel log. A hard mount that loses its server logs a clear message, and another when the server returns:

sudo journalctl -k --since "1 hour ago" | grep -iE 'nfs|rpc|lockd'
# nfs: server nfs1.example.com not responding, still trying
# nfs: server nfs1.example.com OK

If you see “not responding” without a matching “OK”, the client cannot reach the NFS service. If there is nothing in the log at all, the hang may be on the server side (overloaded nfsd, slow back-end storage, a held lock) rather than the network.

Step 3: Find processes stuck in D state

Processes blocked on NFS sit in the D (uninterruptible sleep) state. List them along with the kernel function they are waiting in:

ps -eo pid,stat,wchan:32,etime,cmd | awk 'NR==1 || $2 ~ /^D/'

A wchan such as rpc_wait_bit_killable, nfs_wait_bit_killable or anything starting with nfs or rpc points straight at NFS. For more detail, read the kernel stack of one process as root:

sudo cat /proc/<PID>/stack

To see which processes hold files open on the mount, use fuser, which reads /proc instead of stat-ing the mount and is usually safe. Avoid lsof without -b, because it can hang on the mount itself:

sudo fuser -vm /mnt/data
sudo lsof -b -w 2>/dev/null | grep /mnt/data

Note that the load average climbs with every D-state process, so a load of 200 on an idle box is a typical sign of a hung NFS mount, not real CPU load.

Step 4: Check basic connectivity to the server

Work up the stack: name resolution, ping, then the NFS port itself. NFSv4 only needs TCP port 2049:

getent hosts nfs1.example.com
ping -c 3 nfs1.example.com
nc -zv -w 3 nfs1.example.com 2049

For NFSv3, the client also needs rpcbind (port 111) and mountd on the server. rpcinfo shows which RPC programs and versions the server offers, and showmount lists exports (v3 only; an NFSv4-only server will not answer it):

rpcinfo -p nfs1.example.com
rpcinfo -t nfs1.example.com nfs 4
showmount -e nfs1.example.com
program vers proto   port  service
    100000    4   tcp    111  portmapper
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100005    3   tcp  20048  mountd
...
program 100003 version 4 ready and waiting

“Connection refused” on 2049 means the server is up but nfs-server is not running or a firewall rejects you. A timeout usually means a firewall drop, a routing problem or a dead server. Also check for an MTU mismatch (common with jumbo frames or VPNs): small requests work and large reads hang. Test with a non-fragmenting ping close to the interface MTU:

ping -c 3 -M do -s 1472 nfs1.example.com   # 1500 MTU minus 28 bytes of headers

Step 5: Look at NFS client statistics

nfsstat -c shows client RPC counters. A growing retrans number while the mount is hung confirms requests are being sent but not answered:

nfsstat -c -r
watch -n 2 'nfsstat -c -r'

Per-mount statistics, including average round-trip time and how long requests spend queued, come from mountstats and nfsiostat. These are the best tools for “the mount is not dead, just painfully slow”:

mountstats /mnt/data
nfsiostat 5 3 /mnt/data

In the nfsiostat output, a high avg RTT points at the network or server, while a large gap between avg exe and avg RTT means requests are queuing on the client.

Step 6: Capture the NFS traffic with tcpdump

When the counters do not explain it, look at the wire. Capture to a file from the client while you reproduce the hang:

sudo tcpdump -i any -nn -s 0 host 192.168.1.20 and port 2049 -w /tmp/nfs-hang.pcap
# reproduce the problem in another session, then press Ctrl+C
sudo tcpdump -nn -r /tmp/nfs-hang.pcap | tail -50

What to look for:

  • Calls leaving the client with no replies at all: the server or something in between is dropping traffic.
  • Repeated TCP SYNs to 2049 with no SYN-ACK: firewall or the service is down.
  • TCP resets ([R]) right after connecting: the server is refusing the client, often an export or firewall rule.
  • Replies arriving, but each one slowly: the server is overloaded, so move on to the server checks below.

Open the capture in Wireshark for decoded NFS operations and error codes such as NFS4ERR_STALE, NFS4ERR_GRACE or NFS4ERR_DELAY.

Step 7: Fix “Stale file handle” errors

Stale file handle (ESTALE) is not a hang but often follows one. The client holds a handle for a file or directory that the server no longer recognizes. Common causes:

  • The file or directory was deleted or replaced on the server (or by another client) while this client had it open or cached.
  • The exported file system was re-created, restored from backup or remounted on a different device, so its file system ID changed.
  • The export was moved to another server or cluster node without a stable fsid.
  • The client is still sitting in the old directory: cd out and back in.

On the client, leave the directory and remount:

cd /
sudo umount /mnt/data || sudo umount -l /mnt/data
sudo mount /mnt/data

On the server, give exports a stable file system ID so handles survive device renumbering or failover, then re-export:

# /etc/exports on nfs1
/srv/nfs/data  192.168.1.0/24(rw,sync,root_squash,fsid=101)
sudo exportfs -ra
sudo exportfs -v

Step 8: Unmount a hung NFS share (force and lazy)

Best option first: bring the server back. A hard mount resumes on its own the moment the server answers, with no lost writes. If the server is gone for good, unmount it in this order:

cd /
# 1. Stop or kill the processes using it (SIGKILL works on most NFS waits)
sudo fuser -vm /mnt/data
sudo fuser -km /mnt/data

# 2. Forced unmount: aborts outstanding RPCs on an unreachable server
sudo umount -f /mnt/data

# 3. Lazy unmount: detach it from the tree now, clean up when no longer busy
sudo umount -l /mnt/data

umount -f tells the NFS client to give up on pending requests, which often succeeds right away. umount -l always “works” because it only detaches the mount point. Processes that already had files open keep their references and stay stuck until the kernel can finish with them, so check again with ps afterwards. Never remount a fresh share on the same path and assume the old processes moved over; they did not.

If processes remain in D state forever even after a lazy unmount, the only clean fixes are restoring the server (even temporarily on the same IP) or rebooting the client. A reboot can hang too when systemd tries to unmount the share; unmount it lazily first, then reboot.

Step 9: Server-side checks

On the NFS server, confirm the service, the exports and the firewall:

sudo systemctl status nfs-server
sudo exportfs -v
sudo firewall-cmd --list-services
# NFSv4 needs "nfs"; NFSv3 also needs "rpc-bind" and "mountd"
sudo firewall-cmd --permanent --add-service={nfs,rpc-bind,mountd}
sudo firewall-cmd --reload

Check the kernel log and SELinux denials on the server too:

sudo journalctl -k --since "1 hour ago" | grep -iE 'nfsd|rpc|lockd'
sudo ausearch -m avc -ts recent

Not enough nfsd threads

A busy server with too few nfsd threads makes every client feel hung. Check the current thread count and raise it in /etc/nfs.conf:

cat /proc/fs/nfsd/threads
sudo grep -A3 '^\[nfsd\]' /etc/nfs.conf
# /etc/nfs.conf
[nfsd]
threads=32
sudo systemctl restart nfs-server
cat /proc/fs/nfsd/threads

Restarting nfs-server briefly pauses clients, so do it in a quiet window. Also check that the back-end storage is healthy: iostat -x 2 from sysstat showing a saturated disk means NFS is only the messenger.

NFSv4 grace period after a server restart

After the NFS server restarts, NFSv4 enters a grace period (90 seconds by default) during which clients can reclaim locks but new opens and locks wait. Clients appear frozen for up to a minute and a half, then recover on their own. You can see the configured values here:

cat /proc/fs/nfsd/nfsv4gracetime
cat /proc/fs/nfsd/nfsv4leasetime

To see which NFSv4 clients the server currently knows about (useful for spotting a client stuck holding state), read the per-client info files:

sudo cat /proc/fs/nfsd/clients/*/info

Step 10: Turn on RPC debugging (briefly)

For hangs you cannot explain, the kernel NFS and RPC client can log every operation with rpcdebug. It is very noisy, so enable it only while reproducing the problem, then turn it off:

sudo rpcdebug -m nfs -s all
sudo rpcdebug -m rpc -s call xprt
# reproduce, then read the log
sudo journalctl -k -n 200
sudo rpcdebug -m nfs -c all
sudo rpcdebug -m rpc -c all

Run the same with -m nfsd on the server to trace the server side.

Step 11: Prevent hung mounts at boot and in daily use

The worst NFS hang is the one that stops a server from booting. Mount NFS shares on demand with systemd so a missing server does not block boot, and keep hard for data you care about:

# /etc/fstab on app1
nfs1.example.com:/srv/nfs/data  /mnt/data  nfs  _netdev,nofail,x-systemd.automount,x-systemd.mount-timeout=30,hard,timeo=600,retrans=2  0 0
sudo systemctl daemon-reload
sudo systemctl restart remote-fs.target
ls /mnt/data   # triggers the mount
  • _netdev waits for the network, nofail lets boot continue if the mount fails, and x-systemd.automount mounts it on first access.
  • Add x-systemd.idle-timeout=600 to unmount the share after 10 idle minutes, so an outage only affects processes that are actually using it.
  • Keep NFS paths out of $PATH, shell prompts and monitoring checks that call df, or one dead server will freeze every login.
  • Monitor with timeout-wrapped checks such as timeout 10 stat -f /mnt/data so the check itself cannot hang.

Verification

Once the mount is healthy again, confirm it end to end:

timeout 5 stat -f /mnt/data && echo OK
nfsstat -m
ps -eo pid,stat,cmd | awk '$2 ~ /^D/'   # should print nothing NFS related
echo test | sudo tee /mnt/data/.nfs-check && sudo rm /mnt/data/.nfs-check
sudo journalctl -k --since "10 min ago" | grep -i 'not responding' || echo "no new NFS timeouts"

Quick troubleshooting reference

  • df or ls freezes: hard mount with an unreachable server. Use timeout, findmnt and nfsstat -m, then check connectivity to port 2049.
  • “server not responding, still trying” in the log: network or server outage. Check nc -zv server 2049, the firewall, MTU and systemctl status nfs-server.
  • Hangs on large files only: almost always an MTU mismatch or a middlebox dropping fragments. Test with ping -M do.
  • Freeze for about 90 seconds after a server reboot: NFSv4 grace period. It clears on its own.
  • Stale file handle: server-side file or file system changed. cd /, remount, and use a stable fsid on the export.
  • umount: target is busy: find users with fuser -vm, kill them, then umount -f or umount -l.
  • Processes stuck in D state forever: bring the server (or its IP) back, or reboot the client after a lazy unmount.
  • Exports work for one client but not another: check the client list in /etc/exports; see NFS export to multiple hosts.

Next steps

Upcoming posts in this NFS series cover Kerberos-secured NFS (sec=krb5p) with MIT Kerberos and with Active Directory, and rotating NFS service keytabs. If you want to prepare, the Kerberos series starts with Set Up an MIT Kerberos KDC on AlmaLinux 10 and Create and Manage Kerberos Keytabs on AlmaLinux 10. For every mount option used here, see the nfs(5), umount(8), nfsstat(8) and rpcdebug(8) man pages, and the Red Hat documentation on mounting NFS shares.

, , , , , , , , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *