Create and Manage Kerberos Keytabs on AlmaLinux 10

October 18, 2026 · by Zwiegnet Consulting · Linux, Security

A keytab (key table) is a file that holds the long-term Kerberos keys for one or more principals. Services such as SSH, Apache, NFS and LDAP use a keytab to prove their identity and decrypt the tickets clients present, and scripts or cron jobs can use one to run kinit without anyone typing a password. Because a keytab is effectively a stored password, you need to create, protect and rotate it carefully.

This guide continues from Set Up an MIT Kerberos KDC on AlmaLinux 10 and shows how to create and manage keytabs on AlmaLinux 10 (10.2 at the time of writing, krb5 1.21) with kadmin, klist and ktutil. Everything here also works on AlmaLinux 9.x.

How keytabs work

Each keytab entry stores a principal name, a key version number (kvno), an encryption type and the key itself. When a client requests a service ticket, the KDC encrypts it with the service’s current key. The service decrypts it with the matching entry in its keytab. If the kvno or key in the keytab does not match what the KDC has, authentication fails, so most keytab problems come down to stale or mismatched keys.

Two rules follow from this. First, the ktadd command in kadmin randomizes the principal’s keys every time it runs and bumps the kvno, which invalidates every other copy of that principal’s keytab. Second, a password change on a user principal invalidates any keytab built from the old password.

Lab layout used in this guide

  • Realm: EXAMPLE.COM
  • KDC and admin server: kdc1.example.com at 192.168.1.10 (built in the previous post)
  • Application server: web1.example.com at 192.168.1.20, AlmaLinux 10.2, SELinux enforcing
  • Admin principal: admin/admin@EXAMPLE.COM, allowed by */admin@EXAMPLE.COM * in kadm5.acl

Replace these names with your own everywhere they appear.

Prerequisites

  • A working MIT Kerberos KDC with krb5kdc and kadmin running. If you do not have one yet, follow Set Up an MIT Kerberos KDC on AlmaLinux 10 first.
  • Forward and reverse DNS (or matching /etc/hosts entries) for web1.example.com and kdc1.example.com.
  • Time synchronized with chrony on both machines. Kerberos rejects requests when clocks differ by more than 5 minutes by default.
  • Port 88 (Kerberos) and 749/tcp (kadmin) on the KDC reachable from web1.
  • Root or sudo access on both machines.

Step 1: Install the Kerberos client tools on the application server

On web1.example.com, install the workstation package, which provides kinit, klist, kadmin, ktutil and kvno:

sudo dnf install -y krb5-workstation

Copy /etc/krb5.conf from the KDC so both machines agree on the realm and KDC location, then confirm you can authenticate as the admin principal:

sudo scp root@kdc1.example.com:/etc/krb5.conf /etc/krb5.conf
sudo restorecon -v /etc/krb5.conf

kinit admin/admin
klist
kdestroy

If kinit fails here, fix name resolution, time or the firewall before going further. Keytabs will not work until plain password authentication does.

Step 2: Create the service principals

Service principals use random keys because no human ever types their password. Create a host principal for web1 (used by SSH and other system services) and an HTTP principal for a web service. You can run kadmin remotely from web1; it prompts for the admin/admin password:

kadmin -p admin/admin -q "addprinc -randkey host/web1.example.com"
kadmin -p admin/admin -q "addprinc -randkey HTTP/web1.example.com"
kadmin -p admin/admin -q "listprincs *web1*"

The service type (host, HTTP, nfs, ldap and so on) is case sensitive and must be followed by the fully qualified hostname clients use to reach the service.

Step 3: Extract the host keytab on the host itself

The MIT documentation recommends running kadmin on the machine where the keytab will live, so the key never crosses the network in a file. Run it with sudo so it can write the default system keytab, /etc/krb5.keytab:

sudo kadmin -p admin/admin -q "ktadd host/web1.example.com"

You will see one line per encryption type, similar to Entry for principal host/web1.example.com with kvno 2, encryption type aes256-cts-hmac-sha384-192 added to keytab FILE:/etc/krb5.keytab. The kvno is 2 because ktadd replaced the random key created by addprinc -randkey.

Lock the file down and make sure it has the correct SELinux label:

sudo chown root:root /etc/krb5.keytab
sudo chmod 600 /etc/krb5.keytab
sudo restorecon -v /etc/krb5.keytab
ls -lZ /etc/krb5.keytab

The SELinux type should be krb5_keytab_t.

Alternative: extract on the KDC and copy the file

If the host cannot reach kadmind, extract the keytab on the KDC with kadmin.local and copy it over an encrypted channel, then delete the temporary copy:

# On kdc1.example.com
sudo kadmin.local -q "ktadd -k /root/web1.keytab host/web1.example.com"
sudo scp /root/web1.keytab root@web1.example.com:/etc/krb5.keytab
sudo shred -u /root/web1.keytab

# On web1.example.com
sudo chmod 600 /etc/krb5.keytab
sudo restorecon -v /etc/krb5.keytab

Step 4: Inspect the keytab

klist -k lists the entries in a keytab. Add -t for timestamps and -e for encryption types:

sudo klist -kte /etc/krb5.keytab

With the AES-only supported_enctypes from the KDC guide, you should see four entries for host/web1.example.com@EXAMPLE.COM, all with the same kvno, one each for aes256-cts-hmac-sha384-192, aes128-cts-hmac-sha256-128, aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96.

Avoid klist -k -K on shared terminals or in tickets and chat logs, because -K prints the raw key material.

Step 5: Verify the keytab against the KDC

A keytab is only useful if its keys match the KDC. Test it in three ways. First, get a ticket using only the keytab, with no password prompt:

sudo kinit -k host/web1.example.com
sudo klist
sudo kdestroy

-k without -t uses the default keytab, /etc/krb5.keytab. Second, compare the kvno the KDC currently has with the kvno in the file:

kinit admin/admin
kvno host/web1.example.com
sudo klist -k /etc/krb5.keytab | grep host/web1

kvno prints a line such as host/web1.example.com@EXAMPLE.COM: kvno = 2. The highest kvno in the keytab must match. Third, you can confirm the keys and kvno on the KDC side:

kadmin -p admin/admin -q "getprinc host/web1.example.com"

Step 6: Create a separate keytab for a service account

Do not put application keys in /etc/krb5.keytab, which only root should read. Give each service its own keytab readable by the service’s user. For example, for Apache with mod_auth_gssapi:

sudo kadmin -p admin/admin -q "ktadd -k /etc/httpd/http.keytab HTTP/web1.example.com"
sudo chown root:apache /etc/httpd/http.keytab
sudo chmod 640 /etc/httpd/http.keytab
sudo restorecon -v /etc/httpd/http.keytab
sudo klist -kt /etc/httpd/http.keytab

Point the service at the file in its own configuration (for mod_auth_gssapi that is GssapiCredStore keytab:/etc/httpd/http.keytab). For a service that uses the standard Kerberos libraries without its own setting, set KRB5_KTNAME with a systemd drop-in:

sudo systemctl edit myapp.service
[Service]
Environment=KRB5_KTNAME=FILE:/etc/myapp/myapp.keytab
sudo systemctl restart myapp.service

If you store a keytab outside a standard location, run restorecon and check ls -Z. If the service is denied access, look for AVC denials with sudo ausearch -m avc -ts recent and give the file a type the service’s SELinux domain can read with semanage fcontext.

Step 7: Use a keytab for scripts and cron jobs

Automated jobs that need to authenticate as a client (for example a backup script that mounts Kerberized NFS or calls a Kerberized API) should use a dedicated principal with a random key. Create it and its keytab:

kadmin -p admin/admin -q "addprinc -randkey svc-backup"

sudo useradd -r -m svc-backup
sudo install -d -o svc-backup -g svc-backup -m 700 /etc/security/keytabs/svc-backup
sudo kadmin -p admin/admin -q "ktadd -k /etc/security/keytabs/svc-backup/svc-backup.keytab svc-backup"
sudo chown svc-backup:svc-backup /etc/security/keytabs/svc-backup/svc-backup.keytab
sudo chmod 600 /etc/security/keytabs/svc-backup/svc-backup.keytab

Test it as the service user:

sudo -u svc-backup kinit -k -t /etc/security/keytabs/svc-backup/svc-backup.keytab svc-backup@EXAMPLE.COM
sudo -u svc-backup klist

In a cron job, give the job its own credential cache so it never collides with an interactive login (the variable line applies to every job in that crontab), and fetch a fresh ticket at the start of each run:

# sudo crontab -e -u svc-backup
KRB5CCNAME=FILE:/tmp/krb5cc_svc-backup
0 2 * * * kinit -k -t /etc/security/keytabs/svc-backup/svc-backup.keytab svc-backup@EXAMPLE.COM && /usr/local/bin/backup.sh

GSSAPI applications can also obtain and refresh credentials from a client keytab automatically. Set KRB5_CLIENT_KTNAME to the keytab path and KRB5CCNAME to a cache used only by that service, as described in the MIT application server documentation linked at the end.

Step 8: Build a keytab from a user password with ktutil

Sometimes you need a keytab for an existing principal without changing its keys, for example a user principal whose password other systems still use. ktutil can derive the keys from the password. You need the principal’s current kvno, which getprinc shows:

kadmin -p admin/admin -q "getprinc alice" | grep -i vno

Then create the keytab. The -f flag fetches the correct salt and encryption types from the KDC, so you do not have to guess them:

ktutil
ktutil:  addent -password -p alice@EXAMPLE.COM -k 1 -f
Password for alice@EXAMPLE.COM:
ktutil:  wkt /home/alice/alice.keytab
ktutil:  quit

chmod 600 /home/alice/alice.keytab
kinit -k -t /home/alice/alice.keytab alice@EXAMPLE.COM

Replace 1 with the kvno from getprinc. Remember that this keytab stops working the next time alice changes her password.

Step 9: Merge, list and clean up keytabs with ktutil

ktutil keeps a list of entries in memory. Read keytabs in with rkt, inspect them with list, remove entries with delent and write the result with wkt. To merge two keytabs into one:

sudo ktutil
ktutil:  rkt /etc/krb5.keytab
ktutil:  rkt /etc/httpd/http.keytab
ktutil:  list -e
ktutil:  wkt /root/combined.keytab
ktutil:  quit

wkt appends to an existing file rather than replacing it, so write to a new path and check the result with klist -kte.

To remove entries from a keytab without touching the KDC, use ktremove in kadmin. It needs no special privileges because it only edits the local file:

# Remove every entry for the HTTP principal from a keytab
sudo kadmin -p admin/admin -q "ktremove -k /etc/httpd/http.keytab HTTP/web1.example.com all"

Step 10: Rotate keytab keys

Rotate service keys periodically, and immediately if a keytab may have been exposed. Running ktadd again generates new random keys, increments the kvno, and appends the new entries to the keytab:

sudo kadmin -p admin/admin -q "ktadd host/web1.example.com"
sudo klist -kt /etc/krb5.keytab

The file now contains both the old and new kvno. Keep the old entries until tickets issued with the old key have expired (24 hours with the max_life from the KDC guide), then remove them. The old keyword deletes every entry except those with the highest kvno:

sudo kadmin -p admin/admin -q "ktremove host/web1.example.com old"
sudo klist -kt /etc/krb5.keytab

If the same principal must exist in more than one keytab, for example a service running on several load-balanced nodes, do not run ktadd separately on each node, because each run invalidates the previous one. Extract the keys once, or export the current keys without randomizing them using -norandkey from kadmin.local on the KDC:

# On kdc1.example.com: export existing keys without changing them
sudo kadmin.local -q "ktadd -norandkey -k /root/http-node2.keytab HTTP/www.example.com"

Keytab security checklist

  • Keep /etc/krb5.keytab owned by root with mode 600.
  • Give each service its own keytab containing only its own principals, readable only by that service’s user or group.
  • Extract keytabs on the host that uses them whenever possible. Never email keytabs, commit them to Git or copy them over unencrypted channels.
  • Exclude keytabs from general-purpose backups, or encrypt those backups. Anyone holding a keytab can impersonate the principal.
  • Use random-key principals for services and automation instead of building keytabs from human passwords.
  • If a keytab leaks, run ktadd (or cpw -randkey) for that principal right away to invalidate the leaked keys.

Verify your setup

Run this quick check on the application server after any keytab change:

# Entries and kvno in the keytab
sudo klist -kte /etc/krb5.keytab

# Keytab-only authentication works
sudo kinit -k host/web1.example.com && sudo klist && sudo kdestroy

# KDC kvno matches the highest kvno in the keytab
kinit admin/admin && kvno host/web1.example.com && kdestroy

# Permissions and SELinux label
ls -lZ /etc/krb5.keytab

Troubleshooting

kinit: Key table file ‘/path/file.keytab’ not found while getting initial credentials

The path is wrong or the running user cannot see it. Check the path you passed with -t, and remember that kinit -k without -t uses /etc/krb5.keytab or the value of KRB5_KTNAME.

kinit: Keytab contains no suitable keys for the principal

The principal name in the command does not match any entry, or none of the entry’s encryption types is allowed. Compare the exact name and realm with klist -kte. Principal names are case sensitive, and HTTP/ is not the same as http/. Older krb5 releases report this as No key table entry found.

kinit: Preauthentication failed while getting initial credentials

The keys in the keytab no longer match the KDC, usually because someone ran ktadd on another machine or changed the principal’s password. Compare kvno with klist -k, then extract a fresh keytab.

Request ticket server … kvno N not found in keytab; ticket is likely out of date

The service’s keytab is older than the key the KDC used to issue the client’s ticket. Re-extract the service keytab, restart the service, and have clients run kdestroy and get new tickets.

Permission denied when kadmin writes or a service reads the keytab

Run ktadd with sudo when writing to a root-owned location, and check file ownership and mode for the service user. On SELinux systems also run sudo ausearch -m avc -ts recent and fix labels with restorecon -v.

Server not found in Kerberos database

The client asked for a principal that does not exist, often because DNS returned a different hostname than the one in the principal. Check getent hosts web1.example.com and the reverse lookup of its IP, and make sure the principal uses the name clients actually connect to.

Next steps

With host and service keytabs in place, the next posts in this series cover adding a replica KDC for redundancy and configuring AlmaLinux clients to authenticate against the realm. For background on where MIT Kerberos fits alongside directory services, see Future Of OpenLDAP and MIT Kerberos.

References: MIT Kerberos: keytab, MIT Kerberos: Application servers, kadmin reference and ktutil reference.

, , , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *