A single KDC is a single point of failure: if it goes down, nobody can get a ticket and every Kerberized SSH login, NFS mount and web service stops authenticating. MIT Kerberos solves this with replica KDCs. The primary KDC holds the writable database and runs kadmind, and one or more replicas hold read-only copies that answer ticket requests. The database is copied from the primary to each replica with kprop and received by the kpropd daemon.
This guide adds a replica KDC to the realm built in Set Up an MIT Kerberos KDC on AlmaLinux 10 and uses the host keytabs covered in Create and Manage Kerberos Keytabs on AlmaLinux 10. It targets AlmaLinux 10 (10.2 at the time of writing, krb5 1.21), and the same steps work on AlmaLinux 9.x. MIT now uses the terms primary and replica for what older documentation calls master and slave.
How primary and replica KDCs work
- Primary KDC: runs
krb5kdcandkadmind. All changes (new principals, password changes, keytab exports) happen here. - Replica KDC: runs
krb5kdcandkpropd, but notkadmind. It issues tickets from its copy of the database and cannot be changed directly. - Propagation: the primary dumps its database with
kdb5_util dumpand pushes it to each replica withkpropover TCP port 754.kpropdon the replica checks that the sender is allowed bykpropd.acl, then loads the dump. - Authentication:
kpropandkpropdauthenticate each other with thehost/keys in each KDC’s/etc/krb5.keytab.
Propagation does not copy configuration files, the ACL file or the master key stash file. You copy those by hand once, during setup.
Lab layout used in this guide
- Realm:
EXAMPLE.COM - Primary KDC:
kdc1.example.comat192.168.1.10(already runningkrb5kdcandkadmin) - New replica KDC:
kdc2.example.comat192.168.1.11, AlmaLinux 10.2 minimal, SELinux enforcing, firewalld running - Admin principal:
admin/admin@EXAMPLE.COM - Test user:
alice@EXAMPLE.COM
Replace these names with your own everywhere they appear.
Prerequisites
- A working primary KDC. If you do not have one, follow Set Up an MIT Kerberos KDC on AlmaLinux 10 first.
- Forward and reverse DNS (or identical
/etc/hostsentries on every machine) for bothkdc1.example.comandkdc2.example.com. - Time synchronized with chrony on both KDCs. Propagation fails with a clock skew error if they drift more than 5 minutes apart.
- Root or sudo access on both KDCs and SSH access between them for copying files.
- Port 749/tcp (kadmin) on the primary reachable from the replica, and port 754/tcp (kprop) on the replica reachable from the primary.
Step 1: Prepare the replica server
On kdc2.example.com, set the hostname, check name resolution and time, and install the same packages as the primary:
sudo hostnamectl set-hostname kdc2.example.com
# Lab only: skip these if DNS already resolves both KDCs
echo "192.168.1.10 kdc1.example.com kdc1" | sudo tee -a /etc/hosts
echo "192.168.1.11 kdc2.example.com kdc2" | sudo tee -a /etc/hosts
hostname -f
getent hosts kdc1.example.com kdc2.example.com
timedatectl | grep synchronized
sudo dnf install -y krb5-server krb5-workstation krb5-libsIf you used /etc/hosts, add the kdc2 line on kdc1 and your clients too.
Step 2: Copy the configuration and stash file from the primary
The replica needs the same krb5.conf, kdc.conf and kadm5.acl as the primary, plus the master key stash file so krb5kdc can decrypt the database it receives. Run these on kdc2:
sudo scp root@kdc1.example.com:/etc/krb5.conf /etc/krb5.conf
sudo scp root@kdc1.example.com:/var/kerberos/krb5kdc/kdc.conf /var/kerberos/krb5kdc/kdc.conf
sudo scp root@kdc1.example.com:/var/kerberos/krb5kdc/kadm5.acl /var/kerberos/krb5kdc/kadm5.acl
sudo scp root@kdc1.example.com:/var/kerberos/krb5kdc/.k5.EXAMPLE.COM /var/kerberos/krb5kdc/.k5.EXAMPLE.COM
sudo chmod 600 /var/kerberos/krb5kdc/.k5.EXAMPLE.COM
sudo restorecon -Rv /etc/krb5.conf /var/kerberos/krb5kdcThe stash file is as sensitive as the master password itself, so only ever move it over an encrypted channel such as scp, and never leave a copy in /tmp. kadm5.acl is not used while kdc2 is a replica, but having it in place lets you promote the replica later without extra work.
Step 3: Create host principals and keytabs for both KDCs
Each KDC needs a host/ principal with its key in /etc/krb5.keytab. kprop on the primary authenticates as host/kdc1.example.com, and kpropd on the replica accepts the connection as host/kdc2.example.com.
On kdc1, check whether its host key is already in the keytab:
sudo klist -k /etc/krb5.keytabIf host/kdc1.example.com@EXAMPLE.COM is missing, create the principal (if needed) and export it locally. Skip the addprinc line if the principal already exists:
sudo kadmin.local -q "addprinc -randkey host/kdc1.example.com"
sudo kadmin.local -q "ktadd host/kdc1.example.com"Remember that ktadd randomizes the key and bumps the kvno, so if another copy of this keytab exists elsewhere it becomes stale. The keytabs guide covers this in detail.
Now, on kdc2, create the replica’s host principal and extract its key straight into the local keytab through kadmind. Extracting on the machine that will use the key avoids copying keytabs around. Each command prompts for the admin/admin password:
sudo kadmin -p admin/admin -q "addprinc -randkey host/kdc2.example.com"
sudo kadmin -p admin/admin -q "ktadd host/kdc2.example.com"
sudo klist -ke /etc/krb5.keytab
sudo restorecon -v /etc/krb5.keytabklist -ke should list host/kdc2.example.com@EXAMPLE.COM with the AES encryption types.
Step 4: Create kpropd.acl
kpropd only accepts a database from principals listed in /var/kerberos/krb5kdc/kpropd.acl. List the host principals of every KDC so that any of them can act as primary later. Create the same file on both kdc1 and kdc2:
sudo tee /var/kerberos/krb5kdc/kpropd.acl > /dev/null <<'EOF'
host/kdc1.example.com@EXAMPLE.COM
host/kdc2.example.com@EXAMPLE.COM
EOF
sudo chmod 600 /var/kerberos/krb5kdc/kpropd.acl
sudo restorecon -v /var/kerberos/krb5kdc/kpropd.aclThe AlmaLinux kprop systemd unit refuses to start if this file does not exist, which is a common first stumbling block.
Step 5: Start kpropd and open the firewall on the replica
On AlmaLinux, kpropd runs as the kprop service. Do not start krb5kdc on the replica yet, because it has no database until the first propagation. On kdc2:
sudo systemctl enable --now kprop
sudo systemctl status kprop --no-pager
sudo ss -tlpn | grep ':754\b'
sudo firewall-cmd --permanent --add-service={kerberos,kprop}
sudo firewall-cmd --reload
sudo firewall-cmd --list-servicesfirewalld’s kprop service opens 754/tcp, and kerberos opens 88/tcp and 88/udp for ticket requests. The replica does not need kadmin or kpasswd open because it does not run kadmind.
Step 6: Propagate the database for the first time
On the primary (kdc1), dump the database to a file and push it to the replica:
sudo kdb5_util dump /var/kerberos/krb5kdc/replica_datatrans
sudo kprop -f /var/kerberos/krb5kdc/replica_datatrans kdc2.example.comA successful run prints:
Database propagation to kdc2.example.com: SUCCEEDEDOn kdc2, confirm the database arrived and can be read with the stashed master key:
sudo ls -la /var/kerberos/krb5kdc/
sudo kadmin.local -q "listprincs"You should see the principal database files and the same principal list as on the primary, including alice@EXAMPLE.COM.
Step 7: Start the KDC on the replica
Now that the replica has a database, start krb5kdc on kdc2 and leave kadmin disabled:
sudo systemctl enable --now krb5kdc
sudo systemctl status krb5kdc --no-pager
sudo systemctl is-enabled kadmin # should report disabled
sudo ss -tulpn | grep ':88\b'Step 8: Automate propagation with a systemd timer
Without incremental propagation, changes made on the primary only reach the replica when you run kprop. Password changes and new principals will not work against the replica until then, so schedule it. On kdc1, create a small script:
sudo tee /usr/local/sbin/kprop-replicas.sh > /dev/null <<'EOF'
#!/bin/bash
# Dump the Kerberos database and push it to every replica KDC.
set -euo pipefail
REPLICAS="kdc2.example.com"
DUMP=/var/kerberos/krb5kdc/replica_datatrans
/usr/sbin/kdb5_util dump "$DUMP"
rc=0
for kdc in $REPLICAS; do
if ! /usr/sbin/kprop -f "$DUMP" "$kdc"; then
echo "Propagation to $kdc FAILED" >&2
rc=1
fi
done
exit $rc
EOF
sudo chmod 700 /usr/local/sbin/kprop-replicas.sh
sudo restorecon -v /usr/local/sbin/kprop-replicas.shThen create a service and a timer that run it every 10 minutes:
sudo tee /etc/systemd/system/kprop-replicas.service > /dev/null <<'EOF'
[Unit]
Description=Propagate Kerberos database to replica KDCs
After=network-online.target krb5kdc.service
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/kprop-replicas.sh
EOF
sudo tee /etc/systemd/system/kprop-replicas.timer > /dev/null <<'EOF'
[Unit]
Description=Run Kerberos database propagation every 10 minutes
[Timer]
OnBootSec=5min
OnUnitActiveSec=10min
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now kprop-replicas.timer
sudo systemctl start kprop-replicas.service
systemctl list-timers kprop-replicas.timer
sudo journalctl -u kprop-replicas.service -n 20 --no-pagerPick an interval that matches how quickly you need changes to reach the replicas. Add more replicas by listing them, separated by spaces, in REPLICAS.
Step 9: Point clients at both KDCs
Clients only use the replica if they know about it. Add a second kdc line to the realm in /etc/krb5.conf on every client and on both KDCs. Keep admin_server pointing at the primary, because only it runs kadmind:
[realms]
EXAMPLE.COM = {
kdc = kdc1.example.com
kdc = kdc2.example.com
admin_server = kdc1.example.com
primary_kdc = kdc1.example.com
}The Kerberos library tries the KDCs in order and moves to the next one if a KDC does not answer. If a replica rejects a request, for example a fresh password change it has not received yet, the library retries against the KDC named in primary_kdc (or the _kerberos-master SRV record), so users are not locked out between propagations.
If you publish DNS SRV records instead, add the replica alongside the primary and keep _kerberos-master, _kerberos-adm and _kpasswd pointing only at kdc1:
_kerberos._udp.example.com. IN SRV 0 0 88 kdc1.example.com.
_kerberos._udp.example.com. IN SRV 0 0 88 kdc2.example.com.
_kerberos._tcp.example.com. IN SRV 0 0 88 kdc1.example.com.
_kerberos._tcp.example.com. IN SRV 0 0 88 kdc2.example.com.
_kerberos-master._udp.example.com. IN SRV 0 0 88 kdc1.example.com.
_kerberos-adm._tcp.example.com. IN SRV 0 0 749 kdc1.example.com.
_kpasswd._udp.example.com. IN SRV 0 0 464 kdc1.example.com.Verify the replica KDC
Get a ticket directly from the replica
Kerberos trace logging shows which KDC answers. On any client, temporarily create a config that lists only the replica and request a ticket:
sed 's/^\(\s*kdc = \)kdc1.example.com/\1kdc2.example.com/' /etc/krb5.conf > /tmp/krb5-replica.conf
grep -A4 'EXAMPLE.COM = {' /tmp/krb5-replica.conf
KRB5_CONFIG=/tmp/krb5-replica.conf KRB5_TRACE=/tmp/kinit-trace.log kinit alice
grep -i 'sending' /tmp/kinit-trace.log
klist
kdestroy
rm /tmp/krb5-replica.conf /tmp/kinit-trace.logThe trace lines should show requests sent to 192.168.1.11:88, and klist should show a krbtgt/EXAMPLE.COM@EXAMPLE.COM ticket.
Test failover
Stop the KDC on the primary, get a ticket with the normal configuration, then start the primary again:
# On kdc1
sudo systemctl stop krb5kdc
# On a client
kdestroy
kinit alice
klist
# On kdc1
sudo systemctl start krb5kdckinit takes a moment longer while the library gives up on kdc1, then succeeds against kdc2. The replica’s log confirms it handled the request:
sudo tail -n 5 /var/log/krb5kdc.log # on kdc2, look for AS_REQ ... ISSUEConfirm changes propagate
# On kdc1: create a test principal and push it right away
sudo kadmin.local -q "addprinc -randkey testprop"
sudo systemctl start kprop-replicas.service
# On kdc2: the principal should now exist
sudo kadmin.local -q "getprinc testprop"
# Clean up on kdc1 (it disappears from kdc2 on the next propagation)
sudo kadmin.local -q "delprinc -force testprop"Optional: incremental propagation
Full dumps are simple and reliable for small and medium realms. For large databases, MIT Kerberos also supports incremental propagation (iprop), where kpropd on each replica polls kadmind on the primary every two minutes by default and fetches only the changes. It requires iprop_enable = true and a matching iprop_port in kdc.conf on both sides, a kiprop/ principal for each KDC in its keytab, and the primary’s ACL granting the replica’s kiprop/ principal the p privilege. When iprop is enabled the regular kprop push is disabled, so remove the timer if you switch. Because kadmind must listen on the iprop port, you may also need to label that port for SELinux with semanage port. See the MIT incremental propagation documentation for the full setup.
Promoting the replica to primary
If the primary is lost or needs maintenance, you can switch roles. Because both KDCs share the same configuration, ACL, stash file and kpropd.acl, it only takes a few steps:
- If the old primary still runs, stop
kadminand thekprop-replicas.timeron it, then runkprop-replicas.serviceonce by hand so the replica has the latest data. - On the new primary, stop
kprop, start and enablekadmin, openkadminandkpasswdin firewalld, and install the propagation script and timer with the old primary listed as the replica. - On the old primary, enable
kpropso it can receive the database. - Update
admin_serverinkrb5.conf, or better, move a DNS CNAME such askerberos.example.comthat clients use for the admin server.
Troubleshooting
kprop: Connection refused while connecting to server
kpropd is not running on the replica. Check systemctl status kprop on kdc2. If it failed with an assertion error, /var/kerberos/krb5kdc/kpropd.acl is missing.
kprop: No route to host while connecting to server
The firewall on the replica is blocking port 754/tcp. Run sudo firewall-cmd --list-services on kdc2 and confirm kprop is listed, then test from the primary with nc -vz kdc2.example.com 754.
kprop: Key table entry not found while getting initial credentials
The primary’s /etc/krb5.keytab has no host/kdc1.example.com key. Check with sudo klist -k and repeat Step 3 on kdc1.
kprop: Server rejected authentication (during sendauth exchange) while authenticating to server
The replica could not verify the request. Common causes are a stale host/kdc2.example.com key in the replica’s keytab (compare sudo klist -k on kdc2 with kadmin.local -q "getprinc host/kdc2.example.com" on kdc1), a missing or mistyped line in kpropd.acl, clock skew, or a hostname that does not resolve to the name in the principal. sudo journalctl -u kprop on the replica usually names the exact problem.
krb5kdc on the replica fails to start
Check sudo journalctl -u krb5kdc. A message about the master key means the stash file is missing or wrong, so copy it again from the primary. A missing database means no propagation has succeeded yet. After copying files, run sudo restorecon -Rv /var/kerberos and look for SELinux denials with sudo ausearch -m avc -ts recent.
Password changes or new principals do not work against the replica
The replica has not received the change yet. Run sudo systemctl start kprop-replicas.service on the primary and check its journal, or shorten the timer interval.
Next steps
Your realm now survives the loss of its primary KDC. Keep monitoring the propagation timer, because a replica that silently stops receiving updates serves stale keys. Continue taking regular kdb5_util dump backups on the primary, and store the master password separately from them. The next post in this series, Configure a Kerberos Client on AlmaLinux 10, covers configuring AlmaLinux clients to authenticate against the realm.
References: MIT Kerberos: Installing KDCs, kprop and kpropd manual pages.