Ceph is the open-source, software-defined storage platform that gives you block devices (RBD), a POSIX file system (CephFS) and S3-compatible object storage from one cluster of ordinary servers. This guide builds a three-node Ceph cluster on AlmaLinux 10 with cephadm, the official deployment and management tool, then creates an RBD block device and a CephFS file system and mounts both on a Linux client.
Versions at the time of writing: AlmaLinux 10.2 and Ceph Tentacle 20.2.4, the newest active stable release. The same steps work on AlmaLinux 9.x, Rocky Linux, Oracle Linux, RHEL and CentOS Stream; on an EL9 host cephadm add-repo simply points at the el9 repository instead of el10.
What you will build
- Three AlmaLinux 10 hosts,
ceph1,ceph2andceph3, each running a monitor and an OSD, with two managers for failover - One OSD per host on a spare, empty disk (
/dev/sdbin the examples) - A replicated RBD pool and a 10 GB block image mapped, formatted and mounted on
ceph1 - A CephFS file system with its own restricted client key, mounted with the kernel client
- The Ceph Dashboard on port 8443 for a web view of the cluster
Prerequisites
- Three AlmaLinux 10 servers (VMs are fine for a lab) with at least 2 vCPUs, 4 GB RAM and 20 GB for the OS each. Plan on roughly 4 GB of RAM per OSD in production.
- One extra, completely empty disk per server for the OSD. It must have no partitions, no LVM metadata and no file system. If you need to add a disk to a VM first, see Add New Hard Drive to CentOS Linux.
- Static IP addresses on the same network. The examples use
192.168.1.41,.42and.43. - Root access over SSH from
ceph1to the other two hosts (cephadm manages hosts as root by default). - Internet access to
download.ceph.comandquay.io, where the Ceph container images live.
Step 1: Prepare all three hosts
Run this section on every node. Set the short hostname (use ceph2 and ceph3 on the other nodes) and make every node resolvable from every other node:
sudo hostnamectl set-hostname ceph1
sudo tee -a /etc/hosts <<'EOF'
192.168.1.41 ceph1
192.168.1.42 ceph2
192.168.1.43 ceph3
EOFcephadm runs every Ceph daemon in a container, so each host needs Podman, LVM2 (OSDs are built on LVM), Python 3 and a working time sync. Monitors are very sensitive to clock skew, so make sure chrony is running:
sudo dnf -y update
sudo dnf -y install podman lvm2 python3 chrony
sudo systemctl enable --now chronyd
chronyc trackingLeave firewalld and SELinux enabled. cephadm opens the firewalld ports for each daemon it deploys (3300 and 6789 for monitors, 6800 to 7300 for OSDs and managers, 8443 for the dashboard), and the Ceph containers run fine with SELinux enforcing.
sudo firewall-cmd --state
getenforceConfirm the spare disk is visible and empty:
lsblk /dev/sdbStep 2: Install cephadm on the first node
The rest of the setup runs on ceph1 as root. Download the standalone cephadm binary for the release you want, then use it to add the official Ceph repository and install the packaged version. add-repo also installs epel-release, which the Ceph client packages depend on:
sudo -i
CEPH_RELEASE=20.2.4
curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el10/noarch/cephadm
chmod +x cephadm
./cephadm add-repo --release tentacle
./cephadm install
rm -f ./cephadm
which cephadm
cephadm versionIf a dependency cannot be found while installing packages, enable the CodeReady Builder repository, which some EPEL packages need, and try again:
dnf config-manager --set-enabled crbStep 3: Bootstrap the cluster
Bootstrapping creates the first monitor and manager on ceph1, generates the cluster SSH key, writes /etc/ceph/ceph.conf and the admin keyring, and starts the dashboard. Pass the IP address of ceph1; cephadm derives the public network from it:
cephadm bootstrap --mon-ip 192.168.1.41The command pulls the Ceph container image and finishes with a summary like this. Save the dashboard password it prints:
Ceph Dashboard is now available at:
URL: https://ceph1:8443/
User: admin
Password: xxxxxxxxxx
Bootstrap complete.Install the Ceph command-line tools on the host so you can run ceph, rbd and mount.ceph directly instead of going through cephadm shell:
cephadm install ceph-common
ceph -v
ceph -sAt this point ceph -s shows one monitor, one manager and HEALTH_WARN because there are no OSDs yet. That is expected.
Step 4: Add the other hosts
cephadm connects to each host over SSH with its own key. Copy the cluster public key to ceph2 and ceph3 (you will be asked for their root passwords once):
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph3Add the hosts to the cluster. The _admin label tells cephadm to keep a copy of ceph.conf and the admin keyring on those hosts as well, so you can manage the cluster from any of them:
ceph orch host add ceph2 192.168.1.42 --labels _admin
ceph orch host add ceph3 192.168.1.43 --labels _admin
ceph orch host lsBy default cephadm deploys up to five monitors and two managers across the available hosts, so after a minute or two you should have a monitor on each of the three nodes and a standby manager:
ceph orch ps --daemon-type mon
ceph orch ps --daemon-type mgrStep 5: Create the OSDs
List the disks cephadm can see. A disk shows Available: Yes only if it is larger than 5 GB, has no partitions, no LVM state, no file system and is not mounted:
ceph orch device lsHOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph1 /dev/sdb hdd 50.0G Yes 2m ago
ceph2 /dev/sdb hdd 50.0G Yes 2m ago
ceph3 /dev/sdb hdd 50.0G Yes 2m agoEither tell cephadm to turn every available disk into an OSD, now and whenever a new empty disk appears:
ceph orch apply osd --all-available-devicesOr add specific disks one at a time, which is the safer choice on servers with disks you want to keep out of Ceph:
ceph orch daemon add osd ceph1:/dev/sdb
ceph orch daemon add osd ceph2:/dev/sdb
ceph orch daemon add osd ceph3:/dev/sdbCheck that all three OSDs are up and in:
ceph osd tree
ceph osd dfStep 6: Verify cluster health
ceph -sA healthy cluster looks like this:
cluster:
id: 6f1c2a7e-...
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 5m)
mgr: ceph1.abcdef(active, since 10m), standbys: ceph2.ghijkl
osd: 3 osds: 3 up (since 2m), 3 in (since 2m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 80 MiB used, 150 GiB / 150 GiB avail
pgs: 1 active+cleanIf the health is anything other than HEALTH_OK, ceph health detail explains why.
Step 7: Create an RBD block device
RBD (RADOS Block Device) gives you thin-provisioned, replicated block devices, the same thing OpenStack, Proxmox and Kubernetes use for VM and container disks. Create a pool, initialise it for RBD and create a 10 GB image:
ceph osd pool create rbd
rbd pool init rbd
rbd create --size 10G rbd/disk01
rbd ls -l rbd
rbd info rbd/disk01Map the image with the kernel RBD driver, then format and mount it like any other disk:
modprobe rbd
rbd map rbd/disk01
rbd showmapped
mkfs.xfs /dev/rbd0
mkdir -p /mnt/rbd01
mount /dev/rbd0 /mnt/rbd01
df -h /mnt/rbd01To map and mount the image automatically at boot, list it in /etc/ceph/rbdmap, enable the rbdmap service and use the stable /dev/rbd/POOL/IMAGE path in /etc/fstab:
echo "rbd/disk01 id=admin,keyring=/etc/ceph/ceph.client.admin.keyring" >> /etc/ceph/rbdmap
systemctl enable rbdmap
echo "/dev/rbd/rbd/disk01 /mnt/rbd01 xfs noauto,_netdev 0 0" >> /etc/fstabThe rbdmap service mounts the noauto entries itself after it maps the images. RBD images can be grown online with rbd resize --size 20G rbd/disk01 followed by xfs_growfs /mnt/rbd01, much like expanding space with LVM.
Step 8: Create and mount a CephFS file system
CephFS is a shared POSIX file system that many clients can mount at once, a scalable alternative to an NFS server. One command creates the data and metadata pools and deploys the metadata server (MDS) daemons:
ceph fs volume create cephfs
ceph fs ls
ceph fs status cephfsDo not mount it with the admin key. Create a client key that can only read and write this file system, and save its keyring where mount.ceph looks for it:
ceph fs authorize cephfs client.fsuser / rw > /etc/ceph/ceph.client.fsuser.keyring
chmod 600 /etc/ceph/ceph.client.fsuser.keyringMount it with the kernel client. The device string is USER@FSID.FSNAME=/PATH; leaving the FSID empty tells mount.ceph to read it and the monitor addresses from /etc/ceph/ceph.conf:
mkdir -p /mnt/cephfs
mount -t ceph fsuser@.cephfs=/ /mnt/cephfs
df -h /mnt/cephfs
echo "hello from $(hostname)" > /mnt/cephfs/test.txtFor a permanent mount, add this line to /etc/fstab:
fsuser@.cephfs=/ /mnt/cephfs ceph noatime,_netdev 0 0Older kernels that do not understand the new device string (for example some EL9 kernels) can use the legacy syntax instead: mount -t ceph :/ /mnt/cephfs -o name=fsuser,fs=cephfs.
Another host can mount the same file system by installing ceph-common from the same repository and copying /etc/ceph/ceph.conf and the fsuser keyring to it. Mount it there and you will see the same test.txt.
Step 9: Open the Ceph Dashboard
Browse to https://ceph1:8443/ (or the IP address), accept the self-signed certificate and log in as admin with the password from the bootstrap output. You will be asked to set a new password. The dashboard shows health, hosts, OSDs, pools, RBD images and CephFS, and can deploy most services. If you lost the password, set a new one from the command line:
echo -n 'N3w-Str0ng-Passw0rd' > /root/dashpass.txt
ceph dashboard ac-user-set-password admin -i /root/dashpass.txt --force-password
rm -f /root/dashpass.txtUseful day-two commands
ceph -s # overall status
ceph health detail # why the cluster is not HEALTH_OK
ceph orch ls # services and their placement
ceph orch ps # every daemon on every host
ceph osd tree # OSD layout and state
ceph df # raw and per-pool usage
ceph osd pool ls detail # pool size, min_size, PG count
ceph log last cephadm # recent cephadm actions and errors
cephadm shell # container with all Ceph toolsTroubleshooting
A disk shows Available: No
Check the reject reasons in ceph orch device ls --wide. Old partitions, LVM volumes or a file system signature make cephadm skip a disk. If you are sure the data on it can be destroyed, wipe it and wait for the next device refresh:
ceph orch device zap ceph2 /dev/sdb --forceHost add fails with an SSH error
Make sure the cluster key from /etc/ceph/ceph.pub is in /root/.ssh/authorized_keys on that host, that root SSH login is allowed (PermitRootLogin prohibit-password is enough for key logins) and that the host name you used resolves. ceph cephadm check-host ceph2 shows exactly what is missing, such as Podman, LVM2 or time sync.
HEALTH_WARN clock skew detected
Monitors need their clocks within a fraction of a second of each other. Check chronyc sources on every node and point them all at the same NTP servers.
HEALTH_WARN pool has no application enabled
A pool created with ceph osd pool create must be tagged with what uses it. rbd pool init does this for RBD; for other pools run ceph osd pool application enable POOL rbd|cephfs|rgw.
PGs stuck undersized or degraded
Replicated pools keep three copies on three different hosts by default, so you need at least three hosts with an up OSD. Check ceph osd tree for down OSDs and ceph orch ps --daemon-type osd for failed containers, then look at the logs with cephadm logs --name osd.1 on the host that runs the OSD.
rbd map fails
Run dmesg | tail right after the failure. If the rbd module cannot be loaded, make sure the full kernel modules package for your running kernel is installed (dnf install kernel-modules-$(uname -r)). If the kernel complains about unsupported image features, disable them on the image, for example rbd feature disable rbd/disk01 object-map fast-diff deep-flatten.
Next steps
From here you can add the Ceph Object Gateway for S3-compatible storage (ceph orch apply rgw), serve CephFS to legacy clients with the NFS service (ceph nfs cluster create), or use the cluster as persistent storage for Kubernetes through the Ceph CSI drivers. If you only need a simple shared folder on one server, a plain NFS export to multiple hosts is still the lighter option.
Official references: the cephadm installation guide, basic RBD commands and mounting CephFS with the kernel driver.