4. Use Ceph

4.1. Dashboard

$ ceph mgr module enable dashboard
module 'dashboard' is already enabled

It is enabled by default, but the listening port is only bind to address of ceph internal network. To make it bind to all addresses:

$ ceph config set mgr mgr/dashboard/server_addr 0.0.0.0

In the dashboard, you can do many things other than monitor the culster if you are admin.

4.2. Show information

$ ceph fsid
990b5070-3964-11f1-8888-476de7d3e05c

$ ceph mon dump
epoch 3
fsid 990b5070-3964-11f1-8888-476de7d3e05c
last_changed 2026-04-16T10:42:57.301680+0000
created 2026-04-16T07:19:36.819230+0000
min_mon_release 19 (squid)
election_strategy: 1
0: [v2:10.225.4.54:3300/0,v1:10.225.4.54:6789/0] mon.las3
1: [v2:10.225.4.53:3300/0,v1:10.225.4.53:6789/0] mon.las2
2: [v2:10.225.4.52:3300/0,v1:10.225.4.52:6789/0] mon.las1
dumped monmap epoch 3

4.3. CephFS

4.3.1. Create

Create a CephFS volume:

$ ceph fs volume create cephfs

List volumes:

$ ceph fs volume ls
[
    {
        "name": "cephfs"
    }
]

The Ceph Orchestrator will automatically create and configure MDS for your file system, show the MDSes:

$ ceph orch ps --daemon_type mds
NAME                    HOST  PORTS  STATUS        REFRESHED  AGE  MEM USE  MEM LIM  VERSION  IMAGE ID      CONTAINER ID  
mds.cephfs.las1.rfnkhk  las1         running (8m)     8m ago   8m    12.8M        -  19.2.3   af0c5903e901  5bd90f5152c2  
mds.cephfs.las3.roitqo  las3         running (8m)     8m ago   8m    12.9M        -  19.2.3   aade1b12b8e6  beaa5fb025d8

Show details about the new volume:

$ ceph fs volume info cephfs
{
    "mon_addrs": [
        "10.225.4.54:6789",
        "10.225.4.53:6789",
        "10.225.4.52:6789"
    ],
    "pools": {
        "data": [
            {
                "avail": 20368515072,
                "name": "cephfs.cephfs.data",
                "used": 0
            }
        ],
        "metadata": [
            {
                "avail": 20368515072,
                "name": "cephfs.cephfs.meta",
                "used": 98304
            }
        ]
    }
}

Show all the pools:

$ ceph osd pool ls
.mgr
cephfs.cephfs.meta
cephfs.cephfs.data

Note the name of data pool and metadata pool.

Create a subvolume group:

$ ceph fs subvolumegroup create cephfs ceph-sg

Subvolume groups can have many policies appied, including a quota of capacity, file mode, file owner, etc.

Show subvolume groups in a volume:

$ ceph fs subvolumegroup ls cephfs
[
    {
        "name": "ceph-sg"
    }
]

Get its path:

$ ceph fs subvolumegroup getpath cephfs ceph-sg
/volumes/ceph-sg

Check if there are some subvolume groups in a volume:

$ ceph fs subvolumegroup exist cephfs
subvolumegroup exists

Create a subvolume in a specified volume:

$ ceph fs subvolume create cephfs ceph-vol --group-name ceph-sg

If --group-name is omitted, the subvolume would be put in a default group _nogroup.

List subvolumes in a volume:

$ ceph fs subvolume ls cephfs ceph-sg
[
    {
        "name": "ceph-vol"
    }
]

If the subvolue is in the default group, the group name can be omitted.

Details of the subvolume:

$ ceph fs subvolume info cephfs ceph-vol ceph-sg
{
    "atime": "2026-04-17 07:35:09",
    "bytes_pcent": "undefined",
    "bytes_quota": "infinite",
    "bytes_used": 0,
    "created_at": "2026-04-17 07:35:09",
    "ctime": "2026-04-17 07:35:09",
    "data_pool": "cephfs.cephfs.data",
    "earmark": "",
    "features": [
        "snapshot-clone",
        "snapshot-autoprotect",
        "snapshot-retention"
    ],
    "flavor": 2,
    "gid": 0,
    "mode": 16877,
    "mon_addrs": [
        "10.225.4.54:6789",
        "10.225.4.53:6789",
        "10.225.4.52:6789"
    ],
    "mtime": "2026-04-17 07:35:09",
    "path": "/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0",
    "pool_namespace": "",
    "state": "complete",
    "type": "subvolume",
    "uid": 0
}

Note

The order of the parameters is <volume-name> <subvolume-name> <group-name>, or you can explicitly use --group-name <group-name>.

Show the path of a subvolume:

$ ceph fs subvolume getpath cephfs ceph-vol ceph-sg
/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0

4.3.2. Mount

For we have ceph.conf and ceph.client.admin.keyring in directory /etc/ceph/, we can:

$ sudo mkdir /mnt/cephfs
$ sudo mount -t ceph admin@.cephfs=/ /mnt/cephfs
$ ls /mnt/cephfs/
volumes

Generally, we do not want to expose admin credentials and want to restrict volume access. We can grant previlige to another client for this:

$ ceph fs subvolume authorize cephfs ceph-vol --group-name ceph-sg xxxx
$ ceph fs subvolume authorized_list cephfs ceph-vol ceph-sg
[
    {
        "xxxx": "rw"
    }
]
$ ceph auth get client.xxxx | sudo tee /etc/ceph/ceph.client.xxxx.keyring
[client.xxxx]
    key = AQCS9OlpUNWZFxAAn6EWmjHRjVRP1FmxhmOISw==
    caps mds = "allow rw path=/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0"
    caps mon = "allow r"
    caps osd = "allow rw pool=cephfs.cephfs.data"

This command create a new user xxxx with rw capabilities to the subvolume. Some meta data is written into the volume for help. Another way to create/update a user is (can apply to any directory):

$ sudo ceph fs authorize cephfs client.xxxx $(ceph fs subvolume getpath cephfs ceph-vol ceph-sg) rw -o /etc/ceph/ceph.client.xxxx.keyring

In this way there are no meta data written, so the client cannot be deauthorized by ceph fs subvolume authorize.

Caution

If the client is existing and the command make no modifications to it the output will be empty and clear the keyring file. So better not use -o option.

Then the new user can be used to mount the subvolume:

$ sudo mount -t ceph xxxx@.cephfs=/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0 /mnt/cephfs

If we do not want to expose the keyring file, we can create a file of the user’s key and specify it in mount options:

$ sudo rm /etc/ceph/ceph.client.xxxx.keyring
$ sudo ceph auth print-key client.xxxx > xxxx.secret
$ sudo mount -t ceph xxxx@.cephfs=/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0 /mnt/cephfs -o secretfile=xxxx.secret

The mount command complains that it cannot find keyring or key files in several places but the operation will succeed.

If the client machine does not have a usable ceph.conf, specify monitor addresses explicitly:

$ sudo mount -t ceph xxxx@.cephfs=/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0 /mnt/cephfs -o mon_addr=10.225.4.52:6789/10.225.4.53:6789/10.225.4.54:6789,secret=AQCS9OlpUNWZFxAAn6EWmjHRjVRP1FmxhmOISw==

This time we use the key directly in the command line.

Tip

The legacy format of this command is:

$ sudo mount -t ceph 10.225.4.52:6789/10.225.4.53:6789/10.225.4.54:6789:/volumes/ceph-sg/ceph-vol/8956fb8b-5a7c-48e7-a5fb-28ad8a0747c0 /mnt/cephfs -o name=xxxx,secret=AQCS9OlpUNWZFxAAn6EWmjHRjVRP1FmxhmOISw==

常用的 -o 选项：

选项	功能
rw	以读写方式挂载
acl	启用 POSIX ACL（更细粒度权限控制，比如给特定用户/组额外权限）
relatime	相对时间更新访问时间（atime），只有在文件被修改后或距离上次更新时间较久时才更新，减少写放大
noatime	完全不更新访问时间（atime），进一步减少元数据写入
_netdev	告诉系统这是网络文件系统，启动/关机时按网络设备处理（例如等网络就绪再挂载，常用于 /etc/fstab）
name=xxxx	CephX 客户端用户名
mds_namespace=cephfs	指定要挂载的 CephFS 命名空间/文件系统（多 CephFS 场景下用来选具体 FS）

Caution

The above command will leak the key via shell history. Do not use it.

Deauthroize the client (delete the client, too):

$ ceph fs subvolume deauthorize cephfs ceph-vol --group-name ceph-sg xxxx

Note

Even the client is not existing, the mounted file system can still be accessed.

4.3.3. Remove

Remove a volume is dangerous, so there is a config to control this:

$ ceph config set mon mon_allow_pool_delete true
$ ceph config get mon mon_allow_pool_delete
true

Now you can delete a volume:

$ ceph fs volume rm cephfs --yes-i-really-mean-it
metadata pool: cephfs.cephfs.meta data pool: ['cephfs.cephfs.data'] removed