ceph-csi-cephfs
Prerequiste
CephFS CSI need a SubvolumeGroup to create volumes for kubernetes. So make one in ceph cluster:
$ ceph fs subvolumegroup create cephfs csi
$ ceph fs authorize cephfs client.csi /volumes/csi rw
[client.csi]
key = AQDe6epp9PNxNhAAxNMCkbqfwUzfhSu9+rsmdg==
caps mds = "allow rw fsname=cephfs path=/volumes/csi"
caps mon = "allow r fsname=cephfs"
caps osd = "allow rw tag cephfs data=cephfs"
A client csi is created for this purpose. But, the driver want to create Subvolumes in this SubvolumeGroup, thus the caps above are not enough. Modify the caps for this client as:
$ ceph auth caps client.csi mon "allow r fsname=cephfs" mgr "allow rw" mds "allow rw fsname=cephfs path=/volumes/csi" osd "allow rw tag cephfs *=cephfs"
Two points:
rwaccess tomgrrwaccess toosdfile systemcephfsnot onlydatapool but alsometapool
Install
Download helm charts:
$ helm repo add ceph-csi https://ceph.github.io/csi-charts/
"ceph-csi" has been added to your repositories
$ helm repo update
⋮
$ helm pull ceph-csi/ceph-csi-cephfs
Write a ceph_csi_cephfs_values.yaml file:
csiConfig:
- clusterID: "990b5070-3964-11f1-8888-476de7d3e05c"
monitors:
- "10.225.4.54:6789"
- "10.225.4.53:6789"
- "10.225.4.52:6789"
cephFS:
subvolumeGroup: "csi"
storageClass:
create: true
clusterID: "990b5070-3964-11f1-8888-476de7d3e05c"
fsName: "cephfs"
# Mount the host /etc/selinux inside pods to support
# selinux-enabled filesystems
selinuxMount: false
secret:
create: true
userID: "csi"
userKey: "AQDe6epp9PNxNhAAxNMCkbqfwUzfhSu9+rsmdg=="
Install:
$ helm install ceph-csi-cephfs ceph-csi-cephfs-3.16.2.tgz -n ceph-csi-cephfs --create-namespace -f ceph_csi_cephfs_values.yaml
NAME: ceph-csi-cephfs
LAST DEPLOYED: Fri Apr 24 10:16:57 2026
NAMESPACE: ceph-csi-cephfs
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Examples on how to configure a storage class and start using the driver are here:
https://github.com/ceph/ceph-csi/tree/v3.16.2/examples/cephfs
Show installed workloads:
$ kubectl get all -n ceph-csi-cephfs
NAME READY STATUS RESTARTS AGE
pod/ceph-csi-cephfs-nodeplugin-s5m52 3/3 Running 0 116s
pod/ceph-csi-cephfs-nodeplugin-tb7dj 3/3 Running 0 116s
pod/ceph-csi-cephfs-nodeplugin-w8vh4 3/3 Running 0 116s
pod/ceph-csi-cephfs-provisioner-849d8678c5-5s78c 6/6 Running 0 116s
pod/ceph-csi-cephfs-provisioner-849d8678c5-8t6pf 6/6 Running 0 116s
pod/ceph-csi-cephfs-provisioner-849d8678c5-vfvlc 6/6 Running 0 116s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ceph-csi-cephfs-nodeplugin-http-metrics ClusterIP 10.105.40.236 <none> 8080/TCP 116s
service/ceph-csi-cephfs-provisioner-http-metrics ClusterIP 10.105.146.166 <none> 8080/TCP 116s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/ceph-csi-cephfs-nodeplugin 3 3 3 3 3 <none> 116s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ceph-csi-cephfs-provisioner 3/3 3 3 116s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ceph-csi-cephfs-provisioner-849d8678c5 3 3 3 116s
Related ConfigMaps:
$ kubectl get cm -n ceph-csi-cephfs
NAME DATA AGE
ceph-config 2 3m3s
ceph-csi-config 1 3m3s
ceph-csi-encryption-kms-config 1 3m3s
kube-root-ca.crt 1 3d15h
The ceph cluster config is in ceph-csi-config.
Created StorageClass:
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-cephfs-sc cephfs.csi.ceph.com Delete Immediate true 5m
Ceph client credentials:
$ kubectl get secret -n ceph-csi-cephfs
NAME TYPE DATA AGE
csi-cephfs-secret Opaque 2 8m33s
sh.helm.release.v1.ceph-csi-cephfs.v1 helm.sh/release.v1 1 8m33s
The userID and userKey in csi-cephfs-secret is only BASE64 coded:
$ kubectl get secret csi-cephfs-secret -n ceph-csi-cephfs -ojsonpath='{.data.userID}' | base64 --decode
csi
Tip
We can restore the information needed to mount a CephFS volume if we can touch the ceph CSI driver in kubernetes.
Allocate PVs
Dynamic
By specify spec.storageClassName in a pvc, the underlying pv can be automatically provisioned. For example, create a pvc and Pod as:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-cephfs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: csi-cephfs-sc
---
apiVersion: v1
kind: Pod
metadata:
name: dynamic-ceph-pv-po
spec:
restartPolicy: OnFailure
containers:
- name: app
image: busybox:1.37.0-glibc
imagePullPolicy: IfNotPresent
command: ["sh", "-c", "trap exit INT TERM; sleep infinity & wait"]
resources:
requests:
cpu: "200m"
memory: "256Mi"
limits:
cpu: "200m"
memory: "256Mi"
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: csi-cephfs-pvc
Show the pvc and created pv, dig out the volume path of the pv:
$ kubectl get pvc csi-cephfs-pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
csi-cephfs-pvc Bound pvc-acacc432-af06-484d-9f32-fca65d7616e1 1Gi RWX csi-cephfs-sc <unset> 32s
$ kubectl get pv pvc-acacc432-af06-484d-9f32-fca65d7616e1
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-acacc432-af06-484d-9f32-fca65d7616e1 1Gi RWX Delete Bound ceph-csi-cephfs/csi-cephfs-pvc csi-cephfs-sc <unset> 63s
$ kubectl get pv pvc-acacc432-af06-484d-9f32-fca65d7616e1 -ojsonpath='{.spec.csi.volumeAttributes.subvolumePath}'
/volumes/csi/csi-vol-7d79a879-0a14-49c9-bfab-0637e5360111/89053c31-1ad3-4f94-8df6-395afcb8b838
Show the create subvolue in ceph:
$ ceph fs subvolume ls cephfs csi
[
{
"name": "csi-vol-7d79a879-0a14-49c9-bfab-0637e5360111"
}
]
$ ceph fs subvolume getpath cephfs csi-vol-7d79a879-0a14-49c9-bfab-0637e5360111 csi
/volumes/csi/csi-vol-7d79a879-0a14-49c9-bfab-0637e5360111/89053c31-1ad3-4f94-8df6-395afcb8b838
If the volumes is mounted on somewhere else, you can see a .meta file in the directory of subvolume:
$ sudo cat .meta
[GLOBAL]
version = 2
type = subvolume
path = /volumes/csi/csi-vol-7d79a879-0a14-49c9-bfab-0637e5360111/89053c31-1ad3-4f94-8df6-395afcb8b838
state = complete
[USER_METADATA]
csi.storage.k8s.io/pvc/name = csi-cephfs-pvc
csi.storage.k8s.io/pvc/namespace = ceph-csi-cephfs
csi.storage.k8s.io/pv/name = pvc-acacc432-af06-484d-9f32-fca65d7616e1
.cephfs.csi.ceph.com/userid/0001-0024-990b5070-3964-11f1-8888-476de7d3e05c-0000000000000003-7d79a879-0a14-49c9-bfab-0637e5360111/las1 = csi
This file cannot be found in the pod, because where the pod used is a subdirectory in this directory.
Dynamically provisioned pv will be deleted after released.
Static
Static pv need a dedicated subvolume in CephFS, create one:
$ ceph fs subvolume create cephfs static_vol csi
Authorize to a new client:
$ ceph fs subvolume authorize cephfs static_vol --group-name csi csi-static
This create a new client csi-static. Get its credentials:
$ ceph auth get client.csi-static
[client.csi-static]
key = AQACJetpyvjeDBAAvHBRzZEaGgE4UKhoJLN1Hg==
caps mds = "allow rw path=/volumes/csi/static_vol/26d36559-7994-47b5-9bde-78ec34486754"
caps mon = "allow r"
caps osd = "allow rw pool=cephfs.cephfs.data"
Now create the pv (also with a Secret to provide credentials for user csi-static):
apiVersion: v1
kind: Secret
metadata:
name: csi-static-secret
namespace: default
stringData:
userID: csi-static
userKey: AQACJetpyvjeDBAAvHBRzZEaGgE4UKhoJLN1Hg==
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: csi-static-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
csi:
driver: cephfs.csi.ceph.com
nodeStageSecretRef:
name: csi-static-secret
namespace: default
volumeAttributes:
# optional file system to be mounted
"fsName": "cephfs"
"clusterID": "990b5070-3964-11f1-8888-476de7d3e05c"
"staticVolume": "true"
"rootPath": /volumes/csi/static_vol/26d36559-7994-47b5-9bde-78ec34486754
# volumeHandle can be anything, need not to be same
# as PV name or volume name. keeping same for brevity
volumeHandle: csi-static-pv
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
The userID and userKey here are plaintext and Kubernetes will base64 encode it and move to data field.
Then the pvc and Pod (show the diff from the dynamic one):
resources:
requests:
storage: 1Gi
- storageClassName: csi-cephfs-sc
+ storageClassName: ""
+ volumeName: csi-static-pv
---
apiVersion: v1
kind: Pod
metadata:
- name: dynamic-ceph-pv-po
+ name: static-ceph-pv-po
spec:
restartPolicy: OnFailure
containers:
Note
We do not need a StorageClass for this, so the empty string is needed here. Because if it is ommited, the default StorageClass will be used if there is.
After delete the Pod, show the status of the pv:
$ kubectl get pv csi-static-pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
csi-static-pv 1Gi RWX Retain Released default/csi-cephfs-pvc <unset> 8m9s
The pv is kept, but cannot be re-mount to another Pod because its status is Released. You must make it Available to use again:
$ kubectl patch pv csi-static-pv --type=json -p='[{"op":"remove","path":"/spec/claimRef"}]'
persistentvolume/csi-static-pv patched
$ kubectl get pv csi-static-pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
csi-static-pv 1Gi RWX Retain Available <unset> 11m