CephFS: a beginner’s guide

Ceph logo

Ceph is the open source Software Defined Storage (SDS) king. By default Ceph stores objects, but did you know that you can use it to store files in a filesystem too? Or even replace Hadoop’s HDFS? Let’s take a look at CephFS.

I don’t have a Ceph cluster

Any sufficiently advanced technology is indistinguishable from magic.Arthur C. Clarke

Ceph is pretty complicated as it is, I highly suggest you to follow this guide with a Ceph test cluster beneath your keyboard.

Ceph pools and CephFS

CephFS is a way to store files within a POSIX-compliant filesystem. CephFS lives on top of a RADOS cluster and can be used to support legacy applications. Another common use for CephFS is to replace Hadoop’s HDFS.

To get started you will need a Ceph Metadata Server (Ceph MDS). This server is responsible for maintaining metadata for CephFS files and enable common operations (ls, find etc.) without weighing down the whole cluster. There should be at least one to get started with CephFS, starting with Ceph Luminous there can be more than one Ceph MDS.

Each CephFS instance will need two pools dedicated to it, much like RBD. You will need two pools instead of one because one will keep the data and the other will keep metadata. Know that if you used ceph-ansible and installed an MDS server you may already have an instance of CephFS running.

Now that you know what CephFS is and what it needs to work: let’s get started. You will need to have a Ceph keyring on the host you’re using. The fastest way to get started is to ssh into a monitor node.

I take NO responsibility of what you do with your machine; use this tutorial as a guide and remember you can possibly cause data loss if you touch things carelessly.

Initializing CephFS

The first thing in order to create a CephFS is to create the two pools:

$ ceph osd pool create cephfs_data <pg_num>
$ ceph osd pool create cephfs_metadata <pg_num>

Once you have them you can initialize the filesystem:

$ ceph fs new cephfs cephfs_metadata cephfs_data

CephFS status

You can verify your CephFS instance by doing:

$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

To get more details you can use:

$ ceph fs status <cephfs>
cephfs - 1 clients
| Rank | State  |    MDS    |    Activity   |  dns  |  inos |
|  0   | active |    mon1   | Reqs:    0 /s |   10  |   12  |
|       Pool      |   type   |  used | avail |
| cephfs_metadata | metadata | 2643  | 93.9G |
|   cephfs_data   |   data   |    0  | 93.9G |

| Standby MDS |
MDS version: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)

Mounting CephFS (Kernel driver)

The kernel driver is a convenient way to mount CephFS. In order to use it you must have installed it, refer to your distribution for more information. Once you have it installed simply do:

$ mount -t ceph -o name=<name>,secret=<secret> <mon>:/ <target>

The name parameter is a Ceph User. mon is the monitor‘s hostname or IP address. The <target> is the local directory where you want to mount the filesystem to.

The secret parameter depends on the user, and it is quite insecure (it will be stored in your shell’s history and/or fstab). To avoid this you can simply use the secretfile parameter and specify a file containing the secret. The secret is not the keyring. In order to extract the secret from the keyring you can do:

$ ceph-authtool -p <keyring>

Although you can use the admin user and its secret, it is highly suggested to create a username/secret dedicated for each clientand to assign it the least privilege.

Mounting permanently (Kernel driver)

Using fstab, directly from the official docs:

{ipaddress}:{port}:/ {mount}/{mountpoint} {filesystem-name}     [name=username,secret=secretkey|secretfile=/path/to/secretfile],[{mount.options}]

Mounting CephFS (FUSE)

The FUSE is another way to mount a CephFS. Where you can’t use the kernel driver you can use this method. You will need to have the ceph.conf and ceph.keyring files containing the cluster configuration and required secrets in /etc/ceph on the client node. Once you have them you can do:

$ sudo ceph-fuse -m <monitor>:<port> <target>

The monitor is monitor’s hostname or IP address, the default port is 6789.

Mounting permanently (FUSE)

Using fstab, directly from the official docs:

none    {mountpoint}  fuse.ceph ceph.id={user-ID}[,ceph.conf={path/to/conf.conf}],_netdev,defaults  0 0
Image courtesy of mark | marksei

You may also like...

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.