Connecting Ceph clients. Block devices – RBD

RBD, aka RADOS Block Device, as you might guess from the name, allows you to allocate space from Ceph and present it to clients as block devices (disks).

RBD can often be found in conjunction with virtualization, in Kubernetes, where disks are connected to containers as PV, and also inside the client OS.

In this case, we will look at how to connect block devices with Ceph to a regular Linux host.

Let me remind you that I previously described the procedure for installing Ceph 18 from scratch in this article. In this case, I use the same platform, as well as a client based on Rocky Linux 9.3.

To briefly describe the process, the algorithm is as follows:

  1. On the Ceph side, a pool is created in which it is planned to place RBD images;
  2. An account is created that is given the permissions to create RBD in the previously created pool;
  3. On the client side, the necessary set of packages and libraries is installed;
  4. On the client side, disks of the required size are created and connected to the OS.

Let’s begin and move on to the node with which we have control of the Ceph cluster. In my case, it’s ceph-mon-01.

First, look at the list of existing pools:

[root@ceph-mon-01 ~]# ceph osd lspools
1 .mgr

Let’s create a new pool in which RBDs will be located in the future:

[root@ceph-mon-01 ~]# ceph osd pool create rbd_images_pool
pool 'rbd_images_pool' created

[root@ceph-mon-01 ~]# ceph osd pool application enable rbd_images_pool rbd
enabled application 'rbd' on pool 'rbd_images_pool'

Please note that without specifying a label, using the second command, you can get a warning message in the cluster status:

1 pool(s) do not have an application enabled

Although this is not related to the main topic of the article, let’s look at some of the pool properties that apply to it by default:

[root@ceph-mon-01 ~]# ceph osd pool get rbd_images_pool size
size: 3

[root@ceph-mon-01 ~]# ceph osd pool get rbd_images_pool min_size
min_size: 2

[root@ceph-mon-01 ~]# ceph osd pool get rbd_images_pool pg_autoscale_mode
pg_autoscale_mode: on

For created pools, Replication Factor 3 is automatically selected, i.e. Two additional copies will be created for each object.

The pg_autoscale parameter is also included, which eliminates the need to calculate PG (Placement Groups) and PGP and set them manually.

Let’s look at the list of pools:

[root@ceph-mon-01 ~]# ceph osd lspools
1 .mgr
2 rbd_images_pool

Now let’s create an account and give it full permissions to the previously created pool:

[root@ceph-mon-01 ~]# ceph auth get-or-create client.vmik mon 'profile rbd' osd 'profile rbd pool=rbd_images_pool' -o /etc/ceph/vmik.keyring
[root@ceph-mon-01 ~]# cat /etc/ceph/vmik.keyring
[client.vmik]
        key = AQBXvMVlEtK9MhAAfOJBCOBW94K+84RwPAMWDQ==

You can view the current permissions in the cluster using ceph auth ls: 

[root@ceph-mon-01 ~]# ceph auth ls
…
client.vmik
        key: AQBXvMVlEtK9MhAAfOJBCOBW94K+84RwPAMWDQ==
        caps: [mon] profile rbd
        caps: [osd] profile rbd pool=rbd_images_pool
…

At this point, we have completed the configuration on the Ceph side. Let’s move on to the client.

On the client side, we need to install the ceph-common package. You can read how to connect Ceph repositories for different distributions here.

I will create the repository file myself:

[root@ceph-client ~]# vi /etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-reef/el9/$basearch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-reef/el9/noarch
enabled=1
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://download.ceph.com/rpm-reef/el9/SRPMS
enabled=0
priority=2
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

Please note that rpm-reef is the latest Ceph release at the time of writing – it may differ in the future. El9 is a common Red Hat family of similar version 9 systems.

Now we should have new repositories:

[root@ceph-client ~]# dnf repolist
repo id                                                                 repo name
appstream                                                               Rocky Linux 9 - AppStream
baseos                                                                  Rocky Linux 9 - BaseOS
ceph                                                                    Ceph packages for x86_64
ceph-noarch                                                             Ceph noarch packages
extras                                                                  Rocky Linux 9 - Extras

We will also connect the Epel repository, which contains dependencies for the ceph-common package:

[root@ceph-client ~]# dnf install epel-release

Install the ceph-common package:

[root@ceph-client ~]# dnf install ceph-common

Now let’s briefly return to the control node. Сopy the previously created keyring file, as well as the Ceph configuration file to /etc/ceph of our client:

[root@ceph-mon-01 ~]# scp /etc/ceph/vmik.keyring /etc/ceph/ceph.conf root@ceph-client.vmik.lab:/etc/ceph

By the way, the ceph.conf file in this case is in a minimal configuration. It contains only the fsid of the cluster, as well as addresses for accessing ceph-monitor via v1 and v2.

Now that all the necessary packages have been installed, and the access key and Ceph configuration file have been copied, you can create the first image:

[root@ceph-client ~]# rbd create --size 2048 rbd_images_pool/test_image_1 --id vmik --keyring=/etc/ceph/vmik.keyring

With this command, we created an image test_image_1 in the rbd_images_pool with a size of 2048 megabytes. The credentials for accessing the pool were also specified here.

Let’s check the list of images in the pool:

[root@ceph-client ~]# rbd ls rbd_images_pool --id vmik --keyring=/etc/ceph/vmik.keyring
test_image_1

Once created, we can connect the block device to the server:

[root@ceph-client ~]# rbd device map rbd_images_pool/test_image_1 --id=vmik --keyring=/etc/ceph/vmik.keyring
/dev/rbd0

The device is identified in the system as rbd0.

You can use the rbd showmapped command to get a list of all connected devices:

[root@ceph-client ~]# rbd showmapped
id  pool             namespace  image         snap  device
0   rbd_images_pool             test_image_1  -     /dev/rbd0

All we need to do now is to create a file system and mount the device:

[root@ceph-client ~]# mkfs.xfs /dev/rbd/rbd_images_pool/test_image_1

Please note that aliases are created for devices with the name of the pool and image.

[root@ceph-client ~]# mkdir /ceph_rbd
[root@ceph-client ~]# mount -t xfs /dev/rbd/rbd_images_pool/test_image_1 /ceph_rbd/
[root@ceph-client ~]# df -h | grep rbd
/dev/rbd0                 2.0G   47M  2.0G   3% /ceph_rbd

So we created a file system on top of the block device and then mounted it.

However, after the reboot, you will have to reconnect and mount the device, which is, of course, inconvenient.

Let’s automate this process, but first unmount the device:

[root@ceph-client ~]# umount /ceph_rbd
[root@ceph-client ~]# rbd unmap /dev/rbd0
[root@ceph-client ~]# rbd showmapped

To automatically connect RBD devices, you can use the rbdmap service, which uses the /etc/ceph/rbdmap file and connects all devices specified in this file.

Let’s edit the file:

[root@ceph-client ~]# vi /etc/ceph/rbdmap
# RbdDevice             Parameters
#poolname/imagename     id=client,keyring=/etc/ceph/ceph.client.keyring
rbd_images_pool/test_image_1    id=vmik,keyring=/etc/ceph/vmik.keyring

The format is quite simple. The full path to the image (the pool and its name) is indicated, as well as a keyring file that has permissions to the pool.

Let’s add the rbdmap service to the startup and launch it immediately:

[root@ceph-client ~]# systemctl enable --now rbdmap
Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /usr/lib/systemd/system/rbdmap.service.

[root@ceph-client ~]# rbd showmapped
id  pool             namespace  image         snap  device
0   rbd_images_pool             test_image_1  -     /dev/rbd0

After starting the service, all images from the rbdmap file were mounted automatically. All that remains is to correct fstab for automatic mounting after rebooting the OS:

[root@ceph-client ~]# vi /etc/fstab
/dev/rbd/rbd_images_pool/test_image_1 /ceph_rbd xfs _netdev 0 0

And reboot to check:

[root@ceph-client ~]# reboot

After a reboot, everything is mounted correctly as expected:

[root@ceph-client ~]# df -h | grep rbd
/dev/rbd0                 2.0G   47M  2.0G   3% /ceph_rbd

This is all for connecting RBD to the client, but a curious reader may ask – given that the client can create RBD devices, can it take up the entire Ceph?

That’s right, so you should first set a quota for the pool. This can be done from the control node:

[root@ceph-mon-01 ~]# ceph osd pool set-quota rbd_images_pool max_bytes 4294967296
set-quota max_bytes = 4294967296 for pool rbd_images_pool

By the way, you don’t have to convert it to bytes but specify it in the max_bytes 4GB format.

From the client side we can view quotas:

[root@ceph-client ~]# ceph osd pool get-quota rbd_images_pool --id vmik --keyring=/etc/ceph/vmik.keyring
quotas for pool 'rbd_images_pool':
  max objects: N/A
  max bytes  : 4 GiB  (current num bytes: 6832325 bytes)

But can’t change:

[root@ceph-client ~]# ceph osd pool set-quota rbd_images_pool max_bytes 42949672960 --id vmik --keyring=/etc/ceph/vmik.keyring
Error EACCES: access denied

It is important to remember that since RBD devices are created in a “thin” format, their specified size can be larger than the quota set for the pool, but this should not be done. This won’t end well:

[root@ceph-client ~]# dmesg
[  776.731856] libceph: FULL or reached pool quota
[  776.731944] libceph: FULL or reached pool quota
[  777.152961] libceph: FULL or reached pool quota
[  777.154086] libceph: FULL or reached pool quota
[  792.835004] libceph: FULL or reached pool quota

Always check the image sizes and quotas set for the pool!

How can I expand the RBD device if necessary? – Quite simple:

[root@ceph-client ~]# rbd resize rbd_images_pool/test_image_1 --size 3G --id vmik --keyring=/etc/ceph/vmik.keyring
Resizing image: 100% complete...done.

[root@ceph-client ~]# xfs_growfs /ceph_rbd/

[root@ceph-client ~]# df -h | grep rbd
/dev/rbd0            3.0G   54M  2.9G   2% /ceph_rbd

Removing it is also not difficult:

[root@ceph-client ~]# umount /ceph_rbd
[root@ceph-client ~]# rbd unmap /dev/rbd0
[root@ceph-client ~]# rbd rm rbd_images_pool/test_image_1 --id vmik --keyring=/etc/ceph/vmik.keyring
Removing image: 100% complete...done.

[root@ceph-client ~]# rbd ls rbd_images_pool --id vmik --keyring=/etc/ceph/vmik.keyring
[root@ceph-client ~]#

That’s all for basic work with RBD. We will look at other connection methods in the future.

Loading

Leave a Reply

Your email address will not be published. Required fields are marked *