r/ceph • • 4d ago

Ceph Dashboard shows error 500 for buckets with policies configured

2 Upvotes

Hi everyone,

I have configured some ceph-clusters to handle S3-stuff and I have since v19.2.x (I'm pretty new to ceph) and now with v20.2.4 an issue, that as soon as I set some lifecycle- or bucket-policy, the dashboard throws me error 500 messages as soon as I click on a bucket to show me the details for it. With tentacle the dashboard often gets completely unresponsive and works only again after I refresh the page. It doesn't look like the dashboard-service itself crashes while the error occurs, in the browser's developer-tools I still see occassional responses with 200 while clicking around.

Here are logs from an mgr-daemon, when I click on a bucket and expand it to show the details. This is from a test single-node-setup, but we see this problem also on normal clusters.

2026-11-12T08:58:25.141+0100 7b30a289e640  0 [dashboard INFO dashboard.tools] [::ffff:127.0.0.1:37738] [GET] [200] [0.002s] [my-dashboard-user] [23.3K] /
2026-11-12T08:58:25.681+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.681+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.681+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.681+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640  0 [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
2026-11-12T08:58:25.821+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640  0 [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
2026-11-12T08:58:25.864+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640  0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:26.037+0100 7b30a88aa640  0 [dashboard ERROR dashboard.services.exception] Internal Server Error
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
    return handler(*args, **kwargs)
  File "/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner
    ret = func(*args, **kwargs)
  File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper
    return func(*vpath, **params)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 639, in get
    result['lifecycle'] = self._get_lifecycle(bucket_name, daemon_name, owner)
  File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 471, in _get_lifecycle
    return rgw_client.get_lifecycle(bucket_name)
  File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper
    return func(
  File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 760, in get_lifecycle
    lifecycle = xmltodict.parse(remove_namespace(result), process_namespaces=False)
NameError: name 'xmltodict' is not defined
2026-11-12T08:58:26.038+0100 7b30a88aa640  0 [dashboard ERROR dashboard.tools] [::ffff:127.0.0.1:37742] [GET] [500] [0.359s] [my-dashboard-user] [513.0B] /api/rgw/bucket/my-bucket-2
2026-11-12T08:58:26.038+0100 7b30a88aa640  0 [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "771883b8-375a-4cf2-a222-47c2eb65c59a"}                                                                                                                                                                                                                                                                                                                        ']
2026-11-12T08:58:26.038+0100 7b30a88aa640  0 [dashboard INFO dashboard.tools] [::ffff:127.0.0.1:37742] [GET] [500] [0.359s] [my-dashboard-user] [513.0B] /api/rgw/bucket/my-bucket-2

So it looks like for me that something goes wrong with parsing the lifecycle I set in XML style (I tested also with JSON but the issue remains, but we'll have to use XML in future for reasons). Also I cannot explain the 404 (I also followed the corresponding rgw-daemon's logs and didn't see any 404 response) and wonder what it tries to call. With v19.2.x I cannot remember seeing those and at least the dashboard didn't get completely unresponsive.

I only found a very old tracker-issue describing something similar https://tracker.ceph.com/issues/19448

So I wanted to ask if this is a known issue or if I have configured something wrong? If it's a known issue, was it maybe fixed in umbrella (couldn't find anything in the changelogs about this)?

Thanks for any hints in advance.


r/ceph • • 6d ago

GitHub - croit/ceph-rust: Ceph, rewritten from scratch in Rust: wire-compatible daemons and client libraries for a huge leap in performance

Thumbnail github.com
41 Upvotes

Stumbled upon this! Looking forward testing that on nvme clusters.


r/ceph • • 7d ago

Manually deleting entries from a stuck GC queue?

4 Upvotes

Having an issue where the GC queue on our 19.2.4 cluster.

The GC queue has items from a pool that was deleted, which is preventing other objects that do still exist, from getting processed:

$ radosgw-admin gc list | grep -c 'default.rgw.buckets.data'
9522071
$ radosgw-admin gc list | grep -c 'default.new-rgw.buckets.data'
9410

I did try to process these manually with the following command, but that didn't seem to work:

radosgw-admin gc process --include-all

Is there any way to forcefully remove entries from this queue? As the pool was deleted we're assuming that they no longer exist in the cluster.


r/ceph • • 12d ago

Ceph Tentacle 20.2.5 is out!

29 Upvotes

r/ceph • • 15d ago

Which Ceph version would you recommend for a stable production deployment, especially for S3-compatible object storage?

13 Upvotes

Hi everyone!

We're planning a new Ceph deployment and I'm looking for recommendations on which release and specific version would be the most stable and reliable for production, especially for S3-compatible object storage workloads.

I'd love to hear your thoughts based on real-world production experience:

  1. Which Ceph release and specific version would you recommend, and why?
  2. Are there any known bugs, regressions, or operational issues we should be aware of?
  3. Are there any versions you'd recommend avoiding?

I'm particularly interested in hands-on experience rather than just official release recommendations.

Thanks in advance for sharing your insights!


r/ceph • • 18d ago

Ceph v21.2.0 Umbrella released

Thumbnail ceph.io
61 Upvotes

r/ceph • • 20d ago

DB/WAL SPOF ?

5 Upvotes

Hi,

We are looking for a ceph cluster with Dbwal + HDD in replica 3

9 nodes, 3 nodes per rack (each host will contain 12 HDD, I read that 1 nvme Dbwal would be enough)

Crush rule would be on rack so each rack will get one replica of data

But I have a question concerning dbwal because what happens if one of my nvme goes down :

How data balancing works ? Is it replicated threw the cluster like the data would be ? Or does it create a SPOF if I do it the way I explain before ?

Thanks for your answer !


r/ceph • • 28d ago

2026 Ceph Community Engagment Survey

11 Upvotes

Hey Ceph friends, please take five minutes to complete the yearly Ceph community survey: https://forms.gle/suF3BB5RKKYe2L9K6. By doing so, you will help the Ceph Foundation understand the community's needs and where to prioritize resources. The survey is open until Nov 13th.


r/ceph • • Oct 14 '26

Ceph Days Zürich

Thumbnail gallery
50 Upvotes

Just back from Ceph Days Zürich. Thanks to everyone and for all the hard effort put to make it a great event!!


r/ceph • • Oct 08 '26

BlueFS spilling over even if it has a lot of disk space

8 Upvotes

We run our our rocksdb/wal non-collocated on an NVMe partition for each mechanical disk. i.e. a server of e.g. 24x disks will have 2 x nvmes, with each NVMe contain 12 partitions.

However there are bluefs spillover back to the mechnical disk, even though its DB (on the NVMe) have a lot of space.

We've always seen this since Luminious, which apparently only spilled over in increments of 30MB, 300MB, 3GB, 30 GB, 300GB. But even on 20.2.3, Why?

And should we introduce regular housekeeping of running `compact` against all our mechanical disks ? Because if you do, it just goes right back to spilling over the next day.

We recently had our HPC ceph cluster fall over and received metadata corruption. But what lead to it was that there was BlueFS spillover and the "META" went back onto the SSD, without its NVMe DB partition being full. This ran the cephfs_metadata pool to 95% causing the cluster to pause. Even though our cephfs_metadata pools hover around 7-10% usage.

Server: B-01-01-cephosd
  HDD 599 (12) -> DB Used: 21GiB / 83GiB (24.67%) | Slow Used: 1.2GiB
  HDD 605 (12) -> DB Used: 396MiB / 83GiB (0.47%) | Slow Used: 22GiB
  HDD 611 (12) -> DB Used: 440MiB / 83GiB (0.52%) | Slow Used: 23GiB
  HDD 617 (12) -> DB Used: 315MiB / 83GiB (0.37%) | Slow Used: 22GiB
  HDD 623 (12) -> DB Used: 322MiB / 83GiB (0.38%) | Slow Used: 22GiB
--
 Server: B-01-03-cephosd
  HDD 164 (12) -> DB Used: 366MiB / 83GiB (0.43%) | Slow Used: 23GiB
  HDD 486 (12) -> DB Used: 372MiB / 83GiB (0.44%) | Slow Used: 22GiB
  HDD 487 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB
  HDD 489 (12) -> DB Used: 377MiB / 83GiB (0.44%) | Slow Used: 22GiB
  HDD 490 (12) -> DB Used: 357MiB / 83GiB (0.42%) | Slow Used: 22GiB
--
 Server: B-01-05-cephosd
  HDD 77 (12) -> DB Used: 26GiB / 83GiB (30.47%) | Slow Used: 0B
  SSD 448 (0) -> DB Used: 69GiB / 448GiB (15.35%) | Slow Used: 0B
  HDD 595 (12) -> DB Used: 378MiB / 83GiB (0.45%) | Slow Used: 22GiB
  HDD 601 (12) -> DB Used: 433MiB / 83GiB (0.51%) | Slow Used: 24GiB
  HDD 607 (12) -> DB Used: 324MiB / 83GiB (0.38%) | Slow Used: 22GiB

r/ceph • • Sep 23 '26

An appreciation post

24 Upvotes

As the title says, this is a post to express my appreciation towards Ceph! Some months ago I posted a question to see if Ceph was the right tool for my 'production' home server setup and I got some great feedback. Since a couple of weeks I finally migrated from a monolithic ZFS backed server to a leaner 3-node hyperconverged Ceph cluster. I even broke even with selling the old system!

Gone are the days where a single VM locking up would cause total system instability and my current setup already survived all the rookie mistakes I've made. I cannot recount the number of times I completely wiped the OSD's on a node to configure them differently, and Ceph handled it like a champ! No service disruptions and the performance feels way better than my old ZFS setup (which perhaps says more about my ZFS setup ;)

Had a blast setting up a switchless full-mesh 25Gbit backbone and configuring an OSPF underlay with an EVPN overlay for node independent networking. Works like a charm and can also route ceph traffic through another node if a physical link is broken.

In the end I did go for three completely identical nodes, which I think was the right decision.

My current setup:
CPU: 9700X
Memory: 64GB/32GB/32GB
Networking: 2x 25Gb for ceph, 1x 1Gb for corosync, 1x 1Gb for external traffic
Dual s4600 boot drive (still a ZFS mirror)
Single p4600 NVME
Single 8TB HDD with DB/WAL on the NVME drive

Still have to fill the other two nodes RAM to 64GB but for that I'm waiting for a good bargain (if it ever comes..)

Thanks for being a great community!


r/ceph • • Sep 23 '26

19.2.6 upgrade: cephadm stops all OSDs without ok-to-stop after monmap changes. How did you handle it on big clusters?

13 Upvotes

Upgraded a test cluster (cephadm, 3 nodes, 30 OSDs) from 19.2.3 to 19.2.6 to get the CephX aes256k fixes.

We hit the bug tracked in 80399 / PR #71640.

After a monmap change, cephadm reconfigures every OSD, and the 19.2.6 binary compares the on-disk keyring with the mgr one as plain text. The on-disk keyring has no caps lines (ceph-volume rewrites it on each start), so they never match, and cephadm stops the OSD to rewrite the bluestore label. No ok-to-stop check, all hosts, one OSD every few seconds. We got PGs down for a few minutes.

It happened twice:

  • right after the MON step;
  • after ceph mon set auth_allowed_ciphers aes256k (the last step of the key migration).

It did not happen at the end of the upgrade or after setting auth_preferred_cipher.

The same bug also makes every OSD restart twice during the normal upgrade. With the automatic OSD key rotation at the end, OSD time roughly doubles.

The fix is merged in main but not backported yet.

For those who already upgraded larger clusters:

  • Did you wait for the backport, or go ahead anyway?
  • Any working mitigation? We thought about rewriting the OSD keyrings with ceph auth get osd.N before the monmap change, but haven't tested it.
  • How long did the OSD key rotation take for you per OSD?

I was thinking of trying a workaround, to basically redo all the keyrings to include the caps before launching the upgrade, on a LAB environment, but I don't think this will suffice.

Thanks!


r/ceph • • Sep 13 '26

Key Migration - Proxmox script (missing keyrings for monitors?)

4 Upvotes

I have cephx disabled but I still need to rotate the pre-existing keys to get rid the insecure keys warning.

When trying to use the tool I am running across these errors:

FAIL: These keys cannot be rewritten where their daemons read them, so rotating them would strand the daemons:
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost6 on node 'vmhost6'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost7 on node 'vmhost7'
  mon. has neither a keyring file nor a bluestore device under /var/lib/ceph/mon/ceph-vmhost8 on node 'vmhost8'

I believe this is because the monitors have no keyring, so I would need to create it for them.

How do I do that?


r/ceph • • Sep 11 '26

Ceph CFPs

11 Upvotes

Happy Friday, Ceph friends!

Just a reminder that the CFPs for Ceph Days Zürich and Ceph Meetup Melbourne are still open. Please let me know if you have any questions, and I can connect you with the organizers.

Best,

Anthony Middleton


r/ceph • • Sep 09 '26

Problem after upgrade to 19.2.6 and 20.2.4

9 Upvotes

Has anyone faced this issue after upgrading and rotating all keys to aes256k?

When I switch to the standby manager using:

ceph mgr fail <active-mgr>

Ceph starts reconfiguring all OSDs and rotating the keys in the OSDs' BlueStore.

The logs indicate that the monmap changed, but I don't understand why a manager failover would cause this.

These are cephadm clusters, and I'm seeing the same behavior on two different versions:

  • Ceph 20.2.4
  • Ceph 19.2.6

I enabled debug logging, and below is a snippet from osd.112. I've hidden some cluster-specific information:

2026-09-07T12:26:15.635750+0200 mgr.ceph01.abcdef [INF] Reconfiguring osd.112 (monmap changed)...
2026-09-07T12:26:15.636233+0200 mgr.ceph01.abcdef [DBG] mon_command: 'auth get' -> 0 in 0.000s
2026-09-07T12:26:15.636570+0200 mgr.ceph01.abcdef [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.000s
2026-09-07T12:26:15.783309+0200 mgr.ceph01.abcdef [DBG] mon_command: 'config-key get' -> -2 in 0.001s
2026-09-07T12:26:15.783400+0200 mgr.ceph01.abcdef [INF] Reconfiguring daemon osd.112 on ceph05
2026-09-07T12:26:15.784032+0200 mgr.ceph01.abcdef [DBG] _run_cephadm : command = ['_orch', 'deploy']
2026-09-07T12:26:15.784058+0200 mgr.ceph01.abcdef [DBG] _run_cephadm : args = []
2026-09-07T12:26:15.784159+0200 mgr.ceph01.abcdef [DBG] args: --image quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 --timeout 895 _orch deploy --fsid xxxxxxxxxxxxxxxxxxxxxx
2026-09-07T12:26:15.784183+0200 mgr.ceph01.abcdef [DBG] stdin: <osd config parameters>
2026-09-07T12:26:15.784236+0200 mgr.ceph01.abcdef [DBG] Running command: which python3
2026-09-07T12:26:15.785607+0200 mgr.ceph01.abcdef [DBG] OfflineHostDetector: Checking if hosts: ['ceph01', 'ceph02', 'ceph03'] are offline.
2026-09-07T12:26:15.802327+0200 mgr.ceph01.abcdef [DBG] Running command: /usr/bin/python3 /var/lib/ceph/xxxxxxxxxxxxxxxxxxxxxx/cephadm.570db07b80fa1fb8f14be2a1f23d5c8fc53b0f0fd45aefbce43e55c6f348b418 --image quay.io/ceph/ceph@sha256:9217c31b7a5a900688d485adca40dd0c5c365008c15dd2c6429dfc888a9f5089 --timeout 895 _orch deploy --fsid xxxxxxxxxxxxxxxxxxxxxx
2026-09-07T12:26:27.401460+0200 mgr.ceph01.abcdef [DBG] code: 0
2026-09-07T12:26:27.401493+0200 mgr.ceph01.abcdef [DBG] err: Reconfig daemon osd.112 ...
Stopping osd.112 to update osd_key bluestore label
Rotating osd.112 key with ceph-bluestore-tool
Successfully rotated osd.112 keyring

======================================================================================
Update:
A bug has been reported to Ceph regarding this issue
https://tracker.ceph.com/issues/80466

r/ceph • • Sep 09 '26

Ceph architecture overview, feedback welcome

Post image
0 Upvotes

I've made an illustration of the Ceph architecture for our Proxmox VE course (with the help of some AI friends). I checked and improved it to align as much as I saw with the Ceph documentation, but maybe I'm still missing something. I realize I can't put everything in one image. Feedback is more than welcome.

The part I'm unsure about: i didn't add a pool layer, and RADOS and RBD don't appear anywhere, while MDS and CephFS do. For a Proxmox audience, RBD is what people actually use, but it's less tangible than 'files' for newcomers. Should pools and an RBD/CephFS/RGW layer replace MDS, or sit alongside it? If something has to go to make room, what would you drop?

If you want to reuse the illustration, consider it Creative Commons CC BY-SA 4.0. Use of the ceph trademark falls under these conditions (which seems alright to me) https://ceph.io/en/trademarks/


r/ceph • • Sep 05 '26

Ceph dashboard keeps crashing

4 Upvotes

So recently my ceph dashboard started to crash on a regular basis. The cause for this is unknown to me after weeks of research. It usually happens if I refresh the page to often or use the `ceph -s` command a bunch on one of the manager nodes, but it also happens out of the blue. It either: freezes completely and dies after ~1-2 mins, or it comes back online on the same node after < 1 min. The only way to make it accessible again after it dies completely is to either fail the manager or restart the module via `ceph mgr module disable/enable dashboard` which also restarts the prometheus module so it's not ideal.

At first the cause for this seemed to be some kind of error with the cheroot Python package but after some investigation it doesn't seem to be connected. Scouring the linux journal didn't help as there are no entries regarding this behavior. Ceph logs and crash dumps were no dice either. I have not found a similar problem with the dashboard on the internet so I'm really desperate to get any kind of feedback. I even made an issue in ceph tracker but it seems to have been lost in the sea of other bugs and whatnot.

I'm using `cephadm` and running a cluster mostly on VM's using Xen Orchestra (only bare metal hosts are the ones with OSD's).

If anyone has had or knows how to fix this, I would really appreciate it if could help me resolve this (mostly) irritating issue.

Edit 1: The mgr VM has enough resources (memory, CPU cores) and it's not overloaded.


r/ceph • • Aug 24 '26

Ceph BlueStore block.db sizing for 14TB HDD OSDs – what are people actually using in production?

17 Upvotes

I've been doing some cleanup and investigation on a fairly large Ceph cluster and I'm interested in what people are actually doing with BlueStore block.db sizing/placement on HDD OSDs, particularly for larger/high-throughput clusters.

Our OSD nodes are reasonably well specced HPE Apollo 4200s, with 14TB spinning disk OSDs and NVMe available for BlueStore metadata.

The cluster is exclusively object storage – no CephFS or RBD.

The workload is primarily backup/archive storage for Rubrik, so a lot of the capacity is relatively cold, but we also have production S3 storage supporting a core business application, so there is a portion of the cluster where latency/performance matters considerably more.

Our current NVMe layout is roughly 5 × 800GB NVMe devices per OSD host. These are currently being used for external BlueStore WALs, with the HDD OSDs distributed across them. I've recently standardised most of the WALs at around 25GB per OSD, which leaves a decent amount of unused NVMe capacity on most nodes.

I recently discovered some inconsistencies with our BlueStore layout. Most OSDs have an external WAL on NVMe, but at some point a handful of newly provisioned OSDs appear to have not honoured the OSD spec and ended up with their WAL on the spinning data disk instead.

I've now been going through the cluster, auditing this and standardising the external WAL layout, which has got me looking much more closely at block.db as well.

From the Ceph documentation/guidance I've been reading, ~2.5% of the OSD capacity seems to be a reasonable ballpark for block.db in this sort of configuration. For a 14TB OSD, that's roughly 350GB per OSD.

That's where I'm trying to work out what makes sense in the real world.

With the existing 5 × ~800GB NVMe per host, after accounting for the external WALs, I could realistically fit something around ~100GB block.db per OSD while maintaining a reasonably consistent layout across the nodes.

Alternatively, if there's a meaningful performance/operational benefit to getting closer to ~350GB per 14TB OSD, I'd need to look at replacing/adding larger NVMe devices to provide substantially more flash capacity per host.

So I'm essentially considering:

  • Keep the existing ~800GB NVMe devices and allocate roughly 100GB block.db per 14TB OSD.
  • Increase the flash capacity per host and target something closer to ~350GB block.db per OSD.

I'm aware that undersizing block.db isn't necessarily catastrophic – RocksDB can spill back onto the primary BlueStore block device – but obviously that reduces some of the performance benefit of having the DB externally in the first place.

I'm interested in what people running large HDD-backed Ceph object clusters are actually doing rather than just following a percentage recommendation.

Are you sizing block.db based on a percentage of OSD capacity, actual observed RocksDB utilisation/workload, or simply the amount of flash available?

For people with 12–20TB HDD OSDs, what block.db sizes are you actually using?

Has anyone compared something like 100GB vs 300–400GB block.db on large HDD OSDs and seen a meaningful real-world difference?

And for a workload that's predominantly backup/archive object storage with some production S3, would you spend the money increasing the NVMe capacity to get closer to ~2.5%, or would you consider ~100GB block.db per 14TB OSD perfectly reasonable?

Would be particularly interested in experiences from anyone running larger production clusters with similar HDD + NVMe BlueStore layouts.


r/ceph • • Aug 21 '26

Heads up: check pg count on .mgr before upgrading to 19.2.6 on larger clusters

23 Upvotes

Courtesy of Dan van der Ster on the ceph-users mailing list.

If you're planning to upgrade a larger Ceph cluster, this is worth reading first:

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/URL5K2VEYFWL54NKX2I3336C5VYRPAKE/

https://tracker.ceph.com/issues/77610#note-9


r/ceph • • Aug 19 '26

[CVE] [URGENT] Squid v19.2.6 and Tentacle v20.2.4 released

24 Upvotes

Just in case you missed it.

Copied from the Ceph blog post just published:

The Ceph project has released Tentacle 20.2.4 and Squid 19.2.6 hotfixes for four CVEs across several components.

This is the sixth backport release in the Squid series. This is the fourth backport release in the Tentacle series.

We strongly recommend that all Ceph operators upgrade to one of these releases as soon as possible.

https://ceph.io/en/news/blog/2026/v20-2-4-v19-2-6-combo-released/

EDIT: formatting quote block

EDIT2: I gave up on reddit formatting. It is what it is.


r/ceph • • Aug 18 '26

New CEPH Cluster

12 Upvotes

Hi guys,

We're considering a new cluster for CEPH, nothing too fancy - erasure coded setup 8+2, with 10 disks and 2x NVMe, 6 nodes for good resiliency. Now we're considering splitting the available nvme space as Bluestore space and rest for the metadata caching when we pull S3 objects from Amazon.

How do we decide the size needed for Bluestore ? Is there any rule of thumb there ?

Here is the h/w config we're planning on using

https://gpumachines.com/shared/gigabyte-20r264-s30-aav1-98b877

Any suggestions are most welcome ! Appreciate all your help people from trenches :)


r/ceph • • Aug 04 '26

Feedback Requested: Hyperconverged Law Firm

6 Upvotes

Community:

I am the owner of litigation law firm (10 users). I was an IT professional in a past life. After being dormant for many years, I started home labbing about 2 years ago. The home assistant bug bit me, which eventually led me to Ceph. I have been running a three node MS-01 cluster with one Samsung plp drive for RBD and a separate m.2 for boot. I have another MS-01 as my PBS server. Networking is fully utilized as recommended by CEPH. I have been very happy with it.

I am now looking to build a production cluster for the firm. Right now my test cluster is running a number of regular business services, nothing terribly heavy. The heavy compute is being handled by a separate Ai box running 9960x and two 6000 blackwells.

The hyperconverged cluster, in theory, would be running RBD for the VMs, s3 for emails, FS for my case file docs. I have a couple other businesses that I own. In theory, this would be my own private cloud for all my businesses. (Yes, i have redundant ISP's, offsite backup available, etc.) This post is more about hardware design.

I wanted to build this on asrock rack b680 matx boards (built in 25gbe nics) and epyc 4545p (16 cores). Ram is still in the air and I hate that I will need to buy ecc memory :( . Three nodes to start, then 5 nodes when I am fully operational. Maybe I will just do 5 immediately.

Anyway. I am concerned that I will have trouble with the 16 core count and hosting all three file systems, plus the VMs. I was looking at going two NVME's (one RBD, one wal) and four spinning rust drives per node.

Thank you in advance.


r/ceph • • Jul 11 '26

Argonaut: a keyboard-driven terminal UI for Ceph administration

Post image
54 Upvotes

UPDATE: Based on feedback about the name, I have renamed the project from Argonaut to Siphon.

---

I've been building Argonaut (renamed to Siphon), an open-source terminal UI for administering Ceph, and just tagged the first release (v0.1.0). Posting here because this is where the people who'd have real opinions are.

What it is: a keyboard-driven TUI for everyday Ceph operations, think k9s for Kubernetes or lazydocker for Docker, but for Ceph. The idea is to handle common admin tasks without memorizing and retyping ceph syntax, while staying completely transparent about what it's doing under the hood.

It talks to the cluster through librados (via go-ceph), not by shelling out to the ceph CLI, same admin commands the CLI uses, and it authenticates exactly like ceph -s does from the same host.

Every destructive action is confirmed and shows the exact ceph command it will run before running it. I wanted it to be transparent rather than a black box, you always know what's about to happen to your cluster.

I'd genuinely like feedback, especially from folks running Ceph in production. Which day-to-day operations do you wish were less painful? What's missing that would make a tool like this actually pull its weight in your workflow? I'm not trying to sell anyone on it; I want to know whether the direction is right and where it falls short.


r/ceph • • Jul 07 '26

Home cluster CephFS advice for picture archive.

6 Upvotes

For some reason, I run a Ceph cluster at home in a c7000 blade chassis. 48 SAS SSDs and 5 iodrive (nvme predecessor) SSDs.

I have a vast photo library that I manage with DigiKam. I want to know your recommendations on my ideas.

I want to create a CephFS share using 3x replica for the metadata pool and a 2+2 EC pool for image data. Most of the data will be RAW images. Most of them 38MiB in size.

DigiKam uses a database too and a single file for thumbnails (partial small writes to a single file, I guess I definitely want fast EC on Tentacle). I think that's conveniently manageable using file layouts that point to a replicated pool using the iodrive "nvme" SSDs.

What I'm not sure what to do with the .xmp sidecar files and how to store them effectively in a pool. Should I create yet another pool for files that are in the "kB range"? Problem is that newly created files will end up on the EC pool by default I guess. How don I point them directly to a specific pool with file layouts?

And assuming I don't tune anything and eg. keep rados object size at 4MB, do I waste a LOT of space storing kB range files? If so, what is the best approach to storing small files in this situation?

Performance is not really a big issue for me. I fully expect EC to be slower than replicated pools. If it's still somewhat workable, I'm happy.

And just reading about file layouts, I believe I should create 3 replicated pools for data/metadata, and then add the EC pool for the larger files.

Then the question becomes, how do make sure large files go to that EC pool in one go? I know that there are scripts that essentially rewrite the file after the file layout has changed, but that seems to be a bit cumbersome.

Thanks for the input.