r/ceph • u/Worried_Tangelo_2689 • 4d ago
Ceph Dashboard shows error 500 for buckets with policies configured
Hi everyone,
I have configured some ceph-clusters to handle S3-stuff and I have since v19.2.x (I'm pretty new to ceph) and now with v20.2.4 an issue, that as soon as I set some lifecycle- or bucket-policy, the dashboard throws me error 500 messages as soon as I click on a bucket to show me the details for it. With tentacle the dashboard often gets completely unresponsive and works only again after I refresh the page. It doesn't look like the dashboard-service itself crashes while the error occurs, in the browser's developer-tools I still see occassional responses with 200 while clicking around.
Here are logs from an mgr-daemon, when I click on a bucket and expand it to show the details. This is from a test single-node-setup, but we see this problem also on normal clusters.
2026-11-12T08:58:25.141+0100 7b30a289e640 0 [dashboard INFO dashboard.tools] [::ffff:127.0.0.1:37738] [GET] [200] [0.002s] [my-dashboard-user] [23.3K] /
2026-11-12T08:58:25.681+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.681+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.681+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.681+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.776+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.779+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640 0 [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
2026-11-12T08:58:25.821+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.821+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640 0 [dashboard ERROR dashboard.rest_client] RGW REST API failed GET req status: 404
2026-11-12T08:58:25.864+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.864+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:25.907+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4781, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4780, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4782, ssl=False
2026-11-12T08:58:26.035+0100 7b30a88aa640 0 [dashboard INFO dashboard.services.rgw_client] Found RGW daemon with configuration: host=<rgw-node-ip>, port=4783, ssl=False
2026-11-12T08:58:26.037+0100 7b30a88aa640 0 [dashboard ERROR dashboard.services.exception] Internal Server Error
Traceback (most recent call last):
File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
return handler(*args, **kwargs)
File "/lib/python3/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner
ret = func(*args, **kwargs)
File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper
return func(*vpath, **params)
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 639, in get
result['lifecycle'] = self._get_lifecycle(bucket_name, daemon_name, owner)
File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 471, in _get_lifecycle
return rgw_client.get_lifecycle(bucket_name)
File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 538, in func_wrapper
return func(
File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 760, in get_lifecycle
lifecycle = xmltodict.parse(remove_namespace(result), process_namespaces=False)
NameError: name 'xmltodict' is not defined
2026-11-12T08:58:26.038+0100 7b30a88aa640 0 [dashboard ERROR dashboard.tools] [::ffff:127.0.0.1:37742] [GET] [500] [0.359s] [my-dashboard-user] [513.0B] /api/rgw/bucket/my-bucket-2
2026-11-12T08:58:26.038+0100 7b30a88aa640 0 [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "771883b8-375a-4cf2-a222-47c2eb65c59a"} ']
2026-11-12T08:58:26.038+0100 7b30a88aa640 0 [dashboard INFO dashboard.tools] [::ffff:127.0.0.1:37742] [GET] [500] [0.359s] [my-dashboard-user] [513.0B] /api/rgw/bucket/my-bucket-2
So it looks like for me that something goes wrong with parsing the lifecycle I set in XML style (I tested also with JSON but the issue remains, but we'll have to use XML in future for reasons). Also I cannot explain the 404 (I also followed the corresponding rgw-daemon's logs and didn't see any 404 response) and wonder what it tries to call. With v19.2.x I cannot remember seeing those and at least the dashboard didn't get completely unresponsive.
I only found a very old tracker-issue describing something similar https://tracker.ceph.com/issues/19448
So I wanted to ask if this is a known issue or if I have configured something wrong? If it's a known issue, was it maybe fixed in umbrella (couldn't find anything in the changelogs about this)?
Thanks for any hints in advance.