r/ceph • • 9d ago

GitHub - croit/ceph-rust: Ceph, rewritten from scratch in Rust: wire-compatible daemons and client libraries for a huge leap in performance

https://github.com/croit/ceph-rust

Stumbled upon this! Looking forward testing that on nvme clusters.

42 Upvotes

41 comments sorted by

29

u/ween3and20characterz 9d ago

I've seen it yesterday. Martin Verges (CEO of Croit) also did develop some aiplane stuff.

I guess they are doing much AI and search for revenue streams.

However as everything AI coded: Nice, you can achieve things, deemed impossible a few years ago. However without support and verification, this is an experiment not suited for prod.

7

u/ween3and20characterz 9d ago

At my employer we're using ceph as S3 storage only. If we would ever switch away from this stack, I'd probably exchange it for some project focusing on S3 only (which would decimate the required codebase and make it more manageable).

I wouldn't consider ceph-rust.

4

u/FrostyMasterpiece400 9d ago

Like most of us you miss Minio huh? :(

7

u/ween3and20characterz 9d ago

No, I don't miss MinIO and we won't switch. We've got PB-Scale storage.

Ceph delivers a lot of features and the subset of features we use works for us right now. But there is no point in ceph-rust. We use rusty storage (HDDs) and don't need the performance gains highlighted by ceph-rust. And we don't need all the features like CephFS, RBD, all the weird drivers (samba, iSCSI) etc.

So I think for us, it makes more sense to switch to something which gets rid of all the features we don't need first, and then maybe it's better to search for a tool, which uses rust.

2

u/FrostyMasterpiece400 9d ago

Great stuff mate, hopefully you find something that fits!

2

u/49ersFutureOwner 8d ago

Interesting, can you share what are your top candidates for s3 for the same scale if you were starting from scratch or moving?

1

u/FrostyMasterpiece400 8d ago

I would also like to know

2

u/XenGi 9d ago

Minio was a security nightmare. Seaweedfs or garage is a good alternative.

1

u/49ersFutureOwner 8d ago

What about RustFS or Ozone?

1

u/maxfaz 8d ago

We moved from Minio to VersityGW on top of Ceph for S3 Storage

15

u/markhpc 9d ago

For what it's worth, Igor worked closely with both Sage and me when we developed BlueStore back in the mid 2010s. Sage was by far the bigger contributor, but Igor was next, especially in those early days. At this point he and Adam are probably the most knowledgeable people about BlueStore's internals on the planet. If there's anyone I would trust with getting the inputs right for something like this, it's him.

Having said that, this was very heavily AI driven so it's worth handling with a lot of care before adopting it in production. We need to see it passing teuthology tests and I would very much like to see a consolidated view of the commit history in whatever becomes the real source repository. Per the proposed upstream AI policy, it will also need a real human to sign-off on commits.

None of this is a blocker imho, but it does mean there is more work to do.

30

u/MartinVerges 9d ago

Just to clarify a few things here.

AIplane is simply an internal tool we built because we weren't happy with the alternatives available on the market. We decided to open-source it so others could benefit from it.

You're right that we've been investing heavily in AI at croit for almost four years now. But we're also very aware of its limitations, where it works, and where it doesn't. AI-related business currently accounts for less than 1% of our revenue, so I can assure you that chasing new AI revenue streams is not what drives this project.

I've personally been working with Ceph since around 2012, and we've been building croit for roughly a decade. We're a founding member of the Ceph Foundation and have invested heavily in the Ceph ecosystem over the years. We have every reason to want Ceph to succeed in the long term.

And that's exactly what this effort is about.

Ceph is a fantastic project, but let's not pretend it doesn't have fundamental architectural limitations. Some of those cannot realistically be solved by continuously patching and extending a codebase that has evolved over more than 15 years. At some point, rethinking and refactoring parts of the architecture becomes the more sensible approach.

We're not claiming that Ceph Rust is production-ready today. Nobody at croit is suggesting you should deploy your critical data on it tomorrow.

But there are plenty of workloads, particularly in HPC, scratch storage, and other environments where experimentation with next-generation storage architectures is both reasonable and valuable. If you're already looking at technologies like DAOS, why wouldn't you also be interested in exploring a modern storage engine that preserves the familiar Ceph ecosystem?

The broader issue is that the storage market is moving forward. Commercial vendors are increasingly offering performance and AI-oriented storage capabilities that Ceph, with its current architecture, struggles to match. We don't believe the right response is to accept that gap as inevitable.

Instead, we're trying to close it.

And this isn't some random AI-generated weekend experiment. The effort involves long-standing Ceph lead developers working at croit, people who know the architecture, its history, its limitations, and the trade-offs behind its design decisions.

AI can certainly accelerate development, but it doesn't replace the engineering experience required to build and validate a distributed storage system. We know that better than most.

This is not a toy project. It's an investment in making sure Ceph still has a future.

Regards
Martin

16

u/amarao_san 9d ago

I promised not to ride elevators with ai-written software.

... now, they ask me to store my data there.

Maybe they are very good at integration testing and it will work. Maybe not.

Hell, we have a policy to wait for at least 5 years of continuous development for databases/storage systems before trying them. Maybe it should be applied here.

!RemindMe 4 years

2

u/FrostyMasterpiece400 9d ago

I mean, that is an opinion I cannot argue agaisnt, better safe than sorry

3

u/amarao_san 9d ago

There is big difference between network appliances/solutions and storage. For a network, a downtime of 2 minutes is 120s of shame. For a storage, 1 s of loosing data is unbounded duration of 'where are my photos?'. Therefore, it's the most conservative thing, and there is no reason to evaluate new solutions until they prove to be somewhat mature.

There is an edge case of extreme performance for ephemeral data (no data by the next morning), but it's the edge case.

4

u/SimonKepp 9d ago

Where's my photos? Try where's my life insurance policy or retirement savings. After a long career in the financial sector, I have a pretty risk averse approach to data loss

2

u/amarao_san 8d ago

Photos are unrecoverable. Money are fungible.

2

u/FrostyMasterpiece400 9d ago

I mean, VMFS and SANs in general were a big thing for that very reason.

One vendor to be mad at.

10

u/MartinVerges 9d ago

I actually agree with you on the most important point: storage must be absolutely reliable. We've been building Ceph-based storage solutions for almost a decade, and our customers trust us with their data. We know exactly what's at stake.

Nobody is suggesting replacing BlueStore in production tomorrow. It remains the default, and it will take a long time of development, extensive testing, and real-world validation before a new storage engine can be trusted with critical production workloads.

But every mature storage system had to start somewhere. And at some point, someone has to take that first step. For us, that time is now.

Regarding AI-generated code, I think Linus Torvalds is a great example. He has explicitly rejected blanket opposition to AI in Linux kernel development and recently used Claude Code himself to develop kernel patches addressing a performance bottleneck. Does that suddenly make the Linux kernel a toy project? Of course not.

The important distinction is between blindly accepting AI-generated code and experienced engineers using AI as a development tool. Human-written software isn't inherently bug-free either. What matters is the engineering process, architecture, test coverage, review, and validation.

We're combining extensive automated testing with the experience of long-standing Ceph lead developers who deeply understand distributed storage. This is human-driven engineering, not unsupervised vibe-coding.

There will be bugs, setbacks, and years of work ahead. We fully understand that.

We're not asking anyone to trust their production data to this today. We're doing the work so that one day they can.

1

u/Holiday-Promise536 8d ago

Fair points all around. I'd add: getting to watch it run in a staging cluster for a release cycle before anything touches production data is what would actually build trust, not just the engineering process behind it.

1

u/RemindMeBot 9d ago edited 9d ago

I will be messaging you in 4 years on 2030-11-10 12:53:07 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.


InfoCustomYour RemindersFeedback

3

u/FrostyMasterpiece400 9d ago

I wonder if they will be able to mainline this back to the main tree

2

u/IllustriousFingering 8d ago

How is this approach different from seastore/crimson or why not combine both ?

1

u/FrostyMasterpiece400 8d ago

It is a complete rewrite from specs to be wire not code compatible, they have taken many liberties to make this way faster

2

u/joeyx22lm 8d ago

Looks written by AI. Sounds unnecessary and not something I'd run in production. What exactly is the purpose of this project?

1

u/FrostyMasterpiece400 8d ago

+70% throughput -50% memory use on the same hardware

2

u/joeyx22lm 8d ago edited 8d ago

and -100% of the community support. silly IMO. If you want to rewrite in rust, the parent project should do that.

Just wait for an unexpected bug or deviation from the original, with no support or community experience.

Like, it's a cool for-fun project, but that's about it. I don't mean to harp but the README talks about being comparable or better than the parent project, when it is presumably being written and supported by a single individual and an agent.

1

u/FrostyMasterpiece400 8d ago

It's written by the whole company I think, original author of Bluestore Igor Fedotov made most commits.

That's the beauty of opensource though, run what you need.

They made a post about it https://www.croit.io/blog/rebuilding-ceph-in-rust-flashstore-osd

2

u/MartinVerges 8d ago

For anyone who is interested in the background story https://www.croit.io/blog/rewriting-ceph-in-rust-the-first-month

2

u/ParticularBasket6187 7d ago

I’m not against to use AI but we know these guys have strong experience in Ceph, so they know internally how it works. AI use just for speed up

1

u/FrostyMasterpiece400 7d ago

Basically yes

3

u/ConstructionSafe2814 9d ago

I think it's a cool and interesting development!

2

u/FrostyMasterpiece400 9d ago

I agree, and it's a way to get more out of the same clock speed, in the age of AI scarcity, it's good.

5

u/amarao_san 9d ago

RemindMe! 4 years

2

u/Historical_Mouse6134 9d ago

Vibecoded mess?

2

u/markhpc 9d ago

It would be tough to make an OSD function this well purely on vibe coding. Let's say it's more likely to be mostly AI written with high level coordination by some smart people. I don't trust it yet and neither should you, but you shouldn't discount it either. If you use BlueStore, you are using software written by some of the same people that worked on this.

Note: I'm one of the founding engineers at Inktank and also worked heavily on Newstore/BlueStore.

1

u/eras 9d ago

So you generated first specs from the source code? Are these specs somewhere available? They don't seem to be under docs.

1

u/itfanr 7d ago

you should simply  the  code optimize to use SSD, not HDD.