People want object storage as the backend because in practice it means that you can decouple compute and storage entirely, it has no requirement to provision space up front, and robust object storage systems with (de facto) standardized APIs like S3's are widely available for all kinds of deployments and from many providers, in many forms. In other words: it works with what people already have and want.

Essentially every standalone or embedded key-value storage solution treats the KV store and its operation like a database, from what I can tell -- which is sensible because that's what they are! But people use object stores exactly because they don't operate like traditional databases.

Now there are problems with object stores (they are very coarse grained and have high per-object overhead, necessitating some design that can reconcile the round hole and the square peg) -- but this is just the reality of what people are working with. If there is some other key-value store server/implementation you know of, one that performs and offers APIs like an actual database (e.g. multi writer, range scans, atomic writes) but with unlimited storage, no provisioning, and it's got over 10+ different widespread implementations across every major compute and cloud provider -- I'm interested in what that project is.

This is a low-level embedded db that would be used by sql databases/query engines/streaming engines/etc rather than something that's likely to make sense for you to use as an application developer. It sits in a similar space to RocksDB and LevelDB.

You generally can't use object storage directly for this stuff; if you have a high volume of writes, it's incredibly slow (and expensive) to write them individually to s3. Similarly, on the read side you want to be able to cache data on local disk & memory to reduce query latency and cost.

Using an s3 object per key would be too expensive for many use cases.

The website is a bit fancy but the readme seems to pretty straightforwardly explain why you might want this. It seems to me like a nice little (13k loc) project that doesn't fit my needs but might come in handy for someone else?

> I was excited at first and thought this was SQL atop S3 et al.

You can check out Neon.tech who makes an OS Postgres-on-s3 and DuckDB who makes an embedded DB with transaction support that can operate over S3

If you want SQLite backed by S3, maybe something like SQLite in :memory: mode with Litestream would work?

Edit: actually not sure if you can use :memory: mode since Litestream uses the WAL (IIRC), so maybe a ramfs instead

Embedded means it runs in your application process not a standalone server / service.

> MemTables are flushed periodically to object storage as a string-sorted table (SST). The flush interval is configurable.

Looks like it has a pretty similar structure under the hood, but DuckDB would get you more powerful queries.

FYI duckdb directly supports writes (and transactions) so you don’t necessarily even need the separate store step.

This is more targeted at OLTP style workloads with mutable data and potentially multiple writers

Rust is just another programming language that’s quite similar to C++. The main difference is there’s like 4 types for String (some are references and some are owned) and methods for a struct go in a `impl StructName` block after the struct definition instead of inside it.

I don’t really know rust either but I’m currently writing some bindings to expose Rust libraries to NodeJS and not having too much trouble.

For rust -> c++ I googled one time and found this tool which Mozilla seems to use to call Rust from C++ in their web browser, maybe it would “just work”: https://github.com/mozilla/cbindgen?tab=readme-ov-file