Relevant blog I found interesting: (Datalevin)
https://yyhh.org/blog/2024/09/competing-for-the-job-with-a-t...
From the perspective of someone not familiar with the topic. And I touches on performance
Datomic (as presented in this video) seems like a great thing. But if it's so great, why are so few people talking about it? Is it because it only exists for Clojure, and few people know or talk about Clojure?
I love Clojure as a language, and I have wanted to use it in production; but there are so few opportunities other than solo building - where you really have to climb a steep wall due to the Clojurist mentality of /Frameworks Bad! Choose all your own libraries good!/. Instead I have found myself happy and productive in the Elixir/BEAM world, of course initially because of Phoenix.
And like many people, I have just been accepting the destructive, non-historic approach to data management with typical RDBMSs. If there were a Datomic for Elixir, I'd probably use it.
Must Datomic (or a data management approach like it) be tightly coupled with Clojure?
I've used Datomic from both Kotin and Groovy (!)
I presented on Datomic at KotlinConf too, with some live coding starting around the 31 minute mark https://www.youtube.com/watch?v=hicQvxdKvnc
You probably need to be on the JVM, as the peer library (i.e. the "good one", where you embed the full query engine and data fetching directly into your business logic) is so far only implemented for the JVM.
I suppose 10+ years of weird license models and a hefty price tag haven't helped. Datomic turned free (but still proprietary) in 2023 though. But why Datomic isn't more widely adopted is a huge mystery to me...
Thanks. I’ll check out the video.
At this point I’m pretty addicted to BEAM and the Elixir ecosystem though. (And LiveView!).
There are plenty of Clojure frameworks. Electric Clojure is the latest but Kit with HugSQL serves all my needs.
Electric looks awesome, but their stern warning of “we are building this for ourselves, and if you get value that’s great” gave me pause.
If it reaches a point of being documented and intended for general public use, then I’ll definitely try a project with it.
That's right, I have a blog post cooking up about why Electric is for experts today. A major factor in this is because, like with Clojure, the users aren't paying us, so the documentation you want cannot yet afford to exist. Another factor is that the demos we've been cooking up internally—that we haven't revealed yet—are so f%cking incredible that everyone is going to be motivated to use it anyway - i am simply setting expectations. For example, we just built the 80% that matters of the “sync engine” value prop in two weeks and 100 LOC. Implementing it in userland requires 1 LOC per query. With differential network traffic for over the wire O(1) remote incremental collection maintenance! for free! And the pattern works with any database!
Statistically (and from experience) I'm probably the weird one here, but I cannot fathom why Datomic isn't more popular.
I get that postgres is a good default in many cases, and I don't expect SQL to die tomorrow. But there are _so many_ apps (most/all backoffice apps I've worked on for example) with 10s or 100s of transactions per second at most, that would love to have the data available directly inside your business logic, and where both business logic and devops would improve by many orders of magnitude by having a full transaction log of all changes done to your data over time.
Is it _just_ because Datomic is different and people don't get it, and that preconceived notions makes you think Datomic is something it isn't?
Here's to the crazy ones!
Professional Clojure dev for some time. Here is what prevents me from using Datomic.
- no Docker image; still distributed as a tarball. Although com.datomic/local exists, it only provides the Client API, so it's mostly suited towards mocking Datomic Cloud.
- Datomic Cloud is designed around AWS; applications wanting to use Datomic Cloud must also design themselves around AWS
- Datomic On-Prem does not scale the same way Datomic Cloud does (e.g. every database has an in-memory index; the transactor and *all peers* need to keep them all in-memory)
- No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query.
In addition to the above four points, I strongly believe the following points prevent others from using Datomic.
- Writing Datomic queries outside of Clojure (e.g. Java) requires writing everything in strings, which feels awful no matter what.
- If you are not using a JVM-based language, then there is no choice but the REST API for interaction. The REST API is orders of magnitude slower than the official Clojure and Java clients.
- Too many tooling exists around the pgwire protocol and various dialects of SQL. Datomic obviously does not fit into either of these categories of tooling.
- Applications like DBeaver do not support Datomic at all. The best you can do is access the datomic_kvs table from a database using JDBC storage engine.
> In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them
Rich's observation was that query optimizers often get things wrong in ways that are hard to predict or control, but he's not fundamentally opposed to their use. That said, building a decent optimizer is a huge undertaking and I think they took the right decision to not attempt to bundle that sort of complexity into the original vision for Datomic otherwise they might never have shipped.
The state-of-the-art commercial engine for query optimization and execution algorithms in this 'triple' space is probably https://relational.ai/
As I see it, Rich's stance is that of an expert in the database that doesn't need to deliver business features using the database. New users are not experts and even experienced users that work for companies have pressure to deliver features. You can get initial popularity by targeting these types of expert users working on more experimental products. However long term growth and popularity requires targeting the other 99.9% of users. I've seen one company adopt Datomic due to this type of user and then a couple years latter rip it out because as it grew it's developers were no longer of this type.
>No query planner whatsoever. In databases like XTDB, the order of :where clauses doesn't matter since the engine is able to optimize them. In Datomic, swapping two :where clauses can transform a 10ms query into a 10s query.
Datomic has query-stats https://docs.datomic.com/reference/query-stats.html
As someone without Clojure experience but eyeing Datomic from the sidelines, thank you for the detailed answer, super interesting!
Would you reach for XTDB instead of Datomic?
For an example of Datomic in containers, https://github.com/carrete/datopro-mic
Part of the reason is that it's been a commercial database until recently. I also think it hasn't helped that it's closely associated with Clojure (for natural reasons), which might make Java developers turn away from it.
It's a shame, though. Datomic is a marvel.
Yeah, the cost is the biggest reason probably. There's probably a lot of in-house developed Datomic clones out there (I know one, that was developed even before Datomic was a thing).
Pricing killed it for my experimental project.
Of course the complexity of having to set a cluster early on doesn’t help for small shops.
Then there’s the complexity of learning datalog - love it, but when I see people already struggling with SQL, I’m not confidant about using it generally !
It's free now - but I suppose 10+ years of being both closed source and paid didn't do wonders for adoption.
What part needs a cluster in your experience? The Datomic deployments I've been involved with have been running on a single server, with a single instance of backing storage etc.
Datalog is interesting indeed... Back when I first started using Datomic in 2012 I had just fundamentally decided to use it, and it took probably a week before the query language "clicked", i.e. to be able to actually compose my own queries and not just copy paste my way to something that works.
Datatomic is not open-source, is it? And I think the licensing model was a bit odd. Those definitely don't help.
I love Clojure and would love to use Datomic. The fact that it is not open source pushes me away. The risk of being locked in is far too great for anything serious. There's no Valkey move if someone captures Datomic.
Absolutely true!
"No Valkey move", I like it! That should be come it's own meme!
Being free but not open source makes no sense at all. It is just our little ego that wants to "keep" what we think of as "ours" to ourselves.
Give all you have to all and progress will go so much faster. Others may learn from the implementation of Datomic and make something better. So your ego is hurt, but humanity wins!!
Give all you have, and it will never be enough! Give anyway!
My best guess is Datomic's coupling to the JVM makes it great for clojure developers (or java, scala etc) but not even considered by python or javascript shops, which make up the vast majority.
If python and java has better interop, I think the story would be a little different maybe? My experience is that java is always a little stubborn as "child code" so libaries like pyspark often involve notoriously difficult set up.
I guess because those of us that are fine with Datomic, also don't have any issues going into IBM, Oracle or Microsoft RDMS portfolio of database products.
“Simple” comes from simplex (Latin for “not braided”), while “complex” means “braided together.” Datomic? Well, it feels like a whole lot of braiding going on. Datomic: complex made hard?
Datomic requires a storage backend (e.g., Cassandra, DynamoDB, MySQL, etc.), transactors, peers, and a separate transaction log... also, it appears the few real users of Datomic are primarily running it in AWS. Trying to run it elsewhere likely introduces more complications and uncertainties. Good luck getting support, too—you might have to wait for Rich Hickey to finish his hammock nap (ba-dum-tss)
While Datomic's immutability, time-travel queries, and Datalog query language are conceptually cool, they seem like niche features. My guess is that in 99% of the cases SQL can do the job just fine. Also, Rich Hickey may not have braids, but those curls come pretty close.
Without deep pockets for high-RAM servers, many tend to steer clear of Java. If Datomic were packaged as a single binary with fewer dependencies (and no JVM requirement), it could attract more users. As it stands, it's a complex setup with limited benefits, primarily suited for specific use cases, I think (financial auditing / historical data analysis?).
---
COLOPHON
(from the Greek word κολοφών [kolophōn], meaning "summit" or "finishing touch")
I don’t have real-world experience with Datomic; I played around with Datascript and ultimately steered away from Datomic after perceiving a low ROI given its complexity.
I hope my fellow fans of Clojure and the venerable 'Simple Made Easy' talk can forgive this little attempt at humor at Rich Hickey's expense! :-p
Datomic Pro doesn't need AWS, I don't see what complications it can cause. It has some helpers for running it in AWS (like a bucket to export to CloudWatch), but none of them are mandatory. And if your use case is small you can run with dev storage (https://www.h2database.com/html/main.html); pretty much the same as you would SQLite.
It might be slightly harder to get started with, but then the simplicity comes in when it is time to solve common business problems. A trivial example would be - we have this nice db, now our clients want reports. You run your reporting as a separate clojure process, it doesn't impact production at all, without needing to setup reporting databases and log-shipping.
> Without deep pockets for high-RAM servers
I just got a dedicated server on Hetzner to test out some things. It's $70/month with 64gb RAM and a CPU that builds a complex C++ thingie in 11 minutes where my laptop spends about an hour :D
Scaling is of course not trivial, but the same set of backoffice apps I've worked with throughout the years that would be a good fit for Datomic, has a working set for the database much smaller than 64GB.
> they seem like niche features
That's the thing, though. Maybe it's because I've use Datomic a bunch. When I'm on projects that use a SQL db, a handful of problems are just fundamentally solved in Datomic, and none of the super knowledgeable devs that know SQL in and out are even aware that they are problems.
Some examples:
- What caused this column of this row to end up with this value?
- Oh no, we were down 3am but when the first person investigated it at 7am (oh those backoffice SLAs...) everything works fine and nobody knows that the db state was at 3am
- When we wrote value X, what other values did we also write at the same time?
- We need to carefully write a transaction that reads some data, writes some data, and reads some more data, and hopefully lock correctly, and hopefully we understood the isolation level (that we probably didn't set anyway) correctly and...
Which makes me think I must be the crazy one...
"complex made hard" would be great. it seems you have missed the point of the talk: complex is always easy, complex is always where the gravitation pulls you towards.
> I get that postgres is a good default in many cases ...
And Datomic can run on top of PostGres anyway right? So it's not as there wasn't a lot of the knowledge that wasn't reusable: not for the queries themselves but everything about managing the database. Or am I wrong here?
Yes - if I understand you correctly :) Datomic writes opaque blobs of index data to existing storage, such as Postgres or dynamodb or a handful of others. But you can't query that data meaningfully directly, it's just a bunch of compressed datomic-specific index data in there, no domain structure or query power etc.
Rich said (in a random podcast, which was not widely circulated - possibly Cognicast) that their mission with Datomic Cloud was for it to be the easiest way to get started with clojure. He even used the word “Easy”! (unless i misremember, this needs to be checked). Anyway, Cloud had consumption based pricing and developers, companies, and even hobbyists are all accustomed to paying for managed cloud infrastructure. To me this seems like a viable strategy, and devs would have forgiven the previous licensing issues -- we just want to get paid to play with cool toys at work, you gotta work the money stuff out with the bosses.
The problem with Cloud was that it didn’t deliver on easy. It was not the Heroku they envisioned, rather it was a scary AWS amalgamation that required deep AWS knowledge to debug their CloudFormation templates to get it to work. This is not even really their fault!, as CloudFormation is a dumpster fire, there are much better cloud orchestration tools now. And while AWS's first few products were rock solid (S3, EC2, DynamoDB), around this time is when AWS turned on the enterprise growth machine and pivoted into box checking and everything cloud began to turn to quicksand underneath their feet.
On top of cloud quicksand, Datomic Cloud had technical scalability challenges which they brute force engineered through, motivating the architectural shift away from the easy synchronous/local entity api to the annoying async/remote client api, and then back again sorta with Ions -- but at the cost of, like, 3 years of product roadmap, during which time Typescript exploded in popularity, the serverless hype cycle began to break and SQL started making a comeback.
Also, the Java API had poor adoption. selling predominantly into Clojure commercial users isn’t a great market so consequently: no VC, small team, no sales/marketing/devrel - at a time when the startup boom was gaining momentum, developer infra was getting funded for the first time so competitors were spending tons of money on beautiful docs, marketing, full time twitter accounts, etc.
I still think there’s elements of an amazing product here, but the business window for a Clojure database has kinda closed - because Clojure itself has lost momentum. Datomic Cloud at its essence is a clojure hosting platform, the business is Clojure itself. I am very bullish on Clojure, I think it never found its killer app, and once found, once a money vector is opened, a bunch of cash (like, $100M†) will make all these problems go away and deliver on Clojure's mission to bring "simple made easy" to mass market application development, which is desperately needed. Maybe Hyperfiddle & Electric?
† Paraphrasing a Materialize press release, “it takes $100M to build a production ready database” -- and for the record, Materialize isn't doing too well, nobody seems to know what it is for and the VCs replaced the CEO earlier this year. Full quote: "Why did we raise $100M? Put quite simply, we believe this is the order of magnitude of investment that it takes to build a production-ready database. Databases are notoriously hard to get right, and we do not intend to cut any corners."
In the end, Cognitect chose to spend their lives doing something hard that matters, and I deeply respect that.
> I still think there’s elements of an amazing product here
Interesting framing. The people behind Clojure and Datomic aren't known for being amazing at scaling and shipping products (i.e. marketing and all that jazz).
This is also not me judging them for it, I haven't built and shipped a Datomic, much less marketed it.
They shipped and scaled Clojure!