This blog post seems to blame GC heavily, but if you look back at their earlier blog post [0], it seems to be more shortcomings in either how they're using Cassandra or how Cassandra handles heavy deletes, or some combination:
"It was at that moment that it became obvious they deleted millions of messages using our API, leaving only 1 message in the channel. If you have been paying attention you might remember how Cassandra handles deletes using tombstones (mentioned in Eventual Consistency). When a user loaded this channel, even though there was only 1 message, Cassandra had to effectively scan millions of message tombstones (generating garbage faster than the JVM could collect it)."
And although the blog post talks about GC tuning, there's mention here [1] that they didn't do much tuning and were actually running on an old version of Cassandra (and presumably JVM) - having just switched over from CMS (!).
0) https://discord.com/blog/how-discord-stores-billions-of-messages
1) https://news.ycombinator.com/item?id=33136453But then it’s still nice that they’re using ScyllaDB and now it’s not a concern at all right?
Even if they were using their original solution wrong, I think the solution that cannot use wrong is superior.
Needs (2023)
That services layer reminds be of a big, fancy, distributed Varnish Cache... they don't mention caching and they chose the word coalesce so I assume it doesn't do much actual caching. But made me think of Varnish's "grace mode" and it's use to prevent the thundering herd problem (which is where I first heard of 'request coalescing') https://varnish-cache.org/docs/6.1/users-guide/vcl-grace.htm...
Also love to see consistent hashing come up again and again. It's a great piece of duct tape that has proven useful in many similar situations. If you know where something should be then you know where everything is gonna come look for it!
Coalescing and “origin shielding” tend to be more common terms for that - I’ve never heard of “grace” until today :)
Very well-written article. I'm happy for them that part of the solution was switching from Cassandra to drop-in replacement Scylla, rather than having to deal with something entirely different.
Anyone else reading this and being quite happy that they're not working at this scale?
I don't mind scale. I mind the bureaucracy and promotion-driven-development that comes with working in a bloated engineering org.
+100
Many companies have products that operate at “scale”. They manage to do so with pretty boring techniques (sharding, autoscaling) and technologies (postgres, cloud storage).
Because of the insane blog driven tech culture, many of these teams get questioned by clueless leadership (who read these blogs) and ask why the company isn’t using cassandra / some other hot technology. And it always causes much consternation and wastage.
But that's where the really fun and complicated problems are. The ones that really make you stop and think, and not just think, but be creative.
95% of the work is still the same "treading in well trod paths", same old same old tech work, but that 5% is really something.
I am happy that I dont have to deal with this.
I am sad that my business aren't as big as this scale.
Honestly, 77 nodes doesn’t sound like a terrific scale? The more I scale things up, the more I realize that the tone of the problems doesn’t really change. You just get more layers to your data structures.
Any time I read anything about any web-adjacent technology I'm incredibly thankful that I don't work anywhere near that industry.
Embedded can be complex, but web stuff is just a Lovecraftian nightmare in comparison
I have stared into the abyss and seen the eyes of cthulu. I am much happier writing embedded drivers than I was trying to make sense of why previous devs thought it was a good idea to move bounded tunable server side api calls to the client, allowing it to effectively write arbitrary sql calls across multiple databases.
Fortunately the web is starting (very slowly) to return to sanity, pushing back towards the simpler server-rendered pattern with Javascript being relegated to specific use cases.
I really like the client rendered UI part. It’s a lot more efficient than sending the whole page again every time.
But you can cache the whole server side page and the cost is once. Whereas if you have the client side do the render then every client wears the cost.
You usually can't because of users who are signed in needing slightly different pages etc.
Iteration speed is significantly fast on the client. Perf is an afterthought — for better or worse
spoken like someone who doesnt deploy clients at discord scale?
the 200 backend nodes surely update significantly faster than the hundreds of millions of clients.
Sounds like a fun time lol
My takeaway from this is maybe somewhat different from what the authors intended:
> The last one? Our friend, cassandra-messages. [...] To start with, it’s a big cluster. With trillions of messages and nearly 200 nodes, any migration was going to be an involved effort.
To me, that's a surprisingly small amount of nodes for message storage, given the size of discord. I had honestly expected a much more intricate architecture, engineered towards quick scalability, involving a lot more moving parts. I'm sure the complexity is higher than stated in the article, but it makes me wonder, given that I've been partially responsible for more than 200 physical nodes that did less, how much of modern cloud architecture is over engineered.
They are talking about 177 database nodes, which is not an indicator of architecture complexity. I assume they have dozens/hundreds of services consisting of multiple highly available nodes each across various geographies.
Having seen a much smaller set of Cassandra nodes used to store billions (rather than trillions) of records, I can say that Cassandra was definitely a total PITA for on-call, and a cause of several major outages.
> ...how much of modern cloud architecture is over engineered.
I would wager a good majority of it is. The Stack Overflow architecture[0] sticks out to me in this regard as an example on the other end of the spectrum.
Also bear in mind that they're now doing the same with just 72 nodes.
> The ScyllaDB team prioritized improvements and implemented performant reverse queries, removing the last database blocker in our migration plan.
I wonder how much they paid ScyllaDB to do this before even using ScyllaDB.
The article says they were using ScyllaDB for everything except the message store two years before they did the migration for messages.
Fun article. Also fun to think about how many people have decided to document their crimes in these Cassandra nodes.
My love of embedded stuff is growing. I'm self teaching C and assembly....to get better at low level programming and interactions with hardware but it all seems much simpler than the big data systems. Granted I'm sure it call be broken down into steps and issues to solve like any programming issue but I'm happy focusing on low level stuff for now.
Is there a fundamental reason you wouldn't use postgres for something like this? Scale certainly wouldn't be it.
I'd guess that Discord's storage systems lean towards processing a lot more writes than reads. Postgres and other databases that use B-tree indexing are ideally suited for read heavy workloads. LSM based databases like Cassandra/Scylla are designed for write intensive workloads and have very good horizontal scaling properties built into the system.
I’m not sure if Postgres would have enough horizontal scaling to accommodate the insane volume of reads and writes. I would be super interested to be proven wrong though… anyone know of a cluster being run at that scale?
> Scale certainly wouldn't be it.
vanilla postgres can't scale to such size, you need some sharding solution on top, which likely will be much harder to maintain than ScyllaDB..
Did they go with ScyllaDB just because it was compatible with Cassandra? Would it make sense to use a totally different solution altogether if they didn't start with that.
Yes, we wanted to migrate all our data stores away from Cassandra due to stability and performance issues. Moving to something that didn't have those issues (or at least had a different set of less severe issues) while also not having to rewrite a bunch of code was a positive.
I see more people mixing up past and present tense randomly, as in this post. It’s confusing to read. Is the concept of tense starting to disappear entirely in US English, I wonder?
Until they don't, or they can't, and they need to start deleting.
(Not trying to undermine the engineering efforts, or the welcoming engineering blog posts though! I really think all these is needed)
Storing is one thing. Performing data mining on them is another
Also, people need to keep in mind that those trillions of messages are archived nowhere. Thanks to the walled gardens we're obsessed with building, far-future anthropologists will know more about Pompeii and Machu Picchu than San Francisco.
I don't think every single individual message ever needs to be archived. Every text, every email, every post-it, every poke, every emoji, every reaction GIF...
Definitely not everything, but it's still wild to me that so many products and services have all their troubleshooting and customer support in a discord server.
Sadly, I can understand why Discord doesn't have a lot of incentive to do this. Maybe the community should popularize an open-source free/low-costing bot and hosting solution for exported chat? (I couldn't find one in a few minutes of searching).
Pretty fun read, even tho I'll never work at such scale lol
Answering directly the title:
With eagerness and a salivating mouth, chomping that sweet, sweet personal user data.
I think it’s annoying they interview engineers like they are Google and reading the blog they made it up and learned some basic “pitfalls” as they went along