Jetstream isn't an official change to the Protocol, it's an optimization I made for my own services that I realized a lot of other devs would appreciate. The major driving force behind it was both the bandwidth savings but also making the Firehose a lot easier to use for devs that aren't familiar with AT Proto and MSTs. Jetstream is a much more approachable way for people to dip their toe into my favorite part of AT Proto: the public event stream.

A non-compressed version is almost certainly cheaper for anything local (ex self-hosting your own services that consume the firehose on the same machine or for testing).

There's not really a good reason to do compression if the stream is just going to be consumed locally. Instead you can skip that step and broadcast over memory to the other local services.

This isn’t compression, they are throwing features of the original stream out.

Centralization on trusted servers is going to happen but if they speak a common protocol, at least they can be swapped out. For JetStream, anyone can run an instance, though it will cost them more.

It’s sort of like the right to fork in Open Source; it doesn’t mean people fork all the time or verify every line of code themselves. There’s still trust involved.

I wonder if some security features could be added back, though?

If you’re going to try data reduction and compression, always try compression first. It may reveal that the 10x reduction you were looking at is only 2x and not worth the trouble.

Reduction first may show the compression is less useful. Verbose, human friendly protocols compressed win out in maintenance tasks, and it’s a marathon not a sprint.

> If everyone is on one server (remains to be seen)

Even internally, BlueSky runs a number of "servers." It's already internally federated. And you can transfer your account to your own server if you want to, though that is very much still beta quality, to be fair.

You're not really "on a server" in the same sense as other things. It's closer to "I create content addressable storage" than "The database on this instance knows my username/password."

We've seen this over and over. If you do things the "right" way devs just don't show up because it's too much work.

Or why can't one verify a msg on it's own isolated from all of the other events on the PDS.

Or in rust haha

The graph is labeled - CPU seconds per second.

> It really doesn't seem realistic for anyone to run any of the infrastructure themselves. You're always going to be reliant on a big player that has the capital to keep everything running smoothly

I run a custom rust-based firehose consumer on the main firehose using a cheap digitalocean droplet and don't cross 15% CPU usage even during the peak 33 Mb/s bandwidth described in this article.

The core team seem to put a lot of intention into being able to host different parts of the network. The most resource intensive of which is the relay which produces the event stream that Jetstream and others consume from. One of their devs did a breakdown of that and showed you could run it at $150 per month which is pricey but not unattainable with grants or crowdfunding https://whtwnd.com/bnewbold.net/entries/Notes%20on%20Running...

Old social media never gave full access to the firehose so there’s a pretty big difference.

If you want large scale social networks, you need to work with a large scale of data. Since federated open queries aren’t feasible, you need big machines.

If you want a smaller scale view of the network, do a crawl of a subset of the users. That’s a perfectly valid usage of atproto, and is how ActivityPub works by nature.

Based on the article OP runs his Jetstream instance with 12 consumers (subsets of the full stream if I understand correctly) on a $5 VPS on OVH

That's only about 2.7 MB/s on average.

If someone wants to run a server, they probably would pay for a VPS with a gigabit connection, which would be able to do 120 MB/s.

You might need to pay for extra bandwidth, but it is probably less than a night out every month.

Sometimes I think all of this could have been avoided if people knew how to use RSS.

I thought this was about BlueSky using NATS!

Why is this being downvoted? Seems like a valid concern to raise if you find two pieces of software somewhat having the same functionality.

My guess was NATS Jetstream [0].

[0] https://docs.nats.io/nats-concepts/jetstream

I thought exactly the same

The 10x surge in traffic was us gaining 3.5M new users over the course of a week (growing the entire userbase by >33%) and all these users have been incredibly active on a daily basis.

Lots of these numbers are public and the impact of the surge can be seen here: https://bskycharts.edavis.dev/static/dynazoom.html?plugin_na...

Note the graphs in that link only show users that take a publicly visible action (i.e. post, like, follow, etc.) and won't show lurkers at all.

24GB/day is very much a "we could host this on one server with a bit of read caching" scale.

Every year or so, you plop in one more 10TB SSD.

> The firehose is all public data going into the network right?

It's the "main subset" of the public data, being "events on the network": for the Bluesky app that's posts, reposts, replies, likes, etc. Most of those records are just metadata (e.g. a repost record references the post being reposted, rather than embedding it). Post / reply records include the post text (limited to 300 graphemes).

In particular, the firehose traffic does _not_ include images or videos; it only includes references to "blob"s. Clients separately fetch those blobs from the PDSs (or from a CDN caching the data).

You're not the only one. I don't get why they couldn't have named it something that wasn't very similar to something already around for several decades, or at least insist on the shortened ATproto name (one word, lower case p). Sure, in practice, no one will actually confuse them, but that could be said for Java and JavaScript.

I wondered something similar when I clicked the link: "who is still using enough AT commands that a compressed representation would matter, and how would you even DO that?" But this is clearly something else.

The article kind of assumes you know what it is in order to be interested in it, but it's the protocol used by Bluesky instead of ActivityPub.

You mean the Hayes AT command set? We didn't call it a protocol back in the say.

The Wikipedia article does a pretty good job at giving an overview for it.

https://en.wikipedia.org/wiki/AT_Protocol

Had same confusion. Wondered why it would need compression...

Was expecting the Hayes modem command language.

Well-configured zstd can save a lot of bandwidth over gzip at this scale without major performance impact, especially with the custom dictionary. Initialising zstd with a custom dictionary also isn't very difficult for the client side.

As for application development, I think web socket APIs are generally exposed much better and used much easier than SSEs. I agree that SSEs are a more appropriate technology to use here, but they're used so little that I don't think the tooling is good. Just about every language has a dedicated websocket client library, but SSEs are usually implemented as a weird side effect of a HTTP connection you need to keep alive manually.

The stored ZSTD objects make sense, as you only need to compress once rather than compress for every stream (as the author details). It also helps store the data collected more efficiently on the server side if that's what you want to do.

I don't have an understanding of SSE in depth, but one of the points the post is arguing for is compress once (using zstd dictionary) and send that to every client.

The dictionary allows for better compression without needing a large amount of data, and sending every client the same compressed binary data saves a lot of CPU time in compression. Streams, usually, require running the compression for each client.

A thing I have admired about the BlueSky development team is that they're always thinking about the future, not just the present. One area in which this is true is the system design: they explicitly considered how they would scale. Sure, at the current cost, this may not be worth it, but as BlueSky continues to grow, work like this will be more and more important.

Also, it's just a good look towards the community. Remember, ideally not all of ATProto's infrastructure is run by BlueSky themselves, but by a diverse set of folks who want to be able to control their own data. These are more likely to be individuals or small groups, not capitalized startups. While the protocol itself is designed to be able to do this, "what happens when BlueSky is huge and running your own infra is too expensive" is a current question that some folks are reasonable skeptical about: it's no good being a federated protocol if nobody can afford to run a node. By doing stuff like this, the BlueSky team is signaling that they're aware of and responsive to these concerns, so don't look at it as trying to save money on bandwidth: look at it as a savvy way to generate some good marketing.

That's my take anyway.

EDIT: scrolling down, you can see this kind of sentiment here: https://news.ycombinator.com/item?id=41637307

The current relay firehose has more than 250 subscribers. It's served more than 8.5Gbps in real-world peak traffic sustained for ~12 hours a day. That being said, Jetstream is a lot more friendly for devs to get started with consuming than the full protocol firehose, and helps grow the ecosystem of cool projects people build on the open network.

Also, this was a fun thing I built mostly in my free time :)

It's a benefit on the receiving side too. And it has ecological benefits. Nothing to sneeze at.