"Cloud 3-2-1" to the rescue...

I take the typical formulation (e.g., [1]), and translate it into:

- Keep 3 copies of your data: production + 2.

- Keep 2 snapshots, stored separately.

- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.

It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.

[1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested.

However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.

[0] https://www.tigrisdata.com

[1] https://www.tigrisdata.com/docs/migration/#starting-the-migr...

Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery.

From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.

I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that.

another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.

S3 durability is pretty much the only thing I consider worth using EC2 for.

S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...

You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.

It just caught those who rely on one provider, which is never a good practice.

rclone to somewhere else at least.

And 3-2-1 rule for backups if you're serious.

One of my VPS was lost in that fire, but since my backups where somewhere else, I didn't lose much.

The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.

I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.

Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.

For comparison, here are the results for Scaleway Object Storage (https://www.scaleway.com/en/object-storage/), competing offer from another european S3 provider:

  +--------------------+-------------------+------------------+
  |        ITEM        |       VALUE       |       COST       |
  +--------------------+-------------------+------------------+
  |     upload objects |       16.72 MiB/s | 956.98 ms/object |
  |   download objects |      109.01 MiB/s | 146.78 ms/object |
  |  put small objects |    15.3 objects/s | 261.99 ms/object |
  |  get small objects |    88.8 objects/s |  45.05 ms/object |
  |       list objects | 6115.93 objects/s |      65.40 ms/op |
  |       head objects |   164.3 objects/s |  24.34 ms/object |
  |     delete objects |   157.9 objects/s |  25.33 ms/object |
  | change permissions |       not support |      not support |
  | change owner/group |       not support |      not support |
  |       update mtime |       not support |      not support |
  +--------------------+-------------------+------------------+

what tool are you using for this wonderful data?

Which OVH storage class were you using? Theres a slow (io) and a fast (highperf) one.

> mutating objects, storage policies, tiering, and conditional writes

My only hope is that they won't make it more complex than it should be in an effort to match AWS.

Working at a startup that deals in large data (our storage is measured in PB), egress is the only thing thing that matters if we're going to choose a new storage provider. And an automated class management system a la S3 or GCS.

In slightly different terms, Hetzner is roughly 5 €/TB/month of storage plus 1 €/TB of egress (ex. VAT).

Having implemented object storage at a service provider - it's extremely rare for any customer to have 1gbit/sec sustained throughput on an account, much less a bucket.

Those that do likely will already have an actual sales rep and the means to get these restrictions lifted, with a direct line to engineering.

Hetzner certainly markets to a different crowd, which I suspect is why these limits are being put in place ahead of time. It's very difficult to design around noisy neighbor, and will take some battle tested time in production to get right for the "average Joe" style customer who won't engage engineering resources to work with a provider to avoid hot spots and impacting other customers.

Presumably it's built on top of their existing VPS and dedicated server boxes, which AFAIK only have 1gbit uplinks, so they'd only be able to exceed that speed when the bucket is split between multiple boxes.

Just got a reply from them regarding the bucket amount and permission amount limits and both are in fact beta limits. No idea about the bandwidth though.

I'm guessing that's because it's in beta, they want to set some low limits and incrementally increase it, is my bet.

Backblaze seems to be 2x less expensive than Contabo.

I can't recommend Contabo, they have terrible support and constant hidden downtime.

Your link returns a 500 right now

> It's interesting that they implement AWS bucket policies, including all AWS-specific syntax/keys/etc, even "arn": https://docs.hetzner.com/storage/object-storage/faq/s3-crede...

It seems the Hetzner's docs say they use MinIO to expose their S3 managed service. I guess everything will be the same as MinIO upstream features and contigurations

They probably used chatgpt to build the product and copied-pasted

> extremely suspicious timing.

I've been wondering for well over a decade why Hetzner didn't get its ass in gear and start offering AWS-like services. Instead all they've offered for years is remote control of VMs. Even now, this new S3 offering is very little, very late -- 18 years after Amazon first offered it and took off like a rocket.

The only odd thing about the timing here is what was keeping them.

> Last week Hetzner sent me an email saying they were closing my account for an unknown reason.

Frankly, your service looks a bit suspicious (free and cheap file sharing). You seem to be aware of the potential for abuse - your DMCA/abuse page even says there are a "large number of abuse reports pixeldrain receives every day".

It sucks that Hetzner is closing your account. But, they are known to be pretty conservative and your users have almost certainly violated their T&C many times even if you are making a good faith effort to prevent and respond to abuse.

So you're paying them between €15,000 (€124 * 120 servers, hetzners cheapest sx server) and over €30,000 per month, and you really believe they sit in their little office trying to find servers of already paying customers, just to snatch them and host their own S3 service? This doesn't sound like a conspiracy theory to me; it sounds more like a serious case of megalomania.

That they don't communicate well with you really sucks, and I will read your blog post. But please stay on the sane side of reasoning.

Have you tried calling them? Could it be linked to pixeldrain and some illegal files which were uploaded and they simply don't want to take the risk?

That's terrible. Please consider documenting everything and blogging about this later on.

Thanks for the warning, was just about to start moving there.

As someone who has used Hetzner in one form or another for close to two decades, I frankly find this behaviour shocking.

Besides the normal support channels, try reaching out on their subreddit and also post in the customer forum - to make this more public.

If they don't want your business, fine. But giving you only a couple of weeks to move 12 PB is unreasonable.

Holy shit, you wrote pixeldrain? FYI (and I’m sure you’re very much aware), every single time I’ve come across your site, it’s always been via DDL/warez sites. I feel like you’re in a tough spot with this but I wish you all the best! I would love to read any write ups if you’ve published any!

>Hetzner's 1 Gbps port speed makes it impossible to transfer the data in less than three weeks

But don't you have that spread over 120 servers?

They have been around for a long time. I heard people praising them already in startup events 10+ year ago (I'm not that active in the scene myself any more). They are known for great price/quality ratio for the dedicated servers. There are some gotchas, as you might get random crappy hardware or whatever. The firms I have been working for and which decided to go build their infra mostly with hetzner, I would estimate that they have saved millions in comparison to common cloud services. The staff often complains because managing your own dedicated servers is a bit more work than going with something like AWS, but if you put in the numbers, the amount of money saved in the end can be insane.

I was with linode for a loooong time for my personal VM-in-cloud (like 15 years). But I got an itch to check around to see if there was a cheaper option earlier this year and found that Hetzner would give me literally twice the machine for about two thirds the cost. ($6/mo vs €4~/mo, 1gb nanode vs. CX22 so 1vcpu/1gb to 2vcpu/4gb and 25gb disk to 40gb)

So far I'm happy with the service. Linode wasn't bad per se, Hetzner just offered more for less. Their admin console is a bit more spartan in comparison to Linode's, but that's not a place I spend much time anyway. If Russia glasses Finland and Germany I'd have to find another provider but I'm pretty sure I'd have bigger problems at that point than where to park my wordpress blahg and irc session, lol.

They've existed since 1997 and have always been a solid host with good pricing on both bare metal and later VPS services. They seem to have managed to survive the Cloud onslaught as well by continuing to provide reliable service for good prices.

Perhaps you heard less about them because they're based in Europe and thus less hype sorrounds them?

They became a serious VPS contender amongst DigitalOcean, Linode, and other similar providers. DigitalOcean raised prices while Hetzner gave more powerful server capacity (read double) at lower price points. Hetzner has been increasingly popular in self-hosting communities.

I started using their services since last year for some of the operations, including business, and it's been rock solid. I plan on increasing usage, but will hold on this new object storage offering.

The pricing is much better than other providers, while being a reputable company. Pretty much the best value for non-enterprise workloads.

I've associated them for a while with uber cheap hosting, while also being allegedlly* reasonable (Most of the time) and reliable

* Couldn't tell you, as they flagged my account and asked me for an ID Document I couldn't provide, Maybe I could have sorted it with Customer Service, but instead I just picked a different german hosting

They are cheap. People have more of a focus on cost recently which pushes back against AWS. As far as vibes go, more people seem to be getting back into managing their own machines and self-hosting. I think Linode used to fill that void but for various reasons they relinquished that opportunity.

Hetzner is well known and beloved in Europe for a decade or two, so it didn't really start out of nowhere. Maybe you (and most of HN) became aware of them, when they opened their American datacenters in 2021 & 2022, and became a viable option for people outside of Europe.

It's very cheap. They've been around for years. I've been using them for my personal server for the past 2-3 years without problems. The only problems I have seen from them are the control plane API going down (so, like, creating new servers isn't available).

I've been with them ~15 years, and they have been solid. Back then I had a dedicated server I was developing an online multiplayer java game on, great fun! until the whole java webstart certificate fiasco. A few years back I was porting that old game over to wasm, and they added the mime type support within a few hours of asking. They are some very knowledgeable engineers, and maybe that is reflected in the simple website design. But I like that simple productive interface, and dread the day the MBAs or marketing teams get their hands on it.

I don't know about the last year, maybe they ramped up their marketing. But they've been pretty popular in my circle for the past 6-7 years, mostly because price/performance ratio is really favorable. I use it for the past ~3y.

great value and reliability

I got a pretty nice dedicated server with 2x3 TB HDD + 20 TB/month of traffic for just 25€/month. Was great for hosting >1 TB of video content publically.

That's been a feature of S3 for quite a long time now, called S3 Select https://docs.aws.amazon.com/AmazonS3/latest/userguide/select...

Despite it being an awesome feature I've been itching to use, I've never actually found a use for it beyond messing around. Most places where S3 Select might make sense seems to be subsumed (for my uses) by Athena. Athena has a rather large amount of conceptual and actual boilerplate to get up and running with, though, S3 Select requires no upfront planning beyond building a fancy query string (or using their SDK wrappers)

Where S3 Select is likely to become fiddly is anywhere multiple files are involved. Athena makes querying large collections of CSVs (etc) straightforward, and handles all the scheduling and results merging for you.

You could use Range HTTP Headers if you know which bit to request.

Kind of simple and not that useful for dynamic queries but quite good for queries where you know the queries beforehand and can index accordingly.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/range-...

What you're describing is targeted by Parquet/Iceberg + an engine like Dremio. But targeted at big data / analytics.

In the AWS world, Athena does more or less what you are describing: but is is targeted (AFAIU) for large datasets. I'm not sure how it would perform on a single (small) CSV to shave off a few bytes in transit.

Look into duckdb. It's not server side, but It's pretty phenomenal for this use case.

Yeah, on AWS you can set up a Glue Data Catalog and query it with Athena.

Unfortunately their signature dirt cheap bandwidth isn't so cheap in Singapore, you only get 1TB inclusive instead of 20TB and the overage rates are 7.4x higher than their EU and NA datacenters.

Please see: https://www.reddit.com/r/hetzner/comments/1cmhvzs/new_accoun... --Katie (Hetzner Online)

Cheap is a bit pejorative, I’d call it reasonable or competitive, ie. not exorbitant prices.

They know what they’re doing and they’re doing it well, cheap is more side effect than primary goal.

I think it's Minio.

Agree that the pricing model is highly unclear (which is usual for cloud services).

It does not tell me if I should count hours in hour-of-the-day, or lapsed time. Also the example tries to demonstrate a case of "you don´t have to pay extra", but then falls silent. Nice, not the info I am looking for.

What about envisioning a customer who asks «what am I going to pay? Specify it right now, right here».

It's not explained very clearly and made more complicated by being charged hourly instead of monthly, but essentially there's a minimum charge of €5 per month, which includes 1TB of storage and about 1TB of bandwidth.

Def needs a calculator if they intend to stay with this model at all IMO.

Yeah we are big fans of Hetzner and this pricing model makes no sense. I'm still trying to wrap my head around it and the strange limits associated with it.

It is for people using hetzner for stuff.

I run minio on hetzner and wouldn't send stuff across some other network to backblaze.

I was thinking of switching to BlackBlaze as I'm currently using Digital Ocean Spaces -- however I might look at Hetzner again -- not revolutionary but quite competitive.

1€ / TB egress is extremely attractive, though. Most other "budget" providers charge at least 10 - 20$.

… might be revolutionary for backblaze…

How much debt do they have on their balance sheet?

If you self hosted minio on their SX nodes you could prob get close to half

Where are you seeing these limits? I'm seeing:

Up to 10 TB per object

Up to 1 GBit/s bandwidth per Bucket

Up to 1024 operations/s per Bucket

Up to 100 TB per Bucket

Up to 100,000,000 objects per Bucket

Up to 100 S3 credentials across all projects

Up to 10 Buckets across all projects

It must be new - the page says they're in beta since two weeks ago and expect it to take a month or two to hit production.

It's nice to see more competition in this space.

Looks like it's in beta https://docs.hetzner.com/storage/object-storage/faq/beta

Pretty new. I enrolled in the beta a few weeks back, they will go GA later.

I'm a big Hetzner fanboy, quite sad that pricing is't that competetive...

Locking semantics? Consistent listing of large numbers of files?

I'm speculating but those are at least the two I can think of that aren't explicitly linked to speed equivalency of a basic filesystem.

I’m a wasabi customer, why is backblaze better as per your research?

If you're trying to sensibly engage people in discussion about this then you probably want to provide some context. I personally have no idea what you're referring to.