At the very bottom of the article, I see this notice:
> This article was originally written in Korean and translated by a bilingual reporter with the help of generative AI tools. It was then edited by a native English-speaking editor. All AI-assisted translations are reviewed and refined by our newsroom.
Especially since LLM tech was originally developed for translation. That’s the original reason so much work was done to create a model that could handle context and it turned out that was helpful in more areas than just translation.
While LLM usage is just spinning up in other areas, for translation they have been doing this job well for over 5 years now.
This is how I’ve done translation for a number of years, even pre-LLM, between the languages I speak natively - machine translation is good enough that it’s faster for me to fix its problems than for me to do it from scratch.
(Whether machine translation uses LLMs or not doesn’t seem especially relevant to the workflow.)
So just a blanket message at the bottom of the page "anything and everyone you read here might be total bullshit"
It's still worse than useless :
https://www.bloodinthemachine.com/p/ai-killed-my-job-transla...
I really don't get this take where people try to downplay AI the most where it is obviously having the most impact. Sure. A billion people are supposed to go back to awful machine translation so that a few tens of thousands can have jobs that were already commodity.
> I'm fine with people using LLMs for natural language related work
phew I'm relieved you're okay with people using modern tools to get their job done
Woah, read the timeline at the top of this. The fire happened the very day the government ordered onsite inspection was supposed to start due to Chinese/NK hacking.
Phrack's timeline may read like it, but it wasn't an onsite inspection due to hacking, but a scheduled maintenance to replace the overdue UPS, hence battery-touching involved. Even the image they linked just says "scheduled maintenance."
So right after the investigation was announced, they suddenly scheduled a UPS battery replacement which happened to start a fire big enough to destroy the entire data centre and all data or evidence?
Yeah, that's way less suspicious, thanks for clearing that up.
My mind initially went to a government cover-up, but then:
> 27th of September 2025, The fire is believed to have been caused while replacing Lithium-ion batteries. The batteries were manufactured by LG, the parent company of LG Uplus (the one that got hacked by the APT).
Could the battery firmware have been sabotaged by the hacker to start the fire?
Supply chain interceptions can happen for batteries and other electronics being used.
look at the timeline again. this is the second fire.
Such coincidences do happen. 20 years ago the plane which was carrying all the top brass of the Russian Black Sea Fleet as well as the Fleet’s accounting documentation for inspection to Moscow burst in flames and fell to the ground while trying to get airborne. Being loaded with fuel it immediately became one large infernal fireball. By some miracle no top brass suffered even minor burn/injury while all the accounting documentation burned completely.
Yeah, this whole thing smells.
Who has the incentive to do this, though? China/North Korea? Or someone in South Korea trying to cover up how bad they messed up? Does adding this additional mess on top mean they looked like they messed up less? (And for that to be true, how horrifically bad does the hack have to be?)
It might be different “they”s. Putting on my tinfoil hat, whoever was going to be in hot water over the hack burns it down and now the blame shifts from them to whoever manages G-drive and don’t have a backup plan.
Not saying I believe this (or even know enough to have an opinion), but it’s always important to not anthropomorphize a large organization. The government isn’t one person (even in totalitarian societies) but an organization that contains large numbers of people who may all have their own motivations.
So, someone figured out how to do backups
"NK hackers" reminds me "my homework was eaten by a dog". It's always NK hackers that steal data/crypto and there is absolutely no possibility to do something with it or restore the data, because you know they transfer the info on a hard disk and they shoot it with an AD! Like that general!
How do we know it's NK? Because there are comments in north-korean language, duh! Why are you asking, are you russian bot or smt??
wow https://x.com/koryodynasty/status/1973956091638890499
> A senior government official overseeing recovery efforts for South Korea's national network crisis has reportedly died by suicide in Sejong.
If the US government and corporate executives had even half this level of shame, we'd have nobody left in those positions!
You should look at the previous president of SK. Maybe a few more too... they frequently land in jail...
I'm not sure Yoon Suk Yeol had any shame
"suicide" in these circumstances is usually something else altogether.
Even in cases it is executed by themselves, shame won't be the primary motivation.
You may want to familiarize yourself more with the culture around this in places like South Korea and Japan.
It can be posed as shame on the front side.
More often than not the suicide covers a whole organization's dirty laundry. You'll have people drunk and driving their cars over cliffs [0], low profile actors ending their life as shit hits the fan [0] etc.
Then some on the lower rank might still end their life to spare their family financially (insurance money) or because they're just so done with it all, which I'd put more on depression than anything.
Us putting it on shame is IMHO looking at it through rose colored glasses and masking the dirtier reality to make it romantic.
[0] https://bunshun.jp/articles/-/76130
[1] https://www.tsukubabank.co.jp/cms/article/a9362e73a19dc0efcf...
In Korea, shame often serves as the primary motivator behind high-profile suicides. It's rooted in the cultural concept of "chemyeon (체면)", which imposes immense pressure to maintain a dignified public image.
Do you have any example of these high profile suicides that can't be better explained as "taking one for the team" for lack of a better idiom.
Shame is a powerful social force throughout the society, but we're talking about systematic screwings more often than not backed by political corruption (letting incompetent entities deal with gov contract on basis of political money and other favors) or straight fraud.
It usually isn't but people do usually imply otherwise.
Silver lining: it's likely that technically there is a backup (section 1.3).
It's just in NK or china.
Yikes.
I don't backup my phone. The NSA does it for me!
The recovery process and customer service around that is near impossible
In the same respect /dev/null can backup mine. Good luck getting data back.
When you see a chronology like that, you don't keep trying to speak truth to power.
You delete your data, trash your gear, and hop on a bus, to start over in some other city, in a different line of work.
And with no technology! Perhaps become some kind of ascetic monk.
s/city/country/
Though this is far from the most important points of this article, why do even the article’s authors defend Proton after having their accounts suspended, and after having seemingly a Korean intelligence official warn them that they weren’t secure? Even if they’re perfectly secure they clearly do not have the moral compass people believe they have.
> 27th of September 2025, The fire is believed to have been caused while replacing Lithium-ion batteries. The batteries were manufactured by LG, the parent company of LG Uplus (the one that got hacked by the APT).
Compromised batteries or battery controllers?
Witness A said, “It appears that the fire started when a spark flew during the process of replacing the uninterruptible power supply,” and added, “Firefighters are currently out there putting out the fire. I hope that this does not lead to any disruption to the national intelligence network, including the government’s 24 channel.”[1]
How large is this UPS that a fire can bring down all 96 servers?
This story is really unbelievable.
I’m no expert but traditional lead acid battery UPS are typically at the bottom of the rack due to weight and concern about leakage. Wouldn’t surprise me if li-ion UPS go at the bottom as well. In that case if uncontrolled it seems pretty easy to torch an entire rack.
96 servers isn’t that many, probably less than 10 racks and given the state of the backups it would track that they didn’t spring for halon.
depends on how many batteries were in the facility, if one goes up chances are the rest go too. Can halon systems not put out lithium fires?
Ohh side note but this was the journalist group which was blocked by proton
The timing as well is very suspicious and I think that there can be a lot of discussion about this
Right now, I am wondering about the name most tbh which might seem silly but "APT down - The North Korean files"
It seems that APT means in this case advanced persistent threat but I am not sure what they mean by Apt Down, like the fact that it got shut down by their journalism or-? I am sorry if this may seem naive and on a serious note this raises so many questions...
“APT Down” is likely a reference to a popular Korean drinking game.
https://www.thetakeout.com/1789352/korea-apt-drinking-game-r...
For a moment there I was wondering if “apt down” was a typo and you meant “ifdown”. ;)
Thanks for this, it gives a lot of extra info and content compared to the original article.
thanks for the info, canceling proton rn
This is the first time I see this site, who/what is phrack? A hacker group?
It’s a zine. Been around since the 80’s. Hackers / security industry types read and publish to it.
For more context, the name derives from "phone hacking" or phreacking. You got your legends like Captain Crunch and many of you big tech players were into this stuff when they were younger, such as Woz
This was also often tied to a big counter culture movement. Which one interesting thing is that many of those people now define the culture. I guess not too unlike how many hippies changed when they grew up
Not sure why people downvoted you as I actually read the wikipedia and learnt a lot about phrack and how their name is sort of inspired by "phreaking,anarchy and cracking" and I think thus the name ph-ra-ck.
This sounds like a real whodunit.
Well, I think we know "who"dunnit it's more of a how-dunnit & are-they-still-in-dunnit
> KIM is heavily working on ToyBox for Android.
2 HN front page articles in 1!
Figures.
The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then...
"The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups."
This is absolutely wild.
The issue here is not refusing to use a foreign third party. That makes sense.
The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk.
That’s gross mismanagement.
why? I can upload backups to 20 third parties and no one has a prayer of getting access to those files. Are you under an impression that's challenging?
As a sysadmin at company that provide fairly sensitive services, I find online cloud backups to be way to slow for the purpose of protecting against something like the server room being destroyed by a fire. Even something like spinning disks at a remote location feel like a risk, as files would need to be copied onto faster disks before services could be restored, and that copying would take precious time during an emergency. When downtime means massive losses of revenue for customers, being down for hours or even days while waiting for the download to finish is not going be accepted.
Restoring from cloud backups is one of those war stories that I occasionally hear, including the occasionally fedex solution of sending the backup disk by carrier.
How’s that? Using encryption, which is known to have backdoors and is vulnerable to nation state cracking?
It is much more likely and cheaper, that US marines will desant and capture your backup facility, than someone would break AES-128.
Sending troops would be an act of war, and definitely not cheap.
Stealing some encryption keys, just another Wednesday.
There might be unknown unknowns....
Please provide any proof or references to what you are claiming.
Can you provide an example of a commonly used cryptography system that is known to be vulnerable to nation state cracking?
As for backdoors, they may exist if you rely on a third party but it's pretty hard to backdoor the relatively simple algorithms used in cryptography
>Using encryption, which is known to have backdoors and is vulnerable to nation state cracking?
WTF are you talking about? There are absolutely zero backdoors of any kind known to be in any standard open source encryption systems, and symmetric cryptography 256-bits or more is not subject to cracking by anyone or anything, not even if general purpose quantum computers are doable and prove scalable. Shor's algorithm applies to public-key not symmetric, where the best that can be done is Grover's quantum search for a square-root speed up. You seem to be crossing a number of streams here in your information.
As someone who’s fairly tech-literate but has a big blind spot in cryptography, I’d love to hear any suggestions you have for articles, blog posts, or smaller books on the topic!
My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable. You sound both very knowledgeable on the topic, and very confident in the safety of modern encryption. I’m thinking maybe my understanding is obsolete!
> My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable
Lol this is woefully misinformed.
Agree completely that it's absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.
Good thing Korea has cloud providers, apparently Kakao has even gone...beyond the cloud!
https://kakaocloud.com/ https://www.nhncloud.com/ https://cloud.kt.com/
To name a few.
Samsung owns Joyent
Nevertheless isn't Joyent registered in the US?
They are overwhelmingly whitelabeled providers. For example, Samsung SDI Cloud (the largest "Korean" cloud) is an AWS white label.
Korea is great at a lot of engineering disciplines. Sadly, software is not one of them, though it's slowly changing. There was a similar issue a couple years ago where the government's internal intranet was down a couple days because someone deployed a switch in front of outbound connections without anyone noticing.
It's not a talent problem but a management problem - similar to Japan's issues, which is unsurprising as Korean institutions and organizations are heavily based on Japanese ones from back in the JETRO era.
I spent a week of my life at a major insurance company in Seoul once, and the military style security, the obsession with corporate espionage, when all they were working on was an internal corporate portal for an insurance company… The developers had to use machines with no Internet access, I wasn’t allowed to bring my laptop with me lest I use it to steal their precious code. A South Korean colleague told me it was this way because South Korean corporate management is stuffed full of ex-military officers who take the attitudes they get from defending against the North with them into the corporate world; no wonder the project was having so many technical problems-but I couldn’t really solve them, because ultimately the problems weren’t really technical
> South Korean corporate management is stuffed full of ex-military officers
The way it was explained to me was different... yes, all able-bodied males do national service. But there's a different phenomenon in which someone serves some years active duty (so this is not their mandatory national service, this is voluntary active duty service), in some relatively prestigious position, and then jumps ship to the corporate world, and they get hired as an executive by their ex-comrades/ex-superiors... so there ends up being a pipeline from more senior volunteer active duty military ranks into corporate executive ranks (especially at large and prestigious firms), and of course that produces a certain culture, which then tends to flow downhill
I've done some work for a large SK company and the security was manageable. Certainly higher than anything I've seen before or after and with security theater aspects, but ultimately it didn't seriously get in the way of getting work done.
I think it makes sense that although this is a widespread problem in South Korea, some places have it worse than others; you obviously worked at a place where the problem was more moderate. And I went there over a decade ago, and maybe even the place I was at has lightened up a bit since.
I see they even amputated your periods, and forced you to use ellipses exclusively.
> Korea is great at a lot of engineering disciplines. Sadly, software is not one of them
I am very happy with the software that powers my Hyundai Tuscon hybrid. (It's a massive system that runs the gas and electric engines, recharging, shifting gears, braking, object detection, and a host of information and entertainment systems.) After 2 years, 0 crashes and no observable errors. Of course, nothing is perfect: maps suck. The navigation is fine; it's the display that is at least 2 decades behind the times.
Encrypted backups would have saved a lot of pain here
Any backup would do at this point. I think the most best is: encrypted, off-site & tested monthly.
> no government should keep critical data on foreign cloud storage
Primary? No. Back-up?
These guys couldn’t provision a back-up for their on-site data. Why do you think it was competently encrypted?
They fucked up, that much is clear but the should not have kept that data on foreign cloud storage regardless. It's not like there are only two choices here.
> the should not have kept that data on foreign cloud storage regardless. It's not like there are only two choices here
Doesn't have to be an American provider (Though anyone else probably increases Seoul's security cross section. America is already its security guarantor, with tens of thousands of troops stationed in Korea.)
And doesn't have to be permanent. Ship encrypted copies to S3 while you get your hardenede-bunker domestic option constructed. Still beats the mess that's about to come for South Korea's population.
I'm aware of a big cloud services provider (I won't name any names but it was IBM) that lost a fairly large amount of data. Permanently. So that too isn't a guarantee. They simply should have made local and off-line backups, that's the gold standard, and to ensure that those backups are complete and can be used to restore from scratch to a complete working service.
>I'm aware of a big cloud services provider (I won't name any names but it was IBM) that lost a fairly large amount of data. Permanently. So that too isn't a guarantee.
Permanently losing data at a given store point isn't relevant to losing data overall. Data store failures are assumed or else there'd be no point in backups. What matters is whether failures in multiple points happen at the same time, which means a major issue is whether "independent" repositories are actually truly independent or whether (and to what extent) they have some degree of correlation. Using one or more completely unique systems done by someone else entirely is a pretty darn good way to bury accidental correlations with your own stuff, including human factors like the same tech people making the same sorts of mistakes or reusing the same components (software, hardware or both). For government that also includes political factors (like any push towards using purely domestic components).
>They simply should have made local and off-line backups
FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.
> Permanently losing data at a given store point isn't relevant to losing data overall.
I can't reveal any details but it was a lot more than just a given storage point. The interesting thing is that there were multiple points along the way where the damage would have been recoverable but their absolute incompetence made matters much worse to the point where there were no options left.
> FWIW there's no "simply" about that though at large scale. I'm not saying it's undoable at all but it's not trivial. As is literally the subject here.
If you can't do the job you should get out of the kitchen.
In this context the entirety of IBM cloud is basically a single storage point.
(If IBM was also running the local storage then we're talking about a very different risk profile from "run your own storage, back up to a cloud" and the anecdote is worth noting but not directly relevant.)
DigitalOcean lost some of my files in their object storage too: https://status.digitalocean.com/incidents/tmnyhddpkyvf
Using a commercial provider is not a guarantee.
DO Spaces, for at least a year after launch, had no durability guarantees whatsoever. Perhaps they do now, but I wouldn’t compare DO in any meaningful way to S3, which has crazy high durability guarantees as well as competent engineering effort expended on designing and validating that durability.
You don’t need cloud when you have the data centre, just backups in physical locations somewhere else
Why not? If the region is in country, encrypted, and with proven security attestations validated by third parties, a backup to a cloud storage would be incredibly wise. Otherwise we might end up reading an article about a fire burning down a single data center
Microsoft has already testified that the American government maintains access to their data centres, in all regions. It likely applies to all American cloud companies.
America is not a stable ally, and has a history of spying on friends.
So unless the whole of your backup is encrypted offline, and you trust the NSA to never break the encryption you chose, its a national security risk.
> France spies on the US just as the US spies on France, the former head of France’s counter-espionage and counter-terrorism agency said Friday, commenting on reports that the US National Security Agency (NSA) recorded millions of French telephone calls.
> Bernard Squarcini, head of the Direction Centrale du Renseignement Intérieur (DCRI) intelligence service until last year, told French daily Le Figaro he was “astonished” when Prime Minister Jean-Marc Ayrault said he was "deeply shocked" by the claims.
> “I am amazed by such disconcerting naiveté,” he said in the interview. “You’d almost think our politicians don’t bother to read the reports they get from the intelligence services.”
> “The French intelligence services know full well that all countries, whether or not they are allies in the fight against terrorism, spy on each other all the time,” he said.
> “The Americans spy on French commercial and industrial interests, and we do the same to them because it’s in the national interest to protect our companies.”
> “There was nothing of any real surprise in this report,” he added. “No one is fooled.”
France has had a reputation for being especially active in industrial espionage since at least the 1990s. Here's an article from 2011 https://www.france24.com/en/20110104-france-industrial-espio...
I always thought it was a little unusual that the state of France owns over 25% of the defense and cyber security company Thales.
> I always thought it was a little unusual that the state of France owns over 25% of the defense and cyber security company Thales.
Unusual from an American perspective, maybe. The French state has stakes in many companies, particularly in critical markets that affect national sovereignty and security, such as defence or energy. There is a government agency to manage this: https://en.wikipedia.org/wiki/Agence_des_participations_de_l... .
> America is not a stable ally, and has a history of spying on friends
America is a shitty ally for many reasons. But spying on allies isn’t one of them. Allies spy on allies to verify they’re still allies. This has been done throughout history and is basic competency in statecraft.
That doesn’t capture the full truth. Since Snowden, we have hard evidence the NSA has been snooping on foreign governments and citizens alike with the purpose of harvesting data and gathering intelligence, not just to verify their loyalty.
No nation should trust the USA, especially not with their state secrets, if they can help it. Not that other countries are inherently more trustworthy, but the US is a known bad actor.
> Since Snowden, we have hard evidence the NSA has been snooping on foreign governments and citizens alike
We also know this is also true for Russia, China and India. Being spied on is part of the cost of relying on external security guarantees.
> Not that other countries are inherently more trustworthy, but the US is a known bad actor
All regional and global powers are known bad actors. That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.
There is no such thing as good or trustworthy actors when it comes to state affairs. Each and every one attempt to spy on the others. Perhaps US have more resources to do so than some others.
You really have no evidence to back up your assertion, because you’d have to be an insider.
Not only does the NSA break encryption but they actually sabotage algorithms to make them easier to break when used.
Can the NSA break the Ed25519 stuff? Like the crypto_box from libsodium?
Large amounts of data, like backups, are encrypted using a symmetric algorithm. Which makes the strength of Ed25519 somewhat unimportant in this context.
ed25519 (and ec25519) are generally understood not to be backdoored by the NSA, or weak in any known sense.
The lack of a backdoor can be proven by choosing parameters according to straightforward reasons that do not allow the possibility for the chooser to insert a backdoor. The curve25519 parameters have good reasons why they are chosen. By contrast, Dual_EC_DRBG contains two random-looking numbers, which the NSA pinky-swears were completely random, but actually they generated them using a private key that only the NSA knows. Since the NSA got to choose any numbers to fit there, they could do that. When something is, like, "the greatest prime number less than 2^255" you can't just insert the public key of your private key into that slot because the chance the NSA can generate a private key whose public key just happens to match the greatest prime number less than 2^255 is zero. These are called "nothing up my sleeve numbers".
This doesn't prove the algorithm isn't just plain old weak, but nobody's been able to break it, either. Or find any reason why it would be breakable. Elliptic curves being unbreakable rests on the discrete logarithm of a random-looking permutation being impossible to efficiently solve, in a similar way to how RSA being unbreakable relies on nobody being able to efficiently factorize very big numbers. The best known algorithms for solving discrete logarithm require O(sqrt(n)) time, so you get half the bits of security as the length of the numbers involved; a 256-bit curve offers 128 bits of security, which is generally considered sufficient.
(Unlike RSA, you can't just arbitrarily increase the bit length but have to choose a completely new curve for each bit length, unfortunately. ed25519 will always be 255 bits, and if a different length is needed, it'll be similar but called something else. On the other hand, that makes it very easy to standardize.)
> but nobody's been able to break it, either.
Absence of evidence is not evidence of absence. It could well be that someone has been able to break it but that they or that organization did not publish.
How could you not!? Think of the bragging rights. Or, perhaps the havoc. That persons could sit on this secret for long periods of time seem... difficult to maintain. If you know it's broken and you've discovered it; surely someone else could too. And they've also kept the secret?
I agree on the evidence/absence of conjecture. However, the impact of the secret feels impossible to keep.
Time will, of course, tell; it wouldn't be the first occasion where that has embarrassed me.
There are a large number mathematicians gainfully employed in breaking such things without talking about it.
Exactly.
Like, don't store it in the cloud of an enemy country of course.
But if it's encrypted and you're keeping a live backup in a second country with a second company, ideally with a different geopolitical alignment, I don't see the problem.
The problem is money,
you are seeing the local storage decision under the lens of security, that is not the real reason for this type of decision.
While it may have been sold that way, reality is more likely the local DC companies just lobbied for it to be kept local and cut as many corners as they needed. Both the fire and architecture show they did cut deeply.
Now why would a local company voluntary cut down its share of the pie by suggesting to backup store in a foreign country. They are going to suggest keep in country or worse as was done here literally the same facility and save/make even more !
The civil service would also prefer everything local either for nationalistic /economic reasons or if corrupt then for all kick backs each step of the way, first for the contract, next for the building permits, utilities and so on.
Enemy country in the current geopolitical climate is an interesting take. Doesn't sound like a great idea to me tbh.
There are a lot of gray relations out there, but there’s almost no way you could morph the current US/SK relations to one of hostility; beyond a negligible minority of citizens in either being super vocal for some perceived slights.
You think when ICE arrested over 300 South Korean citizens who were setting up a Georgia Hyundai plant and subjected them to alleged human rights abuses, it was only a perceived slight?
https://www.huffpost.com/entry/south-korea-human-rights-inve...
How Trump’s ICE Raid Triggered Nationwide Outrage in South Korea
https://www.newsweek.com/trump-ice-raid-hyundai-outrage-sout...
'The raid "will do lasting damage to America's credibility," John Delury, a senior fellow at the Asia Society think tank, told Bloomberg. "How can a government that treats Koreans this way be relied upon as an 'ironclad' ally in a crisis?"'
One could have said the exact same thing about US-EU relations just a couple of years ago. And yet, here we are.
A year ago, I would have easily claimed the same thing about Denmark.
I don't follow. Can you share more context?
Trump will find a way, just as he did with Canada for example (i mean, Canada of all places). Things are way more in flux than they used to be. There’s no stability anymore.
A country can become an adversary faster than a government can migrate away from it.
Hence a backup country. I already covered that.
But while countries go from unfriendly to attacking you overnight, they don't generally go from friendly to attacking you overnight.
Overnight, Canada went from being an ally of the US to being threatened by annexation (and target #1 of an economic war).
If the US wants its state-puppet corporations to be used for integral infrastructure by foreign governments, it's going to need to provide some better legal assurances than 'trust me bro'.
(Some laws on the books, and a congress and a SCOTUS that has demonstrated a willingness to enforce those laws against a rogue executive would be a good start.)
From the perspective of securing your data, what's the practical difference between a second country and an enemy country? None. Even if it's encrypted data, all encryption can be broken, and so we must assume it will be broken. Sensitive data shouldn't touch outside systems, period, no matter what encryption.
Any even remotely proper symmetric encryption scheme "can be broken" but only if you have a theoretical adversary with nearly infinite power and time, which is in practice absolutely utterly impossible.
I'm sure cryptographers would love to know what makes it possible for you to assume that say AES-256 or AES-512 can be broken in practice for you to include it in your risk assessment.
You’re assuming we don’t get better at building faster computers and decryption techniques. If an adversary gets hold of your encrypted data now, they can just shelf it until cracking becomes eventually possible in a few decades. And as we’re talking about literal state secrets here, they may very well still be valuable by then.
Barring any theoretical breakthroughs, AES can't be broken any time soon even if you turned every atom in the universe into a computer and had them all cracking all the time. There was a paper that does the math.
You make an incorrect assumption about my assumptions. Faster computers or decryption techniques will never fundamentally "break" symmetric encryption. There's no discrete logarithm or factorization problem to speed up. Someone might find ways to make for example AES key recovery somewhat faster, but the margin of safety in those cases is still incredibly vast. In the end there's such an unfathomably vast key space to search through.
The risk that the key leaks through an implementation bug or a human intelligence source.
Exfiltrating terabytes of data is difficult, exfiltrating 32 bytes is much less so.
That's very far from the encryption itself being broken though. If that were the claim, I would have had no complaints.
A statement like "all encryption can be broken" is about as useful as "all systems can be hacked" in which case, not putting data in the cloud isn't really a useful argument.
> From the perspective of securing your data, what's the practical difference between a second country and an enemy country? None.
Huh? An enemy country will shut off your access. Friendly countries don't.
> Even if it's encrypted data, all encryption can be broken, and so we must assume it will be broken.
This is a very, very hot take.
And which organization has every file, from each of their applications using the cloud, encrypted *before* it is sent to the cloud?
They're talking about backups. you can absolutely send an updated copy every night.
True, the user I was replying to only mentioned backups.
For those there's sure no problem
If you can’t encrypt your backups such that you could store them tatooed on Putin’s ass, you need to learn about backups more.
And yet here is an example where keeping critical data off public cloud storage has been significantly worse for them in the short term.
Not that they should just go all in on it, but an encrypted copy on S3 or GCS would seem really useful right about now.
You can do a bad job with public or private cloud. What if they would have had the backup and lost the encryption key?
Cost wise probably having even a Korean different data center backup would not have been huge effort, but not doing it exposed them to a huge risk.
We’ve had Byzantine crypto key solutions since at least 2007 when I was evaluating one for code signing for commercial airplanes. You could put an access key on k:n smart cards, so that you could extract it from one piece of hardware to put on another, or you could put the actual key on the cards so burning down the data center only lost you the key if you locked half the card holders in before setting it on fire.
SSS is from 1979. https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing
Using the cloud would have been the easiest way to achieve the necessary redundancy, but by far not the only one. This is just a flawed concept from the start, with no real redundancy.
But not security. And for governmental data security is a far more important consideration.
not losing data and keeping untrusted parties out of your data is a hard problem, that "cloud" aka "stored somewhere that is accessible by agents of a foreign nation" does not solve.
It's the government of South Korea, which has a nearly 2 trillion dollar GDP. Surely they could have built a few more data centers connected with their own fiber if they were that paranoid about it.
As OP says, cloud is not the only solution, just the easiest. They should probably have had a second backup in a different building. It would probably require a bit more involvement, but def doable.
The simple solution here would have been something like a bunch of netapps with snapmirrors to a secondary backup site.
Or ZFS or DRBD or whatever homegrown or equivalent non-proprietart alternative is available these days and you prefer.
Because these companies never lose data, like during some lightning strikes, oh wait: https://www.bbc.com/news/technology-33989384
As a government you should not be putting your stuff in an environment under control of some other nation, period. That is a completely different issue and does not really relate to making backups.
“The BBC understands that customers, through various backup technologies, external, were able to recover all lost data.”
You backup stuff. To other regions.
But the Korean government didn't backup, that's the problem in the first place here…
Sure. Using a cloud can make that more convenient. But obviously not so if you then keep all your data in the same region, or even “availability-zone” (which seems to be the case for the all “lost to lightening strikes” data here).
>As a government you should not be putting your stuff in an environment under control of some other nation, period.
Why? If you encrypt it yourself before transfer, the only possible control some_other_nation will have over you or your data is availability.
You're forgetting that you're talking nation states, here. Breaking encryption is in fact the role of the people you are giving access.
Sovereign delivery makes sense for _nations_.
You can use and abuse encrypted one time pads and multiple countries to guarantee it’s not retrievable.
You're assuming a level of competency that's hard to warrant at this point.
If your threat model is this high that you assume encryption breaking to be into your threat model, then maybe you do need a level of comeptency in the process as well.
They have 2 Trillion $ economy. I am sure that competency shouldn't be the thing that they should be worrying at that scale but at the same time I know those 2 trillion $ don't really make them more competent but I just want to share that it was very possible for them to teach/learn the competency
Maybe this incident teaches us atleast something. Definitely something to learn here though. I am interested in how the parent comment suggests sharing one time pad or rather a practical way for them to do so I suppose since I am genuinely curious as most others refer to using the cloud like aws etc. and I am not sure how much they can share something like one time pad and at the scale of petabytes and more, I can maybe understand it but I would love if the GP can tell me a practical way of doing so to atleast have more safety I suppose than encryption methods I suppose..
I think it doesn't need to be the encryption breaking per se.
It could be a gov laptop with the encryption keys left at a bar. Or the wrong keys saved on the system and the backups can't actually be decrypted. Or the keys being reused at large scale and leaked/guessed from lower security area. etc.
Relying on encryption requires operation knowledge and discipline. At some point, a base level of competency is required anyway, I'm not just sure encryption would have saved them as much as we'd wish it would.
To your point, I'd assume high profile incidents like this one will put more pressure to do radical changes, and in particular to treat digital data as a more critical asset that you can't hand down to the crookest corrupt entity willy nilly just for the kickback.
South Korea doesn't lack competent people, but hiring them and letting them at the helm sounds like a tough task.
Using a OTP in your backup strategy adds way more complexity, failure modes, and costs with literally no improvement in your situation.
First of all, you cannot do much if you keep all the data encrypted on the cloud (basically just backing things up, and hope you don't have to fetch it given the egress cost). Also, availability is exactly the kind of issue that a fire cause…
Yeah backups would’ve been totally useless in this case. All South Korea could’ve done is restore their data from the backups and avoid data loss.
Yeah, I heard that consumer clouds are only locally redundant and there aren't even backups. So big DC damage could result in data loss.
By default, Amazon S3 stores data across at least separate datacenters that are in the same region, but are physically separate from each other:
Amazon S3 provides a highly durable storage infrastructure designed for mission-critical and primary data storage. S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.
You can save a little money by giving up that redundancy and having your data i a single AZ:
The S3 One Zone-IA storage class stores data redundantly across multiple devices within a single Availability Zone
For further redundancy you can set up replication to another region, but if I needed that level of redundancy, I'd probably store another copy of data with a different cloud provider so an AWS global failure (or more likely, a billing issue) doesn't leave my data trapped in one vendor).
I believe Google and Azure have similar levels of redundancy levels in their cloud storage.
What do you mean by "consumer clouds"?
I mean… at the risk of misinterpreting sarcasm—
Except for the backup strategy said consumers apply to their data themselves, right?
If I use a service called “it is stored in a datacenter in Virginia” then I will not be surprised when the meteor that hits Virginia destroys my data. For that reason I might also store copies of important things using the “it is stored in a datacenter in Oregon” service or something.
For this reason, Microsoft has Azure US Government, Azure China etc
...on a single-zone persistent disk: https://status.cloud.google.com/incident/compute/15056#57195...
> GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters.
Of course, it's perfectly possible to have proper distributed storage without using a cloud provider. It happens to be hard to implement correctly, so apparently, the SK government team in question just decided... not to?
Not sure “sane backup strategy” and “park your whole government in a private company under American jurisdiction” are mutually exclusive. I feel like I can think of a bunch of things that a nation would be sad to lose, but would be even sadder to have adversaries rifling through at will. Or, for that matter, extort favors under threat of cutting off your access.
At least in this case you can track down said officials in their foxholes and give them a good talking-to. Good luck holding AWS/GCP/Azure accountable…
I know there is legit hate for VMWare/Broadcom but there is a legit case to be made for VCF with an equivalent DR setup where you have replication enabled by Superna and Dell PowerProtect Data Domain protecting both local and remote with Thales Luna K160 KMIP for the data at rest encryption for the vSAN.
To add, use F710s, H710s and then add ObjectScale storage for your Kubernetes workloads.
This setup repatriates your data and gives you a Cloud like experience. Pair it with like EKS-A and you have a really good on premises Private Cloud that is resilient.
He may or may not have been right, but it's besides the point.
The 3-2-1 backup rule is basic.
cloud will also not back up your stuff if you configure it wrong so not sure how's that related
Well it is just malpractise. Even when I was an first semester art student I knew about the concept of off-site backups.
Rightfully did not trust these companies. Sure what happened is a disaster for them, but you cant simply trust Amazon & Microsoft.
Why not? You can easily encrypt your data before sending it for storage on on S3, for example.
You and I can encrypt our data before saving it into the cloud, because we have nothing of value or interest to someone with the resources of a state.
Sometimes sensitive data at the government level has a pretty long shelf life; you may want it to remain secret in 30, 50, 70 years.
Is encryption, almost any form, really reliable protection for a countries' government entire data? I mean, this is _the_ ultimate playground for "state level actors" -- if someday there's a hole and it turns out it takes only 20 years to decrypt the data with a country-sized supercomputer, you can bet _this_ is what multiple alien countries will try to decrypt first.
You're assuming that this needs to protect...
> ... a countries' government entire data?
But the bulk of the data is "boring": important to individuals, but not state security ("sorry Jiyeong, the computer doesn't know if you are a government employee. Apologies if you have rent to make this month!")
There likely exists data where the risk calculation ends up differently, so that you wouldn't store it in this system. For example, for nuke launch codes, they might rather lose than loose them. Better to risk having to reset and re-arm them than to have them hijacked
> Is encryption, [in?] any form, really reliable protection
There's always residual risk. E.g.: can you guarantee that every set of guards that you have watching national datacenters is immune from being bribed?
Copying data around on your own territory thus also carries risks, but you cannot get around it if you want backups for (parts of) the data
People in this thread are discussing specific cryptographic primitives that they think are trustworthy, which I think goes a bit deeper than makes sense here. Readily evident is that there are ciphers trusted by different governments around the world for their communication and storage, and that you can layer them such that all need to be broken before arriving at the plain, original data. There is also evidence in the Snowden archives that (iirc) e.g. PGP could not be broken by the NSA at the time. Several ciphers held up for the last 25+ years and are not expected to be broken by quantum computers either. All of these sources can be drawn upon to arrive at a solid choice for an encryption scheme
A foreign gov getting all your security researchers and staff's personal info with their family and tax and medical records doesn't sound great.
That's just from the top of my head. Exploiting such a trove of data doesn't sound complicated.
Yeah that ignores about two thirds of my point, including that it would never get to the "Exploiting such a trove of data doesn't sound complicated" stage with a higher probability than storing it within one's own territory
I'm in agreement with your second point, I think moving data in the country isn't trivial either and requires a pretty strong system. I just don't have much to say on that side, so didn't comment on it.
You can encrypt them at rest, but data that lies encrypted and is never touched, is useless data. You need to decrypt them as well. Also, plenty of incompetent devops around, and writing a decryption toolchain can be difficult.
Am I missing something? If you ever need to use this data, obviously you transfer it back to your premises and then decrypt it. Whether it's stored at Amazon or North Korean Government Cloud makes no difference whatsoever if you encrypt before and decrypt after transfer.
They can take the data hostage, the foreign nation would have no recourse.
Have it in multiple countries with multiple providers if money isn't a concern.
And are we forgetting that they can literally have a multi cloud backup setup in their own country as well or incentivize companies to build their datacenters there in partnership with them of sorts with a multi cloud setup as I said earlier?
Encryption only protects data for an unknown period of time, not indefinately.
If your threat model includes the TLA types, then backup to a physical server you control in a location geographically isolated from your main location. Or to a local set of drives that you physically rotate to remote locations.
Decryption is not usually an issue if you encrypt locally.
Tools like Kopia, Borg and Restic handle this and also include deduplication and other advanced features.
Really no excuse for large orgs or even small businesses and somewhat tech literate public.
Why write one when there are tools like “restic”?
For sure the only error here is zero redundancy.
S3 features have saved our bacon a number of times. Perhaps your experience and usage is different. They are worth trusting with business critical data as long as you're following their guidance. GCP though have not proven it, their data loss news is still fresh in my mind.
Were you talking about this incidence? https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...
I am currently evaluating between GCP and AWS right now.
I read the article and it seems that, that thing happened because their account got deleted and here is something from the article you linked
Google Cloud is supposed to have safeguards that don't allow account deletion, but none of them worked apparently, and the only option was a restore from a separate cloud provider (shoutout to the hero at UniSuper who chose a multi-cloud solution).
If you are working with really important software, please follow the 3-2-1 EVEN with cloud providers I suppose if you genuinely want ABSOLUTE guarantee I suppose, but it depends on how important the data is I suppose for the prices.
I have thought about using some cheap like backblaze and wasabi and others for the 3-2-1 for backups I suppose I am not sure but I do think that this incident was definitely a bit interesting to read into and I will read more about it, I do remember it from kevin fang's video but this article is seriously good and I will read it later, bookmarked.
On the Microsoft side CVE-2025–55241 is still pretty recent.
What a lame excuse. “The G-Drive’s structure did not allow for backups” is a blatant lie. It’s code for, “I don’t value other employees’ time and efforts enough to figure out a reliable backup system; I have better things to do.”
Whoever made this excuse should be demoted to a journeyman ops engineer. Firing would be too good for them.
It could be accurate. Let’s say, for whatever reason, it is.
Ok.
Then it wasn’t a workable design.
The idea of “backup sites” has existed forever. The fact you use the word “cloud” to describe your personal collection of servers doesn’t suddenly mean you don’t need backups in a separate physical site.
If the government mandates its use, it should have a hot site at a minimum. Even without that a physical backup in a separate physical location in case of fire/attack/tsunami/large band of hungry squirrels is a total must-have.
However it was decided that not having that was OK, that decision was negligence.
You could be right, but it could also be a bad summary or bad translation.
We shouldn't rush to judgement.
> The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information
They were still right though: it's absolutely clear without an ounce of doubt that whatever you put on an US cloud is being accessible by the US government, who can also decide to sanction you and deprive you from your ability to access the data yourself.
Not having backups is entirely retarded, but also completely orthogonal.
The U.S. Government can’t decrypt data for which it does not possess the key (assuming the encryption used is good).
In theory. I'm very much happier to have my encrypted data also not be available to adversaries.
Yeah let’s fax all government data to the Trump administration.
> However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost.
Yikes. You'd think they would at least have one redundant copy of it all.
> erasing work files saved individually by some 750,000 civil servants
> 30 gigabytes of storage per person
That's 22,500 terabytes, about 50 Backblaze storage pods.
Or even just mirrored locally.
It's even worse. According to other articles [1], the total data of "G drive" was 858 TB.
It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month).
They did have backup of the data ... in the same server room that burned down [2].
[1] https://www.hankyung.com/article/2025100115651
[2] https://www.hani.co.kr/arti/area/area_general/1221873.html
(both in Korean)
I made an 840TB storage server last month for $15,000.
How does this even make sense business wise for AWS?
Is their cost per unit so low?
This is just the storage cost. That is they will keep your data on their servers, nothing more.
Now if you want to do something with the data, that's where you need to hold your wallet. Either you get their compute ($$$ for Amazon) or you send it to your data centre (egress means $$$ for Amazon).
When you start to do math, hard drive are cheap when you go for capacity and not performance.
0.00099*1000 is 0.99. So about 12$ a year. Now extrapolate something like 5 year period or 10 year period. And you get to 60 to 120$ for TB. Even at 3 to 5x redundancy those numbers start to add up.
S3 does not spend 3x drives to provide redundancy. Probably 20% more drives or something like that. They split data to chunks and use erasure coding to store them in multiple drives with little overhead.
wait, can you elaborate on how this works?
They charge little for storage and upload, but download, so getting your data back, is pricey.
Mate, this is better than an entire nation's data getting burned.
Yes its pricey but possible.
Now its literally impossible.
I think that AWS Glacier at that scale should be the thing preferred as they had their own in house data too but they still should've wanted an external backup and they are literally by the govt. so they of all people shouldn't worry about prices.
Have secure encrypted backups in aws and other possibilities too and try to create a system depending on how important the treat model is in the sense that absolutely filter out THE MOST important stuff out of those databases but that would require them to label it which I suppose would make them gather even more attention to somehow exfiltrate / send them to things like north korea/china so its definitely a mixed bag.
my question as I said multiple times, why didn't they build a backup in south korea only and used some other datacentre in south korea only as the backup to not have to worry about encryption thing but I don't really know and imo it would make more sense for them to actually have a backup in aws and not worry about encryption personally since I find the tangents of breaking encryption a bit unreasonable since if that's the case, then all bets are off and the servers would get hacked too and that was the point of phrack with the advanced persistent threat and so much more...
are we all forgetting that intel has a proprietory os minix running in the most privileged state which can even take java bytecode through net and execute it and its all proprietory. That is a bigger security threat model personally to me if they indeed are using that which I suppose they might be using.
It's expensive if you calculate what it would cost for a third party to compete with. Or see e.g. this graph from a recent HN submission: https://si.inc/posts/the-heap/#the-cost-breakdown-cloud-alte...
Couldn’t even be bothered to do a basic 3-2-1! Wow
Did you expect government IT in a hierarchical respect-your-superiors-even-when-wrong society to be competent?
South Korea isn't some sort of backwards nation and I'm sure it's chaebols share the same culture.
Having had unfortunate encounters with government IT in other countries I can bet that the root cause wasn't the national culture. It was the internal culture of "I want to do the same exact same thing I've always done until the day I retire."
Absent outside pressure, civil services across the word tend advance scientifically - one funeral (or retirement) at a time.
I mean...I feel you but holy hell dude. Nothing? Boggles the mind.
Edit: my bad backups in the room is something, somehow just forgot about that part
It wasn’t nothing. They had backups, according to yongjik above.
Do backups in the same room count as backups?
That's unfortunate.
It's incompetent really.
No. Fortuna had nothing to do with this, this is called bad planning.
You're assuming average worker utilized the full 30G of storage. More likely average was at like 0.3G.
Article comments aside, it is entirely unclear to me whether or not there was no backups. Certainly no "external" backups, but potentially "internal" backups. My thinking is that not actually allowing backups and forcing all data there creates a prime target for the PRK folks right? I've been in low level national defense meetings about security where things like "you cannot backup off site" are discussed but there are often fire vaults[1] on site which are designed to withstand destruction of the facility by explosive force (aka a bomb) or fire or flood Etc.
That said, people do make bad calls, and this would be an epically bad one, if they really don't have any form of backup.
[1] These days creating such a facility for archiving an exabyte of essentially write mostly data are quite feasible. See this paper from nearly 20 years ago: https://research.ibm.com/publications/ibm-intelligent-bricks...
Funny, because the same thing happened in Nepal a few weeks ago. Protestors/rioters burned some government buildings, along with the tech infrastructure within them, so now almost all electronic data is gone.
Would this have been any different if these documents were stored non-electronically though? I understand that the whole point of electronic data is that it can be backed up, but if the alternative were simply an analog system then it would have fared no better.
Paper records are usually distributed both by agency and by locality.
It would have been better if storage was distributed.
Anti authoritarian patriots?
Happened in Bladerunner too
And Fight Club
Saw a few days ago that the application site for the GKS, the most important scholarship for international students in Korea, went offline for multiple days, surprising to hear that they really lost all of the data though. Great opportunity to build a better website now?
But yeah it's a big problem in Korea right now, lots of important information just vanished, many are talking about it.
Must have been a program without much trickle down into gov tech
A lot of folks are arguing that the real problem is that they refused to use US cloud providers. No, that's not the issue. It's a perfectly reasonable choice to build your own storage infrastructure if it is needed.
But the problem is they sacrificed "Availability" in pursuit of security and privacy. Losing your data to natural and man-made disasters is one of the biggest risks facing any storage infrastructure. Any system that cannot protect your data against those should never be deployed.
"The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups."
This is not a surprise to them. They had knowingly accepted the risk of infrastructure being destroyed by natural and man-made disasters. I mean, WTF!
The lack of backups makes my blood boil. However, from my own experience, I want to know more before I assign blame.
The very first "computer guy" job I had starting in about 1990/1991, my mentor gave me a piece of advice that I remember to this day: "Your job is to make sure the backups are working; everything else is gravy."
While I worked in that job, we outgrew the tape backup system we were using, so I started replicating critical data between our two sites (using 14400 bps Shiva NetModems), and every month I'd write a memo requesting a working backup system and explaining the situation. Business was too cheap to buy it.
We had a hard drive failure on one of our servers, I requested permission to invalidate the drive's warranty because I was pretty sure it was a bad bearing; I got it working for a few weeks by opening the case and spinning the platter with my finger to get it started. I made sure a manager was present so that they could understand how wack the situation was- they bought me a new drive but not the extras that I asked for, in order to mirror.
After I left that job, a friend of mine called me a month later and told me that they had a server failure and were trying to blame the lack of backups on me; fortunately my successor found my stack of memos.
Some more details in this article: https://www.chosun.com/english/national-en/2025/10/02/FPWGFS...
> The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets.
This attempt at putting it in perspective makes me wonder what would put it in perspective. "100M sets of harry potter novels" would be one step in the right direction, but nobody can imagine 100M of anything either. Something like "a million movies" wouldn't work because they are very different from text media in terms of how much information is in one, even if the bulk of the data is likely media. It's an interesting problem even if this article's attempt is so bad it's almost funny
Good article otherwise though, indeed a lot more detail than the OP. It should probably replace the submission. Edit: dang was 1 minute faster than me :)
Thanks! we've added that link to the toptext as well
While I am sure a huge portion of valuable work will be lost, I am smirking thinking of management making a call, "So, if there is any shadow IT who has been running mirror databases of valuable infrastructure, we would have a no questions asked policy on sharing that right now".
I know that I have had to keep informal copies of valuable systems because the real source of truth is continually patched,offline,churn,whatever.
> The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups.
This is why I don't really want to run my own cloud :)
Actually testing the backups is boring.
That said, ones the flames are out, they might actually be able to recover some of it.
Testing backups is boring. If you want exciting, test restores!
Hm, care to elaborate. I kinda liked this idea even though I know that it shouldn't make much sense but still lol, would this have any benefits over testing backups other than the excitement lol
I like to think that at least one worker was loafing on a project that was due the next day and there was no way it was going to get done. Their job was riding on it. They got drunk to embrace the doom that faces them, only to wake up with this news. Free to loaf another day!
just his luck
> There is a cert and private key for rc.kt.co.kr, South Korea Telecom's Remote Control Service. It runs remote support backend from https://www.rsupport.com. Kim may have access to any company that Korea Telecom was providing remote support for.
It's hard to believe this happened. South Korea has tech giants like Samsung, and yet this is how the government runs? Is the US government any better?
Software and information technology in Korea just sucks.
buttons are jpegs/gifs, everything is on Java EE and on vulnerable old webservers etc... A lot of government stuff supports only Internet Explorer even though it's long dead
Remember Log4j vulnerability? A lot of the Korea governmental sites weren't affected because the Java version was too old :)
Don't even get me started on ActiveX.
The first thing that comes to mind when I think of the South Korean government is the storied tradition of physical confrontation in their parliament along with more than a few viral videos of brawls and such over the years. It used to be better in the US, but with the intensity of discord in our government lately, I don't think anyone really knows anymore.
> The first thing that comes to mind when I think of the South Korean government is the storied tradition of physical confrontation in their parliament along with more than a few viral videos of brawls and such over the years
You're thinking of Taiwan, not South Korea.
No South Korea has the same thing. It doesn't happen yearly but has happened quite a bit. We lovingly call it parliament siege raid.
https://m.blog.naver.com/gard7251/221339784832 (a random blog with gifs)
South Korean IT seemed to be stuck in 2007 just not too long ago, would be surprised if it has changed much in the last few years. Do the websites still require you to use internet explorer?
Yes. The US government requires offsite backups .
They also require routine testing distaster recovery plans.
I participated in so many different programs over the years with those tests.
Tests that would roll over to facilities across the country
Our incompetence in the US is much more distributed. It wouldn't surprise me if the same kind of data isn't backed up, but at least it's dozens of separate federal agencies not-backing up their data in different physical places.
The US government still relies heavily on physical records.
Didn't Elon shut that down?
[0]: https://www.cnbc.com/2025/02/13/company-ripped-by-elon-musk-...
Why is there a "still" in there?
Samsung's software is generally terrible; they're decent at hardware, not software.
I was going to say, Samsung anything immediately makes me assume the software is awful. With a dose of zero privacy, cloud enabled door-knob or something.
Well, Elon has a recent copy of everything at least.
This is literally comic. The plot of the live action comic book movie "Danger: Diabolik" [1] has a segment where the a country's tax records are destroyed, thus making it impossible for the government to collect taxes from its citizens.
Meanwhile, Estonia has a "data embassy" in Luxembourg: https://e-estonia.com/solutions/e-governance/data-embassy/
TL;DR: Estonia operates a Tier 4 (highest security) data center in Luxembourg with diplomatic immunity. Can actively run critical government services in real-time, not just backups.
This is because everything is in digital form. Essentially all government systems are digital-first, and for the citizen, often digital-only. If the data is lost, there may be no paper records to restore everything from land registry, business registry (operating agreements, ownership records), etc.
Without an out-of-country backup, a reversion to previous statuses means the country is lost (Estonia has been occupied a lot). With it, much of the government can continue to function, as an expat government until freedom and independence is restored.
That is absolutely delightful. Estonia is just _good_ at this stuff. Admirable.
> Estonia follows the “once-only” principle: citizens provide their data just once, and government agencies re-use it securely. The next step is proactive services—where the government initiates service delivery based on existing data, without waiting for a citizen’s request.
I wish the same concept was in Canada as well. You absolutely have to resubmit all your information every time you do a request. On top of that, federal government agencies still mail each other the information, so what usually can be done in 1 day takes a whole month to process, assuming the mail post isn't on strike (spoiler: they are now).
I think Canada is one of the worst countries in efficiency and useless bureaucracy among 1st world countries.
This comment is in some way more interesting than the topic of the article.
Definitely. Especially when considering that there were 95 other systems in this datacentre which do have backups and
> The actual number of users is about 17% of all central government officials
Far from all, and they're not sure what's recoverable yet ("“It’s difficult to determine exactly what data has been lost.”")
Which is not to say that it's not big news ("the damage to small business owners who have entered amounts to 12.6 billion Korean won.” The ‘National Happiness Card,’ used for paying childcare fees, etc., is still ‘non-functional.’"), but to put it a bit in perspective and not just "all was lost" as the original submission basically stated
Quotes from https://www.chosun.com/english/national-en/2025/10/02/FPWGFS... as linked by u/layer8 elsewhere in this thread
Totally, backup disasters are a regular occurence (maybe not to the degree of negligence) but the Estonia DR is wild.
"secured against cyberattacks or crisis situations with KSI Blockchain technology"
hmmmm
I was in Korea during the Kakao fire incident and thought it was astounding that they had no failovers. However, I thought it'd be a wake up call!
I guess not.
S. Korea has the most backward infosec requirements. It's wild
Having just visited South Korea last year, one thing that sort of caught me off guard was the lack of Google Maps or other major direction system. I wasn’t aware but turns out anything considered “detailed mapping” infrastructure has to be ran stored and on South Korean soil, probably lots of other requirements. So you’re stuck with some shotty local mapping systems that are just bad.
There may be a point in time it made sense but high resolution detailed satellite imagery is plenty accessible and someone could put a road and basically planning structure atop it, especially a foreign nation wishing to invade or whatever they’re protecting against.
Some argument may be made that it would be a heavy lift for North Korea but I don’t buy it, incredibly inconvenient for tourists for no obvious reason.
Several other countries have similar requirements with regards to storing and serving maps locally.
If you take a moment to think about it, what's weird is that so many countries have simply resorted to relying on Google Maps for everyday mapping and navigation needs. This has become such a necessity nowadays that relying on a foreign private corporation for it sounds like a liability.
OSM is competitive with google maps in most places. Even if a person uses google maps, its inaccurate to say they "rely" on it when they could fail over to osm if google maps went down.
Local mapping efforts and allowing Google Maps to operate aren't mutually exclusive though. I don't see how it's weird that people can choose which map app they use.
Why didn't you use Kakao Maps or Naver Maps? They're not shotty and work just fine, even if you don't read Korean, you can quickly guess the UI based on the icons.
>So you’re stuck with some shotty local mapping systems that are just bad.
What made you think of them as bad? Could you be more specific? I use them almost daily and I find them very good.
I was there few months ago and I found them to be quite good too, both in coverage (shops, bus/metro networks) and accuracy. Obviously, not the apps I'm used to so & the language but otherwise, it was okay.
In my experience Open Street Maps was very good there.
I must say, at least for me personally when I hear about such levels of incompetence it rings alarm bells in my head making me think that maybe intentional malice was involved. Like someone higher up had set up the whole thing to happen in such a matter because there was a benefit to this happening we are unaware of. I think this belief maybe stems from lack of imagination on how really stupid humans can get.
Most people overestimate the prevalence of malice, und underestimate the prevalence of incompetence
I would love to know how a fire of this magnitude could happen in a modern data center.
Assume the PHB's who wouldn't spring for off-site backups (vs. excuses are "free") also wouldn't spring for fire walls, decently-trained staff, or other basics of physical security.
Often poor planning or just lithium based batteries far too close to the physical servers.
OVH's massive fire a couple of years ago in one of the most modern DC's at the time was a prime example of just how wrong it can go.
Allegedly from replacing batteries.
Well I'll be. Backup is a discipline to not be taken lightly by any organization, specially a government. Fire? This is backup 101: files should be backed up and copies should be physically apart to avoid losing data.
There are some in this threading pointing out that this would be handled by cloud providers. That bad - you can't hope for transparent backup, you need to actively have a discipline over it.
My fear is that our profession has become very amateurish over the past decade and a lot of people are vulnerable to this kind of threat.
Theoretically, they still have the primary copies (on each individual person's "cloud-enabled" device).
> The Ministry of the Interior and Safety also issued guidelines to each ministry stating, “All work materials should not be stored on office PCs but should be stored on the G-Drive.”
They very well might have only been saving to this storage system. It was probably mapped as a drive or shared folder on the PC.
Do they? It's not clear if this was two-way sync or access on-demand.
Like, I use Google Drive for Desktop but it only downloads the files I access. If I don't touch a file for a few days it's removed from my local cache.
One of the workers jumped off a building. [1] They say the person was not being investigated for the incident. But I can’t help but think he was a put under intense pressure to be scapegoat for how fucked up Korea can be in situations like this.
To be some context on Korea IT scene, you get pretty good pay and benefits if you work for a big product company, but will be treated like dogshit inside subcontracting hell if you work anywhere else.
[1] https://www.hani.co.kr/arti/society/society_general/1222145....
I'm sure they had dozens of process heavy cybersecurity committees producing hundreds if not thousands of powerpoints and word documents outlining procedures and best practices over the last decade.
There is this weird divide between the certified class of non-technical consultants and actual overworked and pushed to corner cut techs.
The data seems secure. No cyberthreat actors can access it now. Effective access control: check.
Ironically, see the phrack article someone linked above
Ironically many of those documents for procedures probably lived on that drive...
Here's a 2024 incident:
> "The outage also hit servers that host procedures meant to overcome such an outage... Company officials had no paper copies of backup procedures, one of the people added, leaving them unable to respond until power was restored."
https://www.reuters.com/technology/space/power-failed-spacex...
I dont know why but cant stop laughing. And the great thing is that they will get paid again to write the same thing.
You jest, but I once had a client who's IaC provisioning code was - you guessed it - stored on the very infrastructure which got destroyed.
Not even one redundant backup? That's unimaginable for me
My guess is someone somewhere is very satisfied that this data is now unrecoverable.
what's the point of a storage system with no back up?
It works fine as long as it doesn't break, and it's cheaper to buy than an equivalently sized system that does have back ups.
Isn't that self-evident? Do you have two microwaves from different batches, regularly tested, solely for the eventuality that one breaks? Systems work fine until some (unlikely) risk manifests...
Idk if this sounds like I'm against backups, I'm not, I'm just surprised by the question
Surely there must be something that's missing in translation? This feels like it simply can't be right.
I agree. No automated fire suppression system for critical infrastructure with no backup?
That may not be a perfect answer. One issue with fire suppression systems and spinning rust drives is that the pressure change etc. from the system can also ‘suppress’ the glass platters in drives as well.
That's why the top-security DCs that my employer operates have large quantities of Nitrogen stored, and use that slightly lower the O2 saturation of the air in the case of fire.
Yes, it's fucking expensive, that's one of the reason you pay more for a VM (or colocation) than at Hetzner or OVH. But I'm also pretty confident that single fire wouldn't destroy all hard drives in that IT space.
Reminds me of the classic video[1] showing how shouting at the harddrives make them go slower.
I'd be interested in if you can even use dry fire suppression on the 5th floor of a building.
Battery fire is impossible to suppress.
That's why in high-quality DCs, battery backup is in a separate room with good fire isolation from the IT space.
Yes, the servers still have some small batteries on their mainboards etc, but it's not too bad.
Much harder, but not impossible.
Because it was arson, not an accident
Arson? Sounds increasingly like espionage.
Are we talking about actual portable thunderbolt 3 connected RAID 5 G-drive arrays with between 70 and 160TB of storage per array? We use that for film shoots to dump TB of raw footage. That G-Drive?? The math checks at 30GB for around 3000 users on a single RAID5 array. This would be truly hilarious if true.
A management / risk issue and NOT and engineering one.
There are two types of people: those who do backups, and those who will do backups.
In a world where data centers burn and cables get severed physically, the entire landscape of tradeoffs is different.
after the kakao fire incident and now this i struggle to understand how they got so advanced in other areas. this is like amateur hour level shit.
It is the same in Japan. They are really good for hardware and other "physical" engineering disciplines, but they are terrible when it comes to software and general IT stuff.
Seriously, I work here as an IT guy and I can't stop wondering how they could become so advance in other areas and stay so backwards in anything software-related except videogames.
In my twenties I worked for a "company" in Mexico that was the official QNX ditribuitor for Mexico and LatAm. I guess the only reason was that Mexico City's Metro used QNX, and every year they bought a new license, I don't know why. We also did a couple of sales in Colombia I think, but was a complete shit show. We really just sent them the software by mail, and they had all sorts of issues getting it out of customs. I did get to go to a QNX training in Canada, which was really cool. Never got to use it though.
I think you meant to post this comment here: https://news.ycombinator.com/item?id=45481892
This is wild. Wilder would be to see that the government runs the same after the loss.
In 2025 data storage used by nation states, exposed to the internet, has no backups.
No offsite backups. No onsite backups. No usb drives laying around unsecure in a closet. Nothing.
What?
I wonder how many IT professionals were begging some incompetent upper management official to do this the right way, but were ignored daily. You'd think there would be concrete policies to prevent these things...
If I worked there I'd have had a hard time believing there were really no backups. Governments can be very nebulous.
> all documents be stored exclusively on G-Drive
Does G-Drive mean Google Drive, or "the drive you see as G:"?
If this is Google Drive, what they had locally were just pointers (for native Google Drive docs), or synchronized documents.
If this means the letter a network disk storage system was mapped to, this is a weird way of presenting the problem (I am typing on the black keyboard and the wooden table, so that you know)
G-drive was simply the name of the storage system
The name G-Drive is said to be derived from the word ‘government’.
It's now derived from the word 'gone'
It's an X-Drive now
'Gone' up in smoke
This is extraordinarily loony shit. Someone designed a system like this without backups? Someone authorized it's use? Someone didn't scream and yell that this was bat and apeshit wacky level crazy? Since 2018? Christ almighty.
This is amazingly incompetent because all the major enterprise storage arrays support automatic replication to remote arrays.
"A source from the Ministry of the Interior and Safety said, “The G-Drive couldn’t have a backup system due to its large capacity” "
:facepalm:
I was the principal consultant at a subcontractor to a contractor for a large state government IT consolidation project, working on (among other things) the data centre design. This included the storage system.
I noticed that someone had daisy-chained petabytes of disk through relatively slow ports and hadn’t enabled the site-to-site replication that they had the hardware for! They had the dark fibre, the long-range SFPs, they even licensed the HA replication feature from the storage array vendor.
I figured that in a disaster just like this, the time to recover from the tape backups — assuming they were rotated off site, which might not have been the case — would have been six to eight weeks minimum, during which a huge chunk of the government would be down. A war might be less disruptive.
I raised a stink and insisted that the drives be rearranged with higher bandwidth and that the site-to-site replication be turned on.
I was a screamed at. I was called unprofessional. “Not a team player.” Several people tried to get me fired.
At one point this all culminated in a meeting where the lead architect stood up in front of dozens of people and calmly told everyone to understand one critical aspect of his beautiful design: No hardware replication!!!
(Remember: they had paid for hardware replication! The kit had arrived! The licenses were installed!)
I was younger and brave enough to put my hand up and ask “why?”
The screeched reply was: The on-prem architecture must be “cloud compatible”. To clarify: He thought that hardware-replicated data couldn’t be replicated to the cloud in the future.
This was some of the dumbest shit I had ever heard in my life, but there you go: decision made.
This. This is how disasters like the one in South Korea happen.
In private organisations you get competent shouty people at the top insisting on a job done right. In government you get incompetent shouty people insisting that the job gets done wrong.
> In private organisations you get competent shouty people at the top insisting on a job done right. In government you get incompetent shouty people insisting that the job gets done wrong.
Great post and story but this conclusion is questionable. These kinds of incompetences or misaligned incentives absolutely happen in private organisations as well.
Much more rarely in my experience, having been at both kinds of organisations.
There’s a sort-of “gradient descent” optimisation in private organisations, established by the profit motive and the competitors nipping at their heels. There’s no such gradient in government, it’s just “flat”. Promotions hence have a much weaker correlation with competence and a stronger correlation with nepotism, political skill, and willingness to participate in corruption.
I’ve worked with may senior leaders in all kinds of organisations, but only in government will you find someone who is functionally illiterate and innumerate in a position of significant power.
Obviously this is just a statistical bias, so there’s overlap and outliers. Large, established monopoly corporations can be nigh indistinguishable from a government agency.
What info needed to be destroyed and who did it implicate?
Does anyone have an understanding of what the impact will be of this, i.e., what kind of government impact scale and type of data are we talking about here?
Is this going to have a real impact in the near term? What kind of data are we’re talking about being permanently lost?
> no back-ups
Top fucking kek. What were they expecting to happen? Like, really? What were they thinking?
I mean ... was making backups on the backlog at least? Can they at least point to the work item that was going to get done soonish?
It got pushed a couple sprints and we've got it on the plan for next quarter as long as no new features come in before then.
If it wasn't it most certainly is now
Why, there's nothing left to backup?
May be a “fast follow”? Right after launch of the “MVP”?
[stub for offtopicness]
Copy/paste:
7 things all kids need to hear
1 I love you
2 I'm proud of you
3 I'm sorry
4 I forgive you
5 I'm listening
6 RAID is not backup. Make offsite backups. Verify backup. Find out restore time. Otherwise, you got what we call Schrödinger backup
7 You've got what it takes
Brilliant.
This deserves its own HN submission. I submitted it but it was flagged due to the title.
Thank you for sharing it on HN.
Technically the data is still in the cloud
I've been putting off a cloud to cloud migration, but apparently it can be done in hours?
You can use accelerants to speed up migration
The egress cost is gonna be a doozie though!
one of many fires to fight in such a fast scenario
Lossy upload though
No information is lost: https://en.wikipedia.org/wiki/No-hiding_theorem#:~:text=info...
Lossy download, no?
Unfortunately, the algorithm to unhash it is written in smoke signals
Cloud of smoke, amirite.
The cloud has materialized
>the G-Drive’s structure did not allow for external backups.
ah the so called schrodingers drive. It's there unless you try to copy it
repeat after me:
multiple copies; multiple locations; multiple formats.
Watching Mr. Robot and seeing the burned batteries the same time...
Good example of a Technology trap
Well that works out doesn’t it? Saves them from discovery.
We will learn nothing
Yikes. That is a nightmare scenario.
Now imagine they had a CBDC.
I thought most liberal governments gave up on those.
This is the reason the 3, 2, 1 rule for backing up exists.
I thought clouds could not burn (:
They are clouds of smoke to begin with. The smoke from the joints of those who believed that storing their data somewhere out of their control was a good idea!
"The day the cloud went up in smoke"
No problem — I'm sure their Supremely nice Leader up north kept a backup. He's thoughtful like that...
They might be singing this song now. (To the tune of 'Yesterday' from the Beatles).
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There’s not half the files there used to be,
And there’s a deadline
hanging over me.
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now my data’s gone
and I long for yesterday-ay-ay-ay.
Yesterday,
The need for back-ups seemed so far away.
Thought all my data was here to stay,
Now I believe in yesterday.For the German enjoyers among us I recommend also this old song: https://www.youtube.com/watch?v=jN5mICXIG9M
Thanks! mmd.
LOL
nice
[flagged]
The Egyptians send their condolences.
Has there been a more recent event, or are you referring to Alexandria?
https://en.wikipedia.org/wiki/Library_of_Alexandria#Burning_...
the destruction of the library of alexandria is under dispute.
I think Alexandria.
touché
I was smirking at this until I remembered that I have just one USB stick as my 'backup'. And that was made a long time ago.
Recently I have been thinking about whether we actually need governments, nation states and all of the hubris that goes with it such as new media. Technically this means 'anarchism' with everyone running riot and chaos. But, that is just a big fear, however, the more I think through the 'no government' idea, the less ludicrous it sounds. Much can be devolved to local government, and so much else isn't really needed.
South Korea's government have kind-of deleted themselves and my suspicion is that, although a bad day for some, life will go on and everything will be just fine. In time some might even be relieved that they don't have this vast data store any more. Regardless, it is an interesting story regarding my thoughts regarding the benefits of no government.
Government is whatever has a monopoly on violence in the area you happen to live. Maybe it’s the South Korean government. Maybe it’s a guy down the street. Whatever the case, it’ll be there.
Mindblowing. Took a walk. All I can say is that if business continues "as usual" and the economy and public services continue largely unaffected then either there were local copies of critical documents, or you can fire a lot of those workers; either one of those ways the "stress test" was a success.
How do you come to the conclusion that because things work without certain documents that you can start laying off workers?
>or you can fire a lot of those workers
Sometimes things can seem to run smoothly for years when neglected... until they suddenly no longer run smoothly!
Long term damage, and risk are two things that don't show up with a test like this. Also, often why things go forward is just momentum, built from the past.
Yeah you can do the same with your car too - just gradually remove parts and see what's really necessary. Seatbelts, horn, rear doors? Gone. Think of the efficiency!
“Final reports and official records submitted to the government are also stored in OnNara, so this is not a total loss”.
Surely having human-resource backups will also help with disaster recovery