Ea loves using generic systems across all their games. When poking around at Madden I found they have a common backend called blaze that has generic web and tcp endpoints. We built out a tool to call these endpoints (having to upload xml) and only later found out that every time we made the call it was crashing their servers but since we were grabbing a new server each request we were crashing all of their madden servers one by one. They ended up building an API to discourage people poking around
Blaze is the name of the C++ framework/service to build custom backend for online games. It allows game team to developp online features in a standard way, it's backed by MySQL.
From what I remember you need roughly one Blaze instance for 5k/10k players.
So, like any sane person would, I overnighted an Xbox, installed Battlefield 2042, and waited for the moment of truth...
I was in!
When I read the line, I thought "that's the spirit!". Kudos to him.
I enjoyed the detailed explanation of how he moved from point to point. I imagine it wasn't as straightforward as is laid out in a blog post
It would be interesting to see what I imagine to be the reams of notes from one of these to show how much time and effort it takes to perform this kind of attack.
For anyone who's enjoyed reading this, there's plenty more to read about on the bug bounty platforms such as HackerOne's "Hacktivity": https://hackerone.com/hacktivity
This is super cool!
What's most wild about all of this to me is that EA has claimed for years a "technical impossibility" to unlink an existing Xbox account and re-link with a new one. (See https://www.reddit.com/r/XboxGamePass/comments/12gsy4i/ea_xb... and many other forum posts on EA). I ran into this wall and after spending hours on support calls with EA they were unable to link a very old Xbox account I had, meaning I can't login to any EA games on Xbox, making the majority of them unplayable on the platform. Yet, here, we see, it is very much possible.
Reminds me of in 2004-2005, I had my typical Hotmail account with 4MB storage, but Microsoft was rolling out a free upgrade for everyone to 250MB. For some reason they were taking an incredibly long time with my account, and I emailed support several times over a year or two about it. Each time they assured me that Microsoft was upgrading accounts as fast as they could, but it was just such a big job that it took years.
Eventually I read on a forum somewhere[1] that you could partly trick the system by temporarily closing your account and re-opening it, which got you a slightly larger 25MB. But still not the promised 250.
All this 2-4MB for existing accounts, 25MB for new accounts, and years-long rollout to 250MB gave the impression that finding spare storage was a huge struggle for Microsoft. Then a few months later they were having to compete with Gmail and they decided that everyone should get 2GB, which was rolled out to every Hotmail account including mine all at once! I can only assume aliens landed and delivered a UFO full of hard drives.
[1] Here's an example of an old forum post about the trick - complete with reply praising the brand new GMail: https://bimmersport.co.nz/topic/5232-hotmail-upgrade-2mb-to-...
Most likely they were in the process of adding this feature to fix your issue when it was sadly exploited before they could announce it
Hahaha!
Probably it’s technically impossible for the customer support to do it.
Likely a bureaucratic problem and much easier for these social media PR teams to pass it off as a system issue
> Unfortunately game entitlements, friends, and game save data for newer cross-platform games like Battlefield 2042 are stored in the EA account itself, not the persona, so that data isn't transferred.
This attack shows us it's possible to change this link and play games, but it's impossible to say what other effects this would have outside of the easily testable scenario laid out in the article. Maybe this change of link invalidates data stored in a billing system, messes up a monthly report that goes to Microsoft's XBox division or causes an internal admin page to crash on load.
I'm not excusing EA but I have worked on plenty of complicated microservice systems and it's not always so straight forward to change the structure of data in one place.
EA level 1 support probably doesn't have access to these controls. They probably stuck whatever tools EA has given them :/
Would've been fun to ban every account and hope they didn't have DB backups
I'd love to see this happen to every billion dollar company that doesn't have a bug bounty program. Offering zero incentive for reporting vulnerability just encourages hackers to exploit it for their own advantage or to wreak havoc.
As a paying customer, I expect better from these companies and personally wouldn't blame the hackers for exploiting their findings if no program exists.
Well the Federal Government certainly wouldn't agree with you. Give it go though!
The Federal Government? Thank goodness these companies only operate in one country. Or we've finally succeeded in uniting under one singular world government
In case you haven't noticed, the FBI charges hackers across the world on a frequent basis. And you should fear them regardless of what country you're in if you're going to be messing with American companies. I've worked at companies where the FBI caught our engineers that were offshore stealing IP. The Company didn't have a clue, they are watching anything and everything that concerns American interest and yes there are no jurisdictions/borders stopping them, outside of Russia, Iran and NK ofc.
Well it might be fun for a sec.
They definately do have backups, no-one is storing 400mm records on a single machine and ultimately you'd just take them offline for an afternoon and then spend 15 years in a federal prison
I thought about this... What would be the outcome here do you think? Ie if this guy didn't report and did decide to mess around for real? Could he have been tracked? Would EA be down for weeks?!
I mean, Kevin Mitnick spent time in jail: https://en.wikipedia.org/wiki/Kevin_Mitnick
I wouldn't mess around with this stuff myself.
Me neither, but would it have been easy to trace him? I mean if he was going to use this for bad, I would assume he would have waited a month or so then done it all via a VPN etc. But point being he _could_ have done this and to be honest for all we know, someone else _has_ been abusing it until it was patched...
VPN is used to bypass regional restrictions.
The VPN provider will share information if an active investigation is underway.
Buy VPS with Monero.
Deploy image containing Tor router and hidden service onion config.
Do above as many times as one needs to feel comfortable.
Use VPSes as proxies intermixed with VPN and Tor legs.
What's lil officer Timmy at CISA gon' do? Netflow you? LOL!
the VPN provider can try to share whatever information but most reputable providers would have nothing
You can pay for a VPN using monero or cash and then connect to the VPN from TOR - VPN provider doesn’t need to know anything
Please don't do this. Most people aren't going to pull this off correctly and if they think this makes it safe for them to go around messing with companies they could really get themselves in a lot of trouble.
If I were to mess around with stuff, and im not. The only way I'm doing that is with a used laptop of Craigslist or whatever and cafe with no cameras, even then idk.
Qubes OS.
Done.
Sorry you will never catch me using hardware to commit a crime that's ever connected to my home or work networks or ever in anyway been associated with my name. IDC about the OS. But I'm not going to commit a crime either so I don't worry about this FYI
If you're already using Tor, what's the point of having a VPN too?
There isn’t a point. It is useless security theatre for those who do not understand how tor works.
Only "benefit" would be that TOR might be blocked but the VPN isnt.
Way more fun to enable every game for every account. Literally. Limited horizons.
And this is why the world has turned against tech...
Because the first thought (at least, the highest rated post right now) it that it would have been "fun" to hurt millions of people to teach the company they were doing business with a lesson.
Sometimes I wonder how it feels to be an engineer at such a company, having all your private APIs, weird bugs and dirty laundry aired in a public breach disclosure.
Though it's likely in a case like this, no single person was responsible for the vulnerability. Probably 5 or 6 different teams owned different parts of what he exploited (which is probably why the exploit existed in the first place - big complex system where everyone only understands their tiny piece of it).
On a team you are emotionally (or maybe even just financially) invested in it feels bad, but when I was at EA they almost worked hard to make it hard to become emotionally invested.
At a company the size of EA almost certainly this will be used to play politics and even if it hurts the company as a whole people will use it to have larger control over the now smaller company.
Bad, especially if you have no control because you don't work in that department but you know you could do better than that department.
In a such large corpo no one gives a shit about it. It's just a job, to get a paycheck. They all are expendable resources, why to be invested emotionally into the job?
I would have a pit in my stomach if I read a post like that knowing I implemented those APIs
What if you implemented the APIs but
- someone else proxied your API to the public
- someone else leaked credentials you assigned them in the public code of a game
As someone working on a team that publishes APIs to the greater large organization, you can't trust other people. People be doing wild things.
I would hope that my employer had a postmortem culture that encouraged looking into every point of failure and identifying process changes that will prevent a repeat of the incident. Instead of pointing the finger at Team X who messed up and/or just "blaming hackers" and continuing on with your defective processes.
Five or six teams is probably an underestimate if they had glued different games into the same system. EA has made a ton of games with online features, bought companies, etc.
The company I work for now likely has weaker security simply from having glued various acquisitions in. We have API endpoints specific to some of them.
Electronic Arts has been, and still is, vulnerable from its Xbox gateway.
I legally/ethically/mentally cannot read this article but if its not related, there is more work to do.
Not that anyone should do it for EA, but for the collective they've swindled.
> It's also disappointing that EA has yet to start a bug bounty program. Without any real incentive to report vulnerabilities, I know people who have instead chosen to keep them to themselves. I would love to see EA follow the rest of the industry's lead here.
Does that mean the author got nothing for reporting this?
It's not true that they got nothing. There's always the possible threat of legal action against them for reporting the vulnerabilities.
I mean, after all, that's what we are all here for right? Fish and legal liability.
It's disappointing how many companies don't offer a bug bounty. I have a hoard of vulnerabilities I've found over the years just sitting in my head. It doesn't help that there are legal risks with reporting them & they can technically sue you to hell (EU/UK)
It's probably the result of some very backward-thinking rationale: "If we get hacked by the bad guys, our shareholders will point to these bounties and say 'wait, you're activetly paying people to hack you and now they did and you're going to have to write down and additional $X Million?'. " Execs afraid of having egg on their face, perhaps.
> Does that mean the author got nothing for reporting this?
Correct.
Is there a best practices guide somewhere on how to setup a bug bounty program?
I work in the field so it's hard to know what info you might be missing. To me it seems quite straightforward: you post to your website somewhere that you're happy to have people probe your technical security provided that they follow coordinated vulnerability disclosure (you'll want to flesh it out a tad more than this one sentence of course) and what kind of reward you're willing to hand out for what kind of bug and in which part of the scope. Any exclusions, such as that you won't pay out to young or old people or if you're born in the wrong country and got sanctioned or so, are also things you'll want to mention up front to prevent sour grapes afterwards
Perhaps I can answer a specific question or look for good pointers if you have a specific question about this?
Thanks! Any good examples?
Valve comes to mind: https://hackerone.com/valve?type=team
I'm not a bounty hunter myself, but trying to think beyond the big names I found the Dutch government's <https://english.ncsc.nl/contact/reporting-a-vulnerability-cv...> as an example that looks good to me. One point of improvement could be that they're not very concrete about any reward (or the lack thereof, also fine, but better to be up front). Some of the exclusions are also a bit broad, e.g. I'd still say XSS on a static site is worth fixing even if it's not a major risk, but I can understand where they're coming from when you consider there's thousands of websites run by the government. On the plus side, they give a clear timeline so you know they're going to pick it up in a timely manner, and they have practical guidelines on what (not) to do
Just remembered: One thing I didn't like about e.g. Google's report mechanism is that it basically required a Google account. There were instructions for if you don't have one, but they didn't work (probably outdated) so you just have to agree with the extremely broad blanket statement that is the Google privacy policy. That could be something to avoid if you're setting up a policy of your own: don't require agreeing to wholly unrelated terms; hackers (in the HN sense of the word) sometimes don't take very well to that
A good experience I had was with Threema (private/encrypted chat application like Wire or Signal). The report process consisted of just sending a service account a chat message (probably there's also other ways), which was nice and easy. My report turned out to be mostly invalid (the risk was real but my imagined fix was flawed and it turned out contact discovery is a hard problem) but their answer was quick and thorough, I was impressed that they didn't just brush it off like so many orgs do.
Being on something like Hackerone, like Valve and Keybase, has pros and cons. I'm probably just old but it feels odd to me to let direct threats to your organisation be handled by a third party, sometimes even having them triage and decide whether to inform your org of a claimed vulnerability at all (recent story on HN; probably it works fine in 99% of cases), as well as it being an instance of having to sign up for something unrelated when I just want to ping an email address with the steps to reproduce. On the other hand, it standardises the whole thing so you know where to find different things if you use it more than the sporadic amount I have. I also wonder if this attracts the beg bounty hunters who see potential easy money, based on that the orgs on Hackerone seem to take reports less seriously when you didn't invest a ton of time in developing an exploit, or if the causality is reversed (maybe they chose Hackerone because they already had too many beg reports, hoping to be able to use accounts' reputation as an indicator for triage)
Is it typical to just present findings and hope to get rewarded? What would the expected reward amount be in similar circumstances where they did pay up? Do companies pay more to prevent articles like this being published? Sorry if these are stupid questions - I know little about this area.
EA doesn't care. They definately should pay and I'd imagine this would be in the high 5 figures or more. Their customers don't care if their code is secure. 99% just want to play Madden.
From the article:
> I had found a way to obtain a privileged access token within the environment (a story for another day, but a certain game's executable had hardcoded credentials!), but I wasn't sure what I could do with it.
Can someone speak to this a bit more? I'm under the impression an executable binary shouldn't be easily read to find such credentials, and I don't know what else a game dev is supposed to do if their executable needs to authenticate itself with a remote server.
The credentials are stored as a string so you can search the binary for a pattern matching what the credential looks like and it will be in there somewhere.
In client server architecture, the client is always untrusted. An executable shouldn't need to authenticate itself to the server. The executable should authenticate as a user or account using details provided by the person.
In cases like telemetry these endpoints usually accept unauthenticated or lightly authenticated data and perform layers of validation to prevent abuse (and are usually write/append only)
> I'm under the impression an executable binary shouldn't be easily read to find such credentials
If the computer can read it, and you have full control of the computer, then you can read it. Physical access is game over. Even if they encrypt it and put the encryption key in an HSM (probably not possible on an arbitrary client's machine anyway), at some point the game is going to decrypt that string and put it in memory. Memory that you can read.
> I'm under the impression an executable binary shouldn't be easily read to find such credentials
Why would you assume that? binaries are perfectly easily readable on non-locked-down platforms.
You'd have to have a system where the executable is encrypted and a secure part of the CPU die handle decryption against a private key, and even then it'd probably be only a matter of time before someone delidded the chip to get the key.
> Why would you assume that?
I thought too highly of modern compiler string literal obfuscation.
Compilers are there to make things more efficient for the machine running the code. Obfuscating a string is the opposite of that. What they actually do much of the time is collect all the string literals into a contiguous pool so that their addresses are fixed and well-packed, providing efficiency at runtime.
It's actually very easy to find string literals in executables because of this, not hard.
Consider the string needs reversible obfuscation or it won't be usable. The only secure way is encryption but you'd need to properly secure the key (probably using some hardware facility that's physically locked down)
> modern compiler string literal obfuscation
the what now?
If they used any open or even popular compiler, then that wouldn't solve anything. Folks would have already figured out how it works, since such encoding would have to be deterministic.
What obfuscation? Do you think that is happening automatically? If you compile literals into your program they are sitting in the data section of your binary verbatim so they can be read directly once the binary itself is memory mapped.
If the program has access to the credential, and the program is running on your computer, you also have access to the credential no matter how they try to obfuscate it.
What the game dev is supposed to do is have an account system on their backend, and ask the player to enter their credentials in the game. The game can then identify itself as this player to the backend servers. That way any actions on the backend can be attributed to a particular player and you have a good basis to make security decisions on.
>I'm under the impression an executable binary shouldn't be easily read to find such credentials
It's hard but not impossible. It's more annoying than trying to extract strings out of a minified js file, but far from impossible. There are tools for it (eg. IDA), so you're not searching for credentials amongst anything that vaguely looks like a string.
>and I don't know what else a game dev is supposed to do if their executable needs to authenticate itself with a remote server.
The problem isn't that that the binary has hardcoded credentials, it's that the credentials are privileged.
The strings command is pretty old can do it if you're naive enough to embed a username and password into the game client.
The main thing is that its privileged - having a token shouldn't let you do anything besides say, report your game stats to a central server or enumerate the server lists, things like that.
TBF strings might not trivially show up the password if you took the most basic of provisions (a non-ascii password, not stored right next to the username separated by a \0), but most programmers likely wouldn't even bother with that.
Even then you can MITM if you have elevated access to the platform and can tinker with the certificate store.
Games like Pokemon Go use a highly obfuscated algorithm to sign requests which makes it much harder to actually use the key if you can retrieve it
What a deeply unserious company.
...and no bounty. I hope they were at least thanked! (great writeup thank you)
EA can get rekt. Their account suspension/banning process is obtuse and opaque. I bought a game, created an online account to play it with a friend, and was banned 2 days later (making my purchase useless.) I did absolutely nothing wrong and their team won't give any details other than "read the account agreement."
Well, fun story: you can just use this api to unban yourself :-D
I am not a lawyer but I bet a sane judge would have little sympathy for a CFAA claim against your own account
It's ridiculous EA didn't pay any bounty for this
BattleDash - "Here's XSS, Account Takeover, Ban Reversal, and a heads up before I publish it"
EA - "So here's $0."
If anyone is at EA, this man just saved the integrity of your entire empire, you might want to give him at least a token amount.
My experience with big companies is even if the whole IT security team thinks this is worthy of a bounty, and the team has plenty of budget they could allocate to it, the process of giving money to an individual is frequently so difficult to get through the bureaucratic purchase order system that it's basically impossible to do unless you are contractually obliged to pay.
Probably easier to hire them on as a consultant than "give them money"
EA is a famously horrible company. I don't think they care much about the "integrity of their empire" because their customers don't care.
When I was in college, I once found some bad exploits in the sims social on Facebook, the subsidiary (Playfish?) behind it asked me for my address and which console I owned and then unexpectedly sent me a huge number of games and goodies. It was great. Better than money, I think (I sold some of those games anyway).
I reached out to employees via unofficial channels. I'm sure if I had spoken to some exec I'd be in jail right now.
Until we see something like this,
The company is liable for $10 per hacked user minus 100X the bounty spend for that year.
Would it be legal to publish any future vulnerabilities without giving them heads up?
Wait this person didn’t get paid for finding this? No bug bounty? Seems like they could at least toss them some free credits or something…
4 months from report to remediation... absolutely pathetic.
This could have been exploited to just unban every account that has ever been banned. This guy would have made a fortune selling just that exploit to cheaters.
Selling the exploit? No. What you do is offer an unban API and charge $1 per call.
Then you're on the hook and the income dries up when they find out. Selling for cash up front means you got 90% of the law if the prosecution decides you've done something wrong and finds you in the first place (the use of exploits is commonly illegalised, and often indirectly the discovery or development, but not the knowledge or sale)
Not that I'd advise either course of action for the players' sake
You would be able to charge much more than $1 per call and the real $ wouldn't come from unbanning but banning instead.
Think about being able to empower a kid to ban anyone they want.
It would turn into chaos but I do not think such a service would be long lived as it would generate so many support tickets and issues that EA would start looking into how it was happening.
If someone was out to maximize chaos and not just make money, this is in all seriousness in the class of problems that someone intelligent could have used to all but destroy EA. You don't offer an API with targeted usage, and you sure don't ban everyone.
There's lots of fun ideas you can go for here, but just as one, suppose I spend a month banning accounts that haven't played much, but more than zero. Then go quiet for a couple of weeks. EA frontline support notices but if you play your cards right they don't put the pieces together and nobody is quite roused to investigate. Then you start up again, somewhat faster, spend a couple of days banning a good chunk of medium sized accounts. Then maybe at the end you ban the biggest accounts as quickly as you can.
Now the bannings are news. EA's PR is probably completely blown out by the crisis and starts saying contradictory things. (My guess is that initially they end up backing their right to ban people and releasing statements to the effect of how right they probably are; this is, in the end, a huge mistake on their part.) Gamers can be reliably expected to start a ton of rumors, take them in the worst way possible, and antagonize EA, and EA is pretty likely to make at least one class-A error in being antagonistic back. (The hackers could even supply some of the rumors and some bots to get them going, though I doubt it'll be necessary. The gamer community is pretty well primed to turn on EA.) A ton of people who are curious but figure this can't be affecting them because they hardly use the service log in and discover they've been banned despite not having done anything on EA in six months. The fire rises as they post to reddit and hundreds of people chime in with "WTF, me too!", even if it's only a small percentage of the total people who check.
Several days later, EA puts all the pieces together confidently enough to be sure that they can announce it's a hack. They're right. Nobody cares. Half of the gamer community doesn't even believe their defense.
It's hard to guess what the upper bound of damage is on this scenario.
I think you are right, banning would cause too many issues and be loud.
I think the real quiet $ maker would be stealing usernames instead.
Like if you wanted the EA gametag of jerf but someone else had it, you could steal it using OPs method if it was still unpatched. A pay service for this would be viable in low volume and on the EA side it would just look like the user did it.
The seller of service would have to implement some kind of checks to make sure for example they weren't stealing the username of a top streamer or etc which would bring heat.
You could turn it into a simple subscription based service: pay to stay (unbanned).
Pretty sure "price restructuring" (price increases) will be paid by most users (cost sunk fallacy).
Yeah, if Alice and Bob are at war, accept a huge payment by Alice to ban Bob, and then ask Bob a small recurrent fee to unban his account til the next payment.
Mafia style. The second part is called "pizzo".
$1 is not very ambitious when people have sometimes thousand of dollars worth of games :D
Not to mention mainstream cheats are going for $50+ a WEEK.
The timeline says that the initial report was 6/16 and the initial patches were 7/8 and 7/18.
It's not clear to me what was exploitable when.
Hmm, someone tried to login to my account few days ago
unrelated