I totally agree with the author. Sadly, I feel like that's not what the majority of LLM users tend to view LLMs. And it's definitely not what AI companies marketing.
> The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters
the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand. I believe that this lack of initial understanding of the user input is what can lead to taking LLM output as factual. If one side of the exchange knows nothing about the subject, the other side can use jargon and even present random facts or lossy facts which can almost guarantee to impress the other side.
> The way to solve this particular problem is to make a correct example available to it.
My question is how much effort would it take to make a correct example available for the LLM before it can output quality and useful data? If the effort I put in is more than what I would get in return, then I feel like it's best to write and reason it myself.
> the user will at least need to know something about the topic beforehand.
I used ChatGPT 5 over the weekend to double check dosing guidelines for a specific medication. "Provide dosage guidelines for medication [insert here]"
It spit back dosing guidelines that were an order of magnitude wrong (suggested 100mcg instead of 1mg). When I saw 100mcg, I was suspicious and said "I don't think that's right" and it quickly corrected itself and provided the correct dosing guidelines.
These are the kind of innocent errors that can be dangerous if users trust it blindly.
The main challenge is LLMs aren't able to gauge confidence in its answers, so it can't adjust how confidently it communicates information back to you. It's like compressing a photo, and a photographer wrongly saying "here's the best quality image I have!" - do you trust the photographer at their word, or do you challenge him to find a better quality image?
What if you had told it again that you don't think that's right? Would it have stuck to it's guns and went "oh, no, I am right here" or would it have backed down and said "Oh, silly me, you're right, here's the real dosage!" and give you again something wrong?
I do agree that to get the full usage out of an LLM you should have some familiarity with what you're asking about. If you didn't already have a sense of what a dosage is already, why wouldn't 100mcg be the right one?
I replied in the same thread "Are you sure that sounds like a low dose". It stuck to the (correct) recommendation in the 2nd response, but added in a few use cases for higher doses. So seems like it stuck to its guns for the most part.
For things like this, it would definitely be better for it to act more like a search engine and direct me to trustworthy sources for the information rather than try to provide the information directly.
Perhaps the absolute worst use-case for an LLM
With search and references, and without search and references are two different tools. They're supposed to be closer to the same thing, but are not. That isn't to say there's a guarantee of correctness with references, but in my experience, accuracy is better, and seeing unexpected references is helpful when confirming.
LANGUAGE model, not FACT model.
I gave LLM a list of python packages and asked it to give me their respective licenses. Obviously it got some of them wrong. I had to manually check with the package's pypi page.
I find if I force thinking mode and then force it to search the web it’s much better.
But at that point wouldn't it be easier to just search the web yourself? Obviously that has its pitfalls too, but I don't see how adding an LLM middleman adds any benefit.
For medication guidelines I'd just do a Google search. But sometimes I want 20 sources and a quick summary of them. Agent mode or deep research is so useful. Saves me so much time every day.
Not always, it can find stuff that is otherwise difficult for me and search engines have become much worse than 15-20 years ago.
Agree, I usually force thinking mode too. I actually like the "Thinking mini" option that was just released recently, good middle ground between getting an instant answer and waiting 1-2 minutes.
Using a LLM for medical research is just as dangerous as Googling it. Always ask your doctors!
I don’t disagree that you should use your doctor as your primary source for medical decision making, but I also think this is kind of an unrealistic take. I should also say that I’m not an AI hype bro. I think we’re a long ways off from true functional AGI and robot doctors.
I have good insurance and have a primary care doctor with whom I have good rapport. But I can’t talk to her every time I have a medical question—it can take weeks to just get a phone call! If I manage to get an appointment, it’s a 15 minute slot, and I have to try to remember all of the relevant info as we speed through possible diagnoses.
Using an llm not for diagnosis but to shape my knowledge means that my questions are better and more pointed, and I have a baseline understanding of the terminology. They’ll steer you wrong on the fine points, but they’ll also steer you _right_ on the general stuff in a way that Dr. Google doesn’t.
One other anecdote. My daughter went to the ER earlier this year with some concerning symptoms. The first panel of doctors dismissed it as normal childhood stuff and sent her home. It took 24 hours, a second visit, and an ambulance ride to a children’s hospital to get to the real cause. Meanwhile, I gave a comprehensive description of her symptoms and history to an llm to try to get a handle on what I should be asking the doctors, and it gave me some possible diagnoses—including a very rare one that turned out to be the cause. (Kid is doing great now). I’m still gonna take my kids to the doctor when they’re sick, of course, but I’m also going to use whatever tools I can to get a better sense of how to manage our health and how to interact with the medical system.
I always thought “ask your doctor” was included for liability reasons and not a thing that people actually could do.
I also have good insurance and a PCP. The idea that I could call them up just to ask “should I start doing this new exercise” or “how much aspirin for this sprained ankle?” is completely divorced from reality.
Yes, exactly this. I am an anxious, detail-focused person. I could call or message for every health-related question that comes to mind, but that would not be a good use of anyone’s time. My doctor is great, but she does not care about the minutiae of my health like I do, nor do I expect her to.
I think "ask your doctor" is for prescription meds since only said doctor can write prescriptions.
And "your doctor" is actually "any doctor that is willing to write you a prescription for our medicine".
> it can take weeks to just get a phone call
> If I manage to get an appointment, it’s a 15 minute slot
I'm sorry that this is what "good insurance" gets you.
This is the terrifying part: doctors do this too! I have an MD friend that told me she uses ChatGPT to retrieve dosing info. I asked her to please, please not do that.
I have a hunch that the whole "chat" interface is a brilliant but somewhat unintentional product design choice that has created this faux trust in LLM's to give back accurate information that others can get from drugs.com or Medline with a text search. This is a terrifying example, and please get her to test it out by second guessing the LLM and watching it flip flop.
Find good doctors. A solution doesn’t have to be perfect. A doctor doing better than regular joe with a computer is much higher as you can see in research around this topic
Almost certainly more I would think, precisely because of magnitude errors.
The ol' "What weighs more, a pound of feathers or two pounds of bricks" trick explains this perfectly to me.
I disagree. I'd wager that state of the art LLMs can beat out of the average doctor at diagnosis given a detailed list of symptoms, especially for conditions the doctor doesn't see on a regular basis.
"Given a detailed list of symptoms" is sure holding a lot of weight in that statement. There's way too much information that doctors tacitly understand from interactions with patients that you really cannot rely on those patients supplying in a "detailed list". Could it diagnose correctly, some of the time? Sure. But the false positive rate would be huge given LLMs suggestible nature. See the half dozen news stories covering AI induced psychosis for reference.
Regardless, it's diagnostic capability is distinct from the dangers it presents, which is what the parent comment was mentioning.
What you're describing, especially with the amount of water "given a detailed list of symptoms" is carrying, is essentially a compute-intensive flowchart with no concept of diagnostic parsimony.
Plot twist, your doctor is looking it up on WebMD themselves
Not really: it's arguably quite a lot worse. Because you can judge the trustworthiness of the source when you follow a link from Google (e.g. I will place quite a lot of faith in pages at an .nhs.uk URL), but nobody knows exactly how that specific LLM response got generated.
> the user will at least need to know something about the topic beforehand.
This is why I've said a few times here on HN and elsewhere, if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. Juniors can do amazing things, they can also goof up hard. What's really funny is you can make them audit their own code in a new context window, and give you a detailed answer as to why that code is awful.
I use it mostly on personal projects especially since I can prototype quickly as needed.
> if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer.
The thing is coding can (and should) be part of the design process. Many times, I though I have a good idea of what the solution should look like, then while coding, I got exposed more to the libraries and other parts of the code, which led me to a more refined approach. This exposure is what you will miss and it will quickly result in unfamiliar code.
I agree. I mostly use it for scaffolding, I don't like letting it do all the work for me.
No friction, no improvements; that only guarantees you'll never find a better way to solve the problem.
>the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand. I believe that this lack of initial understanding of the user input
I think there's a parallel here for the internet as an i formation source. It delivered on "unlimited knowledge at the tip of everyone's fingertips" but lowering the bar also lowered the bar.
That access "works" only when the user is capable of doing their part too. Evaluating sources, integrating knowledge. Validating. Cross examining.
Now we are just more used to recognizing that accessibility comes with its own problem.
Some of this is down to general education. Some to domain expertize. Personality plays a big part.
The biggest factor is, i think, intelligence. There's a lot of 2nd and 3rd order thinking required to simultaneously entertain a curiosity, consider of how the LLM works, and exercise different levels of skepticism depending on the types of errors LLMs are likely to make.
Using LLMs correctly and incorrectly is.. subtle.
> The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters
It's also useful to have an intuition for what things an LLM is liable to get wrong/hallucinate, one of which is questions where the question itself suggests one or more obvious answers (which may or may not be correct), which the LLM may well then hallucinate, and sound reasonable, if it doesn't "know".
LLMs are very sensitive to leading questions. A small hint of that the expected answer looks like will tend to produce exactly that answer.
As a consequence LLMs are extremely unlikely to recognize an X-Y problem.
You don't even need a leading direct question. You can easily lead an LLM just by having some statements (even at times single words) in the context window.
> the problem is that in order to develop an intuition for questions that LLMs can answer, the user will at least need to know something about the topic beforehand
This is why simonw (The author) has his "pelican on a bike" -test, it's not 100% accurate but it is a good indicator.
I have a set of my own standard queries and problems (no counting characters or algebra crap) I feed to new LLMs I'm testing
None of the questions exist outside of my own Obsidian note so they can't be gamed by LLM authors. And I've tested multiple different LLMs using them so I have a "feeling" on what the answer should look like. And I personally know the correct answer so I can immediately validate them.
They are training on your queries. So they may have some exposure to them going forward.
Even if your queries are hidden via a local running model you must have some humility that your queries are not actually unique. For this reason I have a very difficult time believing that a basic LLM will be able to properly reason about complex topics, it can regurgitate to whatever level its been trained. That doesn't make it less useful though. But on the edge case how do we know the query its ingesting gets trained with a suitable answer? Wouldn't this constitute an over-fitting in these cases and be terribly self-reinforcing?
Not if one ollama pull to ur machine.
Please, everybody, preserve your records. Preserve your books, preserve your downloaded files (that can't be tampered with), keep everything. AI is going to make it harder and harder to find out the truth about anything over the next few years.
You have a moral duty to keep your books, and keep your locally-stored information.
I get very annoyed when llms respond with quotes around certain things I ask for, then when I say what is the source of that quote? they say oh I was paraphrasing and that isnt a real quote.
At least wikipedia has sources that probably support what it says and normally the quotes are real quotes. LLMs just seem to add quotation marks as, "proof" that its confident something is correct.
do you have examples of these ?
To that end, it seems as though archive.org will important for an entirely new reason. Not for the loss of information, but the degradation of it.
A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time.
When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, then reopen the image and get a clear image of a pile of dirt.
Furthermore, an encyclopaedia is something you can reference and learn from without a goal, it allows you to peruse information you have no concept of. Not so with LLMs, which you have to query to get an answer.
Lossy compression does make things up. We call them compression artefacts.
In compressed audio these can be things like clicks and boings and echoes and pre-echoes. In compressed images they can be ripply effects near edges, banding in smoothly varying regions, but there are also things like https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... where one digit is replaced with a nice clean version of a different digit, which is pretty on-the-nose for the LLM failure mode you're talking about.
Compression artefacts generally affect small parts of the image or audio or video rather than replacing the whole thing -- but in the analogy, "the whole thing" is an encyclopaedia and the artefacts are affecting little bits of that.
Of course the analogy isn't exact. That would be why S.W. opens his post by saying "Since I love collecting questionable analogies for LLMs,".
> Lossy compression does make things up. We call them compression artefacts.
I don’t think this is a great analogy.
Lossy compression of images or signals tends to throw out information based on how humans perceive it, focusing on the most important perceptual parts and discarding the less important parts. For example, JPEG essentially removes high frequency components from an image because more information is present with the low frequency parts. Similarly, POTS phone encoding and mp3 both compress audio signals based on how humans perceive audio frequency.
The perceived degradation of most lossy compression is gradual with the amount of compression and not typically what someone means when they say “make things up.”
LLM hallucinations aren’t gradual and the compression doesn’t seem to follow human perception.
You are right and the idea of LLMs as lossy compression has lots of problems in general (LLMs are a statistical model, a function approximating the data generating process).
Compression artifacts (which are deterministic distortions in reconstruction) are not the same as hallucinations (plausible samples from a generative model; even when greedy, this is still sampling from the conditional distribution). A better identification is with super-resolution. If we use a generative model, the result will be clearer than a normal blotchy resize but a lot of details about the image will have changed as the model provides its best guesses at what the missing information could have been. LLMs aren't meant to reconstruct a source even though we can attempt to sample their distribution for snippets that are reasonable facsimiles from the original data.
An LLM provides a way to compute the probability of given strings. Once paired with entropy coding, on-line learning on the target data allows us to arrive at the correct MDL based lossless compression view of LLMs.
LLM confabulations might as well be gradual in the latent space. I don’t think lossy is synonymous to perceptual and the high frequency components rather easily translate to less popular data.
I feel like my comment is pretty clear that a compression artefact is not the same thing as making the whole thing up.
> Of course the analogy isn't exact.
And I don’t expect it to be, which is something I’ve made clear several times before, including on this very thread.
Interesting, in the LLM case these compression artefacts then get fed into the generating process of the next token, hence the errors compound.
Not really. The whole "inference errors will always compound" idea was popular in GPT-3.5 days, and it seems like a lot of people just never updated their knowledge since.
It was quickly discovered that LLMs are capable of re-checking their own solutions if prompted - and, with the right prompts, are capable of spotting and correcting their own errors at a significantly-greater-than-chance rate. They just don't do it unprompted.
Eventually, it was found that reasoning RLVR consistently gets LLMs to check themselves and backtrack. It was also confirmed that this latent "error detection and correction" capability is present even at base model level, but is almost never exposed - not in base models and not in non-reasoning instruct-tuned LLMs.
The hypothesis I subscribe to is that any LLM has a strong "character self-consistency drive". This makes it reluctant to say "wait, no, maybe I was wrong just now", even if latent awareness of "past reasoning look sketchy as fuck" is already present within the LLM. Reasoning RLVR encourages going against that drive and utilizing those latent error-correction capabilities.
You seem to be responding to a strawman, and assuming I think something I don't think.
As of today, 'bad' generations early in the sequence still do tend towards responses that are distant to the ideal response. This is testable/verifiable by pre-filling responses, which I'd advise you to experiment with for yourself.
'Bad' generations early in the output sequence are somewhat mitigatable by injecting self-reflection tokens like 'wait', or with more sophisticated test-time compute techniques. However, those remedies can simultaneously turn 'good' generations into bad, they are post-hoc heuristics which treat symptoms not causes.
In general, as the models become larger they are able to compress more of their training data. So yes, using the terminology of the commenter I was responding to, larger models should tend to have fewer 'compression artefacts' than smaller models.
With better reasoning training, the models mitigate more and more of that entirely by themselves. They "diverge into a ditch" less, and "converge towards the right answer" more. They are able to use more and more test-time compute effectively. They bring their own supply of "wait".
OpenAI's in-house reasoning training is probably best in class, but even lesser naive implementations go a long way.
The problem is that language doesn't produce itself. Re-checking, correcting error is not relevant. Error minimization is not the fount of survival, remaining variable for tasks is. The lossy encyclopedia is neither here nor there, it's a mistaken path:
"Language, Halliday argues, "cannot be equated with 'the set of all grammatical sentences', whether that set is conceived of as finite or infinite". He rejects the use of formal logic in linguistic theories as "irrelevant to the understanding of language" and the use of such approaches as "disastrous for linguistics"."
Sorry, what? This is borderline incoherent.
The units themselves are meaningless without context. The point of existence, action, tasks is to solve the arbitrariness in language. Tasks refute language, not the other way around. This may be incoherent as the explanation is scientific, based in the latest conceptualization of linguistics.
CS never solved the incoherence of language, conduit metaphor paradox. It's stuck behind language's bottleneck, and it do so willingly blind-eyed.
What? This is even less coherent.
You weren't talking to GPT-4o about philosophy recently, were you?
I'd know cutting-edge linguistics and signaling theory well beyond Shannon to parse this, not NLP or engineering reduction. What I've stated is extremely coherent to Systemic Functional Linguists.
Beyond this point engineers actually have to know what signaling is, rather than 'information.'
https://www.sciencedirect.com/science/article/abs/pii/S00033...
Ultimately, engineering chose the wrong approach to automating language, and it sinks the field. It's irreversible.
One of the main takeaways from The Bitter Lesson was that you should fire your linguists. GPT-2 knows more about human language than any linguist could ever hope to be able to convey.
If you're hitching your wagon to human linguists, you'll always find yourself in a ditch in the end.
Sorry, 2 billion years of neurobiology beats 60 years of NLP/LLMs which knows less to nothing about language since "arbitrary points can never be refined or defined to specifics" check your corners and know your inputs.
The bill is due on NLP.
Incoherent drivel.
If not language what training substrate do you suggest? Also not strong ideas are expressible coherently. You have an ironic pattern in your comments of getting lost in the very language morass you propose to deprecate. If we don't train models on language what do we train them on? I have some ideas of my own but I am interested if you can clearly express yours.
Neural/spatial syntax. Analoga of differentials. The code to operate this gets built before the component.
If language doesn't really mean anything, then automating it in geometry is worse than problematic.
The solution is starting over at 1947: measurement not counting.
The semantic meaning of your words here is non-existent. It is unclear to me how else you can communicate in a text based forum if not by using words. Since you can't despite your best effort I am left to conclude you are psychotic and should probably be banned and seek medical help.
Engineers are so close-minded, you can't see the freight train bearing down on the industry. All to science's advantage replacing engineers. Interestingly, if you dissect that last entry, I've just made the case measurement (analog computation) is superior to counting (binary computation) and laid out the strategy how. All it takes is brains, or an LLM to decipher what it states.
https://pmc.ncbi.nlm.nih.gov/articles/PMC3005627/
"First, cell assemblies are best understood in light of their output product, as detected by ‘reader-actuator’ mechanisms. Second, I suggest that the hierarchical organization of cell assemblies may be regarded as a neural syntax. Third, constituents of the neural syntax are linked together by dynamically changing constellations of synaptic weights (‘synapsembles’). Existing support for this tripartite framework is reviewed and strategies for experimental testing of its predictions are discussed."
I 100% agree analog computing would be better at simulating intelligence than binary. Why don't you state that rather than burying it under a mountain of psychobabble?
Listing the conditions, dichotomizing the frameworks counting/measurement is the farthest from psycho-babble. Anyone with knowledge of analog knows these terms. And enough to know analog doesn't simulate anything. And intelligence isn't what's being targeted.
I don't think there is a singular "should" that fits every use case.
E.g. a Bloom filter also doesn't "know" what it knows.
I don’t understand the point you’re trying to make. The given example confused me further, since nothing in my argument is concerned with the tool “knowing” anything, that has no relation to the idea I’m expressing.
I do understand and agree with a different point you’re making somewhere else in this thread, but it doesn’t seem related to what you’re saying here.
The argument is that a banana is a squishy hammer.
You're saying hammers shouldn't be squishy.
Simon is saying don't use a banana as a hammer.
> You're saying hammers shouldn't be squishy.
No, that is not what I’m saying. My point is closer to “the words chosen to describe the made up concept do not translate to the idea being conveyed”. I tried to make that fit into your idea of the banana and squishy hammer, but now we’re several levels of abstraction deep using analogies to discuss analogies so it’s getting complicated to communicate clearly.
> Simon is saying don't use a banana as a hammer.
Which I agree with.
This is the type of comment that has been killing HN lately. “I agree with you but I want to disagree because I’m generally just that type of person. Also I am unable to tell my disagreeing point adds nothing.”
Except that’s not what I’m saying at all. If anything, the “type of comment that has been killing HN” (and any community) are those who misunderstand and criticise what someone else says without providing any insight while engaging in ad hominem attacks (which are explicitly against the HN guidelines). It is profoundly ironic you are actively attacking others for the exact behaviour you are engaging in. I will kindly ask you do not do that. You are the first person in this immediate thread being rude and not adding to the collective understanding of the argument.
We are all free to agree with one part of an argument while disagreeing with another. That’s what healthy discourse is, life is not black and white. As way of example, if one says “apples are tasty because they are red”, it is perfectly congruent to agree apples are tasty but disagree that their colour is the reason. And by doing so we engage in a conversation to correct a misconception.
Yeah an LLM is an unreliable librarian, if anything.
I actually disagree. Modern encoding formats can, and do, hallucinate blocks.
It’s a lot less visible and I guess dramatic than LLMs but it happens frequently enough that I feel like at every major event there are false conspiracies based on video « proofs » that are just encoding artifacts
I think you are missing the point of the analogy: a lossy encyclopedia is obviously a bad idea, because encyclopedias are meant to be reliable places to look up facts.
And my point is that “lossy” does not mean “unreliable”. LLMs aren’t reliable sources of facts, no argument there, but a true lossy encyclopaedia might be. Lossy algorithms don’t just make up and change information, they remove it from places where they might not make a difference to the whole. A lossy encyclopaedia might be one where, for example, you remove the images plus gramatical and phonetic information. Eventually you might compress the information where the entry for “dog” only reads “four legged creature”—which is correct but not terribly helpful—but you wouldn’t get “space mollusk”.
I don't think a "true lossy encylopedia" is a thing that has ever existed.
Since sibling comments all seem to have concentrated on idealistic good intent, I would also like to point out a different side of things.
I grew up in socialism. Since we've transitioned to democracy, I learned that I have to unlearn some things. Our encyclopedias were not inaccurate but were not complete. It's like lying through omission. And as the old saying goes, half-truths are worse than lies.
Whether this would be deemed as a lossy encyclopedia, I don't know. What I am certain of, however, is that it was accurate but omitted important additional facts.
And that is what I see in LLMs as well. Overall, it's accurate, except in cases where an additional fact would alter the conclusion. So, it either could not find arguments with that fact, or it chose to ignore them to give an answer and could be prompted into taking them into account or whatever.
What I do know is that LLMs of today give me the same hibbie-jibbies that rereading those encyclopedias of my youth give me.
All encyclopedias are lossy. They curate the info they include, only choosing important topics. Wikipedia is lossy. They delete whole articles for irrelevance. They edit changes to make them more concise. They require sources for facts. All good things, but Wikipedia is a subset of human knowledge.
One could argue that’s what a pocket encyclopaedia (those exist) is. But even if we say they don’t, when you make up a term by mushing two existing words together it helps if the term makes sense. Otherwise, why even use the existing words? You called it a “lossy enyclopedia” and not a “spaghetti ice cream” for a reason, presumably so the term evokes an image or concept in the mind of the reader. If it’s bringing up a different image than what you intended, perhaps it’s not a good term.
I remember you being surprised when the term “vibe coding” deviated from its original intention (I know you didn’t come up with it). But frankly I was surprised at your surprise—it was entirely predictable and obvious how the term was going to be used. The concept I’m attempting to communicate to you is that when you make up a term you have to think not only of the thing in your head but also of the image it conjures up in other people’s minds. Communication is a two-way street.
I think you're saying that "pocket encyclopedia" is one definition of "lossy encyclopedia" that may occur to people (or that may get marketed on purpose). But that's a very poor definition of LLMs. And so the danger is that people may lock onto a wildly misleading definition. Am I getting the point?
A lossy encyclopedia which you can talk to and it can look up facts in the lossless version while having a conversation OTOH is... not a bad idea at all, and hundreds of millions of people agree if traffic numbers are to be believed.
(but it isn't and won't ever be an oracle and apparently that's a challenge for human psychology.)
Completely agree with you - LLMs with access to search tools that know how to use them (o3, GPT-5, Claude 4 are particularly good at this) mostly paper over the problems caused by a lossy set of knowledge in the model weights themselves.
But... end users need to understand this in order to use it effectively. They need to know if the LLM system they are talking to has access to a credible search engine and is good at distinguishing reliable sources from junk.
That's advanced knowledge at the moment!
From earlier today:
Me: How do I change the language settings on YouTube?
Claude: Scroll to the bottom of the page and click the language button on the footer.
Me: YouTube pages scroll infinitely.
Claude: Sorry! Just click on the footer without scrolling, or navigate to a page where you can scroll to the bottom like a video.
(Videos pages also scroll indefinitely through comments)
Me: There is no footer, you're just making shit up
Claude: [finally uses a search engine to find the right answer]
IME, eventually, after a long time, the scrolling stops and you can get to the footer. YMMV!
Slightly off topic, but my experience is that they are pretty terrible at using search tools..
They can often reason themselves into some very stupid direction, burning all the tokens for no reason and failing to reply in the end.
I don't like the confident hallucinations of LLMs either, but don't they rewrite and add entries in the encyclopedia every few years? Implicitly that makes your old copy "lossy"
Again, never really want a confidently-wrong encyclopedia, though
Aren't all encyclopedias 'lossy'? They are all partial collections of information; none have all of the facts.
I am sympathetic to your analogy. I think it works well enough.
But it falls a bit short in that encyclopedias, lossy or not, shouldn't affirmatively contain false information. The way I would picture a lossy encyclopedia is that it can misdirect by omission, but it would not change A to ¬A.
Maybe a truthy-roulette enclyclopedia?
I guarantee every encyclopedia has mistakes.
I remember a study where they checked if wikipedia had more errors than paper encyclopedias, and they found there were about as many errors in both.
That study ended the "you can't trust wikipedia" argument, you can't trust anything but wikipedia is an as good as it gets second hand reference.
> You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, then reopen the image and get a clear image of a pile of dirt.
Oh but it's much worse than that: because most LLMs aren't deterministic in the way they operate [1], you can get a pristine image of a different pile of dirt every single time you ask.
[1] there are models where if you have the "model + prompt + seed" you're at least guaranteed to get the same output every single time. FWIW I use LLMs but I cannot integrate them in anything I produce when what they output ain't deterministic.
"Deterministic" is overrated.
Computers are deterministic. Most of the time. If you really don't think about all the times they aren't. But if you leave the CPU-land and go out into the real world, you don't have the privilege of working with deterministic systems at all.
Engineering with LLMs is closer to "designing a robust industrial process that's going to be performed by unskilled minimum wage workers" than it is to "writing a software algorithm". It's still an engineering problem - but of the kind that requires an entirely different frame of mind to tackle.
And one major issue is that LLMs are largely being sold and understood more like reliable algorithms than what they really are.
If everyone understood the distinction and their limitations, they wouldn’t be enjoying this level of hype, or leading to teen suicides and people giving themselves centuries-old psychiatric illnesses. If you “go out into the real world” you learn people do not understand LLMs aren’t deterministic and that they shouldn’t blindly accept their outputs.
https://archive.ph/20241023235325/https://www.nytimes.com/20...
https://archive.ph/20250808145022/https://www.404media.co/gu...
It's nothing new. LLMs are unreliable, but in the same ways humans are.
But LLMs output is not being treated the same as human output, and that comparison is both tired and harmful. People are routinely acting like “this is true because ChatGPT said so” while they wouldn’t do the same for any random human.
LLMs aren’t being sold as unreliable. On the contrary, they are being sold as the tool which will replace everyone and do a better job at a fraction of the piece.
That comparison is more useful than the alternatives. Anthropomorphic framing is one of the best framings we have for understanding what properties LLMs have.
"LLM is like an overconfident human" certainly beats both "LLM is like a computer program" and "LLM is like a machine god". It's not perfect, but it's the best fit at 2 words or less.
Um, no. They are unreliable at a much faster pace and larger scale than any human. They are more confident while being unreliable than most humans (politicians and other bullshitters aside, most humans admit when they aren't sure about something).
> you can get a pristine image of a different pile of dirt every single time you ask.
That’s what I was trying to convey with the “then reopen the image” bit. But I chose a different image of a different thing rather than a different image of a similar thing.
An encyclopaedia also can't win a gold medals at the IMO and IOI. So yeah, they're not the same thing, even though the analogy is pretty good.
Of course they’re not the same thing, the goal of an analogy is not to be perfect but to provide a point of comparison to explain an idea.
My point is that I find the chosen term inadequate. The author made it up from combining two existing words, where one of them is a poor fit for what they’re aiming to convey.
Thinking of an LLM as any kind of encyclopedia is probably the wrong model. LLMs are information presentation/processing tools that incidentally, as a consequence of the method by which they are built to do that, may occasionally produce factual information that is not directly prompted.
If you want an LLM to be part of a tool that is intended to provide access to (presumably with some added value) encyclopedic information, it is best not to consider the LLM as providing any part of the encyclopedic information function of the system, but instead as providing part of the user interface of the system. The encyclopedic information should be provided by appropriate tooling that, at request by an appropriately prompted LLM or at direction of an orchestration layer with access to user requests (and both kinds of tooling might be used in the same system) provides relevant factual data which is inserted into the LLM’s context.
The correct modifier to insert into the sentence “An LLM is an encyclopedia” is “not”, not “lossy”.
Using artificial neural networks directly for information storage and retrieval (i.e. not just leveraging them as tools accessing other types of storage) is currently infeasible, agreed.
On the other hand, biological neural networks are doing it all the time :) And there might well be an advantage to it (or a hybrid method), once we can make it more economical.
After all, the embedding vector space is shaped by the distribution of training data, and if you have out-of-distribution data coming in due to a new or changed environment, RAG using pre-trained models and their vector spaces will only go so far.
eh, bio neural networks aren't doing that all the time. Memmories are notorious for being "rebuilt" constantly.
I think an LLM can be used as a kind of lossy encyclopedia, but equating it directly to one isn't entirely accurate. The human mind is also, in a sense, a lossy encyclopedia.
I prefer to think of LLMs as lossy predictors. If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordingly and survive.
In the real world, with countless fuzzy factors, no predictor can ever be perfectly lossless. The only real difference, for me, is that LLMs are lossier predictors than human minds (for now). That's all there is to it.
Whatever analogy you use, it comes down to the realization that there's always some lossiness involved, whether you frame it as an encyclopedia or not.
Are LLMs really lossier than humans? I think it depends on the context. Given any particular example, LLMs might hallucinate more and a human might do a better job at accuracy. But overall LLMs will remember far more things than a human. Ask a human to reproduce what they read in a book last year and there's a good chance you'll get either absolutely nothing or just a vague idea of what the book was about - in this context they can be up to 100% lossy. The difference here is that human memory decays over time while a LLM's memory is hardwired.
I think what trips people up is that LLMs and humans are both lossy, but in different ways.
The intuitions that we've developed around previous interactions are very misleading when applied to LLMs. When interacting with a human, we're used to being able to ask a question about topic X in context Y and assume that if you can answer it we can rely on you to be able to talk about it in the very similar context Z.
But LLMs are bad at commutative facts; A=B and B=A can have different performance characteristics. Just because it can answer A=B does not mean it is good at answering B=A; you have to test them separately.
I've seen researchers who should really know better screw this up, rendering their methodology useless for the claim they're trying to validate. Our intuition for how humans do things can be very misleading when working with LLMs.
That's not exactly true. Every time you start a new conversation; you get a new LLM for all intents. Asking an LLM about an unrelated topic towards the end of a ~500 page conversation will get you vastly different results than at the beginning. If we could get to multi-thousand page contexts, it would probably be less accurate than a human, tbh.
Yes, I should have clarified that I was referring to memory of training data, not of conversations.
>Given any particular example, LLMs might hallucinate more and a human might do a better job at accuracy
This drastically depends on the example. For average trivia questions, modern LLMs (even smaller, open ones) beat humans easily.
Lossy is an incomplete characterization. LLMs are also much more fluctuating and fuzzy. You can get wildly varying output depending on prompting, for what should be the same (even if lossy) knowledge. There is not just loss during the training, but also loss and variation during inference. An LLM overall is a much less coherent and consistent thing than most humans, in terms of knowledge, mindset, and elucidations.
> If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordingly and survive.
Yes.
Human intelligence consists of three things.
First, groundedness: The ability to form a representation of the world and one’s place in it.
Second, a temporal-spatial sense: A subjective and bounded idea of self in objective space and time.
Third: A general predictive function which is capable of broad abstraction.
At its most basic level, this third element enables man to acquire, process, store, represent, and continually re-acquire knowledge which is external to that man's subjective existence. This is calculation in the strictest sense.
And it is the third element -- the strength, speed, and breadth of the predictive function -- which is synonymous with the word "intelligence." Higher animals have all three elements, but they're pretty hazy -- especially the third. And, in humans, short time horizons are synonymous with intellectual dullness.
All of this is to say that if you have a "prediction machine" you're 90% of the way to a true "intelligence machine." It also, I think, suggests routes that might lead to more robust AI in the future. (Ground the AI, give it a limited physical presence in time and space, match its clocks to the outside world.)
"Prediction" is hardly more than another term for inference. It's the very essence of machine learning. There is nothing new or useful in this concept.
Point is that it's also exactly analogous to human intelligence. There's almost nothing else to it.
This is how you spot hype nonsense - claims that anything is analogous to human intelligence. Even absent all other objections, we don't understand the human mind well enough to make a claim like that.
You don't need to understand the human mind on a mechanistic level. You only need to examine how the whole organism learns, acts, and reacts to stimulus and situation.
Even something as simple as catching a ball is basically predictive. You predict where the ball will be along its arc when it reaches a point in space where you can catch it. Then, strictly informed by that prediction, you solve a problem of motion through space -- and some very simple-seeming problems of motion through space can't be cracked in a general case without a very powerful supercomputer -- to physically catch the ball.
That's a very simple example. The major component of what we call intelligence is purely predictive. Of course Bayesian inference also works the same way, etc.
> The major component of what we call intelligence is purely predictive
Then what is creativity? Creativity is not predictive and is the most important part of human intelligence, since it isn't about figuring out if a situation leads to good things, its about finding a new kind of situation that leads to good things.
Don't say "we do totally random things and try to predict those outcomes", there is nothing supporting that since we have tried that with computers and that doesn't result in creativity anything like humans, we don't know how human creativity works.
So we've got a temporal-spatial sense, a general predictive function, the capacity for abstraction, independent volition, and a sense of relational context.
Creativity shows up when an agent uses that predictive machinery not only to forecast immediate sensory consequences, but to (a) simulate many alternative internal models or actions (counterfactuals), usually in a self-directed way with an end or goal in mind, (b) predict how those alternatives will be interpreted by other agents or by itself in the future, and (c) select from those alternatives according to an intrinsic/extrinsic valuation that rewards novelty, surprise, utility, or aesthetic pleasure. In other words it's a form of guided meta-prediction.
From a very different perspective, the TRIZ guys have tried to figure out creativity, with results that are at least interesting. Ultimately, what they have to teach is that non-artistic creativity also takes certain characteristic forms.
> The major component of what we call intelligence is purely predictive.
Making more unfounded, nonsensical claims does not reinforce your first unfounded, nonsensical claim.
I'm sure statisticians would love it if the human mind were an inference machine, but that doesn't make it one. Your point of view on this is faith-based.
His view aligns both with a leading neuroscience explanation of the brain (predictive coding [1]) and with Active Inference / the Free Energy principle [2] from optimal control theory. A similar theory of intelligence, called H-JEPA (hierarchical joint embedding predictive architecture) [3] is also put forward by Yann LeCun, a major AI pioneer. Another AI pioneer, Jürgen Schmidhuber, subsequently criticized LeCun's theory, but not for being faulty, but for rehashing ideas published in several papers from the 1990s onward. [4]
1: https://en.wikipedia.org/wiki/Predictive_coding
2: https://en.wikipedia.org/wiki/Free_energy_principle#Active_i...
3: https://openreview.net/forum?id=BZ5a1r-kVsf
4: https://people.idsia.ch/~juergen/lecun-rehash-1990-2022.html
Imagine having the world's most comprehensive encyclopedia at your literal fingertips, 24 hours a day, but being so lazy that you offload the hard work of thinking by letting retarded software pathologically lie to you and then blindly accepting the non-answers it spits at you rather than typing in two or three keywords to Wikipedia and skimming the top paragraph.
>I prefer to think of LLMs as lossy predictors.
I've started to call them the Great Filter.
In the latest issue of the comic book Lex Luthor attempts to exterminate humanity by hacking the LLM and having it inform humanity that they can hold their breath underwater for 17 hours.
> you build a world model
The foundational conceit (if you will) of LLMs is that they build a semantic (world) model to 'make sense' of their training. However it is much more likely that they are simply building a syntactic model in response to the training. As far as I know there is no evidence of a semantic model emerging.
There's some evidence of valid relationships: you can build a map of Manhattan by asking about directions from each street corner and plotting the relations.
This is still entirely referential, but in a way that a human would see some relation to the actual thing, albeit in a somewhat weird and alien way.
Maybe I don’t have a precise enough definition of syntax and semantics, but it seems like it’s more than just syntactic since interchangeable tokens in the same syntax affect the semantics of the sentence. Or do you view completing a prompt such as “The president of the United States is?” as a syntax question?
Is this not addressed by othellogpt?
Another difference is that you are predicting future sensory experiences in real-time, while LLMs "predict" text which a "helpful, honest, harmless" assistant would produce.
There are a lot of parallels between AI and compression.
In fact the best compression algorithms and LLMs have in common that they work by predicting the next word. Compression algorithms take an extra step called entropy coding to encode the difference between the prediction and the actual data efficiently, and the better the prediction, the better the compression ratio.
What makes a LLM "lossy" is that you don't have the "encode the difference" step.
And yes, it means you can turn a LLM into a (lossless) compression algorithm, and I think a really good one in term of compression ratio on huge data sets. You can also turn a compression algorithm like gzip into a language model! A very terrible one, but the output is better than a random stream of bytes.
I suspect this ends up being pretty important for the next advancements in AI, specifically LLM-based AI. To me, the transformer architecture is a sort of compression algorithm that is being exploited for emergent behavior at the margins. But I think this is more like stream of consciousness than premeditated thought. Eventually I think we figure out a way to "think" in latent space and have our existing AI models be just the mouthpiece.
In my experience as a human, the more you know about a subject, or even the more you have simply seen content about it, the easier it is to ramble on about it convincingly. It's like a mirroring skill, and it does not actually mean you understand what you're saying.
LLMs seem to do the same thing, I think. At scale this is widely useful, though, I am not discounting it. Just think it's an order of magnitude below what's possible and all this talk of existing stream-of-consciousness-like LLMs creating AGI seems like a miss
One difference is that compression gives you one and only one thing when decompressing. Decompression isn't a function taking arbitrary additional input and producing potentially arbitrary, nondeterministic output based on it.
We would have very different conversations if LLMs were things that merely exploded into a singular lossy-expanded version of Wikipedia, but where looking at the article for any topic X would give you the exact same article each time.
LLMs deliberately insert randomness. If you run a model locally (or sometimes via API), you can turn that off and get the same response for the same input every time.
True, but I'd argue that you can't get the definite knowledge of an LLM by turning off randomness, or fixing the seed. Otherwise that would be a routinely employed feature, to determine what an LLM "truly knows", removing any random noise distorting that knowledge, and instead randomness would only be turned on for tasks requiring creativity, not when merely asking factual questions. But it doesn’t work that way. Different seeds and will uncover different "knowledge", and it's not the case that one is a truer representation of an LLM's knowledge than another.
Furthernore, even in the absence of randomness, asking an LLM the same question in different ways can yield different, potentially contradictory answers, even when the difference in prompting is perfectly benign.
This is because the knowledge is encoded in a multi-dimensional space, and a seed doesn’t change the knowledge, only the expression of it. If you ask me what E=mc^2 means, I’ll give you different answers depending on whether I think you are a curious lay-person vs. a physicist testing my response.
You see this with humans who encode physical space to physical matrix in our brain. When asking for directions, people have to traverse this matrix until it is memorized, then it isn’t used any longer; only the rote data is referenced.
With a handy trick called arithmetic coding, you can actually turn an LLM into a lossless compression algorithm!
Indeed, see https://bellard.org/nncp/ for an example.
Accurate!
An llm is also a more convenient encyclopedia.
I'm not surprised a large portion of people choose convenience over correctness. I do not necessarily agree with the choice, but looking at historical trends, I do not find it surprising that it's a popular choice.
> "...They have a huge array of facts compressed into them but that compression is lossy (see also Ted Chiang)"
indeed, Ted's piece (ChatGPT Is a Blurry JPEG of the Web) is here:
Worth highlighting - 2023.
Yes, LLM is a lossy encyclopedia with a human-language answering interface. This has some benefits, mostly in terms of convenience. You don't have to browse or read through so many pages of a real encyclopedia to get a quick answer. However, there is also a clear downside. Currently, LLM is unable to judge if your question is formulated incorrectly or if your question opens up more questions that should be answered first. It always jumps to answering something. A real human would assess the questioner first and usually ask for more details before answering. I feel this is the predominant reason why LLM answers feel so dumb at times. It never asks for clarification.
I don't think that's universally true with the new models - I've seen Claude 4 and GPT-5 ask for clarification on questions with obvious gaps.
With GPT-5 I sometimes see it spot a question that needs clarifying in its thinking trace, then pick the most likely answer, then spit out an answer later that says "assuming you meant X ..." - I've even had it provide an answer in two sections for each branch of a clear ambiguity.
A lot of the touted "fundamental limitations of LLMs" are less "fundamental" and more "you're training them wrong".
So there are improvements version to version - from both increases in raw model capabilities and better training methods being used.
I'm frustrated by the number of times I encounter people assuming that the current model behavior is inevitable. There's been hundreds of billions of dollars spent on training LLMs to do specific things. What exactly they've been trained on matters; they could have been trained to do something else.
Interacting with a base model versus an instruction tuned model will quickly show you the difference between the innate language faculties and the post-trained behavior.
Some of the Anthropic guys have said that the core thing holding the models back is training, and they're confident the gains will keep coming as they figure out how to onboard more and more training data. So yeah, Claude might suck at reading and writing plumbing diagrams, but they claim the barrier is simply a function of training, not any kind of architectural limitation.
I agree with the general idea, but "sucks at reading plumbing diagrams" is the one specific example where Claude is actually choked by its unfortunate architecture.
The "naive" vision implementation for LLMs is: break the input image down into N tokens and cram those tokens into the context window. The "break the input image down" part is completely unaware of the LLM's context, and doesn't know what data would be useful to the LLM at all. Often, the vision frontend just tries to convey the general "vibes" of the image to the LLM backend, and hopes that the LLM can pick out something useful from that.
Which is "good enough" for a lot of tasks, but not all of them, not at all.
GPT-5 is seriously annoying. It asks not just one but multiple clarifying questions, while I just want my answer.
If you don't want to answer clarifying questions, then what use is the answer???
Put another way, if you don't care about details that change the answer, it directly implies you don't actually care about the answer.
Related silliness is how people force LLMs to give one word answers to underspecified comparisons. Something along the lines of "@Grok is China or US better, one word answer only."
At that point, just flip a coin. You obviously can't conclude anything useful with the response.
No, I don't think GPT-5 clarifying questions actually do what you think they do. They just made the model ask clarifying questions for the sake of asking clarifying questions. I'm sure GPT-4o would have given me the answer I wanted without clarifying questions.
This is also why the Kagi Assistant is still be the AI tool I’ve found. The failure state is the same as a search results, it either can’t find anything, finds something irrelevant, or finds material that contradicts the premise of your question.
It seems to me the more you can pin it to another data set, the better.
That AI is closely related to compression is a well established idea. E.g. http://prize.hutter1.net/
It seems reasonable to argue that LLMs are a form of lossy compression of text that preserves important text features.
There is a precedent of distributing low quality lossy compressed versions of copyrighted work being considered illegal.
An LLM is a lossy Borges' Library of Babel
"Though the vast majority of the books in this universe are pure gibberish, the laws of probability dictate that the library also must contain, somewhere, every coherent book ever written, or that might ever be written, and every possible permutation or slightly erroneous version of every one of those books. " -https://en.wikipedia.org/wiki/The_Library_of_Babel
It's a version of the library of babel filtered only to the books which plausibly consist of prose. The set is still incomprehensibly vast, and all the more treacherous for generally being "readable".
Perhaps this also explains why I almost think LLMs are not helpful in engineering-level projects:
1. My project involved programming languages and APIs that iterated several levels faster than LLM could publish a book
2. I have lost faith in LLMs developing software.
An example is the famous Unity game engine, but LLM has not helped its Unity DOTS architecture (ECS mode compared with GameObject mode). Although I have a basic understanding of it, both the entities API documentation and the LLM answers are terrible. I chose Unity because I heard it is mature so I think LLMs would be helpful with so much materials. Sadly, for ECS it doesn't. So I chose Bevy, a game engine that I can understand and apply by reading documents and can solve problems without the help of LLM.
I disagree with that analogy, because LLMs have a lot of connections between text fragments, which an encyclopedia doesn't have to such a deep degree. An encyclopedia also can't interpret and output relevant knowledge from an input prompt.
It helps a lot to set expectations though, especially when thinking of an encyclopedia not as a row of dusty old books but as an 'archive of human knowledge'.
A slightly more precise analogy is probably 'a lossily compressed snapshot of the web'. Or maybe the Librarian from Snow Crash - but at least that one knew when it didn't know ;)
It’s an analogy.
And intelligence is an analogy of lossless compression.
It's also important to say what it isn't. LLM detractors, for lack of a better word, expect an oracle and then when they find out it's just a lossily compressed blob of human knowledge with natural language as a query interface they say the tool is useless.
I've got my opinion on whether that's useful or not and it's quite a bit more nuanced. You don't zoom-enhance JPEGs for a reason either.
An oracle was expected because that's what everyone kept saying it was or would be. If LLMs were shown and demonstrated realistically people would think they were really neat and find ways to use them. Instead I'm told I have phd™ level intelligence in my pocket. So of course people are going to be mad when it gets stumped on problems my 4yo could figure out.
There are several human endeavors for which we select people of high aptitude and crush their souls in very demanding postgraduate professional education so they can remix a knowledge base that's difficult for humans to master and impossible for humans to fully encompass.
The worm in that apple is that you still need educated humans to catch the erroneous LLM output.
Aren't the detractors the ones who know for a fact that it's a lossily compressed blob of knowledge and don't blindly fall for the hype?
most of us here know that (I hope), the difference is in the declaration of uselessness.
The problem is that figuring out which bits are the wrong ones is as much or more work than reading the relevant documentation. I think the main value is that it has a unified interface rather than 5000 different websites that you need to learn how to navigate.
> You don't zoom-enhance JPEGs for a reason either.
Tell that to the Google Pixel product team:
That's both perfect for this thread and terrible in general.
If you can see the analogy between text and pictures, it drives the point exactly the right way: in both cases you expect a database to know things it either can't or forgot. If it had a good picture of the zoomed in background it could probably generate a very good representation of what the cropped part would look like; same thing works with text.
Mental model:
A LLM is basically a program runtime. Code in -> output. There's a P(correct output|program), and better your model or the program, the higher it is. Even a bad model can produce the right output if you feed it the right program -- the hardest output is easy if the program is just "here's the output I want you to produce, parrot it verbatim". The key is being able to search for a program that has the highest marginal P(success) efficiently.
I recently remembered a question I had a decade ago.
Why 1 nostril is always clogged up when breathing and it seems to switch now and then?
It's magnificent that I can finally get an answer for that and would never imagine it is completely natural. Never learned about it at school.
No way I can find this via search engine as it just gives me SEO garbage or anecdotal silliness.
I've been going back and getting answers to many questions I previously couldn't.
Do you care if the answers are right?
I just Googled the exact phrase "Why 1 nostril is always clogged up when breathing and it seems to switch now and then", and the first result was an explanation from a major medical center about it (https://health.clevelandclinic.org/why-do-i-sometimes-get-st...). Perhaps there's other questions search engines don't do well on - but why don't you consider the AI answer to be "anecdotal silliness" in the same way as you would if it were just some guy on Reddit spouting off?
Stop “encyclopedizing” LLMs.
Why do we treat confident delivery as a proxy for accuracy or truth?
It could be non-lossy if it would actually reach out to an encyclopedia.
If one takes it as a language engine which translates human language into API calls, and API call results to human language, it would appear to be a non-lossy encyclopedia.
It is the basic building block which enables computers to handle natural language.
The simulated intelligence is proof of its capability as a language model, but it is often so dumb that it is doesn't feel like a "knowledge model".
It annoys me greatly that LLMs are not used primarily for search in this way, but rather search is used to disguise them as something else for the purpose of fooling people into parting with cash.
Even the actual search engines aren't using them this way. Google's "AI Overview" is actively harmful to trying to learn anything you aren't already familiar with.
RAG is one of the coolest things I've ever used with an LLM, and it would be exponentially more helpful to me in the majority of the AI tools marketed to me than the nonsense they do implement.
The first thing I tell the juniors under my supervision: any LLM is not a fact machine, even though sometimes it pretends to be. Double check everything!
The thing I always tell those who heavily trust its output is to ask it something you either already know the answer to or are something of an expert in; the flaws become much more evident.
The old joke is that you can get away with anything with a hi-vis vest and enough confidence, and LLM's pretty much work on that principle
A super heavy overconfidence of any LLM is what confuses a lot of people.
My company went head first into AI integration into everything.
I'm counting down the days until some important business decision is based on AI output that is wrong.
That had happened already.
I am continually amazed by how often something mentioned on Hacker News comes within an unreasonable proximity to something that I have done recently.
I couldn't get something like that done one-shot with Claude. On the other hand, Claude did give me a lot of assistance at writing this
https://gist.github.com/Lerc/43540d8d581b2be8155a6a4e6e85c94...
Which is a Micropython setup of a ST7789 SPI display on a RP2350 using multiple DMA channels to provide a live updating paletted frame buffer. Once setup, you write to the SRAM, it appears on the display, without CPU involvement.
I started by feeding it the source of [Dmitry's](https://dmitry.gr/?r=06.%20Thoughts&proj=09.ComplexPioMachin...) C version of the paletted technique.
The chatbot, of course, emitted something completely broken, but it was enough for me to see where it was headed. By the time I got it working there were maybe no lines of it's original output left, but much of what replaced it was also LLM generated. Given I was pretty much new to MicroPython, SPI, the ST7789, and the Pico's PIO, it let me build something that I suspect If it were doing it alone, I would have given up before getting it working. (probably when I put my thumbnail through Display #1)
When I get a chance, I'll tidy it up properly, and put it on github.
At the moment I'm playing with Gemini to see if I can make a tile+sprites mode that generates the scanlines as they go to the display (without using CPU)
I think that the natural language understanding capability of current LLMs is undervalued.
To understand what the user meant before LLM's we had to train several NLP+ML models in order to get something going but in my experience we'll never get close to what LLM's do now.
I remember the first time I tried ChatGPT and I was surprised by how well it understood every input.
It's parsing. It's tokenizing. But it's a stretch to call it understanding. It creates a pattern that it can use to compose a response. Ensuring the response is factual is not fundamental to LLM algorithms.
In other words, it's not thinking. The fact that it can simulate a conversation between thinking humans without thinking is remarkable. It should tell us something about the facility for language. But it's not understanding or thinking.
I know that the "understanding" is a stretch, but I refer to the Understanding of the NLU that wasn't really understanding either.
I’ll add my own questionable analogy:
Physicians were (one of the) first LLMs (in terms of functionality).
Their job is to handle common cases, and have broad enough knowledge to point you in the right direction as needed
Less than 1% of an LLM is a lossy encyclopedia.
The other 99+% is all of the lossy knowledge that isn't even in encyclopedias in the first place.
Including going much, much, much deeper than e.g. Wikipedia in many areas. So there it's not "lossy" -- it's effectively the opposite, i.e. "super resolution".
And very, very little of what I look up using LLM's is anywhere in Wikipedia to begin with.
Outdated or terrible documentation would leads LLM giving a unexpected answer, and that would mislead me!
You think there aren't outdated or terrible articles on Wikipedia? Or the internet in general?
The internet misleads you. Which is why we develop good BS detectors, and double-check information as needed. There isn't perfect information anywhere. Even official docs are often riddled with errors and inconsistencies.
I did a similar experiment and found that GPT5 hallucinates upto 20% in domains like cricket stats where there is too much info to memorize. However interestingly the mini version refuses to answer most of the time which is a better approach imho. https://kaamvaam.com/machine-learning-ai/llm-eval-hallucinat...
I have been explaining LLMs to friends and family by comparing them to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better.
I just posted a full write up of the idea to HN: https://news.ycombinator.com/item?id=45103597
Its way more than this, today I got it to write some code, and a bit was wrong, I told it why it was wrong and it said oh ok, and toddled off and fixed it - in all the places that were wrong. This is way more than an encyclopaedia could, lossless or not.
The more I use it, the more surprised I am at its capabilities, it really is like a beginner dev, but one that doesn't learn from its mistakes (yet anyway). I find myself asking it to do more and more.
Not sure this is a novel perspective, yet the author in his normal style claims he has discovered the atom.
How I like explaining them to people:
"An LLM is a search engine that can remix."
That's how I've always characterized them. But if you think about it, it's not really true.
The LLM is "lossily" containing things an encyclopedia would never contain. An encyclopedia, no matter how large, would never contain the entire text of every textbook it deems worth of inclusion. It would always contain a summary and/or discussion of the contents. The LLM does, though it "compresses" over it, so that it, too, only has the gist at whatever granularity it's big enough to contain.
So in that sense, an encyclopedia is also a lossy encyclopedia.
I agree and like the analogy. And it's lossy to become useful as an AI in the first place. The "Learning" process has two effects on a machine learning model: In the beginning, it memorizes the facts it is trained on. But at some critical point, when it has no more capacity to memorize more facts, it starts to generalize. (This is why it's harder to train large models - large training datasets are needed.) And generalization is where AI models become very useful: For coding, writing poems, or any other task where memorization is not sufficient.
What I tell my non-technical friends when they ask about AI -
The goal of an LLM is not to give you 100% accurate answers. The goal of an LLM is to continue the conversation.
Exactly: at their core the only task they ate capable of is "complete this document."
It just turned out that document completion was far more effective than anyone anticipated.
A lossy encyclopedia might be the most useless thing ever.
An encyclopaedia is a lossy representation of reality.
I have a nvidia jetson orin nano with llama.ccp/ollama. Gemma3:4b / Gemma3-4b-it is awesome, reasonable fast (even with vision - i think its like 15t/s) and all that on a raspberry sized microcontroller.
Simons llm client tool is on every machine and I use it daily
It is but when you take that stochastic token prediction machine and combine it with post, you can extract a graph that resembles intelligence that is then stored in a very disorganized fashion. This works because it is how we process and express /communicate information.
I think we will start seeing stateful AI models within the next couple of years and that will be a major milestone that could shake up the space. LLM is merely a stepping stone.
I like this lossy compression / decompression analogy for coding too: when you prompt for a feature, you are basically asking to decompress the meaning of your ask into your existing code. Any semantic gap in your prompt will be filled with plausible glue, ie. the LLM makes decisions for you. A good prompt minimizes the glue needed and reduces the potential for really crappy outcome, but it's always a possibility!
Great analogy! Puts a succinct labele to my mental model around it Will definitely use this.
Though, with lossy media it is obvious when it is lossy. Yet LLMs will exhibit overconfidence to tell you facts that don't exist. Not suggesting LLMs exhibit human characteristics, just that there is yet a better analogy out there :)
I find this a very useful analogy. Although it does not factor all the LLMs capabilties in.
I can also see them as very clever search engines, since this is one way I use them a lot: ask hard questions about a huge and legacy codebase.
These analogies do not really work for generating new code. A new metaphor I am starting to use is "translator engine": it is translating from human language to programming language. It in a way explains a lot of the stupidity I am seeing.
I'm getting so annoyed with the omnipresent mainstream model trend of cramming more and more data in models and advertising that as an improvement.
One, that's got to be a recipe for All Overfit All The Time, or at least I don't understand how you avoid overfit when the expected output is a reconstruction of atomic, individual facts. And two, this mass of embedded parameters has got to make them costlier, less efficient to run, as well as plain less useful, than if they were backed by e.g. knowledge graphs (ideally annotated with sources of truth), and were optimized toward querying such graphs robustly as opposed to trying and necessarily failing to remember the contents in exhaustive detail.
Model weights are a terrible way to store data. Surely I can't be the only nerd out there who feels that a model should not try to be an encyclopedia and should certainly never pretend to be one?
I suppose it boils down to marketing. Models are sold as "smart", and what smart is supposed to look like in Western culture is confidently spouting fact-shaped sentences about any topic. So that's what we're getting. What a waste.
I think it’s an old analogy, and a good one. LLMs are for knowledge what mp3s were for audio.
This was widely discussed in the past years as well.
The older analogy was to JPEG compression - I linked to that in my post (the Ted Chiang link). https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
This analogy has been used for machine learning since way before ChatGPT, my co workers and I were discussing this idea but for LSTM models in roughly 2018.
What’s old is new again.
Are you talking about lossy compression or a lossy encyclopedia?
I work in the AI field and I've heard every analogy possible for LLMs since ChatGPT was released, including many variants of Encylopedias (Grolier/Encarta etc), although, analogies to encyclopedias have always been (for me) quite limited as encyclopedias are just static data-stores and also are riddled with errors and out of date (just like LLMs). LLMs however can provide output which is completely novel and has never been seen before.
Pre LLMs we had already been working on content generation using prior tech, including texture generation pre diffusion models and voice generation (although it sounded terrible). At my company we spent hours discussing the difference between various data compression algorithms and ML techniques/model architectures and what was happening inside ML models and also, inside our brains! But even then we didn't think anything we were discussing was novel at all, these ideas were (and still are) obvious.
Anyway, back on the topic, of the LLM as encyclopedia, you can USE an LLM for encyclopedia-like workloads, and in some cases it is better or worse than an actual encyclopedia. But in the end, encyclopedias are written by flawed humans just like all the data that went into training the LLM was written by flawed humans. Both encyclopedias and LLMs are flawed and in different ways, but LLMs at least can do new things.
I actually think a better analogy to an LLM is to the human brain than an encyclopedia, lossy or not. I think we massively overrate our brains and underrate LLMs. The older I've gotten the more I realize the vast majority of people talk absolute rubbish most of the time, exaggerate their knowledge, spout "truths" which are totally inaccurate, and fake it till they make it throughout most of their life. If you were fact checking the entire population on everything they said on a day to day basis, I think the level of "hallucination" would be much higher than Claude Opus 4.1. That is, I think our level of scrutiny is MUCH higher for LLMs than it is for our friends and co-workers. We tend to assume that if another human says something to us like "New York has a higher level of crime than Buenos Aires", we take them at face level usually, due to various psychological and social priming. But we fact check our LLMs on statements such as these.
Also: "An LLM is a lying encyclopedia"
Using his analogy I would see it more like this : for the same overall quantity of bytes it is more broader but also lossy as being compressed. You have ten wikipedia without the precision of one.
Encyclopedias are already lossy. Top LLMs seem to be less lossy than top encyclopedias tbh.
It's also like someone who knows lots of facts but bad at remembering where they exactly learned it from.
And likely when mid-story to invent parts of the story to fill the gaps, rather than admit it is wrong.
Maybe the LLMs aren't so different from us.
and frequently confuses them with other facts they know
I have this idea that I'm working on that I can't decide if it is just a silly idea or actually applies directly here: https://github.com/Kilo-Org/alex-treBENCH
I think models will also become snapshots or time capsules (with obvious and non-obvious biases) that archaeologists of tomorrow (like in 500 years) will use to understand us and the change in society (e.g. how the models themselves change over timefrom encyclopaedic standpoint).
Every encyclopedia is lossy, by definition. Even the most expansive holds a tiny fraction of human knowledge (which is a fraction of what we could know).
On the other hand, it’s not worse than other analogies.
This.
Also every encyclopedia is full of things that are wrong. People seem to be forgetting this basic issue. Any given authoritative, well respected source will contain mistakes, errors of omission, and downright lies. Part of a proper education used to be that when writing things, you need to site your sources, and the sources can't just be an encyclopedia. I use LLMs a lot, but if anything is really important, I'm going to fact check it and look for other sources.
It's an encyclopedia with noise injected. Which devalues the whole concept of an encyclopedia. It could be 10% of what it is, as long as it would be correct it would increase in value.
A "language model" describes itself already; it is a "model" of human languages.
I feel like thats also true for my brain :)
The Infinite Monkey Theory for LLMs.
It’s a lossy encyclopedia that can lie to and manipulate you. In that use case, it’s fairly useless because you cannot intrinsically trust its answers without performing additional testing and research, in which case you would’ve been better off learning new things than making sure an LLM wasn’t lying to you.
> It’s a lossy encyclopedia that can lie to and manipulate you.
So can a traditional encyclopedia.
We're at such a strange point where even school children knows that something like Wikipedia isn't necessarily factually correct and that you need to double check. They then go and ask ChatGPT, as if it wasn't trained on Wikipedia.
We haven't reached the stage yet where the majority of people are as sceptical of chatbots as they are of Wikipedia.
I get that even if people know not to trust a wiki, they might anyway, because, meh, good enough, but I still like us to move into a stage where the majority is at least somewhat aware that the chatbot might be wrong.
To be fair, most people aren’t even critical of Wikipedia. They read an article, consume its content, and believe themselves competent experts without digging into the sources, the papers, or the talk pages for discourse and dissent.
Giving LLMs credibility as “lossless encyclopedias” is tacit approval of further dumbing-down of humanity through answer engines instead of building critical thinking skills.
No, I agree that most people aren't critical (critical enough) of Wikipedia. My point is that many of them know that they should be.
True, but in that case we call it “errors” or “propaganda”, depending on the context and source. Plus the steep costs of traditional encyclopedias, the need to refresh collections with new data periodically, and the role of librarians, all acted as a deterrent against lying (since they’re reference material).
Wikipedia can also lie, obviously, but it at least requires sources to be cited, and I can dig deeper into topics at my leisure or need in order to improve my knowledge.
I cannot do either with an LLM. It is not obligated to cite sources, and even if it is it can just make shit up that’s impossible to follow or leads back to AI-generated slop - self-referencing, in other words. It also doesn’t teach you (by default, and my opinions of its teaching skills are an entirely different topic), but instead gives you an authoritative answer in tone, but not in practice.
Normalizing LLMs as “lossy encyclopedias” is a dangerous trend in my opinion, because it effectively handwaves the need for critical thinking skills associated with research and complex task execution, something in sore supply in the modern, Western world.
> Normalizing LLMs as “lossy encyclopedias” is a dangerous trend in my opinion, because it effectively handwaves the need for critical thinking skills associated with research and complex task execution
Calling them "lossy encyclopedias" isn't intended as a compliment! The whole point of the analogy is to emphasize that using them in place of an encyclopedia is a bad way to apply them.
That might’ve been the author’s intent, but the comments in this thread (and downvotes of my opinions) suggest that a non-zero number of people believe that analogy to be the best justification yet for LLMs-as-answer-engines that shouldn’t be assailable by dissenters or critics.
So long as people are dumb enough to gleefully cede their expertise and sovereignty to a chatbot, I’ll keep desperately screaming into the void that they’re idiots for doing so.
To put even more simply, and IMO more interestingly:
LLMs are compression algorithms
Lossy encyclopedia is an understatement. It merges pieces of information from different contexts to create new ones that look plausible.
That’s not the title of the article (granted, it doesn’t have one), and the author calls the analogy “questionable”.
Most analogies are questionable in my experience - I find calling something a "questionable analogy" makes it a tiny bit less likely that people will pick it apart with thousands of reasons it's not an exact match for what it's describing.
In the HTML's <title>-tag, it's called "Lossy encyclopedia".
That’s exactly what I called it when a friend showed me it first time.
An LLM is a lossy compression before all else. Then after u can call it names.
Interestingly almost every single comment on this post is either trivial or incoherent compared to HN discussions from 10 years ago. I wonder if interecting with LLMs degrades the human capacity for coherent thought?
I think its more that most reasonable people already stopped going into these LLM discussion threads a long time ago.
At the start of a hype cycle there is a lot of good discussions, then most reasonable people have established their opinions and stop engaging with it.
More like a fuzzy encyclopedia
I have another analogy.
LLM are animatronic rubber ducks.
https://en.wikipedia.org/wiki/Rubber_duck_debugging
( and obviously like all analogies - this one is lossy )
I definitely agree with that, at least this is how I use chatGPT in 99% of the cases.
Chain of thought seems to be an extraction algorithm for information buried deeper.
The models hold more information than they can immediately extract, but CoT can find a key to look it up or synthesise by applying some learned generalisations.
As far as analogies go I prefer approximate database
lossy encycopledia that can also do some short-term memory (RAG) things.
If I encounter someone today who repeats this I’m going to call them a scholastic parrot.
Wouldn't it be better for users if, rather than having to puzzle this out (how to provide examples, etc) the LLM was somehow aware of its lossy areas, and replied "I don't know"?
Or maybe, to be more useful: "I don't know, but if you give me an example maybe we can figure it out"?
The problem is not only that it resembles a "lossy encyclopedia", but also that it's an extremely confident encyclopedia that doubles down on the confidence even when it doesn't have the data.
Pretty sure that's why we're all building RAGs, to account for the missing information in the LLMs direct memory...
A "JPEG for knowledge" is how I've put it.
Another metaphor: LLMs are sketches, not technical drawings. A sketch is not supposed to be the final product; it is exploratory, not definitive.
If you used sketches to build a house, it has a nonzero chance of falling down. Likewise, if you made technical drawings as a way to brainstorm house designs, the process would be overly rigid and extremely inefficient.
Characteristic of an encyclopedia is that is has structure on several levels.
The output of an LLM does not. It can be coerced into faking structure, but that is quite brittle and still just an emulation.
So, it's basically useless or even harmful.
Yes, if you try to use it as if it was an actual lossless encyclopedia.
One of the reasons I like this analogy is that it hints at the fact that you need to use them in a different way - you shouldn't be looking up specific facts in an unassisted LLM outside of things that even lossy compression would capture (like the capital cities of countries).
The only usages I found so far that are somewhat useful is to generate plots with python and how to use the various libraries for machine learning. Also massage some hastily written text. Both involved haste as I needed some result fast.
Everything else is mostly playing around and harmful to learning.
This sounds pretty helpful. If I'm trying a new lib that I want to do something specific with, I paste all the documentation in and interrogate the LLM about it, then cross-reference with the docs. Usually much faster to do what I want than just CTRL+F or writing a SO question that gets immediately marked as duplicate because some other question is vaguely related.
For language learning, it's terrible and will try to teach me wrong things if it's unguided. But pasting e.g. a lesson transcript that I just finished, then asking for exercises based on it helps solidify what I learned if the material doesn't come with drills.
I think writing is one of the things it's kind of terrible at. It's often way too verbose and has a particular 'voice' that I think leaves a bad taste in peoples' mouths. At least this issue has given me the confidence to finally just send single sentence emails so people know I don't use LLMs for this.
My frustrations with LLMs from years ago has largely chilled out as I've gotten better at using them and understanding that they aren't people who I can trust to give solid advice. If you're careful about what you put in and careful about what you take out you can get decent value.
Just like a hammer if you're trying to cook with it.
Yes and working out how to disentangle the information storage mechanisms from say language processing is a massive area of interest. Only problem with Attention Transformers imo is that they're a bit too good :p
Imagine a slightly lossy compression algorithm which can store 10x, 100x the current best lossless and be able to maintain 99.999% fidelity when recalling that information. Probably, very probably a pipe dream. But why do large on device models seem to be able to remember adjust everything from Wikipedia and store that in smaller format than a direct archive of the source Material. (Look at the current best from diffusion models as well)
If the world could just internalize this simple fact - it's a search engine of all of the knowledge it has ingested, with a human-language interface (and quite a good one at that). It doesn't think. It isn't intelligence. Calling it "AI" from the beginning was a grift from the companies developing the tech.
Since late 2022, I've used LLMs extensively for coding, copywriting, research, and everything in between and I've slowly gone from "this is amazing" to "this is extremely useful but probably extremely overhyped" to "this might not actually be all that useful at all". Where accuracy matters, fact checking these things takes as much time as just doing the work manually. I think its most useful application is as a tool for spammers and bots, and that doesn't exactly bode well for the companies spending hundreds of billions of dollars on the tech.
agent reframing:
llm is a pretty good librarian who has read a ton of books (and doesn't have perfect memory)
even more useful when allowed to think-aloud
even more useful when allowed to write stuff down and check in library db
even more useful when allowed to go browse and pick up some books
even more useful when given a budget for travel and access to other archives
even more useful when …
brrrrt
Encyclopedias are lossy Encyclopedias.
Also, humans hallucinate more than LLMs.
Do they hallucinate more? How many words per day do humans spit out vs the LLMs, and what is the hallucination rate? Also, don't count humans saying, "I'm not sure but I think..." as a hallucination, because it's not.