How does ChatGPT read your text: RAG and Context Window

How ChatGPT does not read: An unhelpful mental model

When people imagine a Large Language Model like the one powering ChatGPT reading through a text to answer their question, they have an image in mind of a little bot going through the text word by word, taking notes and then writing a little report.

This is not what it is doing at all. The LLM does not do anything resembling what a human does when they read a text. What it is doing is weird and hard to wrap your head around. And what's worse, knowing how it reads text does not necessarily help you predict exactly what it will and will not be able to do.

Two settings

And what's even worser, is that in different settings, it's using very different approaches to "reading" your text. And they're both weird in different ways. The two settings are:

1. Context window: Text that the LLM can see at once as a prompt (this could be quite long but rarely longer than a short book - different models have different limits but most of the time, think about the length of a newspaper article or at best an academic paper). This will usually happen inside a chat.
2. RAG (Retrieval Augmented Generation): Texts or a collection of texts that are too long to fit into the context window. For example, the files you might upload into a custom GPT, Poe bot or a Claude project. The LLM does not do this on its own but it has to be marshalled to do it by the programmers of the interface.

How does an LLM answer questions about your prompt? Attention is all it needs

Metaphor

So, this is weird. The way an LLM reads the text you give it is not the way you would do it. The closest comparison to what a person may have an experienced is writing down a phone number. Unless, they really focus, most people cannot write down a whole phone number at once after a single look. So they, glance at the number, try to recognise a pattern and write down as many digits as possible, glance again and write down some more. They try to combine attention and pattern recognition in a superfast process.

That's exactly what an LLM is doing. It glances at the whole text as you would glance at a phone number, tries to quickly recognise a bunch of patterns, writes down one tiny bit in response, glances again, writes down another bit, glances again (including the things it has already written down), and so on. Usually, these bits are parts of words or sequences of characters called tokens.

Reality

This metaphor is fairly general and leaves out many important details. The three concepts that are key to understanding here are tokens (chunks of words that the LLM sees and generates), embeddings (large vector representing the 'meaning' of each token that the LLM actually uses), attention (a process through which the LLM decides which tokens are more important for it to decide what to generate next).

All of those are tied together through many many layers of engineering and math glue. For example, the token vectors are multiplied by each other in one layer to determine which ones are more relevant. But that often generates weirdly large numbers, so every so often there's a normalisation layers, that just cuts off the peaks (kind of link normalisation in audio editing).

Or, when the tokens from the prompt are sent to the next layer, the system has no idea which order they are in. So the order has to be injected back in through a layer called positional embedding.

And finally, when the system has gone through about 30 layers of the process of vector multiplication and normalisation, the result is converted into a probability distribution over all 50-100 thousand tokens, the top k are chosen and the system essentially rolls a pair of dice to pick one at random.

And all of this does not generate the whole text but just one more token. When that token is added to what's been there. The system forgets everything that came before, takes a quick glance at the whole text (including the token it just added) and starts over.

Yes, this is slow and computationally expensive. But there are lots of tricks that can be used. Such as caching of previous activations or using a cheaper process to quickly generate a bunch of tokens and only use the more expensive process to check if there's a problem. But ultimately, under the hood of a Large Language Model is a lot of matrix multiplication and weird tricks for it not go completely off the rails.

Implications

So this is weird, interesting but also not very useful. Because LLMs can do all the thing we all know they can do with just this method. If all you knew about them was that this is how they process text, you'd not think they'd be able to succeed. (And I've kept some of the juicier details to myself, here).

But the one big lesson here is, that the LLM does not have an inner sketchpad for keeping ideas to use in a while and also is never doing anything resembling "copy/paste". Everything it's telling you about the text, it's arrived at the same you would by glancing at a phone number one or two digits at a time.

This also makes things like slight errors in quotes or outright hallucination make sense. The LLM does not actually have any component to do the sort of things that a person reading would do. And if you're asking "why don't they just build it in", you're welcome to try and become very rich or famous.

How does an LLM answer questions about a collection of texts? It is not what you think

Metaphor

Imagine you wanted to find an answer in a book or a from a stack of papers on your desk. But there's a limit. No matter how complicated the question, you'd only have one method of coming up with the answer.

You can never read the text first or do a search for key words, instead you have a look at the table of contents and have to guess which chapter the answer might be in and then read that. Not a very reliable way of doing that.

But even that would be too easy. Instead, the table of contents is not of chapters but of paragraphs. And each paragraph has a only a quick description that does not contain any of the words in it, just a general description that can help you guess if it may be a general match for the question you're asking.

And then, you don't get to choose how many you look at. You have a set limit - maybe 5 or 10 but probably not more than 20. Then you get to read those and formulate the answer based on what's in them. Nothing else and you never get to go back.

Reality

This is pretty much what happens inside a collection of documents you are using LLMs to answer questions about using a method called Retrieval Augmented Generation (RAG). There will be many more tricks to make it work but essentially, this is it. There are three components to RAG:

1. Preprocessing: This mostly happens before but could happen during the process. During this process, the text is broken up into chunks (the size and methods vary but usually about a paragraph length and there are often overlaps). Then each chunk is given an embedding (a large vector - about 3,000 dimensions - representing the overall meaning). That number is then stored in a special vector database (store) that makes it easy to search through them quickly and find ones that are similar to each other. You can think of it as a sort of semantic fingerprint.
2. Retrieval: When the user submits their question, the question is given an embedding exactly the same way the chunks are. That is, it's given a large number representing a vector in a semantic space. Then the system searches the vector store for embeddings that are near it in the semantic space. If you're wondering how it knows, it's exactly the same algebra used for tiny vectors with two numbers. One simple method is example cosine similarity. This process retrieves a set number of chunks (possibly 10-20)
3. Generation: Then your query and the retrieved chunks are sent as a prompt to a large language model which will then read them and write your response pretty much in the way described above. Essentially, the LLM is responding to a prompt about the length of a newspaper article composed of disconnected chunks. This is often surprisingly good but it has limits.

Implications of RAG

If that sounds very ineffective and unreliable that's because it is. There are some consequences:

1. RAG is not good at finding pieces that don't fit. With the first approach you can ask things like "What is out of place". But that's not possible with RAG because it never sees all of the texts.
2. RAG can miss important chunks or retrieve chunks that are the opposite of what you want. The system knows the embeddings (semantic fingerprints) are near each other in a semantic space but not what's in them. Which is why antonyms and synonyms will often be retrieved together.
3. The final step is subject to all the limitations of the normal generation such as hallucination. So you may get an answer that is 'true' but not contained in the text or one that is true in general but the text says something different.