"Training a model", I don't think those words mean what you think they mean

TLDR;

(Claude generated) AI model "training" involves five distinct layers:

1. Pre-training: Creates the foundational model (months, expensive)
2. Fine-tuning: Adapts the model for specific tasks (days, moderate cost)
3. Preference optimization: Aligns model outputs with human values (weeks, moderate cost)
4. Prompting: Guides model behavior at runtime (seconds, cheap)
5. RAG: Incorporates external knowledge on-the-fly (minutes, low cost)

Only the first three actually change the model. The last two are runtime modifications accessible to most users.

Why it matters: Understanding these layers is crucial for making informed decisions about AI implementation, resource allocation, and ethical considerations. It helps clarify what's possible at different stages of AI development and use.

Who should pay attention: If you're involved in decision-making about AI, whether as a policymaker, business leader, developer, educator, or concerned citizen, this knowledge will help you navigate the AI landscape more effectively.

"Training a model", I don't think those words mean what you think they mean

Do you know what "model training" means when you hear people say things like "the AI companies are training models on the user data", "the model is trained not to respond to illegal queries" or "I trained the model to on the company documents"? Don't worry, there's a good chance they don't either. Model training can me a lot of different things in different contexts.

Meaning of "training" before Large Language Models

Before Large Language Models people would say things like "we trained a model to predict what movies you will like based on what you watched" or "the model learns what the customer likes as they shop". Or "the model is trained to predict how likely you're to buy a house". This all sounds very fancy but in fact, it was no more sophisticated than slightly more complicated line going through a bunch of data points on a chart in Excel. You have a function that plots that graph, you feed it one number and it spits out another.

As you can imagine, it's easy enough putting more data into an Excel spreadsheet and improving the function to fit a better line to it as we go. In those days of data science all of those sentences sort of meant this one thing. Fitting a better line to a bunch of data points so that when we had a new input on the X axis we'd get a better prediction of the corresponding point on the Y axis. The reason we needed data science and learning was that we needed more axes and actually deciding what the units on those axes were was a big deal. Back then it was call feature engineering.

How LLMs changed the meaning of "training"

NONE of that is true of Large Language Models (for the most part). People still occasionally talk about functions and curves in this context such as the loss function or the activation function but they play a different role than the functions of old that plotted lines through a bunch of data points.

There is a very easy technical way to talk about training and learning Large Language Models but in effect none of it is like the learning and training the old models. This is a bold statement, because in fact, there is a lot of continuity in the mathematics and logic behind these models - it is easy to point to papers going back to 1950s that contain the same ideas. But for a non-expert, it is much easier to jettison any of the old ideas about machine learning or model training and start over.

So what does model training mean? There are two ways to answer this question.

1. What is actually the thing that "training" is and
2. What are the different meanings associated with the common contexts of "model is trained" in common parlance such as the statements I mentioned above.

Here I try to answer question 2 to hopefully give you a better sense of 1.

When people say "model learned" or "model was trained", they usually mean what was the thing that made the model to behave one way or another. And there are broadly five completely different things that can do that and some of those have almost nothing to do with 'training'. Here are the five:

1. Pre-training
2. Fine-tuning
3. Preference optimization
4. Prompting
5. Retrieval Augmented Generation

Five layers of what people call Large Language Model training (not all of which are actually training)

Summary table

(Generated by Claude)

Pre-training

This is the P we know from ChatGPT. Pre-training is what gives the model its raw power: the ability to generate coherent text in many languages in response to a prompt (which could include computer languages and the prompt could include things like reasoning puzzles). It is the most expensive and time consuming part of the process.

Some of the largest models are trained on trillions of words and the training takes months. This is done once and there is no live updating of this model while it is being used. But most importantly, this is NOT the model you are using when you're talking to ChatGPT because it would often give you gibberish.

Can you or should you think about training your own model in this sense? Almost certainly not. If you're having to read this, you should probably never consider training your own model. You almost certainly do not have enough of your own data, available computing power and enough engineers. But most importantly, once you had this model, it would not be any better than the models others have trained because the things you're hoping you would fix by training a model on your data are inherent to the model architecture. And if they could have been fixed, somebody who has all the data, compute and engineers would have done it.

But having said that, if you have a million dollars (or more) and lots of good data in a narrow domain that may not be represented in the general data, OpenAI will work with you and help you develop a custom model not pre-trained exactly from scratch but enhanced with it (this gets into finer details of pre-training because even that is not exactly one thing). The one case study of this is the legal model Harvey. Harvey worked because it had lots of US case law data which is a relatively narrow domain. Bloomberg tried it with business data and while it sort of worked for them, it was not the same success.

Terminological note: This stage of the training is done using a Machine Learning architecture called a Transformer (the T in ChatGPT). If you know ML history, you may remember that one of the first things people talked about was the difference between unsupervised, supervised and reinforcement learning. This stage is not exactly unsupervised because the training objective is to learn to complete text that the model sees. But because it is done without curated and labeled examples, it is not exactly supervised either. Sometimes this is referred to as self-supervised but you may also hear that it is unsupervised. This would not be that important on its own except, the next two phases are called supervised and reinforcement, it may be useful to be able to slot this into that relatively famous distinction.

Fine-tuning (SFT)

The pre-trained model is powerful but despite all those trillions of words it was trained on over months, it does not really know that it should always answer a question with an answer or what exactly a good essay looks like. This is where fine-tuning comes in. Or more accurately supervised fine tuning (SFT).

Because the finetuning is instructing the model is giving it instructions about what output should look like, it is often called 'instruction finetuning' and models that have been instruction fine-tuned often have the tag 'instruct' applied to the name. In fact, the breakthrough came in early 2022 when OpenAI announced InstructGPT which later became ChatGPT.

How does it work? In machine learning, supervised training simply means training on labelled examples. And that's what is done here. After all those trillions of words, the model is given millions or even thousands of carefully curated examples, often hand-crafted by experts, to learn how to do things with its pre-trained raw power that we as users actually find useful.

You can think of the Large Language Model as layers of sieves into which you pour at every step about 100 thousand possible next words and slowly that filter out the ones that don't best. Fine-tuning modifies the final layers of the sieves.

Finetuning is much cheaper and quicker but is also not done live. It does not need nearly as much data but it needs to be very high quality data, often curated or even created by experts in the domain just to fit the need. But it is not data with 'reliable facts'. Finetuning does NOT (for the most part) give the model new skills or knowledge. It just gives it examples of good outputs for given inputs. For example, questions and answers, reliable JSON formatting, what a script looks like. The model has the raw power from pre-training to generate all of those but not without a lot of prompting.

This is what made ChatGPT possible. GPT-3 was available for 2 years but because if was not instruction fine-tuned, very few people could do something useful with it. If you are using a model inside a chatbot, you are always using one that has been instruction-tuned. It would be useless without it. Most models that are released have been fine-tuned but some smaller and open source models are released in both versions and sometimes without any fine-tuning at all.

Finetuning is something that is within reach of more people but most of the time you should not consider it. The two reasons not to consider finetuning are:

1. It does not actually give the model new 'knowledge' or 'skill', it just tells it what good output looks like. And most of the time people want the knowledge and skill.
2. It requires high quality, curated data set that most people who think they have lots of good data do not actually have. But even if you do, it will not 'teach' the model 'facts' in that data, just format.

Not all fine-tuning is alike and even if you're reading this, you can try a very basic version of it yourself through OpenAI Playground. OpenAI will let you upload labeled examples (100s will do) and then you will have a custom API you can call to try if it's better. The API is more expensive and you cannot do it all on the models but for some people it is a useful solution. However, you are unlikely to be the one but trying it will show you that getting even 100 good examples is a lot of work even if you started out thinking you have lots of great data.

Note: Finetuning is not always straightforward and can lead to unintended consequences. For instance, it can make the model perform slightly worse than its pre-trained variant. Which is why sometimes models without fine-tuning can be more useful. But in practice, a non-finetuned model is almost always less useful and harder to sue.

Preference optimization (RLHF or PPO/DPO)

Most commonly this is called RLHF which stands for Reinforcement Learning from Human Feedback. What this does is give a chance to the model to learn a policy (technical term) about which direction to choose when there are many options to complete text from the perspective of textual accuracy. In other words, things like should the response be long or short, should it be offensive or not, should it express a political opinion?

In a general sense, this layer is done to change the bias the model may have learned from the raw data. In the most stereotypical case, this is done by humans reading two outputs of the model to the same prompt and choosing one they like better. Humans are giving the model feedback, which is where the HF in RLHF comes from.

This is both incredibly expensive but also fiddly. Reinforcement learning is hard to control and the model can learn unintended policies to lead it astray. For example, human readers tend to interpret longer text as more reliable, which is one of the reasons ChatGPT often says a lot more than is required to answer the question.

Because of the expense of getting human feedback but even more because of the arcane and often tacit expertise required to manage reinforcement learning in this context, RLHF was really only available to the biggest companies such as OpenAI or Google. But new methods were developed where even open source projects could use public datasets of human preferences (not opinion surveys but pairs or responses) to change the preference policy of the model directly.

This is now called DPO which stands for Direct Preference Optimization in contrast to the more intensive method now sometimes called PPO or Proximal Policy Optimization. It turns out that DPO may be better but than PPO but they will probably both continue to play a role. This is why I choose the label for this phase to be Preference Optimization rather than RLHF which is how it is often referred to.

Any model you will use inside a chatbot will have had its preference at least somewhat optimized and this is what can give the model slight political leanings. This is usually in part inadvertent and partly done on purpose by the companies.

Prompting (In-Context Learning)

Now we get to the first layer of model "training" that does not change the underlying model in any way. And people in the industry would not actually call that training. It's what happens when you give the model a prompt and it "learns" from it what you want to do. I know that sounds very boring (but it's actually very exciting) and not much like training (which it is not).

The original confusion came in 2020 when OpenAI released GPT-3 using a paper entitled "Language models are few-shot learners". This title may have done more damage to the subsequent discourse than any talk of hallucination. It introduced the concept of "in-context learning" (ICL) or "few-shot learning". The problem is that the "learning" in ICL is not the same as the "learning" in Machine Learning.

The "learning" of in-context learning has a very specific and dumb meaning: including examples of what you want the output to look like in your prompt. Each one of those examples is called a "shot". So "0-shot learning" is simply asking the model what you want - the sort of thing we do with ChatGPT, "1-shot learning" is giving it one example, "five-shot" five examples, etc. That is all. If you look at some benchmarks of model performance, they may include something like 5-shot next to some scores. That means the model achieved the score after being given 5 examples of the problem in the same prompt.

Now, this is an incredibly exciting and useful property of Large Language Models. You don't just have to describe to them what you want, you can give them examples. Even though you don't have to give as many examples to ChatGPT or Claude, examples are still one of the most underused prompting techniques.

In fact, when GPT-3 was released in 2020, that was the only way you could make it do anything useful such as give you code or have a conversation. It simply did not know what 'conversation' was. It could do it but because a question can be followed by other things than answers, it would very happily give you another answer or something even more random seeming if you gave it a question.

The only reason you can have a reasonable interaction with ChatGPT is because it was fine-tuned and does not need examples of every type of possible interaction. Also, because there is a limit to how long the prompt can be and sometimes you need dozens or even hundreds of examples for the model to follow them reliably, relying solely on in-context learning would be very limiting.

While few-shot learning refers to giving examples, you can use any "prompts" to "train the model" to do or not do something. You can include text such as "When the user asks you whether you are conscious, always respond by saying 'I don't think so'." When you interact with any model via a chatbot or even another app, it will always have what is called a "system prompt" which is essentially instructions like this. This is usually attached to any chat you're having without you ever seeing it.

As most people have experienced, prompting (with examples or without) is a very imperfect way to get the model do exactly what you want, but it's also surprisingly effective in many situations. Of course, none of the "learning" or "training" approaches on this list are perfect and often need to be used in complement. So in fact, when you are using ChatGPT you are experiencing all of the four layers up to know in action.

Retrieval Augmented Generation (RAG) or Talk to Your PDFs

I described how RAG works elsewhere so, let's be brief.

Often people want their Large Language Model to have their own 'local' knowledge such as information about their company's procedures or recent emails. And they say, they would like to train their model on that data. As I tried to show, none of the solutions up to know is suitable for that. The only way to do that is Retrieval Augmented Generation (RAG).

Very briefly and crudely, RAG splits up all of your documents into smaller chunks, gives them all a sort of semantic fingerprint using vectors and stores the vectors in a special database. It then assigns your query the same kind of semantic fingerprint and retrieves a number of chunks with a similar semantic fingerprint from the database. It then sends you query and those chunks to the LLM along with a prompt that tells the model to generate an answer from these.

This has many limitations but can be a very effective way of giving the model some knowledge. Anyone can try a basic version of RAG if they have a paid subscription to ChatGPT or Claude. ChatGPT calls their RAG systems "GPTs" and Claude calls them Projects. They both approach this slightly differently but they both let you upload a bunch of documents and add instructions how to answer queries based on them.

Note: There are many flavours of RAG. While most of the time when people talk about RAG, the model is not actually changed, the original proposal was to use RAG for a sort of finetuning of the model.

Read more about RAG here.