Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Fine-Tuning Overview

TLDR

Why Fine-Tune?

Fine-tuning is what creates the models that most users want to use. These days, pretrained models are not the models people think of when they hear LLM or AI. Heavily fine-tuned models like ChatGPT, Bard, or Claude are what come to mind.

That’s often because pretrained models:

Here’s an example of the LLAMA2 pretrained model versus the fine-tuned model on the same prompt. Note the difference in the response and the usefulness of the fine-tuned response versus the pretrained model.



You can think of a pretrained model like a half-built car with no bodywork, safety systems, or steering wheel. You have all the basics of what makes a car “go,” such as the engine, suspension, chassis. Technically this is all you need to start moving. But it’s uncomfortable to drive, dangerous as there are no safety systems added, and it lacks the styling that differentiates it from any other car.

A pretrained LLM on the left (minus the steering wheel), a fine-tuned LLM on the right

Figure 1:A pretrained LLM on the left (minus the steering wheel), a fine-tuned LLM on the right

Why learn finetuning?

In my opinion if you are looking for a job in the AI space and can only pick between learning pretraining and finetuning, I’d suggest learning finetuning. As a process finetuning happens many more times than pretraining, and by a wider group of people. Sticking with the car analogy you can think of it this way. There are only a handful of companies in the world that produce cars from scratch. That is from the ground up thinking through engine design, suspension, electronics. There are many more organizations that take existing cars and specialize them for a purpose. Think race cars, police cars, or even minor changes like repainting or changing out the stereo.

It’s the same with models. 10 or so companies produce high quality base models, but there are tens of thousands of fine-tuned variants built on top of them.

Even within those companies producing top quality models, like Meta, fine tuning happens many more times than pretraining. Meta explains this in their LLAMA 2 paper, where finetuning occurs 10+ times over three different model variants (Chat/Instruction tuned, and two reward models), over 4 parameter sizes (70b, 34b, 13b and 7b). Contrast this to pretraining which only occurs once.

Fine-Tuning Fundamentals

Fine-tuning specifically refers to modifying the model’s weights to elicit the type of responses we want from the model. Just like pretraining, and all statistical model training, the procedure is as follows:

  1. A model generates an output.

  2. A measure of how “wrong” the answer is, is calculated.

  3. Weights are updated accordingly.

How all fine-tuning methods work

Figure 2:How all fine-tuning methods work

The difference lies in the fact that, instead of creating examples through self-supervision and calculating loss on the next token in pretraining, fine-tuning tends to use different training example creation methods and full responses from the model. Fine-tuning is typically applied to a pretrained model, though fine-tuned models can themselves be further fine-tuned. For example, the LLAMA 2 developers started with a pretrained model, but then fine-tuned their model through 5 iterations using multiple techniques, sometimes employing different techniques in different iterations.

Fine-tuning is not just one method but an entire class of methods. The specific one used depends on numerous factors. Common considerations include:

Fine-Tuning Methodologies

Below is a list of common fine-tuning methodologies used in practice. Due to the sheer amount of written content on fine-tuning, references for each method are inline with each section.

Know that fine-tuning is a very active field, both in terms of fundamental research and empirical experimentation. Expect the methods here to rise and fall in favor, and new methods to be discovered soon.

Supervised Fine-Tuning

For people coming from a machine learning background, Supervised Fine-tuning will feel familiar. A “training set” of prompt and output response pairs is created. The model is then prompted with the training prompt, and the “predicted” response is compared. The corresponding loss is used to update the model weights.

In Supervised Tuning a prompt and a “true” response pair is used to calculate the loss for weight updates

Figure 3:In Supervised Tuning a prompt and a “true” response pair is used to calculate the loss for weight updates

The prompt and response pairs are constructed manually, and the loss is typically calculated over the entire output phrase. The weights are updated and another training loop is performed.

The benefits of supervised fine-tuning:

The downsides of creating supervised fine-tuning data are:

Supervised Fine-Tuning References

Reinforcement Learning with Human feedback

RLHF is currently one of the most popular fine-tuning methods. The core idea behind RLHF is “Instead of humans telling the model what to say, what if humans indicated what they prefer?” To implement RLHF, The model is prompted and one or more responses are generated. Users then provide their preference to be used for model reweighting.

There are two patterns: Single-sided, where only one model output is rated,
In a single-sided setup, raters rate a single response.

Figure 4:In a single-sided setup, raters rate a single response.

and double-sided, or more, where multiple outputs are ranked and/or rated.
In a side-by-side setup, raters choose between two or more different versions.

Figure 5:In a side-by-side setup, raters choose between two or more different versions.

The benefits of RLHF are:

The downsides for RLHF are

RLHF References

Reward Models

Reward models are models that rank the outputs of user models that are being trained. With multiple models, it can be easy to mix them up. From this point, we’ll make this distinction:

User model - Models that will be shipped to users
Reward Model - Models trained specifically to provide rewards to other models

Reward models can be created in various ways and essentially are fine-tuned models in their own right. For example, RLHF is often used to train the reward model, with the idea that human preferences can be learned and then reused in automated training tools. For GPT-4 training, OpenAI configured GPT-4 instances into rule-based reward models. These rule-based models used multiple sources of information (prompts, outputs from a policy model, and a human rule) to calculate the score for the GPT-4 user model output.

A model provides a preference score. This is a single-sided example. Models that can rank side-by-side examples also exist.

Figure 6:A model provides a preference score. This is a single-sided example. Models that can rank side-by-side examples also exist.

The reward model is often a copy of the pretrained model, which is fine-tuned specifically for reward scoring. One distinct difference is that the token prediction head from the pretrained model is replaced with a regression head that outputs ratings. This doesn’t necessarily need to be the case, though in my reading so far, it seems that for the largest model training at companies like Meta and OpenAI, there is a benefit for the reward model being the same parameter size as the user model it is fine-tuning.

The benefits of reward models are:

The downsides:

Reward Model References

Constitutional AI

Constitutional AI is a method developed by Anthropic to reduce human harm and further scale AI preference training. The basic idea is, instead of needing humans to specifically provide a preference between output prompts, models can be provided with a constitution of values, from which they can self learn what appropriate responses are. The novel idea here is that the model(s) critique, rewrite responses, and train themselves based on a small constitution provided by humans. The constitutional AI training process itself has many steps, and involves more than 5 models.

The multi-step constitutional AI training process. Source: Bai et al., Constitutional AI: Harmlessness from AI Feedback (2022)

Figure 7:The multi-step constitutional AI training process. Source: Bai et al., Constitutional AI: Harmlessness from AI Feedback (2022)

The benefits of this more complex process are:

  1. Writing a small set of principles is a less daunting task than rating thousands of responses.

  2. Humans are exposed to fewer harms during the training process.

  3. The training process is more scalable.

In practice, Anthropic demonstrates that this method is effective through their assessment, and puts it into practice with their own AI Agent Claude. The key question here then largely falls on writing the constitution itself. Anthropic has experimented with a number of ways to do so.

Constitutional AI references

Parameter Efficient Fine Tuning (PEFT)

As models become larger, updating the weights becomes more computationally and economically challenging. However, a simple trick to mitigate this is by not updating the weights, or at least not all of them. Similar to fine-tuning, there are numerous strategies being developed and tested for this methodology. These methods are particularly appealing to open source or “home” LLM builders as they enable significant changes in model behavior with relatively minimal cost and effort. Like fine-tuning, there are many methods and more are emerging each day. We’ll cover the two that have been most widely used, with links to other methods.

Prompt Tuning

In any prompted LLM response, there are two things that affect the output more than any other factor:

  1. The model weights

  2. The prompt

Given this, the idea of prompt tuning is to add a new set of weights between the prompt and the model, and only tune those. In the paper introducing this method, the T5 model weights were 11,000,000,000 parameters, and the added prompt weights were 20,400 parameters. Yet, the research showed similar performance with prompt tuning to full model weight fine-tuning on a benchmark dataset.

The original model weights stay untouched, a smaller amount of weights are introduced and tuned.

Figure 8:The original model weights stay untouched, a smaller amount of weights are introduced and tuned.

Prompt Engineering

Prompt Engineering, or prompt design, or system prompts are similar ideas. The model is prompted to “act like a pirate” or “be a math expert”. This method is quite popular and accessible because of how intuitive and easy it is; any user can adjust the behavior as part of their prompt. It’s also quite effective, changing the output generation dramatically. It’s so effective that deployed models like ChatGPT have a pre-prompt that is hidden from the user but often exposed through clever prompting. However, this isn’t categorized as fine-tuning because no weights are updated.

The benefits of this method are its ease for anyone to update or generate as these are written in natural language. It doesn’t require any additional compute at train time, which also makes them easy to iterate upon. The downsides are that they take up portions of the context window and compute at inference time, and that they “wear off” as they don’t affect behavior as strongly as fine-tuning.

For examples, refer to Anthropic’s red teaming paper and the LLAMA 2 Paper. Anthropic’s red teaming paper showed how prompt tuning basically made a model no safer than a baseline pretrained model. In LLAMA 2, Meta also showed how prompt tuning can fail quite readily, so much so that they invented a new fine-tuning method called Ghost Attention to make fine-tuning more effective.

Prompt Tuning References

Low-Rank Adaptation of Large Language Models

LORA, as abbreviated, is a technique that reduces the number of parameters that need to be trained by reducing the dimensionality using a linear algebra technique called Singular Value Decomposition, also known as Rank Decomposition.

A full explanation would require a course in linear algebra, but here are two pieces of intuition:

The first piece of insightful explanation comes from an LLM:

Dimensionality Reduction: Singular values can be used to reduce the dimensionality of data by projecting the data onto a lower-dimensional subspace. This technique is useful in machine learning and data analysis, where high-dimensional data can be simplified for efficient processing and analysis.

The other is by seeing a basic example. Consider these two matrices:

matrix_1 = [[1, 2],
            [3, 6]]

matrix_2 = [[1, 2],
            [3, 5]]

In matrix_1, the second row is a linear combination of the first row. Specifically, if you multiply [1,2] by three, you get the second row. In this case, matrix_1 has a rank of 1. This means the second row provides no additional information. Contrast this with matrix_2, where no linear multiplication of either row will yield the first. In this case, matrix_2 has a rank of 2.

You can verify this yourself with numpy:

f"Rank matrix_1 {np.linalg.matrix_rank(matrix_1)}"
f"Rank matrix_2 {np.linalg.matrix_rank(matrix_2)}"

Rank matrix_1 1
Rank matrix_2 2

Now, this is an oversimplified example that’s leaving out a lot of the fundamental details. Try running U, S, Vh = np.linalg.svd(matrix_1) or reading the docs for the svd method and you’ll see how much deeper this topic goes.

However, even without the specific details, the dimensionality reduction intuition is the same idea with 11 billion parameter LLMs. The pretrained model’s parameters are in a super high dimensional space, but for fine-tuning, you likely can fine-tune many fewer parameters in a compressed space and influence the final output behavior drastically. The trick the authors use is creating a new set of LORA parameters that are much smaller than the others, but are shaped in a way that can be easily combined without changing the input and output sizes of certain parts of the model.

Here’s a visualization, adapted from the original paper, where A and B are the LORA parameters. The dimension x is the original input dimension and r is the reduced dimension:

An adapted diagram from the original LORA paper. A and B are the weights added to the LLM system trained. Within a pretrained LLM that subset may only map to subset of the original models.

Figure 9:An adapted diagram from the original LORA paper. A and B are the weights added to the LLM system trained. Within a pretrained LLM that subset may only map to subset of the original models.

Aside from the mathematical properties, LORA has a number of impressive engineering properties. Apart from fine-tuning efficiency, LORA weights can be swapped in and out for different tasks for the same user, or different LORA weights can be stored as different model implementations for different users. LORA methods are also actively being improved with advancements like Qlora and DyLORA, which provide improvements upon the original.

LORA References

Distillation

Text Models
Pretraining
Text Models
Model Evaluation