Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

(Small) Language Model Intuition

TLDR:

What we do with languages and want from a language model

Here are some examples

The basic ___ of a language model

The basic purpose of a language model is to predict a word given other words. You as a human can try this, what is the missing word from the title above? Is it premise? Or function? Or idea? Or banana? All those words would work. The last one feels improbable though.

If you’re a native English speaker this assessment of probability may feel automatic for you but if we perform this exercise in a language that you’re not fluent in this would be challenging. You would neither know the words that surround the missing word, or what the missing word is itself.

This is the general intuition of language models.

Training Small Language Models

We can formalize this idea with this formula.

P(wm)=P(w1,…,wm)P(w_m) = P(w_1,\ldots,w_m)
(1)

That is, given a sequence of words what’s the most likely. Sometimes you see this formula written in its conditional form which is

P(wm∣w1,…,wm)P(w_m | w_1,\ldots,w_m)
(2)

The way a language model does this is by learning over a corpus of text. Let’s say we have the following

Training ExampleText
1My name is Ravin
2My name is Scott
3My dogs name is Spot

We may want to predict the next word after name P(w_next∣name) P(w\_{next}| name ).

With this text corpus above we get

P(is∣name)=1P(my∣name)=0P(is | name ) = 1 \\ P(my | name ) = 0
(3)

Why is that? Because is always after name.

Let’s take two different words

P(name∣my)=2/3P(Ravin∣is)=1/3P( name | my ) = 2/3 \\ P( Ravin | is ) = 1/3
(4)

Verify yourself why this is the case. If you see this then you understand the basic premise of a language model. Consider these learned probabilities to be the trained model Now this is a very simple model, in practice models can be trained on the last two words, or the word before or the word after, or other combinations of things. But the idea is all the same, the corpus of text is used as a training example to learn the structure of that same text.

All the things we can do

Now that we have a trained model we can use it in many different ways.

It could be used for grammatical checks. If this system saw the text Name my is Ravin, we can “ask” the model what it thinks about the word name preceding the word my. In this case we would get

P(next_word=my∣current_word=name)=0P( \text{next\_word}=my \mid current\_word=name ) = 0
(5)

This would indicate this sequence of words as unlikely indicating there is an issue.

It could also be used for text generation, We could pick a random word to start with like dog and see what the model picks to be the next word. Then with that prediction the model continues on and predicts the word after that and so on and so forth. This is in essence how generative models like Bard or ChatGPT work.

Problems with small language models and systems

You can think of what we did as “small” language model. This simplistic model provides an intuition of how these models work. In practice these single input word probability models don’t work well.

This is because

Over the years a number of different model techniques have been applied to alleviate some of these issues, such as n grams or Markov Chain approaches or Recurrent Neural Networks. That’s because as I’m writing this the type of model that dominates in language is the Neural Network Transformer architecture. I do encourage you to familiarize yourself with what I’m calling foundational methods. I’ve provided references below to help.

References

Footnotes

  1. Language model

    ↩
The Fundamentals
Neural Network Quickstart
Text Models
How we got here