1. What Is a Large Language Model?

A Large Language Model (LLM) is a deep neural network trained on massive amounts of text data to predict the next token in a sequence. Despite the simple training objective, this approach produces systems capable of reasoning, coding, translation, and creative writing.

Key facts about modern LLMs:

Mental model

Think of an LLM as a massive compression of the internet's text into a function that, given any text prefix, produces a plausible continuation. The "intelligence" emerges from the patterns it has compressed.

2. How LLMs Work — Behind the Scenes

The complete pipeline from user input to model output:

Raw Text → Tokenization → Embeddings → Transformer Layers → Logits → Softmax → Token Output → Repeat

Let's break down each step:

Step 1: Tokenization

Raw text is split into tokens — subword units that the model can process. Tokenizers like BPE (Byte Pair Encoding) or SentencePiece break text into common subword chunks.

# Example tokenization
"ChatGPT is amazing" → ["Chat", "G", "PT", " is", " amazing"]
"unhappiness"        → ["un", "happiness"]

# Typical vocabulary sizes:
# GPT-4: ~100K tokens
# LLaMA: ~32K tokens
# Claude: ~100K tokens

Each token maps to an integer ID. The tokenizer maintains a fixed vocabulary — any text is decomposed into sequences from this vocabulary.

Step 2: Embedding

Each token ID is mapped to a high-dimensional dense vector (typically 768 to 4096 dimensions for large models). These embeddings are learned during training and capture semantic relationships between tokens.

Step 3: Positional Encoding

Transformers process all tokens in parallel — they have no inherent sense of order. Positional encodings inject sequence position information. Modern LLMs use Rotary Position Embeddings (RoPE), which encode relative positions and enable length generalization.

Step 4: Transformer Layers

The core of the model. Each layer applies:

  1. Self-Attention: Each token attends to all previous tokens (in decoder-only models), computing relevance-weighted information.
  2. Feed-Forward Network (FFN): A two-layer MLP that processes each token independently, adding non-linear transformations.
  3. Layer Normalization: Stabilizes activations for smoother training.
  4. Residual Connections: Skip connections that allow gradients to flow and enable deeper networks.

A typical large model has 32–80 transformer layers stacked sequentially.

Step 5: Logits and Softmax

The final hidden state is projected through the output head (a linear layer) to produce logits — one score per vocabulary token. Softmax converts these into a probability distribution over the entire vocabulary.

Step 6: Sampling / Decoding

A token is selected from the probability distribution using a decoding strategy (greedy, top-k, top-p, etc.). This token is appended to the sequence, and the process repeats autoregressively until a stop condition is met.

3. Key Components of an LLM

Component Role
Tokenizer Converts text to token IDs and back. Defines the model's vocabulary.
Embedding Layer Maps token IDs to dense vectors that encode meaning.
Attention Heads Focus on relevant parts of the context. Multiple heads capture different relationship types.
FFN (Feed Forward) Non-linear transformation per token. Often stores factual knowledge.
Layer Norm Normalizes activations to stabilize and accelerate training.
KV Cache Caches past key/value attention states to speed up autoregressive inference.
Context Window Maximum number of tokens the model can "see" at once (4K to 1M+).
Output Head Linear projection from hidden states to vocabulary probabilities.

4. The Transformer Architecture

Introduced in "Attention Is All You Need" (Vaswani et al., 2017), the Transformer replaced RNNs and LSTMs as the dominant sequence model. Key advantages:

Architecture Variants

Type Examples Best For
Encoder Only BERT, RoBERTa, DeBERTa Classification, embeddings, NER
Decoder Only GPT-4, LLaMA, Mistral, Claude Text generation, chat, reasoning
Encoder-Decoder T5, BART, Flan-T5 Translation, summarization
Industry trend

Modern LLMs are almost exclusively Decoder-Only. The autoregressive generation paradigm has proven most versatile — a single decoder model can handle classification, generation, reasoning, and tool use through appropriate prompting.

5. Self-Attention — The Core Mechanism

Self-attention is what allows each token to "look at" and gather information from other tokens in the sequence. Here is the core formula:

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V

Where:
  Q (Query)  — "What am I looking for?" (from current token)
  K (Key)    — "What do I have to offer?" (from all tokens)
  V (Value)  — "What information do I carry?" (from all tokens)
  d_k        — Dimension of keys (scaling factor prevents vanishing gradients)

How it works intuitively:

  1. Each token generates a Query vector (what it wants to know).
  2. Each token generates Key and Value vectors (what it can tell others).
  3. Dot product of Q and K gives attention scores — how relevant each other token is.
  4. Softmax normalizes scores to a probability distribution.
  5. Weighted sum of Value vectors gives the output — a context-aware representation.

Multi-Head Attention

Instead of one attention operation, the model runs H parallel attention "heads" with different learned projections. Each head can specialize in different relationship types (syntax, semantics, coreference, etc.). Outputs are concatenated and projected back.

MultiHead(Q, K, V) = Concat(head_1, ..., head_H) * W_O
where head_i = Attention(Q * W_Q_i, K * W_K_i, V * W_V_i)

# Typical: 32-64 attention heads per layer

6. Autoregressive Generation and Decoding Strategies

LLMs generate text one token at a time, where each new token is conditioned on all previously generated tokens. The choice of decoding strategy dramatically affects output quality and style.

Strategy How It Works Use Case
Greedy Always pick the highest probability token Fast, deterministic outputs
Beam Search Keep top-N candidate sequences, expand in parallel Translation, structured outputs
Temperature Scale logits before softmax (T<1 = focused, T>1 = creative) Chat, creative writing
Top-K Sample only from the top K highest-probability tokens Controlled randomness
Top-P (Nucleus) Sample from the smallest set whose cumulative probability exceeds P Open-ended generation
Repetition Penalty Reduce probability of tokens already generated Avoid repetitive loops

Key Parameters for API Calls

# Common API parameters
{
  "temperature": 0.7,       # 0.0 = deterministic, 2.0 = very random
  "top_p": 0.9,             # Nucleus sampling threshold
  "top_k": 50,              # Top-K filter (not all APIs support this)
  "max_tokens": 4096,       # Maximum output length
  "stop": ["\n\n", "END"], # Stop sequences
  "frequency_penalty": 0.3  # Penalize repeated tokens
}
Rule of thumb

For factual/analytical tasks, use temperature 0.0-0.3. For creative tasks, use 0.7-1.0. Avoid temperature above 1.2 unless you specifically want highly random output. Top-P of 0.9 is a solid default for most applications.

7. KV Cache — Why Inference Is Fast

The Key-Value (KV) Cache is crucial for efficient autoregressive generation. Without it, generating each new token would require recomputing attention over the entire sequence from scratch.

Without KV Cache

For each new token, recompute all Key and Value vectors for all previous tokens. Complexity: O(n^2) per generation step, O(n^3) total for a full sequence.

With KV Cache

Cache the Key and Value vectors from previous steps. For each new token, only compute its Query, then attend to the cached Keys and Values. Complexity: O(n) per step, O(n^2) total.

# Pseudocode for KV-cached generation
kv_cache = {}

for each new token:
    q = compute_query(new_token)
    k, v = compute_key_value(new_token)
    kv_cache.append(k, v)           # Store for future tokens
    attention_out = attend(q, kv_cache)  # Attend to all cached KVs
    next_token = decode(attention_out)

The Memory Problem

KV cache grows linearly with sequence length and is the primary memory bottleneck during inference. For a 70B model with 128K context, the KV cache alone can consume 40+ GB of GPU memory.

Optimization Techniques

8. Training: Pre-training, SFT, RLHF, DPO

Modern LLMs go through multiple training phases, each building on the previous one:

Phase 1: Pre-training

The foundation. Train on massive, diverse text data with a simple objective: predict the next token.

Phase 2: Supervised Fine-Tuning (SFT)

Train the base model on curated instruction-response pairs to make it follow instructions and behave like an assistant.

Phase 3: RLHF (Reinforcement Learning from Human Feedback)

Align the model with human preferences using reinforcement learning:

  1. Generate multiple responses to the same prompt.
  2. Human raters rank the responses by quality.
  3. Train a reward model on these rankings.
  4. Use PPO (Proximal Policy Optimization) to optimize the LLM to maximize the reward model's score.

RLHF is effective but complex — it requires training three models (policy, reward, reference) and is prone to reward hacking.

Phase 4: DPO (Direct Preference Optimization)

A simpler alternative to RLHF that eliminates the separate reward model:

# DPO loss (simplified)
loss = -log(sigmoid(beta * (log_prob_chosen - log_prob_rejected)))

# Where beta controls how much to deviate from the reference model

9. Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning updates all model parameters — expensive and requires massive GPU memory. PEFT methods train only a tiny fraction of parameters while achieving comparable results.

Method Description Use Case
LoRA Low-rank decomposition of weight updates (A * B matrices) Most popular, general-purpose fine-tuning
QLoRA LoRA applied to 4-bit quantized base model Fine-tune 70B models on consumer GPUs (24GB)
Prefix Tuning Prepend trainable "virtual tokens" to the input Task-specific adaptation without modifying weights
Adapter Layers Insert small bottleneck layers between transformer blocks Multi-task learning with shared base
DoRA Decompose weights into magnitude and direction components Improved quality over standard LoRA

LoRA Explained

Instead of updating the full weight matrix W, LoRA adds a low-rank update:

# LoRA core idea
W_new = W_frozen + delta_W
delta_W = A * B

# Where:
#   W_frozen: original weights (not trained, saves memory)
#   A: shape (d, r) — trainable
#   B: shape (r, d) — trainable
#   r: rank (typically 8-64, much smaller than d which is 4096+)

# Parameters trained: 2 * d * r (instead of d * d)
# Example: d=4096, r=16 → 131K params vs 16.7M params per layer
Practical tip

QLoRA lets you fine-tune a 70B-parameter model on a single 24GB GPU. Use rank 16-64, target the attention projection layers (q_proj, v_proj), and train for 1-3 epochs on your dataset. Tools: Hugging Face PEFT, Unsloth, Axolotl.

10. Important Numbers to Know

Reference numbers for building intuition about LLM scale and costs:

Context Windows

Model Context Length Approx. Pages of Text
GPT-3 4K tokens ~6 pages
GPT-4 8K / 32K tokens ~12 / ~50 pages
GPT-4o 128K tokens ~200 pages
Claude (2025+) 200K tokens ~300 pages
Gemini 1.5 Pro 1M+ tokens ~1500 pages

Model Sizes

Category Parameters GPU Memory (FP16) Examples
Small 1B–7B 2–14 GB Phi-3, Mistral 7B, LLaMA 3 8B
Medium 13B–70B 26–140 GB LLaMA 3 70B, Mixtral 8x22B
Frontier 200B–1T+ 400+ GB GPT-4, Claude, Gemini Ultra

Training Scale

Token Approximations

What's Next?

This cheatsheet covers the fundamentals of how LLMs work under the hood. To go further and build production systems with LLMs, explore these PathtoAI Academy courses:

The field moves fast, but the fundamentals covered here — transformers, attention, training phases, and inference optimization — remain the foundation everything else is built on. Master these concepts and you will be equipped to understand any new development in the LLM space.

"The Transformer is not just a model architecture — it's the engine of a new computing paradigm."