Having reproduced GPT-2 small from scratch, I wanted to do something with the weights other than admire the loss curve. The obvious move — and the natural companion to my post on how base models become assistants — was to actually run the pipeline: supervised fine-tuning, then DPO. Could a 124M model become a chat assistant? Short answer: a toy one — it follows simple instructions and holds the format, but it’s shallow and hallucinates confidently. That’s exactly what makes it a good thing to have built and poked at. As before, worked out with Claude.

Try it in your browser → (runs the model client-side with transformers.js — compare base vs SFT vs DPO).

Stage 1 — SFT: the big jump

SFT is, mechanically, the same next-token objective as pretraining, on different data — instruction→response pairs instead of raw web text. I used Alpaca-cleaned (~52k examples) with TRL’s SFTTrainer, formatting each row as a prompt/completion pair so the loss is masked to the response only (the model learns to produce answers, not echo prompts):

### Instruction:
{instruction}

### Response:
{output}          ← loss computed here only

Three epochs on the 124M base (~8 minutes on one H100), and instruction-following simply appears. The contrast with the base model is stark:

It still hallucinates — asked for a motivational quote it invented an attribution, “— Paulo Freirer (2006)” — but it’s unmistakably following instructions now. That’s the SFT payoff: format and instruction-following are cheap to teach; factual reliability is not.

Stage 2 — DPO: the alignment tax, in miniature

Direct Preference Optimization nudges the SFT model toward preferred responses using (prompt, chosen, rejected) triples — no reward model, no PPO. I used UltraFeedback (15k pairs) with TRL’s DPOTrainer. The training metrics did exactly what they should: the reward margin rose to ~0.28 and the model preferred “chosen” over “rejected” ~63% of the time.

And the behavior did change — DPO made responses longer, more detailed, more “assistant-like”. This is the well-documented UltraFeedback length/verbosity preference: annotators (and the models that labeled the data) tend to prefer fuller answers, so DPO pushes the model to be verbose.

Here’s the catch, and the most interesting result of the whole exercise: at 124M, verbosity isn’t quality — it’s more room to hallucinate. DPO optimized the preference signal it was given, but that signal is a proxy for “good,” not ground truth. So on some prompts it regressed:

The SFT model gave the right one-word answer; the DPO model, optimized to be more elaborate and “helpful-sounding,” talked itself out of it. That’s the alignment tax / reward-vs-truth gap you read about in the RLHF literature, reproduced here on a tiny model you can train in an afternoon: making a model sound more aligned to a preference is not the same as making it more correct, especially when the base model lacks the capacity to back up the extra words.

Making it multi-turn

Both models above are single-turn — Alpaca is one instruction → one response, with no notion of a conversation. To get an actual (toy) chatbot, I ran a separate SFT pass on a conversational dataset, smol-smoltalk (built for small models), using a ChatML-style chat template and assistant_only_loss — so the loss falls on every assistant turn across a multi-turn dialogue and the model learns the turn boundaries (<|user|> / <|assistant|>, each turn ending in EOS).

It does hold a running conversation and reference earlier turns:

User: Give me three tips for studying. Assistant: When you read your text, be sure to carefully listen with attention and focus on the words that are important… User: Can you expand on the first one? Assistant: While it’s possible for readers to comprehend the first paragraph more effectively once they’ve read each other’s sentences…

Notice it picks up “the first one” and runs with it — real (if wobbly) use of context. But with a 1024-token window and 124M parameters, coherence degrades fast and it hallucinates freely; it’s a chatbot in form, not in reliability. The multi-turn model is gpt2-small-fineweb-edu-10b-chat, and it’s the default in the in-browser demo.

Takeaways

• SFT is where the transformation happens. A near-instant jump from “document completer” to “instruction follower,” using the same objective as pretraining.
• DPO changes style, and style ≠ substance. It reliably shifted the model toward longer, more assistant-like output — and just as reliably exposed that a 124M model has nothing to fill that extra length with but confident guesses.
• Watching the alignment tax happen on a model small enough to fully understand is worth more than reading about it. The reward metric went up; the answers, on balance, did not.

For a single-turn instruction-follower, reach for the SFT model; to actually chat, use the multi-turn chat model. All are on the Hub: SFT · DPO · chat · base — with all the training code in the repo under posttraining/, and an in-browser demo to try them.