Welcome to Posttraining

Reading depth

Choose A Pass

Post-training is where raw model capability is shaped into useful behavior. It is the long, practical phase after pretraining where preference data, reinforcement learning, evaluations, tool use, product constraints, and deployment feedback all start to matter.

This site is for writing about that phase with an engineering bias. Expect notes on evals, RLHF and RLAIF, agents, alignment, reliability, product behavior, and the tradeoffs that appear when model behavior has to survive contact with real users.

Every box is an anchor into the article, so the diagram acts as static navigation instead of a runtime widget.

Data Mixture

The post-training dataset decides what the model rehearses. It can include human preferences, expert demonstrations, synthetic traces, red-team prompts, support conversations, and task-specific examples where the desired behavior is narrower than generic helpfulness.

Preference Model

Preference models compress comparative judgments into a training signal. They are useful because many quality dimensions are easier to rank than to specify directly.

Loss Function Calculation

The loss turns the preference signal into a policy update. In practice, this is where teams balance improvement against drift: enough optimization to make the model more useful, but not so much that it loses capabilities or style constraints that already worked.

Code-shaped sketch

type RewardTerms = {
  preferenceReward: number;
  klPenalty: number;
  safetyPenalty: number;
};

export function postTrainingObjective(terms: RewardTerms) {
  return terms.preferenceReward - terms.klPenalty - terms.safetyPenalty;
}

Policy Update

A policy update is a behavior edit. The simplest way to think about it: the base model already knows many possible continuations, and post-training changes which continuations become more likely under particular contexts.

Evaluation Gates

Evals are the release valves. They catch regressions in factuality, style, tool use, safety, latency, and task success before a training win becomes a product regression.

Product Feedback

The product is part of the training environment. Users reveal confusing instructions, brittle tool contracts, overlong answers, unsafe affordances, and workflows where the model succeeds technically but fails practically.

The canonical archive for these essays will live at posttraining.dev. Substack will remain the email distribution channel for readers who prefer to subscribe by inbox.