Fine-Tuning: Supervised Fine-Tuning Interview Answer Framework

The Interview Question, Restated

“When would you fine-tune a model instead of using prompt engineering or retrieval, and how would you structure the supervised fine-tuning (SFT) pipeline?” This question tests whether a candidate treats fine-tuning as a default lever or as a specific, expensive tool reached for only after cheaper options are exhausted. Interviewers use it to filter candidates who jump to “just fine-tune it” without a cost-benefit argument.

Answer First

Supervised fine-tuning is the process of continuing to train a pretrained language model on a curated dataset of (input, desired-output) pairs so the model’s output distribution shifts toward that dataset’s style, format, or task behavior. The correct interview answer states three things in order: (1) fine-tuning solves a distribution-matching problem, not a knowledge problem, so it is the wrong tool when the goal is “give the model facts it doesn’t know” (retrieval solves that); (2) fine-tuning is justified when prompt engineering has hit a ceiling on consistency, format adherence, or latency (a fine-tuned model needs a shorter prompt because instructions are baked into weights); and (3) the pipeline has four required stages — data curation, training run, evaluation against a held-out set, and a rollback-capable deployment gate.

Scope and Assumptions

This page covers full or parameter-efficient (LoRA/QLoRA) supervised fine-tuning of an existing pretrained or instruction-tuned LLM using labeled input-output pairs. It does not cover reinforcement learning from human feedback (RLHF/DPO), pretraining from scratch, or continued pretraining on unlabeled text. The interview format assumed: a live or take-home system design question where the candidate must justify SFT as a design decision, not just explain the algorithm. Example scenario used throughout: “our support-ticket classifier needs to output a structured JSON category and priority, and the current prompted GPT-4-class model gets the format right only 85% of the time.”

Core Framework: The Build-vs-Prompt-vs-Fine-Tune Decision Tree

The single most common mistake in this interview answer is starting with the fine-tuning mechanics before establishing that fine-tuning is the right choice. Walk the decision tree out loud first.

Is the failure a KNOWLEDGE gap (model doesn't know a fact)?
  -> YES: Use retrieval (RAG) or a tool call. Fine-tuning does not reliably inject new facts and risks hallucination if used for this purpose.
  -> NO, continue.

Is the failure a FORMAT/CONSISTENCY gap (model knows the answer but wraps it inconsistently)?
  -> YES, and prompt engineering (few-shot examples, structured output constraints, function calling) has been tried and still fails above an acceptable error rate:
       -> Fine-tuning is justified. Proceed to data curation.
  -> YES, but prompt engineering has NOT been tried yet:
       -> Try prompt engineering first. It is orders of magnitude cheaper and faster to iterate.

Is the failure a LATENCY/COST gap (a large general-purpose model works but is too slow or expensive at your call volume)?
  -> YES: Fine-tune a smaller model on distilled outputs from the larger model. This is a legitimate and common SFT use case — model distillation via SFT.

State this tree explicitly in the interview. It signals you know fine-tuning is not a universal upgrade — it is a specific fix for format, consistency, and cost, not a knowledge upgrade.

Worked Example: Support-Ticket Classifier SFT Pipeline

Stage 1 — Data curation. Collect 500-2,000 examples of (ticket text, correct structured output) pairs. Source: human-corrected outputs from the current prompted model’s failures, not synthetic data alone — synthetic-only datasets tend to reinforce the base model’s existing biases rather than correct them. Deduplicate near-identical tickets (common in support data — the same three issues generate 40% of volume) to avoid the model overfitting to the most frequent categories at the expense of rare-but-important ones.

# Minimal SFT data format (JSONL), one example per line
# Input: raw ticket text. Output: target structured completion.
{"messages": [
  {"role": "system", "content": "Classify the support ticket."},
  {"role": "user", "content": "My payment failed twice and I was charged both times."},
  {"role": "assistant", "content": "{\"category\": \"billing_double_charge\", \"priority\": \"high\"}"}
]}

Stage 2 — Training run. For a 7B-13B parameter base model and a dataset in the low thousands of examples, LoRA (Low-Rank Adaptation) fine-tuning is the default choice over full fine-tuning: it trains a small set of low-rank adapter matrices injected into the attention layers rather than updating all model weights, which cuts GPU memory requirements roughly 3-4x and lets the base model be shared across multiple task-specific adapters. Typical LoRA hyperparameters for this scale: rank r=16, alpha=32, learning rate 1e-4 to 2e-4, 2-3 epochs — more epochs on a small dataset risks overfitting to the exact phrasing of training examples rather than generalizing the format.

Stage 3 — Evaluation. Hold out 15-20% of the curated data as a test set never seen during training. Measure exact-match rate on the structured JSON output (does it parse, does the category match, does the priority match) separately from measuring whether the underlying classification judgment is correct — a model can learn the JSON format perfectly while still misclassifying tickets, and conflating these two failure types hides which one you actually fixed.

Stage 4 — Deployment gate. Run the fine-tuned model in shadow mode against production traffic for a defined window (commonly 1-2 weeks depending on ticket volume) before cutting over, comparing its structured-output accuracy and category distribution against the existing prompted model on the same live tickets. Keep the prompted model as an instant rollback path — fine-tuned models fail silently in ways prompted models do not, because there is no prompt to inspect and adjust; a bad fine-tune requires a new training run to fix, not a text edit.

Trade-offs Table: SFT Approach by Constraint

Decision Rubric

Choose full fine-tuning only when: you have tens of thousands of labeled examples, a dedicated ML infrastructure team, and a documented case that LoRA underperformed on your task (this is rare for a mid-size product team and should be treated as a fallback, not a default).

Choose LoRA/QLoRA when: you have 500+ labeled examples, need consistent structured output or domain-specific tone, and want to keep the ability to serve multiple task adapters off one base model.

Choose prompt engineering (no fine-tuning) when: you have fewer than 500 labeled examples, the failure is intermittent rather than systematic, or you have not yet tried structured output constraints (JSON mode, function calling schemas) and few-shot examples.

Choose distillation SFT when: a larger model already solves the task correctly but is too slow or costly at your call volume — fine-tune a smaller model on the larger model’s outputs.

Interview Scorecard

Book Sample

The 0→1 AI Engineer Interview Playbook (ASIN B0H2CML9XD) walks through this exact SFT-vs-prompting decision tree as a live interview transcript with interviewer follow-ups on data curation and evaluation design. The 0→1 Machine Learning Engineer Interview Playbook (ASIN B0H256Z1MF) covers the underlying LoRA math (low-rank decomposition, why rank and alpha are chosen the way they are) in more depth for candidates who need the theory before the applied interview framing.

Get the AI Engineer Interview Playbook: /go/B0H2CML9XD?source=ai-engineers-blog&page=aie-sft-001

Get the Machine Learning Engineer Interview Playbook: /go/B0H256Z1MF?source=ai-engineers-blog&page=aie-sft-001

Common Follow-Up Questions and How to Handle Them

“How do you know the fine-tuned model didn’t just memorize the training examples instead of learning the general format?” Give a concrete detection method, not a reassurance. Split the held-out test set so that no ticket in the test set is a near-duplicate of any training example (deduplicate across the train/test boundary, not just within each set), then check whether test-set accuracy is meaningfully lower than training-set accuracy. A large gap between training and test accuracy is the direct signal of memorization rather than generalization, exactly analogous to overfitting in classical ML, and candidates who can name this check signal real fine-tuning experience rather than surface familiarity with the term LoRA.

“What if the base model gets updated by the vendor — does your fine-tune still work?” This is a maintenance-cost question disguised as a technical question. State plainly that a LoRA adapter is trained against a specific base model checkpoint and is not guaranteed to transfer cleanly to a new base model version, because the adapter’s low-rank matrices were learned to correct that specific checkpoint’s weight geometry. This means every base model upgrade requires re-evaluating (and likely retraining) the adapter — a real operational cost that should be factored into the build-vs-prompt decision at the start, since a prompt-only solution updates for free when the vendor ships a better base model, while a fine-tuned adapter does not.

“Your fine-tuned model’s format accuracy is 98% but the underlying classification accuracy only improved from 85% to 87%. What does that tell you?” This is testing whether you separate the two failure types cleanly. The correct read: fine-tuning successfully solved the format-consistency problem (98% is a large jump from the original 85% format-adherence baseline) but barely touched the underlying judgment problem, which suggests the training data taught the model how to format an answer more than it taught the model which answer is correct. The fix is not more epochs on the same data — it is auditing whether the training examples actually contain enough signal to teach correct classification, which often means the labels themselves need review before blaming the training process.

Cost Modeling: LoRA Training Cost at Different Data Scales

Interviewers testing seniority often ask you to reason about cost before committing to a fine-tuning project. A rough but defensible cost model for a 7B-parameter LoRA fine-tune:

Training compute scales roughly with: (number of training examples) × (epochs) × (sequence length)
  divided by (effective throughput of the training hardware).

At 2,000 examples, 3 epochs, average 300-token sequences, a single modern GPU (24-80GB class)
completes this run in a small number of GPU-hours — commonly under 2 hours for LoRA at this
scale, versus a full fine-tune of the same model requiring several times more GPU-hours and
several times more GPU memory due to storing full optimizer state for all model parameters
rather than just the adapter's low-rank matrices.

The recurring cost is not the training run itself — it is the evaluation and iteration cycle:
every prompt-format change to the training data requires a new training run to test, so the
real cost driver at small-to-medium scale is engineering time spent iterating on data quality,
not raw compute spend.

State this explicitly when asked to estimate cost: for teams under roughly 10,000 training examples, the dominant cost is data curation and evaluation iteration, not GPU spend — a claim that reframes the “is fine-tuning worth it” question away from a compute-budget question and toward a data-quality and engineering-time question, which is the more accurate framing for most production teams at this scale.

Sources and Freshness

LoRA hyperparameter ranges reflect commonly published configurations for 7B-13B parameter models in open fine-tuning literature and framework documentation (e.g., Hugging Face PEFT), not a single benchmark run performed for this page. No proprietary company training data or interview outcome data is used. Last reviewed: 2026-07-15. Next scheduled review: quarterly.

Recommended Resource

If you’re actively preparing for this process, the 0→1 AI Engineer Playbook covers the judgment frameworks, real question patterns, and structured answers this article draws on — useful when you want a complete preparation system rather than scattered tips.