· AI Engineers Editorial · AI Engineering · 7 min read
LLM Fine-Tuning: Interview Answer Framework
A structured framework for answering LLM fine-tuning interview questions, covering LoRA, QLoRA, full fine-tuning, data preparation, evaluation, and when to fine-tune versus prompt.
LLM Fine-Tuning: Interview Answer Framework
Fine-tuning questions in AI engineering interviews test a specific instinct: do you reach for fine-tuning as a last resort, or a first instinct? Strong candidates treat fine-tuning as an expensive, high-commitment decision that only makes sense once prompting and retrieval have been tried and found insufficient. This article gives you a framework for answering fine-tuning questions, covering LoRA, QLoRA, full fine-tuning, data preparation, evaluation, and — the question interviewers ask most — when to fine-tune versus when to just write a better prompt.
Why Interviewers Ask About Fine-Tuning
Fine-tuning is expensive in engineering time, compute, and ongoing maintenance (a fine-tuned model needs its own eval suite, retraining cadence, and versioning). Interviewers use fine-tuning questions to check whether you understand this cost structure, or whether you’ll reach for the most technically interesting solution instead of the most appropriate one. The strongest answers open by questioning whether fine-tuning is even necessary before describing how you’d do it.
The Decision Framework: When to Fine-Tune vs. Prompt
Before any fine-tuning question, state your decision criteria out loud. Fine-tuning is justified when:
- The task requires a new skill or format the base model can’t reliably produce, even with extensive prompting and few-shot examples (e.g., a highly specific structured output format, a domain-specific writing style, or a classification taxonomy with hundreds of fine-grained labels).
- Latency or cost at scale makes long prompts uneconomical. If a well-engineered prompt with few-shot examples is 3,000 tokens and you’re calling it millions of times a day, fine-tuning a smaller model to bake in that behavior can be cheaper long-term.
- You need to change model behavior that RAG and prompting can’t touch — tone, refusal patterns, or reasoning style, rather than factual knowledge (which retrieval handles better and more cheaply than fine-tuning).
If none of these apply, the correct interview answer is “I’d start with prompt engineering and RAG, and only fine-tune if evaluation shows a gap those can’t close.” Interviewers consistently rate this answer higher than diving straight into training details.
LoRA (Low-Rank Adaptation)
What it does: LoRA freezes the pretrained model weights and injects small trainable low-rank matrices into specific layers (typically attention projections), training only those matrices. This cuts trainable parameters by orders of magnitude versus full fine-tuning while recovering most of the performance gain.
When to use it: LoRA is the default choice for most fine-tuning use cases in 2026 — domain adaptation, style transfer, task-specific behavior — where you have a few thousand to a few hundred thousand training examples and want to iterate quickly without needing a large GPU cluster.
Interview talking point: Mention rank (r) and alpha as the key hyperparameters, and that you’d sweep rank (commonly 8-64) against a validation set rather than assume higher rank is always better — higher rank increases capacity but also overfitting risk on small datasets.
QLoRA (Quantized LoRA)
What it does: QLoRA combines LoRA with 4-bit quantization of the frozen base model weights, dramatically reducing the GPU memory required to fine-tune large models — the technique that made fine-tuning 65B+ parameter models feasible on a single consumer or prosumer GPU.
When to use it: QLoRA is the right answer when the interviewer asks “how would you fine-tune a large model with limited GPU budget.” It’s a memory optimization, not a fundamentally different training approach from LoRA — being clear about that distinction, rather than treating QLoRA as a separate technique, signals you understand the mechanism rather than just the name.
Interview talking point: Flag the tradeoff: quantization introduces a small quality degradation versus full-precision LoRA, and training is somewhat slower due to dequantization overhead during the forward pass. For most practical use cases this tradeoff is worth it given the memory savings, but say that explicitly rather than presenting QLoRA as strictly better.
Full Fine-Tuning
What it does: Full fine-tuning updates all model parameters, requiring significantly more GPU memory (to store gradients and optimizer states for every parameter) and more training data to avoid catastrophic forgetting of the model’s general capabilities.
When to use it: Full fine-tuning is justified when you have a large, high-quality dataset (often 100K+ examples) and need the model to fundamentally shift its behavior in ways that low-rank adaptation can’t capture — or when you’re training a smaller model from a strong base specifically for one narrow task and don’t need to preserve general capability.
Interview talking point: The most common mistake candidates make is assuming full fine-tuning is always “better” because it’s more thorough. In practice, LoRA/QLoRA match or beat full fine-tuning on most task-specific benchmarks while being far cheaper and less prone to catastrophic forgetting — say this to show you’ve internalized the actual tradeoff rather than the intuitive-but-wrong default.
Data Preparation
What it does: Fine-tuning data quality determines the ceiling on fine-tuning results more than any hyperparameter choice. This means deduplication, format consistency, label/output verification, and — critically — filtering out examples where the “ground truth” output is itself mediocre, since the model will learn to imitate whatever quality level you feed it.
Interview talking point: Describe a concrete data pipeline: collect raw examples, deduplicate near-identical entries (which cause overfitting to specific phrasings), run a subset through human or LLM-judge quality review, and hold out a stratified validation split that mirrors the distribution of production traffic, not just the training distribution. Interviewers specifically probe whether you’ve thought about train/production distribution mismatch.
Evaluation
What it does: Fine-tuning evaluation needs two layers: a held-out task-specific eval set measuring whether the fine-tune improved on the target task, and a general-capability regression suite checking the fine-tune didn’t degrade unrelated capabilities (catastrophic forgetting).
Interview talking point: Name specific failure modes you’d watch for: a fine-tune that overfits to training data formatting quirks, a fine-tune that improves the target metric but degrades instruction-following on out-of-distribution prompts, and a fine-tune that looks good on automated metrics but fails a human spot-check. Mention that you’d run the eval before and after fine-tuning on the same fixed prompt set, comparing side by side rather than trusting a single aggregate score.
Comparison Table
| Approach | Trainable params | GPU memory need | Best for | Main risk |
|---|---|---|---|---|
| LoRA | Small (rank-dependent) | Moderate | Domain/style adaptation, fast iteration | Underfitting if rank too low |
| QLoRA | Small (rank-dependent) | Low (4-bit base) | Large models on limited GPU budget | Slight quality loss vs. full precision |
| Full fine-tuning | All parameters | High | Large high-quality datasets, narrow deployment | Catastrophic forgetting, cost |
| Prompt engineering / RAG (baseline) | None | None | Most tasks before considering fine-tuning | Context length limits, latency |
How to Structure Your Interview Answer Out Loud
When asked “how would you fine-tune a model for X,” don’t start with hyperparameters. Start with: (1) why prompting/RAG alone isn’t sufficient here, (2) what data you’d need and how you’d verify its quality, (3) LoRA/QLoRA as the default technique with full fine-tuning as the exception, (4) the specific evaluation you’d run before shipping. This sequence is what separates a senior answer from a junior one.
Mistakes Candidates Make
The most common mistake is jumping straight to “I’d fine-tune with LoRA” without ever questioning whether fine-tuning is the right tool. A close second is describing fine-tuning mechanics in detail while saying nothing about data quality, which is usually the actual bottleneck in production fine-tuning projects. The third is failing to mention a regression eval for general capabilities, treating fine-tuning as risk-free.
Practice Questions
- “A team wants to fine-tune a model because prompting ‘isn’t working.’ What questions do you ask before agreeing?”
- “Walk me through your data pipeline for a fine-tuning project from raw logs to training-ready dataset.”
- “How would you detect catastrophic forgetting after a fine-tune, and what would you do about it?”
Rehearse these with the “question the premise first” framing before your next interview.
For a complete structured walkthrough of AI engineering interview questions, including fine-tuning, prompt engineering, and system design, see The 0-to-1 AI Engineer Interview Playbook (Amazon: https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20).