Ai Engineer Interview Batch Vs Streaming Inference

Why This Is One of the Most Asked System Design Questions in 2026

“Would you serve this model with batch or streaming inference?” is close to a universal question across AI engineer interviews in 2026, because it forces a candidate to reason simultaneously about latency requirements, cost structure, hardware utilization, and product constraints — all in a few minutes of whiteboard time. It’s a favorite precisely because there’s no single correct answer; the interviewer is scoring your reasoning process, not a memorized conclusion.

The rise of LLM-serving infrastructure has made this question sharper than it was in the classical ML era. With classical models, “batch vs. streaming” mostly meant “cron job vs. REST API.” With LLMs, the question now also covers continuous batching (vLLM, TensorRT-LLM style dynamic batching), speculative decoding tradeoffs, and how token-level streaming to the end user interacts with server-side request batching — two related but distinct concepts candidates frequently conflate.

Defining Terms Precisely (Where Most Candidates Lose Points)

Batch inference: requests are collected over a window (minutes to hours), processed together against the model, and results are written to storage for later consumption. No individual request expects a synchronous response. Classic use cases: nightly recommendation score refresh, embedding regeneration for a document corpus, offline evaluation runs.

Online/real-time inference: a single request triggers a synchronous or near-synchronous model call, typically behind a REST or gRPC endpoint, response expected within a strict SLA (often sub-second to a few seconds).

Streaming inference (the term causing the most confusion): this has two meanings candidates must disambiguate on the spot.

1. Server-side continuous batching — the serving engine (vLLM, TGI, TensorRT-LLM) dynamically groups incoming requests into batches at the token level, so GPU utilization stays high even under variable request arrival rates. This is an infrastructure/throughput concept.
2. Client-facing token streaming — the user sees tokens appear incrementally (as with ChatGPT-style UIs) rather than waiting for the full response. This is a UX/perceived-latency concept.

A response that only addresses one of these two meanings when asked “how does streaming inference work for LLMs” reads as incomplete to an interviewer in 2026, since production LLM serving stacks now do both simultaneously and the interaction between them (e.g., how continuous batching affects per-token latency variance seen by the streaming client) is itself a common follow-up.

Decision Framework: When to Choose Which

The strongest interview answers use an explicit framework rather than jumping to a conclusion:

1. Latency SLA — sub-second requirement almost always rules out pure batch. Multi-hour acceptable latency almost always favors batch for cost reasons.
2. Request volume predictability — highly bursty, unpredictable traffic favors an autoscaled online serving layer with continuous batching; steady, poolable volume (e.g., re-scoring your entire user base nightly) favors batch, where you can use spot/preemptible GPU capacity at 60-70% cost savings.
3. Cost per inference at scale — batch inference on preemptible accelerators, with large static batch sizes, routinely achieves 3-8x lower cost per token/inference than always-on online serving provisioned for peak load.
4. Freshness requirements — does the answer need today’s data, or is a nightly snapshot acceptable? Fraud detection needs real-time; a weekly content-recommendation refresh does not.
5. Failure blast radius — a failed batch job can be retried with no user-facing impact; a failed online inference call needs graceful degradation (cached response, fallback model, or explicit error) built into the serving path.

Walking through these five dimensions explicitly, then landing on a recommendation, is what turns a two-sentence answer into a senior-level system design response.

Worked Example: Designing Inference for a Content Moderation System

A frequently used interview prompt: “Design the inference architecture for a content moderation system that must flag policy-violating posts before they’re publicly visible, at 2 million posts/day.”

Strong candidates propose a hybrid architecture:

• A fast, small classifier (distilled model, sub-50ms) runs online/streaming on every post at publish time — this is the SLA-critical path, blocking publish only on high-confidence violations.
• Borderline-confidence posts are queued and processed in near-real-time (seconds, via continuous-batched serving) by a larger, more accurate model — a second-tier check that doesn’t block the initial publish but can retroactively hide content.
• A nightly batch job re-scores a random sample plus all borderline cases from the day using the largest available model, both for quality monitoring and for generating hard-negative training data for the next fine-tune cycle.

This three-tier design demonstrates you can combine batch and streaming rather than treating the question as binary — a signal that consistently correlates with senior-level hiring decisions in 2026 debriefs.

Comparison Table: Batch vs. Streaming/Online Inference

Follow-Up Questions Interviewers Layer On Top

Once you’ve answered the core batch-vs-streaming question, expect layered follow-ups:

• “How would continuous batching change your cost estimate for the online tier?” — expect you to mention that continuous batching can close much of the cost gap versus naive request-per-call serving, though it rarely fully closes it versus true offline batch.
• “What happens to your architecture if request volume grows 20x overnight?” — tests whether you’ve actually designed for autoscaling versus a fixed-capacity assumption.
• “How do you monitor for model drift differently in a batch vs. streaming setup?” — batch systems get natural checkpoints for drift monitoring at each run; streaming systems need continuous, windowed drift detection (often itself a small streaming pipeline).

Rehearsing this multi-layer questioning pattern in advance, rather than encountering it cold, is exactly the kind of preparation covered in The 0-to-1 AI Engineer Interview Playbook (https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20), which includes full interview transcripts showing how these follow-ups typically unfold.

FAQ

Q: Is “streaming inference” the same as “real-time inference”? Interviewers seem to use them interchangeably. A: They overlap but aren’t identical. Real-time/online inference refers to the latency contract (fast, synchronous). Streaming inference, in the LLM-serving context, more specifically refers to either continuous request batching on the server or incremental token delivery to the client — always clarify which meaning is intended before answering.

Q: Do I need hands-on vLLM/TensorRT-LLM experience to answer this well? A: Not strictly, but naming the actual tools (vLLM’s PagedAttention and continuous batching, TensorRT-LLM’s in-flight batching) signals current, practical knowledge versus theoretical-only understanding, which matters more in 2026 interviews than it did two years ago.

Q: What’s the biggest mistake candidates make on this question? A: Treating it as strictly binary. The strongest answers, as shown in the content moderation example above, propose hybrid tiered architectures matched to different latency/cost/accuracy requirements within the same system.