Ai Engineer Kubernetes Gpu Scheduling

Why GPU Scheduling Is Its Own Discipline

Kubernetes was designed around CPU and memory as the primary schedulable resources, and GPUs were bolted on afterward via the device plugin framework. That mismatch creates real operational pain that AI infrastructure interviews probe directly: GPUs can’t be subdivided by the default scheduler the way CPU cores can, utilization visibility is poor without extra tooling, and a single misconfigured pod spec can silently waste an entire expensive GPU node. Any AI engineer interviewing for a role that touches training or inference infrastructure in 2026 is expected to know this terrain beyond “I set nvidia.com/gpu: 1 in the pod spec.”

The core architectural fact to internalize: Kubernetes treats GPUs as opaque, indivisible resources through the NVIDIA device plugin. Unlike CPU (which can be requested in fractional millicores) or memory (in bytes), a pod requesting nvidia.com/gpu: 1 gets an entire physical GPU, even if its workload only uses 15% of its compute and memory. This is the single fact interviewers most often build follow-up questions around.

The Device Plugin and Node-Level GPU Discovery

The NVIDIA device plugin (or AMD’s equivalent, or Intel’s, depending on hardware) runs as a DaemonSet on every GPU node, advertises available GPUs to the kubelet, and handles the actual device mounting into containers. Candidates should know the request flow: the scheduler places a pod on a node with sufficient advertised nvidia.com/gpu capacity, the kubelet requests device allocation from the device plugin, and the plugin returns the device paths and environment variables (like NVIDIA_VISIBLE_DEVICES) that get injected into the container.

A common production bug worth knowing: if the device plugin pod crashes or is evicted, the node’s advertised GPU capacity can drop to zero even though the physical GPUs are fine, causing the scheduler to stop placing new pods on an otherwise healthy node. Diagnosing this requires checking kubectl describe node for allocatable GPU count, not just checking nvidia-smi on the node directly.

MIG: Fractional GPUs Done Properly

NVIDIA’s Multi-Instance GPU (MIG) technology, available on A100, H100, and newer architectures, is the primary answer to the “GPUs are indivisible” problem. MIG partitions a physical GPU into up to seven isolated instances, each with dedicated compute cores, memory, and cache, presented to Kubernetes as distinct schedulable resources (e.g., nvidia.com/mig-1g.10gb).

This matters enormously for inference workloads, where a single small model rarely needs a full H100. Teams running many small-to-medium inference services in 2026 typically partition GPUs into MIG slices sized to their actual model memory footprint, dramatically improving GPU utilization and reducing per-service cost. The tradeoff candidates should be able to name: MIG partitions are fixed at profile-creation time (usually requiring a node reboot or at minimum plugin restart to reconfigure), so it doesn’t help with bursty, unpredictable workloads the way time-slicing does.

Time-slicing is the lighter-weight alternative — multiple pods share a single physical GPU without hardware isolation, trading isolation and predictable performance for zero reconfiguration overhead. It’s appropriate for training/dev environments and low-priority batch jobs, not for latency-sensitive production inference sharing a GPU with other tenants.

Bin Packing, Node Affinity, and Cost Efficiency

GPU nodes are the most expensive line item in most AI infrastructure budgets, so bin packing efficiency directly affects cost. The default Kubernetes scheduler optimizes for spreading pods across nodes (for availability), which is often the wrong objective for GPU workloads where you want to pack pods tightly onto fewer nodes so idle GPU nodes can be scaled down entirely.

The standard 2026 pattern uses a combination of:

• Node affinity and taints/tolerations to keep GPU workloads segregated from CPU-only workloads and prevent accidental scheduling of non-GPU pods onto expensive GPU nodes
• Custom schedulers or scheduler extensions (like the Kubernetes scheduler framework’s score plugins, or third-party schedulers like Volcano and Kueue) that explicitly bin-pack GPU workloads to minimize the number of partially-utilized nodes
• Cluster autoscaler tuning with GPU-aware scale-down policies, since the default autoscaler can be slow to recognize a GPU node with low utilization as a scale-down candidate if any pod is still running on it

Gang Scheduling for Distributed Training

A distinct failure mode from inference scheduling: distributed training jobs (multi-node data or model parallelism) need all their pods scheduled simultaneously, or none at all — partial scheduling wastes GPU-hours on the pods that did get placed while waiting for the rest. Default Kubernetes scheduling has no concept of this “all or nothing” requirement, which is why gang scheduling frameworks like Kueue (now the CNCF-endorsed standard as of 2025-2026) and Volcano exist.

Candidates working with large training clusters should be able to explain gang scheduling’s core mechanism: the job is only admitted to the cluster once enough resources are confirmed available for the entire pod group, preventing the classic scenario where 6 of 8 required nodes get GPU pods placed and then sit idle waiting for 2 more nodes to free up.

Frequently Asked Questions

Q: Do I need deep Kubernetes GPU scheduling knowledge for an AI engineer role, or is this more of an ML infra/platform engineer topic? A: It depends on the role, but in 2026 the line has blurred significantly — many “AI engineer” postings expect at least conversational fluency in how their inference workloads get scheduled, since cost and latency both trace back to scheduling decisions. Platform-heavy roles expect hands-on MIG/Kueue configuration experience.

Q: What’s the single most common GPU scheduling mistake teams make? A: Requesting whole GPUs for inference workloads that use a fraction of the memory and compute, then wondering why GPU costs are high relative to traffic. The fix — MIG partitioning or better bin packing — is usually straightforward once someone actually measures per-pod GPU utilization, which most teams don’t do by default.

Q: How should I demonstrate Kubernetes GPU scheduling knowledge if I’ve only worked with managed inference platforms (SageMaker, Vertex AI) that abstract this away? A: Be honest about the abstraction layer you’ve worked at, but show you understand what’s happening underneath — that’s often the differentiator interviewers are looking for. The 0-to-1 AI Engineer Interview Playbook (https://www.amazon.com/dp/B0H2CML9XD?tag=sirjohnnymai-20) covers how to translate managed-platform experience into infrastructure-level talking points that hold up under follow-up questions.

Closing Notes

Kubernetes GPU scheduling interview questions reward specificity: naming MIG versus time-slicing tradeoffs, explaining why gang scheduling exists, or describing a real utilization number you improved. Generic answers about “setting resource limits” signal a candidate who hasn’t operated GPU infrastructure under real cost pressure.