Kubernetes Scheduling and GPU Virtualization for LLM Training: A New Grad PM's Education Guide

The candidates who prepare the most often perform the worst. I saw this during a Q3 2023 hiring loop at NVIDIA for a Technical PM role in the DGX Cloud team. We had a candidate from Stanford who could recite the entire Kubernetes documentation but failed the loop because he treated a scheduling problem as a software bug rather than a resource contention crisis.

He spent 15 minutes explaining Pod affinity rules while the hiring manager was waiting for him to address the actual bottleneck: GPU memory fragmentation. The result was a unanimous No Hire. He knew the syntax, but he had zero judgment on how hardware constraints dictate product roadmaps.

Why does Kubernetes scheduling fail during large-scale LLM training?

Kubernetes fails because its default scheduler is designed for microservices, not for the monolithic, tightly coupled resource demands of an LLM training cluster. In a 2022 debrief at Meta for the PyTorch infrastructure team, we discussed a candidate who suggested using standard K8s Horizontal Pod Autoscaler for a 175B parameter model training run.

The lead engineer shut him down immediately. LLM training isn’t about scaling out; it is about gang scheduling. If you have 512 H100 GPUs and 511 are allocated but one is stuck in a pending state, the entire training job sits idle while burning $12,000 per hour in idle compute.

The problem isn’t your scheduling logic—it’s your lack of awareness regarding the cost of synchronization. In the Meta loop, the distinction that separates an L4 from an L5 PM is the understanding that LLM training is not X (a distributed system of independent tasks), but Y (a single massive computation split across a fabric). A standard K8s scheduler treats a GPU as a generic resource.

In reality, if two GPUs are on different PCIe switches, the latency spikes. At Google Cloud, during a Vertex AI debrief, a candidate was rejected because he didn’t mention NVLink or GPUDirect RDMA when asked how to optimize a training cluster. He focused on the software layer, but the bottleneck was the physical topology.

The judgment here is simple: if you are designing for LLM training, you ignore the default K8s scheduler and move toward specialized orchestrators like Volcano or Kueue. In a real-world scenario at an AI startup in SF, we saw a team lose three weeks of training time because they used standard K8s pods that were scattered across different racks.

The inter-node communication latency killed their throughput. They were seeing 40% GPU utilization because the GPUs were waiting for data to travel across the network. The fix wasn’t more code; it was implementing topology-aware scheduling to ensure pods were physically adjacent.

How does GPU virtualization actually impact LLM inference and training costs?

GPU virtualization is a trade-off between utilization and performance, where the goal is to move from the waste of dedicated GPUs to the efficiency of fractional sharing. In a 2024 hiring committee at OpenAI, we debated a candidate’s proposal for a multi-tenant inference API. He suggested using NVIDIA MIG (Multi-Instance GPU) for everything.

The senior architects pushed back because MIG provides hardware-level isolation that is too rigid for dynamic LLM workloads. MIG is not for flexibility; it is for guaranteed SLAs. If you carve an H100 into seven instances, you’ve locked in the memory, which is a death sentence for a model that needs a sudden burst of VRAM for a long-context window.

The core insight is that the problem isn’t the virtualization technology—it’s the memory overhead. At a mid-sized AI lab in Palo Alto, I saw a PM try to save $200,000 in monthly cloud spend by implementing time-slicing for GPU sharing.

The result was a catastrophic drop in tokens-per-second because of the context-switching overhead. They discovered that for LLM inference, the cost of swapping weights in and out of VRAM is higher than the cost of just buying more GPUs. The judgment: for training, virtualization is almost always a mistake; for inference, it is a requirement for margin.

Most new grads confuse vGPU (virtual GPU) with MIG. In a technical screen for a Core Infra role at Azure, a candidate claimed vGPU would allow them to “overprovision” training clusters. The interviewer’s reaction was cold.

You cannot overprovision a training job. If a model requires 80GB of VRAM to hold the weights and gradients, providing 40GB via virtualization doesn’t make it “cheaper”—it makes the job crash with an Out-of-Memory (OOM) error. The contrast is clear: virtualization is for the “long tail” of small inference requests, not for the “heavy lift” of pre-training.

What is the real-world trade-off between MIG and Time-Slicing for AI products?

The trade-off is a choice between deterministic performance (MIG) and maximum flexibility (Time-Slicing), and choosing the wrong one kills your unit economics. During a product review at a stealth-mode LLM startup, the PM proposed Time-Slicing to allow 10 researchers to share one A100. The lead engineer pointed out that since Time-Slicing doesn’t isolate memory, one researcher’s buggy script could trigger a CUDA OOM that crashed the other nine users’ sessions. This is the “noisy neighbor” problem. In a production environment, this isn’t a technical glitch; it’s a product failure.

At NVIDIA, the internal rubric for GPU product management emphasizes that MIG is for “hard isolation.” If you are building a managed service where customers pay for a guaranteed 20GB of VRAM, you use MIG. If you are building an internal playground for data scientists who don’t care if their job takes 10 minutes longer, you use Time-Slicing.

I recall a debrief where a candidate suggested a “hybrid approach” without defining the trigger for switching. The verdict was “No Hire” because the candidate was trying to avoid making a hard product decision.

The organizational psychology of GPU allocation is about power and priority.

In a Q1 2024 loop at Anthropic, the interview question was: “How do you allocate 1,000 GPUs between a research team and a product team?” The failing answer was “I’d use a fair-share scheduler.” The winning answer was “I’d implement a preemptible priority queue where product jobs have a hard SLA and research jobs are killed the moment a product job enters the queue.” This demonstrates an understanding that in the GPU world, the scheduler is not a tool for fairness, but a tool for revenue protection.

What do FAANG hiring committees look for in a Technical PM for AI Infra?

Hiring committees look for the ability to map hardware constraints to business P&L, not the ability to write YAML files. In a Google Cloud HC, a candidate’s design for a GPU-as-a-Service product was rejected because he spent 20 minutes on the API definition and zero minutes on the power and cooling constraints of the data center.

He treated the GPU as a cloud abstraction, but at the scale of 10,000 H100s, the product is actually a power-grid and thermal-management problem. The HC’s verdict: “Candidate is a software PM, not an infra PM.”

The difference is that a software PM thinks about “latency,” while an infra PM thinks about “TFLOPS per Watt.” In a 2023 interview at AWS for the Trainium team, a candidate was asked how to handle a failing GPU in a cluster of 2,048.

The candidate suggested “restarting the pod.” The interviewer corrected him: in a distributed training run, a single GPU failure can corrupt the entire checkpoint. You don’t just restart a pod; you have to roll back the entire cluster to the last global checkpoint, which could mean losing 4 hours of compute—roughly $50,000 in wasted spend.

The signal we look for is “hardware empathy.” I once saw a candidate at Meta who described the exact cost of a 400Gbps InfiniBand cable versus a standard Ethernet cable. He explained how the choice of fabric affects the “all-reduce” operation in distributed training. That level of specificity shifted the vote from a “Leaning Yes” to a “Strong Hire.” He didn’t just know K8s; he knew why K8s is often the wrong tool for the job. He understood that the network is the computer.

How do you negotiate a compensation package for an AI Infra PM role?

Negotiating an AI Infra role requires leveraging the scarcity of “full-stack” talent—those who understand both the CUDA kernel and the K8s orchestrator. In a 2024 negotiation for a Senior PM role at a Tier-1 AI lab, the candidate had an offer for $190,000 base and 0.02% equity.

He countered by citing a competing offer from a cloud provider with a $215,000 base and a $75,000 sign-on bonus. He didn’t just ask for more money; he argued that his ability to reduce GPU idle time by 15% through better scheduling would save the company $2M in OpEx per year. That specific value proposition pushed the equity to 0.05%.

The mistake most new grads make is negotiating based on “market average” instead of “replacement cost.” In a conversation with a candidate who had a $165,000 base offer from a mid-tier company, I told him: “You aren’t a PM; you are a resource optimizer.” I advised him to frame his value in terms of “GPU utilization rate.” If you can prove you can move a cluster from 60% to 80% utilization, you aren’t asking for a $20k raise; you are delivering millions in found capacity.

For a New Grad role at a company like Google or Meta, the base is usually non-negotiable (e.g., $145,000 to $160,000), but the sign-on and equity are where the war is won. In one case, a candidate secured a $50,000 sign-on bonus by proving they had a relationship with a key vendor at NVIDIA that could accelerate their H100 allocation. The company didn’t pay for the candidate’s skills; they paid for the candidate’s access. In the AI gold rush, access to compute is the only currency that matters.

Preparation Checklist

• Map the data path from the NVMe drive to the GPU HBM3 memory to understand where the actual bottlenecks occur (the PM Interview Playbook covers the system design patterns for these bottlenecks with real debrief examples).
• Analyze the cost of a single H100 node ($30k - $40k) and calculate the hourly burn rate of a 1,000-node cluster to understand the stakes of a scheduler failure.
• Compare the latency of PCIe Gen5 versus NVLink 4.0 to explain why pod placement in K8s must be topology-aware.
• Draft a “preemption policy” for a multi-tenant cluster, defining exactly who gets killed first when a high-priority job arrives.
• Research the difference between synchronous and asynchronous SGD (Stochastic Gradient Descent) to understand why “gang scheduling” is mandatory for LLMs.
• Review the “all-reduce” and “all-gather” communication primitives to explain why network bandwidth is the primary constraint for scaling LLM training.

Mistakes to Avoid

• Confusing Utilization with Efficiency. BAD: “I will increase GPU utilization to 95% to save money.” GOOD: “I will increase MFU (Model Flops Utilization) by optimizing the batch size and reducing communication overhead, which reduces the total training time from 30 days to 22 days.”

• Treating GPUs as Generic Cloud Resources. BAD: “I’ll use K8s autoscaling to add more GPUs when the queue grows.” GOOD: “I’ll implement a reservation system for H100 clusters because the lead time for hardware is 6 months, making real-time autoscaling an illusion.”

• Over-indexing on the UI/UX of the Orchestrator. BAD: “I will build a dashboard so researchers can see their GPU usage in real-time.” GOOD: “I will build an automated health-check agent that kills and replaces ‘zombie’ GPUs that are idling due to Xid errors, preventing the entire training job from hanging.”

FAQ

What is the most important metric for an LLM Infra PM? Model Flops Utilization (MFU). Everything else is vanity. If your GPUs are 100% utilized but your MFU is 30%, you are wasting 70% of your spend on communication overhead and memory stalls.

Can you use standard Kubernetes for LLM training? Yes, but only as a wrapper. You must use a secondary scheduler like Volcano or Kueue to handle gang scheduling and priority queues, or you will face massive resource fragmentation and wasted compute.

Is MIG better than Time-Slicing? MIG is better for production SLAs and multi-tenant stability. Time-Slicing is better for internal development and maximum flexibility. Use MIG for the customer; use Time-Slicing for the intern.amazon.com/dp/B0GWWJQ2S3).