Custom Routing for Inference Optimization in Meta Recommendation Systems

CustomRouting for Inference Optimization in Meta Recommendation Systems

In a July 2023 architecture review at Meta’s Menlo Park campus, the lead inference engineer for the Feed ranking system presented a prototype custom routing layer that promised to cut tail latency by 22% while preserving overall throughput. The room included senior engineers from the AI Infra team, product managers from the News Feed group, and a visiting researcher from FAIR. When the engineer showed a trace of a request being shuttled between three different GPU kernels, the head of ranking asked whether the added software complexity would outweigh the latency gains during peak traffic. This moment captures the tension that drives Meta’s investment in custom routing: the need to squeeze every millisecond out of inference without destabilizing a system that serves hundreds of billions of requests each day.

What is custom routing and why does Meta need it for inference optimization?
Custom routing is a software layer that dynamically directs incoming inference requests to specific compute kernels, memory pools, or hardware accelerators based on runtime characteristics such as feature sparsity, model sub‑graph complexity, or real‑time load signals. Meta’s recommendation stacks—especially those powering the Feed, Reels, and Ads—run thousands of model variants that differ in embedding size, depth, and activation patterns. A static routing policy forces every request to follow the same path, causing under‑utilized resources when a request is simple and overload when it is complex. By contrast, custom routing lets the system match each request to the most efficient execution route, which internal measurements showed could reduce average latency by 15‑25% for long‑tail queries without sacrificing p99 throughput. The need arises because Meta’s inference fleet mixes legacy GPU generations (V100, A100) with newer MTIA ASICs and FPGA‑based accelerators; a one‑size‑fits‑all scheduler cannot exploit the heterogeneity effectively.

In a debrief following the July review, the hiring committee noted that the candidate’s explanation of routing criteria included concrete numbers: “We route requests with embedding sparsity above 70% to the sparse‑kernel pool, which cuts compute cycles by 18% on A100s.” This specificity impressed the panel because it tied a design decision to measurable hardware behavior rather than vague performance hopes. The committee also recalled a prior incident where a generic load‑balancer caused a 40% spike in queue depth during a holiday traffic surge, reinforcing why Meta insists on routing decisions that incorporate model‑level metadata, not just system‑level metrics.

How does custom routing reduce latency and improve throughput in recommendation models?
Custom routing attacks latency at three stages: kernel selection, memory placement, and pipeline scheduling. First, the routing layer inspects the input feature vector to detect sparsity patterns; if the vector contains fewer than 5% non‑zero entries, it dispatches the request to a specialized sparse‑matrix multiplication kernel that avoids unnecessary multiply‑add operations. Internal benchmarks from the Q1 2024 performance sweep showed that this path cut kernel execution time from 1.2 ms to 0.9 ms for 78% of Feed requests. Second, the layer assigns each request to a memory pool that matches its activation size; large activation tensors are pinned to HBM2e banks on MTIA chips, while smaller tensors stay in L2 cache, reducing cache‑miss penalties by an estimated 12%. Third, the routing decision feeds into a dynamic pipeline scheduler that can overlap kernel execution with data transfer, effectively hiding latency behind compute.

Throughput gains emerge because the routing layer prevents resource contention. When a burst of complex requests arrives, the router redirects simpler requests to idle cores on older GPUs, keeping the high‑priority MTIA lanes free for heavy workloads. A production experiment conducted during the Black Friday 2023 traffic window demonstrated a 1.3× increase in requests processed per second when custom routing was enabled, while the 99th‑percentile latency stayed within the 100 ms SLA. The engineer who presented the prototype summed it up in a follow‑up tech talk: “We are not making every request faster; we are making the right request run on the right hardware, which lifts the whole system.”

What are the key components of Meta’s custom routing infrastructure?
The infrastructure consists of four tightly integrated pieces: a feature extractor, a routing policy engine, a hardware abstraction layer, and a feedback controller. The feature extractor runs on the CPU shortly after request deserialization and computes lightweight signals such as embedding norm, sparsity ratio, and predicted compute cost using a tiny regression model trained on offline profiling data. This model is only 2 KB in size and adds less than 0.05 ms overhead, as logged in the inference telemetry dashboard for the Q3 2023 rollout.

The routing policy engine consumes those signals and applies a decision tree that maps to concrete compute queues. The tree is versioned and stored in a distributed configuration service; updates are pushed via a canary mechanism that first exposes the new policy to 5% of traffic in the staging cluster. In the October 2023 policy update, the team added a branch that sends requests with high attention‑head variance to the FPGA‑accelerated transformer kernel, a change that reduced latency for those requests by 31% according to the A/B test results.

The hardware abstraction layer presents a uniform API to the routing engine, hiding differences between CUDA kernels on GPUs, MLIR‑based kernels on MTIA, and OpenCL kernels on FPGAs. It also handles memory registration, ensuring that tensors are placed in the correct memory domain before kernel launch. Finally, the feedback controller monitors real‑time metrics—queue depth, kernel execution time, and memory bandwidth usage—and adjusts the policy thresholds every few minutes. During a nightly maintenance window in December 2023, the controller detected a sudden rise in HBM2e latency and automatically shifted 12% of traffic to the L2‑resident path, preventing a potential SLA breach.

How do teams evaluate the trade-offs between routing complexity and performance gains?
Teams weigh three dimensions: development overhead, runtime overhead, and risk of regression. Development overhead is measured in engineer‑weeks required to add a new routing branch; the Meta infra team estimates that each additional branch consumes roughly two weeks of effort, including unit tests, integration tests, and documentation. Runtime overhead is the extra CPU cycles spent in the feature extractor and policy engine; telemetry from the Feed ranking service shows that the current routing layer adds a median of 0.07 ms per request, which is negligible compared to the 1‑2 ms kernel execution window.

Risk of regression is evaluated through shadow mode experiments, where the new routing policy runs in parallel with the production policy without affecting user‑facing outcomes. The shadow run logs discrepancies in kernel selection and latency; if the divergence exceeds a 5% threshold for more than 1% of traffic, the rollout is paused. In the March 2024 rollout of a branch for multimodal embeddings, the shadow mode revealed a 9% increase in CPU usage due to a poorly optimized sparsity estimator, prompting the team to replace the estimator with a lookup table before promoting the branch to canary.

A counter‑intuitive truth that emerged from these evaluations is that the biggest performance wins often come from routing decisions that appear trivial at first glance. For example, directing requests with a bias term larger than 0.8 to a kernel that pre‑adds the bias in hardware saved only 0.04 ms per request, but because such requests accounted for 22% of the total volume, the aggregate latency reduction reached 0.9 ms across the fleet—a gain that justified the engineering effort despite the modest per‑request impact.

Preparation Checklist

• Study Meta’s public AI Infra blog posts from 2022‑2023 to understand the evolution of TorchRec, FBGEMM, and MTIA integration.
• Reproduce the sparse‑kernel routing experiment using the open‑source TorchRec tutorial; modify the sparsity threshold and measure latency on a local A100 instance.
• Review the MLSys 2022 paper “Heterogeneous Scheduling for Large‑Scale Recommendation Models” to grasp the academic foundations that Meta’s routing layer builds on.
• Practice explaining a routing decision with concrete numbers: “If embedding sparsity > 70%, we route to the sparse‑kernel pool, which cuts compute cycles by 18% on A100s.”
• Work through a structured preparation system (the PM Interview Playbook covers ML inference optimization case studies with real debrief examples).
• Prepare to discuss trade‑off frameworks: list three metrics you would track (latency, throughput, error rate) and how you would weigh them against engineering cost.
• Run a small shadow‑mode simulation using Meta’s open‑source “RoutingSim” tool to see how policy changes affect queue depth under synthetic traffic bursts.

Mistakes to Avoid

BAD: Proposing a routing policy that relies solely on system‑level metrics like overall GPU utilization, ignoring model‑specific features such as embedding sparsity or activation size. This approach caused a production incident in early 2023 where the router sent all requests to the newest MTIA chips, creating a bottleneck and increasing tail latency by 18% during a traffic spike.
GROUND: Always enrich the routing signal with at least one model‑level characteristic; the Feed team’s routing engine combines sparsity ratio with predicted compute cost from a lightweight regression model, which prevented the MTIA overload during the same traffic spike.

BAD: Adding a new routing branch without measuring its runtime overhead, assuming the extra CPU cost is negligible. In a July 2023 experiment, a branch that performed a dense matrix multiplication to predict cache‑miss penalty added 0.3 ms per request, erasing the latency gains from the kernel switch and leading to a rollback.
GROUND: Profile the feature extractor and policy engine in isolation before integration; the routing team now requires a sub‑0.1 ms overhead budget for any new signal, verified with microbenchmarks on the target CPU.

GROUND: Skipping shadow‑mode validation and pushing a routing change directly to canary based solely on offline A/B test results. The September 2023 rollout of a branch for quantized embeddings passed offline tests but caused a 12% increase in memory bandwidth usage in production, triggering an automatic rollback after five minutes.
BAD: Treat shadow mode as optional; the routing team’s post‑mortem mandated that every policy change must run in shadow for at least 15 minutes of live traffic and show no statistically significant deviation in latency or error rate before promotion to canary.

FAQ

How does custom routing differ from traditional load balancing in Meta’s inference stack?
Traditional load balancers distribute requests based on coarse signals like current queue length or CPU utilization, treating all inference jobs as identical. Custom routing, by contrast, inspects fine‑grained model attributes—such as embedding sparsity, activation size, and predicted kernel cost—to send each request to the hardware execution path best suited for its computational profile. This model‑aware dispatch reduces wasted cycles and prevents resource contention, yielding latency improvements of 15‑25% for long‑tail queries in production feeds.

What specific hardware does Meta’s custom routing layer target today?
The layer routes requests to three main classes of accelerators: NVIDIA A100 GPUs for dense matrix‑multiply workloads, Meta’s MTIA ASICs for sparse embedding lookups and low‑precision convolutions, and field‑programmable gate arrays (FPGAs) for custom transformer attention kernels. Routing decisions also consider memory hierarchy placement, directing large activation tensors to HBM2e banks on MTIA and smaller tensors to on‑die L2 cache to minimize memory‑latency stalls.

Can the techniques described be applied outside Meta’s recommendation systems?
Yes, the core idea—using lightweight runtime features to select optimal compute kernels—transfers to any setting with heterogeneous hardware and variable workloads, such as LLM serving, ad‑click prediction, or video transcoding. Teams at other companies have replicated the approach by extracting simple signals like token length or frame resolution and mapping them to pre‑profil­ed kernel choices, achieving latency cuts of 10‑20% without major system redesigns. The key is to invest in a low‑overhead feature extractor and a versioned policy engine that can be updated safely via shadow mode and canary rollouts.amazon.com/dp/B0GWWJQ2S3).