· Valenx Press · 2 min read
More PM Career Resources
Explore frameworks, salary data, and interview guides from a Silicon Valley Product Leader.
FAQ
How often should LLM regression tests run in Meta’s CI/CD pipeline?
Regression tests should run synchronously on every merge to the model serving branch (typically 15-20 minutes of CI time) and asynchronously on a nightly basis for the full benchmark suite. Synchronous tests catch regressions in code changes; nightly tests catch regressions from training runs. For features with high traffic volume (Instagram, WhatsApp), synchronous gates are non-negotiable. For internal tooling features, async gates with morning review are acceptable.
What do I do if a regression gate fails and engineering wants to override it?
Do not override regression gates without documented justification and sign-off from your director. At Meta, override requires a written exception request that specifies which metrics are failing, why the failure is acceptable, and what user impact mitigation is in place. The override goes to the ML platform lead and the product director. PMs who push overrides without this process have been flagged in performance reviews. The gate exists to catch failures before users do — overriding it transfers the testing burden to production traffic.
How do I handle regression testing for features that use prompt engineering rather than model fine-tuning?
Prompt engineering changes require their own regression suite because they alter behavior without changing model weights. Build a prompt-specific test case library with 100-200 test cases covering the intended behavior change. Run these test cases in the CI/CD pipeline alongside the model evaluation suite. PMs who ship prompt changes without dedicated regression tests consistently discover edge cases in production — the most common incident type for AI features at Meta in 2024.
