Oral

Optimizing Deployment Configurations for LLM Inference

Sungmin Cho ⋅ Jaewon Lee ⋅ Chunqiang Tang ⋅ Yejin Lee ⋅ Geonhwa Jeong ⋅ ⋅ Scott Batura ⋅ ⋅ ⋅ ⋅ Sijia Chen ⋅ ⋅ Bradley Davis ⋅ Summer Deng ⋅ ⋅ Emad El-Haraty ⋅ ⋅ Lu Fang ⋅ Lu Fang ⋅ Joshua Fromm ⋅ ⋅ ⋅ Liangpeng Guo ⋅ ⋅ ⋅ Jianyu Huang ⋅ Aya Ibrahim ⋅ ⋅ Hongyi Jia ⋅ Changkyu Kim ⋅ ⋅ ⋅ ⋅ ⋅ Xiaozhu Meng ⋅ Vlad Tiberiu Mihailescu ⋅ ⋅ Maxim Naumov ⋅ Michal Ostrowski ⋅ ⋅ ⋅ Sarunya Pumma ⋅ ⋅ ⋅ Jeremy Francis Reizenstein ⋅ Rajasi Saha ⋅ ⋅ ⋅ Ruan Silva ⋅ ⋅ Jon Swenson ⋅ ⋅ Chris Thi ⋅ ⋅ Yunfan Wang ⋅ Pengchao Wang ⋅ Wenchen Wang ⋅ ⋅ Bram Wasti ⋅ ⋅ ⋅ Jingyi Yang ⋅ ⋅ ⋅ Jing Zhang ⋅ Yi Zhen ⋅

Abstract

Meta's Large Language Models (LLMs)---the Llama model family---serve nearly one billion monthly active users. Deploying these models for inference involved navigating a complex design space that spanned diverse hardware options (e.g., H100, H200, MI300X), multiple parallelism strategies (tensor, pipeline, expert, context, and data parallelism), and nuanced runtime choices (e.g., continuous batching versus prefill-decode disaggregation)---all while leveraging workload-specific characteristics and meeting stringent service level objectives (SLOs). This paper presents insights we gained from developing and applying a systematic approach to analyze millions of deployment configurations and identify those that maximize throughput while meeting latency SLOs. We share lessons learned from our experience operating Llama inference at scale, including trade-offs among runtime designs, the phase-specific nature of parallelism strategies, opportunities for leveraging hardware heterogeneity, platform scaling behaviors, and system-level implications of model architectures such as Mixture-of-Experts (MoE). We hope our production experience offers practical insights for the broader LLM inference community.

Chat is not available.