Poster 11

FaaScale: Unlocking Fast LLM Scaling for Serverless Inference

Minchen Yu ⋅ Rui Yang ⋅ Chaobo Jia ⋅ Zhaoyuan Su ⋅ Sheng Yao ⋅ Tingfeng Lan ⋅ Yuchen Yang ⋅ Zirui Wang ⋅ Yue Cheng ⋅ Wei Wang ⋅ Ao Wang ⋅ Ruichuan Chen

Abstract

Serverless computing is an attractive paradigm for cloud-based large language model (LLM) inference, but scaling LLMs on demand remains a major challenge due to high data transfer cost. We present FaaScale, a serverless LLM system that enables fast and resource-efficient model scaling. The key idea is a co-design principle—pipelined multicast inference—which synergizes multicast with dynamic, cross-node pipeline-parallel execution during model transfer. FaaScale implements this design through PipeCast, a model scaling scheme that adaptively multicasts model blocks and dynamically forms inference pipelines on the fly. Coupled with efficient memory management across GPU and host memory, FaaScale handles bursty LLM inference workloads effectively, achieving up to 5× lower tail time-to-first-token latency and 31.3% cost reduction on real-world LLM traces.