Oral

Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel

Hongyi Jin ⋅ Bohan Hou ⋅ Guanjie Wang ⋅ Ruihang Lai ⋅ Jinqi Chen ⋅ Zihao Ye ⋅ Yaxing Cai ⋅ Yixin Dong ⋅ Xinhao Cheng ⋅ Zhihao Zhang ⋅ Yilong Zhao ⋅ Yingyi Huang ⋅ Lijie Yang ⋅ Jinchen Jiang ⋅ Gabriele Oliaro ⋅ ⋅ Xupeng Miao ⋅ Vinod Grover ⋅ Todd Mowry ⋅ Zhihao Jia ⋅ Tianqi Chen

Abstract

Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent megakernel techniques fuse multiple operators into a single persistent kernel to eliminate launch gaps and expose inter-kernel parallelism, but struggle to handle dynamic shapes and data-dependent computation in real workloads. We present Event Tensor, a unified compiler abstraction for dynamic megakernels. Event Tensor encodes dependencies between tiled tasks, and enables first-class support for both shape and data-dependent dynamism. Built atop this abstraction, our Event Tensor Compiler (ETC) applies static and dynamic scheduling transformations to generate high-performance persistent kernels. Evaluations show that ETC achieves state-of-the-art LLM serving latency while significantly reducing system warmup overhead.

Chat is not available.