A growing number of applications depends on Machine Learning (ML) functionality and benefits from bothhigher quality of ML predictions and better timeliness (latency) at the same time. A growing body of research incomputer architecture, ML, and systems software literature focuses on reaching better latency/accuracy tradeoffsfor ML models. Efforts include compression, quantization, pruning, early-exit models, mixed DNN precision, aswell as ML inference accelerator designs that minimize latency and energy, while preserving delivered accuracy.All of them, however, yield improvements for a single point in the latency/accuracy tradeoff space. We make acase for applications that operate in dynamically changing deployment scenarios, where no single point is optimal.We draw on recently proposed weight-shared SuperNet mechanism to enable serving a stream of queries that uses(activates) different SubNets within this weight-shared construct. This creates an opportunity to exploit the inherenttemporal locality with our proposed SubGraph Stationary (SGS) optimization. We take a hardware-softwareco-design approach with a real implementation of SGS in SushiAccel and the implementation of a softwarescheduler SushiSched controlling which SubNets to serve and what to cache in real time. Combined, they arevertically integrated into SUSHI—an inference serving stack, which yields up to 41% improvement in latency,0.98% increase in served accuracy, and achieves up to 52.6% saved energy.