SpecMD: A Comprehensive Study on Speculative Expert Prefetching

SpecMD: A Comprehensive Study on Speculative Expert Prefetching

AuthorsDuc Hoang, Ajay Jaiswal, Mohammad Samragh Razlighi, Minsik Cho

View publication

Copy Bibtex

Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model’s parameters is used during each inference. However, to translate this sparsity into practical performance, an expert caching mechanism is required. Previous works have proposed hardware-centric caching policies, but how these various caching policies interact with each other and different hardware specification remains poorly understood. To address this gap, we develop SpecMD, a standardized framework for benchmarking ad-hoc cache policies on various hardware configurations. Using SpecMD, we perform an exhaustive benchmarking of several MoE caching strategies, reproducing and extending prior approaches in controlled settings with realistic constraints. Our experiments reveal that MoE expert access is not consistent with temporal locality assumptions (e.g LRU, LFU). Motivated by this observation, we propose Least-Stale, a novel eviction policy that exploits MoE’s predictable expert access patterns to reduce collision misses by up to 85× over LRU. With such gains, we achieve over 88% hit rates with up to 34.7% Time-to-first-token (TTFT) reduction on OLMoE at only 5% or 0.6GB of VRAM cache capacity.

Related readings and updates.

EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

May 19, 2026research area Methods and Algorithms, research area Speech and Natural Language Processingconference ICML