Topic

The Evolution of Multi-Tier Storage in Mooncake Store: Tiered KV Cache and Large-Scale Model Weight Management

This talk will introduce the recent evolution of Mooncake Store for managing large-scale model data. For KV Cache, DRAM continues to provide low-latency access to hot data, while SSDs expand the retained KV working set at a lower cost, reducing data eviction and improving cache hit rates for multi-turn conversations and long-context workloads. Mooncake has evolved its Store from distributed DRAM to local commodity NVMe, NVMe KV, shared NVMe-oF, and remote DFS, while using object replica management, asynchronous flushing, eviction-to-disk, SSD cache promotion, and batched I/O to efficiently orchestrate data movement across storage tiers. For large-scale model weights, trillion-parameter models such as Kimi K3 introduce new challenges in TB-scale checkpoint loading, cross-node distribution, and parallel layout transformation. Mooncake addresses these challenges through Manifests, chunked objects, range reads, and multi-buffer transfers, reducing redundant disk reads and intermediate data copies. To handle layout changes across Tensor Parallelism (TP), Pipeline Parallelism (PP), Expert Parallelism (EP), and Data Parallelism (DP), Mooncake further uses N-dimensional overlap planning and runtime binding to generate efficient Reshard transfer plans. The talk will demonstrate how Mooncake Store provides a unified storage layer for both tiered KV data management and large-scale model weight movement, enabling efficient data orchestration across heterogeneous storage resources.