maas-base/docs/performance-lab/references/evaluating-lmcache-prefill-acceleration-in-vllm.md

Evaluating LMCache Prefill Acceleration in vLLM

LMCache is an extensible KV Cache Layer for LLM inference designed to address key challenges in large-scale deployment scenarios. This documentation evaluates the performance impact of LMCache on vLLM inference, particularly focusing on prefill stage acceleration and its implications for various workload patterns.

Conclusions

1. LMCache provides significant prefill acceleration in scenarios with high cache hit rates, achieving up to +355.3% input TPS improvement and -58.8% reduction in TTFT for long-context (20K tokens) multi-turn conversations in the experiments.

2. Performance benefits are highly workload-dependent:

   • Optimal scenarios: Multi-turn conversations with shared prefixes and repeated patterns
   • Suboptimal scenarios: Random inputs with no cache reuse patterns

3. Chunk size optimization The default 256 chunk size shows the optimal results in tested configurations.

4. Cache miss scenarios incur overhead, showing -3% to -15% performance degradation when no cache reuse occurs, making LMCache most suitable for workloads with predictable prefix patterns.

Technical Background

LMCache Overview

LMCache extends vLLM's KV cache capabilities through:

Component	Description
CPU Offloading	Extends cache capacity beyond GPU VRAM limits
Chunk-based Management	Efficient cache storage and retrieval with configurable chunk sizes
Multiple Backends	Support for local storage, Redis, and custom backends like Mooncake
Distributed KV Cache	Shared cache across multiple vLLM instances

Key Use Cases

1. Low Prefix Cache Hit Rates: Mitigates GPU VRAM limitations and cache eviction issues
2. Distributed Cache Sharing: Enables cache sharing across multiple vLLM instances
3. PD Disaggregation: Supports disaggregated deployment architectures

Experimental Setup

• Model: Qwen3-8B
• Hardware: NVIDIA RTX 4090 24GB
• vLLM Version: v0.10.1.1
• Benchmark Method: Multi-turn conversation benchmark

??? info "Serving Commands"

```bash
# Standard vLLM serving
vllm serve Qwen/Qwen3-8B

# LMCache-enabled serving
##### lmcache_config.yaml
chunk_size: 256
local_cpu: true
max_local_cpu_size: 50
#####
LMCACHE_CONFIG_FILE=/root/lmcache_config.yaml vllm serve /root/Qwen3-8B \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```

??? info "Benchmark Scripts"

```bash
# Multi-turn bench scripts
# Ref: https://github.com/vllm-project/vllm/tree/main/benchmarks/multi_turn

##### generate_multi_turn.json
{
    "filetype": "generate_conversations",
    "num_conversations": 24,
    "text_files": ["pg1184.txt"],
    "print_stats": false,
    "prompt_input": {
        "num_turns": {
            "distribution": "uniform",
            "min": 12,
            "max": 18
        },
        "common_prefix_num_tokens": {
            "distribution": "constant",
            "value": 500
        },
        "prefix_num_tokens": {
            "distribution": "lognormal",
            "average": 4000,
            "max": 20000
        },
        "num_tokens": {
            "distribution": "uniform",
            "min": 120,
            "max": 160
        }
    },
    "prompt_output": {
        "num_tokens": {
            "distribution": "uniform",
            "min": 80,
            "max": 120
        }
    }
}
#####

python benchmark_serving_multi_turn.py --model $MODEL_PATH --input-file generate_multi_turn.json --num-clients 10 --max-active-conversations 10
```

Experimental Results

Multi-turn Conversation Performance

5K Input Tokens

Configuration	Input TPS	Total TPS	Mean TTFT (ms)	Mean TPOT (ms)
Without LMCache	5849	5957	4350.48	48.47
With LMCache	9426 (+61.2%)	9592	2646.09 (-39.2%)	30.60 (-36.9%)

20K Input Tokens

Configuration	Input TPS	Total TPS	Mean TTFT (ms)	Mean TPOT (ms)
Without LMCache	4312.17	4335.71	5070.52	33.91
With LMCache	7750.60 (+79.7%)	7792.92	2091.00 (-58.8%)	25.83 (-23.8%)

20K Input Tokens + 1 Output Token

Configuration	Input TPS	Total TPS	Mean TTFT (ms)
Without LMCache	7443.2	7443.6	4658.66
With LMCache	33887.9 (+355.3%)	33889.8	980.87

Tuning Chunk Size

Chunk Size	Input TPS	Performance Gain	Mean TTFT (ms)
64	33820.3	+354.4%	985.28
256	33887.9	+355.3%	980.87
1024	31634.0	+325.0%	1055.69

Cache Miss Scenarios (Random Dataset)

??? info "Benchmark Scripts"

```
vllm bench serve --model Qwen/Qwen3-8B --endpoint-type openai-chat --endpoint /v1/chat/completions --dataset-name random --random-input-len 1024 --random-output-len 128 --num-prompts 100 --seed 40
```

1K Input Tokens

Metric	Without LMCache	With LMCache	Change
Output TPS	579.86	561.44	-3.2%
Total TPS	5212.32	5046.72	-3.2%
Mean TTFT (ms)	8886.36	9242.72	+4.0%
Mean TPOT (ms)	42.08	43.47	+3.3%

8K Input Tokens

Metric	Without LMCache	With LMCache	Change
Output TPS	77.87	66.77	-14.3%
Total TPS	5060.79	4338.96	-14.3%
Mean TTFT (ms)	80610.70	92682.22	+15.0%
Mean TPOT (ms)	43.33	42.27	-2.4%

20K Input Tokens

Metric	Without LMCache	With LMCache	Change
Output TPS	22.97	21.77	-5.2%
Total TPS	3698.09	3504.41	-5.2%
Mean TTFT (ms)	277456.13	292811.62	+5.5%
Mean TPOT (ms)	31.68	32.80	+3.5%

All VRAM KV Cache Hit Scenarios

1K Input Tokens

Metric	Without LMCache	With LMCache	Change
Output TPS	5954.33	5752.71	-3.3%
Total TPS	53589.01	51802.45	-3.3%
Mean TTFT (ms)	3052.08	3247.10	+6.4%
Mean TPOT (ms)	38.40	39.04	+1.7%

8K Input Tokens

Metric	Without LMCache	With LMCache	Change
Output TPS	3676.71	3656.30	-0.6%
Total TPS	238986.41	237659.44	-0.6%
Mean TTFT (ms)	5060.41	5326.37	+5.3%
Mean TPOT (ms)	54.37	53.86	-1.0%

20K Input Tokens

Metric	Without LMCache	With LMCache	Change
Output TPS	2213.12	1972.32	-10.9%
Total TPS	356312.70	317543.74	-10.9%
Mean TTFT (ms)	9649.76	10109.51	+4.8%
Mean TPOT (ms)	87.10	94.26	+8.2%

Remote Storage Backend Performance (20K Tokens TTFT)

Backend	Cache Miss (s)	Cache Hit (s)	Performance Boost
lmcache_server	0.739	0.324	2.28x
Redis	0.746	0.388	1.92x
Mooncake (TCP)	0.759	0.362	2.10x