How much context fits in 24 GB of VRAM?
Tested Nov 16, 2026 on a used RTX 3090 24 GB (300 W limit), llama.cpp b12023, CUDA build
On a 24 GB RTX 3090, Qwen3.6 27B loaded with 64K tokens of context using the default cache and 128K with an 8-bit (q8_0) cache. Qwen3.5 9B handled the largest size we tried, 256K, with either cache. Qwen3.6 35B-A3B topped out at 64K and 128K. Gemma 4 31B topped out at 32K and 64K.
| Model | Default cache (f16) | VRAM GB | 8-bit cache (q8_0) | VRAM GB |
|---|---|---|---|---|
| Qwen3.5 9B | 256K | 14.0 | 256K | 11.2 |
| Qwen3.6 27B | 64K | 20.1 | 128K | 20.9 |
| Qwen3.6 35B-A3B | 64K | 22.4 | 128K | 22.7 |
| Gemma 4 31B | 32K | 21.7 | 64K | 21.8 |
Buy a used RTX 3090 24GB at a fair price
Typical sold price $1,650. Most sell between $1,583 and $1,749, shipping included.
From 258 completed US eBay sales · price updated Nov 18 · price history
We may earn a commission if you buy through these links. It does not change your price or the sold prices on this page. Disclosure
VRAM used at each context size
| Model | KV cache | 16K | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|---|
| Qwen3.5 9B | default (f16) | 6.2 GB | 6.7 GB | 7.8 GB | 9.8 GB | 14.0 GB |
| Qwen3.5 9B | 8-bit (q8_0) | 6.0 GB | 6.4 GB | 7.0 GB | 8.4 GB | 11.2 GB |
| Qwen3.6 27B | default (f16) | 17.1 GB | 18.1 GB | 20.1 GB | out of VRAM | |
| Qwen3.6 27B | 8-bit (q8_0) | 16.7 GB | 17.3 GB | 18.5 GB | 20.9 GB | out of VRAM |
| Qwen3.6 35B-A3B | default (f16) | 21.4 GB | 21.7 GB | 22.4 GB | out of VRAM | |
| Qwen3.6 35B-A3B | 8-bit (q8_0) | 21.3 GB | 21.5 GB | 21.9 GB | 22.7 GB | out of VRAM |
| Gemma 4 31B | default (f16) | 20.5 GB | 21.7 GB | out of VRAM | ||
| Gemma 4 31B | 8-bit (q8_0) | 19.4 GB | 20.2 GB | 21.8 GB | out of VRAM |
How we tested
llama-server -ngl 99 -fa on -np 1 -c N -ctk/-ctv f16|q8_0; 'fits' = server started and allocated the full context; next size up ran out of VRAM
Does an 8-bit cache slow things down?
At 32K tokens into a conversation, the 8-bit cache cost Qwen3.6 27B about 10% of its speed (35.8 → 32.1 tok/s) and Qwen3.6 35B-A3B about 14% of its speed (134.1 → 115.1 tok/s).
| Model | Cache | tok/s at start | at 8K | at 32K |
|---|---|---|---|---|
| Qwen3.6 27B | default (f16) | 39.3 | 38.5 | 35.8 |
| Qwen3.6 27B | 8-bit (q8_0) | 39.4 | 37.1 | 32.1 |
| Qwen3.6 35B-A3B | default (f16) | 149.0 | 147.3 | 134.1 |
| Qwen3.6 35B-A3B | 8-bit (q8_0) | 147.7 | 138.0 | 115.1 |
Rule of thumb
- The model file has to fit first, plus room for buffers. In our test, VRAM at 16K was 0.8–3.4 GB more than the model file.
- Context then fills what is left. VRAM grows with context: Qwen3.6 27B used 17.1 GB at 16K and 20.1 GB at 64K with the default cache.
- For 3 of the 4 models we tried, the 8-bit cache doubled the largest context that loaded. Qwen3.5 9B fit the largest size we tried with either cache.
What about 70B models?
We have not measured a dense 70B yet. The largest models we ran were mixture-of-experts with part of the experts in system RAM: gpt-oss 120B REAP (58B) at 34.7 tok/s and Qwen3 Coder Next 80B-A3B at 41.4 tok/s. A 2-bit 70B test is scheduled. Running models bigger than 24 GB
Questions
Does q8_0 KV cache fit more context?
Yes, in our test. For 3 of the 4 models we tried, the 8-bit cache doubled the largest context that loaded. Qwen3.5 9B fit the largest size we tried with either cache.
How much VRAM does 128K context need?
It depends on the model and cache. In our test: Qwen3.5 9B used 9.8 GB at 128K with f16; Qwen3.5 9B used 8.4 GB at 128K with q8_0; Qwen3.6 27B used 20.9 GB at 128K with q8_0; Qwen3.6 35B-A3B used 22.7 GB at 128K with q8_0.
Why does a 9B model fit 256K?
Because the model itself is small, so most of the card is left for context. Qwen3.5 9B used 11.2–14.0 GB at 256K in our test.