GPU Sold

How much context fits in 24 GB of VRAM?

Tested Nov 16, 2026 on a used RTX 3090 24 GB (300 W limit), llama.cpp b12023, CUDA build

On a 24 GB RTX 3090, Qwen3.6 27B loaded with 64K tokens of context using the default cache and 128K with an 8-bit (q8_0) cache. Qwen3.5 9B handled the largest size we tried, 256K, with either cache. Qwen3.6 35B-A3B topped out at 64K and 128K. Gemma 4 31B topped out at 32K and 64K.

Largest context that loaded on a 24 GB RTX 3090, and the VRAM it used
ModelDefault cache (f16)VRAM GB8-bit cache (q8_0)VRAM GB
Qwen3.5 9B256K14.0256K11.2
Qwen3.6 27B64K20.1128K20.9
Qwen3.6 35B-A3B64K22.4128K22.7
Gemma 4 31B32K21.764K21.8

Buy a used RTX 3090 24GB at a fair price

Typical sold price $1,650. Most sell between $1,583 and $1,749, shipping included.

From 258 completed US eBay sales · price updated Nov 18 · price history

See used RTX 3090s on eBay

We may earn a commission if you buy through these links. It does not change your price or the sold prices on this page. Disclosure

VRAM used at each context size

0481216202416K32K64K128K256K24 GB card limitGB
Qwen3.5 9B, default (f16)Qwen3.5 9B, 8-bit (q8_0)Qwen3.6 27B, default (f16)Qwen3.6 27B, 8-bit (q8_0)Qwen3.6 35B-A3B, default (f16)Qwen3.6 35B-A3B, 8-bit (q8_0)Gemma 4 31B, default (f16)Gemma 4 31B, 8-bit (q8_0)× = ran out of VRAM at that size
VRAM used in GB at each context size. “out of VRAM” is the first size that did not load.
ModelKV cache16K32K64K128K256K
Qwen3.5 9Bdefault (f16)6.2 GB6.7 GB7.8 GB9.8 GB14.0 GB
Qwen3.5 9B8-bit (q8_0)6.0 GB6.4 GB7.0 GB8.4 GB11.2 GB
Qwen3.6 27Bdefault (f16)17.1 GB18.1 GB20.1 GBout of VRAM
Qwen3.6 27B8-bit (q8_0)16.7 GB17.3 GB18.5 GB20.9 GBout of VRAM
Qwen3.6 35B-A3Bdefault (f16)21.4 GB21.7 GB22.4 GBout of VRAM
Qwen3.6 35B-A3B8-bit (q8_0)21.3 GB21.5 GB21.9 GB22.7 GBout of VRAM
Gemma 4 31Bdefault (f16)20.5 GB21.7 GBout of VRAM
Gemma 4 31B8-bit (q8_0)19.4 GB20.2 GB21.8 GBout of VRAM
How we tested

llama-server -ngl 99 -fa on -np 1 -c N -ctk/-ctv f16|q8_0; 'fits' = server started and allocated the full context; next size up ran out of VRAM

Full method

Does an 8-bit cache slow things down?

At 32K tokens into a conversation, the 8-bit cache cost Qwen3.6 27B about 10% of its speed (35.8 → 32.1 tok/s) and Qwen3.6 35B-A3B about 14% of its speed (134.1 → 115.1 tok/s).

Generation tok/s at different points in a conversation, one chat
ModelCachetok/s at startat 8Kat 32K
Qwen3.6 27Bdefault (f16)39.338.535.8
Qwen3.6 27B8-bit (q8_0)39.437.132.1
Qwen3.6 35B-A3Bdefault (f16)149.0147.3134.1
Qwen3.6 35B-A3B8-bit (q8_0)147.7138.0115.1
How we tested

llama-bench -d 0,8192,32768 -p 512 -n 128

Full method

Rule of thumb

What about 70B models?

We have not measured a dense 70B yet. The largest models we ran were mixture-of-experts with part of the experts in system RAM: gpt-oss 120B REAP (58B) at 34.7 tok/s and Qwen3 Coder Next 80B-A3B at 41.4 tok/s. A 2-bit 70B test is scheduled. Running models bigger than 24 GB

Questions

Does q8_0 KV cache fit more context?

Yes, in our test. For 3 of the 4 models we tried, the 8-bit cache doubled the largest context that loaded. Qwen3.5 9B fit the largest size we tried with either cache.

How much VRAM does 128K context need?

It depends on the model and cache. In our test: Qwen3.5 9B used 9.8 GB at 128K with f16; Qwen3.5 9B used 8.4 GB at 128K with q8_0; Qwen3.6 27B used 20.9 GB at 128K with q8_0; Qwen3.6 35B-A3B used 22.7 GB at 128K with q8_0.

Why does a 9B model fit 256K?

Because the model itself is small, so most of the card is left for context. Qwen3.5 9B used 11.2–14.0 GB at 256K in our test.

How Much Context Fits in 24 GB of VRAM? (Tested)