Local AI one year on: from an unusable 15 tokens/second to 400 tok/s on two RTX 5070 Ti cards

At the start of this year, running AI on a home machine was still a frustrating experience: small models were weak, large ones crawled at 15 tokens per second with context shrunk to 8K — not enough to even read a code repository. But according to a hands-on report just shared by a GPU enthusiast in Vietnam, the game has completely changed within a year: on a rig with two RTX 5070 Ti 16GB cards — costing less in total than a single RTX 5090 — a Qwen 27B model peaks at 400 tokens/second generating code, keeps context above 200K, and reaches quality close to leading cloud models.

Quick answer

• Early 2026 (the author's own tests): a single RTX 5080 16GB via llama.cpp only handled small models well; 7B–14B models ran at ~70 tokens/second with up to 32K context but could barely code anything real, while larger models dropped to ~15 tok/s with context cut to just 8K.
• Two 5080s did not save it either: the PCIe x8/x8 bottleneck dragged speed down to ~15–20 tok/s — far too slow for real tasks. Back then, serious local AI work demanded an RTX 5090 32GB or a professional Quadro, expensive enough that local AI was practically reserved for large companies.
• Today: on a Core Ultra 9 285K rig with 128GB RAM and 2× RTX 5070 Ti 16GB, the Qwen 3.8 27B model peaks at ~400 tokens/second when generating code, averages around 140 tok/s, and still holds context above 200K — by the author's account matching a single 5090 32GB despite the PCIe x8 link.
• Model quality is the biggest change: the Flash variant, per the author, approaches Claude Sonnet, while the 27B "works on par with frontier models" — plus unlimited tokens, running 24/7 with no quota interruptions.
• Every figure in this article is the author's personal hands-on measurement on his own and an acquaintance's hardware, not vendor-published data.

Early on: why local AI was "just a toy"

The author — someone with plenty of PC parts and GPUs to tinker with — recalls the start of the year with a single RTX 5080 16GB running through llama.cpp. Two big problems showed up immediately:

• Small models (7B, 9B, 14B): decent speed at ~70 tokens/second and up to ~32K context — but only capable of chatbots and trivial tasks, not real coding.
• Large models: speed fell to ~15 tok/s and context shrank severely to ~8K — the context overflowed before a repository could even be read.

Splitting one model across two 5080 cards did not help: limited by the PCIe slots running at x8/x8, speed dropped hard to ~15–20 tok/s — too slow to assign real work.

The conclusion at the time was blunt: real local AI work required at least an RTX 5090 32GB or a professional Quadro, and with one card already very expensive before the rest of the system, local AI was nearly exclusive to large enterprises. Compared with cloud AI back then, beyond data privacy there was almost no reason to spend the huge sum: about $100/month of cloud was enough for daily work.

Today: a rig "cheaper than one 5090" that does real work

The turning point came when the author got hands-on time with a machine owned by an acquaintance who has run local AI for over six months:

Component	Tested configuration
CPU / RAM	Intel Core Ultra 9 285K / 128GB RAM
GPU	2× RTX 5070 Ti 16GB (running x8/x8, same as earlier tests)
Models tested	Qwen 3.8 27B and Qwen 3.8 Flash, jointly served across both GPUs
Optimizations	Several local-AI optimization repos, notably capy-flash and a deepseek harness
Total cost	The whole PC costs less than a single RTX 5090 at current prices (per the author)

The measured results surprised even the author:

• Qwen 3.8 27B peaks at ~400 tokens/second when generating code, the Flash variant at ~200 tok/s, with an average around 140 tok/s — by his account faster than today's frontier models and entirely fast enough for real work.
• Context stays above 200K while still fitting on the two GPUs; despite the PCIe x8 link, speed is nearly equal to a single 5090 32GB card. The Flash model gets even larger context thanks to the 128GB of system RAM helping with offload.
• Model quality is transformed: Qwen 3.8 Flash, per the author, approaches Sonnet, while the 27B genuinely works on par with frontier models; the upcoming Qwen 4.0 27B is expected to be even hotter.

The benchmark table attached to the original post paints the same picture from other runs: average decode speed of about 248 tokens/second across a mixture of code, prose and Vietnamese, with Vietnamese alone at ~113 tok/s — a detail worth noting for Vietnamese users, since Vietnamese is usually the slowest part of multilingual benchmarks.

Three new local-AI advantages beyond privacy

Previously, the nearly sole reason to bear local AI's cost was data privacy. In this account, the balance now includes very practical advantages:

• Unlimited tokens, never interrupted: no hourly or monthly quotas like cloud plans; the machine can work 24/7, taking long tasks overnight without hitting a ceiling.
• Speed beyond what real work needs: at an average of ~140 tok/s and peaks of 400 tok/s, waiting for the model to produce code is essentially no longer the bottleneck.
• Discrete GPUs win on image and video: media models on dedicated-VRAM GPUs run far faster than on unified-memory machines (Mac, DGX Spark...), and users can always add more GPUs for more VRAM and larger models.

Conversely, unified-memory machines are rated by the rig's owner as quite slow compared with VRAM GPUs (the author has not tried them himself), though they fit very large models thanks to the big unified pool; image processing in particular is severely limited.

What limitations remain?

The author is candid about the downside: local AI pricing is still not for most people, especially with component prices having multiplied. Two mid-range GPUs plus 128GB of RAM remain a significant investment, before electricity and tuning effort. For now, it is a worthwhile long-term investment for heavy users — or simply a tinkering hobby — not yet a cloud replacement for the majority.

Frequently asked questions

Does running two GPUs over PCIe x8/x8 bottleneck performance?

Yes — early in the year it was exactly what dragged a 2× RTX 5080 setup down to ~15–20 tok/s. But in the new test with 2× RTX 5070 Ti, overall speed still came close to a single 5090 — because models, inference software and optimization repos have all improved, the PCIe bottleneck no longer dominates as it did. When adding GPUs for more VRAM to serve larger models, the author likewise rates the inter-slot link speed as far less of an issue than before.

How much VRAM does real local-AI work need?

From this report: 16GB (one 5080) only suits small chatbot-grade models; to run a ~27B model at usable speed and context, today's practical mark is about 32GB of total VRAM (here, two 16GB cards jointly serving one model) plus large system RAM for offloading when needed. Previously that mark meant a single 5090 32GB or a professional Quadro.

What does tokens/second (tok/s) mean for actual work?

It is the speed at which the model generates text/code: ~15 tok/s is slower than reading speed and makes long tasks feel like waiting; ~70 tok/s suffices for chatbots; at 140–400 tok/s like the new test, code generation feels nearly instant and the assign–wait–review loop is no longer blocked by the model.

Has local AI replaced cloud AI?

Not for most users, because the hardware price barrier is still high. But for heavy users the advantages now go beyond privacy: unlimited tokens, 24/7 operation, high speed, and data staying in-house. The economics flip accordingly: instead of paying cloud fees indefinitely, you pay once for hardware and use it without limits — whether that is worth it depends on your actual usage volume.

What about unified-memory machines (Mac, DGX Spark)?

Their strength is the large unified memory pool, fitting very large models that mainstream GPUs cannot hold. Their weakness, per the assessment relayed in the post: inference speed is notably slower than VRAM GPUs, and image processing is very limited. They fit when running extremely large models matters more than speed.

Conclusion

The change over one year is not a single breakthrough but three cumulative ones: mid-size open models became good enough for real work, inference software and optimization repos learned to squeeze mainstream hardware far better, and a two-card mid-range rig now delivers speed that used to require a flagship card costing far more. Local AI is thus moving from "a toy for people with spare GPUs" to a serious work option — with the remaining barrier being almost purely component prices.

Source: a hands-on experience post by Le The Vinh in the AI Vietnam Facebook group (6 Nov 2026), edited and structured by Wotech Electronics. All performance figures are the author's real-world measurements on personal hardware, not manufacturer-published data. Illustration: the benchmark table attached to the original post.