AI models like the one behind "thinking mode" chatbots write out long chains of reasoning before they answer. That makes them more accurate, but it is expensive. When many people ask questions at the same time, the server runs out of room and most people get no answer at all. This page shows the problem, our fix, and what happened when we tested it.
Ask the model (Qwen3-4B) a math question and you can let it answer right away, or let it think first. Thinking writes thousands of extra words, called tokens. Every token the model is still working on takes up GPU memory.
And most of the time, thinking doesn't change the answer. When we answered each question both ways:
With one user, wasted thinking only costs time. With a crowd, it is worse. The GPU's memory fills with half-finished reasoning, the server starts throwing work away and redoing it, and new questions wait in line until they time out. On a live GPU, a server where every question thinks answered only 25% of hard questions correctly in time under heavy load, even though it gets 70% right when it does answer.
Every request goes through the same three steps.
Get a fast answer with no thinking. It is cheap, and right about 86% of the time.
Every 256 tokens, ask the thinking model for its current answer. When it agrees with the quick answer, stop: that answer is right 96% of the time.
A small optimizer re-solved every half second puts a price on thinking based on how full the server is. When it's busy, fewer questions get to think.
Two real questions, with the thinking model's answer at each check-in. A green chip is right, a red chip is wrong, and an orange underline means it matches the quick answer.
The price works like a dial. It doesn't stop each question at a fixed point. It decides how much thinking the whole server can afford right now.
Math questions arrive at one GPU: a quiet stretch, a rush at 2.25 questions per second, then quiet again. The left server lets every question think to the end. The right server uses our method. Each square is one question. Press Play, or drag the slider. Click any square to see its question and answers.
How our server answered the questions that arrived in each 20-second window. During the rush it moves questions from thinking to quick answers. When the line clears, thinking comes back.
Correct answers in each phase, averaged over 8 different random streams of questions. The last column is the middle (median) wait from asking to getting an answer during the rush.
"Always quick" is fastest and nearly as accurate on these mostly easy questions. Our server gets slightly more right but makes people wait longer during the rush.
How this replay was made: each square is a real recorded Qwen3-4B answer (MATH-500 and GSM8K), timed by a simulator of a vLLM server on one A100 GPU slice. On the same loads, the simulator lands within about 2% of live server runs. The replay uses the earlier version of our method, which looks up its price in a table built in advance.
We then ran the newest version live: it re-solves the price every half second from what is actually on the server. Test: 300 hard competition math problems (MATH Level 5) the method had never seen, Qwen3-4B on one A100 GPU slice, each question gets 5 minutes, two runs per point. The score is the share of all incoming questions answered correctly within 5 minutes.
Each cell is the share of all incoming questions answered correctly within 5 minutes, averaged over two runs. Gaps of a few points are within noise.
Today, users choose how much a model thinks ("reasoning effort", "thinking budget"). Nothing adjusts it from how busy the server is. Our method lets the server decide how much thinking it can afford right now, so a spike becomes slightly shorter answers instead of an outage.
They run models on their own GPUs for privacy and can't add hardware on demand. When everyone logs in at 9am, everyone still gets an answer.
Traffic jumps the night before an exam. Quick answers during the rush and deep thinking once it's quiet beats error pages for half the students.
New GPUs take minutes to come online; spikes hit in seconds. Instead of "server busy" errors, everyone gets a slightly shorter answer.
The same price can come from a spending budget: spend thinking where it changes the answer, per customer tier.
No method guarantees correct answers: thinking every time still gets 20–30% of hard problems wrong. Under heavy load the real choice is a quick answer or no answer. A thinking-first server got 70% right on what it answered but answered too few in time, so only 25% of all questions got a correct answer. Ours answered everyone and got 65% right. When there is room, ours thinks and scores at or above every other policy.
We trade a few points of accuracy for answering everyone, and only when the server is overloaded.
Most research on reasoning models asks when one question should stop thinking. We ask how a busy server should share thinking across everyone. Seen that way, the server breaks because it runs out of memory, not compute. And the fix is deciding who gets to think, not trimming each answer.
Next: more runs of the traffic-burst test, since that is where the live price clearly beats a hand-tuned rule.