Tokens Per Watt: Why Your Context Window Is a Power Decision

Two power walls, twenty-two years apart

In May 2004, Intel cancelled Tejas and Jayhawk. The chips worked on paper. They just couldn’t be cooled. Dennard scaling (the rule that let you shrink transistors, drop voltage, and keep total chip power flat) had stopped holding, leakage current was climbing exponentially, and power density was heading past what air cooling could handle. Clock frequency, the number the entire industry had been marketing on for two decades, stopped going up.

Here’s June 2026. Gartner forecasts global data center electricity consumption at 565 TWh this year, up from 447 TWh in 2025, with power demand hitting 132 GW and projected to reach 290 GW by 2030. AI-optimized servers account for 31% of data center power consumption in 2026 and will pass conventional servers in 2027. The quote from Gartner’s Linglan Wang is the one to hold onto: AI capacity is now constrained by power availability.

TSMC says the same thing from the supply side. At the company’s Amsterdam symposium on May 28, Kevin Zhang told reporters that the improvement customers most want is energy efficiency: across edge, mobile, IoT, and high-performance AI data centers alike. Not performance. Efficiency. TSMC is targeting 30% efficiency improvement per generation while shipping 1,000W chips and looking at megawatt-class systems before the decade is out.

When the constraint moves from “how much compute can I buy” to “how much power can I plug in,” the metric moves with it. That metric is tokens per watt.

OpenAI responded from the demand side: its Jalapeño ASIC, disclosed at Hot Chips in August, ships at 700W where NVIDIA’s equivalents draw 1,200 to 1,400W. The watt budget drove the architecture.

I’ve written before about why flat multi-agent orchestration collapses and how hierarchical chip design fixes it. Short version: I built a 12-agent research system, watched two agents argue politely for 47 LLM calls and $200, and fixed it by grouping them into three subsystems behind coordinators. That post was about correctness and coordination overhead. This one is about the thing I didn’t understand at the time: the same restructuring was an energy intervention, and the energy math is far less intuitive than the cost math.

The Number: Tokens per watt halves when context doubles.

The paper is The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency (Chen et al., March 2026). Here’s the core table for Llama-3.1-70B on H100-SXM5, TP=8, fp16:

Serving context	Concurrent sequences	Power at saturation	tok/W
2K	512	598 W	35.0
4K	256	593 W	17.6
8K	128	583 W	8.97
16K	64	557 W	4.69
32K	32	507 W	2.58
64K	16	435 W	1.50
128K	8	369 W	0.88

Read the power column. It barely moves. The GPU burns roughly the same watts holding 512 sequences as holding 8. What collapses is the numerator: a fixed KV-cache budget holds fewer concurrent sequences as per-token memory footprint grows, so throughput falls linearly with context while power stays flat. Across the full 2K–128K range, the spread is close to 40x.

There’s a second-order effect worth noticing in that table. At 128K the GPU is drawing 369W to deliver 0.88 tok/W. A significant share of that is just the cost of being powered on. The paper measures H100 idle power at 300W, about 43% of TDP. A long-context pool that’s lightly loaded is a space heater with a job title.

Jalapeño’s NUMA architecture pairs each of its 64 core slices with a dedicated HBM4 slice to keep KV-cache local, and it still draws ~550W sustained. Purpose-built silicon shifts the curve; it doesn’t flatten it.

One clarification before someone objects, because it matters for how you use this. The law is about the serving context window a pool is configured for, not the length of any individual prompt. If you rent an API, you don’t set routing topology. What you do control is the context-length distribution your workload presents. That distribution is exactly what decides which part of this curve your requests land on, whether you’re self-hosting or paying someone else’s electricity bill through a per-token price.

Why agents are the worst-case workload for this law

Chat is one turn. Question in, answer out, and most chat interfaces truncate or summarize old turns, so input length stays roughly bounded.

An agent doesn’t get that. To pick its next action, it needs the whole task state: original instructions, tool definitions, every tool result so far, and its own prior reasoning. Most LLM APIs are stateless, so every step re-sends the entire accumulated history as input. The history only grows. That’s not a prompt-engineering problem; it’s the shape of the loop.

The measurements are consistent and unflattering:

Gartner puts agentic models at 5 to 30x more tokens per task than a standard GenAI chatbot.
OpenRouter data compiled by a16z showed agentic requests consuming roughly 15x more tokens than standard human queries, with daily agentic token volume on the platform reaching 7.3 trillion in early August 2026, about 14x the level six months earlier. Agentic usage passed human usage in February.
For coding agents specifically, Bai et al. found agentic tasks consuming roughly 1000x more tokens than code chat or code reasoning on SWE-bench Verified, with input tokens rather than output tokens driving the cost (arXiv:2604.22750).
OpenAI designed Jalapeño explicitly for ‘interactive agents where delays compound across an entire task,’ breaking inference into three phases (prefill, draft, verification) because agentic workloads shift unpredictably between them. The chip is the hardware confession that agents are a structurally different workload.

That last paper contains the finding that should change how you budget. Token usage on the same task varied by up to 30x between runs, and higher token usage did not translate into higher accuracy. Accuracy often peaked at intermediate cost and then saturated. The models also systematically underestimated their own token consumption when asked to predict it in advance.

So: agents burn 5 to 30x (sometimes 1000x) the tokens, the extra tokens are mostly re-read input, the extra spend buys you nothing past a point, and the model can’t tell you in advance how much it’s about to spend. Now put that workload on a curve where efficiency halves every time context doubles.

What chip designers did in 2004

They did not build a better transistor. The transistor wasn’t the problem; the power budget was, and the power budget was fixed.

What they built was architecture:

Voltage islands and power domains. Stop running the whole die at one voltage. Partition it, and let each partition run at the voltage its work actually requires.

DVFS. Scale voltage and frequency dynamically per domain. The GPU block in your phone runs hot when you’re gaming and drops near zero when you’re not. The always-on sensor hub runs at minimum voltage forever. This is why the battery lasts a day.

Clock and power gating. Don’t just slow an idle block down. Stop its clock. Cut its supply entirely. The cheapest joule is the one you never spend.

Dark silicon, eventually. By the early 2010s, the industry accepted that at a fixed power budget, a growing fraction of a chip simply cannot be powered on at once. Design shifted to deciding which parts get to be lit.

The through-line: when the scarce resource stopped being transistors and started being watts, the discipline moved from “make the component faster” to “allocate the budget deliberately.” That’s the transfer.

Three patterns that carry over.

1. Voltage islands → model tiers per subsystem. Not every agent call needs the same model. A formatter is deterministic; it belongs in the cheapest tier. A synthesizer resolving contradictory evidence belongs in the expensive one. In my rebuilt system, the fact-checker handles roughly seven out of ten checks on a small fast model and escalates only the ambiguous remainder.

Be careful about the justification, though, because the 1/W paper is more measured here than the pitch usually is. It compares context-window routing against semantic routing (small model for easy queries, large for hard ones) and finds the long pool is the binding constraint either way; both land at 1.52 tok/W at 64K. The real case for model tiering is per-physical-GPU capacity and cost, plus whether the small model meets your quality bar. It is not that a small model is magically more energy-efficient at the same context length. Say the true thing; it’s still a good argument.

2. Clock gating → don’t wake the expensive path. In silicon, the biggest wins come from blocks that are fully off, not blocks running slowly. The agentic equivalent is a deterministic gate in front of the LLM call: schema validation, a cached result, a rules check, a confidence threshold that terminates instead of retrying. Every LLM call you don’t make is worth more than every call you optimize.

3. Level shifters → compression at domain crossings. A coordinator that summarizes before forwarding isn’t just saving tokens. It’s moving every downstream call to a shorter context window, which on the 1/W curve is where the efficiency lives. Compressing a 50K-token history to 2K at a subsystem boundary moves downstream calls several doublings to the left. That’s the single highest-impact architectural move available to most teams, and it’s free of any hardware decision.

The paper’s fleet-level numbers back the general principle hard: two-pool context routing delivers roughly 2.5x better tokens per watt, an H100→B200 hardware upgrade delivers roughly 1.7x, and because the two levers are independent, they multiply to about 4.25x. Topology beats hardware. Neither alone gets you halfway.

Where the analogy breaks (three places).

This is the part I’d want to read if someone else wrote this post, so here it is.

Power domains are static and formally verified. Context growth is not. In chip design, power intent is captured in a UPF/CPF file, and the isolation cells and level shifters at every domain crossing are formally verified before tapeout. You know at design time what can be off when. Agent context length is dynamic, data-dependent, and (per Bai et al.) stochastic enough to vary 30x between runs of the same task. You cannot statically verify a token budget. The equivalent discipline has to be runtime enforcement: hard per-task ceilings, kill switches, and instrumentation. Design-time verification isn’t available to you.

A level shifter is lossless. A summarizer is not. A level shifter changes the voltage of a signal without changing the bit. Boundary summarization throws information away, and it throws it away non-deterministically. When the research coordinator compresses 50K tokens into a 2K report, sometimes the thing the composition subsystem needed is in the discarded 48K.

Chip designers get to treat their domain crossings as contracts with proven properties. You have to treat yours as a lossy channel and design for the loss. That means the summary schema needs an explicit “what I dropped and why” field, and downstream agents need a way to request the full record.

Your idle domain still burns most of its power. An H100 pulls 300W at idle. Jalapeño sustains ~550W under load against a 700W TDP, but the idle-to-active ratio hasn’t been disclosed. Even on purpose-built silicon, the always-on cost doesn’t vanish; it just shifts.

And here’s where it gets genuinely uncomfortable for the thesis: the 1/W paper explicitly classifies agent-heavy traces as the “dispersed” workload archetype, where 74% of requests fit inside 8K but the remaining 26% stretch toward 64K with a p99 around 32K.

For that shape, the long pool dominates GPU-hours even with optimal routing, because a real fraction of the traffic genuinely needs it. Two-pool routing helps least exactly where agents live. The paper’s suggested lever for that archetype is model architecture: sparse MoE models, whose per-iteration decode time scales with active parameters rather than total, benefit at every context length rather than only in the short tail.

Which is a more useful conclusion than a clean one would have been. If you run agents, routing tricks buy you less than the headline number, and the two things that actually move your curve are compressing the tail and choosing an architecture that’s cheap at long context.

Tokens per watt is a yield metric.

Semiconductor yield is the fraction of dies on a wafer that work. Every defective die wasted the silicon, the power, and the machine time that went into it. The industry didn’t fix yield by inventing a better transistor. It fixed yield by moving test upstream so defects stopped propagating, and by containing failures so one bad block didn’t take the die with it.

Tokens per watt is the same shape of metric, and the denominator is the interesting part. A token spent on a routing decision a better architecture would have avoided is a defective die. An 80K-token context stuffed with history from five agents ago is wasted wafer area. A full-reasoning call for a task a schema check could have settled is a wafer that should have been caught at incoming inspection.

Which means the number to report is not tokens per watt. It’s tokens per completed, accepted task per watt. A summarization that emits 200 tokens and gets thrown away is more expensive than one that emits 400 and gets used. Most observability stacks have the inference telemetry and the outcome telemetry sitting in separate systems and never join them. Join them. That join is your yield number.

For engineering leaders: If your team ships agentic systems, tok/W belongs in your design review the way latency budgets do. Ask your architects to report the context-length distribution of every agent workflow, not just the token total or dollar cost. That single metric tells you whether your system is operating on the efficient part of the curve or burning watts on coordination overhead.

What to do Monday

Before your next design review, run every agent call through four questions:

Q1: Does this call need an LLM at all? If a schema check, cache lookup, or rules engine can answer it, stop here. This is clock gating.
Q2: What’s the shortest context this call can work with? Compress the input to that length before sending it. This is level shifting.
Q3: Which model tier does the quality requirement actually demand? Don’t default to the biggest model. This is voltage islands. (OpenAI took this to the logical extreme: an entirely separate chip architecture for inference, while keeping NVIDIA GPUs for training. That’s not a voltage island; it’s a separate power rail entirely.)
Q4: What’s the hard token ceiling for the task this call belongs to? Set it. Enforce it. This is a power budget.

Pin that on the wiki. Then the specifics:

Plot your context-length distribution, not your token total. p50, p90, p99. Under the 1/W law, your p99 sets the efficiency floor of whatever fleet serves you. A workload with a fat 64K tail is paying roughly 12x per token against a 4K workload on identical hardware.
Compress at every subsystem boundary, and instrument the loss. Cross-boundary messages in kilobytes, not full histories. Log the compression ratio and how often a downstream agent had to ask for something the summary dropped. If that second number is zero, you’re probably not compressing hard enough.
Put a deterministic gate in front of every expensive call. Cache, schema check, confidence threshold. The cheapest token is the one never generated.
Set a hard per-task token ceiling with a kill switch. Bai et al. found accuracy peaking at intermediate cost and saturating after. Unbounded loops don’t buy accuracy; they buy variance and a bill.
Tier your models on quality and per-GPU cost, not on a tok/W claim you can’t support. If your workload is large enough, tiering may eventually mean different silicon entirely – OpenAI’s split between NVIDIA for training and Jalapeño for inference is the extreme version of this. For the rest of us, it means picking the right model size per subsystem and being honest about why.
If you self-host, split your context pools. Roughly 2.5x, independent of and multiplicative with any hardware upgrade. If you’re on an API, the equivalent lever is keeping your requests out of the long tail.

The semiconductor industry spent two decades learning that the scarce resource wasn’t transistors. It was the watts to run them and the methodology to allocate those watts deliberately. Frequency stopped being the headline number and efficiency took over, because physics stopped negotiating.

The AI industry is midway through the same lesson. The scarce resource isn’t model intelligence. It’s the power to run inference, and the architecture that decides how much of that power goes to coordination overhead instead of output.

I’d like to hear how other people are instrumenting this, particularly anyone who has actually joined inference telemetry to outcome telemetry in production, because that’s the number I most want to see and least often find. And if you’ve found prior art outside semiconductors that maps onto the power-budget problem, I’d like to hear that too.

Sources

Chen, H., Liu, X., Liu, Y., Jiang, J., He, B., Liu, X. “The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency.” arXiv:2603.17280, March 2026. Analytical study; H100 figures calibrated to published measurements, B200 figures projected with ±20% uncertainty.
Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., Pei, J. “How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks.” arXiv:2604.22750, April 2026.
Kim, Y. et al. “Towards a Science of Scaling Agent Systems.” Google Research / DeepMind / MIT. arXiv:2512.08296, December 2025.
Gartner, “Data Center Electricity Consumption to Grow 26% in 2026,” June 10, 2026.
Data Center Dynamics, “Energy efficient compute is most important attribute for customers, TSMC claims,” May 29, 2026.

About the author: Ajay Kumar Govindaram is a Solutions Architect who spent a decade in semiconductor process engineering and EDA before moving to cloud architecture. This cross-domain background informs a growing body of published work exploring structural parallels between chip design methodology and modern AI systems, including his prior piece on hierarchical agent orchestration through the semiconductor lens. He writes independently on topics at the intersection of hardware design thinking and software architecture.

The views expressed in this article are my own. All examples are from personal projects, publicly available research, and general industry knowledge. No proprietary or customer-specific information is disclosed.