The standard assumption about maturing technology is that hardware demand flattens. Software gets more efficient, silicon gets denser, and the same work needs fewer parts. That has been broadly true for compute. It has not been true for memory, and the reasons are structural rather than cyclical.
The short version: as AI moves from training to inference, the binding constraint moves from arithmetic to data movement. Memory sits on the wrong side of that shift, which is to say the profitable side.
Inference Is a Bandwidth Problem
Generating a token requires streaming the active model weights out of memory and into the accelerator. The arithmetic involved is trivial by modern standards. The data movement is the entire cost. A GPU waiting on memory is a GPU doing nothing, which is why accelerator vendors keep adding stacks rather than adding cores.
Training is episodic and capex-lumpy. It happens in bursts, tied to model release schedules and cluster build-outs. Inference is continuous and scales with adoption. TrendForce expects inference to overtake training as the primary driver of AI server demand before the end of the decade, and every forecast that moves in that direction moves demand toward bandwidth and away from raw compute.
The State Scales With Users, Not With Models
This is the part most demand models understate. Every active session holds a key-value cache, and that cache grows linearly with context length and with the number of concurrent users. It has nothing to do with how large the model is.
Three things are pushing it hard right now. Context windows keep expanding. Reasoning models generate an order of magnitude more tokens per query than single-shot answers did, and every one of those tokens extends the cache before the user sees a word. Agentic workloads hold sessions open for minutes or hours instead of seconds.
So even in a world where model architectures froze tomorrow, per-user memory consumption would keep climbing on adoption alone. Industry estimates put real-time memory demand across the major inference platforms at roughly 750 petabytes, and closer to 1.5 exabytes once you count the redundancy and headroom any production deployment actually requires.
Efficiency Moves Demand, It Does Not Remove It
Each efficiency technique that gets cited as a threat to memory demand turns out, on inspection, to relocate it.
Mixture-of-experts architectures explicitly trade compute for capacity. You hold every expert resident in memory and activate a small fraction per token. Sparse models are more memory-hungry per unit of compute, not less. Quantization cuts the cost per token, which raises token volume, which is the oldest pattern in the industry. Distillation produces smaller models that get deployed in far more places.
The classic Jevons shape applies. Cheaper inference means more inference.
The Wafer Math Amplifies Everything
Bit shipments understate what AI is doing to supply, because high-speed memory is far more expensive to manufacture per bit. A gigabyte of HBM consumes roughly four times the fab capacity of standard DRAM once you account for die area, through-silicon vias, and stacking yield. GDDR7 runs around 1.7 times.
Total DRAM wafer starts are growing something like six to eight percent annually. Every wafer allocated to HBM is a wafer not producing conventional DDR5. That crowding-out effect is why commodity DRAM pricing has moved with the AI cycle despite having almost nothing to do with AI workloads, and why memory has started behaving like an allocation market rather than a spot commodity market.
Where the Thesis Can Still Cost You
Structural demand and a non-cyclical stock are different objects, and conflating them is how people get hurt in this sector.
The supply response is late, not absent. New fabs take two to three years from commitment to meaningful output. The historical failure mode has never been that demand disappeared. It is that peak capacity arrived into a demand pause. Korean cluster announcements and Micron’s Japanese expansion will not add bits before 2027 or 2028, which supports the near term and complicates the one after it.
CXMT is currently more useful as a negotiating lever for buyers than as a genuine supply threat. It lags badly in HBM, it lacks EUV, and its mobile mix skews to older LPDDR generations. That could change slowly. It is unlikely to change suddenly.
The real technical risk to the thesis is architectural. Attention variants that carry constant-size state, or aggressive key-value compression, would cut per-token memory intensity in a way no process node ever could. Low probability on a two-year view. Not zero on a five-year one.
For anyone watching the cycle rather than the secular story, the 2027 HBM4 contract negotiations running through the second half of this year are the first real price discovery since suppliers sold out. Firm pricing validates the extended-supercycle case. Any concession pattern shows up two to three quarters before it appears in reported results.
The level rises structurally. The cycle still exists. Those are separate trades.
Leave a Reply