Saram ConsultingEvery model release collapses into a single question: “How many parameters?” It is the clock-speed debate of the 2020s — a proxy metric that was once useful, then misleading, and is now actively dangerous to the decision-making of anyone building production AI systems.
A model with 70 billion parameters trained on 2 trillion tokens is a fundamentally different artifact from a model with 70 billion parameters trained on 15 trillion tokens. They occupy different points on different scaling curves. They will behave differently on reasoning tasks, memorization tasks, long-context tasks, and tasks that require rare knowledge. Collapsing both into “70B” discards almost everything that matters.
Jie Tang — Tsinghua professor and co-founder of Zhipu AI — recently published a post that compresses five years of scaling law evolution into a precise, practitioner-oriented argument: parameters are not the metric. The allocation is the metric. And the allocation has more than one dial.
His post is both a historical corrective and a product manifesto. GLM-5.3, released August 14, 2026, is the controlled experiment behind the argument. Same base model as GLM-5.2. Same architecture. Same total parameters. Same activated parameters. One month of post-training. The gains are not marginal.
Here is the full story — what the field got wrong, what it corrected, and where the frontier actually sits now.
The Kaplan Detour: When the Exponent Was Wrong
In 2020, Kaplan et al. at OpenAI published the first systematic scaling laws for transformer language models. They fit power laws to loss as a function of parameters ($N$), dataset size ($D$), and compute ($C$). The exponents were unequal — parameters had a steeper exponent than data — and the practical recommendation was clear: when you get more compute, spend it mostly on making the model bigger.
The industry followed this recipe faithfully.
GPT-3: 175 billion parameters, ~300 billion training tokens. That is roughly 1.7 tokens per parameter. Gopher: 280 billion parameters, ~300 billion tokens. Megatron-Turing NLG: 530 billion parameters, ~271 billion tokens — roughly 0.5 tokens per parameter.
These models had the capacity to know more, but they had not been shown enough data to actually learn it. They were massive, sprawling warehouses with only a few boxes on the shelves.
The error in Kaplan’s fit was subtle: the exponents were locally valid but extrapolated poorly. Small errors in the exponents compound dramatically when you push across orders of magnitude of compute. Every model that followed Kaplan’s ratio more faithfully was more wrong, not less, because the error compounds with scale rather than washing out.
The trillion-parameter round that followed was, as Tang puts it, “a detour the whole field took together and then reversed.”
Chinchilla: The Correction That Changed Everything
In 2022, Hoffmann et al. at DeepMind re-ran the scaling experiment with a key methodological difference: over 400 models spanning 70M to 16B+ parameters, trained on 5B to 500B tokens. They used three independent estimation approaches — fixed compute budgets varying $N$ and $D$, isoFLOP curves, and parametric loss fitting. All three converged on a dramatically different answer.
The compute-optimal ratio was not parameter-heavy. It was roughly balanced: ~20 tokens per parameter, with parameters and data scaling at close to a 1:1 ratio as compute increases.
The parametric form they fit:
$$L(N,D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$
The result was spectacular. Chinchilla — 70 billion parameters trained on 1.4 trillion tokens — outperformed Gopher (280 billion parameters, 300 billion tokens) on nearly every benchmark, despite being 4x smaller and requiring 4x less inference compute. It was simply trained correctly.
GPT-3, at 1.7 tokens per parameter, was wildly undertrained. A Chinchilla-optimal model with the same compute budget would have been far smaller and far better. The trillion-parameter race was a multi-million-dollar correction the entire industry had to absorb.
But Chinchilla had a blind spot. It optimized for a world where the dominant cost is training — where a model is trained once, evaluated on benchmarks, and put in a showcase. That world no longer exists.
The Inference Pivot: Why 20:1 Is Not the Answer Either
Today, a frontier model is called billions of times a day. The total cost of a model’s life is:
$$C_{total} = C_{train} + (C_{inference} \times N_{queries})$$
When $N_{queries}$ is enormous — which it is for any widely deployed model — inference cost dominates the ledger. A model that costs $2 million to train but $50 million per year to serve has a very different optimization landscape than one that costs $10 million to train and $5 million per year to serve.
Put inference into the objective and the math changes fundamentally. The optimum shifts toward smaller models trained far longer — deliberate over-training. It is cheaper to pay extra once in training to save on every query forever.
The empirical evidence is now overwhelming:
| Model | Parameters | Training Tokens | Tokens per Parameter |
|---|---|---|---|
| GPT-3 | 175B | 300B | ~1.7 |
| Chinchilla | 70B | 1.4T | ~20 |
| Llama-2-7B | 7B | 2T | ~290 |
| Gemma-2-9B | 9B | 8T | ~889 |
| Llama 3 8B | 8B | 15T | ~1,875 |
The field moved from 1.7 tokens per parameter to nearly 1,875 in four years — a thousandfold increase. Each step represented a shift in what was being optimized: from “minimize training loss at fixed compute” to “minimize total cost-of-ownership at fixed capability.”
Researchers training 47 models from 150M to 6B parameters found no evidence of a saturation point even at 10,000 tokens per parameter. Models keep getting better as you feed them more data, even far past the Chinchilla-optimal point. The “over” in “over-training” is a misnomer — it is only over-training if you ignore inference.
Industry shorthand: Chinchilla-optimal is ~20:1. Inference-optimal is 100:1 to 1,000:1. And practitioners increasingly pick model size based on what fits on the deployment device, not on a theoretical optimum.
MoE Breaks the Single-Number Illusion
Just as the industry settled into the inference-optimal regime, Mixture-of-Experts architectures broke the math entirely. In a dense model, “parameter count” is one number. In an MoE model, it is at least two:
Total parameters ($N_{total}$): The full set of weights across all experts. This governs how much the model can hold — knowledge, facts, the long-tail of world information. Think of it as the Library.
Activated parameters ($N_{active}$): The subset routed to for any single token. This governs how much computation happens per forward pass — how many steps of a causal chain the model can carry before the thread breaks. Think of it as the Workbench.
A model like GLM-5 has ~744 billion total parameters but only ~40 billion activated per token. You pay for 40B. You store 744B. The inference cost is that of a 40B dense model; the knowledge capacity is that of something far larger.
This separation has a profound consequence: the old 20:1 dense ratio does not transfer. There is no single “tokens per parameter” number that means the same thing in an MoE as in a dense model, because “parameters” now means two different things.
The Task-Dependent Scaling Law
Roberts et al. (2025) made this concrete. They showed that the optimal tokens-per-parameter ratio is skill-dependent:
- Memorization tasks (factual recall, knowledge-intensive QA) benefit from more parameters — lower tokens per parameter. You need the parameter space to store the knowledge.
- Reasoning tasks (mathematical proof, code generation, multi-step inference) benefit from more data — higher tokens per parameter. Reasoning patterns are more data-hungry than facts.
A misspecified validation set whose skill mix does not match the target can shift the apparent compute-optimal parameter count by nearly 50%. There is no universal optimum. The optimum depends on what you want the model to do.
The Counterintuitive MoE Result
Perhaps the most striking finding: at fixed tokens-per-parameter, pushing total parameters higher (by adding more experts) can actually degrade reasoning performance, while activating more experts reliably improves it.
Why? More experts means more storage — better knowledge. More activated experts means more compute per token — better reasoning. If you add experts but keep the number activated fixed, you expand storage without expanding reasoning capacity. The added routing complexity may actually hurt.
This means “bigger MoE” is not automatically “smarter MoE.” Unless the extra capacity is spent on activation, not storage.
Finding a Vulnerability Is Not a Retrieval Problem
Tang uses a specific example to make the argument concrete, and it cuts to the heart of why parameter count is the wrong metric for the tasks that matter most.
If you ask a model to find a known CVE, that is retrieval. You need a model with a massive Library (total parameters) that has memorized the database of known flaws. Scale total parameters and you improve at this task.
But finding a novel zero-day vulnerability — a complex logic flaw buried in a codebase — is a long-horizon reasoning problem. It requires:
- Understanding the system’s architecture
- Identifying a potential attack surface
- Tracing untrusted data flows across multiple files
- Constructing a multi-step exploit chain
- Verifying that each step follows logically from the previous one
- Carrying the chain to completion without losing coherence
Steps 3 through 6 are reasoning. They require depth of computation per token, not breadth of stored knowledge. A model that has memorized every CVE from 2015 to 2024 still cannot find a new one if it cannot carry a twenty-step chain of inference to the end.
This is why Retrieval-Augmented Generation fails at high-level cybersecurity. Giving an LLM a database of CVEs does not help it find a new one. The capability lives in effective depth and reasoning — exactly the dimensions that post-training on long-horizon environments is designed to move.
GLM-5.3: The Controlled Experiment
This is where the post becomes a live case study rather than a literature review.
GLM-5.3 uses the exact same base model as GLM-5.2. Same architecture. Same ~744B total parameters. Same ~40B activated per token. Same 1M token context window. Every improvement comes from one month of scaling post-training: more diverse long-horizon environments, more reinforcement learning, continued use of their SAO (Scalable Agentic Optimization) and slime asynchronous RL infrastructure.
The technical stack:
- IndexShare: Long-context technique for coherent reasoning across sprawling codebases
- SAO: RL method for long-horizon tasks where rewards span hundreds of steps
- Slime: Open-source asynchronous RL infrastructure for parallel environment rollouts
- Environment generation: Automated pipelines where research agents synthesize environments from real work patterns, judge agents verify solvability, and verifiers are synthesized without reference solutions to prevent reward hacking
The results:
| Benchmark | GLM-5.2 | GLM-5.3 | Change |
|---|---|---|---|
| Terminal-Bench 3.0 | 4.6 | 28.3 | +515% |
| DeepSWE v1.1 | 46.2 | 66.9 | +45% |
| Agents’ Last Exam (CLI) | 23.8 | 28.5 | +20% |
| CyberGym | 77.2% | 84.5% | +7.3 pts |
| ExploitBench | 24.4% | 54.4% | +123% |
The cybersecurity gains were emergent and unplanned. Z.ai added vulnerability-discovery environments expecting incremental bug-finding improvements. Instead, the model developed multi-step exploit chaining — a qualitative capability jump that appeared only at scale. The team flagged it as a surprise: capability kept compounding as training scaled, with no signs of saturation.
On the efficiency side, GLM-5.3 closes a task using roughly 50,000 output tokens versus approximately 120,000 for Claude Opus 4.8 on comparable work. Shorter trajectories mean lower per-task cost for anyone paying by the token on agentic workloads.
The emergent cyber capabilities were significant enough that Z.ai delayed open-weight release by approximately two weeks for safety review — one of the first instances of a Chinese lab citing emergent capability risk as a release consideration.
An honest caveat: The benchmark gains are vendor-reported, and weights were not yet released for independent verification at the time of the announcement. The magnitude of the Terminal-Bench jump (4.6 to 28.3) is large enough that even generous discounting for self-reporting bias leaves a real effect. But “same base, same architecture, one month of post-training, non-marginal gains” is Z.ai’s controlled-experiment framing, not yet a community-replicated result.
The Multi-Dial Framework
Tang’s central argument is that scaling has multiple independent dials:
| Dial | What It Buys | Current State |
|---|---|---|
| Base model size | Knowledge capacity | Heavily optimized |
| Pretraining data | Statistical learning | Heavily optimized |
| Over-training ratio | Inference cost reduction | Widely adopted |
| Active compute / depth | Per-token reasoning | MoE-dependent |
| Post-training (RL + environments) | Behavioral competence | Most remaining slack |
| Test-time compute | Additional reasoning at inference | Rapidly evolving |
| Mid-training | Domain specialization | Next frontier |
The key insight: these dials do not have to be turned together. You can hold four constant and turn the fifth, measure the result, and then decide which dial to turn next.
“The dials do not have to be turned together, and the one worth turning next is rarely the one that was worth turning last.”
This is essentially gradient descent applied to research resource allocation: identify the dimension with the steepest current gradient, invest there until the gradient flattens, then move to the next dimension.
The field has cycled through several single-dial obsessions:
- 2020–2021: Parameters (blind spot: data starvation)
- 2022–2023: Data / Chinchilla (blind spot: inference cost)
- 2023–2024: Over-training (blind spot: post-training / RL)
- 2024–2025: MoE total parameters (blind spot: active compute / task-dependence)
- 2025–present: Post-training environments (blind spot: ???)
Each dial has diminishing returns when turned in isolation, but the next dial often has low-hanging fruit. GLM-5.3 is the empirical demonstration that post-training — specifically long-horizon RL on rich environments — can yield frontier-level gains without touching pre-training at all.
Why This Matters If You Are Building Agents
The implications extend far beyond model benchmarking. If you are building production AI agents — coding agents, CSV agents, security agents — the scaling law evolution changes how you should think about your entire system.
The Unit of Data Is Changing
For LLM pre-training, the unit is a token. For reasoning, it is a problem. For agentic RL, it is a trajectory — a sequence of actions, observations, and outcomes. For real-world agents, it is an episode — a complete task from goal to completion.
Each step up this hierarchy contains more information about what to do, not just what is true. And each requires different scaling strategies.
Trajectory Length Is the New Bottleneck
An agent with 99% per-step accuracy has a 36.6% chance of completing a 100-step task correctly. The compounding error over long horizons is brutal. This means that for agentic workloads, the relevant scaling question is not “how smart is the model on a single step?” but “how reliably can it maintain coherence over a full trajectory?”
That is a depth problem, not a breadth problem. It lives in activated parameters, effective depth, and post-training — not in total parameter count.
Environments Are Becoming Training Assets
Historically, data was the scarce resource. For agentic systems, high-quality environments may be the scarce resource. For coding: compilers, tests, CI/CD pipelines. For security: vulnerable machines, exploit validators. For regulated industries: validated systems, QMS records, SOP repositories, test evidence, deviation histories.
The company with the best proprietary environments may have an AI training advantage even without the biggest model. That is a fundamentally different competitive moat than parameter count.
Total vs. Active Maps onto Agent Architecture
The MoE distinction between total and activated parameters has a direct analogue in multi-agent system design. Your retrieval layer (total knowledge capacity — BM25, dense search, the full corpus) plays the role of total parameters. Your subagent’s reasoning chain (how many steps of causal logic it can carry without losing the thread) plays the role of activated parameters.
A subagent overloaded with too much retrieved context tends to lose the causal thread the same way a memorization-heavy MoE config degrades on reasoning benchmarks. Same underlying tension — capacity to hold vs. capacity to chain — showing up at the system architecture layer instead of the model layer.
The Bottom Line
The question everyone asks at model release — “how many parameters?” — is the least informative question you can ask. The real question is: which dial had the most slack left, and did the developer turn it far enough?
Five years ago, there was one dial: parameter count. Today, the lab with the best intuition about which dial to turn next wins. GLM-5.3 is proof that you can get frontier-level capability jumps without touching the base model at all. The slack has moved — from parameters to data to inference cost to post-training — and it will move again.
The era of “just make it bigger” is over. The era of surgical, multi-dimensional scaling is here. And the scarce resource is gradually shifting from parameters to compute to experience.
For anyone building AI systems today — especially agents that need to reason over long horizons in specialized domains — the implication is clear. Stop counting parameters. Start measuring where the compute goes.
Research notes: [[scaling-laws-multi-dial-era-2026]]
Source: Jie Tang, “Thoughts About Scaling Law” (August 2026). Tang is co-founder of Zhipu AI (Z.ai) and professor at Tsinghua University.