An observed-range language-model scaling fit

Kaplan et al. measured held-out next-token cross-entropy for decoder-only autoregressive Transformer language models trained on WebText2. Here decoder-only means a stack of causal self-attention and positionwise feedforward blocks with residual connections and rowwise layer normalization, omitting the encoder and encoder–decoder cross-attention sublayers. With positional information and a vocabulary-probability output, it assigns a next-token conditional distribution given earlier tokens; its sequence likelihood factors into those next-token probabilities. Let $N$ be the paper’s non-embedding parameter count, $D$ the training dataset size in tokens, and $L$ the held-out test loss. The tested model sizes range from $768$ to $1.5\times {10}^9$ non-embedding parameters, and tested dataset sizes range from $22\times {10}^6$ to $23\times {10}^9$ tokens. WebText2 contains $20.3$ million documents and $2.29\times {10}^{10}$ tokens, of which $6.6\times {10}^8$ were reserved for testing.

The paper reports three separate empirical one-variable fits, each for its stated regime, rather than one joint formula. For parameter-limited models trained to convergence on sufficiently large data,

$$L(N)\approx {\left(\frac{N_c}{N}\right)}^{{\alpha }_N},{\alpha }_N\approx 0.076,N_c\approx 8.8\times {10}^{13}\text{ non-embedding parameters}\mathrm{.}$$

For large models trained on a limited dataset with early stopping,

$$L(D)\approx {\left(\frac{D_c}{D}\right)}^{{\alpha }_D},{\alpha }_D\approx 0.095,D_c\approx 5.4\times {10}^{13}\text{ tokens}\mathrm{.}$$

For compute-efficient training with a sufficiently large dataset and an optimally sized model, the source uses a batch-adjusted non-embedding compute estimate. At a fixed target loss, let $B>0$ be the actual batch size in tokens, $S>0$ the actual number of training steps, and $B_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}>0$ the critical batch size in tokens. In the paper’s empirical batch-size model, set

$$C=6NBS,S_{\min }=\frac{S}{1+B_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}\mathrm{/}B},C_{\min }=\frac{C}{1+B\mathrm{/}{B}_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}}=6N{B}_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}{S}_{\min }\mathrm{.}$$

Here $C$ estimates actual non-embedding training FLOPs, $S_{\min }$ estimates the minimum step count at large batch size, and $C_{\min }$ estimates the minimum compute at small batch size. These are model-based adjustments, not exact laws for arbitrary training runs; compute is converted to PF-days for the fitted curve below. Its fitted compute curve is

$$L(C_{\min })\approx {\left(\frac{C_c^{\min }}{C_{\min }}\right)}^{{\alpha }_{C_{\min }}},{\alpha }_{C_{\min }}\approx 0.050,C_c^{\min }\approx 3.1\times {10}^8\text{ PF-days}\mathrm{.}$$

The compute fit spans eight orders of magnitude in $C_{\min }$. The reported compute-allocation fits are $N_{\mathrm{o}\mathrm{p}\mathrm{t}}\propto C_{\min }^{}$, $B_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}\propto C_{\min }^{}$, and $S_{\min }\propto C_{\min }^{}$; the last exponent is sufficiently small that the authors say it may be consistent with zero. The paper’s table labels the reported S-min relation as a lower bound. The paper also gives a separate model-derived compute-loss exponent of about $0.054$; it is not the empirical fit value $0.050$. The scale constants depend on vocabulary and tokenization.

For any doubling that stays within a fit’s measured range, the corresponding fitted curve predicts a loss ratio of $2^{-0.076}$, $2^{-0.095}$, or $2^{-0.050}$ in its respective regime. Applying any curve beyond its reported range, or interpreting a smooth loss fit as a prediction about an unmeasured capability, is extrapolation. The paper reports similar model-size trends on a finite set of other text datasets after WebText2 training, but this is not a universal transfer or reliability guarantee under distribution shift. These are empirical fits, not general theorems about emergent capability, downstream-task performance, other modalities, or universally optimal compute allocation.

Foundation models as broad-pretraining and adaptation systems: gives the report’s qualitative foundation-model usage and states that pretraining loss entails no general capability or transfer guarantee.

Transformer encoder, decoder, and fixed-width expressivity blocks: supplies the causal self-attention, positionwise feedforward, residual, and normalization components; its encoder–decoder layer additionally has cross-attention, which the decoder-only variant described here omits. This example imports no expressivity or universality theorem.

Kaplan et al., Scaling Laws for Neural Language Models, §§1.2, 2.1–2.3, 3.2–3.3, 5.1, 6.1–6.3, §8, and Appendix A: empirical one-variable fits, parameter and dataset ranges, batch-adjusted compute definition, allocation fits, and stated limits.