Foundation models, transfer, and scaling evidence — Examples

Kaplan et al. measured held-out next-token cross-entropy for decoder-only autoregressive Transformer language models trained on WebText2. Here decoder-only means a stack of causal self-attention and positionwise feedforward blocks with residual connections and rowwise layer normalization, omitting the encoder and encoder–decoder cross-attention sublayers. With positional information and a vocabulary-probability output, it assigns a next-token conditional distribution given earlier tokens; its sequence likelihood factors into those next-token probabilities. Let $N$ be the paper’s non-embedding parameter count, $D$ the training dataset size in tokens, and $L$ the held-out test loss. The tested model sizes range from $768$ to $1.5\times {10}^9$ non-embedding parameters, and tested dataset sizes range from $22\times {10}^6$ to $23\times {10}^9$ tokens. WebText2 contains $20.3$ million documents and $2.29\times {10}^{10}$ tokens, of which $6.6\times {10}^8$ were reserved for testing.

The paper reports three separate empirical one-variable fits, each for its stated regime, rather than one joint formula. For parameter-limited models trained to convergence on sufficiently large data,

For large models trained on a limited dataset with early stopping,

For compute-efficient training with a sufficiently large dataset and an optimally sized model, the source uses a batch-adjusted non-embedding compute estimate. At a fixed target loss, let $B>0$ be the actual batch size in tokens, $S>0$ the actual number of training steps, and $B_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}>0$ the critical batch size in tokens. In the paper’s empirical batch-size model, set

Here $C$ estimates actual non-embedding training FLOPs, $S_{\min }$ estimates the minimum step count at large batch size, and $C_{\min }$ estimates the minimum compute at small batch size. These are model-based adjustments, not exact laws for arbitrary training runs; compute is converted to PF-days for the fitted curve below. Its fitted compute curve is

The compute fit spans eight orders of magnitude in $C_{\min }$. The reported compute-allocation fits are $N_{\mathrm{o}\mathrm{p}\mathrm{t}}\propto C_{\min }^{}$, $B_{\mathrm{c}\mathrm{r}\mathrm{i}\mathrm{t}}\propto C_{\min }^{}$, and $S_{\min }\propto C_{\min }^{}$; the last exponent is sufficiently small that the authors say it may be consistent with zero. The paper’s table labels the reported S-min relation as a lower bound. The paper also gives a separate model-derived compute-loss exponent of about $0.054$; it is not the empirical fit value $0.050$. The scale constants depend on vocabulary and tokenization.

For any doubling that stays within a fit’s measured range, the corresponding fitted curve predicts a loss ratio of $2^{-0.076}$, $2^{-0.095}$, or $2^{-0.050}$ in its respective regime. Applying any curve beyond its reported range, or interpreting a smooth loss fit as a prediction about an unmeasured capability, is extrapolation. The paper reports similar model-size trends on a finite set of other text datasets after WebText2 training, but this is not a universal transfer or reliability guarantee under distribution shift. These are empirical fits, not general theorems about emergent capability, downstream-task performance, other modalities, or universally optimal compute allocation.

Foundation models as broad-pretraining and adaptation systems: gives the report’s qualitative foundation-model usage and states that pretraining loss entails no general capability or transfer guarantee.

Transformer encoder, decoder, and fixed-width expressivity blocks: supplies the causal self-attention, positionwise feedforward, residual, and normalization components; its encoder–decoder layer additionally has cross-attention, which the decoder-only variant described here omits. This example imports no expressivity or universality theorem.

Kaplan et al., Scaling Laws for Neural Language Models, §§1.2, 2.1–2.3, 3.2–3.3, 5.1, 6.1–6.3, §8, and Appendix A: empirical one-variable fits, parameter and dataset ranges, batch-adjusted compute definition, allocation fits, and stated limits.

Let $\mathsf{X}=\{a,b\}$ and $\mathsf{Y}=\{0,1\}$, each with its full power-set sigma-algebra. Give the source and target examples the joint laws $P_S={\delta }_{(a,0)}$ and $P_T={\delta }_{(b,1)}$, so their input marginals are ${\mu }_S={\delta }_a$ and ${\mu }_T={\delta }_b$. Here ${\delta }_z$ denotes unit point mass at $z$. Supply the conditional label kernels $K_S(x,\cdot )={\delta }_0$ and $K_T(x,\cdot )={\delta }_1$ for each $x\in \mathsf{X}$. This is a finite train/evaluation distribution shift with disjoint input support; it is not covariate shift.

For a downstream training sample $\mathcal{D}=\{(x_i,y_i){\}}_{i=1}^n$ drawn from $P_S$, with $n\ge 1$, a simple parameter empirical-risk-minimization formulation chooses $\gamma$ from a nonempty parameter set $\mathrm{\Gamma }$ to minimize ${\widehat{R}}_{\mathcal{D}}(\gamma )=n^{-1}{\sum }_{i=1}^n{\mathrm{\ell }}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}(f_{\gamma }(x_i),y_i)$ when that infimum is attained; $f_{\gamma }:\mathsf{X}\to \mathsf{Y}$ is a task predictor and ${\mathrm{\ell }}_{\mathrm{t}\mathrm{a}\mathrm{s}\mathrm{k}}:\mathsf{Y}\times \mathsf{Y}\to [0,\mathrm{\infty })$ is a task loss. Fine-tuning is a parameter-based adaptation that updates some or all pretrained parameters using task information. In-context learning instead adds task demonstrations and a test input to the prompt while keeping the pretrained parameters fixed. It is an adaptation route by input conditioning, not empirical-risk minimization over changed model parameters.

The report’s 2021 theory discussion asks when a small population pretraining loss implies a small minimal adaptation loss and identifies that interface as a theory gap involving the two data distributions, objectives, training methods, and architecture. A pretraining cross-entropy scaling fit does not by itself give a downstream task-loss or reliability guarantee for this example. This is a conceptual interface example and makes no empirical comparison of prompting and fine-tuning.

The diagram shows the support points of the two laws with their unit masses. It is only a support illustration; the verification derives the shift condition directly from the singleton set $\{b\}$.

diagram failed to render:
\begin{tikzpicture}[>=stealth,scale=0.9,transform shape,every node/.style={draw,rounded corners,align=center,font=\small,minimum width=1.7cm,minimum height=0.7cm}]
  \node (source) at (0,1) {$P_S$};
  \node (sourcepoint) at (3,1) {$(a,0)$};
  \node (target) at (0,-1) {$P_T$};
  \node (targetpoint) at (3,-1) {$(b,1)$};
  \draw[->] (source) -- node[above,draw=none] {mass $1$} (sourcepoint);
  \draw[->] (target) -- node[below,draw=none] {mass $1$} (targetpoint);
\end{tikzpicture}

Foundation models as broad-pretraining and adaptation systems: distinguishes prompt conditioning from parameter-based adaptation and records that the pretraining-loss-to-adaptation-loss relationship is not a general guarantee.

Source and target joint laws, risks, and covariate shift: defines source and target joint laws, input marginals, supplied conditional label kernels, and the additional domination and kernel-equality hypotheses for covariate shift.

Bommasani et al., On the Opportunities and Risks of Foundation Models, §§4.3 and 4.10.1–4.10.3: adaptation modes, the optimization-based interface, in-context learning without parameter optimization, and the report’s theory question.

Stanford CS324, Large Language Models, “Adaptation” lecture notes: probing, fine-tuning, prompting, and downstream task adaptation distinctions.