Fixed-width Transformer approximation on compact domains

First consider the compactly supported equivariant map. If $f=0$, the zero Transformer suffices. Otherwise its support $K$ is a nonempty compact permutation-invariant set. Compactness bounds all scalar entries of matrices in $K$; choose real $a<b$ with every entry in $[a,b]$. Since a nonzero continuous function is nonzero on an open set, $K$ has positive measure and the bounds can be chosen with $a<b$. The common affine map $T(X)=(X-a\mathbf{1})\mathrm{/}(b-a)$ sends $K$ into $[0,1{]}^{d\times n}$ and commutes with column permutations. A residual width-four ReLU sublayer implements $z\mapsto \alpha z+\beta$ on every feature by $z+(\alpha -1)(\mathrm{ReLU}(z)-\mathrm{ReLU}(-z))+\beta =\alpha z+\beta ;$ two hidden units handle one feature and finitely many layers handle all $d$ features. Thus $T$ can be inserted in the Yun architecture.

Now consider continuous $f$ on compact $K$ without symmetry. If $K$ has zero Lebesgue measure, the integral is zero for any network. Otherwise let $a,b$ be the minimum and maximum of its scalar entries; again $a<b$. Put $\alpha =(b-a{)}^{-1}$ and choose $E_{rj}=(b-a)(j-1)-a(1\le r\le d,1\le j\le n)\mathrm{.}$ The first residual width-four sublayers scale $X+E$ by $\alpha$, producing entries $(X_{rj}-a)\mathrm{/}(b-a)+(j-1)$. For any mesh $\delta =1\mathrm{/}m$ used below, put ${\eta }_{\delta }(t)=-t$ for $0\le t<\delta$ and ${\eta }_{\delta }(t)=0$ otherwise. For each feature row, apply the residual updates $t\mapsto t+{\eta }_{\delta }(t-k)$ in increasing order for $k=0,\delta ,\dots ,n-\delta$. The $k$th update sends exactly $[k,k+\delta )$ to $k$ and leaves values outside that interval unchanged; hence every column-$j$ value in $[j-1,j)$ is mapped to one of $j-1,j-1+\delta ,\dots ,j-\delta$. Entries exactly at an upper endpoint lie on finitely many coordinate hyperplanes of measure zero and do not affect the integral. Thus the quantized matrices lie in $G_{\delta ,\mathrm{p}\mathrm{o}\mathrm{s}}$ from [F3], where position offsets make every column distinct. For each grid cell meeting $K$, choose a representative $X_L\in K$ and set $A(L)=f(X_L)$; set arbitrary values, for example zero, on cells disjoint from $K$. The position-tagged assertion of [F3] realizes this finite table exactly.

It remains to replace the modified gates and uniform hardmax in [F3] by the standard ReLU and softmax of ${\mathcal{T}}^{2,1,4}$. Consider any scalar gate $\varphi$ that is affine on the three intervals cut by $c_1<c_2$, with formulas $a_{1}t+b_1$, $a_{2}t+b_2$, and $a_{3}t+b_3$, and with at least one $a_i=0$. Fix $0<\rho <(c_2-c_1)\mathrm{/}2$. Replace the parts on $[c_1-\rho ,c_1]$ and $[c_2-\rho ,c_2]$ by linear ramps joining the left, middle, and right affine pieces, and leave $\varphi$ unchanged elsewhere. The resulting continuous piecewise-affine map has knots $r_1=c_1-\rho$, $r_2=c_1$, $r_3=c_2-\rho$, $r_4=c_2$ and successive slopes $s_0=a_1$, $s_1$, $s_2=a_2$, $s_3$, $s_4=a_3$. Put ${\mathrm{\Delta }}_i=s_i-s_{i-1}$. It is a constant plus four ReLUs: use ${\mathrm{\Delta }}_i\mathrm{ReLU}(t-r_i)$ at knots not in $S$ and ${\mathrm{\Delta }}_i\mathrm{ReLU}(r_i-t)$ at knots in $S$, choosing any applicable case $S=\mathrm{\varnothing }$ if $a_1=0$, $S=\{1,2\}$ if $a_2=0$, or $S=\{1,2,3,4\}$ if $a_3=0$; add the constant that matches the left affine piece. In each case the left-tail slope is $-{\sum }_{i\in S}{\mathrm{\Delta }}_i=a_1$ and the right-tail slope is ${\sum }_{i\mathrm{\notin }S}{\mathrm{\Delta }}_i=a_3$, while the slope jump at each knot is ${\mathrm{\Delta }}_i$, so this formula is exact outside the two ramp intervals. It covers the middle-constant sentinel gate as well as the constant-ended pulse and clamp gates used below. The two-piece clipping gate $\varphi (t)=-t$ for $t<0$ and $0$ for $t\ge 0$ is exactly $\mathrm{ReLU}(-t)$. A width-four ReLU sublayer realizes each ramped scalar gate, including its constant bias, in the modified residual update. On bounded input regions the ramp error is confined to strips whose widths tend to zero with $\rho$. For any finite score vector $s$, softmax of $\lambda s$ converges as $\lambda \to \mathrm{\infty }$ to equal weights on its maximizing coordinates and zero elsewhere; hence it converges to the uniform hardmax used in [F3].

Choose $\delta =1\mathrm{/}m$ with integer $m$ large enough that $n\le {\delta }^{-d}$, the duplicate-cell bound below is small, and $M={\delta }^{-nd}>d+1+\log (n)\mathrm{/}\lambda$ for a fixed $\lambda >0$. In the modified unpositioned construction, first apply to each feature row the residual gate $t\mapsto t+{\chi }_M(t)$, where ${\chi }_M(t)=-t-M$ for $t<0$, ${\chi }_M(t)=0$ for $0\le t<1$, and ${\chi }_M(t)=-t-M$ for $t\ge 1$. This sends every out-of-range coordinate to the sentinel $-M$. Then, in increasing order for $k=0,1,\dots ,m-1$, apply $t\mapsto t+{\eta }_{\delta }(t-k\delta )$, with ${\eta }_{\delta }(s)=-s$ for $0\le s<\delta$ and $0$ otherwise. The $k$th clamp sends precisely $[k\delta ,(k+1)\delta )$ to $k\delta$ and leaves values below or above that interval unchanged; after all gates each coordinate is a grid value or the sentinel. These are finite compositions of the three-piece gates in step 1.3, so the exact modified quantizer maps into $G_{\delta }^{+}$. For the unpositioned branch, partition the normalized cube into half-open grid cells of side $\delta$. Their preimages under $T$ have side $h=(b-a)\delta$. By [F4], the oscillation of $f$ between two points of $K$ at distance at most $\sqrt{dn}h$ tends to zero with $\delta$. For each orbit of distinct-column grid matrices $L$ under column permutations, if the closed preimage cell meets $K$, choose one representative $X_L$ in that intersection and put $A(L)=f(X_L)$; transport the choice by permutations and set $A(L)=0$ on orbits whose cells miss $K$. This is a finite choice and gives $A(LP)=A(L)P$. Apply the first assertion of [F3] to this table.

By [F4], for sufficiently small $\delta$ every point of $K$ and the representative in its grid cell have outputs within any prescribed tolerance. The cell diameter in the original coordinates is at most $\sqrt{dn}(b-a)\delta$, and $K$ has finite measure because it is compact and bounded. Thus the integrated $L_p(K)$ error of the modified position-tagged lookup can be made arbitrarily small.

The table network agrees with $f$ on every point of $K$ in a distinct-column cell up to the oscillation in step 2.1. The union of cells whose grid matrices have duplicate columns has measure at most $\left(\genfrac{}{}{0px}{}{n}{2}\right)m^{d(n-1)}{h}^{dn}=\left(\genfrac{}{}{0px}{}{n}{2}\right)(b-a{)}^{dn}{\delta }^d,$ because for each pair of columns, equality leaves only $m^{d(n-1)}$ grid matrices. The table is zero on cells not meeting $K$; cells meeting $K$ but lying partly outside it are contained in the $\sqrt{dn}h$-neighborhood of $K$. Intersect these neighborhoods with one fixed bounded box containing $K$. Along any sequence $h_j\downarrow 0$, the resulting closed neighborhoods decrease to $K$ and have finite measure; [F5]–[F6] imply that their excess measure over $K$ tends to zero. The table values are bounded because $f$ is continuous on compact $K$. Outside the normalized cube the sentinel branch of [F3] returns zero, as does $f$. Therefore the integrated $L_p$ error of the modified lookup tends to zero as $\delta \downarrow 0$.

To control the unbounded input domain in the first assertion, insert an exact sentinel gate before simulating the lookup. For a quantized matrix $Z$, put $s_j={\sum }_{r=1}^d{Z}_{rj}$ and $w_j=\frac{e^{-\lambda s_j}}{{\sum }_{k=1}^n{e}^{-\lambda s_k}},\bar{s}={\sum }_{j=1}^n{w}_j{s}_j\mathrm{.}$ A size-one softmax head with constant query $1$, key $-\lambda s_j$, value $s_j$, and output vector ${\mathbf{1}}_d$ sends $Z$ to $Z+\bar{s}{\mathbf{1}}_{d\times n}$. If all entries of $Z$ are in $[0,1]$, then every $s_j\ge 0$, so every shifted entry is nonnegative. If any entry is the sentinel $-M$, then ${\min }_j{s}_j\le -M+d-1$. The entropy identity $-{\sum }_j{w}_j\log w_j=\lambda \bar{s}+\log {\sum }_{k=1}^n{e}^{-\lambda s_k}$ and the bounds $-{\sum }_j{w}_j\log w_j\le \log n$ and ${\sum }_j{e}^{-\lambda s_j}\ge e^{-\lambda {\min }_j{s}_j}$ imply $\bar{s}\le {\min }_j{s}_j+(\log n)\mathrm{/}\lambda <-1$, by the choice in step 2.1. Since every entry of $Z$ is at most $1$, every shifted entry is then negative. Apply coordinatewise ReLU, implemented by the residual update $z_r\mapsto z_r+\mathrm{ReLU}(-z_r)=\mathrm{ReLU}(z_r)$ one feature at a time. It leaves every shifted valid state unchanged and sends every shifted sentinel state to zero. A second size-one softmax head exactly inverts the shift on nonnegative states: the shifted column sums are $s_j+d\bar{s}$, so their weights remain $w_j$ and their weighted mean is $(d+1)\bar{s}$; add $-(d+1{)}^{-1}$ times this mean to each feature. This restores $Z$, and sends the zero state to zero. These two attention updates and the coordinatewise clipping are ordinary blocks in ${\mathcal{T}}^{2,1,4}$.

Fix the mesh, finite table, and modified lookup from steps 1.2 and 2.1. Choose the lookup ramp widths so small that their ramp intervals avoid every scalar gate input on the finitely many exact lookup trajectories. The detector boundaries and pulse boundaries have positive clearance from these inputs, and clipping is the exact map $z_r\mapsto z_r+\mathrm{ReLU}(-z_r)$. Replace each hardmax head by scaled softmax, scaling only its scores. In the contextual sweeps a query stays away from its half-grid thresholds; maximizing keys have the same scalar code, which is also their scalar value. Thus small perturbations of tied maximizing keys cannot change the limiting value. In the final head, the only zero-query state is the all-zero matrix, where all values vanish. Positive score gaps to nonmaximizing keys suppress their weights exponentially. These observations give convergence of each head output near each exact trajectory state; the fixed ramped tokenwise maps are continuous. Induction through the finite composition therefore approximates the lookup arbitrarily closely on all required finite grid states.

In the unpositioned branch, this simulation can and will fix the zero matrix exactly. Every contextual attention update sends zero to zero, for every softmax scale, since its value projection vanishes there. In the lookup construction of [F3], the invalid-code detector equals one at scalar code zero, since the interval of valid codes is strictly positive. Its ramped version also equals one there when its width is sufficiently small. Subtracting the detector's fixed positive constant makes every coordinate negative, and the exact coordinatewise clipping then returns the zero matrix. Every subsequent lookup pulse is zero at scalar code zero, as its positive center is at least the least valid code. Small enough pulse ramp widths preserve this equality. Thus the entire simulated lookup sends zero exactly to zero, including its detector layers; this is stronger than merely approximating zero. The exact filter in step 3.2 followed by this lookup consequently gives zero whenever the quantizer has an exact sentinel coordinate.

Use a separate width $\rho <\min (1,\delta \mathrm{/}2)$ for the quantizer ramps. Away from finitely many scalar boundary strips of width at most $\rho$, these quantizers give exactly the grids in steps 1.2 and 2.1; the upper-endpoint hyperplanes are null. The unpositioned quantizer has every output entry in $[-M,1]$: its initial residual sentinel ramp joins output values $-M$ and $0$, or $1-\rho$ and $-M$, and each later clamp ramp stays between its adjacent grid endpoints. A raw normalized coordinate outside $[-\rho ,1]$ is an exact sentinel and stays so through the clamps. By step 4.1, all inputs outside the fixed box $[-1,1{]}^{d\times n}$ therefore have exactly zero output. Within this box, the set where the quantizer differs from its discrete version is contained in finitely many coordinate strips of total measure tending to zero with $\rho$; previously clamped grid points stay fixed under later clamps. The position-tagged input region is already bounded, and the same strip estimate applies there. There is a common bound on all simulated outputs, independent of $\rho$ and of the attention scales: on any bounded scalar interval a ramped gate lies between values of its two fixed adjacent affine pieces; attention outputs are convex combinations of fixed linear value projections, irrespective of score scale. Residual addition and induction over the fixed finite list of layers give a finite common bound, including the exact filter. Thus each transition-region error integral is bounded by a fixed constant times its strip measure. First choose the mesh so the modified lookup error in steps 2.2 and 3.1 is small; fix its table and coefficients and the resulting common bound; then shrink the quantizer strips and choose lookup simulation accuracy on the finite states as in step 3.3. The finite-grid simulation error is integrated over a fixed finite-volume box, while the unbounded exterior contributes exactly zero by step 4.1. Taking all these contributions sufficiently small, with the affine volume factor $(b-a{)}^{dn}$ included, makes the total error less than $\epsilon$ in both assertions.