Mean-field training and feature learning

If $f^{-1}(y)\cap U=\mathrm{\varnothing }$, then $K=\mathrm{\varnothing }$ and $H_d(U,U\setminus K)=H_d(U,U)=0$, so $D(f,U,y)=0$. For the identity and $y\in U$, $K=\{y\}$ and the induced map sends the chosen local orientation generator to itself, so $D(\mathrm{id},U,y)=1$.

Under the additivity hypotheses, $K$ is the disjoint union of $K_i=K\cap U_i$. The closed set $Z=U\setminus (U_1\cup U_2)$ has closure in $U$ disjoint from $K$: the compact set $K$ lies in the open set $U_1\cup U_2$. Excision [F3] identifies $H_d(U,U\setminus K)$ with $H_d(U_1\cup U_2,(U_1\cup U_2)\setminus K)$. Since $U_1,U_2$ are disjoint open sets, every singular simplex in their union lies in exactly one of them, so this relative group and its chains split as the direct sum of the two relative groups. The unique orientation class [F1] therefore corresponds to $[U_1{]}_{K_1}+[U_2{]}_{K_2}$. The restrictions of $f$ commute with the inclusions of these pairs, so functoriality [F2] maps this sum to the sum of the two target classes. Their integer coefficients consequently satisfy $D(f,U,y)=D(f,U_1,y)+D(f,U_2,y)$.

For moving-target invariance set $g_t(x)=H_t(x)-y(t)$; translation identifies the target generator with the standard generator at $0$. The zero set $S=\{(t,x):g_t(x)=0\}$ is a closed subset of compact $[0,1]\times \bar{U}$ and is disjoint from $[0,1]\times \mathrm{\partial }U$. Thus its projection $K_{\ast }\subset U$ is compact. Every $g_t$ defines a map of the fixed pair $(U,U\setminus K_{\ast })$ into $({\mathbb{R}}^d,{\mathbb{R}}^d\setminus \{0\})$. The singular prism operator for the homotopy satisfies $\mathrm{\partial }P+P\mathrm{\partial }=(g_1{)}_{\mathrm{\#}}-(g_0{)}_{\mathrm{\#}}$; each prism over a simplex in $U\setminus K_{\ast }$ avoids $0$, so the chain homotopy descends to relative chains. Hence $(g_0{)}_{\ast }=(g_1{)}_{\ast }$ on $H_d(U,U\setminus K_{\ast })$. Restriction compatibility in [F1] sends $[U{]}_{K_{\ast }}$ to $[U{]}_{K_t}$ for the actual time-$t$ zero set, so these two induced integers are exactly $D(H_0,U,y(0))$ and $D(H_1,U,y(1))$. They are equal, proving moving-target invariance.

The target group is $\mathbb{Z}$ by [F4]; the source class is unique by [F1]; and the induced map exists by [F2]. Thus the integer in the Statement is well-defined. Steps 1.1–3.1 establish all asserted properties.

Suppose first that $M>0$ and $w$ has not reached zero by $t_0+2M\mathrm{/}\eta$. Integrating $w^{\mathrm{\prime }}\ge \eta \mathrm{/}2$ gives $w(t_0+2M\mathrm{/}\eta )\ge w(t_0)+M\ge 0$, contradicting the assumed strict negativity before the first crossing. Hence the first crossing time $\tau$ exists and satisfies the stated bound. If $M=0$, then $w(t_0)=0$ and $\tau =t_0$.

On $[t_0,\tau ]$, $w$ is strictly increasing with $w^{\mathrm{\prime }}\ge \eta \mathrm{/}2$, so it is an admissible absolutely continuous time change. Its inverse satisfies $dt\mathrm{/}dw\le 2\mathrm{/}\eta$. If $L>0$, set $p=ϵ+Lq$. The inequality for $q$ gives $dp\mathrm{/}dw\le (2L\mathrm{/}\eta )\mathrm{\mid }w\mathrm{\mid }p$, and integration from $w(t_0)$ to $0$ yields $p(\tau )\le ϵe^{L{M}^2\mathrm{/}\eta }$. Rearranging gives $q(\tau )\le ϵ(e^{L{M}^2\mathrm{/}\eta }-1)\mathrm{/}L$. If $L=0$, direct integration gives $q(\tau )\le (2ϵ\mathrm{/}\eta ){\int }_{w(t_0)}^0\mathrm{\mid }s\mathrm{\mid }ds=ϵw(t_0{)}^2\mathrm{/}\eta \le ϵM^2\mathrm{/}\eta$. The same bounds hold with zero right-hand side when $M=0$.

If $y=f(x)$ for $x\in \mathrm{spt}\mu$ and $V$ is any open neighborhood of $y$, continuity makes $f^{-1}(V)$ an open neighborhood of $x$. Hence $f_{\mathrm{\#}}\mu (V)=\mu (f^{-1}(V))>0$, so $y\in \mathrm{spt}(f_{\mathrm{\#}}\mu )$. Taking the closure relative to $Y$, since support is closed, $\overline{f(\mathrm{spt}\mu )}\subseteq \mathrm{spt}(f_{\mathrm{\#}}\mu )$.

The complement of $\mathrm{spt}\mu$ has $\mu$-measure zero: every point in it lies in a zero-measure open neighborhood, and the countable relative Euclidean basis gives a countable cover by zero-measure basis sets. If $y\mathrm{\notin }\overline{f(\mathrm{spt}\mu )}$, choose an open neighborhood $V$ disjoint from that closure. Then $f^{-1}(V)\subseteq X\setminus \mathrm{spt}\mu$, so $f_{\mathrm{\#}}\mu (V)=0$ and $y\mathrm{\notin }\mathrm{spt}(f_{\mathrm{\#}}\mu )$. Thus $\mathrm{spt}(f_{\mathrm{\#}}\mu )\subseteq \overline{f(\mathrm{spt}\mu )}$.

The coordinate $w$ is measurable, so $w_{+}$ and $w_{-}$ are nonnegative measurable functions. By [F1], ${\alpha }_{\mu }$ and ${\beta }_{\mu }$ are measures, and [F2] gives ${\alpha }_{\mu }(\mathrm{\Theta }),{\beta }_{\mu }(\mathrm{\Theta })\le \int \mathrm{\mid }w\mathrm{\mid }d\mu <\mathrm{\infty }$.

For each measurable $B$, $w=w_{+}-w_{-}$ and both integrals are finite, so ${\int }_{\mathbb{R}\times B}wd\mu ={\alpha }_{\mu }(B)-{\beta }_{\mu }(B)$. Countable additivity of both finite measures makes their difference countably additive with finite real values, hence a finite signed measure by [F3]. For every $B\in \mathcal{T}$ and every countable measurable partition $(B_j)$ of $B$, ${\sum }_j\mathrm{\mid }{h}^1(\mu )(B_j)\mathrm{\mid }\le {\sum }_j({\alpha }_{\mu }(B_j)+{\beta }_{\mu }(B_j))={\alpha }_{\mu }(B)+{\beta }_{\mu }(B)$; taking the supremum in [F4] proves the stated pointwise total-variation bound. The weighted penalty inequality follows by monotonicity of the nonnegative integral, first for simple nonnegative weights and then by increasing simple approximation. For a finite-valued simple $\phi$, linearity [F5] and the definitions of ${\alpha }_{\mu },{\beta }_{\mu }$ give $\int \phi d{\alpha }_{\mu }-\int \phi d{\beta }_{\mu }=\int w\phi d\mu$. For general bounded measurable $\phi$, choose $M$ with $\mathrm{\mid }\phi \mathrm{\mid }\le M$ and partition $[-M,M]$ into finitely many intervals of length at most $1\mathrm{/}n$; assigning one endpoint on each measurable inverse image gives a finite-valued simple ${\phi }_n$ with $\mathrm{\parallel }{\phi }_n-\phi {\mathrm{\parallel }}_{\mathrm{\infty }}\le 1\mathrm{/}n$. The errors in the two ${\alpha }_{\mu },{\beta }_{\mu }$ integrals are at most $({\alpha }_{\mu }(\mathrm{\Theta })+{\beta }_{\mu }(\mathrm{\Theta }))\mathrm{/}n$, and the error in the $w\phi$ integral is at most $\int \mathrm{\mid }w\mathrm{\mid }d\mu \mathrm{/}n$. Letting $n\to \mathrm{\infty }$ proves the identity.

Let $\phi :\mathrm{\Theta }\to \mathcal{H}$ have the bounded everywhere simple approximation specified in the Definition. Its approximants pulled back to the product, multiplied by scalar simple functions converging everywhere to $w$, show that $w\phi (\theta )$ is strongly measurable; moreover $\mathrm{\parallel }w\phi (\theta )\mathrm{\parallel }\le \mathrm{\mid }w\mathrm{\mid }\mathrm{\parallel }\phi {\mathrm{\parallel }}_{\mathrm{\infty }}$, so [F6] makes it Bochner integrable. For each $z\in \mathcal{H}$, [F7] moves the bounded functional $x\mapsto ⟨z,x⟩$ inside that Bochner integral, and step 2.1 identifies the resulting scalar integral with the pairing in the Definition. Equality of all inner products proves the displayed feature-moment factorization.

For $x,y\in Q_r$, the segment between them lies in $Q_r$. The bound on $d\mathrm{\Phi }$ and the fundamental theorem of calculus on this segment give $\mathrm{\parallel }\mathrm{\Phi }(x)-\mathrm{\Phi }(y)\mathrm{\parallel }\le L_{\mathrm{\Phi }}\mathrm{\mid }x-y\mathrm{\mid }$. For $V$, divide the segment into $N$ equal pieces. The two ${\lambda }_V$-support inequalities at consecutive endpoints and the uniform bound $M_V$ on subgradients yield $\mathrm{\mid }V(x_{j+1})-V(x_j)\mathrm{\mid }\le M_V\mathrm{\mid }x-y\mathrm{\mid }\mathrm{/}N+\mathrm{\mid }{\lambda }_V\mathrm{\mid }\mathrm{\mid }x-y{\mathrm{\mid }}^2\mathrm{/}(2N^2)$. Summing and letting $N\to \mathrm{\infty }$ gives $\mathrm{\mid }V(x)-V(y)\mathrm{\mid }\le M_V\mathrm{\mid }x-y\mathrm{\mid }$.

For any coupling $\pi \in \mathrm{\Gamma }({\mu }_n,\mu )$, [F3] and [F1] give $\mathrm{\parallel }\int \mathrm{\Phi }d{\mu }_n-\int \mathrm{\Phi }d\mu \mathrm{\parallel }\le L_{\mathrm{\Phi }}\int \mathrm{\mid }x-y\mathrm{\mid }d\pi \le L_{\mathrm{\Phi }}(\int \mathrm{\mid }x-y{\mathrm{\mid }}^{2}d\pi {)}^{1\mathrm{/}2}$. The same coupling gives $\mathrm{\mid }\int Vd{\mu }_n-\int Vd\mu \mathrm{\mid }\le M_V(\int \mathrm{\mid }x-y{\mathrm{\mid }}^{2}d\pi {)}^{1\mathrm{/}2}$. Taking the infimum over couplings and using [F2] makes both differences tend to zero. Continuity of $R$ then gives $F({\mu }_n)\to F(\mu )$.

Fix $u\in \mathrm{\Omega }$. For $t\ge 0$, the measure $\mu +t{\delta }_u$ from [F3] is nonnegative and $F(\mu +t{\delta }_u)$ is finite for sufficiently small $t$. Its right derivative at $t=0$ is $d{R}_{m_{\mu }}[\mathrm{\Phi }(u)]+V(u)=G_{\mu }(u)$. If $\mu$ minimizes $F$, this derivative is nonnegative. Since $u$ was arbitrary, $G_{\mu }\ge 0$ everywhere.

For $t$ in a neighborhood of $0$ with $1+t>0$, the measure $(1+t)\mu$ is admissible. Differentiating $F((1+t)\mu )$ at $0$ gives $d{R}_{m_{\mu }}[m_{\mu }]+\int Vd\mu =\int G_{\mu }d\mu$ by [F2]. A minimum at the interior parameter $t=0$ forces this derivative to be zero. Since $G_{\mu }$ is nonnegative and integrable, $\int G_{\mu }d\mu =0$ implies $G_{\mu }=0$ $\mu$-almost everywhere.

For any finite-energy nonnegative $\nu$, convexity of $R$ gives $R(m_{\nu })\ge R(m_{\mu })+d{R}_{m_{\mu }}[m_{\nu }-m_{\mu }]$. By [F2], after adding the $V$ terms this is $F(\nu )-F(\mu )\ge \int G_{\mu }d\nu -\int G_{\mu }d\mu \ge 0$, using the assumed nonnegativity and $\mu$-almost-everywhere vanishing of $G_{\mu }$. Infinite-energy competitors cannot lower $F$, so $\mu$ is a global minimizer.

First, $\mathrm{\Phi }$ is Borel measurable with values in the separable space $\mathcal{H}$. Fix a countable dense sequence $(h_j)$ in $\mathcal{H}$; the map assigning to $\mathrm{\Phi }(u)$ a nearest point among $h_1,\dots ,h_k$, breaking ties by least index, is finite-valued and measurable, and its error tends to zero as $k\to \mathrm{\infty }$. Thus $\mathrm{\Phi }$ is strongly measurable. Moreover, $\mathrm{\parallel }\mathrm{\Phi }(w,\theta )\mathrm{\parallel }\le \mathrm{\parallel }\phi {\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\mid }w\mathrm{\mid }$. A probability law with finite second moment has $\int \mathrm{\mid }w\mathrm{\mid }d\mu \le (\int \mathrm{\mid }w{\mathrm{\mid }}^{2}d\mu {)}^{1\mathrm{/}2}<\mathrm{\infty }$ by Cauchy–Schwarz. Thus [F3] gives existence of its feature moment, while $0\le V\le \mathrm{\parallel }\tilde{V}{\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\mid }w\mathrm{\mid }$ gives a finite penalty; $R$ is finite-valued. This proves the asserted well-definedness for every law in the Statement. By [A1] and [F1] choose couplings ${\pi }_n$ of ${\mu }_n$ and $\mu$ with ${\delta }_n^2=\int (\mathrm{\mid }w-w^{\mathrm{\prime }}{\mathrm{\mid }}^2+\mathrm{\mid }\theta -{\theta }^{\mathrm{\prime }}{\mathrm{\mid }}^2)d{\pi }_n\to 0$. By [F2], the coupling marginal identities hold also for these Bochner feature integrals: apply a bounded linear functional, use the scalar marginal identity, and identify the vectors by all such evaluations. Consequently [F4] and the pointwise bound $\mathrm{\parallel }w\phi (\theta )-w^{\mathrm{\prime }}\phi ({\theta }^{\mathrm{\prime }})\mathrm{\parallel }\le \mathrm{\parallel }\phi {\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\mid }w-w^{\mathrm{\prime }}\mathrm{\mid }+\mathrm{Lip}(\phi )\mathrm{\mid }{w}^{\mathrm{\prime }}\mathrm{\mid }\mathrm{\mid }\theta -{\theta }^{\mathrm{\prime }}\mathrm{\mid }$ give $\mathrm{\parallel }{m}_{{\mu }_n}-m_{\mu }\mathrm{\parallel }\le (\mathrm{\parallel }\phi {\mathrm{\parallel }}_{\mathrm{\infty }}+\mathrm{Lip}(\phi )(\int \mathrm{\mid }{w}^{\mathrm{\prime }}{\mathrm{\mid }}^{2}d\mu (w^{\mathrm{\prime }},{\theta }^{\mathrm{\prime }}){)}^{1\mathrm{/}2}){\delta }_n⟶0.$ The second-moment factor belongs to the fixed limit marginal and is finite, with no bound on the diameter or interior of $\mathrm{\Theta }$ required.

Since $\mid \mathrm{\mid }w\mathrm{\mid }-\mathrm{\mid }{w}^{\mathrm{\prime }}\mathrm{\mid }\mid \le \mathrm{\mid }w-w^{\mathrm{\prime }}\mathrm{\mid }$, the same estimate with scalar $\tilde{V}$ in place of $\phi$ gives $\int Vd{\mu }_n\to \int Vd\mu$. Step 1.1 gives convergence of feature moments in $\mathcal{H}$; continuity of $R$ therefore gives convergence of the risk terms. Adding the two limits proves $F({\mu }_n)\to F(\mu )$. Under the full feature-energy hypotheses, the identical feature and penalty integrals are finite by step 1.1, so the finite branch in that definition gives exactly this $F$. If $\mathrm{\Theta }=\mathrm{\varnothing }$, no law in the Statement exists and the assertion is vacuous; zero maps, zero weights and singleton or lower-dimensional $\mathrm{\Theta }$ are included in the estimates.

Let $\alpha ,\beta$ be the raw measures in [F1]. For every countable measurable partition $(B_j)$ of a measurable set $B$, ${\sum }_j\mathrm{\mid }\nu (B_j)\mathrm{\mid }={\sum }_j\mathrm{\mid }\alpha (B_j)-\beta (B_j)\mathrm{\mid }\le \alpha (B)+\beta (B)$. Taking the supremum in [F3] gives $\mathrm{\mid }\nu \mathrm{\mid }(B)\le (\alpha +\beta )(B)$. For a nonnegative simple $\tilde{V}$, the definitions of $\alpha ,\beta$ give $\int \tilde{V}d(\alpha +\beta )=\int \mathrm{\mid }w\mathrm{\mid }\tilde{V}(\theta )d\mu$. If $M$ bounds $\tilde{V}$, partition $[0,M]$ into finitely many intervals of length at most $1\mathrm{/}n$ and assign an endpoint on each measurable inverse image. This gives nonnegative simple $V_n$ with $\mathrm{\parallel }{V}_n-\tilde{V}{\mathrm{\parallel }}_{\mathrm{\infty }}\le 1\mathrm{/}n$. The error on the left is at most $(\alpha (\mathrm{\Theta })+\beta (\mathrm{\Theta }))\mathrm{/}n$, and the error on the right is at most $\int \mathrm{\mid }w\mathrm{\mid }d\mu \mathrm{/}n$; these bounds agree because $\alpha (\mathrm{\Theta })+\beta (\mathrm{\Theta })=\int \mathrm{\mid }w\mathrm{\mid }d\mu$. Taking $n\to \mathrm{\infty }$ proves the identity for $\tilde{V}$. Monotonicity [F4] and the measure domination, with $\mathrm{\mid }\nu \mathrm{\mid }$ a measure by [F7], give $\int \tilde{V}d\mathrm{\mid }\nu \mathrm{\mid }\le \int \tilde{V}d(\alpha +\beta )=\int \mathrm{\mid }w\mathrm{\mid }\tilde{V}(\theta )d\mu$.

The pointwise simple approximation in the Definition makes $\phi$ strongly measurable for the finite measure $\lambda ={\alpha }_{\mu }+{\beta }_{\mu }+{\nu }^{+}+{\nu }^{-}$. Multiplying those approximants by scalar simple functions converging pointwise to $w$ gives a pointwise simple approximation to $w\phi (\theta )$ on the product. Its norm is bounded by $\mathrm{\parallel }\phi {\mathrm{\parallel }}_{\mathrm{\infty }}\mathrm{\mid }w\mathrm{\mid }$, so both this weighted feature and $\phi$ are Bochner integrable for the relevant measures by [F6]. For every $z\in \mathcal{H}$, the function ${\psi }_z(\theta )=⟨z,\phi (\theta )⟩$ is bounded and measurable. The pairing [F1] and equality $h^1(\mu )=\nu$ give $\int w{\psi }_z(\theta )d\mu =\int {\psi }_{z}d\nu$. By [F2] and [F5], these are respectively $⟨z,\int w\phi d\mu ⟩$ and $⟨z,m_{\nu }⟩$, so equality for every $z$ proves equality of feature moments. Adding the risk term to step 1.1 proves $F(\mu )\ge J(\nu )$. The section maps $\theta \mapsto (1,\theta )$ and $\theta \mapsto (-1,\theta )$ are measurable: preimages of measurable rectangles are either empty or measurable subsets of $\mathrm{\Theta }$. Their preimages preserve disjoint unions, so the displayed formula defines a finite nonnegative measure, of mass ${\nu }^{+}(\mathrm{\Theta })+{\nu }^{-}(\mathrm{\Theta })=\mathrm{\mid }\nu \mathrm{\mid }(\mathrm{\Theta })$. The definition of $h^1$ gives $h^1({\mu }_{\nu })={\nu }^{+}-{\nu }^{-}=\nu$ by [A1]. Direct integration on the two sections gives $\int \mathrm{\mid }w\mathrm{\mid }\tilde{V}d{\mu }_{\nu }=\int \tilde{V}d{\nu }^{+}+\int \tilde{V}d{\nu }^{-}=\int \tilde{V}d\mathrm{\mid }\nu \mathrm{\mid }$ and $\int w\phi d{\mu }_{\nu }=m_{\nu }$. Hence $F({\mu }_{\nu })=J(\nu )$. This proves the infimum identity and its attainment.