Latent-class contrastive representation learning

Let $(\mathcal{C},{\mathcal{C}}_0,\rho )$ be a measurable probability space and $(\mathcal{X},\mathcal{A})$ a measurable space. For each class $c\in \mathcal{C}$, let $D_c$ be a probability measure on $(\mathcal{X},\mathcal{A})$ and assume $c\mapsto D_c(A)$ is measurable for every $A\in \mathcal{A}$. Assume the displayed mixtures define probability measures. The positive-pair law $D_{\mathrm{s}\mathrm{i}\mathrm{m}}$ and negative marginal $D_{\mathrm{n}\mathrm{e}\mathrm{g}}$ are specified by

$D_{\mathrm{s}\mathrm{i}\mathrm{m}}(A\times B):={\int }_{\mathcal{C}}{D}_c(A)D_c(B)\rho (dc),D_{\mathrm{n}\mathrm{e}\mathrm{g}}(A):={\int }_{\mathcal{C}}{D}_c(A)\rho (dc)$

for measurable $A,B\in \mathcal{A}$. Equivalently, draw $C\sim \rho$ and then draw $X,X^{+}$ independently with law $D_C$ for a positive pair; draw a fresh independent class $C^{\mathrm{\prime }}\sim \rho$ and $X^{-}\sim D_{C^{\mathrm{\prime }}}$ for one negative. The class labels are latent: the learner observes the pair and negative data, not their classes.

Let $d\ge 1$, $R>0$, and let $\mathcal{F}$ be a nonempty class of measurable maps $f:\mathcal{X}\to {\mathbb{R}}^d$ satisfying ${\sup }_{x\in \mathcal{X}}\parallel f(x){\parallel }_2\le R$. Use either one-negative loss ${\mathrm{\ell }}_{\mathrm{h}}(t)=(1+t{)}_{+}$ or ${\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}}(t)={\log }_2(1+e^t)$ from the immediately preceding loss-properties lemma. The contrastive score and population loss are

$q_f(x,x^{+},x^{-}):=f(x{)}^{\mathrm{\top }}(f(x^{-})-f(x^{+})),L_{\mathrm{u}\mathrm{n}}(f):={\mathbb{E}}_{(X,X^{+})\sim D_{\mathrm{s}\mathrm{i}\mathrm{m}},X^{-}\sim D_{\mathrm{n}\mathrm{e}\mathrm{g}}}[\mathrm{\ell }(q_f(X,X^{+},X^{-}))]\mathrm{.}$

For $M\ge 1$, an observed sample is $S=((x_j,x_j^{+},x_j^{-}){)}_{j=1}^M$ of $M$ independent triplets with common law $D_{\mathrm{s}\mathrm{i}\mathrm{m}}\otimes D_{\mathrm{n}\mathrm{e}\mathrm{g}}$, and its empirical loss is ${\widehat{L}}_{\mathrm{u}\mathrm{n}}(f):=M^{-1}{\sum }_{j=1}^M\mathrm{\ell }(q_f(x_j,x_j^{+},x_j^{-}))$. For this sample define the unnormalized vector Rademacher complexity

${\mathcal{R}}_S(\mathcal{F}):={\mathbb{E}}_{\epsilon }{\sup }_{f\in \mathcal{F}}{\sum }_{j=1}^M{\sum }_{t=1}^d({\epsilon }_{jt0}{f}_t(x_j)+{\epsilon }_{jt+}{f}_t(x_j^{+})+{\epsilon }_{jt-}{f}_t(x_j^{-})),$

where every ${\epsilon }_{jt0},{\epsilon }_{jt+},{\epsilon }_{jt-}$ is an independent uniform sign. This is the one-sided convention with no absolute value and no $1\mathrm{/}M$ factor.

All vector and matrix expectations in this definition are taken entrywise in finite dimensions. The bound on $f$ makes each coordinate of $f(X)$ and every product defining a covariance entry integrable.

Let $C_{+},C_{-}\sim \rho$ be independent class draws and set $\tau :=\mathrm{Pr}(C_{+}=C_{-})<1$. Assume the diagonal $\mathrm{\Delta }:=\{(c,c):c\in \mathcal{C}\}$ and each ${\mathrm{\Delta }}_A:=\{(c,c):c\in A\}$, $A\in {\mathcal{C}}_0$, are measurable. The average binary task draws $(C_{+},C_{-})$ conditional on $C_{+}\mathrm{\ne }{C}_{-}$, then samples a label $Y$ uniformly from $\{+1,-1\}$ and an input $X$ from $D_{C_{+}}$ if $Y=+1$ or $D_{C_{-}}$ if $Y=-1$. For a vector representation $f$, a binary linear classifier uses a weight vector $w\in {\mathbb{R}}^d$ and loss $\mathrm{\ell }(-Y{w}^{\mathrm{\top }}f(X))$. Define $L_{\mathrm{s}\mathrm{u}\mathrm{p}}(f)$ as the average over tasks of the infimum of this risk over $w\in {\mathbb{R}}^d$. Write ${\mu }_c:={\mathbb{E}}_{X\sim D_c}f(X)$ and define the class-mean classifier by $w_{\mu }={\mu }_{C_{+}}-{\mu }_{C_{-}}$; its average risk is $L_{\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)$. For $\gamma >0$, write $L_{\gamma ,\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)$ for the same mean-classifier risk with margin loss ${\mathrm{\ell }}_{\gamma }(t):=(1+t\mathrm{/}\gamma {)}_{+}$ in place of $\mathrm{\ell }$.

For $\tau >0$, let $L_{\mathrm{u}\mathrm{n}}^{=}(f)$ and $L_{\mathrm{u}\mathrm{n}}^{\mathrm{\ne }}(f)$ be the conditional contrastive risks given $C_{+}=C_{-}$ and $C_{+}\mathrm{\ne }{C}_{-}$, respectively. Let $\nu$ be the class law conditioned on the collision event, namely $\nu (A):=(\rho \otimes \rho )({\mathrm{\Delta }}_A)\mathrm{/}\tau$ for $A\in {\mathcal{C}}_0$; for an atomic class law this gives $\nu (\{c\})=\rho (\{c\}{)}^2\mathrm{/}\tau$. Define ${\mathrm{\Sigma }}_f(c):={\mathbb{E}}_{X\sim D_c}[(f(X)-{\mu }_c)(f(X)-{\mu }_c{)}^{\mathrm{\top }}]$ and

$s(f):={\mathbb{E}}_{C\sim \nu }[\sqrt{\parallel {\mathrm{\Sigma }}_f(C){\parallel }_{\mathrm{o}\mathrm{p}}}{\mathbb{E}}_{X\sim D_C}\parallel f(X){\parallel }_2],\parallel \mathrm{\Sigma }{\parallel }_{\mathrm{o}\mathrm{p}}:={\max }_{\parallel u{\parallel }_2=1}\parallel \mathrm{\Sigma }u{\parallel }_2\mathrm{.}$

The assumed measurability of $\mathrm{\Delta }$ and every ${\mathrm{\Delta }}_A$ makes the displayed collision event and conditional class law well-defined; disjoint class sets give disjoint ${\mathrm{\Delta }}_A$, so the formula for $\nu$ is countably additive and has total mass one. Assume also that $c\mapsto {\mu }_c$ and $c\mapsto {\mathrm{\Sigma }}_f(c)$ are measurable for each $f$, and that the displayed class-conditional risks and random suprema used below are measurable.

If $\tau =0$, there is no collision term; set $s(f):=0$ and use $L_{\mathrm{u}\mathrm{n}}(f)=L_{\mathrm{u}\mathrm{n}}^{\mathrm{\ne }}(f)$. Any empirical minimizer or approximate minimizer used below is separately assumed to be a measurable selector; no selector-existence claim is part of this definition.

Probability measures and probability spaces: fixes probability laws and their total mass.

Expectation of a nonnegative or integrable random variable: supplies scalar expectations; vector and matrix expectations use the entrywise convention stated above.

Supervised prediction problems

Population and empirical risk: supply the measurable prediction-problem and population/empirical risk conventions.

Population and empirical risks are measurable: establishes measurability of fixed-predictor risks in the supervised interface.

Identical distribution and IID families

Independent random elements have product joint law: fix IID sampling and the product-law convention for independent draws.

One-sided Rademacher complexity: supplies the one-sided sign-supremum convention; the vector form above is unnormalized as in the source.

Moments, variance, and covariance on a probability space

The operator norm is zero on the zero domain and otherwise is max_{||v||=1} ||Tv||: define the within-class covariance matrix and its operator norm.

Convexity, Lipschitz bounds, and oscillation of the source contrastive losses: defines the source hinge/logistic losses and the common constant $L_0$.

Saunshi et al., Sections 2–5, latent classes, pair/triplet sampling, mean classifier and one-negative objective.

Lei et al., Sections 2–4, contrastive sampling and a distinct unsupervised-loss generalization analysis.

The two class roots are sampled independently. The arrows show conditional sampling laws, not deterministic functions; $X$ and $X^{+}$ are independent conditional on $C_{+}$, and the negative example has its own independent class draw.

diagram failed to render:
\begin{tikzpicture}[>=stealth]
  \node[draw, rounded corners] (cp) at (0,0) {$C_+\sim\rho$};
  \node[draw, rounded corners] (x) at (-1.2,-1.2) {$X\sim D_{C_+}$};
  \node[draw, rounded corners] (xp) at (1.2,-1.2) {$X^+\sim D_{C_+}$};
  \node[draw, rounded corners] (cm) at (4,0) {$C_-\sim\rho$};
  \node[draw, rounded corners] (xm) at (4,-1.2) {$X^-\sim D_{C_-}$};
  \draw[->] (cp) -- (x);
  \draw[->] (cp) -- (xp);
  \draw[->] (cm) -- (xm);
\end{tikzpicture}

The transfer claims below concern the source's average binary tasks, one independent marginal negative, the stated hinge/logistic losses, and the class-mean linear classifier. They do not assert multiclass top-1 accuracy, universal invariance, an attained population minimizer, or a guarantee for empirical SimCLR outcomes. The source treats multiple-negative and block-similarity extensions separately.

Saunshi et al., Sections 4–6 and Supplementary Appendix B.

Chen et al., Sections 2–4, SimCLR's batch NT-Xent objective and reported experiments.