Self-supervised and contrastive representations — Examples

Let the latent classes be $a,b$ with equal probabilities, let each class contain one deterministic input, and define a scalar representation by $f(a)=0$ and $f(b)=1$. Use one independent marginal negative and the hinge loss. The probability that the negative class equals the positive-pair class is $\tau =1\mathrm{/}2$. Conditional on such a collision, the contrastive score is always zero and the conditional loss is $1$. Conditional on distinct classes, the ordered pair $(a,b)$ has loss $1$ and $(b,a)$ has loss $0$, so $L_{\mathrm{u}\mathrm{n}}^{\mathrm{\ne }}(f)=1\mathrm{/}2$ and the full contrastive risk is $L_{\mathrm{u}\mathrm{n}}(f)=3\mathrm{/}4$.

Latent-class contrastive representation learning: defines the positive-pair and independent-negative sampling laws.

Convexity, Lipschitz bounds, and oscillation of the source contrastive losses: defines the hinge loss used in the calculation.

Saunshi et al., Section 4.2 and Equation (8): decomposes the objective by whether the negative class collides with the positive class.

Let $C=\{a,b\}$ be uniform and let each class contain two equally likely inputs indexed by $\eta \in \{-1,+1\}$. In ${\mathbb{R}}^2$, define $f(a,\eta )=(1,\eta r)$ and $f(b,\eta )=(-1,\eta r)$ for $r\ge 1$. Each class mean is respectively $(1,0)$ and $(-1,0)$, so the mean classifier separates both classes with hinge loss zero. Yet for one independent marginal negative and hinge contrastive loss, the total risk is $L_{\mathrm{u}\mathrm{n}}(f)=\frac{1}{4}+\frac{1}{2}{r}^2$, which grows without bound as the within-class variance $r^2$ increases. The norm bound holds with $R=\sqrt{1+r^2}$ for each fixed $r$.

Latent-class contrastive representation learning: defines the latent-class triplet law, mean classifier, covariance and contrastive risk.

Convexity, Lipschitz bounds, and oscillation of the source contrastive losses: defines the hinge loss.

Saunshi et al., Section 5.1 and Figure 1(b): gives a two-class example where high within-class variation raises the unsupervised loss despite a good mean classifier.

Let the raw input be $(c,n)\in \{-1,+1{\}}^2$, where the latent class is $c$ and, conditional on either class, the nuisance coordinate $n$ is uniform on $\{-1,+1\}$. Apply the augmentation $T(c,n)=n$ before forming positive pairs and negative samples. The resulting input law $D_c$ is the same uniform law for both classes, so $D_{\mathrm{s}\mathrm{i}\mathrm{m}}=D_{\mathrm{n}\mathrm{e}\mathrm{g}}\otimes D_{\mathrm{n}\mathrm{e}\mathrm{g}}$ and the observed triplets carry no class-dependent distributional signal. Consequently, every deterministic representation of the augmented input is independent of the binary task label, and its optimal average hinge risk for a uniform-label two-class task is exactly $1$; the erased class coordinate is unavailable to both the contrastive learner and the downstream linear classifier.

Latent-class contrastive representation learning: defines the class-conditional post-augmentation laws and binary tasks.

Convexity, Lipschitz bounds, and oscillation of the source contrastive losses: defines the hinge loss used to compute the downstream risk.

Chen et al., Section 3: explains how augmentation choices define the predictive tasks in contrastive learning.

In SimCLR's ImageNet experiments, a projection head is a map $g$ applied to the encoder representation $h$ to produce $z=g(h)$; linear evaluation trains a linear classifier on a fixed representation. The authors report a comparison among an identity head, a linear projection head, and a nonlinear projection head with one additional hidden layer and ReLU. In their Figure 8, the nonlinear head's pre-projection representation $h$ scores about three percentage points above the linear-head condition and more than ten points above the no-head condition; they also report that $h$ scores more than ten points above the projected representation $z=g(h)$. Their default setup uses a ResNet-50 encoder, a two-layer MLP projection head to a 128-dimensional latent space, NT-Xent (normalized temperature-scaled cross-entropy), batch size 4096 and 100 training epochs. These are reported empirical results under the paper's evaluation protocol, not a theorem that projection heads universally improve representations.

Latent-class contrastive representation learning: supplies the representation-learning context only; the reported benchmark is empirical.

Chen et al., Section 4 default setup and Section 4.2, Figure 8: reports the encoder/head configuration and linear-evaluation comparisons.

Chen et al., Supplementary Appendix B.4: provides additional projection-head analysis.

Consider two equally likely deterministic classes $a,b$ and the scalar representation $f(a)=2$, $f(b)=4$, so $R=4$, $\tau =1\mathrm{/}2$, and the classwise centered representation has sub-Gaussian proxy ${\sigma }^2=0$. For the distinct-class tasks, the mean classifier has logistic risk

$L_{\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)=\frac{1}{2}[{\log }_2(1+e^4)+{\log }_2(1+e^{-8})]>\frac{5}{2},$

while its source hinge margin risk at $\gamma =1$ is exactly $L_{1,\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)=5\mathrm{/}2$. The logistic contrastive risk conditioned on distinct classes has the same value as the displayed logistic mean-classifier risk. Consequently, for every $0<ϵ\le 1$ smaller than the positive gap, the source hinge margin comparison $L_{\mathrm{u}\mathrm{n}}^{\mathrm{\ne }}(f)\le \gamma L_{\gamma ,\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)+ϵ$ fails for logistic contrastive loss. In particular, the hinge-margin corollary cannot be transferred unchanged to logistic training.

For the optional transfer comparison, assume the Axiom of Choice, fix $M\ge 1$ and $0<\delta <1$, impose the corollary's remaining model, risk and supremum measurability, and measurable exact-empirical-minimizer hypotheses, and use the singleton $\mathcal{F}=\{f\}$. The proposed extension is

$L_{\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(\widehat{f})\le \gamma L_{\gamma ,\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)+ϵ+\frac{\sqrt{2}{L}_0\tau s(f)+2{\mathrm{\Gamma }}_M^{\mathrm{l}\mathrm{o}\mathrm{g}}(S,\delta )}{1-\tau },$

where $L_{\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(\widehat{f})$ uses logistic task loss, $L_{\gamma ,\mathrm{s}\mathrm{u}\mathrm{p}}^{\mu }(f)$ uses the scaled hinge margin loss, $\gamma =1$ here, and

${\mathrm{\Gamma }}_M^{\mathrm{l}\mathrm{o}\mathrm{g}}(S,\delta ):=\frac{2\sqrt{18}{L}_{0}R{\mathcal{R}}_S(\mathcal{F})}{M}+3B_{{\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}},R}\sqrt{\frac{\ln (4\mathrm{/}\delta )}{2M}}$

uses the logistic loss oscillation. The example tests this claimed mean-classifier bound with logistic contrastive training.

Latent-class contrastive representation learning: defines the latent-class risks, scores and centered classwise representations.

Convexity, Lipschitz bounds, and oscillation of the source contrastive losses: defines the base-2 logistic loss and hinge margin loss.

Class-conditional sub-Gaussian concentration yields a contrastive margin bound: states the one-negative margin inequality for hinge contrastive loss.

Sub-Gaussian contrastive transfer with a mean-classifier margin loss: states the source margin transfer corollary with its hinge-only scope.

The Axiom of Choice: supplies the choice assumption for the optional transfer comparison.

Saunshi et al., Lemma 5.1 and Corollary 5.1.1: states the margin comparison in the hinge-loss section.