Multimodal contrastive alignment and transfer

Let $\mathsf{X}$ and $\mathsf{T}$ be nonempty sets of image and text inputs, let $d,N\ge 1$, and let $((x_i,t_i){)}_{i=1}^N$ be an indexed batch whose $i$th image and $i$th text are the designated matched pair. Let $f_{\theta }:\mathsf{X}\to {\mathbb{R}}^d$ and $g_{\varphi }:\mathsf{T}\to {\mathbb{R}}^d$ be image and text encoder maps into one shared $d$-dimensional space; any learned projection into that space is included in the corresponding map. A finite feedforward network on vectorized inputs is one possible encoder realization. Assume $f_{\theta }(x_i)\mathrm{\ne }0$ and $g_{\varphi }(t_i)\mathrm{\ne }0$ for every $i$. For $a,b\in {\mathbb{R}}^d$, set $⟨a,b⟩:={\sum }_{r=1}^d{a}_r{b}_r$ and $\mathrm{\parallel }a{\mathrm{\parallel }}_2:=\sqrt{⟨a,a⟩}$. Define normalized embeddings

$u_i:=\frac{f_{\theta }(x_i)}{\mathrm{\parallel }{f}_{\theta }(x_i){\mathrm{\parallel }}_2},v_i:=\frac{g_{\varphi }(t_i)}{\mathrm{\parallel }{g}_{\varphi }(t_i){\mathrm{\parallel }}_2}\mathrm{.}$

For a learned real log-scale $\tau$, set $s:=\exp (\tau )>0$ and define the logits ${\mathrm{\ell }}_{ij}:=s⟨u_i,v_j⟩$. For each image-to-text row and text-to-image column, respectively, define

$p_{ij}^{I\to T}:=\frac{\exp ({\mathrm{\ell }}_{ij})}{{\sum }_{k=1}^N\exp ({\mathrm{\ell }}_{ik})},p_{ji}^{T\to I}:=\frac{\exp ({\mathrm{\ell }}_{ij})}{{\sum }_{k=1}^N\exp ({\mathrm{\ell }}_{kj})}\mathrm{.}$

The two directional cross-entropies use the matched diagonal as their labels:

$L_{I\to T}:=-\frac{1}{N}{\sum }_{i=1}^N\log p_{ii}^{I\to T},L_{T\to I}:=-\frac{1}{N}{\sum }_{j=1}^N\log p_{jj}^{T\to I}\mathrm{.}$

The symmetric image-text contrastive loss on this batch is

$L_{\mathrm{s}\mathrm{y}\mathrm{m}}:=\frac{1}{2}(L_{I\to T}+L_{T\to I})\mathrm{.}$

Since the batch is nonempty, the embeddings are nonzero, the exponential is finite and positive on real inputs, and the logarithm is defined on positive reals, every displayed normalization and loss term is well-defined. Each off-diagonal candidate in a row or column denominator is treated as a negative for that anchor, whether or not the corresponding image and text might also be semantically compatible. This is an index-matching objective on the supplied batch; it makes no assumption about latent classes, sampling, or semantic alignment.

For $N=3$, the similarity matrix has this pattern. Row $i$ indexes image $i$ and column $j$ indexes text $j$; outlined diagonal cells are designated matches and dashed off-diagonal cells are other batch candidates. The definition and its formula do not rely on the diagram.

diagram failed to render:
\begin{tikzpicture}[x=1cm,y=1cm]
  \node at (1.2,1.3) {text index $j$};
  \node at (-1.25,-0.8) {image index $i$};
  \node at (0,0.65) {$1$};
  \node at (1.2,0.65) {$2$};
  \node at (2.4,0.65) {$3$};
  \node at (-0.55,0) {$1$};
  \node at (-0.55,-0.8) {$2$};
  \node at (-0.55,-1.6) {$3$};
  \node[draw,thick,minimum width=1cm,minimum height=0.55cm] at (0,0) {$\ell_{11}$};
  \node[draw,dashed,minimum width=1cm,minimum height=0.55cm] at (1.2,0) {$\ell_{12}$};
  \node[draw,dashed,minimum width=1cm,minimum height=0.55cm] at (2.4,0) {$\ell_{13}$};
  \node[draw,dashed,minimum width=1cm,minimum height=0.55cm] at (0,-0.8) {$\ell_{21}$};
  \node[draw,thick,minimum width=1cm,minimum height=0.55cm] at (1.2,-0.8) {$\ell_{22}$};
  \node[draw,dashed,minimum width=1cm,minimum height=0.55cm] at (2.4,-0.8) {$\ell_{23}$};
  \node[draw,dashed,minimum width=1cm,minimum height=0.55cm] at (0,-1.6) {$\ell_{31}$};
  \node[draw,dashed,minimum width=1cm,minimum height=0.55cm] at (1.2,-1.6) {$\ell_{32}$};
  \node[draw,thick,minimum width=1cm,minimum height=0.55cm] at (2.4,-1.6) {$\ell_{33}$};
\end{tikzpicture}

Radford et al., §§2.3–2.5 and Figure 3: batch matching objective, encoders, and training setup

Machine Learning II lecture notes, “The contrastive objective” and “Encoders and scaling”

Finite feedforward computation graphs and layered MLPs: a finite acyclic weighted/activation graph evaluates to an input-output map; this is one possible encoder model class here.

The real exponential function and the number $e$ by a power series: defines the real exponential on every real input by its convergent power series.

The exponential is a continuous bijection from $\mathbb{R}$ onto $(0,\mathrm{\infty })$: the real exponential has positive range, so the scale and softmax denominators are positive.

The natural logarithm as the inverse of the exponential function: defines the logarithm on positive reals as the inverse of the exponential.

Fix image and text encoder maps $f_{\theta }:\mathsf{X}\to {\mathbb{R}}^d$ and $g_{\varphi }:\mathsf{T}\to {\mathbb{R}}^d$ into one shared embedding space, and a real log-scale $\tau$ with $s:=\exp (\tau )>0$. The inputs are an image $x\in \mathsf{X}$, an ordered nonempty list of distinct class descriptions $C=(c_1,\dots ,c_K)$, and a finite nonempty list of prompt templates $T=(t_1,\dots ,t_M)$, where each $t_m$ maps a class description to a text input. The admissible input conditions are $f_{\theta }(x)\mathrm{\ne }0$, $g_{\varphi }(t_m(c_k))\mathrm{\ne }0$ for every $k,m$, and a nonzero average text embedding for every class.

1. Normalize the image embedding: $u:=f_{\theta }(x)\mathrm{/}\mathrm{\parallel }{f}_{\theta }(x){\mathrm{\parallel }}_2$.

2. For each class $c_k$ and template $t_m$, form the text embedding $v_{km}:=g_{\varphi }(t_m(c_k))\mathrm{/}\mathrm{\parallel }{g}_{\varphi }(t_m(c_k)){\mathrm{\parallel }}_2$.

3. Average the normalized prompt embeddings in the shared embedding space and normalize the average to obtain the class prototype $q_k:=\left(\frac{1}{M}{\sum }_{m=1}^M{v}_{km}\right)/{\parallel \frac{1}{M}{\sum }_{m=1}^M{v}_{km}\parallel }_2$. This is prompt-template averaging; it averages embeddings before classification, not class probabilities.

4. For each class compute the scaled similarity score $z_k:=s⟨u,q_k⟩$ and softmax probability $p_k:=\exp (z_k)\mathrm{/}{\sum }_{r=1}^K\exp (z_r)$.

5. Return the vector $(p_1,\dots ,p_K)$ and class $c_{k^{\ast }}$, where $k^{\ast }$ is the least index maximizing $z_k$.

The returned classifier is zero-shot in the precise sense that no labeled examples from the target classification dataset are used to fit its class prototypes or classifier. The encoder maps, log-scale, class descriptions, and templates are fixed inputs to this inference procedure; it performs no training or parameter update. The procedure makes no accuracy or distribution-generalization guarantee.

Radford et al., §3.1.2: text-class candidates, scaled cosine scores, and softmax classification; §3.1.4: prompt engineering and embedding-space ensembling

Machine Learning II lecture notes, “Zero-shot transfer” and “Prompt engineering”/“Prompt ensembling”

Symmetric image-text contrastive loss: supplies the shared image/text embedding space, normalized inner-product score convention, and positive learned exponential scale.

The real exponential function and the number $e$ by a power series: defines the real exponential used in the logit scale and softmax.

The exponential is a continuous bijection from $\mathbb{R}$ onto $(0,\mathrm{\infty })$: gives positivity of exponential scores and softmax denominators.