Symmetric image-text contrastive loss

Let $\mathsf{X}$ and $\mathsf{T}$ be nonempty sets of image and text inputs, let $d,N\ge 1$, and let $((x_i,t_i){)}_{i=1}^N$ be an indexed batch whose $i$th image and $i$th text are the designated matched pair. Let $f_{\theta }:\mathsf{X}\to {\mathbb{R}}^d$ and $g_{\varphi }:\mathsf{T}\to {\mathbb{R}}^d$ be image and text encoder maps into one shared $d$-dimensional space; any learned projection into that space is included in the corresponding map. A finite feedforward network on vectorized inputs is one possible encoder realization. Assume $f_{\theta }(x_i)\mathrm{\ne }0$ and $g_{\varphi }(t_i)\mathrm{\ne }0$ for every $i$. For $a,b\in {\mathbb{R}}^d$, set $⟨a,b⟩:={\sum }_{r=1}^d{a}_r{b}_r$ and $\mathrm{\parallel }a{\mathrm{\parallel }}_2:=\sqrt{⟨a,a⟩}$. Define normalized embeddings

$u_i:=\frac{f_{\theta }(x_i)}{\mathrm{\parallel }{f}_{\theta }(x_i){\mathrm{\parallel }}_2},v_i:=\frac{g_{\varphi }(t_i)}{\mathrm{\parallel }{g}_{\varphi }(t_i){\mathrm{\parallel }}_2}\mathrm{.}$

For a learned real log-scale $\tau$, set $s:=\exp (\tau )>0$ and define the logits ${\mathrm{\ell }}_{ij}:=s⟨u_i,v_j⟩$. For each image-to-text row and text-to-image column, respectively, define

$p_{ij}^{I\to T}:=\frac{\exp ({\mathrm{\ell }}_{ij})}{{\sum }_{k=1}^N\exp ({\mathrm{\ell }}_{ik})},p_{ji}^{T\to I}:=\frac{\exp ({\mathrm{\ell }}_{ij})}{{\sum }_{k=1}^N\exp ({\mathrm{\ell }}_{kj})}\mathrm{.}$

The two directional cross-entropies use the matched diagonal as their labels:

$L_{I\to T}:=-\frac{1}{N}{\sum }_{i=1}^N\log p_{ii}^{I\to T},L_{T\to I}:=-\frac{1}{N}{\sum }_{j=1}^N\log p_{jj}^{T\to I}\mathrm{.}$

The symmetric image-text contrastive loss on this batch is

$L_{\mathrm{s}\mathrm{y}\mathrm{m}}:=\frac{1}{2}(L_{I\to T}+L_{T\to I})\mathrm{.}$

Since the batch is nonempty, the embeddings are nonzero, the exponential is finite and positive on real inputs, and the logarithm is defined on positive reals, every displayed normalization and loss term is well-defined. Each off-diagonal candidate in a row or column denominator is treated as a negative for that anchor, whether or not the corresponding image and text might also be semantically compatible. This is an index-matching objective on the supplied batch; it makes no assumption about latent classes, sampling, or semantic alignment.

For $N=3$, the similarity matrix has this pattern. Row $i$ indexes image $i$ and column $j$ indexes text $j$; outlined diagonal cells are designated matches and dashed off-diagonal cells are other batch candidates. The definition and its formula do not rely on the diagram.

Radford et al., §§2.3–2.5 and Figure 3: batch matching objective, encoders, and training setup

Machine Learning II lecture notes, “The contrastive objective” and “Encoders and scaling”

Finite feedforward computation graphs and layered MLPs: a finite acyclic weighted/activation graph evaluates to an input-output map; this is one possible encoder model class here.

The real exponential function and the number $e$ by a power series: defines the real exponential on every real input by its convergent power series.

The exponential is a continuous bijection from $\mathbb{R}$ onto $(0,\mathrm{\infty })$: the real exponential has positive range, so the scale and softmax denominators are positive.

The natural logarithm as the inverse of the exponential function: defines the logarithm on positive reals as the inverse of the exponential.