Finite feedforward computation graphs and layered MLPs

Fix integers $d,r\ge 1$. A finite feedforward computation graph from ${\mathbb{R}}^d$ to ${\mathbb{R}}^r$ consists of a finite directed graph $(V,E)$ with no directed cycle; pairwise disjoint input, hidden, and output vertex sets $I,H,O$ whose union is $V$; exactly $d$ input vertices and $r$ output vertices; and a specified ordering of $V$ in which the tail of every directed edge precedes its head. The input vertices are labelled bijectively by the coordinates $1,\dots ,d$, and input vertex $i$ has value $x_i$ for $x\in {\mathbb{R}}^d$. Each non-input vertex $v$ has a real bias $b_v$, a real weight $w_{uv}$ for every edge $u\to v$, and a real activation function ${\rho }_v$. In the specified order, its value is

$z_v={\rho }_v\text{\hspace{0.17em}⁣}(b_v+{\sum }_{u:u\to v}{w}_{uv}{z}_u)\mathrm{.}$

The network output is the vector of values at $O$. A layered multilayer perceptron is the specialization with layers $V_0,\dots ,V_L$, where $V_0=I$, $V_L=O$, and every edge goes from $V_{\mathrm{\ell }-1}$ to $V_{\mathrm{\ell }}$ for some $\mathrm{\ell }$; the hidden vertices use a shared activation and the output vertices use the identity activation. A fully connected layered perceptron includes every edge between consecutive layers. The fixed ordering makes the recursive evaluation unambiguous and is part of the graph data.

The diagram shows the edge direction and evaluation convention for a fully connected scalar-output perceptron with one hidden layer. Each circle is one scalar-valued node; the hidden nodes apply $\sigma$ and the output node is linear. The diagram illustrates the definition and is not used in any proof.

diagram failed to render:
\begin{tikzpicture}[>=stealth, node distance=1.5cm]
  \node[draw,circle] (x1) {$x_1$};
  \node[draw,circle,below of=x1] (x2) {$x_2$};
  \node[draw,circle,right of=x1,yshift=-0.35cm] (h1) {$\sigma$};
  \node[draw,circle,below of=h1] (h2) {$\sigma$};
  \node[draw,circle,right of=h1,yshift=-0.35cm] (y) {$y$};
  \draw[->] (x1) -- (h1);
  \draw[->] (x1) -- (h2);
  \draw[->] (x2) -- (h1);
  \draw[->] (x2) -- (h2);
  \draw[->] (h1) -- (y);
  \draw[->] (h2) -- (y);
\end{tikzpicture}

Leshno et al., §2: one-hidden-layer feedforward network convention

Goodfellow et al., §6.4.1: feedforward network representation scope