Structure of connected nilpotent groups and the maximal-torus criterion

We first establish the rigidity input ${\mathrm{Hom}}_{R\text{-groups}}(M_R,U_R)=0$ for every $k$-algebra $R$, every multiplicative-type group $M$, and every unipotent group $U$. Faithfully flat base change splits $M$, so a homomorphism $M_R\to {\mathbf{G}}_{a,R}$ corresponds to a primitive element in $R[X(M)]$. Comparing the coefficients in $\mathrm{\Delta }(\sum a_m{e}_m)=\sum a_m{e}_m\otimes e_m$ and $\sum a_m{e}_m\otimes 1+1\otimes \sum a_m{e}_m$ forces every coefficient, including $a_0$, to vanish. For a general $U$, choose a minimal closed $k$-subgroup $H\subseteq U$ through which a proposed morphism factors; such a minimal subgroup exists by the descending chain condition on closed subschemes of $U$. If $H\mathrm{\ne }1$, the first nontrivial coordinate in an upper-unitriangular normal series gives a nonzero homomorphism $\beta :H\to {\mathbf{G}}_a$ with proper kernel. The composite ${\beta }_{R}f$ is zero by the primitive-element calculation, so $f$ factors through this proper kernel, contradicting minimality. Thus $f=0$. This argument includes arbitrary nonreduced $R$.

Put $S=Z(G{)}_s$ as supplied by [F2]. It is characteristic in $Z(G)$, hence normal in $G$, and central. Let $N$ be the inverse image in $G$ of $Z(G\mathrm{/}S{)}_s$. The two normal subgroups $S\subseteq N$ have $S$ central and both $S$ and $N\mathrm{/}S$ of multiplicative type. By [F3], $N$ is central and of multiplicative type. Maximality in $Z(G)$ then gives $N=S$, so $Z(G\mathrm{/}S{)}_s=1$. By [F2] the centre of $G\mathrm{/}S$ is unipotent. [F2, F3, F5].

A nilpotent affine group with unipotent centre is unipotent, as we now prove by induction on its finite nilpotence class. In the commutative case it is its own centre. Otherwise write $Z=Z(G)$ and $Q=G\mathrm{/}Z$, and let $N$ be the inverse image of $M=Z(Q{)}_s$. For every $R$ and $g\in G(R)$, the commutator $n\mapsto [g,n]$ has values in the central group $Z_R$, since $N\mathrm{/}Z$ is central in $Q$. It is a group homomorphism, is trivial on $Z_R$, and therefore descends to a homomorphism $M_R\to Z_R$. Step 1.1 makes it zero. Thus $N$ is central in $G$, so $N\subseteq Z$ and $M=1$. By [F2] the centre of $Q$ is unipotent; its nilpotence class is smaller, so induction makes $Q$ unipotent. As $Z$ is unipotent, the extension $G$ is unipotent by [F1]. Applying this conclusion to $G\mathrm{/}S$ from step 1.2 proves that $G\mathrm{/}S$ is unipotent. The induction is on class, so it also covers finite and infinitesimal groups whose centres need not lower dimension. [F1, F2, F5, step 1.1, step 1.2].

Every multiplicative-type subgroup $M\subseteq G$ maps trivially to the unipotent group $G\mathrm{/}S$ by step 1.1, and therefore lies in $S$. This proves the asserted largest-subgroup property. It also proves characteristicity as a group-scheme property: for any $R$ and any group automorphism of $G_R$, its composite $S_R\to (G\mathrm{/}S{)}_R$ is zero by step 1.1; the inverse automorphism supplies the reverse inclusion. Thus $S$ is preserved over every base algebra. [step 1.1, step 2.1].

Suppose $G$ becomes trigonalizable over a separable extension. By [F4] there is a normal unipotent subgroup $U=G_u$ with $G\mathrm{/}U$ of multiplicative type. Since $S\cap U=1$, normality implies that $S$ and $U$ commute. The product map $S\times U\to G$ is a homomorphism with trivial kernel; its image is normal, and its quotient is both a quotient of the unipotent group $G\mathrm{/}S$ and a quotient of the multiplicative-type group $G\mathrm{/}U$. This quotient is trivial by step 1.1. Hence the product map is an isomorphism. For smooth connected $G$ over a perfect field, [F4] applies, and the two product factors are smooth and connected, so $S$ is a torus. [F1, F4, F5, step 2.1, step 3.1].

For smooth $G$ over arbitrary $k$, pass to an algebraic closure. The formation of the centre commutes with field extension, as does its multiplicative-type factor by [F2]. Step 4.1 over this perfect field shows that $S_{k^{\mathrm{a}}}$ is a torus. Thus $S$ is a torus over $k$. Over a perfect field step 4.1 gives the stated decomposition $G=U\times T$ with $U$ smooth connected unipotent and $T$ a torus. Conversely, when $U$ is smooth and connected, such a product is smooth connected nilpotent: a central normal series for $U$, obtained from its upper-unitriangular representation, together with the central factor $T$ gives a central series for the product. [F1, F2, step 4.1].

If a torus $T$ properly contains $S$, its commutativity gives $T\subseteq C_G(S)$ and its nontrivial torus quotient $T\mathrm{/}S\subseteq C_G(S)\mathrm{/}S$. Conversely let a nontrivial torus $D$ lie in this quotient and let $P$ be its inverse image. The exact sequence $1\to S\to P\to D\to 1$ has smooth connected kernel and quotient, so $P$ is smooth and connected. The subgroup $S$ is central in $P$, since $P\subseteq C_G(S)$; apply [F3] to the action of connected $P$ on itself to see that $P$ is commutative and of multiplicative type. Smoothness and connectedness then make $P$ a torus. Since $D\mathrm{\ne }1$, it properly contains $S$, proving both directions of (b). No commutativity of the whole centralizer is required.