Rosenlicht dichotomy for smooth connected algebraic groups

Suppose $G$ is nonproper and choose its proper normal completion $X$ from [F1]. Its nonempty boundary is defined by a coherent ideal sheaf $I$. Blow up $I$, then normalize, obtaining a proper normal integral modification still containing $G$. Here is the needed boundary construction: on an affine chart with $I=(a_1,\dots ,a_r)$ the blowup is $\mathrm{Proj}{⨁}_{n\ge 0}{I}^n$, covered by the rings $R[I\mathrm{/}{a}_i]$; these agree after localization, and on each chart $I$ is the principal ideal $(a_i)$. The Proj is a closed subscheme of ${\mathbb{P}}_R^{r-1}$ and hence proper over that chart by [F2]. Its map is an isomorphism where $I=R$. The invertible nonzero ideal $I\mathcal{O}$ defines precisely the inverse image of the boundary; it remains invertible and nonzero after finite normalization. That inverse image is nonempty, since the proper birational map is surjective (its closed image contains the dense $G$). On a normal affine chart each minimal prime over its nonzero principal boundary equation has height one, so choose a boundary prime divisor $E$. Replace $X$ by this modified completion.

Left translation on $G$ induces a faithful rational action $\alpha :G\times X⇢X$. It is defined at the generic point of $G\times E$. To see this without assuming the whole product normal, take an affine chart $\mathrm{Spec}B$ of $X$ meeting ${\eta }_E$ and an affine chart $\mathrm{Spec}A$ of $G$. Localize $A{\otimes }_{k}B$ at the prime defining $G\times E$. It is a Noetherian local domain, by the affine specialization argument proved in Composition at points in the domain of a rational group action. After inverting $B\setminus {\mathfrak{p}}_E$ its maximal ideal is generated by the uniformizer of $B_{{\mathfrak{p}}_E}$; its quotient is the field $k(G\times E)$. It is therefore a DVR by [F2], and the valuative criterion extends $\alpha$ there. Images of finitely many affine target generators then extend to a neighbourhood of this generic point. This restriction cannot dominate $X$: otherwise its general value lies in $G$, where inverse translation is defined; [F3] would give $g^{-1}\cdot (g\cdot e)=e\in G$ for general $e\in E$, contradicting that $E$ is boundary.

Normalize $G\times X$. This is a finite modification, an isomorphism near the generic point of $G\times E$ because that local ring is a DVR. Apply [F3] to the lifted dominant map to $X$ and the divisor above $G\times E$. Its restriction is nondominant by step 2.1, so a proper normal modification $\varphi :X^{\mathrm{\prime }}\to X$ makes its image a prime divisor $D\subset X^{\mathrm{\prime }}$. The strict transform $E^{\mathrm{\prime }}$ of $E$ is birational to $E$: a proper birational modification of a normal variety is an isomorphism near a height-one generic point for this graph construction, since at its DVR one rescales the finitely many projective coordinates by their minimum valuation, making one a unit; the graph is then a morphism there, and finite normalization leaves that normal open unchanged. Transfer $\alpha$ birationally to $X^{\mathrm{\prime }}$. The restriction $G\times E^{\mathrm{\prime }}⇢X^{\mathrm{\prime }}$ dominates $D$.

The transferred action is defined at the generic point of $G\times D$ by the same DVR argument as step 2.1. For general $(g,h,y)\in G\times G\times E^{\mathrm{\prime }}$ put $x=h\cdot y$. Whenever the values are defined, [F3] gives $g\cdot x=(gh)\cdot y\in D$. The map $(g,h,y)\mapsto (g,h\cdot y)$ dominates $G\times D$, so the action restricts to a rational map $G\times D⇢D$. Its generic inverse is $(g,x)\mapsto (g^{-1},g\cdot x)$, and the group law restricts as a rational identity. This yields birational transformations of $D$ on a symmetric dense open $W$ of $G$: the inverse identity on $X^{\mathrm{\prime }}$ restricts wherever both successive values are defined, so the maps for $a$ and $a^{-1}$ are inverse on $D$. When $a,b,ab\in W$, [F3] gives $\rho (a)\rho (b)=\rho (ab)$. Extend to each $g$ by choosing $h$ with $h,gh\in W$ and setting $\rho (g)=\rho (gh)\rho (h{)}^{-1}$. This is independent of $h$: for another choice $h^{\mathrm{\prime }}$, choose $t$ in the intersection of the dense opens requiring $t,gt,h^{-1}t,(h^{\mathrm{\prime }}{)}^{-1}t\in W$. The partial identities $\rho (gh)\rho (h^{-1}t)=\rho (gt)$ and $\rho (h)\rho (h^{-1}t)=\rho (t)$ identify the expression with $\rho (gt)\rho (t{)}^{-1}$, and the same holds for $h^{\mathrm{\prime }}$. To verify the group law for arbitrary $g_1,g_2$, choose $t$ with $t,g_{2}t,g_1{g}_{2}t\in W$. Using these choices in the defining formula gives $\rho (g_1)\rho (g_2)=\rho (g_1{g}_2)$. Thus the restriction is a rational action of $G$ on $D$. Thus $X^{\mathrm{\prime }}$ has a stable boundary divisor for its rational translation action. No regular action on the complete model is asserted.

Let $U$ be the action domain in $G\times X^{\mathrm{\prime }}$. Its intersection with $G\times D$ contains a dense open. The birational involution $(g,x)\mapsto (g^{-1},g\cdot x)$ on $G\times D$ shows that the subset on which both $g\cdot x$ and $g^{-1}\cdot (g\cdot x)$ are defined is open dense. By [F5] choose a rational point $(g_0,x_0)$ in it. Its fibre over $x_0$ is a nonempty open $V\subset G$ containing $g_0$. Consider the morphism $\beta :V\to D$, $g\mapsto g\cdot x_0$, and put $F={\beta }^{-1}(\beta (g_0))$ with reduced structure. Applying [F4] to its image closure, every component of $F$ has dimension at least $\dim G-\dim D=1$. Choose a component $C$ containing $g_0$. For each $g\in C(k)$, [F3] gives $(g_0^{-1}g)\cdot x_0=x_0$, since both $g\cdot x_0$ and $g_0^{-1}\cdot (g_0\cdot x_0)$ are defined. It also gives $(g^{-1}{g}_0)\cdot x_0=x_0$, using $g^{-1}\cdot (g\cdot x_0)$, which is defined by the choice of $V$. Hence both $S=g_0^{-1}C$ and $S^{-1}$ fix $x_0$, with total action defined there.

We prove that the subgroup generated by $S(k)$ is closed; it is not enough merely to take an abstract subgroup. The constructible set $T=S{S}^{-1}$ is irreducible, symmetric, contains the identity, and contains $S$. The closures $H_n=\overline{T^n}$ are irreducible and increasing. Their dimensions are bounded by $\dim G$, so for some $n$ they stabilize: once $H_n=H_{n+1}$, multiplication by $T$ preserves that closure, hence all subsequent closures equal $H=H_n$. Density of $T^n\times T^n$ in $H\times H$ gives $HH\subset H$; symmetry gives $H^{-1}=H$. With reduced structure $H$ is a closed group scheme: the multiplication and inverse maps factor through its defining ideals because they vanish on all closed points of the reduced products. By [F4], $T^n$ contains a nonempty open $O$ of $H$. For every $h\in H(k)$ the two nonempty opens $O$ and $hO$ intersect, so $h=a{b}^{-1}$ with $a,b\in O$. Thus $H(k)$ is exactly the abstract subgroup generated by $S(k)$. It is irreducible and has positive dimension because it contains $S$. By [F5], $H$ has a smooth point; translating it to every closed point makes the entire $H$ smooth, since a nonempty closed nonsmooth locus would have a closed point.

Every finite product of elements of $S(k)$ and $S(k{)}^{-1}$ fixes $x_0$ with its total action defined, by repeated application of [F3]. Step 6.1 therefore shows that all of $H(k)$ does so. The regular action domain contains $H\times \{x_0\}$, and its restriction there is the constant morphism, since $H$ is reduced and coordinate differences vanish at all closed points. Restricting the rational action to $H$ is scheme-faithful: via the birational identification of $X^{\mathrm{\prime }}$ with $G$ it is ordinary left translation. Equality with the identity rational transformation for a test-scheme point of $H$ becomes equality of left translation and projection on a schematically dense open of that base change of $G$; the two regular morphisms then agree everywhere, by coordinate localization and integrality of $G$. Evaluating at the identity of $G$ shows that the group point is the identity. Thus [F6] makes $H$ affine. It is a smooth connected closed subgroup of positive dimension, proving the nonproper alternative.

If $G$ is proper, it is an abelian variety by [F1]. An affine closed smooth connected subgroup would also be proper and geometrically integral, so [F2] makes it a point. The alternatives are therefore exclusive. For a disconnected smooth group, its finitely many open and closed components are translates of the identity component, so it is proper precisely when that component is proper. Apply the connected result to that component if $G$ is nonproper. AC is inherited from [F1]–[F6].