Journal of Lie Theory Volume 10 (2000) 345–357 ◯^{C}2000 Heldermann Verlag

On the stability of subgroup actions

on certain quasihomogeneous G-varieties

Ivan Arzhantsev ^{∗}

Communicated by E. B. Vinberg

Abstract. Let G be a simple algebraic group, U be a maximal unipotent subgroup of G, and X be the affine G-variety corresponding to the algebra of regular functions on the homogeneous space G/U. We classify all subgroups H of G such that the restricted action H : X is stable, and that H is either a simple or a semisimple irreducible subgroup of G.

Key words: reductive algebraic groups, homogeneous spaces, stable actions. AMS 1991 Subject Classification: primary 14L30; secondary 14M17, 17B10.

1. Introduction

Let G be a reductive algebraic group acting regularly on an affine algebraic variety X. The ground field K is supposed to be an algebraically closed field of characteristic zero. Let us recall that the action G : X is stable if there exists a non-empty open G-invariant subset X_{0} ⊂ X such that any G-orbit in X_{0} is closed in X. For more details on stable actions see [11], [12], [8], [15]. Stable actions, in some sense, are the best actions from the point of view of invariant theory and that is why it is important to know whether an action is stable or not. In the recent work [15] E. B. Vinberg obtained a sufficient condition for an action G : X to be stable in terms of some weight semigroups assigned to the action (Theorem 3.2). The present paper can be considered as an application of this result.

Suppose that H and F are reductive subgroups of G and consider the H - action on the affine homogeneous space G/F by left multiplication. A well-known result due to D. Luna [8] states that the action H : G/F is stable. It is natural to consider the same action for a non-reductive subgroup F. But the problem is that the homogeneous space G/F is not affine. Hovewer if G/F is quasi-affine and X is an affine G-equivariant embedding of G/F then the question about stability for the action H : X makes sense.

ISSN 0949–5932 / $2.50 ◯^{C} Heldermann Verlag

∗ This work was supported by the European Community Network “Algebraic groups”

ERDCHRXCT 940697 and by INTAS-OPEN-97-1570.

Let U be a maximal unipotent subgroup in G and let X = \overline{G/U} = Spec K[G/U] be the affine G-variety corresponding to the algebra of regular functions on the homogeneous space G/U. The variety \overline{G/U} is the smallest affine embedding of G/U. In [15], a criterion for the action H : \overline{G/U} to be stable is given. Namely, this action is stable if and only if the rank semigroup Ξ(G/H) contains a strictly dominant weight. E. B. Vinberg formulated the problem of classifying of all connected reductive subgroups H in G with this property.

We say that a reductive subgroup H is large in G if the action H : \overline{G/U} is not stable. As a motivation for this term let us note that if the action H : \overline{G/U} is stable then the action H_{1} : \overline{G/U} is also stable for any reductive subgroup H_{1} ⊂ H, see Corollary 5.4. Some equivalent conditions for a reductive subgroup to be large are given (Theorem 5.1) in terms of the co-isotropy representation of the pair (G, H). As a corollary, one obtains that a reductive subgroup H is large in G if and only if the maximal semisimple subgroup of the identity component H^{0} is large in G. Here we make use of results from [10]. So the classification problem reduces to the case when H is connected and semisimple. If G is a simple group and H is either a simple or a semisimple irreducible subgroup of G we give a complete classification of large subgroups, see Tables 1-3. These results show that there are not too many large subgroups and “in general” the action H : \overline{G/U} is stable.

The author is grateful to E. B. Vinberg for setting the problem and to M. Brion and D. A. Timashev for useful discussions.

This paper was written during a stay at Institut Fourier (Saint Martin d’Hères, France). The author wishes to thank this institution and especially M. Brion for invitation and hospitality.

2. Notations

We consider algebraic varieties and algebraic groups over an algebraically closed field K of characteristic zero. All topological terms refer to the Zariski topology.

G is a connected reductive group;

F^{0} is the identity component of an algebraic group F; T ⊂ B are a maximal torus and a Borel subgroup in G; U is the maximal unipotent subgroup in B; Ξ(G)_{+} is the semigroup of all dominant weights of G; V_{λ} is the irreducible G-module with highest weight λ; V_{λ^{∗}} = V_{λ}^{∗} is the dual module to V_{λ}; V^{F} is the subspace of F -fixed vectors in a F -module V ; {ω_{1}, ω_{2}, …, ω_{r}} is the set of all fundamental weights of a semisimple group G, r = rk G. For a subset {λ_{1}, …, λ_{k}} in a semigroup Ξ we denote by < λ_{1}, …, λ_{k} > the smallest subsemigroup with zero containing λ_{1}, …, λ_{k}.

3. Preliminaries

1) Let X be an affine G-variety. The decomposition of the algebra of regular functions

K[X] = ⨁_{λ∈Ξ(X)} K[X]_{λ},

where λ ∈ Ξ(G)_{+} and K[X]_{λ} is the sum of all irreducible G-submodules in K[X] isomorphic to V_{λ}, is called the isotypic decomposition of the algebra K[X]. Here Ξ(X) is the subset in Ξ(G)_{+} consisting of all dominant weights such that K[X]_{λ} ≠ {0}. This subset is a subsemigroup in Ξ(G)_{+}, see [13]. The semigroup Ξ(X) is called the rank semigroup of a G-variety X. In particular, if H is a reductive subgroup of G then we get the rank semigroup Ξ(G/H) for the affine homogeneous space G/H. It follows from Frobenius reciprocity that

Ξ(G/H) = {λ ∈ Ξ(G)_{+} | V_{λ^{∗}}^{H} ≠ 0 }.

By definition, the rank of an affine G-variety X is the rank of Ξ(X) as of a subsemigroup in a free abelian group.

A reductive subgroup H ⊂ G is said to be spherical if any isotypic component of the G-algebra K[G/H] is an irreducible G-module, or, equivalently, the restricted B -action on G/H is quasihomogeneous, see [6, III. 3. 6 ].

2) Let us define a linear representation of H corresponding to the pair (G, H), which will play an important role in the sequel. Denote by 𝔤 and 𝔥 the Lie algebras of the groups G and H respectively. One has the natural H - representation in the quotient space 𝔤/𝔥. Let 𝔪 be the dual space to 𝔤/𝔥.

Definition 3.1. The linear representation (H : 𝔪) is said to be the co-isotropy representation for the pair (G, H).

The H -module 𝔪 can be identified with the H -module 𝔥^{⊥} = {x ∈ 𝔤 | (x, 𝔥) = 0}, where ( , ) is a G-invariant non-degenerate scalar product on the Lie algebra 𝔤.

In [10], the close connections between the rank semigroup Ξ(G/H) and the stabilizer of general position (the s.g.p.) S for the co-isotropy representation are investigated.

3) Now let us recall Theorem 10 from [15].

Theorem 3.2. Let G : X be an action of a connected reductive group on an affine variety and let H be a reductive subgroup of G. Assume that

Ξ(X) − Ξ(G/H) = { λ − μ | λ ∈ Ξ(X), μ ∈ Ξ(G/H) }

is a group. Then the restricted action H : X is stable.

Note that, for two semigroups Ξ_{1}, Ξ_{2} ⊂ ℤ^{n} ⊂ ℝ^{n}, the condition “Ξ_{1} − Ξ_{2} is a group” is equivalent to each of the following ones:

1) Ξ_{1} and Ξ_{2} are not separated by a hyperplane in their linear span;

2) the convex cones spanned by Ξ_{1} and Ξ_{2} have a common relatively interior point.

Remark 3.3. We say that a point λ ∈ Ξ is a relatively interior point of the cone C(Ξ) spanned by Ξ if λ belongs to the interior of the cone C(Ξ) in the subspace spanned by Ξ.

4. Special actions

An action G : X is called special if the stabilizer of any point in X contains a maximal unipotent subgroup of the group G. In terms of isotypic decomposition it turns out that the action G : X is special if and only if

K[X]_{λ}K[X]_{μ} ⊂ K[X]_{λ+μ} for any λ, μ ∈ Ξ(X),

see [13, Theorem 5]. In particular, any action of a torus is special.

A G-variety X is said to be a S -variety if the action G : X is special and G has a dense orbit in X, see [14].

Consider the irreducible G-modules V_{λ_{1}} , V_{λ_{2}} ,…, V_{λ_{k}} with highest weights λ_{1}, λ_{2}, …, λ_{k} ∈ Ξ(G)_{+} and the highest weight vectors v_{λ_{1}} ∈ V_{λ_{1}} , v_{λ_{2}} ∈ V_{λ_{2}} ,…, v_{λ_{k}} ∈ V_{λ_{k}}. Let X(λ_{1}, λ_{2}, …, λ_{k}) be the closure of the G-orbit of the vector v_{λ_{1}} + v_{λ_{2}} + … + v_{λ_{k}} in the G-module V_{λ_{1}} ⊕ V_{λ_{2}} ⊕ … ⊕ V_{λ_{k}}. This variety is a S -variety.

It is shown in [14] that any S -variety is isomorphic to a variety X(λ_{1}, λ_{2}, …, λ_{k}). If k = 1 then the variety X(λ) is the closure of the highest vector orbit and is called a HV -variety.

Let H be a reductive subgroup in G. We are interested in stability of the restricted action H : X(λ_{1}, λ_{2}, …, λ_{k}). Here the semigroup Ξ(X) is < λ_{1}^{∗}, λ_{2}^{∗}, …, λ_{k}^{∗} >, see [14].

Proposition 4.1. [15, Proposition 7] Let G : X be a special action. If the restricted action H : X is stable then Ξ(X) − (Ξ(X) ∩ Ξ(G/H)) is a group.

If the cones spanned by < λ_{1}, λ_{2}, …, λ_{k} > and Ξ(G/H) have a common relatively interior point then the action H : X(λ_{1}, λ_{2}, …, λ_{k}) is stable (see Theorem 3.2). If the cone spanned by Ξ(G/H) does not intersect the relative interior of the cone spanned by < λ_{1}, λ_{2}, …, λ_{k} > then the action H : X(λ_{1}, λ_{2}, …, λ_{k}) is not stable (see Proposition 4.1). But these two cases do not exhaust all possibilities.

A subgroup H ⊂ G is said to be symmetric if there exists an involutive automorphism σ of the group G such that H = G^{σ}. It is proved in [17] that the action H : X(λ) for a symmetric subgroup H is stable if and only if < λ > ∩ Ξ(G/H) ≠ {0}. Moreover, in this case the action is either quasihomogeneous or stable with one-parameter family of closed orbits of maximal dimension.

This nice alternative does not hold for a non-symmetric spherical subgroup of G. As an example, one can consider the natural action SL(n) : K^{n} and restrict this action to SL(n − 1) ⊂ SL(n).

Problem 4.2. Let H be a reductive spherical subgroup in G. When is the restricted action H : X(λ_{1}, λ_{2}, …, λ_{k}) stable ?

There is one case, where Theorem 3.2 and Proposition 4.1 give a criterion for stability. Namely, suppose that G is semisimple and set X = X(ω_{1}, ω_{2}, …, ω_{r}). The dense G-orbit in X is isomorphic to G/U and X can be considered as the smallest affine embedding of the quasiaffine homogeneous space G/U. This embedding plays an important role in representation theory. Sometimes it is denoted by \overline{G/U}. Recall that a dominant weight is said to be strictly dominant if it is not orthogonal to any simple root.

Here Ξ(X) = Ξ(G)_{+} and we have

Proposition 4.3. [15, Corollary 1] The action H : X(ω_{1}, ω_{2}, …, ω_{r}) is stable if and only if the semigroup Ξ(G/H) contains a strictly dominant weight.

In the next section we study subgroups with this property.

Proposition 4.4. Let H be a reductive spherical subgroup in a semisimple group G. Suppose that λ_{1}, λ_{2}. . . . .λ_{k} are dominant weights and the intersection A = Ξ(G/H)∩ < λ_{1}, λ_{2}, …, λ_{k} > is a free semigroup. Then for X = X(λ_{1}, λ_{2}, …, λ_{k}) the algebra of invariants K[X]^{H} is free.

Proof. We have K[X] = ⊕K[X]_{λ}, dim K[X]_{λ}^{H} ≤ 1, and dim K[X]_{λ}^{H} = 1 iff λ ∈ A. The product fg of two non-zero H -invariants f ∈ V_{λ}^{H} and g ∈ V_{μ}^{H} is a non-zero element of K[X]_{λ+μ}^{H}. Hence the algebra K[X]^{H} is the semigroup algebra for the semigroup A.

Corollary 4.5. Let H be a connected semisimple spherical subgroup in a simply connected semisimple group G. Suppose that X = X(ω_{1}, ω_{2}, …, ω_{r}). Then the algebra K[X]^{H} is free.

Proof. In this case A = Ξ(G/H) and this is a free semigroup, see [10].

5. Main results

Let us propose some characterizations of the class of reductive subgroups we are interested in. In this section we suppose G to be a connected semisimple group and H to be a reductive subgroup of G. Set X = X(ω_{1}, ω_{2}, …, ω_{r}). Say that a G-invariant subvariety Z of a G-module V is essential if it generates the vector space V .

Theorem 5.1. The following properties are equivalent:

(1) The action H : X is not stable;

(2) There exists an essential subvariety Z in V_{λ_{1}} ⊕ … ⊕ V_{λ_{r}} ⊕ W, where λ_{i} = n_{i}ω_{i}, n_{i} > 0 and W is a G-module, such that the restricted action H : Z is not stable;

(3) The semigroup Ξ(G/H) is contained in a proper face of the positive Weyl chamber;

(4) The stabilizer of general position S for the co-isotropy representation 𝔪 of the pair (G, H) has positive semisimple rank; (5) gHg^{−1} ∩ U ≠ {e} for any g ∈ G. If, in addition, H is a spherical semisimple subgroup in G then conditions

(1)-(5) are equivalent to (6) dim H + rk(G/H) > dim B.

Definition 5.2. A reductive subgroup H of G is called large if it satisfies the conditions (1)-(5).

Remark 5.3. H is large if and only if its maximal connected semisimple subgroup is large. This follows from the property (4). In the sequel, we can suppose H to be connected and semisimple.

Corollary 5.4. Let H_{2} ⊂ H_{1} ⊂ G be a chain of reductive groups.

1) If either H_{2} is not large in H_{1} or H_{1} is not large in G then H_{2} is not large in G;

2) If H_{2} is large in G then H_{1} is large in G.

Proof. 1) follows from statement (4) of Theorem 5.1 and 2) follows from statement (3).

This corollary can be considered as a motivation for the term “large”.

Proof of Theorem 5.1. (1) ⇔ (3) follows from Proposition 4.3.

(1) ⇒ (2). It follows from the construction that X is an essential subvariety in V_{ω_{1}} ⊕ … ⊕ V_{ω_{r}}.

(2) ⇒ (3). For an essential subvariety Z the semigroup Ξ(Z) contains a non-zero multiple of any fundamental weight. Suppose that the action H : Z is not stable. If Ξ(G/H) contains a strictly dominant weight then Ξ(Z) and Ξ(G/H) have a common relatively interior point. This contradicts Theorem 3.2.

(3) ⇔ (4) follows from results of [10]. Namely, S^{0} is a regular reductive subgroup in G [10, Th.1 and Th.3]. The semigroup Ξ(G/H) is contained in a proper face of the positive Weyl chamber if and only if there is a simple co-root that is orthogonal to all elements of Ξ(G/H). This is the case iff the corresponding SL_{2}-triple is contained in S [10, 5.1].

(1) ⇔ (5). If the action is stable then the stabilizer of general position is reductive. But this stabilizer is gHg^{−1} ∩ U for some g ∈ G, so it is a unipotent subgroup and that is why it is trivial. Conversely, if the stabilizer of general position is trivial and H is semisimple, we can apply the Popov criterion for stability [11]. Here the variety X is factorial, see [14]. It is obvious that (5) holds iff it holds for the maximal connected semisimple subgroup of H. So Remark 5.3 concludes the proof.

(1) ⇔ (6). It is well known that H -invariants separate generic H -orbits on a factorial variety. So the action H : X is stable iff the dimension of the generic fiber for the quotient morphism X → X//H is equal to dim H. This means that dim X = dim H + dim X//H. We can assume without loss of generality that G is simply connected. Then dim X = dim B and dim X//H = rk(G/H), see the proof of Proposition 4.4 and Corollary 4.5.

6. The classification

In order to classify all large subgroups we need just to classify all connected semisimple large subgroups, see Remark 5.3. In this section such a classification is obtained in cases, where G is simple and H is simple or semisimple and irreducible. The latter condition means that the minimal representation of G restricted to H is irreducible. For exceptional simple groups we list all connected semisimple large subgroups.

Theorem 6.1. 1) All connected simple irreducible proper large subgroups of classical groups are listed in Table 1;

2) All connected simple reducible large subgroups of classical groups are listed in Table 2;

3) All irreducible large semisimple subgroups of simple groups are simple;

4) All connected semisimple proper large subgroups of exceptional groups are listed in Table 3.

Comments. We use notation and numeration of simple roots as in [9]. Let π_{1}, …, π_{k} be the fundamental weights of H and R(ϕ) be the irreducible H - module with highest weight ϕ. Denote by I a one-dimensional H -module and by m R(ϕ) the module R(ϕ) ⊕ … ⊕ R(ϕ) (m times). Let R(ϕ)R′(ψ) be the tensor product of representations of two simple groups, where ′ corresponds to the second factor.

The column “Embedding” contains the H -decomposition of the first fundamental representation of the group G. In the column “𝔪” the co-isotropy representation for the pair (G, H) is indicated. In the column “s.g.p.(H : 𝔪)” the Lie algebra of s.g.p. is given. In the last column the rank rk(G/H) is indicated.

TABLE 1

G  H  embedding 𝔪 s.g.p.(H : 𝔪) r

1 SL(2n), n > 1 Sp(2n)  R(π_{1}) R(π_{2})  n A_{1}  n − 1

2  SO(7)  G_{2}  R(π_{1}) R(π_{1})  A_{2}  1

3  SO(8)  Spin(7) R(π_{3}) R(π_{1})  D_{3}  1

Remark 6.2. There is a misprint in Table 1 from [7] concerning the rank semigroup of the spherical pair (E_{6}, SL(6) × SL(2)). The rank semigroup here is generated by ω_{1} + ω_{5}, ω_{2} + ω_{4}, 2ω_{3} and 2ω_{6}. To show this one has to note that this pair is symmetric (the involution is defined by the Weyl group element of maximal length) and the results of [17] can be applied. Hence SL(6) × SL(2) is not large in E_{6}.

TABLE 2

g  h  embedding  𝔪  s.g.p.(h : 𝔪)  r

1 sl(n)  sl(n),  r(π_{1}) ⊕ m i, m r(π_{1}) ⊕ m r(π_{n−1})⊕  a_{2n−n−1} 2n − 2n
n < 2n − 1 m = n − n  ⊕ m^{2} i

2 sp(2n) sp(2n),  r(π_{1}) ⊕ 2m i, 2m r(π_{1}) ⊕ m(2m + 1) i  c_{2n−n}  2n − 2n
n < 2n  m = n − n

3 so(n)  so(n),  r(π_{1}) ⊕ m i,  m r(π_{1}) ⊕ \frac{m(m−1)}{2} i  𝔰o(2n − n) n − n
n < 2n − 2 m = n − n

4 so(2n)  sl(n)  r(π_{1}) ⊕ r(π_{n−1})  r(π_{2}) ⊕ r(π_{n−2}) ⊕ i  [\frac{n}{2}] a_{1}  [\frac{n+1}{2}]

5 so(8)  g_{2}  r(π_{1}) ⊕ i  2 r(π_{1})  a_{1}  3

6 so(9)  spin(7)  r(π_{3}) ⊕ i  r(π_{1}) ⊕ r(π_{3})  a_{2}  2

7 so(10) spin(7)  r(π_{3}) ⊕ 2 i  r(π_{1}) ⊕ 2 r(π_{3}) ⊕ i  a_{1}  4

Remark 6.3. It is sufficient to find all minimal large subgroups in G (see Corollary 5.4) but for convenience of the reader we list all subgroups.

Proof of Theorem 6.1. We shall use the characterization (4) from Theorem 5.1. Our first goal is to find all pairs (G, H) such that the stabilizer of general position for the co-isotropy representation is not finite.

Let us say that an irreducible H -submodule W in the adjoint module Ad(G) is nice if the s.g.p. for the representation H : W is finite.

To check this condition, one can apply Elashvili’s Tables [3], [4], where all representations of simple groups and all irreducible representations of semisimple groups with infinite s.g.p. are listed.

If W is a nice submodule in Ad(G) then it is not isomorphic to a submodule of the adjoint module Ad(H) and W is contained in the co-isotropy module 𝔪. Thus H is not a large subgroup in G in this situaton.

TABLE 3

G  H  embedding  𝔪  s.g.p.(H : 𝔪) r

1 G_{2}  SL(3)  R(π_{1}) ⊕ R(π_{2}) ⊕ I  R(π_{1}) ⊕ R(π_{2})  A_{1}  1

2 F_{4}  Spin(9)  R(π_{1}) ⊕ R(π_{4}) ⊕ I  R(π_{4})  B_{3}  1

3 F_{4}  Spin(8)  R(π_{1}) ⊕ R(π_{3}) ⊕ R(π_{4}) ⊕ 2 I R(π_{1}) ⊕ R(π_{3}) ⊕ R(π_{4})  A_{2}  2

4 E_{6}  Spin(10)  R(π_{1}) ⊕ R(π_{5}) ⊕ I  R(π_{4}) ⊕ R(π_{5}) ⊕ I  A_{3}  3

5 E_{6}  F_{4}  R(π_{1}) ⊕ I  R(π_{1})  D_{4}  2

6 E_{6}  Spin(9)  R(π_{1}) ⊕ R(π_{4}) ⊕ 2 I  R(π_{1}) ⊕ 2 R(π_{4}) ⊕ I  A_{1}  5

7 E_{7}  E_{6}  R(π_{1}) ⊕ R(π_{5}) ⊕ 2 I  R(π_{1}) ⊕ R(π_{5}) ⊕ I  D_{4}  3

8 E_{7} Spin(12) × SL(2)  R(π_{1})R′(π_{1}) ⊕ R(π_{6})  R(π_{5})R′(π_{1})  3 A_{1}  4

9 E_{7}  Spin(12)  2 R(π_{1}) ⊕ R(π_{6})  2 R(π_{5}) ⊕ 3 I  3 A_{1}  4

10 E_{8}  E_{7} × SL(2)  R(π_{1})R′(π_{1}) ⊕ R(π_{6}) ⊕ R′(2π_{1})  R(π_{1})R′(π_{1})  D_{4}  4

11 E_{8}  E_{7}  2 R(π_{1}) ⊕ R(π_{6}) ⊕ 3 I  2 R(π_{1}) ⊕ 3 I  D_{4}  4

Otherwise we calculate explicitely the co-isotropy representation and find the s.g.p.. Here Table 5 from [9] is very useful.

Denote by V the first fundamental representation of the group G, V = R(ϕ_{1}) ⊕ … ⊕ R(ϕ_{k}) for H, and L a vector space with trivial H -action.

Case 1.1. G = SL(N) and H is a simple irreducible subgroup.

Here Ad (SL(N)) ≅ V ⊗ V^{∗} − I. By assumption, V = R(ϕ) for some highest weight ϕ of the group H. Then R(ϕ + ϕ^{∗}) is contained in Ad (SL(N)) and is a nice submodule with only one exception: H = Sp(2n) and V = R(π_{1}), see Table 1.

Case 1.2. G = Sp(2N) and H is a simple irreducible subgroup.

Here Ad (Sp(2N)) ≅ S^{2}V , V = R(ϕ), and R(2ϕ) is a nice submodule in Ad (Sp(2N)).

Case 1.3. G = SO(N), N > 6 and H is a simple irreducible subgroup. Here Ad (SO(N)) ≅ Λ^{2}V and if V = R(ϕ) and α is a simple root that

is not orthogonal to ϕ then R(2ϕ − α) is a H -submodule in Ad (SO(N)). It is possible to find a simple root α such that R(2ϕ − α) is a nice submodule in all cases except for:

1) H = Spin(7) and V = R(π_{3}), see Table 1.

2) H = G_{2} and V = R(π_{1}), see Table 1.

In order to avoid many cases, one can use the criterion for an irreducible representation of a simple group to be orthogonal, see [1].

Case 2.1. G = SL(N) and H is a simple reducible subgroup.

Set V = R(ϕ_{1}) ⊕ … ⊕ R(ϕ_{k}). The module Ad(SL(N)) contains the H - submodule R(ϕ_{1} + ϕ′_{1}), which is nice unless ϕ_{1} = π_{1}, H = SL(n) or Sp(2n). Assume ϕ_{1} = π_{1}.

For H = SL(n) if ϕ_{2} = π_{1} then 𝔪 contains the submodule 3 R(π_{1} + π_{n−1}) and the s.g.p. is finite. If ϕ_{2} = π_{n−1} then 𝔪 contains R(π_{1} + π_{n−1}) ⊕ R(2π_{1}) ⊕ R(2π_{n−1}) and the s.g.p. is finite. If V = R(π_{1}) ⊕ L then it is easy to compute 𝔪 as a H -module and all cases with an infinite s.g.p. are given in Table 2. In other cases 𝔪 contains a submodule R(π_{1} + ϕ_{2}) with finite s.g.p.

For H = Sp(2n) if ϕ_{2} = π_{1} then 𝔪 contains 3 R(2π_{1}) and if V = R(π_{1})⊕L then 𝔪 contains R(π_{2})⊕2 R(π_{1}). In other cases 𝔪 contains a submodule R(π_{1}+ϕ_{2}) with finite s.g.p.

Case 2.2. G = Sp(2N) and H is a simple reducible subgroup.

Analogous to the previous case.

Case 2.3. G = SO(N), N > 6 and H is a simple reducible subgroup. Here the submodule R(2ϕ_{1} − α) or the submodule R(ϕ_{1} + ϕ_{2}) is nice in the

majority of cases. In all remaining cases explicit computations allow us to find 𝔪.

For the classification of maximal semisimple irreducible subgroups in classical groups see [1].

Case 3.1. G = SL(N) and H = SL(n) ⊗ SL(m), nm = N.

Here V = R(π_{1})R′(π_{1}). Then R(π_{1} + π_{n−1})R′(π_{1} + π_{m−1}) ⊂ 𝔪 and this is a nice submodule.

Case 3.2. G = Sp(2N) and H = Sp(2n) ⊗ SO(m), N = 2nm, n ≥ 1, m ≥ 3, m ≠ 4 or n = 1, m = 4.

Here R(2π_{1})R′(2π_{1}) ⊂ 𝔪 is a nice submodule with some low-dimensional exceptions.

Case 3.3.1. G = SO(N) and H = Sp(2n) ⊗ Sp(2m), n ≥ 1, m ≥ 1, N = 4nm.

Case 3.3.2. G = SO(N) and H = SO(n) ⊗ SO(m), n ≥ 3, m ≥ 3, n, m ≠ 4, N = nm.

In these two cases the submodule R(2ϕ_{1} − α)R′(2ϕ_{2}) or the submodule R(2ϕ_{1})R′(2ϕ_{2} − α) in 𝔪 is nice with some low-dimensional exceptions. For example, for SL(2) × Sp(4) ⊂ SO(8) the Lie algebra of the s.g.p. is one-dimensional.

Let G be an exceptional simple group. The classification of all semisimple subgroups in G is obtained in [2]. Firstly maximal semisimple subgroups are considered. There are three types.

Type I. H is a maximal regular semisimple subgroup of maximal rank, see Table 5 in [5]. There exist 15 possibilities. The co-isotropy representation can be found with the help of Table from [16]. In four cases we obtain large subgroups, see 1, 2, 8 and 10 in Table 3.

Type II. H is a maximal regular semisimple subgroup of non-maximal rank (it is contained in a maximal reductive subgroup of maximal rank). There are two possibilities, see Table 6 from [5]. It is not difficult to compute 𝔪 here and to show that these subgroups are large, see 4 and 7 in Table 3.

Type III. H is a maximal L-subgroup, see [5, p. 207]. There are 14 possibilities (we do not consider subgroups of rank one as they cannot be large). The co-isotropy representations are given in [2], Tables 24 and 35. Using Elashvili’s Tables one can check that there is only one large subgroup, see 5 in Table 3.

Secondly we have to consider all maximal semisimple large (see Corollary 5.4) subgroups in cases 2, 4, 5, 7, 8 and 10 in Table 5. Case 2 leads to 3, cases 4 and 5 lead to 6, case 8 produces 9 and 10 produces 11. All other possibilities correspond to non large subgroups. To complete the classification, we have to consider maximal reducible semisimple (not simple) subgroups in spinor groups, but in these low-dimensional cases it can be checked that the s.g.p. is finite.

The consideration of maximal large subgroups in subgroups 3, 6, 9 and 11 gives no large subgroups in the initial group.

The proof is completed.

Corollary 6.4. (of the proof) For a simple irreducible subgroup H in a simple group G the following conditions are equivalent:

(1) H is a large subgroup in G;

(2) rk(G/H) < rk(G).

These conditions are not equivalent in all other cases. Namely, consider 1) SO(5) ⊂ SO(8), here rk(G/H) = 3;

2) GL(2) ⊂ SL_{3}, here rk(G/H) = 1;

3) SL(2) ⊗ Sp(4) ⊂ SO(8), here rk(G/H) = 3.

Here H is not a large subgroup. These cases are exactly the cases, where the identity component of the s.g.p.(H : 𝔪) is a torus.

7. Remarks

The previous results give a geometrical description of the H -action on the variety X = \overline{G/U}. Namely, suppose that a reductive subgroup H ⊂ G is not large. Then the generic fiber of the quotient morphism X → X//H is isomorphic to H. Moreover, if H is a spherical semisimple connected subgroup in a semisimple simply connected group G then X//H is an affine space (Corollary 4.5) and one can consider X as a total space for a multiparameter contraction of the action H : H by left multiplication. In particular, the quasi-affine homogeneous space G/U contains an open affine subset that fibers over (K^{∗})^{r} with the fiber H.

Example 7.1. Set G = SL(2n + 1) and H = Sp(2n) with the natural inclusion. Here Ξ(G/H) is Ξ(G)_{+} and the G-equivariant action of a maximal torus T ⊂ G on X defines an effective action T : X//H. This implies that G/U contains an open affine subset isomorphic to Sp(2n) × (K^{∗})^{2n}.

Finally, consider another consequence of Theorem 3.2.

Proposition 7.2. Let H be the stabilizer of general position for an action G : X of a reductive group G on an affine variety X. Suppose that H is reductive. Then for any reductive subgroup H_{1} ⊂ H the induced action H_{1} : X is stable.

Proof. By the Luna theorem [8], one has to consider only the case H_{1} = H. Moreover, one can suppose that the action G : X is quasihomogeneous. Then G/H is an open orbit in X and the groups of rational B -semi-invariants K(X)^{(B)} and K(G/H)^{(B)} coincide. Any rational B -semi-invariant is a quotient of two regular B -semi-invariants. Hence the semigroups Ξ(X) ⊂ Ξ(G/H) have the same rank and contain a common relatively interior point.

If H is not reductive then the statement does not hold. The action SL(n) : K^{n} and the restricted action SL(n − 1) : K^{n} provide a counter-example.

References

[1] Dynkin, E. B., Maximal subgroups of the classical groups, Tr. Mosk. Mat. O-va 1 (1952), 39–166 (in Russian); English trans.: Amer. Math. Soc. Transl., II. Ser. 6 (1957), 245–378.

[2] —, Semisimple subalgebras of semisimple Lie algebras, Mat. Sb., Nov. Ser. 30 (1952), 349–462 (in Russian); English trans.: Amer. Math. Soc. Transl., II. Ser. 6 (1957), 111–243.

[3] Elashvili, A. G., Canonical forms and stationary subalgebras of points of general position for simple linear Lie groups, Funktsional’nyi Analiz i ego Prilog. 6(1) (1972), 51–62 (in Russian); English Transl.: Func. Anal. and its Appl. 6(1) (1972), 44–53.

[4] —, Stationary subalgebras of points of the common state for irreducible linear Lie groups, Funktsional’nyi Analiz i ego Prilog. 6(2) (1972), 65– 78 (in Russian); English Transl.: Func. Anal. and its Appl. 6(2) (1972), 139–148.

[5] Gorbatsevich, V. V, A. L. Onishchik, and E. B. Vinberg, “Lie Groups and Lie Algebras III, ” Itogi Nauki i Tekhniki, Sovr. Probl. Mat., Fund. Naprav., vol. 41, VINITI, Moscow 1990 (in Russian); English Transl.: Berlin Heidelberg New York, Springer 1994.

[6] Kraft, H., “ Geometrische Methoden in der Invariantentheorie, ” Vieweg Verlag, Braunschweig 1985; Russian Transl.: Mir, Moscow 1987.

[7] Krämer, M., Sphärische Untergruppen in kompakten zusammenhängenden Liegruppen, Composit. Math. 38 (1979), 129–153.

[8] Luna, D., Sur les orbites fermées des groupes algébriques reductifs, Invent. Math. 16 (1972), 1–5.

[9] Onishchik A. L., and E. B. Vinberg, “Seminar on Lie groups and algebraic groups, ” Moscow: Nauka 1988 (in Russian); English Transl.:Lie groups and algebraic groups, Berlin Heidelberg New York, Springer 1990.

[10] Panyushev, D. I., Complexity and rank of homogeneous spaces, Geometriae Dedicata 34 (1990), 249–269.

[11] Popov, V. L., Stability criteria for an action of a semisimple group on a factorial manifold, Izv. Akd. Nauk SSSR, Ser. Mat. 34 (1970), 523–531 (in Russian); English Transl.: Math. USSR, Izv. 4 (1970), 527–535.

[12] —, On stability of an action of an algebraic group on an algebraic variety, Izv. Akd. Nauk SSSR, Ser. Mat. 36 (1972), 371–385 (in Russian); English Transl.: Math. USSR, Izv. 6 (1972), 367–379.

[13] —, Contractions of the actions of reductive algebraic groups, Mat. Sbornik 130 (1986), 310–334 (in Russian); English Transl.: Math. USSR-Sb. 58 (1987), 311–335.

[14] Popov, V. L., and E. B. Vinberg, On a class of quasihomogeneous affine varieties, Izv. Akad. Nauk SSSR, Ser. Mat. 36 (1972), 749–764 (in Russian); English Transl.: Math. USSR, Izv. 6 (1973), 743–758.

[15] Vinberg, E. B., On stability of actions of reductive algebraic groups, to appear in Proceedings of the Second International Algebraic Conference on Taiwan (January, 1997).

[16] —, The Weyl group of a graded Lie algebra, Izv. Akad. Nauk SSSR, Ser. Mat. 40 (1976), 488–526 (in Russian); English Transl.: Math. USSR, Izv. 10 (1977), 463–495.

[17] Vust, Th., Opération de groupes réductifs dans un type de cônes presque homogènes, Bull. Soc. math. France 102 (1974), 317–333.

Department of Mathematics Moscow State Pedagogical University Krasnoprudnaia 14, Moscow, Russia ivan@arjantse.mccme.ru

Received April 19, 1999 and in final form May 15, 1999