Journal of Lie Theory Volume 12 (2002) 357–368 \begin{matrix}c \\ ◯ \end{matrix} 2002 Heldermann Verlag

The Outer Automorphism Group of Normalizers of Maximal Tori in Connected Compact Lie Groups

Jean-François Hämmerli^{∗}

Communicated by A. Valette

Abstract. Let G be a connected compact Lie group, T a maximal torus of G, N = N_{G}(T) its normalizer and W = N/T the Weyl group of G. We show that the outer automorphism group of N canonically decomposes as a semidirect product, where the normal subgroup is given by the cohomology group H^{1}(W; T) and the complement subgroup is the outer automorphism group of G .

1. Introduction

The origin of the present work lies in a well-known theorem of Curtis, Wiederhold and Williams [3], saying that two connected compact Lie groups are isomorphic if and only if the normalizers of their maximal tori are isomorphic (see the paper by Osse [10] for a proof of the general, non-semisimple, case). Here, we are interested in the following natural question, formulated by Osse and closely related to this theorem:

What is the relationship between the automorphisms of the normalizer of a maximal torus and those of the corresponding connected compact Lie group?

Every automorphism of a connected compact Lie group preserves some maximal torus (see [4] or [2]), thus its normalizer, and yields, therefore, an automorphism of the normalizer by restriction. Conversely, can every automorphism of the normalizer be extended to an automorphism of the whole group? The following motivating example shows that the answer is no!

Example 1.1. Let G = SU(2m) with m ⩾ 2 and let T ⊂ G be a maximal torus. The normalizer N = N_{G}(T) of T fits into the extension T ↪ N ↠^{π} Σ_{2m} , where Σ_{2m} denotes the symmetric group on 2m letters. Then the map

ψ : N ⟶ N , n ⟼ (−𝟙)^{ϵ(π(n))}·n ,

ISSN 0949–5932 / $2.50 \begin{matrix}c \\ ◯ \end{matrix} Heldermann Verlag where −𝟙 = diag(−1, …, −1) ∈ SU(2m) and ϵ : Σ_{2m} → ℤ/2 = {0, 1} denotes the signature homomorphism, defines an automorphism of the normalizer. We investigate ψ for the case of the standard maximal torus consisting of the diagonal matrices of determinant 1 . By contradiction, suppose that ψ can be extended to SU(2m) . As it is trivial on the maximal torus, this extension is a conjugation by an element t ∈ T [2, Proposition 9, p. 30]. We now check what it implies on the maximal torus and on an element n ∈ N that exchanges the first two elements on the diagonal, i.e.

∗ Supported by the Japanese Society for the Promotion of Science.

n·diag(z_{1}, z_{2}, z_{3}, …, z_{2m})·n^{−1} = diag(z_{2}, z_{1}, z_{3}, …, z_{2m}) .

We have t·n·t^{−1} = ψ(n) = −n, which is equivalent to n^{−1}·t·n = n·t·n^{−1} = −t. As m ⩾ 2 , this is clearly in contradiction with how n is acting on T .

Our main theorem gives a complete description of the automorphisms of the normalizer that are not extendable, identifying them as a cohomology group of degree 1 , and answers the above question in terms of outer automorphism groups.

Theorem 1.2. Let G be a connected compact Lie group, T a maximal torus in G , and N the normalizer of T in G . Let W = N/T be the Weyl group of G . Then the outer automorphism group of the Lie group N canonically decomposes as the semidirect product

Out(N) ≅ H^{1}(W; T) ⋊ Out(G) ,

where the W -module T is endowed with the natural action induced by the extension T ↪ N ↠ W .

In section 2 we introduce the material on root diagrams that is needed for our proof of theorem 1.2. Section 3 deals with some properties of the automorphisms of N that induce the identity on T . It then recalls how the cohomology group H^{1}(W; T) can be identified as the subgroup of Out(N) consisting of elements that have a representative in Aut(N) inducing the identity on both T and W , as typically depicted in example 1.1. Finally section 4 is devoted to the proof of the main theorem. Once Out(G) has also been identified as a subgroup of Out(N) in proposition 4.1, it shows in particular that its action on H^{1}(W; T) , which is not a priori clear, can be interpreted as conjugation in Out(N) .

Remark 1.3. The question addressed in the present work has a natural formulation in the context of p-compact groups; we refer to [9] for related results and to [5] for an account concerning the importance of normalizers of maximal tori in the theory of p-compact groups.

2. Root diagrams

The notion of root diagram generalizes that of root system and allows to treat the case of connected compact Lie group that are not semisimple. Root diagrams are introduced and studied in Bourbaki [2, §4, No. 8-9] , where it is shown that the classification of connected compact Lie groups can be done in this purely algebraic setting. The idea of the slightly modified version of the definition of root diagram that we present here is due to Osse; its advantage is twofold: it is more tractable than the definition in [2], and it more clearly shows that a root diagram is a generalization of a root system. We will see that the two definitions are basically equivalent.

Definition 2.1. A (reduced) root diagram is a triple D = (V, M, Φ) satisfying:

1. V is a euclidean space, M is a lattice in it, and Φ is a finite subset of M \ {0};

2. for all α ∈ Φ and for all k ∈ ℤ, kα ∈ Φ if and only if k = ±1 ;

3. for all α ∈ Φ , the map

s_{α} : V ⟶ V , x ⟼ x − \frac{2(x , α)}{(α , α)} α ,

called reflection of vector α , leaves the set Φ globally invariant;

4. for all α ∈ Φ and x ∈ M , \frac{2(x , α)}{(α , α)} ∈ ℤ.

The group of automorphisms of V generated by the reflections s_{α} is called the Weyl group of D and is denoted by W (D) . The inner product (. , .) is obviously W (D)-invariant. Hence the set Φ is a reduced root system in the euclidean space ℝΦ , whose Weyl group is precisely W (D) .

The following lemma shows that the definition presented here is equivalent to Bourbaki’s, once, without loss of generality, a W (D)-invariant inner product has been fixed in the latter one.

Lemma 2.2. Let D = (V, M; Φ) be a root diagram. Consider the set

M_{o} = {x ∈ M : (x , α) = 0, ∀ α ∈ Φ} .

Then M_{o} is the subgroup M^{W (D)} of W (D)-invariants of M . Moreover it is a direct summand in M and the set M_{o} ∪ Φ spans the vector space V .

Proof. It is clear that M_{o} = M^{W (D)} .

Since M_{o} is a subgroup of the free abelian group M of rank n, it is itself a free abelian group of rank r ⩽ n. By the elementary divisors theorem, there exist bases {e_{1}, …, e_{n}} of M and {ϵ_{1}, …, ϵ_{r}} of M_{o} , and integers d_{1}, …, d_{r} such that ϵ_{j} = d_{j}·e_{j} , for all j = 1, …, r . Let j ∈ {1, …, r}; by definition of M_{o} , we have (ϵ_{j} , α) = 0 for all α ∈ Φ , hence the same holds for e_{j} , and therefore e_{j} ∈ M_{o} and d_{j} = 1 . This proves that M_{o} is a direct summand in M .

Let us now show that M_{o} ∪ Φ spans V . Let x ∈ M ; for all α ∈ Φ , we have s_{α}(x) ≡ x (mod ℤΦ) . As W = W (D) is generated by the s_{α}’s and ℤΦ is globally W -invariant, it implies w · x ≡ x (mod ℤΦ) for all w ∈ W . Define y = ∑_{w∈W} w·x ∈ M^{W} ; obviously, y ≡ |W |·x (mod ℤΦ) , and therefore x = (1/|W |)·y + z with y ∈ M_{o} and z ∈ ℤΦ .

The example that will interest us is the (covariant) root diagram of a connected compact Lie group G with respect to a fixed maximal torus T: it is given by D(G) = (LT, Γ(T), R^{∨}) , where Γ(T) is the integral lattice in the Lie algebra LT of the maximal torus, and R^{∨} is the set of coroots of G. The Weyl group W (D(G)) is canonically isomorphic to the Weyl group W of G. In this particular case, M_{o} is equal to the intersection Γ(T)∩ℨ(LG) of the integral lattice with the center of the Lie algebra LG of G, i.e. the non-semisimple part of the integral lattice.

Let us now define the notion of isomorphism of root diagrams (compare with [10, Def. 4.2]).

Definition 2.3. Let D = (V, M, Φ) and D′ = (V′, M′, Φ′) be root diagrams. An isomorphism of root diagrams between D and D′ is an isomorphism of vector spaces f : V ⟶ V′ (which is not necessarily an isometry) mapping M bijectively onto M′ , Φ bijectively onto Φ′ , and satisfying, for all x ∈ M and all α ∈ Φ , the compatibility condition

\frac{2(x , α)}{(α , α)} = \frac{2(f(x) , f(α))}{(f(α) , f(α))} .

As an immediate consequence of this definition, the inner product on V is more or less irrelevant. More precisely, two W (D)-invariant inner products on V yield two isomorphic root diagrams.

The automorphism group of a root diagram will be denoted by Aut(D) . We end up this section by recalling a key result on Aut(D) , which will be needed to prove theorem 1.2 (see [2, Prop. 18, p. 42]).

Proposition 2.4. (Bourbaki) For any connected compact Lie group G , there is a canonical isomorphism

Aut(D(G))/W (D(G)) ≅ Out(G) .

3. Automorphisms of N inducing the identity on T

We start by fixing some notations for the last two sections. The Lie algebra of a Lie group K will be denoted by LK and the differential at the unit element of a Lie group homomorphism φ : K_{1} → K_{2} by φ_{∗} : LK_{1} → LK_{2} . For an element g in K , we will write c_{g} for conjugation by g , i.e. c_{g}(x) = gxg^{−1} , for all x in K . We also assume the notations of theorem 1.2.

As T is the component of the identity of N , an automorphism ψ ∈ Aut(N) of the normalizer induces two automorphisms ψ_{|T} and ψ̄ fitting in the commutative diagram

[figure: T; ⊂; N; π; W; T; ≅ ψ_{|T}; ⊂; N; ≅ ψ; π; W; ≅ ψ̄]

The automorphism ψ also induces a vector space automorphism ψ_{∗} : LT → LT on the Lie algebra of the torus.

Before stating the first lemma in this section, let us recall that for a root α ∈ R , its associated coroot α^{∨} ∈ LT and reflection s_{α} , the Lie algebra of the torus decomposes as LT = ℝα^{∨} ⊕ H_{α} , where H_{α} denotes the hyperplane fixed by s_{α} , i.e. the hyperplane orthogonal to α^{∨} .

Lemma 3.1. The automorphism ψ̄ permutes the reflections in W . Moreover for s_{β} = ψ̄(s_{α}) , one has

\begin{matrix}(i) & s_{β} = ψ_{∗} ∘ s_{α} ∘ ψ_{∗}^{−1} ; \\ (ii) & ψ_{∗}(H_{α}) = H_{β}; \\ (iii) & ψ_{∗}(ℝα^{∨}) = ℝβ^{∨} . \end{matrix}

Proof. Let s_{α} ∈ W be a reflection, and let q_{α} ∈ N be an element such that π(q_{α}) = s_{α} . Let us denote w = ψ̄(s_{α}) = π(ψ(q_{α})) , and H = ψ_{∗}(H_{α}) . Let also Y = ψ_{∗}(X) ∈ H , with X ∈ H_{α} . The following calculation shows that w fixes H pointwise:

w·Y = (c_{ψ(q_{α})})_{∗}(Y)
= (ψ ∘ c_{q_{α}} ∘ ψ^{−1})_{∗}(Y)
= ψ_{∗} ∘ (c_{q_{α}})_{∗} ∘ ψ_{∗}^{−1}(Y)
= ψ_{∗} ∘ s_{α} ∘ ψ_{∗}^{−1}(Y)
= ψ_{∗} ∘ s_{α} ∘ ψ_{∗}^{−1} ∘ ψ_{∗}(X)
= ψ_{∗}(X)
= Y .

Now the subgroup of W that fixes H pointwise has the form ⟨s_{β}⟩ = {id, s_{β}} for some reflection s_{β} [7, Theorem 1.12 (d), p.22, and Proposition 1.14, p.24]. Since w is an element of order 2 , we can conclude that w = s_{β} . Moreover the previous calculation shows that (i) and (ii) are verified.

The last assertion follows from (i) and (ii): for all X ∈ LT we have

s_{β} ∘ ψ_{∗}(X) = ψ_{∗} ∘ s_{α}(X)
⟺ ψ_{∗}(X) − 2\frac{(ψ_{∗}(X) , β^{∨})}{(β^{∨} , β^{∨})} ·β^{∨} = ψ_{∗}(X − 2\frac{(X , α^{∨})}{(α^{∨} , α^{∨})} ·α^{∨})
= ψ_{∗}(X) − 2\frac{(X , α^{∨})}{(α^{∨} , α^{∨})} ·ψ_{∗}(α^{∨})
⟺  \frac{(ψ_{∗}(X) , β^{∨})}{(β^{∨} , β^{∨})} ·β^{∨} = \frac{(X , α^{∨})}{(α^{∨} , α^{∨})} ·ψ_{∗}(α^{∨}) .

Considering an element X ∈ LT \H_{α} , we have ψ_{∗}(X) ∈ LT \H_{β} , and the last equality shows that ψ_{∗}(α^{∨}) is a (non-zero) multiple of β^{∨} .

Corollary 3.2. If ψ_{|T} is the identity of T , then ψ̄ is the identity of W .

Proof. Let us write s_{β} = ψ̄(s_{α}) . As ψ_{|T} = id_{T} , we also have ψ_{∗} = id_{LT} . Therefore H_{β} = ψ_{∗}(H_{α}) = H_{α} , and thus s_{β} = s_{α} .

We end up this section by recalling a relationship between the automorphisms of an extension inducing the identity, on both the kernel and the quotient, and a group of cohomology, which will clarify the appearance of the term H^{1}(W; T) in theorem 1.2. This relationship was deduced in [11] by exploiting an exact sequence introduced by Wells [12]. Let A ↪ E ↠ Q be a group extension, where A is abelian. Let θ : Q → Aut(A) denote the natural action induced by the extension. Let also Aut(E, A, Q) be the subgroup consisting of elements ψ in Aut(E) that fix A pointwise and such that the induced automorphism ψ̄ : Q → Q is also the identity, i.e.

Aut(E, A, Q) = {ψ ∈ Aut(E) : ψ_{|A} = id_{A} and ψ̄ = id_{Q}} .

We summarize the results from [11, 12] that we need in the following theorem. Another approach can be found in [1, pp. 87-88].

Theorem 3.3. Let A ↪ E ↠ Q a group extension with A abelian.

(1) There is an identification

H^{1}(Q; A) = Aut(E, A, Q)/ Inn_{E}(A) ,

where Inn_{E}(A) = {ψ ∈ Aut(E, A, Q) : ψ = c_{a} , with a ∈ A} .

(2) Moreover if A is a maximal abelian normal subgroup of E (this is the case if and only if the restriction of θ to the center of Q is an injection), then the induced application

H^{1}(Q; A) ⟶ Out(E)

is injective.

Since T is self-centralizing in G, it is a maximal abelian normal subgroup of N . Thus this theorem says that the cohomology group H^{1}(W; T) can be seen as the subgroup of Out(N) given by

H^{1}(W; T) = Aut(N, T, W)/ Inn_{N}(T) ≅ Aut(N, T, W)/(T/Z(N)) ,

where Z(N) denotes the center of N .

Remark 3.4. Let s : W → N be a set-theoretic splitting of the canonical projection π : N ↠ W . Then the bijection between the set of normalized cocycles Z^{1}(W; T) = {f : W → T | x·f(y)f(xy)^{−1}f(x) = e} and Aut(N, T, W) is explicitly given by

f ⟼ (ψ : s(x)t ⟼ f(x)s(x)t) .

As W is finite, this ensures that the elements of Aut(N, T, W) really are Lie group automorphisms of N .

4. Proof of the main theorem

We will prove theorem 1.2 by showing that there exists a split short exact sequence

H^{1}(W; T) ↪ Out(N) ↠ Out(G) .

The first step will consist in showing that there is a canonical injection of the outer automorphism group Out(G) into Out(N) . Then, in the most important step, we will construct a surjective homomorphism Out(N) ↠ Out(G) , for which the injection in the first step is a splitting. Finally, we will identify the kernel of this surjection with the cohomology group H^{1}(W; T) . We subdivide these three steps in as many propositions.

Proposition 4.1. There is a canonical inclusion

i : Out(G) ↪ Out(N) .

of the outer automorphism group of G into that of N .

Proof. Let us consider Aut(G, T) = {φ ∈ Aut(G) : φ(T) = T } , the subgroup of automorphisms of G preserving T . As φ(N) = N_{G}(φ(T)) = N for all φ ∈ Aut(G, T) , we get a canonical homomorphism ψ : Aut(G, T) → Aut(N) ↠ Out(N) . On the other hand it is known [2, Proposition 18, p. 42] that

Aut(G, T)/(Aut(G, T) ∩ Inn(G)) ≅ Out(G) ,

where Aut(G, T) ∩ Inn(G) = {φ ∈ Aut(G) : ∃n ∈ N with φ = c_{n}} . Thus we get an induced map

Aut(G, T) \overline{^{ψ}} Out(N)

[figure: Out(G); ψ̄=i]

We conclude by showing that i is injective. Let φ ∈ Aut(G, T) ; we have [φ] ∈ ker i if and only if φ|_{N} ∈ Inn(N) . Therefore we find n ∈ N such that φ|_{N} = c_{n} . So c_{n}^{−1} ∘ φ|_{T} = id_{T} and thus the automorphism c_{n}^{−1} ∘ φ is a conjugation by an element t ∈ T [2, Proposition 9, p. 30]. Hence [φ] = [c_{nt}] is trivial in Aut(G, T)/(Aut(G, T) ∩ Inn(G)) ≅ Out(G) and i is injective.

Proposition 4.2. There exists a surjective homomorphism

p : Out(N) ↠ Out(G)

for which the injection i is a splitting, i.e. such that p∘i is the identity of Out(G) . Before proving this proposition, we first introduce two lemmas.

Lemma 4.3. Let W be a finite real reflection group and let W = W_{1}×W_{2}×…× W_{n} be its decomposition into irreducible components. Let φ be an automorphism of W that preserves the set of reflections in W (i.e. for any reflection s_{α} , the element φ(s_{α}) is a reflection). Then φ preserves the decomposition of W into irreducible components, i.e. for all i we have φ(W_{i}) = W_{j} , for some j .

Proof. By definition of an irreducible component, and by the fact that any irreducible component is generated by the reflections it contains, two reflections s_{α} and s_{α′} are in the same component if and only if there exists m reflections s_{α_{1}} = s_{α}, …, s_{α_{m}} such that s_{α_{i}} and s_{α_{i+1}} do not commute, for all 1 ⩽ i ⩽ m − 1 , and such that s_{α′} = s_{α_{m}} … s_{α_{2}}s_{α_{1}} . As φ preserves reflections, the previous equivalence implies that φ(s_{α}) and φ(s_{α′}) are in the same component if and only if s_{α} and s_{α′} are in the same component.

Remark 4.4. One cannot drop the hypothesis on φ in this lemma: the Weyl group of SU(2) × SU(2) has two irreducible components; it is isomorphic to the Klein group whose automorphisms permute the three non-trivial elements, but it contains only two reflections.

Direct factors isomorphic to SO(2ℓ + 1) in G will play a special role in the proof of proposition 4.2 (see [10, 8] for related special appearances of this direct factors). The following lemma asserts that the corresponding direct factors in N are well-behaved with respect to automorphisms.

Lemma 4.5. Let ψ ∈ Aut(N) . Then ψ permutes the maximal tori of the direct factors in N that correspond to direct factors isomorphic to SO(2ℓ + 1) in G .

Proof. We assume that G has at least one direct factor isomorphic to SO(2ℓ + 1) . By the decomposition theorem of connected compact Lie groups, we can write G = H ×G_{s} , where G_{s} = SO(2ℓ+1) , and H = (S×G_{1}×…×G_{s−1})/K , with S a torus, G_{1}, …, G_{s−1} simple 1-connected compact Lie groups and K a finite central subgroup of S × G_{1} × … × G_{s−1} . This also implies the following decompositions: T = (S×T_{1}×…×T_{s−1})/K×T_{s} , LT = LS⊕LT_{1}⊕…⊕LT_{s−1}⊕LT_{s} for the Lie algebra of T , N = (S × N_{1} × … × N_{s−1})/K × N_{s} , and W = W_{1} × … × W_{s−1} × W_{s} . By lemma 3.1, the automorphism ψ̄ : W → W permutes the reflections, and thus, by lemma 4.3, it permutes the irreducible components of W . Suppose, without loss of generality, that ψ̄(W_{1}) = W_{s} . By lemma 3.1, we therefore also have ψ_{∗}(LT_{1}) = LT_{s} . Let us consider the composite homomorphism

κ : G_{1} ↪ S × G_{1} × … × G_{s−1} ↠ (S × G_{1} × … × G_{s−1})/K ↪ G ,

which factorizes through the injection α : Ḡ_{1} = G_{1}/ ker κ ↪ G , with ker κ central. By restriction, we also have α_{|N̄_{1}} : N̄_{1} = N_{1}/ ker κ ↪ N , and α_{|T̄_{1}} : T̄_{1} = T_{1}/ ker κ ↪ T . Let us then consider the composition

ψ_{1} : N̄_{1} ↪^{α} N →^{ψ} N ↠^{p_{s}} N_{s} ,

where p_{s} denotes the canonical projection. Claim. ψ_{1} is an isomorphism. First we show that ψ_{1} is surjective. ¿From the commutative diagram

[figure: LT_{1}; α_{∗} ; ≅; LT_{1}; ψ_{∗|LT_{1}} ; ≅; LT_{s}; (p_{s})_{∗|LTs} ; ≅; LT_{s}; T̄_{1}; ψ_{1|T̄_{1}}; T_{s}]

we get that ψ_{1|T̄_{1}} : T̄_{1} → T_{s} is surjective. On the other hand, we have

ψ̄ ∘ ᾱ : W̄_{1} = N̄_{1}/T̄_{1} ⟶^{≅} W_{1} ⟶^{≅} W_{s} ,

and we deduce that im ψ_{1} meets every connected component of N_{s} . So im ψ_{1} is a subgroup of N_{s} that contains T_{s} and that meets all components, therefore it is equal to N_{s} and we get the surjectivity of ψ_{1} . For the injectivity, we first observe that the situation we have just described at the level of Weyl groups implies that ker ψ_{1} ⊂ T̄_{1} . Now ψ ∘ α is clearly injective, and maps T̄_{1} bijectively onto T_{s} . As p_{s} maps T_{s} bijectively onto itself, we get that ψ_{1} is injective. This proves our claim.

Then, by the theorem of Curtis-Wiederhold-Williams [3, 10], recalled in the introduction, Ḡ_{1} must be isomorphic to SO(2ℓ+1) . It follows that the group Ḡ_{1} , which is of adjoint type, is a direct factor in G; its maximal torus T̄_{1} is mapped by ψ isomorphically onto T_{s} .

Proof of proposition 4.2. Let ψ ∈ Aut(N) ; the main part of the proof consists in showing that the vector space automorphism ψ_{∗} : LT → LT is an automorphism of the root diagram D(G). As (ψ_{1} ∘ ψ_{2})_{∗} = (ψ_{1})_{∗} ∘ (ψ_{2})_{∗} , it will yield, by theorem 2.4 , a homomorphism p̄ : Aut(N) → Out(G) . We finally check that p̄ factorizes through a homomorphism p : Out(N) → Out(G) with the required properties.

Let us show that ψ_{∗} ∈ Aut(D(G)) in several steps. Step 1: The naturality of the exponential implies that ψ_{∗} maps Γ(T) bijectively onto itself. Step 2: We derive some formulae that will be useful in the next step. In particular, we get that the the compatibility condition in definition 2.3 is satisfied. Let α^{∨} ∈ R^{∨} ; fix an element X ∈ LT and decompose it as X = Y + Z , with Y ∈ ℝα^{∨} and Z ∈ H_{α} . Explicitly, we have Y = \frac{(X ,α^{∨})}{(α^{∨} ,α^{∨})} ·α^{∨} , and thus

ψ_{∗}(Y) = \frac{(X , α^{∨})}{(α^{∨} , α^{∨})} ·ψ_{∗}(α^{∨}) .

On the other hand, we have ψ_{∗}(X) = ψ_{∗}(Y) + ψ_{∗}(Z) , with, again by lemma 3.1 , ψ_{∗}(Y) ∈ ℝβ^{∨} and ψ_{∗}(Z) ∈ H_{β} . Therefore we have

ψ_{∗}(Y) = \frac{(ψ_{∗}(X) , ψ_{∗}(α^{∨}))}{(ψ_{∗}(α^{∨}) , ψ_{∗}(α^{∨}))} ·ψ_{∗}(α^{∨}) .

Comparing these two expressions for ψ_{∗}(Y) , we get that

\frac{(X , α^{∨})}{(α^{∨} , α^{∨})} = \frac{(ψ_{∗}(X) , ψ_{∗}(α^{∨}))}{(ψ_{∗}(α^{∨}) , ψ_{∗}(α^{∨}))} , ∀ X ∈ LT , ∀ α^{∨} ∈ R^{∨} .

In particular, this equality shows that the compatibility condition in definition 2.3 is satisfied. Now setting X = β^{∨} ∈ R^{∨} , we get

\frac{∥β^{∨}∥}{∥α^{∨}∥} ·cos θ = \frac{∥ψ_{∗}(β^{∨})∥}{∥ψ_{∗}(α^{∨})∥} ·cos ϕ ,

where θ = ∡(α^{∨} , β^{∨}) et ϕ = ∡(ψ_{∗}(α^{∨}) , ψ_{∗}(β^{∨})) . Symmetrically we also get

\frac{∥α^{∨}∥}{∥β^{∨}∥} ·cos θ = \frac{∥ψ_{∗}(α^{∨})∥}{∥ψ_{∗}(β^{∨})∥} ·cos ϕ .

For two non-orthogonal coroots α^{∨} and β^{∨} , these last two expressions finally yield

\frac{∥α^{∨}∥}{∥β^{∨}∥} = \frac{∥ψ_{∗}(α^{∨})∥}{∥ψ_{∗}(β^{∨})∥} .  (∗)

Step 3: In this step we show that ψ_{∗} maps the set of coroots R^{∨} bijectively onto itself. We first observe that the troublesome cases are those where G has direct factors of type SO(2ℓ + 1) . By lemma 3.1, ψ_{∗} permutes the directions generated by the coroots. We know from step 1 that ψ_{∗} preserves Γ(T). Moreover the set R^{∨} is contained in the integral lattice Γ(T) , and it is almost always characterized by the fact that α^{∨} is minimal, i.e. ℝα^{∨} ∩ Γ(T) = ℤα^{∨} for all α^{∨} ∈ R^{∨} . The only exception is the following case (see [8, Proposition 3.2.] or [10, pp. 86-87]): R has an irreducible component of type B_{ℓ} (ℓ ⩾ 1 , B_{1} = A_{1}) that corresponds in G, to a direct factor SO(2ℓ + 1) , and α is a short root. For R^{∨} , this translates into a dual irreducible component of type C_{ℓ} , with a long coroot α^{∨} ; in this case 1/2·α^{∨} is the minimal element. By minimality of coroots in all the other cases, it only remains to treat this exceptional case. Lemma 4.5 further reduces the situation to the case where N is the normalizer of a maximal torus in a single copy of SO(2ℓ + 1) . The case ℓ = 1 is straightforward: the integral lattice is isomorphic to ℤ and is preserved by ψ_{∗}, thus the generator 1/2·α^{∨} is mapped to ±1/2·α^{∨} , and therefore ψ_{∗} preserves R^{∨} = {±α^{∨}}. If ℓ > 1 , there are two possible lengths for the coroots, and the factor between the norms of a long and a short coroot is \sqrt{2} . The only pathological case that could occur is the following: a long coroot α^{∨} (i.e. a non-minimal one) maps to twice a short coroot. A short coroot could then map to either a short coroot or half a long coroot (because ψ_{∗}(Γ(T)) = Γ(T)) . Now α^{∨} = α_{ℓ}^{∨} is an element of some basis {α_{1}^{∨}, …, α_{ℓ−1}^{∨}, α_{ℓ}^{∨}} of R^{∨} , with α_{ℓ}^{∨} and α_{ℓ−1}^{∨} non-orthogonal and α_{ℓ−1}^{∨} necessarily short, as depicted in the following Dynkin diagram of type C_{ℓ}:

[figure: <; ···; α_{1}^{∨}; α_{2}^{∨}; α_{ℓ−1}^{∨}; α_{ℓ}^{∨}]

Let us write 2 · β_{ℓ}^{∨} = ψ_{∗}(α_{ℓ}^{∨}) , for some short coroot β_{ℓ}^{∨} . We apply (∗) with α^{∨} = α_{ℓ}^{∨} and β^{∨} = α_{ℓ−1}^{∨} . Distinguishing the two possible cases for the length of the image of the short coroot α_{ℓ−1}^{∨} , we get

\sqrt{2} = \frac{∥α_{ℓ}^{∨}∥}{∥α_{ℓ−1}^{∨}∥} = \frac{∥2·β_{ℓ}^{∨}∥}{∥ψ_{∗}(α_{ℓ−1}^{∨})∥} = \frac{2∥α_{ℓ−1}^{∨}∥}{∥ψ_{∗}(α_{ℓ−1}^{∨})∥} = \{ \begin{matrix}2\frac{∥α_{ℓ−1}^{∨}∥}{∥α_{ℓ−1}^{∨}∥} & = 2 \\ 2\frac{∥α_{ℓ−1}^{∨}∥}{\frac{1}{2} ∥α_{ℓ}^{∨}∥} & = 2\sqrt{2} , \end{matrix}

which is manifestly impossible and ends up the proof of step 3.

Steps 1,2 and 3 together show that ψ_{∗} is an automorphism of D(G) . Finally, it is clear that inner automorphisms of N act as elements of the Weyl group of D(G) . By theorem 2.4, this implies that the homomorphism p̄ factorizes through p : Out(N) → Out(G) . By construction, it is also clear that the composition p ∘ i : Out(G) ↪ Out(N) → Out(G) is the identity. This shows that p is split-surjective, with i as a splitting, and completes the proof.

Proposition 4.6. The kernel of p is

ker p = H^{1}(W; T) .

Proof. By definitions:

[ψ] ∈ ker p ⟺ p[ψ] ∈ W (D)
⟺ (ψ_{∗} : LT → LT ) ∈ W .

So we find n ∈ N such that ψ_{∗} = (c_{n})_{∗} . Clearly [ψ] = [c_{n}^{−1} ∘ ψ] ∈ Out(N) . Now (c_{n}^{−1} ∘ ψ)_{∗} = id_{LT} and thus c_{n}^{−1} ∘ ψ_{|T} = id_{T} . Therefore, by corollary 3.2 , the induced automorphism on W is also the identity, and we can conclude by invoking theorem 3.3.

Proof of theorem 1.2. Propositions 4.1, 4.2 and 4.6 clearly constitute a proof.

Remark 4.7. We mention here a few properties and results related to the group H^{1}(W; T) . These are taken from the author’s PhD thesis [6] and will be the object of a later publication. Using a presentation of the normalizer due to Tits, it is shown that H^{1}(W; T) is a (finite) elementary abelian 2-group. Explicit calculations of this group are then carried out: for example if G = SU(n) then H^{1}(W; T) ≅ ℤ/2 for n = 2m ⩾ 4 , generated by the class of the automorphism of example 1.1, and trivial in the remaining cases. Together with some recent work of Suter and Matthey, these explicit calculations cover all simple connected compact Lie groups.

Remark 4.8. The explicit description of Out(N) in theorem 1.2 turns out to be a key ingredient in [6] for proving a generalization of the theorem of Curtis, Wiederhold and Williams, to nonconnected compact Lie groups. This generalization then allows us to get a new proof, in the nonconnected case, of the fact that the classifying space BG characterizes the compact Lie group G.

Acknowledgments. The material in this paper is taken from my PhD thesis at the University of Neuchâtel, Switzerland, and it is a pleasure to warmly thank my advisor Professor U. Suter for his guidance and constant encouragements. I am also greatly indebted to M. Matthey and A. Osse for many fruitful discussions and suggestions. In particular, the contents of section 2 were discussed at a seminar in which Suter, Matthey, Osse and myself participated. I also gratefully acknowledge the financial support of the Swiss National Science Foundation, under grant no. 2000-056816.99, and the current financial support of the Japan Society for the Promotion of Science.

References

[1] Adem, A., and R. J. Milgram, “Cohomology of Finite Groups,” Springer, 1994.

[2]  Bourbaki, N., “Groupes et algèbres de Lie, ch. IX,” Masson, 1983.

[3] Curtis, M., A. Wiederhold, and B. Williams, Normalizers of Maximal Tori, in: Localization in Group Theory and Homotopy Theory, Springer LNM 418 (1974), 31–47.

[4] de Siebenthal, J., Sur les groupes de Lie compacts non connexes, Comment. Math. Helv. 31 (1956/57), 41–89.

[5] Dwyer, W. G., Lie Groups and p-Compact Groups, in: Proceedings of the International Congress of Mathematicians, Berlin 1998, 2, 433–442, Documenta Mathematica, 1998.

[6] Hämmerli, J.-F., “Normalizers of Maximal Tori and Classifying Spaces of Compact Lie Groups,” Ph.D. Thesis, Université de Neuchâtel, Switzerland, 2000.

[7] Humphreys, J. E., “Reflection Groups and Coxeter Groups,” Cambridge University Press, 1990.

[8] Jackowski, S., J. McClure, and B. Oliver, Self Homotopy Equivalences of Classifying Spaces of Compact Connected Lie Groups, Fund. Math. 147 (1995), 99–126.

[9] Møller, J. M., Deterministic p-Compact Groups, in: Stable and unstable homotopy (Toronto, ON, 1996), 255–278, Amer. Math. Soc., 1998.

[10] Osse, A., λ-Structures and Representation Rings of Compact Connected Lie Groups, J. Pure Appl. Algebra 121 (1997), 69–93.

[11] Robinson, D. J. S., Automorphisms of Group Extensions, in: Algebra and its Applications, Marcel Dekker Lecture Notes in Pure and Applied Mathematics 91 (1984), 163–167.

[12] Wells, C., Automorphisms of Group Extensions, Trans. Amer. Math. Soc. 155(1) (1971), 189–194.

Jean-François Hämmerli Department of Mathematics Faculty of Science Okayama University 3-1 Tsushima-naka JP-700-8530 Okayama hammerli@math.okayama-u.ac.jp

Received February 16, 2001 and in final form October 30, 2001