Journal of Lie Theory Volume 10 (2000) 399–410 C ◯2000 Heldermann Verlag

Separation of Unitary Representations of Exponential Lie Groups

Didier Arnal, Ali Baklouti, Jean Ludwig and Mohamed Selmi^{∗}

Communicated by J. Faraut

Abstract. We show that every irreducible unitary representation π of an exponential Lie group G = exp 𝔤 is characterized by its generalized moment set.

1. Introduction

Let G be a real Lie group with Lie algebra 𝔤, (π, H_{π}) a unitary representation of G and H_{π}^{∞} the space of C^{∞} vectors of π. Let 𝔤^{∗} be the dual space of 𝔤. In [8], Wildberger has introduced the moment map Ψ_{π} of π. For all ξ in H_{π}^{∞} \{0}, Ψ_{π}(ξ) in 𝔤^{∗} is defined by:

Ψ_{π}(ξ)(X) = \frac{1}{i} \frac{⟨dπ(X)ξ, ξ⟩}{⟨ξ, ξ⟩}, X ∈ 𝔤.  (1.1)

The moment set I_{π} of the representation π is by definition the closure in 𝔤^{∗} of the image of the moment map:

Ψ_{π} : H^{∞} \ {0} ⟶ 𝔤^{∗}.

Wildberger has given an explicit description of the moment set I_{π} when G is a connected simply connected nilpotent Lie group. More precisely, he shows (Theorem 4.2 in [8]) that I_{π} is the closure of the convex hull of the coadjoint orbit O_{π} associated to π via the Kirillov theory, i.e.

I_{π} = \overline{conv(O_{π})}.

This result has been generalized by Arnal and Ludwig [1] for connected solvable Lie groups. Nevertheless, as shown in [8], the moment set does not characterize the representation for nilpotent Lie groups.

ISSN 0949–5932 / $2.50 ^{_{C} ◯} Heldermann Verlag

∗ This work was supported by the CMCU: F97/1501. D. Arnal and J. Ludwig thank the Department of Mathematics of the University of Sfax for its kind hospitality during their stay.

We can extend the moment map to the dual of the complex universal enveloping algebra U(𝔤) of 𝔤 as follows:

Ψ̃_{π}(ξ)(A) = ℜ𝔢 (\frac{1}{i} \frac{⟨dπ(A)ξ, ξ⟩}{⟨ξ, ξ⟩}) , ξ ∈ H^{∞} \ \{0\}, A ∈ U(𝔤)  (1.2)

and consider the convex hull J(π) of the image of this generalized moment map Ψ̃_{π}. Let us call the set J(π) the generalized moment set of π, i.e.

J(π) = conv (Ψ̃_{π}(H^{∞} \ {0})) .

Let U_{n} be the subspace of U(𝔤) consisting of elements A of degree less or equal to n. By restriction to U_{n}, we define:

J(π)^{n} = {f|_{U_{n}}, f ∈ J(π)}.

In [2], we show:

Theorem 0. Let G be a simply connected nilpotent Lie group. Then there exists an integer n such that, for any unitary irreducible representations π and ρ, one has J(π)^{n} = J(ρ)^{n} if and only if π and ρ are equivalent.

In this paper, we use the set J(π) to characterize the irreducible unitary representations of exponential groups. More precisely, we prove the following:

Theorem 1. Let G = exp 𝔤 be an exponential Lie group. Let π and ρ be two unitary irreducible representations. Then π ∼ ρ if and only if J(π) = J(ρ).

2. Preliminaries

Let 𝔤 be a real Lie algebra, G its simply connected Lie group and let exp : 𝔤 → G be the exponential mapping. The group G is called an exponential group if the exponential mapping exp is a diffeomorphism. In particular this condition implies that G is solvable. Throughout this paper, we assume that G is an exponential group.

It is well known that each irreducible unitary representation π of G is associated to a coadjoint orbit O_{π} in 𝔤^{∗} by the following construction:

Let f be in 𝔤^{∗}. We take polarizations in f, i.e. subalgebras 𝔥 such that:

⟨f, [𝔥, 𝔥]⟩ = 0

with maximal dimension. Among those polarizations, we can find one satisfying the Pukanszky’s condition:

H.f = f + 𝔥^{⊥} (here H = exp(𝔥)).

For such a Pukanszky polarization, we define π_{f,𝔥} by:

π_{f,𝔥} = Ind_{H}^{G}χ_{f}

where χ_{f} denotes the unitary character:

χ_{f}(exp X) = e^{if(X)} ∀X ∈ 𝔥

of H.

Then π_{f,𝔥} is irreducible and the class π of π_{f,𝔥} depends only of the coadjoint orbit of f.

Moreover, the map f ⟼ [π_{f,𝔥}] defines a bijection K (the Kirillov correspondance) between the space 𝔤^{∗}/G of orbits and the the space Ĝ of classes of irreducible unitary representation of G [4], [5], [7].

Let now H be a closed subgroup of G and π_{0} a unitary representation of H on the space H_{π_{0}} . Let us describe explicitely the induced representation π = Ind_{H}^{G}π_{0} . Let dg be a left Haar measure on G and Δ_{G} the modular function of G. Note Δ_{H,G} the positive character of H defined by:

Δ_{H,G}(h) = \frac{Δ_{H}(h)}{Δ_{G}(h)}.

Let K(G, H) be the space of continuous functions F on G, with compact support modulo H and which verify:

F (gh) = Δ_{H,G}(h)F (g) (g ∈ G, h ∈ H).

G acts on this space by left translation. It is well known ([4]) that, up to a multiplication by a scalar, there exists a unique G-invariant positive linear form:

F ↦ ∮_{G/H} F (g) dġ

on K(G, H).

Let K(G, H, π_{0}) be the space of continuous mappings F : G ⟶ H_{π_{0}} with compact support modulo H and which verify:

F (gh) = Δ_{H,G}^{1/2}(h)π_{0}(h)^{−1}F (g) (g ∈ G, h ∈ H).

If F belongs to K(G, H, π_{0}), the function g ↦ ∥F (g)∥_{π_{0}}^{2} belongs to K(G, H). Set:

∥F ∥_{π}^{2} = ∮_{G/H} ∥F (g)∥_{π_{0}}^{2} dġ.

The induced representation π = Ind_{H}^{G} π_{0} of G is realized by left translation on the Hilbert space H_{π} obtained by the completion of K(G, H, π_{0}) with the norm defined above.

Definition 2.1 Let 𝔥 be a subalgebra of 𝔤, a set B = {X_{1}, ⋯, X_{d}} of elements of 𝔤 is called a coexponential basis to 𝔥 in 𝔤 if the map:

(t_{1}, t_{2}, ⋯, t_{d}; Y) → E_{B}(t_{1}, t_{2}, ⋯, t_{d}; Y) = exp t_{1}X_{1} exp t_{2}X_{2}, …, exp t_{d}X_{d} exp Y

is a diffeomorphism from ℝ^{d} × 𝔥 onto G.

For any subalgebra 𝔥, there exist coexponential basis (see [4], [5]). With such a basis, we can realize the space H_{π} of an induced representation π = Ind_{H}^{G} π_{0} as the space L^{2}(ℝ^{d}, H_{π_{0}}).

3. Main lemma

The proof of theorem 1 will use an induction on the dimension of 𝔤. The following lemmata allow us to pass from 𝔤 to some subalgebra 𝔤_{0} of lower dimension.

Lemma 3.1. Let G be an exponential Lie group. Let π and ρ be two irreducible representations of G such that J(π) = J(ρ). Suppose there exists a connected subgroup G_{0} = exp(𝔤_{0}) of codimension 1 and two irreducible representations π_{0} and ρ_{0} of G_{0} such that:

π = Ind_{G_{0}}^{G}π_{0} and ρ = Ind_{G_{0}}^{G}ρ_{0}.

Choose a coexponential basis {X} to 𝔤_{0} in 𝔤, realize π (resp. ρ) on the space L^{2}(ℝ, H_{π_{0}}) (resp. L^{2}(ℝ, H_{ρ_{0}})). Suppose furthermore that there exists u in U(𝔤_{0}) such that:

(dπ(u)(ξ))(t) = h(t)ξ(t) and (dρ(u)(η))(t) = h(t)η(t),

for each ξ in H_{π}^{∞} and η in H_{ρ}^{∞}, for every t, where h is a strictly increasing C^{∞} function with h(0) ≥ 0. Then:

J(π_{0}) = J(ρ_{0}).

Proof. We recall that all C^{∞} vectors of π are C^{∞} functions of the variable t in ℝ with value in the space H_{π_{0}}^{∞}. Moreover, if ξ_{0} is a vector in H_{π_{0}}^{∞} and φ a C^{∞} complex valued map with compact support, then ξ(t) = φ(t)ξ_{0} is a C^{∞} vector of π [6].

We fix such a vector ξ, so that:

φ(0) = 1, ∥ξ_{0}∥_{π_{0}} = 1 and ∥ξ∥_{π} = 1.

Since J(π) = J(ρ), there exist normalized C^{∞} vectors η_{1}, …, η_{n} in H_{ρ}^{∞} and λ_{1}, …, λ_{n} in ]0,1[ with ∑_{j=1}^{n} λ_{j} = 1 such that for all A in U(𝔤):

ℜ𝔢 (\frac{1}{i}⟨dπ(A)ξ, ξ⟩) = ℜ𝔢 (\frac{1}{i} ∑_{j=1}^{n} λ_{j}⟨dρ(A)η_{j}, η_{j}⟩) .

This implies:

⟨dπ(A)ξ, ξ⟩ = ∑_{j=1}^{n} λ_{j}⟨dρ(A)η_{j}, η_{j}⟩ for all A in U(𝔤).  (3.1)

In particular, for any integer k, we have:

⟨dπ(u^{k})ξ, ξ⟩ = ∑_{j=1}^{n} λ_{j}⟨dρ(u^{k})η_{j}, η_{j}⟩.

Choose M > 0 and m < M such that the support of φ is contained in [m, M]. Then, for every odd k,

⟨dπ(u^{k})ξ, ξ⟩ = ∫_{ℝ} h^{k}(t)∥ξ(t)∥_{π_{0}}^{2}dt
= ∫_{m}^{M} h^{k}(t)φ^{2}(t)∥ξ_{0}∥_{π_{0}}^{2}dt
≤ (h(M))^{k}∥ξ∥_{π}^{2} = (h(M))^{k}.

Hence:

\begin{aligned}∑_{j=1}^{n} λ_{j} ∫_{−∞}^{+∞} h^{k}(t)∥η_{j}(t)∥_{ρ}^{2} dt & = ∑_{j=1}^{n} λ_{j}⟨dρ(u^{k})η_{j}, η_{j}⟩ \\ & = ⟨dπ(u^{k})ξ, ξ⟩ ≤ (h(M))^{k}. \end{aligned}

Thus, for each ε > 0, for all M + ε ≤ t, we have 0 < h(M + ε) ≤ h(t) and:

h(M + ε)^{k} ∑_{j=1}^{n} λ_{j} ∫_{M+ε}^{+∞} ∥η_{j}(t)∥_{ρ}^{2} dt ≤ ∑_{j=1}^{n} λ_{j} ∫_{M+ε}^{+∞} h^{k}(t)∥η_{j}(t)∥_{ρ}^{2} dt ≤ h(M)^{k}.

This shows that:

∑_{j=1}^{n} λ_{j} ∫_{M+ε}^{+∞} ∥η_{j}(t)∥_{ρ}^{2} dt ≤ lim_{{k→∞ \atop k odd}} (\frac{h(M)}{h(M + ε)})^{k} = 0.

Hence supp η_{j} ⊂] − ∞, M], for all j. Using the negative increasing function t ⟼ h(t) − h(M) on ] − ∞, M], one gets by (3.1):

∑_{j=1}^{n} λ_{j} ∫_{−∞}^{M}(h(t) − h(M))^{2k}∥η_{j}(t)∥_{ρ}^{2} dt = ∑_{j=1}^{n} λ_{j}⟨dρ(u − h(M)1)^{2k}η_{j}, η_{j}⟩
= ⟨dπ(u − h(M)1)^{2k}ξ, ξ⟩
= ∫_{m}^{M}(h(t) − h(M))^{2k}φ^{2}(t)dt.

The function t ↦ (h(t) − h(M))^{2k} being decreasing,

\begin{aligned}∑_{j=1}^{n} λ_{j} ∫_{−∞}^{M}(h(t) − h(M))^{2k}∥η_{j}(t)∥_{ρ}^{2} dt & = ∫_{m}^{M}(h(t) − h(M))^{2k}φ^{2}(t)dt \\ & ≤ (h(m) − h(M))^{2k}. \end{aligned}

404 Arnal, Baklouti, Ludwig and Selmi Then for each ε > 0:

∑_{j=1}^{n} λ_{j} ∫_{−∞}^{m−ε}(h(t) − h(M))^{2k}∥η_{j}(t)∥_{ρ}^{2} dt ≤ (h(m) − h(M))^{2k}.

If t ≤ m − ε then (h(m − ε) − h(M))^{2k} ≤ (h(t) − h(M))^{2k} thus:

∑_{j=1}^{n} λ_{j} ∫_{−∞}^{m−ε} ∥η_{j}(t)∥^{2}dt ≤ lim_{k→+∞} (\frac{h(m) − h(M)}{h(m − ε) − h(M)})^{2k} = 0.

Hence supp η_{j} ⊂ [m, M] for each j.

Let now A be in U(𝔤_{0}) and x a real number, for each integer k:

∫_{m}^{M} \frac{(ixh(t))^{k}}{k!} ⟨(dπ(A)ξ)(t), ξ(t)⟩_{π_{0}} dt =

= ⟨\frac{1}{k!}(ixdπ(u))^{k}dπ(A)ξ, ξ⟩

= ∑_{j=1}^{n} λ_{j}⟨\frac{1}{k!}(ixdπ(u))^{k}dρ(A)η_{j}, η_{j}⟩

= ∑_{j=1}^{n} λ_{j} ∫_{m}^{M} \frac{(ixh(t))^{k}}{k!} ⟨(dρ(A)η_{j})(t), η_{j}(t)⟩_{ρ_{0}} dt.

Let g(t) = ⟨(dπ(A)ξ)(t), ξ(t)⟩_{π_{0}} and g_{j}(t) = ⟨(dρ(A)η_{j})(t), η_{j}(t)⟩_{ρ_{0}} , then g and g_{j} are C^{∞} functions with compact support. Hence:

∫_{ℝ} e^{ixh(t)}g(t)dt = ∑_{j=1}^{n} λ_{j} ∫_{ℝ} e^{ixh(t)}g_{j}(t)dt.

Let us make the change of variables s = h(t). We set θ(s) = \frac{g(t)}{h′(t)} and θ_{j}(s) = \frac{g_{j}(t)}{h′(t)} , so:

θ̂(x) = ∫_{ℝ} e^{ixs}θ(s)ds = ∑_{j=1}^{n} λ_{j} ∫_{ℝ} e^{ixs}θ_{j}(s)ds = ∑_{j=1}^{n} λ_{j}θ̂_{j}(x).

Then θ(x) = ∑_{j=1}^{n} λ_{j}θ_{j}(x), by the injectivity of the Fourier transform. In particular θ(h(0)) = ∑_{j=1}^{n} λ_{j}θ_{j}(h(0)), i.e.:

\frac{1}{h′(0)}⟨(dπ(A)ξ)(0), ξ(0)⟩_{π_{0}} = \frac{1}{h′(0)} ∑_{j=1}^{n} λ_{j}⟨(dρ(A)η_{j})(0), η_{j}(0)⟩_{ρ_{0}}.

Since

(dπ(A)ξ)(0) = dπ_{0}(A)(ξ(0)) = dπ_{0}(A)ξ_{0}

and similarly for the representation ρ, we have:

⟨dπ_{0}(A)(ξ_{0}), ξ_{0}⟩_{π_{0}} = ∑_{j=1}^{n} λ_{j}∥η_{j}(0)∥_{ρ_{0}}^{2}.\frac{⟨dρ_{0}(A)η_{j}(0), η_{j}(0)⟩_{ρ_{0}}}{∥η_{j}(0)∥_{ρ_{0}}^{2}}  (3.2)

When A = 1, the formula (3.2) gives us

1 = ∑_{j=1}^{n} λ_{j}∥η_{j}(0)∥_{ρ_{0}}^{2}.

Hence A ↦ ⟨dπ_{0}(A)(ξ_{0}), ξ_{0}⟩_{π_{0}} is a convex combination of some elements in Ψ̃_{ρ_{0}}(H_{ρ_{0}}^{∞}). So,

J(π_{0}) ⊂ J(ρ_{0}).

By exchanging the roles of π and ρ, we are done.

By exactly the same arguments we can prove the following lemma:

Lemma 3.2. Let G be an exponential Lie group. Let π and ρ be two irreducible representations of G such that J(π) = J(ρ). Suppose that there exists a connected subgroup G_{0} = exp(𝔤_{0}) of codimension 2 and two irreducible representations π_{0} and ρ_{0} of G_{0} such that:

π = Ind_{G_{0}}^{G}π_{0} and ρ = Ind_{G_{0}}^{G}ρ_{0}.

Choose a coexponential basis {X_{1}, X_{2}} to 𝔤_{0} in 𝔤, realize π (resp. ρ) on the space L^{2}(ℝ^{2}, H_{π_{0}}) (resp. L^{2}(ℝ^{2}, H_{ρ_{0}})). Suppose furthermore that there exist u_{1} , u_{2} in U(𝔤_{0}) and strictly increasing real C^{∞} functions h_{1} , h_{2} such that h_{j}(0) ≥ 0 and:

\begin{matrix}(dπ(u_{1})(ξ))(t_{1}, t_{2}) = h_{1}(t_{1})ξ(t_{1}, t_{2}) & and (dρ(u_{1})(η))(t_{1}, t_{2}) = h_{1}(t_{1})η(t_{1}, t_{2}) \\ (dπ(u_{2})(ξ))(t_{1}, t_{2}) = h_{2}(t_{2})ξ(t_{1}, t_{2}) & and (dρ(u_{2})(η))(t_{1}, t_{2}) = h_{2}(t_{2})η(t_{1}, t_{2}). \end{matrix}

for every ξ in H_{π}^{∞}, η in H_{ρ}^{∞} and real numbers t_{1} , t_{2} . Then:

J(π_{0}) = J(ρ_{0}).

4. An example

Before we give the proof of theorem 1, we present an example where there is no polynomial invariant to separate the generic orbits. Let 𝔤_{α} be the Lie algebra spanned by X, Y_{1} , Y_{2} such that:

[X, Y_{1}] = Y_{1} − αY_{2}, [X, Y_{2}] = αY_{1} + Y_{2} and [Y_{1}, Y_{2}] = 0.

The group G_{α} = exp 𝔤_{α} is called the Grélaud group. Let {X^{∗}, Y_{1}^{∗}, Y_{2}^{∗}} be the dual basis of {X, Y_{1}, Y_{2}} in 𝔤_{α}^{∗}. The coadjoint orbits of G_{α} which are not reduced to a point are the orbits O_{θ} of f_{θ} = cos θY_{1}^{∗} + sin θY_{2}^{∗} (0 < θ ≤ 2π). It is easy to see that:

O_{θ} = {sX^{∗} + e^{−t} cos(θ − αt)Y_{1}^{∗} + e^{−t} sin(θ − αt)Y_{2}^{∗}, s, t ∈ ℝ} .

We consider the Pukanszky polarisation 𝔥 = ℝY_{1} ⊕ ℝY_{2} in f_{θ}, we choose {−X} as coexponential basis to 𝔥 in 𝔤. The unitary irreducible representation π, associated to the orbit O_{θ} acts on the space L^{2}(ℝ) as:

π(exp xX)ξ(t) = ξ(t + x)
π(exp y_{1}Y_{1})ξ(t) = e^{−iy_{1}e^{t} cos(θ+αt)}ξ(t)
π(exp y_{2}Y_{2})ξ(t) = e^{−iy_{2}e^{t} sin(θ+αt)}ξ(t).

The self-adjoint operators :

idπ(X) = i\frac{d}{dt}, idπ(Y_{1}) = e^{t} cos(θ + αt) and idπ(Y_{2}) = e^{t} sin(θ + αt)

on L^{2}(ℝ) characterize the representation π. Since idπ(Y_{1}) and idπ(Y_{2}) are multiplication operators, we could obtain the value of θ which characterizes the orbit O_{θ}, directly from these operators by the following transcendental expression:

e^{iθ}Id = \frac{idπ(Y_{1}) − dπ(Y_{2})}{\sqrt{−dπ(Y_{1})^{2} − dπ(Y_{2})^{2}}} e^{− \frac{i}{2} α log(−dπ(Y_{1})^{2}−dπ(Y_{2})^{2})}.  (4.1)

But we can also use lemma 3.1. Indeed let f_{ψ} = cos ψY_{1}^{∗} + sin ψY_{2}^{∗} and ρ the representation associated to O_{ψ}. Let us put:

𝔤_{0} = 𝔥, π_{0} = χ_{f_{θ}} ρ_{0} = χ_{f_{ψ}}, u = (iX)^{2} + (iY)^{2}, h(t) = e^{2t}.

If J(π) = J(ρ), then with lemma 3.1:

J(π_{0}) = J(χ_{f_{θ}}) = J(ρ_{0}) = J(χ_{f_{ψ}})

and f_{θ} = f_{ψ}, π ∼ ρ.

5. Proof of theorem 1

Now we are going to prove theorem 1. It is clear that if π ∼ ρ then J(π) = J(ρ). We shall show the converse by induction on the dimension of G. It is obviously true for one dimensional G.

Let f (resp. g) be an element of the Kirillov orbit O_{π} of π (resp. O_{ρ} of ρ). If there exists a non trivial ideal 𝔞 of 𝔤 on which f vanishes, then dπ(X) = 0 for all X in 𝔞 and since J(π) = J(ρ):

⟨dρ(X)η, η⟩ = ∑_{j=1}^{n} λ_{j}⟨dπ(X)ξ_{i}, ξ_{j}⟩ = 0, (X ∈ 𝔞, η ∈ H_{ρ}).

Hence dρ(X) = 0 for each X in 𝔞 and g|_{𝔞} vanishes. Let Ḡ = G/A where A = exp 𝔞 and p : G → Ḡ be the canonical projection. This p defines projections (still denoted p) for the corresponding Lie algebras and enveloping algebras. We can write f = f̄_{∘}p = p^{t}(f̄) and π = π̄_{∘}p (resp. g = ḡ_{∘}p, ρ = ρ̄_{∘}p). By construction:

J(π̄) = p^{t} (J(π)) = p^{t} (J(ρ)) = J(ρ̄)

and π̄ and ρ̄ are equivalent by the induction hypothesis. Finally we get π ∼ ρ.

We can thus assume than f does not vanish on any non trivial ideal. In particular the center 𝔷(𝔤) of 𝔤 satisfies dim 𝔷(𝔤) ≤ 1 and if 𝔷(𝔤) is one dimensional, f is non trivial on 𝔷(𝔤).

Let 𝔞 be a non central minimal ideal of 𝔤. Then 𝔞 is abelian and of dimension equal to one, two or three (see [3]). Let us call 𝔞^{f} the subspace:

𝔞^{f} = {T ∈ 𝔤, ⟨f, [T, 𝔞]⟩ = 0} .

We shall prove in the sequel that 𝔞^{f} = 𝔞^{g}. Then according to [3], π and ρ are induced from the subgroup G_{0} = exp 𝔞^{f} = exp 𝔞^{g}:

π = Ind_{G_{0}}^{G}π_{0}, ρ = Ind_{G_{0}}^{G}ρ_{0}.

Case 1 dim 𝔷(𝔤) = 0: Then the dimension of 𝔞 is one or two.

Subcase 1.1 dim(𝔞) = 1: Take Y in 𝔞 such that f(Y) = 1, then g(Y) = α ≠ 0 and so:

𝔞^{f} = {U ∈ 𝔤, [U, Y] = 0} = 𝔞^{g}.

Choose X in 𝔤 such that [X, Y] = −Y as coexponential basis to 𝔞^{f}, then we have (see the notations of lemma 3.1):

(dπ(Y)ξ) (t) = ie^{t}ξ(t), (dρ(Y)η) (t) = ie^{t}η(t) (ξ ∈ H_{π}^{∞}, η ∈ H_{ρ}^{∞}).

Then:

\begin{aligned}α ∫ e^{t}∥η(t)∥_{ρ_{0}}^{2} dt & = \frac{1}{i}⟨dρ(Y)η, η⟩ = ∑_{j=1}^{n} λ_{j} \frac{1}{i}⟨dπ(Y)ξ_{j}, ξ_{j}⟩ \\ & = ∑_{j=1}^{n} λ_{j} ∫ e^{t}∥ξ_{j}(t)∥_{π_{0}}^{2} dt. \end{aligned}

Hence α is non negative and if we replace g by Ad^{∗}(exp log αX)g, one can suppose that g(Y) = 1.

Now with h(t) = e^{t}, we can apply lemma 3.1 and the induction hypothesis, thus π ∼ ρ.

Subcase 1.2 dim 𝔞 = 2: Then there exist X, Y_{1} , Y_{2} such that 𝔞 = ℝY_{1} ⊕ ℝY_{2} with the brackets:

[X, Y_{1}] = Y_{1} − αY_{2}, [X, Y_{2}] = Y_{2} + αY_{1}, α ∈ ℝ \ {0}.

This is a generalization of the Grélaud group case. As in subcase 1.1, we have:

𝔞^{f} = {U ∈ 𝔤, [U, Y_{1}] = [U, Y_{2}] = 0} = 𝔞^{g}

and

π = Ind_{G_{0}}^{G}π_{0}, ρ = Ind_{G_{0}}^{G}ρ_{0} (𝔤_{0} = 𝔞^{f} = 𝔞^{g}).

Let us put f_{1} = f(Y_{1}), f_{2} = f(Y_{2}). Then f_{1}^{2} + f_{2}^{2} ≠ 0 and since

f′_{1}^{2} + f′_{2}^{2} = e^{−2x}(f_{1}^{2} + f_{2}^{2}) if f′ = Ad^{∗}(exp xX)f,

we can suppose that f_{1}^{2} + f_{2}^{2} = 1 and similarly g_{1}^{2} + g_{2}^{2} = 1 if g_{1} = g(Y_{1}), g_{2} = g(Y_{2}). Then:

−dπ(Y_{1}^{2} +Y_{2}^{2})ξ(t) = e^{2t}ξ(t), −dρ(Y_{1}^{2} +Y_{2}^{2})η(t) = e^{2t}η(t) (ξ ∈ H_{π}^{∞}, η ∈ H_{ρ}^{∞}).

As above, we get π ∼ ρ.

Case 2 dim 𝔷(𝔤) = 1: Let 𝔷(𝔤) = ℝZ. Then both f(Z) and g(Z) are not zero and since J(π) = J(ρ), we can choose Z such that f(Z) = g(Z) = 1.

Subcase 2.1 dim 𝔞 = 2: We choose a basis {Y, Z} of 𝔞 such that f(Y) = 0 (Z is central and f(Z) = 1). Thus:

[U, Y] = λ(U)Y + μ(U)Z (U ∈ 𝔤)

with either λ and μ linearly independant in 𝔤^{∗} or λ = 0 and μ ≠ 0 (see [4]). Hence:

𝔞^{f} = {U ∈ 𝔤, μ(U) = 0} = 𝔞^{g}.

We choose X in 𝔤 such that λ(X) = 0 and μ(X) = 1. Then acting on g with Ad^{∗}(exp g(Y)X), we can suppose g(Y) = 0. Then:

π = Ind_{G_{0}}^{G}π_{0}, ρ = Ind_{G_{0}}^{G}ρ_{0} (𝔤_{0} = 𝔞^{f} = 𝔞^{g}).

Moreover {X} is a coexponential basis to 𝔤_{0} and:

dπ(−iY)ξ(t) = tξ(t),  dρ(−iY)η(t) = tη(t)  (ξ ∈ H_{π}^{∞}, η ∈ H_{ρ}^{∞}).

and by lemma 3.1 and induction hypothesis, π ∼ ρ.

Subcase 2.2 dim 𝔞 = 3: We choose a basis {Y_{1}, Y_{2}, Z} of 𝔞 such that f(Y_{1}) = f(Y_{2}) = 0 (Z is central and f(Z) = 1). Thus for each U in 𝔤:

[U, Y_{1}] = λ(U) (Y_{1} + αY_{2}) + μ_{1}(U)Z, [U, Y_{2}] = λ(U) (Y_{2} − αY_{1}) + μ_{2}(U)Z

with λ, μ_{1} and μ_{2} linearly independant in 𝔤^{∗} (see [4]). Then:

𝔞^{f} = {U ∈ 𝔤, μ_{1}(U) = μ_{2}(U) = 0} = 𝔞^{g}.

Choose X_{1} , X_{2} in 𝔤 such that:

λ(X_{1}) = λ(X_{2}) = 0 and μ_{i}(X_{j}) = δ_{ij}.

Then {X_{1}, X_{2}} is a coexponential basis to 𝔤_{0} , acting with Ad^{∗}(exp g(Y_{1})X_{1}) and Ad^{∗}(exp g(Y_{2})X_{2}) on g, we can suppose g(Y_{1}) = g(Y_{2}) = 0. Thus for any ξ in H_{π}^{∞} and η in H_{ρ}^{∞}:

\begin{matrix}dπ(−iY_{1})ξ(t_{1}, t_{2}) = t_{1}ξ(t_{1}, t_{2}), & dρ(−iY_{1})η(t_{1}, t_{2}) = t_{1}η(t_{1}, t_{2}), \\ dπ(−iY_{2})ξ(t_{1}, t_{2}) = t_{2}ξ(t_{1}, t_{2}), & dρ(−iY_{2})η(t_{1}, t_{2}) = t_{2}η(t_{1}, t_{2}). \end{matrix}

This ends the proof of theorem 1 by using lemma 3.2 and the induction hypothesis.

References

[1] Arnal D. et J. Ludwig, La convexité de l’application moment d’un groupe de Lie, Journal of Funct. Anal. 105 (1992), 256–300.

[2] Baklouti A., J. Ludwig et M. Selmi, Séparation des représentations unitaires des groupes de Lie nilpotents, Proceedings of the II International Workshop, Claustal, Germany 17-20 August 1997, Lie theory and its applications in Physics II.

[3] Bernat P., Sur les représentations unitaires des groupes de Lie résolubles, Ann. Scient. Ec. Norm. Sup. 82 (1965), 37–99.

[4] Bernat P. and al., “Représentations des groupes de Lie résolubles,” Dunod, Paris, 1972.

[5] Leptin H. and J. Ludwig, Unitary representations theory of exponential Lie groups, De Gruyter, Berlin, New York, 1994.

[6] Poulsen N., On C^{∞} vectors and intertwining bilinear forms for representations of Lie groups, Journal of Funct. Anal. 9 (1972), 87–120.

[7] Pukanszky L., Unitary representation of solvable Lie group, Ann. Sci. Ec. Norm. Sup.(4) 4 (1973), 457–608.

[8] Wildberger N. J., Convexity and unitary representations of nilpotent Lie groups, Invent. Math. 98 (1989), 281–292.

410 Arnal, Baklouti, Ludwig and Selmi

D. Arnal and J. Ludwig Université de Metz Département de Mathématiques Ile du Saulcy, 57045 METZ Cedex 01 France arnal@poncelet.sciences.univ-metz.fr ludwig@poncelet.sciences.univ-metz.fr A. Baklouti and M. Selmi Faculté des Sciences de Sfax Département de Mathématiques Route de Soukra 3038 SFAX Tunisie Ali.Baklouti@fss.rnu.tn Mohamed.Selmi@fss.rnu.tn

Received July 20, 1999 and in final form November 10, 1999