Journal of Lie Theory Volume 11 (2001) 135–154 \begin{matrix}c \\ ◯ \end{matrix} 2001 Heldermann Verlag Berlin

Moment sets and the unitary dual of a nilpotent Lie group

Ali Baklouti, Chal Benson, and Gail Ratcliff

Communicated by J. Ludwig

Abstract. Let G be a connected and simply connected nilpotent Lie group with Lie algebra 𝔤 and unitary dual Ĝ. The moment map for π ∈ Ĝ sends smooth vectors in the representation space of π to 𝔤^{∗}. The closure of the image of the moment map for π is called its moment set. N. Wildberger has proved that the moment set for π coincides with the closure of the convex hull of the corresponding coadjoint orbit. We say that Ĝ is moment separable when the moment sets differ for any pair of distinct irreducible unitary representations. Our main results provide sufficient and necessary conditions for moment separability in a restricted class of nilpotent groups.

1. Introduction

Let G be a real Lie group with Lie algebra 𝔤 and π be a (strongly continuous) unitary representation of G in some Hilbert space H. The moment map for π is defined as

ψ_{π} : H^{∞} \ \{0\} → 𝔤^{∗}, ψ_{π}(v)(X) = \frac{1}{i} \frac{⟨dπ(X)v, v⟩}{⟨v, v⟩},  (1)

where dπ denotes the derived representation of 𝔤 in the space H^{∞} of smooth vectors. When π is finite dimensional, this notion, which is due to N. Wildberger, reduces to that of the usual moment map for the Hamiltonian action of G via π on the projective space P (H) [11]. The moment set for π is defined as

I_{π} = \overline{\{ψ_{π}(v) : v ∈ H^{∞} \ \{0\}\}},

the closure of the image of ψ_{π} in 𝔤^{∗}. We let Ĝ denote the set of irreducible unitary representations of G (up to unitary equivalence) and consider the moment sets I_{π} for π ∈ Ĝ. One says that Ĝ is moment separable if I_{π} ≠ I_{π′} for all π, π′ ∈ Ĝ with π ≄ π′.

In the case of a compact group G, the moment set I_{π} of an irreducible representation π need not be convex. Wildberger has shown, however, that the set of extremal points of the convex hull of I_{π} is a single coadjoint orbit, namely the orbit through the highest weight of the representation π. Thus the moment set completely determines the representation.

ISSN 0949–5932 / $2.50 \begin{matrix}c \\ ◯ \end{matrix} Heldermann Verlag

In this paper we consider the situation where G is a connected and simply connected nilpotent Lie group. In this setting, a fundamental result of Wildberger relates the moment set I_{π} for π ∈ Ĝ to the coadjoint orbit O ⊂ 𝔤^{∗} associated to π via the Kirillov method [4]. Namely

I_{π} = \overline{Conv(O)},  (2)

the closure of the convex hull of O in 𝔤^{∗} [10]. We remark that this result has been generalized to encompass connected solvable Lie groups by D. Arnal and J. Ludwig in [1]. In view of Equation 2, Ĝ is moment separable if and only if

\overline{Conv(O)} = \overline{Conv(O′)} ⇒ O = O′

for all coadjoint orbits O, O′ ⊂ 𝔤^{∗}.

In [10], Wildberger presents an example which shows that Ĝ need not be moment separable in the nilpotent case. It is thus natural to seek a characterization of the class of connected and simply connected nilpotent Lie groups for which the moment sets do separate Ĝ. Our main results in this direction are given below in Theorems 3.5 and 3.6. These provide sufficient and necessary conditions, respectively, for moment separability of Ĝ. In fact, our results apply only to a restricted class of nilpotent groups, namely those which satisfy Condition (C) formulated below in Section 3..

The sufficient and necessary conditions in Theorems 3.5 and 3.6 involve properties of the Pukanszky polynomials which parameterize the coadjoint orbits. We review this orbit parameterization in Section 2. in order to introduce notation needed to formulate our results. The orbits are grouped in layers and the polynomial functions which determine the orbits within a layer have a common domain. It is, however, possible for the convex hull of a coadjoint orbit to pass though other orbit layers. This fact greatly complicates the use of Pukanszky polynomials in this problem. We are led to formulate Condition (C) for use as a hypothesis in Theorems 3.5 and 3.6. This condition implies that the moment sets for representations whose coadjoint orbits lie in different layers are necessarily distinct. We show that all three step groups satisfy Condition (C), so these are among the groups to which our results apply.

We present several examples in Section 4.. These show that the sufficient condition for separability in Theorem 3.5 is not necessary and that the necessary condition in Theorem 3.6 is not sufficient. In addition, we present an example to show that the convex hull of a coadjoint orbit need not lie within a layer. In Section 5. we discuss some questions that remain open in this area.

We conclude this introduction by noting that the definition given in Equation 1 extends to yield a generalized moment map,

Ψ_{π} : H^{∞} \ {0} → U (𝔤)^{∗},

where U (𝔤) denotes the complexified universal enveloping algebra. One can define the generalized moment set J_{π} ⊂ U (𝔤)^{∗} as

J_{π} = \overline{Conv(Ψ_{π}(H^{∞} \ \{0\}))}.

It is shown in [2] that such generalized moment sets always separate the unitary dual Ĝ of any connected and simply connected nilpotent Lie group.

2. Preliminaries on Orbit Parameterization

We begin this section by reviewing Pukanszky’s parameterization of the coadjoint orbits for a nilpotent Lie group. This material is quite standard. We refer the reader to [9], [8] or [3] for details. Throughout, G will always denote a connected and simply connected nilpotent Lie group with Lie algebra 𝔤. Let

{0} = 𝔤_{0} ⊂ 𝔤_{1} ⊂ ⋯ ⊂ 𝔤_{n} = 𝔤

be any fixed Jordan-Hölder sequence in 𝔤 and

X_{1}, X_{2}, …, X_{n}

be an associated strong Malcev basis with X_{j} ∈ 𝔤_{j} \ 𝔤_{j−1}. We denote by

X_{1}^{∗}, X_{2}^{∗}, …, X_{n}^{∗}

the dual basis for 𝔤^{∗}.

For ℓ ∈ 𝔤^{∗} we let O_{ℓ} = Ad^{∗}(G)ℓ denote the coadjoint orbit through ℓ and

𝔤(ℓ) = {X ∈ 𝔤 : ℓ[X, 𝔤] = {0}},

the Lie algebra of the stabilizer of ℓ. A value j ∈ {1, …, n} is a jump index for ℓ if

X_{j} ∉ 𝔤(ℓ) + 𝔤_{j−1}.

We let

e(ℓ) = {j : j is a jump index for ℓ}, ẽ(ℓ) = {1, …, n} \ e(ℓ)

and

E = {e(ℓ) : ℓ ∈ 𝔤^{∗}}.

The set e(ℓ) contains exactly dim(O_{ℓ}) indices, which is necessarily an even number. The sets

Ω_{e} = {ℓ ∈ 𝔤^{∗} : e(ℓ) = e}

for each e ∈ E are the layers in 𝔤^{∗}. One has O_{ℓ} ⊂ Ω_{e(ℓ)} for ℓ ∈ 𝔤^{∗}.

Each layer Ω_{e} is a semi-algebraic set in 𝔤^{∗}. To understand this fact, one first introduces the strict total ordering ≺ on E defined as follows. For e, e′ ∈ E we have e ≺ e′ if either

1. e = {j_{1} < j_{2} < ⋯ < j_{d}}, e′ = {j′_{1} < j′_{2} < ⋯ < j′_{d′}} where j_{1} = j′_{1}, …, j_{k−1} = j′_{k−1} and j_{k} < j′_{k} for some k ≤ min(d, d′), or

2. e′ ⊊ e.

Note that, in view of the second condition, the empty set e = Ø is the maximal element in E. The layer Ω_{Ø} corresponds to the one-dimensional representations in Ĝ. The layer Ω_{e} given by the minimal element e in E contains the generic orbits and forms a Zariski open set in 𝔤^{∗}. More generally, one has that for e ∈ E , the layer Ω_{e} is the intersection of a Zariski open set with ⋃_{e′⪰e} Ω_{e′}, which is Zariski closed. In fact, there are Ad^{∗}(G)-invariant polynomial functions

P_{e} : 𝔤^{∗} → ℝ

for each e ∈ E with the property that

Ω_{e} = {ℓ ∈ 𝔤^{∗} : P_{e}(ℓ) ≠ 0 and P_{e′}(ℓ) = 0 for e′ ≺ e}.

These are defined explicitly as P_{Ø} = 1 and

P_{e}(ℓ) = P f(M_{e}(ℓ)), where M_{e}(ℓ) = (ℓ[X_{i}, X_{j}])_{i,j∈e}  (3)

for e ≺ Ø. That is, P_{e}(ℓ) is the Pfaffian of the even skew-symmetric matrix M_{e}(ℓ). Given any subset S of {1, 2, …, m}, we let

V_{S} = Span{X_{j}^{∗} : j ∈ S} ⊂ 𝔤^{∗}.

For e ∈ E and ℓ ∈ Ω_{e}, there is a polynomial function

r^{e}(ℓ, ·) : V_{e} → Vẽ

for which r^{e}(ℓ, 0) = 0 and

O_{ℓ} = ℓ + Graph(r^{e}(ℓ, ·)) = ℓ + {x + r^{e}(ℓ, x) : x ∈ V_{e}}.  (4)

Writing

r^{e}(ℓ, x) = ∑_{j∈ẽ} r_{j}^{e}(ℓ, x)X_{j}^{∗},

the polynomial r_{j}^{e}(ℓ, ·) : V_{e} → ℝ depends only on ℓ|_{𝔤_{j−1}} and on the value of x on 𝔤_{j−1}.

For ℓ ∈ 𝔤^{∗} we define
ẽ_{1}(ℓ) = {j ∈ ẽ(ℓ) : r_{j}^{e(ℓ)}(ℓ, x) = 0 ∀x ∈ V_{e(ℓ)}}, ẽ_{2}(ℓ) = ẽ(ℓ) \ ẽ_{1}(ℓ).

The orbit O_{ℓ} is constant in the directions X_{j}^{∗} for each j ∈ ẽ_{1}(ℓ). For e = e(ℓ), the polynomial r^{e}(ℓ, x) = ∑_{j∈ẽ(ℓ)} r_{j}^{e}(ℓ, x)X_{j}^{∗} = ∑_{j∈ẽ_{2}(ℓ)} r_{j}^{e}(ℓ, x)X_{j}^{∗} takes values in the subspace V_{ẽ_{2}(ℓ)} of V_{ẽ(ℓ)}. We will also write this polynomial as

r_{ℓ} : V_{e(ℓ)} → V_{ẽ_{2}(ℓ)}.

For e ∈ E , the set

U_{e} = Ω_{e} ∩ Vẽ

is a cross section to the coadjoint orbits in Ω_{e}. That is, O_{ℓ} meets U_{e(ℓ)} in exactly one point. For ℓ = ∑_{j∈ẽ} ℓ_{j}X_{j}^{∗} ∈ U_{e} we write

ℓ^{(1)} = ∑_{j∈ẽ_{1}(ℓ)} ℓ_{j}X_{j}^{∗} ∈ V_{ẽ_{1}(ℓ)}, ℓ^{(2)} = ∑_{j∈ẽ_{2}(ℓ)} ℓ_{j}X_{j}^{∗} ∈ V_{ẽ_{2}(ℓ)},

so that ℓ = ℓ^{(1)} + ℓ^{(2)}. With this notation, Equation 4 now yields

O_{ℓ} = ℓ^{(1)} + (ℓ^{(2)} + Graph(r_{ℓ})) ⊂ V_{ẽ_{1}(ℓ)} ⊕ (V_{e} ⊕ V_{ẽ_{2}(ℓ)}),  (5)
Conv(O_{ℓ}) = ℓ^{(1)} + (ℓ^{(2)} + Conv(Graph(r_{ℓ}))), and  (6)
\overline{Conv(O_{ℓ})} = ℓ^{(1)} + (ℓ^{(2)} + \overline{Conv(Graph(r_{ℓ}))}).  (7)

The indices ẽ_{1}(ℓ) can be characterized algebraically as follows.

Lemma 2.1. ẽ_{1}(ℓ) = {j ∈ ẽ(ℓ) : X_{j} ∈ 𝔞(ℓ)} where 𝔞(ℓ) denotes the largest ideal of 𝔤 that is contained in 𝔤(ℓ).

Proof. Note that the largest ideal in 𝔤 that is contained in 𝔤(ℓ) is

𝔞(ℓ) = ⋂_{g∈G} Ad(g)(𝔤(ℓ)) = {X ∈ 𝔤 : Ad(g)(X) ∈ 𝔤(ℓ) ∀g ∈ G}.

Let j ∈ ẽ(ℓ). Now j ∈ ẽ_{1}(ℓ) if and only if the map G → ℝ, g ↦ (Ad^{∗}(g)ℓ)(X_{j}) is constant. Equivalently, we must have

\frac{d}{dt}|_{t=0} (Ad^{∗}(exp(tY)g)ℓ)(X_{j}) = 0

for all g ∈ G and all Y ∈ 𝔤. That is, j ∈ ẽ_{1}(ℓ) if and only if (Ad^{∗}(g)ℓ)[X_{j}, Y] = 0 for all g, Y . Equivalently, X_{j} ∈ 𝔤(Ad^{∗}(g)ℓ) = Ad(g)(𝔤(ℓ)) for all g ∈ G.

Lemma 2.1 shows that we can have e(ℓ) = e(ℓ′) but ẽ_{1}(ℓ) ≠ ẽ_{1}(ℓ′). The ideal 𝔞(ℓ) can be characterized in terms of the representation π asociated to the coadjoint orbit O_{ℓ} through ℓ:

𝔞(ℓ) = {X ∈ 𝔤 : dπ(X) is a scalar operator} = {X ∈ 𝔤 : dπ(X) = iℓ(X)I}.

(8)

We refer the reader to [5] for this fact, the proof of which requires details concerning the Kirillov correspondence. Lemma 2.3 below recasts Equation 8 in terms of the moment map for π. The proof uses a preliminary result.

Lemma 2.2. Let X ∈ 𝔤, π ∈ Ĝ and λ ∈ ℝ be given. We have ψ_{π}(v)(X) = λ for all v ∈ H_{π}^{∞} \ {0} if and only if dπ(X) = iλI.

Proof. It is clear that if dπ(X) = iλId, then ψ_{π}(v)(X) = λ for all v ∈ H_{π}^{∞} \ {0}. Conversely, suppose that ψ_{π}(v)(X) = λ for all v ∈ H_{π}^{∞} \ {0}. Thus

⟨dπ(X)v, v⟩ = iλ∥v∥^{2}.  (9)

Replacing v by u + v in this equation (u, v ∈ H_{π}^{∞} \ {0} with v ≠ −u) gives

Im(⟨dπ(X)v, u⟩) = λRe(⟨v, u⟩).

On the other hand, replacing v in Equation 9 by u + iv yields

Re(⟨dπ(X)v, u⟩) = −λIm(⟨v, u⟩).

Thus

⟨dπ(X)v, u⟩ = iλ⟨v, u⟩,

and hence dπ(X)v = iλv for all v ∈ H_{π}^{∞} \ {0}.

Lemma 2.3. Let ℓ ∈ 𝔤^{∗} and π ∈ Ĝ be the representation corresponding to O_{ℓ}. Then the largest ideal 𝔞(ℓ) in 𝔤 contained in 𝔤(ℓ) is

𝔞(ℓ) = {X ∈ 𝔤 : v ↦ ψ_{π}(v)(X) is constant on H_{π}^{∞}}
= {X ∈ 𝔤 | ψ_{π}(v)(X) = ℓ(X) ∀ v ∈ H_{π}^{∞}}.

Proof. Lemma 2.2 shows that v ↦ ψ_{π}(v)(X) is constant on H_{π}^{∞} if and only if dπ(X) is a scalar operator. The result now follows immediately from Equation 8.

Lemma 2.4. Let π, π′ ∈ Ĝ correspond to coadjoint orbits O_{ℓ} and O_{ℓ′}. If I_{π} = I_{π′} then 𝔞(ℓ) = 𝔞(ℓ′) and ℓ = ℓ′ on 𝔞(ℓ) = 𝔞(ℓ′).

Proof. Let X ∈ 𝔞(ℓ). We will show that X ∈ 𝔞(ℓ′) and that ℓ′(X) = ℓ(X). Let v′ ∈ H_{π′}^{∞}\{0}. Since ψ_{π′}(v′) ∈ I_{π′} and I_{π′} = I_{π} we have that ψ_{π}(u_{n}) → ψ_{π′}(v′) for some sequence (u_{n})_{n=1}^{∞} in H_{π}^{∞} \ {0}. Using Lemma 2.3 and the fact that X ∈ 𝔞(ℓ) we obtain

ψ_{π′}(v′)(X) = lim_{n→∞} ψ_{π}(u_{n})(X) = lim_{n→∞} ℓ(X) = ℓ(X).

Thus v′ ↦ ψ_{π′}(v′) takes the constant value ℓ(X) on H_{π′}^{∞} \ {0}. Lemma 2.3 now shows that X ∈ 𝔞(ℓ′) and that ℓ′(X) = ℓ(X).

Corollary 2.5. If ℓ, ℓ′ ∈ 𝔤^{∗} satisfy e(ℓ) = e(ℓ′) and \overline{Conv(O_{ℓ})} = \overline{Conv(O_{ℓ′})} then ẽ_{1}(ℓ) = ẽ_{1}(ℓ′) and ẽ_{2}(ℓ) = ẽ_{2}(ℓ′).

Proof. In view of Equation 2, this result is an immediate consequence of Lemmas 2.1 and 2.4.

3. Moment Separability

In this section we will consider connected and simply connected nilpotent Lie groups G that satisfy the following condition:

(C)  For every ℓ ∈ 𝔤^{∗}, Conv(O_{ℓ}) ⊂ ⋃_{e′⪰e(ℓ)} Ω_{e′} .

Example 4.5 in Section 4. shows that Condition (C) need not hold in general. Since ⋃_{e′⪰e(ℓ)} Ω_{e′} is closed, we also will have \overline{Conv(O_{ℓ})} ⊂ ⋃_{e′⪰e(ℓ)} Ω_{e′} when Condition (C) holds. Note that each of the following conditions imply (C):

• Conv(O_{ℓ}) ⊂ Ω_{e(ℓ)} for all ℓ ∈ 𝔤^{∗}.

• The connected components of each layer Ω_{e} (e ∈ E ) are convex.

In many examples it is easy to verify these latter properties. For example, the coadjoint orbits, layers and Pukanszky polynomials for all groups G of dimension at most 6 are computed explicitly by O. Nielsen in [7]. One sees very easily from this that the connected components of the layers are convex in all but two of these cases. The two exceptions are the generic layers for the six dimensional groups denoted G_{6,17} and G_{6,23} in [7]. The orbits in these layers are, however, flat (of dimension 4) and hence we have Conv(O_{ℓ}) = O_{ℓ} ⊂ Ω_{e(ℓ)} for such orbits. Thus we see that Condition (C) holds for all cases where dim(G) ≤ 6. The following proposition gives two further classes of examples which satisfy Condition (C).

Proposition 3.1. a) If G is at most step three then G satisfies Condition (C).

b) If all coadjoint orbits for G have dimension at most two then G satisfies (C).

Proof. Suppose that G is nilpotent of step at most three and let ℓ ∈ 𝔤^{∗}. We will show that Conv(O_{ℓ}) ⊂ Ω_{e(ℓ)}. For ℓ′ ∈ Conv(O_{ℓ}) we have

ℓ′ = λ_{1}ℓ ∘ Ad(exp(Y_{1})) + ⋯ + λ_{N}ℓ ∘ Ad(exp(Y_{N}))

for some Y_{1}, …, Y_{N} ∈ 𝔤 and values λ_{1}, …, λ_{N} ∈ (0, 1) with λ_{1} + ⋯ + λ_{N} = 1. Since the layers are determined by the polynomials {P_{e} : e ∈ E }, we need only show that P_{e}(ℓ′) = P_{e}(ℓ) for all e ∈ E . This is immediate for e = Ø. For e ≠ Ø we have P_{e}(ℓ′) = P f(M_{e}(ℓ′)) where M_{e}(ℓ′) = (ℓ′[X_{i}, X_{j}])_{i,j∈e}. As G is at most

3-step, we have

ℓ′[X_{i}, X_{j}] = ∑_{k=1}^{N} λ_{k}ℓ(Ad(exp(Y_{k}))[X_{i}, X_{j}])

= ∑_{k=1}^{N} λ_{k}ℓ([X_{i}, X_{j}] + [Y_{k}, [X_{i}, X_{j}]])

= ℓ[X_{i}, X_{j}] + ℓ [∑_{k=1}^{N} λ_{k}Y_{k}, [X_{i}, X_{j}]]

= ℓ ∘ Ad (exp (∑_{k=1}^{N} λ_{k}Y_{k})) [X_{i}, X_{j}].

Letting g = exp(− ∑ λ_{k}Y_{k}) we have show that

M_{e}(ℓ′) = M_{e}(Ad^{∗}(g)ℓ).

Thus also P_{e}(ℓ′) = P_{e}(Ad^{∗}(g)ℓ) = P_{e}(ℓ) by Ad^{∗}(G)-invariance of P_{e}. This proves the first assertion in Proposition 3.1

Next suppose that all coadjoint orbits for G are at most two dimensional. For e ∈ E let H_{e} = {ℓ ∈ 𝔤^{∗} : P_{e}(ℓ) = 0}. If e ≺ Ø then #(e) = 2 since the orbits in Ω_{e} are two dimensional. If e = {j_{1} < j_{2}} then we see that P_{e}(ℓ) = ℓ[X_{j_{1}}, X_{j_{2}}]. Thus H_{e} is a codimension 1 subspace of 𝔤^{∗} for each e ≺ Ø. We have Ω_{e} = (⋂_{e′≺e} H_{e′}) ∩ (𝔤^{∗} \ H_{e}) for e ≺ Ø and Ω_{Ø} = ⋂_{e′≺Ø} H_{e′}. We see that the connected components for each layer Ω_{e} are convex sets.

Remark 3.2. In [10] it is shown that any pair of two-dimensional coadjoint orbits O_{1}, O_{2} is “convex distinguishable”. That is, Conv(O_{1}) = Conv(O_{2}) only when O_{1} = O_{2}. It is thus probably true that Ĝ is moment separable when all coadjoint orbits have dimension at most 2. In particular, this is the case if, in addition, the convex hull of each coadjoint orbit is closed. On the other hand, examples presented below in Section 4. show that Ĝ may or may not be moment separable when G is 3-step.

Proposition 3.3. If G satisfies Condition (C), then Ĝ is moment separable if and only if the following condition holds: For each e ∈ E , given f, g ∈ U_{e}, we have \overline{Conv(O_{f})} = \overline{Conv(O_{g})} implies f = g.

Proof. The condition is clearly necessary for moment separability. Now suppose that the condition holds, and we are given f, g ∈ 𝔤^{∗} with \overline{Conv(O_{f})} = \overline{Conv(O_{g})}. In view of Condition (C) one has

g ∈ \overline{Conv(O_{g})} = \overline{Conv(O_{f})} ⊂ ⋃_{e′⪰e(f)} Ω_{e′}

and hence e(g) ⪰ e(f). Likewise e(f) ⪰ e(g) and hence e(g) = e(f).

Now we can take f′ in U_{e} ∩ O_{f} and g′ in U_{e} ∩ O_{g}. Then our condition gives f′ = g′, and hence O_{f} = O_{g}. Thus G is moment separable.

Lemma 3.4. Let ℓ, ℓ′ be points in a coadjoint orbit O ⊂ Ω_{e}. Then

r^{e}(ℓ′, x) = r^{e}(ℓ, x + y) − r^{e}(ℓ, y)

for some y in V_{e}.

Proof. By our description of the coadjoint orbit O = O_{ℓ}, we have ℓ′ = ℓ + y + r^{e}(ℓ, y) for some y in V_{e}. On the other hand, an arbitrary point in O = O_{ℓ′} is of the form ℓ′ + x′ + r^{e}(ℓ′, x′) or

ℓ + x + r^{e}(ℓ, x) = ℓ′ − y − r^{e}(ℓ, y) + x + r^{e}(ℓ, x)
= ℓ′ + x − y + r^{e}(ℓ, x) − r^{e}(ℓ, y).

Thus x′ = x − y, and r^{e}(ℓ′, x′) = r^{e}(ℓ, x) − r^{e}(ℓ, y) = r^{e}(ℓ′, x′ + y) − r^{e}(ℓ, y).

Our main results concerning groups G subject to Condition (C) are Theorems 3.5 and 3.6 below. These provide a sufficient and a necessary condition, respectively, for moment separability of Ĝ. We will see via examples in Section 4. that the sufficient condition provided in the first of these theorems is not necessary and that the necessary condition in the second theorem is not sufficient.

Theorem 3.5. Suppose that G satisfies Condition (C) and, for all ℓ ∈ 𝔤^{∗}, e = e(ℓ), we have r_{j}^{e}(ℓ, V_{e}) ≠ ℝ for all j ∈ ẽ. Then Ĝ is moment separable.

Proof. In light of Lemma 3.4, we see that the condition on r_{j}^{e}(ℓ, ·) holds for all ℓ in a layer Ω_{e} iff it holds for all ℓ in the cross-section U_{e}. Thus we consider only points in the cross-sections.

Given e ∈ E , take f, g ∈ U_{e} with \overline{Conv(O_{g})} = \overline{Conv(O_{f})} . We will show that f = g. Corollary 2.5 yields ẽ_{1}(f) = ẽ_{1}(g) and ẽ_{2}(f) = ẽ_{2}(g). Equation 7 implies, moreover, that f^{(1)} = g^{(1)} must hold. It remains to show that f^{(2)} = g^{(2)}.

We write ẽ_{1} and ẽ_{2} for the common sets ẽ_{1}(f) = ẽ_{1}(g) and ẽ_{2}(f) = ẽ_{2}(g) respectively. We suppose below that ẽ_{2} ≠ Ø as otherwise we are done. Let

C_{f} = Conv(Graph(r_{f})), C_{g} = Conv(Graph(r_{g})) ⊂ V_{e} ⊕ Vẽ_{2}.

From above we have that

f^{(2)} + \overline{C}_{f} = g^{(2)} + \overline{C}_{g}.

For j ∈ ẽ_{2}, let p_{j} : V_{e} ⊕ Vẽ_{2} → ℝ be the X_{j}^{∗}-coordinate map. That is, p_{j}(ℓ) = ℓ(X_{j}^{∗}). As p_{j} is linear, we have

p_{j}(f^{(2)} + C_{f}) = f_{j} + p_{j}(C_{f})
= f_{j} + Conv(p_{j}(Graph(r_{f})))
= f_{j} + Conv(r_{j}^{e}(f, V_{e}))
= f_{j} + r_{j}^{e}(f, V_{e}).

Since j ∈ ẽ_{2}, r_{j}^{e}(f, ·) : V_{e} → ℝ is a non-constant polynomial, hence r_{j}^{e}(f, V_{e}) is an unbounded interval in ℝ. By assumption we have r_{j}^{e}(f, V_{e}) ≠ ℝ, so r_{j}^{e}(f, V_{e}) is a proper unbounded interval, (−∞, a), (−∞, a], (a, ∞), or [a, ∞) for some a ∈ ℝ.

We also have

f_{j} + r_{j}^{e}(f, V_{e}) ⊂ p_{j}(f^{(2)} + \overline{C}_{f}) ⊂ f_{j} + \overline{p_{j}(C_{f})} ⊂ f_{j} + \overline{r_{j}^{e}(f, V_{e})}

where r_{j}^{e}(f, V_{e}) is a proper unbounded interval in ℝ, and a similar string of inclusions for g. Since f^{(2)} + \overline{C}_{f} = g^{(2)} + \overline{C}_{g} we conclude that

f_{j} + \overline{r_{j}^{e}(f, V_{e})} = g_{j} + \overline{r_{j}^{e}(g, V_{e})}  (10)

for all j ∈ ẽ_{2}.

Let j ∈ ẽ_{2} and assume inductively that f_{i} = g_{i} for all i ∈ ẽ_{2} with i < j. (In particular, this is automatic when j = min(ẽ_{2}).) We have that f|_{𝔤_{j−1}} = g|_{𝔤_{j−1}} . Indeed, for i < j we have either

• i ∈ e, in which case f_{i} = 0 = g_{i} since f, g ∈ U_{e},

• i ∈ ẽ_{1}, in which case f_{i} = g_{i} since f^{(1)} = g^{(1)}, or

• i ∈ ẽ_{2}, in which case f_{i} = g_{i} by the inductive hypothesis.

As r_{j}(f, ·) and r_{j}(g, ·) depend only on f|_{𝔤_{j−1}} and g|_{𝔤_{j−1}} respectively, we conclude that r_{j}^{e}(f, V_{e}) = r_{j}^{e}(g, V_{e}). Letting A = \overline{r_{j}^{e}(f, V_{e})} = \overline{r_{j}^{e}(g, V_{e})}, Equation 10 becomes

(f_{j} − g_{j}) + A = A.

As A is of the form (−∞, a] or [a, ∞) we now conclude that f_{j} = g_{j} as desired.

Theorem 3.6. Suppose that G satisfies Condition (C) and that Ĝ is moment separable. Then for all ℓ ∈ 𝔤^{∗}, either ẽ_{2}(ℓ) = Ø or Conv(Graph(r_{ℓ})) ≠ V_{e(ℓ)} ⊕ V_{ẽ_{2}(ℓ)}.

Note that when ẽ_{2}(ℓ) = Ø we have O_{ℓ} = ℓ + V_{e}. These are flat orbits and the associated representations are square integrable modulo their kernels [6]. In such cases we have, in particular, that O_{ℓ} is convex.

When ẽ_{2}(ℓ) ≠ Ø then Conv(Graph(r_{ℓ})) ≠ V_{e(ℓ)} ⊕ V_{ẽ_{2}(ℓ)} if and only if Graph(r_{ℓ}) lies to one side of a hyperplane in V_{e(ℓ)} ⊕ V_{ẽ_{2}(ℓ)}. Thus the condition in Theorem 3.6 can also be stated as:

• For all ℓ ∈ 𝔤^{∗} either ẽ_{2}(ℓ) = Ø or there exist real numbers {a_{j} : j ∈ e(ℓ)}, {b_{j} : j ∈ ẽ_{2}(ℓ)}, c ∈ ℝ with

∑_{j∈e(ℓ)} a_{j}x_{j} + ∑_{j∈ẽ_{2}(ℓ)} b_{j}r_{j}^{e(ℓ)}(ℓ, x) > c

for all x = ∑_{j∈e(ℓ)} x_{j}X_{j}^{∗} ∈ V_{e(ℓ)}.

Equivalently:

• For all ℓ ∈ 𝔤^{∗} either ẽ_{2}(ℓ) = Ø or there exist elements A ∈ Span{X_{j} : j ∈ e(ℓ)}, B ∈ Span{X_{j} : j ∈ ẽ_{2}(ℓ)}, c ∈ ℝ with

x(A) + (r_{ℓ}(x))(B) > c

for all x ∈ V_{e(ℓ)}.

Proof. As in the proof of Theorem 3.5, by Lemma 3.4 we only need to consider points in the cross-sections U_{e}. Suppose that G satisfies Condition (C) and ẽ_{2}(f) ≠ Ø for some f ∈ U_{e}, but that Conv(Graph(r_{f})) = V_{e(f)} ⊕ V_{ẽ_{2}(f)}. Let

g = f + εX_{k}^{∗}

where k = max(ẽ_{2}(f)) and ε > 0 is chosen small enough so that P_{e}(g) ≠ 0. The latter is possible since P_{e}(f) ≠ 0 and P_{e} is continuous. We will show that g ∈ U_{e} and that Conv(O_{g}) = Conv(O_{f}). As g ≠ f and U_{e} is a cross section for Ω_{e}, this shows that that Ĝ is not moment separable.

From Equation 7 we obtain

Conv(O_{f}) = f^{(1)} + (f^{(2)} + Conv(r_{f})) = f^{(1)} + (V_{e} ⊕ V_{ẽ_{2}(f)}).

As g = f^{(1)} + (f^{(2)} + εX_{k}^{∗}) ∈ f^{(1)} + (V_{e} ⊕ V_{ẽ_{2}(f)}), we see that g ∈ Conv(O_{f}). Condition (C) now implies that g ∈ ⋃_{e′⪰e} Ω_{e′} and hence P_{e′}(g) = 0 for all e′ ∈ E with e′ ≺ e. Since P_{e}(g) ≠ 0 we now conclude that g ∈ Ω_{e} = {ℓ ∈ 𝔤^{∗} : P_{e}(ℓ) ≠ 0 and P_{e′}(ℓ) = 0 for e′ ≺ e}. Moreover, g = f + εX_{k}^{∗} ∈ Vẽ so g ∈ U_{e} = Ω_{e} ∩ Vẽ.

We claim that r^{e}(g, ·) = r^{e}(f, ·), or equivalently,

r_{j}^{e}(g, ·) = r_{j}^{e}(f, ·) for all j ∈ ẽ.

For this we consider two cases:

1. If j ≤ k then g|_{𝔤_{j−1}} = f|_{𝔤_{j−1}} and hence r_{j}^{e}(g, ·) = r_{j}^{e}(f, ·).

2. If j > k then j ∈ ẽ_{1}(f) since k = max(ẽ_{2}(f)). But as noted above, g ∈ Conv(O_{f}) = f^{(1)} + (V_{e} ⊕ V_{ẽ_{2}(f)}) and as Conv(O_{f}) is an Ad^{∗}(G)- invariant set we have

O_{g} ⊂ f^{(1)} + (V_{e} ⊕ V_{ẽ_{2}(f)}).

Hence the polynomial g_{j} + r_{j}^{e}(g, ·) = f_{j} + r_{j}^{e}(g, ·) which gives the orbit O_{g} in the X_{j}^{∗}-direction must be constant (g_{j} = f_{j}). That is, r_{j}^{e}(g, ·) = 0 = r_{j}^{e}(f, ·) in this case.

It now follows that ẽ_{1}(g) = ẽ_{1}(f), ẽ_{2}(g) = ẽ_{2}(f) and r_{g} = r_{f}. Finally we compute

Conv(O_{g}) = g^{(1)} + (g^{(2)} + Conv(Graph(r_{g})))
= f^{(1)} + (g^{(2)} + Conv(Graph(r_{f})))
= f^{(1)} + (g^{(2)} + V_{e} ⊕ V_{ẽ_{2}(f)})
= f^{(1)} + (V_{e} ⊕ V_{ẽ_{2}(f)})
= Conv(O_{f}).

Examples in Section 4. show that the converses for Theorems 3.5 and 3.6 do not hold in general. The necessary condition from Theorem 3.6 is, however, sufficient in the special case where each coadjoint orbit has at most one nonconstant direction:

Proposition 3.7. Suppose that G satisfies Condition (C) and that #(ẽ_{2}(ℓ)) ≤ 1 for all ℓ ∈ 𝔤^{∗}. Then Ĝ is moment separable if and only if Conv(Graph(r_{ℓ})) ≠ V_{e(ℓ)} ⊕ V_{ẽ_{2}(ℓ)} for all ℓ ∈ 𝔤^{∗} with ẽ_{2}(ℓ) ≠ Ø.

Proof. In view of Theorem 3.6, we need only prove that O_{f} = O_{g} follows from \overline{Conv(O_{f})} = \overline{Conv(O_{g})}. By Proposition 3.3, we can assume that f, g ∈ U_{e} for some e ∈ E . As in the proof of Theorem 3.5, we obtain ẽ_{1}(f) = ẽ_{1}(g) and f^{(1)} = g^{(1)}. We suppose that ẽ_{2}(f) = ẽ_{2}(g) = {j}, and wish to show that f_{j} = g_{j}. As in the proof of Theorem 3.5 we have

f_{j}X_{j}^{∗} + \overline{Conv(Graph(r_{j}^{e}(f, ·)X_{j}^{∗}))} = g_{j}X_{j}^{∗} + \overline{Conv(Graph(r_{j}^{e}(g, ·)X_{j}^{∗}))}.

Moreover, as f^{(1)} = g^{(1)} and f, g ∈ U_{e} we have f|𝔤_{j−1} = g|𝔤_{j−1} and hence r_{j}^{e}(f, ·) = r_{j}^{e}(g, ·). Thus

Conv(Graph(r_{j}^{e}(f, ·)X_{j}^{∗})) = Conv(Graph(r_{j}^{e}(g, ·)X_{j}^{∗})) = C

say. We have now

(f_{j} − g_{j})X_{j}^{∗} + \overline{C} = \overline{C}

and, by iteration, n(f_{j} − g_{j})X_{j}^{∗} + \overline{C} = \overline{C} for all n ∈ ℤ.

Now suppose that f_{j} ≠ g_{j}. Then, by convexity, we have ℝX_{j}^{∗} + \overline{C} = \overline{C}. Since C is the graph of a function from V_{e} to ℝX_{j}^{∗}, we conclude that \overline{C} = V_{e}⊕ ℝX_{j}^{∗}. This contradicts the hypothesis that C ≠ V_{e}⊕ ℝX_{j}^{∗} and hence f_{j} = g_{j}.

4. Examples

As remarked in [10], Ĝ is moment separable for all connected and simply connected nilpotent groups G with dim(G) ≤ 5. Examples 4.1 and 4.2 below illustrate the application of Theorems 3.5 and 3.6 to groups of dimension 6. When dim(G) = 6, as is noted in [10], Ĝ may or may not be moment separable.

Examples 4.3, 4.4 and 4.5 demonstrate some limitations to the results obtained in this paper. Example 4.3 shows that Ĝ may be moment separable in cases where Condition (C) holds but G does not satisfy the remaining hypothesis in Theorem 3.5. Example 4.4 shows that G may satisfy Condition (C) and the conclusion of Theorem 3.6 but Ĝ may fail to be moment separable. Example 4.5 shows that not all nilpotent groups satisfy Condition (C).

Example 4.6 suggests that some of our results should carry over to the setting of exponential solvable Lie groups.

Example 4.1. Let G be the 6-dimensional group with Lie algebra 𝔤 having strong Malcev basis X_{1}, …, X_{6} where

[X_{6}, X_{5}] = X_{4}, [X_{6}, X_{4}] = X_{1}, [X_{3}, X_{2}] = X_{1},

and other brackets of basis elements vanish. This is the group denoted G_{6,1} in [7]. G is a 3-step group with center ℝX_{1}. Condition (C) holds in view of Proposition 3.1. In fact, the jump sets and layers are, from [7]:

E = {{2, 3, 4, 6} ≺ {5, 6} ≺ Ø}, where

Ω_{{2,3,4,6}} = {ℓ : ℓ_{1} ≠ 0}, Ω_{{5,6}} = {ℓ : ℓ_{1} = 0, ℓ_{4} ≠ 0}, Ω_{Ø} = {ℓ : ℓ_{1} = ℓ_{4} = 0}.

Here we write elements ℓ ∈ 𝔤^{∗} as ℓ = ℓ_{1}X_{1}^{∗} + ⋯ + ℓ_{6}X_{6}^{∗}. For e = {2, 3, 4, 6} and ℓ ∈ U_{e} = {ℓ_{1}X_{1}^{∗} + ℓ_{5}X_{5}^{∗} : ℓ_{1} ≠ 0}, one has ẽ_{1}(ℓ) = {1}, ẽ_{2}(ℓ) = {5} and r_{5}^{e}(ℓ, ·) : V_{e} → ℝ is

r_{5}^{e}(ℓ, x_{2}, x_{3}, x_{4}, x_{6}) = \frac{1}{2ℓ_{1}} x_{4}^{2}

Thus r_{5}^{e}(ℓ, V_{e}) ≠ ℝ. For ℓ ∈ Ω_{{5,6}} ∪ Ω_{Ø} we have ẽ_{2}(ℓ) = Ø. (The orbit O_{ℓ} is flat of dimension two or is a single point.) Thus Theorem 3.5 shows that Ĝ is moment separable.

Example 4.2. Next consider the 6-dimensional group G whose Lie algebra 𝔤 is given by the strong Malcev basis X_{1}, …, X_{6} with non-zero brackets

[X_{6}, X_{5}] = X_{3}, [X_{6}, X_{4}] = X_{2}, [X_{5}, X_{2}] = X_{1}, [X_{4}, X_{3}] = X_{1}.

This is the group G_{6,4} in [7]. As in Example 4.1, 𝔤 is 3-step with one dimensional center ℝX_{1}. Condition (C) holds and we have

E = {{2, 3, 4, 5} ≺ {4, 6} ≺ {5, 6} ≺ Ø}, where

Ω_{{2,3,4,5}} = {ℓ : ℓ_{1} ≠ 0},
Ω_{{4,6}} = {ℓ : ℓ_{1} = 0, ℓ_{2} ≠ 0},
Ω_{{5,6}} = {ℓ : ℓ_{1} = ℓ_{2} = 0, ℓ_{3} ≠ 0},
Ω_{Ø} = {ℓ : ℓ_{1} = ℓ_{2} = ℓ_{3} = 0}.

For e = {2, 3, 4, 5} and ℓ ∈ U_{e} = {ℓ_{1}X_{1}^{∗} + ℓ_{6}X_{6}^{∗} : ℓ_{1} ≠ 0} one has ẽ_{1}(ℓ) = {1}, ẽ_{2}(ℓ) = {6} and r_{ℓ} : V_{e} → V_{ẽ_{2}(ℓ)} is the map

r_{ℓ}(x_{2}, x_{3}, x_{4}, x_{5}) = −\frac{1}{ℓ_{1}} x_{2}x_{3}X_{6}^{∗}.

This gives a saddle in the the space spanned by {X_{2}^{∗}, X_{3}^{∗}, X_{6}^{∗}}. It follows that Conv(Graph(r_{ℓ})) = V_{e} ⊕ V_{ẽ_{2}(ℓ)} for ℓ ∈ Ω_{{2,3,4,5}}. Theorem 3.6 now implies that Ĝ is not moment separable.

We remark that Examples 4.1 and 4.2 taken together show that the unitary dual of a 3-step group may or may not be moment separable.

Example 4.3. The n-step ladder group G_{n} (n ≥ 2) has Lie algebra 𝔤_{n} of dimension n + 1 with strong Malcev basis X_{1}, …, X_{n+1} where

[X_{n+1}, X_{j}] = X_{j−1}

for j = 2, …, n. In this example one has

E = {e_{1} ≺ e_{2} ≺ … ≺ e_{n}} where e_{j} = {j + 1, n + 1}

for j = 1, …, n − 1 and e_{n} = Ø. The layers Ω_{e_{j}} are

Ω_{e_{j}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1} = ⋯ = ℓ_{j−1} = 0, ℓ_{j} ≠ 0}

for j = 1, …, n − 1, and

Ω_{e_{n}} = Ω_{Ø} = {ℓ ∈ 𝔤^{∗} : ℓ_{1} = ⋯ = ℓ_{n−1} = 0} = {ℓ_{n}X_{n}^{∗}+ℓ_{n+1}X_{n+1}^{∗} : ℓ_{n}, ℓ_{n+1} ∈ ℝ}.

We see that the connected components of each layer are convex, so Condition (C) holds. Alternatively, this follows from Proposition 3.1 since each coadjoint orbit has dimension at most two. We’ll show that Ĝ_{n} is moment separable but that for n ≥ 4 the condition in Theorem 3.5 does not hold. The case n = 2, the three dimensional Heisenberg group, is transparent. So we suppose below that n ≥ 3.

First note that there is an obvious algebra isomorphism 𝔤_{n}/ℝX_{1} ≅ 𝔤_{n−1}. Coadjoint orbits in layers Ω_{e_{j}} with j > 1 are diffeomorphic to coadjoint orbits in 𝔤_{n−1}^{∗} via the associated linear map 𝔤_{n−1}^{∗} → 𝔤_{n}^{∗}. Thus an inductive argument shows that Ĝ_{n} is moment separable if and only if \overline{Conv(O)} = \overline{Conv(O′)} ⇒ O = O′ for coadjoint orbits O, O′ ⊂ Ω_{e_{1}} .

Writing e = e_{1} and functionals ℓ = ℓ_{1}X_{1}^{∗} + ⋯ + ℓ_{n+1}X_{n+1}^{∗} as ℓ = (ℓ_{1}, …, ℓ_{n+1}), we have

U_{e} = {(ℓ_{1}, 0, ℓ_{3}, …, ℓ_{n}, 0) : ℓ_{1} ≠ 0}.

A brute-force computation shows that for ℓ ∈ U_{e}, ℓ = (ℓ_{1}, 0, ℓ_{3}, …, ℓ_{n}, 0), one has

O_{ℓ} = \{(ℓ_{1}, x_{2}, ℓ_{3} + \frac{1}{2ℓ_{1}} x_{2}^{2}, ℓ_{4} + \frac{ℓ_{3}}{ℓ_{1}} x_{2} + \frac{1}{6ℓ_{1}^{2}} x_{2}^{3}, ℓ_{5} + \frac{ℓ_{4}}{ℓ_{1}} x_{2} + \frac{ℓ_{3}}{2ℓ_{1}^{2}} x_{2}^{2} + \frac{1}{24ℓ_{1}^{3}} x_{2}^{4}, …,

ℓ_{n} + \frac{ℓ_{n−1}}{ℓ_{1}} x_{2} + \frac{ℓ_{n−2}}{2ℓ_{1}^{2}} x_{2}^{2} + ⋯ + \frac{ℓ_{3}}{(n − 3)!ℓ_{1}^{n−3}} x_{2}^{n−3} + \frac{1}{(n − 1)!ℓ_{1}^{n−2}} x_{2}^{n−1}, x_{n+1})

: x_{2}, x_{n+1} ∈ ℝ}.

The Pukanszky polynomials r_{j}^{e}(ℓ, ·) : V_{e} → ℝ are here given by

r_{j}^{e}(ℓ, x_{2}, x_{n+1}) = ∑_{i=1}^{j−1} \frac{ℓ_{j−i}}{i!ℓ_{1}^{i}} x_{2}^{i} = ∑_{i=1}^{j−3} \frac{ℓ_{j−i}}{i!ℓ_{1}^{i}} x_{2}^{i} + \frac{1}{(j − 1)!ℓ_{1}^{j−2}} x_{2}^{j−1}.

We have ẽ_{1}(ℓ) = {1} and ẽ_{2}(ℓ) = {3, …, n}.

Suppose now that ℓ = (ℓ_{1}, 0, ℓ_{3}, …, ℓ_{n}, 0) and ℓ′ = (ℓ′_{1}, 0, ℓ′_{3}, …, ℓ′_{n}, 0) are two points in U_{e} with \overline{Conv(O_{ℓ′})} = \overline{Conv(O_{ℓ})}. It follows that ℓ′_{1} = ℓ_{1}. We can suppose here that ℓ_{1} > 0. All points ℓ′′ in O_{ℓ} have ℓ′′_{3} ≥ ℓ_{3}. Hence the same is true for all points ℓ′′ in \overline{Conv(O_{ℓ})}. Thus we have ℓ′_{3} ≥ ℓ_{3}. Interchanging the roles of ℓ and ℓ′ we obtain also ℓ_{3} ≥ ℓ′_{3} and hence ℓ′_{3} = ℓ_{3}. We note, moreover, that all points ℓ′′ ∈ \overline{Conv(O_{ℓ})} with ℓ′′ ≠ ℓ have ℓ′′_{3} > ℓ_{3}. Since ℓ′ ∈ \overline{Conv(O_{ℓ})} and ℓ′_{3} = ℓ_{3}, we now conclude that ℓ′ = ℓ must hold. This shows that Ĝ_{n} is moment separable.

On the other hand, for n ≥ 4 we see that r_{4}^{e}(ℓ, x) = (ℓ_{3}/ℓ_{1})x_{2} + (1/6ℓ_{1}^{2})x_{2}^{3} has r_{4}^{e}(ℓ, V_{e}) = ℝ for all ℓ ∈ Ω_{e}. Thus the condition in Theorem 3.5 does not hold here.

Example 4.4. Consider the 7-dimensional group G with Lie algebra 𝔤 given by the strong Malcev basis X_{1}, …, X_{7} where:

[X_{5}, X_{4}] = X_{3}, [X_{5}, X_{3}] = X_{2}, [X_{5}, X_{2}] = X_{1},
[X_{6}, X_{5}] = X_{3}, [X_{6}, X_{4}] = X_{2}, [X_{6}, X_{3}] = X_{1},
[X_{7}, X_{5}] = X_{1},

and other brackets of basis elements are zero. 𝔤 is 4-step with center ℝX_{1}. The jump sets for this example are

E = {{2, 3, 5, 6} ≺ {3, 4, 5, 6} ≺ {4, 5} ≺ Ø}

with layers given by:

Ω_{{2,3,5,6}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1} ≠ 0},
Ω_{{3,4,5,6}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1} = 0, ℓ_{2} ≠ 0},
Ω_{{4,5}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1} = ℓ_{2} = 0, ℓ_{3} ≠ 0},
Ω_{Ø} = {ℓ ∈ 𝔤^{∗} : ℓ_{1} = ℓ_{2} = ℓ_{3} = 0},

where ℓ = ℓ_{1}X_{1}^{∗} + ⋯ + ℓ_{7}X_{7}^{∗}. Condition (C) holds because the connected components of each layer are convex. We will show that the condition in Theorem 3.6 holds but that Ĝ is not moment separable.

The cross sections U_{e} are given by

U_{{2,3,5,6}} = {ℓ_{1}X_{1}^{∗} + ℓ_{4}X_{4}^{∗} + ℓ_{7}X_{7}^{∗} : ℓ_{1} ≠ 0},
U_{{3,4,5,6}} = {ℓ_{2}X_{2}^{∗} + ℓ_{7}X_{7}^{∗} : ℓ_{2} ≠ 0},
U_{{4,5}} = {ℓ_{3}X_{3}^{∗} + ℓ_{6}X_{6}^{∗} + ℓ_{7}X_{7}^{∗} : ℓ_{3} ≠ 0},
U_{Ø} = {ℓ_{4}X_{4}^{∗} + ⋯ ℓ_{7}X_{7}^{∗}}.

Writing elements ℓ ∈ 𝔤^{∗} as “(ℓ_{1}, …, ℓ_{7})”, the coadjoint orbit through a point ℓ in U_{{2,3,5,6}}, U_{{3,4,5,6}}, U_{{4,5}}, or U_{Ø} is:

O_{ℓ} = {(ℓ_{1}, x_{2}, x_{3}, ℓ_{4} + x_{2}x_{3}/ℓ_{1} − x_{2}^{3}/(3ℓ_{1}^{2}), x_{5}, x_{6}, ℓ_{7} − x_{2}) : x_{2}, x_{3}, x_{5}, x_{6} ∈ ℝ},
O_{ℓ} = {(0, ℓ_{2}, x_{3}, x_{4}, x_{5}, x_{6}, ℓ_{7}) : x_{3}, x_{4}, x_{5}, x_{6} ∈ ℝ},
O_{ℓ} = {(0, 0, ℓ_{3}, x_{4}, x_{5}, ℓ_{6} − x_{4}, ℓ_{7}) : x_{4}, x_{5} ∈ ℝ}, or
O_{ℓ} = {ℓ}

respectively.

One can read off the Pukanszky polynomials from these orbit descriptions. For e = {2, 3, 5, 6} and ℓ ∈ U_{e}, we see that ẽ_{2}(ℓ) = {4, 7} and that the map r_{ℓ} : V_{e} → V_{ẽ_{2}(ℓ)} is

r_{ℓ}(x_{2}, x_{3}, x_{5}, x_{6}) = (\frac{x_{2}x_{3}}{ℓ_{1}} − \frac{x_{2}^{3}}{3ℓ_{1}^{2}} ) X_{4}^{∗} − x_{2}X_{7}^{∗}.

Thus Graph(r_{ℓ}) is contained in the linear subspace {ℓ′ : ℓ′_{2} + ℓ′_{7} = 0} of V_{e} ⊕ V_{ẽ_{2}(ℓ)}. Hence the same is true for Conv(Graph(r_{ℓ})). This shows that Conv(Graph(r_{ℓ})) ≠ V_{e} ⊕ V_{ẽ_{2}(ℓ)} for all ℓ ∈ U_{{2,3,5,6}}. For e = {4, 5} and ℓ ∈ U_{e}, we have ẽ_{2}(ℓ) = {6} and r_{ℓ} : V_{e} → V_{ẽ_{2}(ℓ)} is

r_{ℓ}(x_{4}, x_{5}) = −x_{4}X_{6}^{∗}.

We see that Conv(Graph(r_{ℓ})) = Graph(r_{ℓ}) ≠ V_{e} ⊕ V_{ẽ_{2}(ℓ)} for all ℓ ∈ Ω_{{4,5}}. For e = {3, 4, 5, 6} or e = Ø and ℓ ∈ U_{e} we have ẽ_{2}(ℓ) = Ø. Hence the condition in the conclusion of Theorem 3.6 holds for this example.

On the other hand, Ĝ is not moment separable. Indeed, let f = X_{1}^{∗} and g = X_{1}^{∗} + X_{4}^{∗}. Since f, g ∈ U_{{2,3,5,6}} and f ≠ g, we have that O_{f} ≠ O_{g}. In fact, these orbits are:

O_{f} = {(1, x_{2}, x_{3}, x_{2}x_{3} − x_{2}^{3}/3, x_{5}, x_{6}, −x_{2}) : x_{2}, x_{3}, x_{5}, x_{6} ∈ ℝ},
O_{g} = {(1, x_{2}, x_{3}, 1 + x_{2}x_{3} − x_{2}^{3}/3, x_{5}, x_{6}, −x_{2}) : x_{2}, x_{3}, x_{5}, x_{6} ∈ ℝ}.

Since (1, 1, 1, 2/3, 0, 0, −1) ∈ O_{f} and (1, −1, −1, 4/3, 0, 0, 1) ∈ O_{f}, we have

g = (1, 0, 0, 1, 0, 0, 0)
= \frac{1}{2}(1, 1, 1, 2/3, 0, 0, −1) + \frac{1}{2}(1, −1, −1, 4/3, 0, 0, 1) ∈ Conv(O_{f}).

Thus Conv(O_{g}) ⊂ Conv(O_{f}). But we also have (1, 1, −1, −1/3, 0, 0, −1) ∈ O_{g} and (1, −1, 1, 1/3, 0, 0, 1) ∈ O_{g}. So

f = (1, 0, 0, 0, 0, 0, 0)
= \frac{1}{2}(1, 1, −1, −1/3, 0, 0, −1) + \frac{1}{2}(1, −1, 1, 1/3, 0, 0, 1) ∈ Conv(O_{g})

and hence Conv(O_{f}) ⊂ Conv(O_{g}). As O_{f} ≠ O_{g} but Conv(O_{f}) = Conv(O_{g}), Ĝ is not moment separable.

This example shows that the necessary condition for moment separability furnished by Theorem 3.6 is not sufficient.

Example 4.5. Let G be the group of 5 × 5 unipotent upper-triangular matrices. This is a 4-step group of dimension 10 with one dimensional center. We’ll show that Condition (C) does not hold for this example. The Lie algebra 𝔤 for G is the set of 5 × 5 upper triangular matrices with 0’s on the diagonal. Let E_{i,j} denote the matrix with a 1 in position (i, j) and other entries zero. Then {E_{i,j} : i < j} is a strong Malcev basis for 𝔤 when ordered as:

E_{1,5}, E_{1,4}, E_{2,5}, E_{1,3}, E_{2,4}, E_{3,5}, E_{1,2}, E_{2,3}, E_{3,4}, E_{4,5}.

The structure equations are

[E_{i,j}, E_{k,ℓ}] = δ_{j,k}E_{i,ℓ}.

There are nine layers Ω_{e} in 𝔤^{∗}. Rather than writing “X_{1} = E_{1,5}, …, X_{10} = E_{4,5}” and listing jump sets e as subsets of {1, …, 10}, we will specify the basis vectors E_{i,j} that give the jump indices in each case. One has

E = {e_{1} ≺ e_{2} ≺ ⋯ ≺ e_{9}}

where:

e_{1} = {E_{1,4}, E_{2,5}, E_{1,3}, E_{3,5}, E_{1,2}, E_{2,3}, E_{3,4}, E_{4,5}},
e_{2} = {E_{1,4}, E_{2,5}, E_{1,3}, E_{3,5}, E_{1,2}, E_{4,5}},
e_{3} = {E_{1,3}, E_{2,4}, E_{3,5}, E_{1,2}, E_{2,3}, E_{3,4}},
e_{4} = {E_{1,3}, E_{2,4}, E_{1,2}, E_{3,4}},
e_{5} = {E_{2,4}, E_{3,5}, E_{2,3}, E_{4,5}},
e_{6} = {E_{1,2}, E_{2,3}, E_{3,4}, E_{4,5}},
e_{7} = {E_{2,3}, E_{3,4}},
e_{8} = {E_{3,4}, E_{4,5}},
e_{9} = Ø.

Writing elements ℓ ∈ 𝔤^{∗} as ℓ = ∑_{i<j} ℓ_{i,j}E_{i,j}^{∗}, the corresponding layers Ω_{e_{j}} are:

Ω_{e_{1}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} ≠ 0, ℓ_{1,4}ℓ_{2,5} − ℓ_{2,4}ℓ_{1,5} ≠ 0},
Ω_{e_{2}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} ≠ 0, ℓ_{1,4}ℓ_{2,5} − ℓ_{2,4}ℓ_{1,5} = 0},
Ω_{e_{3}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = 0, ℓ_{1,4}ℓ_{2,5} ≠ 0},
Ω_{e_{4}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = ℓ_{2,5} = 0, ℓ_{1,4} ≠ 0},
Ω_{e_{5}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = ℓ_{1,4} = 0, ℓ_{2,5} ≠ 0},
Ω_{e_{6}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = ℓ_{1,4} = ℓ_{2,5} = 0, ℓ_{1,3}ℓ_{3,5} ≠ 0},
Ω_{e_{7}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = ℓ_{1,4} = ℓ_{2,5} = ℓ_{1,3} = 0, ℓ_{2,4} ≠ 0},
Ω_{e_{8}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = ℓ_{1,4} = ℓ_{2,5} = ℓ_{1,3} = ℓ_{2,4} = 0, ℓ_{3,5} ≠ 0},
Ω_{e_{9}} = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = ℓ_{1,4} = ℓ_{2,5} = ℓ_{1,3} = ℓ_{2,4} = ℓ_{3,5} = 0}.

Orbits in Ω_{e_{1}} are 8-dimensional with one non-constant direction (E_{2,4}^{∗}). Orbits in Ω_{e_{2}} are 6-dimensional with three non-constant directions (E_{2,4}^{∗}, E_{2,3}^{∗}, E_{3,4}^{∗}). Orbits in Ω_{e_{3}} are 6-dimensional with one non-constant direction (E_{4,5}^{∗}). Orbits in Ω_{e_{4}} are 4-dimensional with one non-constant direction (E_{2,3}^{∗}). Orbits in Ω_{e_{5}} are 4dimensional with one non-constant direction (E_{3,4}^{∗}). Orbits in Ω_{e_{6}} are flat and 4-dimensional. Orbits in Ω_{e_{7}} are 2-dimensional with one non-constant direction (E_{4,5}^{∗}). Orbits in Ω_{e_{8}} are flat and 2-dimensional. Orbits in Ω_{e_{9}} = Ω_{Ø} are single points.

From above we see that the cross sections for the layers Ω_{e_{1}} and Ω_{e_{2}} are:

U_{e_{1}} = {aE_{1,5}^{∗} + bE_{2,4}^{∗} : a ≠ 0, b ≠ 0}, U_{e_{2}} = {aE_{1,5}^{∗} + bE_{2,3}^{∗} + cE_{3,4}^{∗} : a ≠ 0}.

One computes that the coadjoint orbit through a point f = aE_{1,5}^{∗} +bE_{2,3}^{∗} +cE_{3,4}^{∗} ∈ U_{e_{2}} (a ≠ 0) is

O_{f} = \{[ \begin{matrix}0 & x_{12} & x_{13} & x_{14} & a \\ 0 & 0 & b + \frac{x_{25}x_{13}}{a} & \frac{x_{14}x_{25}}{a} & x_{25} \\ 0 & 0 & 0 & c + \frac{x_{14}x_{35}}{a} & x_{35} \\ 0 & 0 & 0 & 0 & x_{45} \\ 0 & 0 & 0 & 0 & 0 \end{matrix} ] : x_{14}, x_{25}, x_{13}, x_{35}, x_{12}, x_{45} ∈ ℝ\} .

Note that both aE_{1,5}^{∗} +2E_{1,4}^{∗} +bE_{2,3}^{∗} +cE_{3,4}^{∗} and aE_{1,5}^{∗} +2E_{2,5}^{∗} +bE_{2,3}^{∗} +cE_{3,4}^{∗} belong to O_{f}. Hence

g = aE_{1,5}^{∗} + E_{1,4}^{∗} + E_{2,5}^{∗} + bE_{2,3}^{∗} + cE_{3,4}^{∗}
= \frac{1}{2}(aE_{1,5}^{∗} + 2E_{1,4}^{∗} + bE_{2,3}^{∗} + cE_{3,4}^{∗}) + \frac{1}{2}(aE_{1,5}^{∗} + 2E_{2,5}^{∗} + bE_{2,3}^{∗} + cE_{3,4}^{∗})
∈ Conv(O_{f}).

Since g ∈ Ω_{e_{1}} we see that

Conv(O_{f}) ⊄ [ ⋃_{e′⪰e(f)} Ω_{e′} = ⋃_{j=2}^{9} Ω_{e_{j}} ] .

Thus, Condition (C) fails for this example.

We note, moreover, that Ĝ fails to be moment separable in a rather spectacular fashion. The coadjoint orbit through a point g = aE_{1,5}^{∗} + bE_{2,4}^{∗} ∈ U_{e_{1}} (a ≠ 0, b ≠ 0) is

O_{g} = \{[ \begin{matrix}0 & x_{12} & x_{13} & x_{14} & a \\ 0 & 0 & x_{23} & b+ \frac{x_{14}x_{25}}{a} & x_{25} \\ 0 & 0 & 0 & x_{34} & x_{35} \\ 0 & 0 & 0 & 0 & x_{45} \\ 0 & 0 & 0 & 0 & 0 \end{matrix} ] : x_{14}, x_{25}, x_{13}, x_{35}, x_{12}, x_{23}, x_{34}, x_{45} ∈ ℝ\} .

From this we see that Conv(O_{g}) = {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = a} for all such g. (The key observation here is that O_{g} forms a saddle in the subspace spanned by {E_{1,4}^{∗}, E_{2,5}^{∗}, E_{2,4}^{∗}}.) But we saw above that for f = aE_{1,5}^{∗} + bE_{2,3}^{∗} + cE_{3,4}^{∗} ∈ U_{e_{2}} , Conv(O_{f}) meets Ω_{e_{1}} . Letting g ∈ Conv(O_{f}) ∩ Ω_{e_{1}} we have

{ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = a} = Conv(O_{g}) ⊂ Conv(O_{f}) ⊂ {ℓ ∈ 𝔤^{∗} : ℓ_{1,5} = a}.

This shows that for any point ℓ ∈ Ω_{e_{1}} ∪ Ω_{e_{2}} ,

Conv(O_{ℓ}) = {ℓ′ ∈ 𝔤^{∗} : ℓ′_{1,5} = ℓ_{1,5}}.

We remark that one can also find pairs of coadjoint orbits in each of the layers Ω_{e_{3}} , Ω_{e_{4}} and Ω_{e_{5}} whose convex hulls coincide.

Example 4.6. The aim here is to examine our results from the nilpotent situation in an exponential solvable case. Another aim is the following: In Remark 3.2 we mention that in [10], it is shown that any pair of two dimensional coadjoint orbits O_{1}, O_{2} is “convex distinguishable”. We are going to show that this does not hold in the more general context of exponential groups.

Let G be the 4-dimensional group with the Lie algebra 𝔤 having strong Malcev basis {Z, X, Y, A} where

[A, X] = −X, [A, Y] = Y, [X, Y] = Z,

and other brackets of basis elements vanish. This group is completely solvable and hence exponential solvable. Let {Z^{∗}, X^{∗}, Y^{∗}, A^{∗}} be the basis of 𝔤^{∗} dual to {Z, X, Y, A}. Let ℓ_{λ,ν,α,η} = λZ^{∗} + νX^{∗} + αY^{∗} + ηA^{∗} and

θ = exp(aA) · exp(xX) · exp(yY) · exp(zZ), a, x, y, z ∈ ℝ.

Then a routine calculation shows that:

Ad^{∗}(θ)ℓ_{λ,ν,α,η} = λZ^{∗} + e^{a}(ν + yλ)X^{∗} + e^{−a}(α − xλ)Y^{∗} + (η − xν + yα − xyλ)A^{∗}.

From this it follows that for λ ≠ 0, the orbit O_{(λ,ν,α,η)} through ℓ_{λ,ν,α,η} is given by:

O_{(λ,ν,α,η)} = \{λZ^{∗} + uX^{∗} + vY^{∗} + (\frac{uv − αν}{λ} + η) A^{∗} : u, v ∈ ℝ\} .

We see that for ℓ = λZ^{∗} + ηA^{∗} with λ ≠ 0 one has e(ℓ) = {2, 3} and ẽ_{2}(ℓ) = {4}. The map r_{ℓ} : V_{e(ℓ)} → V_{ẽ_{2}(ℓ)} is here

r_{ℓ}(u, v) = \frac{uv}{λ} A^{∗}.

We see that Conv(Graph(r_{ℓ})) = V_{e(ℓ)} ⊕ V_{ẽ_{2}(ℓ)}. In view of Theorem 3.6, we expect that Ĝ is not moment separable. This is indeed the case. In fact, the orbits O_{(λ,0,0,η)} for fixed λ ≠ 0 are not convex distinguishable.

Consider for example the following pair of coadjoint orbits:

O = O_{(1,0,0,0)} = {ℓ_{(1,u,v,uv)} : u, v ∈ ℝ},
O′ = O_{(1,0,0,1)} = {ℓ_{(1,u,v,uv+1)} : u, v ∈ ℝ}.

We see that O′ ≠ O since ℓ_{(1,0,0,1)} ∉ O. As ℓ_{(1,1,1,1)} ∈ O and ℓ_{(1,−1,−1,1)} ∈ O satisfy

\frac{1}{2}ℓ_{(1,1,1,1)} + \frac{1}{2}ℓ_{(1,−1,−1,1)} = ℓ_{(1,0,0,1)}

we have Conv(O′) ⊂ Conv(O). On the other hand

\frac{1}{2}ℓ_{(1,1,−1,0)} + \frac{1}{2}ℓ_{(1,−1,1,0)} = ℓ_{(1,0,0,0)}

with ℓ_{(1,1,−1,0)}, ℓ_{(1,−1,1,0)} ∈ O′ and thus Conv(O) ⊂ Conv(O′). This shows that Conv(O) = Conv(O′), so that O, O′ are a pair of (two dimensional) coadjoint orbits which are not convex distinguishable.

5. Concluding Remarks and Questions

For a given group G, the conditions in Theorems 3.5 and 3.6 are usually easy to check. This is illustrated by the examples described above in Section 4.. It would, however, be desirable to have a single condition which is both necessary and sufficient for moment separability. One might also hope to eliminate the use of Condition (C) as a hypothesis, although this seems difficult provided one sticks to the framework of Pukanszky polynomials. In this regard, we ask whether Ĝ necessarily fails to be moment separable whenever Condition (C) fails.

Finally, we remark that the closure in Equation 2 plays a mysterious role. In fact, we know of no example where the convex hull Conv(O) of some coadjoint orbit is not itself closed. This is a subtle issue because the convex hull for the graph of a polynomial function ℝ^{n} → ℝ^{m} can fail to be closed. (The polynomial p : ℝ^{2} → ℝ^{2} defined as p(x, y) = (x^{2}, x^{2}y^{2}) is one example.) In any case, we ask whether convex distinguishability implies moment separability. That is, if Conv(O) = Conv(O′) ⇒ O = O′ for all coadjoint orbits O, O′ ⊂ 𝔤^{∗} then is Ĝ moment separable?

References

[1] Arnal, D., and J. Ludwig, La convexité de l’application moment d’un groupe de Lie, J. Funct. Anal. 105 (1992), 256–300.

[2] Baklouti, A., J. Ludwig, and M. Selmi, Séparation des représentations unitaires irréductibles des groupes de Lie nilpotents, In Doebner, H-D., Hilgert, J., and Dobrev, V. editors, “Lie Theory and its Applications in Physics II,” 1998.

[3] Corwin, L., and F. Greenleaf, “Representations of nilpotent Lie groups and their applications, Part 1: Basic theory and examples,” volume 18 of Studies in Advanced Math., Cambridge Univ. Press, Cambridge, 1990.

[4] Kirillov, A. A., Unitary representations of nilpotent Lie groups, Russian Math. Surveys 17 (1962), 57–110.

[5] Ludwig, J., Good ideals in the group algebra of a nilpotent Lie group, Math. Z. 161 (1978), 195–210.

[6] Moore, C., and J. Wolf, Square integrable representations of nilpotent groups, Trans. Amer. Math. Soc. 185 (1973), 445–462.

[7] Nielsen, O., Unitary Representations and Coadjoint Orbits of Low-Dimensional Nilpotent Lie Groups, Queen’s Papers in Pure and Applied Mathematics, Queen’s University, Kingston, Ont. Canada, 1983.

[8] Pedersen, N., On the infinitesimal kernel of irreducible representations of nilpotent Lie groups, Bull. Soc. Math. France 112 (1984), 423–467.

[9] Pukanszky, L., “Leçons sur les représentations des groupes,” Dunod, Paris, 154 Baklouti, Benson, Ratcliff

[10] Wildberger, N., Convexity and unitary representations of nilpotent Lie groups, Invent. Math. 98 (1989), 281–292.

[11] Wildberger, N., The moment map for a Lie group representation, Trans. Amer. Math. Soc. 330 (1992), 257–268.

Ali Baklouti Faculté de Sciences de Sfax, Département de Mathématiques, 3038 Sfax, Tunisia, Ali.Baklouti@fss.rnu.tn Chal Benson Dept. Math. and Comp. Sci., Univ. of Missouri-St. Louis, St. Louis, MO 63121, U.S.A., benson@arch.umsl.edu Gail Ratcliff Dept. Math. and Comp. Sci., Univ. of Missouri-St. Louis, St. Louis, MO 63121, U.S.A., ratcliff@arch.umsl.edu

Received January 10, 2000 and in final form April 17, 2000