Journal of Lie Theory Volume 13 (2003) 77–89 ◯^{C}2003 Heldermann Verlag

Lattices and Harmonic Analysis on Some 2-Step Solvable Lie Groups

Huale Huang

Communicated by M. Moskowitz

Abstract. This paper deals with a class of 2-step solvable simply connected Lie groups G in which we construct certain lattices Γ. It turns out that these are all the lattices in such groups. We then apply our results to study the decomposition of the quasi-regular representation of G on G/Γ.

1. Introduction

Given a Lie group, it is often useful to have a parametrization of the set of all its lattices. In Euclidean space ℝ^{n}, for example, each lattice corresponds to a basis, and any lattice is equivalent to the standard integer lattice under an automorphism in GL(n, ℝ). In the nilpotent case, the lattices of the Heisenberg groups are classified, up to automorphisms, by certain sequences of positive integers with divisibility conditions (see [1]). In [7], R. Mosak and M. Moskowitz studied the set of lattices in a class of simply connected, solvable, but not nilpotent Lie groups G. Their construction of G depends on a diagonal n × n matrix Δ with distinct non-zero eigenvalues, of trace 0. In this environment they define the 1-parameter subgroup η(t) = e^{tΔ} in GL(n, ℝ) and G is the semi-direct product ℝ^{n} ×_{η} ℝ. The Lie group G is connected and simply connected, solvable, but not nilpotent. Because no d_{i} is 0, G has trivial center. Here among other things, we shall consider a more general setting. Let Δ be an n × n upper triangular matrix of trace 0 in 𝔤𝔩(n, ℝ), the full Lie algebra of real n × n matrices, with at least one non-zero element on the diagonal. As in [7] we can define a 1-parameter subgroup η(t) = exp(tΔ) in GL(n, ℝ) and construct a similar Lie group G. We define in G a class of distinguished lattices L(A, σ), for A in SL(n, ℤ), and for certain σ in GL(n, ℝ). In [7] it was proven (Theorem 1) that up to commensurability, every lattice in G differs from one of these by an automorphism of G, and two such lattices, L(A, σ) and L(B, τ) are equivalent by an automorphism of G if and only if A and B (or A and B^{−1}) are conjugate in GL(n, ℤ).

We will prove generalizations as well as strenghtenings of the results of [7]. In fact, we shall prove that up to isomorphism, those lattices comprise all the

ISSN 0949–5932 / $2.50 ◯^{C} Heldermann Verlag lattices in G (whereas in [7] this was done only up to commensurability.) Another result which we have strenghtened is Corollary 5 in [7] in which a tricotomy of possibilities is significantly reduced. Here we also mention a conjecture which we have proven in case n = 2, but not in general.

We then turn to some related questions of concerning the decomposition of the quasi-regular representation for such groups; that is where the group operates by right translation on the space of C^{∞} functions on the homogeneous space G/Γ, Γ a lattice in G. Here we show that when n = 2 the quasi-regular representation decomposes into a direct sum of indecomposable subrepresentations in such a way that although each of these indecomposable subrepresentation occurs with finite muliplicity, the multiplicity function itself is always unbounded.

Finally, the author would like to thank Prof. M. Moskowitz for proposing the problem and for various suggestions and advice given in the course of the preparation of this paper.

2. Construction and Classification of Lattices

In order to construct these lattices in G, we make the following

Definition 2.1. Let A be a matrix in SL(n, ℤ), and σ a matrix in GL(n, ℝ). We shall say that the pair (A, σ) is Δ-compatible if σ^{−1}Aσ is upper triangular and there exists a number g ∈ ℝ such that

σ^{−1}Aσ = exp(gΔ)

To construct our lattice, let the pair (A, σ) be Δ-compatible. We denote by L(A, σ) the semi-direct product σ^{−1}ℤ^{n} ×_{η} ℤg, which is a lattice in G. Indeed, η(g) = exp(gΔ) = σ^{−1}Aσ leaves L = σ^{−1}ℤ^{n} stable, since

η(g)L = σ^{−1}AσL = σ^{−1}Aℤ^{n} = σ^{−1}ℤ^{n} = L.

Consequently L(A, σ) is a subgroup in G, and is obviously discrete and cocompact. Our first result is that up to isomophism, the lattices L(A, σ) are all the lattices in G. In order to prove this fact, we begin with the definition of the roots of a solvable Lie algebra. Let 𝔤 be a solvable Lie algebra over the complex number field. By Lie’s Theorem, there exists a decreasing sequence 𝔤_{0}, 𝔤_{1}, ⋯, 𝔤_{n} of ideals of 𝔤, such that 𝔤_{0} = 𝔤, 𝔤_{n} = {0}, dim 𝔤_{i−1} −dim 𝔤_{i} = 1. The representation of 𝔤 on 𝔤_{i−1}/𝔤_{i} induced by the adjoint representation of 𝔤 gives rise to linear forms λ_{i} on 𝔤. These n forms are called the roots of 𝔤. Now suppose 𝔤 is a real Lie algebra, the roots of 𝔤 are by definition the restriction to 𝔤 of the roots of its complexification 𝔤^{ℂ}. Hence the values of λ_{1}, ⋯, λ_{n} for X of 𝔤 are the eigenvalues of ad(X). Let G be a simply connected solvable real Lie group, and 𝔤 its Lie algebra. We call G of real type, if for any i, all values of λ_{i} are real. For such Lie groups, Saito [10] proved the following rigidity theorem: (see also Corollary 11 of [6] for a generalization)

Theorem 2.2. Let G be a simply connected solvable real Lie group of real type, and Γ_{1} and Γ_{2} be two lattices in G. If they are isomorphic, then this isomorphism extends to an automorphism of G.

Now we can state our results. From now on, we let G = ℝ^{n} ×_{η} ℝ, where η(t) = exp(tΔ), and Δ is an upper triangular matrix in 𝔰𝔩(n, ℝ) with at least one nonzero element on the diagonal. Note that G is solvable of real type, but not nilpotent, since the roots of its Lie algebra 𝔤 are those elements on the diagonal of Δ together with 0. Hence all the roots of G are real and at least one root is nonzero, therefore 𝔤 is not nilpotent.

Theorem 2.3. Every lattice L in G is isomorphic to L(A, σ), for some Δcompatible pair (A, σ).

Proof. Suppose L is a lattice in G. Since ℝ^{n} is the nilradical of G, L ∩ ℝ^{n} is a lattice L in ℝ^{n} (see [9]), and L/L ∩ ℝ^{n} ≃ Lℝ^{n}/ℝ^{n} is a lattice of G/ℝ^{n} ≃ ℝ. Hence L/L ∩ ℝ^{n} is isomorphic to ℤ. Let g ∈ L/L ∩ ℝ^{n} ⊂ ℝ be a generator of the group. Then we have a split short exact sequence

[figure: \{1\}; L ∩ ℝ^{n}; i; L; π; \underline{s}; ℤg; \{1\}]

For any (x_{0}, g) ∈ L, define s(g) = (x_{0}, g). Then s(g)^{−1} = (−η(−g)x_{0}, −g). For any (x, 0) ∈ L, we have

s(g)(x, 0)s(g)^{−1} = (x_{0}, g)(x, 0)(−η(−g)x_{0}, −g)

= (x_{0} + η(g)x, g)(−η(−g)x_{0}, −g) = (η(g)x, 0).

Hence (η(g)x, 0) ∈ L. Similarly (η(−g)x, 0) ∈ L. Thus η(g)L = L. Since L is a lattice in ℝ^{n}, we can write L = σ^{−1}ℤ^{n}, for some σ ∈ GL(n, ℝ). So A = ση(g)σ^{−1} preserves ℤ^{n}. Since Δ is upper triangular and tr Δ = 0, det(A) = det(η(g)) = 1, so A is in SL(n, ℤ), and by construction, (A, σ) is compatible with Δ. Thus L ≃ L(A, σ). By Saito’s rigidity theorem, these lattices must differ by an automorphism of G.

The construction of the lattice L(A, σ) obviously depends on the choice of A, as well as the choice of diagonalizing matrix σ. However, if B in SL(n, ℤ) is conjugate to A or A^{−1} in GL(n, ℤ), and τ diagonalizes B, we shall show that the lattice L(B, τ) differs from L(A, σ) by an automorphism of G. More precisely, (see [7]):

Definition 2.4. Let A and B be elements of SL(n, ℤ). We say that A and B are extendedly conjugate if B is conjugate to A or A^{−1} in GL(n, ℤ).

Theorem 2.5. Let (A, σ) and (B, τ) be Δ-compatible pairs, where

σAσ^{−1} = exp(gΔ), τBτ^{−1} = exp(hΔ)

Then the lattices L(A, σ) and L(B, τ) differ by an automorphism of G iff A is extendedly conjugate to B in GL(n, ℤ).

Proof. Suppose φ ∈ Aut(G), and φ(L(A, σ)) = L(B, τ). Since φ maps the nilradical ℝ^{n} to itself, it takes σ^{−1}ℤ^{n} = L(A, σ) ∩ ℝ^{n} to τ^{−1}ℤ^{n} = L(B, τ) ∩ ℝ^{n} and the inverse φ^{−1} of φ takes τ^{−1}ℤ^{n} to σ^{−1}ℤ^{n}. Let α = φ|_{σ^{−1}ℤ^{n}}, then α^{−1} = φ^{−1}|_{τ^{−1}ℤ^{n}}. Hence α is an isomorphism and we have the following commutative diagram

[figure: \{1\}; σ^{−1}ℤ^{n}; i; L(A, σ); π; \underline{s}; ℤg; \{1\}; \{1\}; τ^{−1}ℤ^{n}; α; i′; L(B, τ); φ; π′; \underline{s′}; ℤh; γ; \{1\}]

It follows that γ must also be an isomorphism, and the splitting can be chosen so that this diagram is commutative also. Indeed, for any splitting s of π in the first row, we define s′(γ(g)) = φs(g). Then π′(s′(γ(g))) = π′(φs(g)) = (π′φ)s(g) = (γπ)s(g) = γ(πs)(g) = γ(g). Hence s′ is indeed a cross setion to π′. From the proof of Theorem 2, we know that η(g)x = s(g)xs(g)^{−1}. Hence we have

α(η(g)x) = φ(s(g)xs(g)^{−1}) = φ(s(g))φ(x)φ(s(g))^{−1}
= s′(γ(g))α(x)s′(γ(g))^{−1} = η(γ(g))α(x).

Thus αη(g) = η(γ(g))α, and since γ is an isomorphism, γ(g) = ±h. First we suppose that η(g) = h. Then αη(g) = η(h)α and since η(g) = σ^{−1}Aσ, and η(h) = τ^{−1}Bτ, it follows that ασ^{−1}Aσ = τ^{−1}Bτα, so τασ^{−1}A = Bτασ^{−1}. Let ρ = τασ^{−1} ∈ GL(n, ℝ). Then ρA = Bρ, since ρ : ℤ^{n} → ℤ^{n} is an isomprphism, we have ρ ∈ GL(n, ℤ). Similarly, if γ(g) = −h, we can prove that there exists an ρ′ such that ρ′A = B^{−1}ρ′, ρ′ ∈ GL(n, ℤ).

On the other hand, if A = ρBρ^{−1}, where ρ ∈ GL(n, ℤ), then ση(g)σ^{−1} = ρτη(h)τ^{−1}ρ^{−1}. Let α = τ^{−1}ρ^{−1}σ, then α : σ^{−1}ℤ^{n} → τ^{−1}ℤ^{n} is an isomorphism, and αη(g) = η(h)α. Hence for any k ∈ ℤ, αη(kg) = η(kh)α. Thus we can define a homormophism φ : σ^{−1}ℤ^{n} ×_{η} ℤg → τ^{−1}ℤ^{n} ×_{η} ℤh, by φ(x, kg) = (α(x), kh). Indeed for (x_{1}, k_{1}g) and (x_{2}, k_{2}g) in σ^{−1}ℤ^{n} ×_{η} ℤg, we have

φ((x_{1}, k_{1}g)(x_{2}, k_{2}g)) = φ(x_{1} + η(k_{1}g)(x_{2}), (k_{1} + k_{2})g)
= (α(x_{1}) + αη(k_{1}g)(x_{2}), (k_{1} + k_{2})h))
= (α(x_{1}) + η(k_{1}h)α(x_{2}), (k_{1} + k_{2})h),

and

φ(x_{1}, k_{1}g)φ(x_{2}, k_{2}g) = (α(x_{1}), k_{1}h)(α(x_{2}), k_{2}h)
= (α(x_{1}) + η(k_{1}h)α(x_{2}), (k_{1} + k_{2})h) = φ((x_{1}, k_{1}g)(x_{2}, k_{2}g)).

We have proven that φ is a homormophism. It is easy to see that φ is actually an isomorphism. Thus L(A, σ) ≃ L(B, τ). If A = ρB^{−1}ρ^{−1}, we proceed similarly. Again, by using Saito’s rigidity theorem, these lattices must differ by an automorphism of G.

3. Commensuralizers of Lattices

For a lattice L in a Lie group G, we shall denote by C(L) the commensuralizer of L, that is, the set of x ∈ G such that the conjugate L^{x} is commensurable with L, i.e. L ∩ L^{x} is also a lattice of G. ¿From now we will assume that all the elements on the diagonal of Δ are nonzero. In this more general setting, Theorem 4 and Corollary 5 in [7] remain valid. Furthermore we can use the corresponding proofs verbatim to get Theorem 3.1 and Corollary 3.2 below.

Theorem 3.1. Let L be a lattice in G. Then 1. The normalizer N(L) of L is also a lattice.

2. If (L_{i})_{i>1} denotes the increasing sequence of normalizers, L_{i} = N(L_{i−1}), with L_{0} = L, then

∪L_{i} ⊂ C(L).

3.If C(L) is discrete, then G contains a lattice which is its own normalizer.

Corollary 3.2. Let L be a lattice in G, and let C = C(L) be its commensuralizer. Then either C is discrete, or it is dense in G, or the closure of C in the Euclidean topology is the semi-direct product of the nil-radical, ℝ^{n} of G with a discrete subgroup of G.

Actually Corolary 3.2, can be strengthened by elimenating one of the possibilities. Namely, the discrete case can not occur when Δ is digonal. In fact we have the following

Theorem 3.3. Let L be a lattice in G, and C = C(L) be its commensuralizer. Then either C is dense in G, or the closure of C in the Euclidean topology is the semi-direct product of the nil-radical ℝ^{n} of G with a discrete subgroup of ℝ. Furthermore, if n=2, C must be dense in G.

Proof. First, by Theorem 2.3, we may suppose that L = L(A, σ), where the pair (A, σ) is Δ-compatible, i.e. σ^{−1}Aσ = D(λ_{1}, ⋯, λ_{n}). Now L(A, σ) = σ^{−1}ℤ^{n} ×_{η} ℤg, so for any (x_{1}, ⋯, x_{n}) ∈ σ^{−1}ℤ^{n} and k ∈ ℤ,

η(kg)(x_{1}, ⋯, x_{n}) = (λ_{1}^{k}x_{1}, ⋯, λ_{n}^{k}x_{n}) ∈ σ^{−1}ℤ^{n}

also. For any (x_{1}, ⋯, x_{n}) ∈ σ^{−1}ℤ^{n} and k, l ∈ ℤ with k ≠ 0, we claim that

X = (\frac{λ_{1}^{l}x_{1}}{1 − λ_{1}^{k}} , ⋯, \frac{λ_{n}^{l}x_{n}}{1 − λ_{n}^{k}} , 0) ∈ C.

Since for any m ∈ ℤ, (x, 0)(y, mkg)(x, 0)^{−1} = (x − η(mkg)(x) + y, mkg), it is easy to see that L^{X} ⊃ σ^{−1}ℤ^{n} ×_{η} ℤkg. Hence

L ∩ L^{X} ⊃ σ^{−1}ℤ^{n} ×_{η} ℤkg,

and X ∈ C. For any given ϵ > 0, we can choose l sufficiently large so that |λ_{i}^{l}x_{i}| < ϵ, if λ_{i} < 1. Then we can choose k large enough, so that λ_{i}^{l−k} < ϵ if λ_{i} > 1 and λ_{i}^{k} < ϵ if λ_{i} < 1. It follows that (0, ⋯, 0) can be approximated by points in C. Thus the discrete case can not occur. By Corollary 3.2, we have proven the first part of this theorem.

Now we consider the case n = 2, with no loss of generality, we can suppose that Δ = D(1, −1), (since Δ = D(d, −d) for some d ∈ ℝ and d ≠ 0) and for any matrix

A = ( \begin{matrix}a & b \\ c & k − a \end{matrix} ) ∈ SL(2, ℤ),

with trace(A) > 2, we have a lattice L = L(A, σ) = σ^{−1}ℤ^{2} ×_{η} ℤg. Construct the matrices

A_{m} = ( \begin{matrix}\frac{2m(2a−k)+m^{2}+k^{2}−4}{m^{2}−k^{2}+4} & \frac{4mb}{m^{2}−k^{2}+4} \\ \frac{4mc}{m^{2}−k^{2}+4} & \frac{m^{2}+k^{2}−4−2m(2a−k)}{m^{2}−k^{2}+4} \end{matrix} )

such that A_{m} ∈ SL(2, ℚ) and A commutes with all A_{m}. Let λ^{±1} be the eigenvalues of A, and λ^{±h_{m}} the eigenvalues of A_{m}. We claim that the lattice (0, h_{m} ln λ)L(0, h_{m} ln λ)^{−1} is commensurable with L. Since if (x, kg) ∈ L, we know that

(0, h_{m} ln λ)(x, kg)(0, h_{m} ln λ)^{−1} = (η(h_{m} ln λ)x, kg) = (σ^{−1}A_{m}σx, kg).

Therefore η(g) leaves σ^{−1}A_{m}σ(σ^{−1}ℤ^{2}) = σ^{−1}A_{m}ℤ^{2} stable. Hence η(g) leaves σ^{−1}ℤ^{2} ∩ σ^{−1}A_{m}ℤ^{2}, which is isomorphic to ℤ^{2}, stable also. This means that

σ^{−1}ℤ^{2} ∩ σ^{−1}A_{m}ℤ^{2} ×_{η} ℤg

is a lattice, which is contained in both (0, h_{m} ln λ)L(0, h_{m} ln λ)^{−1} and L(A, σ). We see easily that lim_{m→∞} h_{m} = 0, so C is dense in G.

¿From the above proof, it follows that the commensuralizer, C, of a lattice L(A, σ) for a compatible Δ-pair (A, σ) depends on those matrices over ℚ lying on the one parameter subgroup η(t) = exp(tΔ). Actually we have the following

Proposition 3.4. Let Δ be an upper triangular matrix in 𝔰𝔩(n, ℝ), with distinct nonzero elements in the diagonal, let η(t) = exp(tΔ), and (A, σ) be a Δcompatible pair. If σ^{−1}Aσ = D(λ_{1}, ⋯, λ_{n}), and

H = {h ∈ ℝ : λ_{i}^{h} = a_{0} + a_{1}λ_{i} + ⋯ + a_{i}λ_{i}^{n−1}},

where a_{i} ∈ ℚ, for 0 < i ≤ n, then H is a subgroup of ℝ containing ℤ .

Proof. First we know that λ_{1}, ⋯, λ_{n} are distinct. Hence the representation of the expression in our proposition is unique. Let f(x) be the characteristic polynomial of A, and g(x) = a_{0} + a_{1}x + ⋯ + a_{n−1}x^{n−1}, where a_{i} ∈ ℚ, g(λ_{i}) = λ_{i}^{h} for any 1 ≤ i ≤ n. Since (f(x), g(x)) = 1, there exist polynomials u(x) and v(x) ∈ ℚ[x] and degree of v(x) < n such that f(x)u(x) + g(x)v(x) = 1. So λ_{i}^{−h} = v(λ_{i}), and −h ∈ H. If h_{1} and h_{2} in H, then there exist u(x) and v(x) ∈ ℚ[x] such that λ_{i}^{h_{1}} = u(λ_{i}) and λ_{i}^{h_{2}} = v(λ_{i}). Let g(x) be the remainder of u(x)v(x) divided by f(x). Then g(x) ∈ ℚ(x) and g(λ_{i}) = λ_{i}^{h_{1}+h_{2}} . Hence h_{1} + h_{2} ∈ H, and H is a subgroup.

Remark 3.5. If we could prove that for a given A, H is not a discrete subgroup of ℝ, then the commensuralizer C of L(A, σ) would actually be dense in G.

As a consequence of our present results we now make the following:

Conjecture 3.6. The commensuralizer C(L) of any lattice L in G is dense in G.

As in [7] (Proposition 6), we have the following:

Proposition 3.7. If the pair (A, σ) is Δ-compatible, then N(L(A, σ)) is of the form L(B, τ) for some Δ-compatible pair (B, τ). More specifically, B can be chosen so that B^{p} = A for some integer p > 0, and τ = (I − A)σ.

By Theorem 2.5, it would be interesting to find when two matrices are conjugate by a matrix of GL(n, ℤ). First they must be similar. We consider the case n = 2. Here two such matrices are similar if and only if they have the same trace. It is natural to ask if they are also conjugate by a matrix in GL(2, ℤ). Unfortunately the answer to this question is no. For let

A = ( \begin{matrix}a & b \\ c & k − a \end{matrix} ) B = ( \begin{matrix}0 & 1 \\ −1 & k \end{matrix} )

be two matrices in SL(2, ℤ) with k > 2. Then we have the following result.

Theorem 3.8. Let A and B be as above. If b = ±1 or c = ±1, then A and B are conjugate by a matrix in GL(2, ℤ).

Proof. First we suppose that b = ±1. It is easily to check that

X = ( \begin{matrix}x & 2y \\ ax + 2cy & bx + 2(k − a)y \end{matrix} )

satisfies XA = BX, and

det(X) = bx^{2} + 2(k − 2a)xy − 4cy^{2} = \frac{(bx + (k − 2a)y)^{2} − (k^{2} − 4)y^{2}}{b} .

We know that the Pell equation x^{2} − (k^{2} − 4)y^{2} = 1 always has infinitely many integer solutions (see [8]). Hence there is a X ∈ GL(2, ℤ) such that XA = BX, i.e. A and B are conjugate by a matrix in GL(2, ℤ). For the case c = ±1, we proceed similarly, we only need to replace x by 2x and 2y by y.

On the other hand, we can find two matrices with same trace, which are not conjugate by any matrix in GL(2, ℤ). Let

A = ( \begin{matrix}2m + 1 & 2m \\ 2m + 2 & 2m + 1 \end{matrix} ) B = ( \begin{matrix}0 & 1 \\ −1 & 4m + 2 \end{matrix} ) ,

where m ∈ ℤ and m > 0. We have the following:

Proposition 3.9. A and B are not extendedly conjugate.

Proof. Suppose that A and B are conjugate by a matrix

X = ( \begin{matrix}x & y \\ z & w \end{matrix} ) ∈ GL(2, ℤ),

i.e. AX = XB. One see easily that y = −(2m + 1)x − 2mz and w = −(2m + 2)x − (2m + 1)z, so det(X) = −(2m + 2)x^{2} + 2mz^{2} is an even integer. Hence X is not in GL(2, ℤ). But

B^{−1} = ( \begin{matrix}4m + 2 & −1 \\ 1 & 0 \end{matrix} ) ,

and by Theorem 3.8, we know that B and B^{−1} are conjugate in GL(2, ℤ). Hence A and B^{−1} are not conjugate in GL(2, ℤ) either.

4. Decomposition of the Quasi-regular Representation

We now turn to some related questions of representation theory for such groups. The representations R we shall consider are the so called quasi-regular representations. These are the ones where the group operates by right translation on some space of functions on the homogeneous space G/Γ, for a lattice Γ in G, as above. Thus R_{g}(f)(x) = f(xg), where g ∈ G, x ∈ G/Γ and f is a real or complex function on G/Γ. In the case that the function space is L^{2}(G/Γ) such representations for nilpotent Lie groups were first studied by C.C. Moore in [4] and also by L. Corwin and F. Greenleaf in [3], where the latter two authors got a formula to compute the multiplicity of every irreducible subrepresentation. In the case the function space is C^{∞}(G/Γ) the problem of decomposition and calculation of multiplicity of the right regular representation was done for certain solvable groups by J. Brezin in [2]. Actually, in [2] when n = 2 the author proves the multiplicity function (see below) is unbounded in certain cases by looking at its average. In what follows we shall extend this result of Brezin, but without averages, to all the groups we have been considering, but also when n = 2 by estimating the multiplicity of an indecomposable representation.

On pg. 27 of [2] the author conjectures that the multiplicity function will be unbounded whenever the solvmanifold is non-degenerate, in the sense that its fundamental group does not contain any normal abelian subgroup of finite index. Since, if our lattice contains such an abelian subgroup, then this subgroup is also a lattice of G, in view of the density theorem of Mosak and Moskowitz see [5], G would be abelian. Therefore this non-degeneracy condition is always satisfied for the groups we consider. Hence, in our Theorem below, we are proving special cases of this conjecture. See below for the relevant definitions.

Theorem 4.1. When n = 2, mult(ω) is an unbounded function on Ω.

Actually, our Theorem 4.1 is likely to be true for arbitrary n. The proof of such a result would depend, among other things, on generalizing Theorem 3.3 to higher n. This would, if true, yield a proof of the conjecture of Brezin in all these additional cases.

As a consequence of Theorem 2.3 above we can now replace our upper triangular matrix Δ by any matrix in 𝔰𝔩(k, ℝ) with only real roots and hencefourth we shall assume this has been done. Actually, a refinement of this statement can be made. It is the following

Proposition 4.2. Let L(A, σ) be a lattice of G = ℝ^{k} ×_{η} ℝ, then there exists an isomorphism of G to G′ such that G′ has a lattice of ℤ^{k} ×_{η′} ℤ, where η′(1) = A.

Proof. let η′(t) = ση(gt)σ^{−1}, then we have the following commutative diagram

[figure: R^{n}; σ; R^{n}; R^{n}; η(tg); σ; R^{n}; η′(t)]

and η′(1) = A. Let G′ = ℝ^{k} ×_{η′} ℝ, then G′ is isomorphisic to G, and ℤ^{k} ×_{η′} ℤ is a lattice of G′.

By Propositon 4.2, we can assume without loss of generality that Γ is the lattice, ℤ^{k} ×_{η} ℤ. Thus, from now on, we shall always consider Γ to be the integer points of our linear Lie group. We shall take for the space of functions C^{∞}(G/Γ), with the Frechet semi-norm topology. This means that a sequence of functions converges if all derivatives of all orders converge uniformly on the compact space G/Γ.

Since the group G = ℝ^{k} ×_{η} ℝ, if f ∈ C^{∞}(G/Γ), then for each fixed t ∈ ℝ, the function \underline{u} → f(\underline{u}, t) on ℝ^{k} is periodic with respect to ℤ^{k}. This is because \underline{u} ∈ ℝ^{k}, \underline{n} ∈ ℤ^{k}, and t ∈ ℝ imply

f(\underline{u} + \underline{n}, t) = f((\underline{n}, 0) · (\underline{u}, t) = f(\underline{u}, t).

Hence, for each fixed t ∈ ℝ, we can expand f(·, t) in Fourier series

f(\underline{u}, t) = ∑_{\underline{n}∈ℤ^{k}} a\underline{n}(t)e(< \underline{n}, \underline{u} >),

in which < \underline{n}, \underline{u} >= n_{1}u_{1} + n_{2}u_{2} + ⋯ + n_{k}u_{k} and e is the function t → exp(2πit) on ℝ. For the functions a\underline{n}(t), we have the following

Proposition 4.3. The function a\underline{n} on ℝ is a bounded C^{∞} function. Furthermore,

a_{\underline{n}η(m)}(t) = a\underline{n}(t + m)

for all m ∈ ℤ and t ∈ ℝ.

Proof. We write a\underline{n} in terms of f as an integral over a fundamental domain,

a\underline{n}(t) = ∫_{0}^{1} ⋯ ∫_{0}^{1} f(\underline{u}, t)e(\underline{n}, −\underline{u})du_{1} ⋯ du_{n}.

It follows that a\underline{n} is C^{∞} and that ||a\underline{n}||_{∞} ≤ ||f||_{∞}.

The relation between a\underline{n} and a_{\underline{n}η(m)}(t) follows from the periodicity of f with respect to the elements (\underline{0}, m) of Γ. Because of left invariance we have

f(\underline{u}, t) = f((\underline{0}, m)(\underline{u}, t)) = f(η(m)\underline{u}, t + m),

which in turn implies that

∑_{\underline{n}} a\underline{n}(t + m)e(< \underline{n}, η(m)\underline{u} >) = ∑_{\underline{n}} a\underline{n}(t)e(\underline{n}, \underline{u}).  (1)

Since < \underline{n}, η(m)\underline{u} >=< \underline{n}η(m), \underline{u} >, we get our formula, by comparing the coefficents both sides of (1).

As a consequence of Proposition 4.3, if a\underline{n} = 0, then a\underline{m} = 0, whenever \underline{m} is in the η(ℤ)-orbit of \underline{n}. Let Ω denote the family of all the η(ℤ) orbits of points of ℤ^{k}. For each S ⊂ Ω, let Z(S) denote those f ∈ C^{∞}(G/Γ) for which a\underline{m} = 0 whenever \underline{m} is not in S. Just as in [2] Proposition 1.8, we have the following

Proposition 4.4. Z(S) is an R−invariant subspace of C^{∞}(G/Γ).

Now among the subspaces Z(S), those of the form Z(ω) for some ω ∈ Ω are minimal. These minimal subspaces also span C^{∞}(G/Γ) in the sense that their closed linear span in the C^{∞} topology is the whole space.

In the following proposition f_{ω} will denote the partial sum

∑_{\underline{n}∈ω} a\underline{n}(t)e(< \underline{n}, \underline{u} >).  (2)

Proposition 4.5. The quasi regular representation decomposes uniquely as a direct sum

C^{∞}(G/Γ) = ∑ ⊕_{ω∈Ω}Z(ω),

in the sense that every f ∈ C^{∞}(G/Γ) can be written in precisely one way as a convergent sum ∑_{ω∈Ω} f_{ω} with f_{ω} ∈ Z(ω).

Proof. It is obvious that the summands f_{ω} are unique, if they exist. The main point is to check that the function f_{ω} is in Z(ω) and that the series ∑_{ω} f_{ω} converges to f (with any ordering of Ω). Now f_{ω} is in Z(ω) because ω is a η(ℤ)- orbit in ℤ^{k}. Since, for each fixed t ∈ ℝ, f(., t) is a C^{∞} function on the k-torus T^{k}, the sum in (2) converges in the C^{∞} topology on C^{∞}(T^{k}) and therefore, can be differentiated term by term with respect to the variables u_{1}, ⋯, u_{k} as often as desired.

Recall that a\underline{n} is gotten by integrating:

a\underline{n}(t) = ∫_{0}^{1} ⋯ ∫_{0}^{1} f(\underline{u}, t)e < \underline{n}, \underline{u} > du_{1} ⋯ du_{k}.  (3)

Let Δ denote the Laplacian \frac{∂^{2}}{∂_{u_{1}}} + ⋯ + \frac{∂^{2}}{∂_{u_{k}}} . Integrating by parts, we see that (3) implies that the l-th derivative a(l) ; \underline{n}(t) of a\underline{n} with respect to t must satisfy

|a(l) ; \underline{n}(t)| ≤ (4π^{2}(n_{1}^{2} + ⋯ + n_{k}^{2}))^{−h} ∫_{0}^{1} ⋯ ∫_{0}^{1} |∂_{t}^{l}Δ^{h}f(\underline{u}, t)|du_{1} ⋯ du_{k},  (4)

for all integers h ≥ 0. If we set C_{h,l} = sup_{0≤u_{1},···,u_{k}≤1}(4π^{2})^{−h}|∂_{t}^{l}Δ^{h}f(\underline{u}, t)|, then (4) can be rewritten as

sup_{0≤t≤1} |a(l) ; \underline{n}(t)| ≤ C_{h,l} < \underline{n}, \underline{n} >^{−h} .  (5)

Using (5) with h = a_{1}+a_{2}+⋯+a_{k} +k, we see that the mixed partial ∂_{t}^{j}∂_{u_{1}}^{a_{1}} ⋯ ∂_{u_{k}}^{a_{k}} of the sum in (2) can be evaluated by term by term differentiation. Hence (2) defines a C^{∞} function on G/Γ. If we let f_{ω} denote that function, then from (5) we have for some C > 0,

sup_{0≤u_{1},···,u_{k}≤1} |\frac{∂^{j}f_{ω}(\underline{u}, t)}{∂_{u_{1}}^{a_{1}} ⋯ ∂_{u_{k}}^{a_{k}}∂_{t}^{i}} | ≤ C ∑_{\underline{n}} < \underline{n}, \underline{n} >^{j−h},  (6)

where j = i + a_{1} + ⋯ + a_{k}. Taking h − j ≥ \frac{m+1}{2} and summing (6) over all ω ∈ Ω, we see that, independent of the order of summation, the series ∑_{ω∈Ω} f_{ω} converges in the C^{∞}-topology to f.

Since the representation we are considering is on a Frechet space, not on a Hilbert space, we can only decompose the representation as a direct sum of indecomposable sub-representations. Let’s recall that an R-invariant subspace V of the space C^{∞}(G/Γ) is indecomposable if it can not be decomposed into two nontrival R-invariant subspaces. This is weaker than the notion of irreducible representation. (An irreducible subspace does not have any nontrival R-invariant subspace.) By Corollary 1.39 in [2], Z(ω) is an indecomposable R-invariant subsapce, when n = 2.

Definition 4.6. Let ω_{1} and ω_{2} be two elements in Ω, if there exists an f ∈ Hom_{R}(Z(ω_{1}), Z(ω_{2})), and f is an isomorphism as linear map, then we call Z(ω_{1}) and Z(ω_{2}) isomorphic. For any ω ∈ Ω, we define the multiplicity, mult(ω) of Zω, to be the number of all the subspaces Z(ω′) isomorphic to Z(ω).

Remark 4.7. By Proposition 1.44 of [2], we see that, when n = 2 mult(ω) is finite .

Our purpose now is to show that when n = 2, the multiplicities mult(ω) in Z(ω) are unbounded. A very particular case of this result was proven in [2]. More precisely, there only the case where A = η(1) is of the form

X = ( \begin{matrix}a & b \\ Db & a \end{matrix} ) ∈ SL(2, ℤ),

where D is a square-free positive integer congruent to 2 or to 3 (mod 4) and a^{2} − Db^{2} = 1 is treated. We now turn to the proof of Theorem 4.1.

Proof. (Of Theorem 4.1 ) Here we use the same notation as in the proof of Theorem 3.3. A = η(1),

( \begin{matrix}λ^{g} & 0 \\ 0 & λ^{−g} \end{matrix} ) = ση(g)σ^{−1},

where σ diagonalizes A and A_{m} and h_{m} have the same meaning as in the proof of Theorem 3.3. Since h_{m} → 0, for any given posive integer number N, there exists an number m such that

0 < |Nh_{m}| < |g|.

We claim that

((m^{2} − k^{2} + 4)^{N}, (m^{2} − k^{2} + 4)^{N})η(ih_{m})η(ℤ) ⊂ ℤ^{2}(0 ≤ i ≤ N)

are disjoint orbits under the action of η(ℤ). Indeed, if

((m^{2} − k^{2} + 4)^{m}, (m^{2} − k^{2} + 4)^{m})η(ih_{m})η(ℤ)

and

((m^{2} − k^{2} + 4)^{m}, (m^{2} − k^{2} + 4)^{m}))η(jh_{m})η(ℤ)

are in the same η(ℤ) orbit, then there exists an integer number k such that

((m^{2} − k^{2} + 4)^{m}, (m^{2} − k^{2} + 4)^{m})η(kg)η(ih_{m}) =
((m^{2} − k^{2} + 4)^{m}, (m^{2} − k^{2} + 4)^{m})η(jh_{m}).

Since

( \begin{matrix}λ^{g} & 0 \\ 0 & λ^{−g} \end{matrix} ) = ση(g)σ^{−1},

we have kg = (j − i)h_{m}(−N ≤ j − i ≤ N). But, this can be true only when k = 0, and i = j. However, these are obviously in the same η(ℝ) orbit. Hence, by Corollary 1.35 of [2], the multiplicity of ((m^{2} − k^{2} + 4)^{N}, (m^{2} − k^{2} + 4)^{N}) is at least N + 1.

References

[1] Auslander, L., Lecture Notes On Nil-Theta Functions, CMBS Reg. Conf. Series Math. 34, Amer. Math. Soc., Providence, 1977.

[2] Brezin, J., Harmonic Analysis on Compact Solvmanifolds, Springer Lecture Notes 602, 1977.

[3] Corwin, L., and F. Greenleaf, Character formulas and spectra of compact nilmanifolds, J. Funct. Anal. 21 (1976) 123–15.

[4] Moore, C. C., Decomposition of Unitary Representations defined by discrete subgroups of nilpotent groups, Ann. of Math. 82 (1965), 146–182.

[5] Mosak, R., and M. Moskowitz, Zariski Density in Lie Groups, Israel Journal of Mathematics 52, (1985), 1–14.

[6] —, Analytical Density in Lie Groups, Israel Journal of Mathematics 58 (1987), 1–9.

[7] —, Lattices in a Split Solvable Lie Group, Math. Proc. of Cambridge Philos. Soc. 122 (1997), 245–250.

[8] Niven, I., and H. Zuckerman, “An Introduction to the Theory of Numbers,” John Wiley & Sons, New York, 1972.

[9] Raghunathan, M. S., “Discrete Subgroups of Lie Groups,” Springer-Verlag, Berlin, 1972.

[10] Saito, M., Sur Certains Groups de Lie Resolubles II, Sci. Papers College Gen. Ed. Univ. Tokyo, 7 (1957), 157–168.

Huale Huang AT&T Labs-Research 180 Park Ave Building 103, Room C258 Florham Park, NJ 07932 email:huale@research.att.com

Received September 21, 2001 and in final form October 24, 2001