Journal of Lie Theory Volume 9 (1999) 461–480 \begin{matrix}c \\ ◯ \end{matrix} 1999 Heldermann Verlag Berlin

Square-integrablity of tensor products

Matthias Mayer

Communicated by A. Valette

Abstract. This paper is concerned with C_{0}−representations of locally compact groups. The focus is on the relationship between the C_{0}−property and square-integrability, the latter meaning that the representation is quasi-equivalent to a subrepresentation of the regular one. We show that for certain real algebraic groups every C_{0}−representation has a square-integrable tensor power and discuss some classes of groups enjoying this property. We point out to which extent this result supports a conjecture of Figà-Talamance and Piccardello concerning the radical of the Fourier algebra in the Fourier-Stietjes algebra. Finally, we give a simple criterion for a C_{0}−representation to be square-integrable.

1. Introduction

In this paper we are concerned with asymptotic properties of (strongly continuous) unitary representations (π, H_{π}) of a locally compact group G and with their relations to square-integrability. By the latter we mean that there is a dense subspace D of the representation space H_{π} , such that for all ξ, η ∈ D the matrix coefficient

φ_{ξη}: G → ℂ, g ↦ ⟨π(g)ξ, η⟩

is in L^{2}(G) . Square-integrable representations have been extensively studied, and it follows from the results of Rieffel [40], Duflo–Moore [15] and others that a representation (π, H_{π}) is square-integrable if and only if it is quasi-equivalent in the sense of ([14],5.3.1) to a subrepresentation of the

regular representation (λ_{G}, L^{2}(G)) . We write π ≤^{q} λ_{G} . Beyond the classical theory, square-integrable representations recently gained interest, since they

ISSN 0949–5932 / $2.50 \begin{matrix}c \\ ◯ \end{matrix} Heldermann Verlag are the basic tool for the construction of continuous wavelet transforms from cyclic representations, see e.g., [29, 31, 25].

Thus it is highly desirable to be able to construct square-integrable representations or to decide when a given representation (π, H_{π}) is squareintegrable and we present results concerning both questions (see Theorems 1.1 and 1.2).

Square-integrable representations are C_{0} –representations, which means that all matrix coefficients of π vanish at infinity, but not vice versa. The main theorem of this paper shows that, for certain real algebraic groups, every C_{0} –representation has a square integrable tensor power.

To be more precise, recall that a connected real algebraic group G has a unique largest unipotent radical N , and decomposes as

G = N ⋊_{φ} H ,

where H is a reductive Levi-complement of N . N and H are Zariskiclosed, N is simply connected with respect to the topology induced by GL(n, ℂ) and H acts algebraically and reductively on the Lie algebra 𝔫 by the derived representation. The centralizer in G of a subset S ∈ G is denoted by Z(S, G) .

Theorem 1.1. Let G be a connected real algebraic group and keep the above notation. Suppose that

(i) Z(H, G) ∩ N = {e} ,

(ii) Z(Z, G) ∩ H is compact, where Z is the center of N .

Then there exists a finite k_{0} ∈ ℕ such that for all k ≥ k_{0} and for all representations (π, H_{π}) whose subrepresentations all have compact kernel

π^{⊗k} ≤^{q} λ_{G} .

Our second theorem gives a criterion for square-integrability of C_{0} – representations. Before formulating the theorem, let us recall that any irreducible representation (π, H_{π}) of a locally compact group G which contains a regularly embedded normal subgroup N is of the following form

(π, H_{π}) = ind_{Gχ̃}^{G}χ .

Here, χ̃ is an irreducible representation of N , Gχ̃ is the group of elements in G which preserve χ̃ under conjugation and χ is an irreducible representation of Gχ̃ such that χ|_{N} is a multiple of χ̃ .

Theorem 1.2. Let N be a regularly embedded simply connected nilpotent normal subgroup of a Lie group G . Let (π, H_{π}) be a unitary representation of G and let π = ∫ ⊕ ; Ĝ m(ϱ)ϱdμ(ϱ) be a decomposition in irreducibles. Then

π ≤^{q} λ_{G} provided that

(i) for μ–almost all ϱ = ind_{Gχ̃}^{G}χ , the fixed group Gχ̃ of the character χ̃ , which χ|_{N} is a multiple of, is a compact extension of N;

(ii) for μ–almost all ϱ the G–orbit of the Kirillov-orbit associated to χ̃ is open in 𝔫^{∗} .

Let us discuss these theorems. In Theorem 1.1, assumption (i) is a bit stronger than the requirement that N does not contain nontrivial subgroups which are central in G (such a group would be non-compact). As to the second requirement, identify G with the outer semidirect product N ⋊_{φ} H and observe that we may split off the kernel of φ from H , yielding

G = (N ⋊ H/ ker φ) × ker φ ,

(at least modulo finite subgroups). See ([27],p.87) for more details. Since the property, that any representation as in the theorem has a square-integrable tensor power, carries over to direct products and finite extensions, we may consider both factors separately. Then ker φ will enjoy this property if its center is compact and it is a Kazhdan group, by well-known results of Cowling [11] and Moore [38]. Our result 1.1 applies to the first factor if the quotient (ker(φ |_{Z}))/(ker φ) is compact. If this requirement is not satisfied one may try to embed H/ ker φ in an appropriate Sp(n, ℝ) and to use a result of Howe and Moore [27]. We discuss some examples in Section 3.

In any case, we may apply Theorem 1.1 to the groups G: = V ⋊ H , where V is a vector space and H the group of diagonal matrices or a semisimple Kazhdan group acting on V without nontrivial fixed points. We will give more elaborate examples in Section 2.

Several examples of groups are known such that, for every C_{0} – representation (π, H_{π}) , a sufficiently large tensor power is square-integrable: This follows easily for semisimple Kazhdan groups from the results of Cowling [11] and Moore [38], for generalized motion groups from those of Liukonnen and Mislove [33, 34] and for G: = ℝ^{2} ⋊ SL(2, ℝ) from [37]. By way of contrast, on groups with non-compact center and on non-compact nilpotent groups there exist C_{0} –representations whose tensor powers are far from being square-integrable. This was shown by Figá-Talamanca and Picardello generalizing methods of Varopoulos [23].

Observe that we do not require the representation π in Theorem 1.1 to be a C_{0} –representation, this is a part of the result. The fact that representations whose subrepresentations all have compact kernel are C_{0} is of considerable interest in many areas, e.g., in ergodic theory ([7], [8]). This C_{0} –property is known for semisimple groups with finite center by the Howe– Moore theorem [27] and for connected so called totally minimal groups, treated below [35, 36].

The proof of Theorem 1.1 is based on the one given by Howe and Moore for the fact that irreducible representations of algebraic groups have a tensor power which is square-integrable modulo the projective kernel ([27],6.1). A somewhat different approach would use the methods in [33, 34], after getting rid of the compactness assumption made there.

Theorem 1.1 also bears significance to a conjecture by Figá-Talamanca and Picardello concerning Fourier-Stieltjes algebras. The Fourier-Stieltjes algebra B(G) is one of the most fruitful constructions of a dual object for a general locally compact group G , due to P. Eymard [16]. B(G) is the space of all matrix coefficients of strongly continuous unitary representations of G . Using sums, tensor products and contragredient representations, it is easily seen that B(G) carries the pointwise structure of an involutive commutative algebra. In addition, B(G) may be identified with the dual space of the group C^{∗}–algebra C^{∗}(G) via

φ_{ξη} ↦ (C^{∗}(G) ∋ T ↦ ⟨π(T)ξ, η⟩) ,

where (π, H_{π}) denotes also the lifting to C^{∗}(G) and ξ, η are vectors in H_{π} . The dual space norm makes B(G) an involutive (semisimple) Banach algebra which the group G acts upon by left and right translations. The matrix coefficients of the regular representation form a closed ideal, the Fourier algebra A(G) . It coincides with the closure of all coefficient functions with compact support in B(G) . For abelian G , Bochner’s theorem identifies B(G) with the measure algebra M(Ĝ) on Ĝ and A(G) with L^{1}(Ĝ) . For all these facts, see [16, 17].

More generally, denote by A_{π} the closure in B(G) of all coefficients of a unitary representation (π, H_{π}) . By Arsac’s theory ([1],3.1.II), A_{π} characterizes (π, H_{π}) up to quasi-equivalence. If

B_{0}(G) : = {ψ ∈ B(G)| ψ vanishes at infinity} ,

the above question on the square-integrability of tensor products is related to the question on the relationship between B_{0}(G) and the radical A_{r}(G) of the Fourier algebra, that is

A_{r}(G) = {ψ ∈ B(G)| ∃k ∈ ℕ : ψ^{k} ∈ A(G)} .

Figà-Talamanca and Picardello [23] showed that A_{r}(G) is not norm dense in B_{0}(G) if the center of G is not compact or if G is a non-compact nilpotent group. Their conjecture reads as follows

Conjecture 1.3. Let G be an analytic group with compact center and without non-compact nilpotent direct factors. Then A_{r}(G) is dense in B_{0}(G) . Observe that A_{r}(G) might be a proper dense subspace of B_{0}(G) . Consider for instance G: = SL(2, ℝ) and

π: = ∫_{[0,1]}^{⊕} κ_{s}ds

where κ_{s}, 0 < s < 1, denotes the complementary series representations (see e.g., [24],p. 246) and ds is a finite measure equivalent to the Lebesguemeasure. Then A_{π} ⊆ \overline{A_{r}(G)} , but A_{π} ⊈ A_{r}(G) . Theorem 1.1 might be seen as a step towards proving Conjecture 1.3. Another class of groups satisfying this conjecture are linear reductive groups (with compact center), as follows immediately from the results of Cowling [11, 12]. Arbitrary compact central extensions of reductive Kazhdan groups are discussed in Chapter 3. Furthermore, in this chapter, we construct more groups supporting Conjecture 1.3, as G: = H_{n} ⋊ Sp(n, ℝ) , where H_{n} is the (2n + 1)–dimensional Heisenberg group with compact center, and related examples.

The proof of Theorem 1.2 is more or less a byproduct of the proof of Theorem 1.1. Let us discuss some appplications. Theorem 1.2 shows that all C_{0} -representations of the affine group of the line are square-integrable. Observing that the proof allows an immediate generalization to arbitrary local fields, we see that the same is true for the Fell group ℚ_{p} ⋊ ℤ_{p} . These two examples are well-known ([30],[41]). To give a new example, consider the following semidirect product: Let ℝ^{+} act on the (2n + 1)–dimensional Heisenberg group (identified with its Lie algebra)

𝔥_{n}: = \{[(x, y), z] : = ( \begin{matrix}0 & x & z \\ 0 & 0 & y \\ 0 & 0 & 0 \end{matrix} ) | x, y ∈ ℝ^{n}, z ∈ ℝ\}

via

t[(x, y), z] : = [(tx, ty), t^{2}z] .

The resulting group G: = 𝔥_{n} ⋊ ℝ^{+} acts on 𝔥_{n}^{∗} via the coadjoint representation. Defining (Z^{∗}, X_{1}^{∗}, …, X_{n}^{∗}, Y_{1}^{∗}, …, Y_{n}^{∗}) to be the dual basis on 𝔥_{n}^{∗} w.r.t. the canonical basis, we see that the C_{0} –representations are induced by representations of 𝔥_{n} corresponding to the linear forms ±Z^{∗} . Observe that ℝ^{+} acts freely and without nontrivial fixed points on 𝔥_{n} . Thus Theorem (1.2) implies that on this group all C_{0} –representations are square-integrable.

The question when C_{0} -representations are square-integrable, has been widely discussed. One of the most important results is that the regular representation splits into irreducibles if every C_{0} –representation is squareintegrable ([22], [3]), but this is only a necessary condition, at least in the nonunimodular case ([4],Ex.4). Baggett [2] showed by means of the Fell topology on the dual that the regular representation of a non-compact analytic unimodular group does not split.

From now on we will assume that G is a second countable and connected locally compact group and that all occuring Hilbert spaces are separable. A representation (π, H_{π}) is a strongly continuous unitary representation on a Hilbert space H_{π} .

Acknowledgement I want to thank Prof. B. Bekka for many valuable ideas, fruitful discussions and a careful reading of the first version of the paper, H. Führ for a lot of critical and helpful remarks and the German Research Foundation (DFG) for supporting my work.

2. Square-Integrability of Tensor Products on Linear Groups

a. The Proof of Theorem 1.1 The group G = N ⋊_{φ} H is by our assumptions a regular semidirect product and we may apply Mackey’s theory to obtain the dual: Every ϱ ∈ Ĝ is of the form

ϱ = ind_{NH_{χ}}^{G}χ′ ⊗ σ ,  (1)

where

• χ ∈ N̂ ,

• H_{χ}: = {h ∈ H| hχ: = χ ∘ φ(h)^{−1} ≃ χ} is the fixed group of χ in H ,

• χ′ and σ are multiplier representations in the following sense:

There is a Borel function ω: H_{χ} × H_{χ} → {z ∈ ℂ| |z| = 1} , constant on the N × N –cosets and satisfying

ω(e, x) = ω(x, e) = 1, ω(xy, z)ω(x, y) = ω(x, yz)ω(y, z)

(e denoting the group identity), such that χ′ is an extension of χ to NH_{χ} with χ′(xy) = \overline{ω(x, y)}χ′(x)χ′(y) and σ(n) = 1, ∀n ∈ N, σ(x, y) = ω(x, y)σ(x)σ(y) .

We denote the Mackey surjection Ĝ → N̂/G, ϱ ↦ G.χ by θ and recall that θ is continuous with respect to the Fell topology ([19],Lemma 3).

To fix notation, we shortly recall the Kirillov picture (for a complete discussion, see e.g., [10]): Let 𝔫^{∗} be the dual of the Lie algebra 𝔫 of N and denote by Ad^{∗} the coadjoint action of N on 𝔫^{∗} : Ad^{∗}(n)f : = f ∘ Ad(n)^{−1} for all n ∈ N, f ∈ 𝔫^{∗} . There is a homeomorphism, the Kirillov-map,

κ: 𝔫^{∗}/N → N̂ ,

realized in the following way: Let f ∈ 𝔫^{∗} and 𝔭 be a real polarization of f . This means that 𝔭 is a maximal subalgebra which is subordinate to f , that is f([𝔭, 𝔭]) = 0 . It should be noted that every subalgebra subordinate to f is contained in a real polarization. Now form the analytic subgroup P : = exp 𝔭 and obtain

χ: = ind_{P}^{N}e^{if(·)} .  (2)

Then χ depends only on the coadjoint orbit of f and (2) defines the Kirillov homeomorphism. Since the coadjoint orbits are closed, there exists a Borel section F : N̂ → 𝔫^{∗} ([42],A.7).

Going back to our group G = N ⋊ H , we may identify 𝔫 with an ideal of the Lie algebra 𝔤 = 𝔫 ⋊ 𝔥 . Hence G acts upon 𝔫^{∗} via

Ad^{∗}(n, h)f : = f ∘ Ad(n, h)^{−1} .

Thus the fixed group H_{χ} consists of the elements h ∈ H satisfying

Ad^{∗}(1, h)F (χ) ∈ N · F (χ) .

Furthermore, since Ad^{∗}gAd^{∗}Nf = Ad^{∗}NAd^{∗}gf , we have χ̄ ∈ Gχ ⟺ F (χ̄) ∈ GF (χ) . Now let G(F (χ)) : = {g ∈ G| Ad^{∗}gF (χ) = F (χ)} be the stabilizer of F (χ) in G . This is an algebraic group and we may choose a Levi-complement H_{2} of G(F (χ)) . As explained in ([27],p.87), we may conjugate F (χ) with a suitable n ∈ N to obtain H_{2} = H_{χ} . Thus we find an F′ ∈ N · F (χ) , which is stabilized by H_{χ} . In particular, N · H_{χ} is algebraic. Richardson’s theorem ([32],p.132) together with the fact that the Zariskitopology is noetherian implies that there are closed subgroups H_{1}, H_{2}, …, H_{s} of H and a finite measurable invariant partition 𝔫^{∗} = D_{1} ∪ D_{2} ∪ … ∪ D_{s} defined by

F ∈ D_{j} ⟺ {h ∈ H|hF ∈ N · F } is conjugate to H_{j}, 1 ≤ j ≤ s .

Since G acts smoothly upon N̂ , we find by the same argument as above a Borel section L: N̂/G → N̂ and the composition map

R: Ĝ → 𝔫^{∗}, ϱ ↦ F ∘ L ∘ θ(ϱ)

is Borel.

Ĝ θN̂/G L  N̂F 𝔫^{∗} R ↑  ⟶ ↓ ⟶^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.} ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}  ^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}^{.}

Hence the mapping R induces a finite measurable partition of Ĝ in

Ĝ_{j} : = R^{−1}(D_{j}), 1 ≤ j ≤ s .

We will keep these notations all through this paper.

The proof of Theorem 1.1 runs roughly as follows: Starting from a direct integral decomposition of a representation (π, H_{π}) as in the theorem, we obtain, by somewhat sophisticated calculations involving Mackey’s tensor product theorem, the fact that π has no subrepresentations with non-compact kernel and requirement (ii), that

π^{⊗k} ≤^{q} ∫ ⊕ ; Ĝ^{k} ind_{N}^{G} ∫ ⊕_{𝔫^{∗}} κ(f)dμ_{ϱ_{1},...,ϱ_{k}}(f)dμ^{⊗k}(ϱ_{1}, …, ϱ_{k}) ,

where k is a natural number independent of π , μ is the spectral measure of π and μ_{ϱ_{1},...,ϱ_{k}} is the measure on the dual 𝔫^{∗} induced by the smooth mapping

G^{k} → 𝔫^{∗}, (g_{1}, g_{2}, …, g_{k}) ↦ ∑_{j=1}^{k} g_{j}R(ϱ_{j}) .

The reductiveness of the H –action, requirement (i) and again the fact that π has no subrepresentations with non-compact kernel yield that this mapping has full rank a.e. μ^{⊗k} , whence μ_{ϱ_{1},...,ϱ_{k}} is absolutely continuous to the Lebesgue-measure on 𝔫^{∗} for μ^{⊗k} – almost all ϱ_{1}, …, ϱ_{k} ∈ Ĝ^{k} . The theorem then follows from induction in stages. Now let (π, H_{π}) be a representation such that all its subrepresentations have at most compact kernel. Since G is type I, π is quasi-equivalent to a multiplicity-free representation ([14],5.4.1). Hence we may assume that

π = ∫ ⊕ ; Ĝ ϱdμ(ϱ)  (3)

to be its decomposition in irreducibles. Thus π = ⨁_{j=1}^{s} π_{j} , where each π_{j}

is supported on Ĝ_{j}, 1 ≤ j ≤ s . If we find a k ∈ ℕ such that π_{j}^{⊗k} ≤^{q} λ_{G} for every 1 ≤ j ≤ s then k_{0} : = ks will do the task for arbitrary representations as in the theorem (recall that the tensor product of a representation which is quasi-contained in the regular representation with any representation of G is quasi-contained in the regular representation, too). Thus we may restrict ourselves to the case, where μ is supported on a single Ĝ_{j_{0}}, 1 ≤ j_{0} ≤ s .

The first lemma repeats an argument of Howe and Moore [27] which we include for completeness:

Lemma 2.1. There is an r_{0} ∈ ℕ, r_{0} ≤ dim H_{j_{0}} , such that there is a Zariski-open subset D ⊆ H_{j_{0}}^{r_{0}} with: For all r ≥ r_{0}, (h_{1}, h_{2}, …, h_{r}) ∈ D × H^{r−r_{0}}

(h_{1}^{−1}H_{j_{0}}h_{1} ∩ h_{2}^{−1}H_{j_{0}}h_{2} ∩ … ∩ h_{r}^{−1}H_{j_{0}}h_{r})_{0} = (H_{j_{0}}^{nor})_{0} .

Here, U_{0} denotes the identity component of a subgroup U and H_{j_{0}}^{nor} the intersection of all conjugates of H_{j_{0}} , which is an algebraic normal subgroup.

Proof. Recall that the dimension function

H^{r} ∋ (h_{1}, h_{2}, …, h_{r}) ↦ dim(h_{1}^{−1}H_{j_{0}}h_{1} ∩ h_{2}^{−1}H_{j_{0}}h_{2} ∩ … ∩ h_{r}^{−1}H_{j_{0}}h_{r})

is upper semicontinuous with respect to the Zariski-topology on H^{r} . Thus the minimum

d(r) : = min{dim(h_{1}^{−1}H_{j_{0}}h_{1}∩h_{2}^{−1}H_{j_{0}}h_{2}∩…∩h_{r}^{−1}H_{j_{0}}h_{r})| h_{1}, h_{2}, …, h_{r} ∈ H} is assumed on a Zariski-open subset D_{r} ⊆ H^{r} . Clearly, after at most dim H_{j_{0}} steps, we have d(r + 1) = d(r) . This implies that the identity component of h_{1}^{−1}H_{j_{0}}h_{1} ∩ h_{2}^{−1}H_{j_{0}}h_{2} ∩ … ∩ h_{r}^{−1}H_{j_{0}}h_{r} is normal for all h_{1}, h_{2}, …, h_{r} ∈ D_{r} . Hence define r_{0} : = r, D: = D_{r} .

For the remainder of this section, recall the measurable mappings

θ: Ĝ → N̂/G, L: N̂/G → N̂, R: Ĝ → 𝔫^{∗}

from the beginning.

Lemma 2.2. Keep the above notations. If S̃ ⊆ Ĝ is μ–measurable and satisfies

W : = Span(G · R(ϱ)| ϱ ∈ S̃) ⪇ 𝔫^{∗}

then μ(S̃) = 0 .

Proof. Let S̃ be as in the lemma and define

M_{W} : = {x ∈ 𝔫| x ∈ ker gR(ϱ) ∀g ∈ G, ϱ ∈ S̃} .

By assumption, M_{W} is a closed non-compact normal subgroup of N and of G (after identifying N with 𝔫), subordinate to gR(ϱ) for all g ∈ G, ϱ ∈ S̃ . For each ϱ ∈ S̃ , choose a real polarization 𝔭 containing M_{W} . Then M_{W} , being normal in N , is contained in the kernel of L∘θ(ϱ) = indN_{𝔭}e^{iR(ϱ)} . Since M_{W} is normal in G and since ϱ = ind_{N}^{G} ((L ∘ θ(ϱ))′ ⊗ σ(ϱ)) , the same argument shows that M_{W} is contained in ker ϱ for all ϱ ∈ S̃ . Thus the kernel of ∫ ⊕ ; S̃ ϱdμ were not compact if μ(S̃) ≠ 0 .

Corollary 2.3. With the assumptions of Theorem 1.1 , we have for all r ≥ r_{0} and all (h_{1}, h_{2}, …, h_{r}) ∈ D × H^{r−r_{0}} (r_{0} and D defined by (2.1) ) that the subgroup

h_{1}^{−1}H_{j_{0}}h_{1} ∩ h_{2}^{−1}H_{j_{0}}h_{2} ∩ … ∩ h_{r}^{−1}H_{j_{0}}h_{r}

is compact.

Proof. Since the groups under consideration are all algebraic, they have only finitely many connected components. By (2.1), it remains to show that (H_{j_{0}}^{nor})_{0} is compact. Indeed, every h_{0} ∈ (H_{j_{0}}^{nor})_{0} satisfies

h_{0}hR(ϱ) ∈ hNR(ϱ) = NhR(ϱ), ∀h ∈ H, ϱ ∈ suppμ ,

in particular

h_{0} (hR(ϱ)|_{Z}) = hR(ϱ)|_{Z}, ∀h ∈ H ,

where Z denotes the center of 𝔫 . By (2.2), the orbits HNR(ϱ) span 𝔫^{∗} . Thus every f ∈ Z^{∗} is a linear combination of elements of HNR(ϱ)|_{Z} and (H_{j_{0}}^{nor})_{0} acts trivially on Z^{∗} , hence on Z . By requirement (ii), (H_{j_{0}}^{nor})_{0} is compact.

The aim of these lemmas is to apply Mackey’s tensor product theorem.

Proposition 2.4. With the above assumptions and notations, we have for k ≥ r_{0} :

π^{⊗k} ≤^{q} ∫ ⊕ ; Ĝ^{k} ind_{N}^{G} (∫_{D×H^{k−r_{0}}}^{⊕} ⨂_{j=1}^{k} h_{j} · (L ∘ θ) (ϱ_{j}) dν^{⊗k}((h_{1}, …, h_{k}))) ×
× dμ^{⊗k}((ϱ_{1}, …, ϱ_{k})) ,

where ν^{k} denotes a finite measure on H^{k} , equivalent to the Haar measure.

Proof. Observe at first that the diagonal group ΔG ◁ G^{k} and (NH_{j_{0}})^{k} are regularly related, since all these groups are algebraic. Thus, let dν̃^{⊗k} be the measure on the double coset space ΔG : (NH_{j_{0}})^{k} induced by a finite measure on G^{k} , equivalent to the Haar measure, and define for g: = (g_{1}, …, g_{k}) ∈ G^{k}

G^{(g)} : = g_{1}^{−1}NH_{j_{0}}g_{1} ∩ … ∩ g_{k}^{−1}NH_{j_{0}}g_{k}

and

ϱ^{(g)} : G^{(g)} → U(⨂_{j=1}^{k} H_{(L∘θ(ϱ_{j}))′⊗σ_{j}} ), x ↦ ⨂_{j=1}^{k}((L ∘ θ(ϱ_{j}))′ ⊗ σ_{j})(g_{j}xg_{j}^{−1}) ,

where again (L ∘ θ(ϱ_{j}))′ denotes the extension of (L ∘ θ)(ϱ_{j}) to NH_{j_{0}} and U(H_{π}) denotes the group of unitary operators on H_{π} . Then, by Mackey’s tensor product theorem,

⨂_{j=1}^{k} ϱ_{j} = ⨂_{j=1}^{k} (ind_{NH_{j_{0}}}^{G} (L ∘ θ(ϱ_{j}))′ ⊗ σ_{j}) = ∫_{ΔG:(NH_{j_{0}})^{k}}^{⊕} ind_{G^{(g)}}^{G}ϱ^{(g)}dν̃^{⊗k} .

But clearly ΔG : (NH_{j_{0}})^{k} is identified with ΔH : H_{j_{0}}^{k} and we may write

⨂_{j=1}^{k} (ind_{NH_{j_{0}}}^{G} (L ∘ θ(ϱ_{j}))′ ⊗ σ_{j}) = ∫_{ΔH:H_{j_{0}}^{k}}^{⊕} ind_{G^{(g)}}^{G}ϱ^{(g)}d\overline{ν}^{⊗k} ,

\overline{ν} being induced by a finite measure ν on H which is equivalent to the Haar measure. Now take D ⊆ H^{r_{0}} as in Lemma 2.1. Then D × H^{k−r_{0}} , being Zariski-open, is conull in H^{k} , hence we have certainly

⨂_{j=1}^{k} (ind_{NH_{j_{0}}}^{G} (L ∘ θ(ϱ_{j}))′ ⊗ σ_{j}) ≤ ∫_{D×H^{k−r_{0}}}^{⊕} ind_{G^{(g)}}^{G}ϱ^{(g)}dν^{⊗k} .

Now, G^{(h)}/N is compact for h: = (h_{1}, …, h_{r}) ∈ D ×H^{k−r_{0}} , by (2.3). Thus ([18],Lemma 4.2) implies ind_{N}^{G^{(h)}}(ϱ^{(h)}|_{N}) = (ind_{N}^{G^{(h)}}1)⊗ϱ^{(h)} ≥ ϱ^{(h)} . On the

other hand, ϱ^{(h)}|_{N} ≃^{q} ⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j}) . Putting all together and using induction in stages, we find (writing ϱ and h instead of (vrh_{1}, …, ϱ_{k}) and (h_{1}, …, h_{k}) , respectively)

π^{⊗k} = ∫ ⊕ ; Ĝ^{k} ⨂_{j=1}^{k} ϱ_{j}dμ^{⊗k}(ϱ) = ∫ ⊕ ; Ĝ^{k} ⨂_{j=1}^{k} (ind_{NH_{j_{0}}}^{G} (L ∘ θ(ϱ_{j}))′ ⊗ σ_{j}) dμ^{⊗k}(ϱ) =

≤ ∫ ⊕ ; Ĝ^{k} ∫_{D×H^{k−r_{0}}}^{⊕} ind_{N}^{G} (ϱ^{(h)} |_{N} ) dν^{⊗k}(h)dμ^{⊗k}(ϱ)
≃^{q} ∫ ⊕ ; Ĝ^{k} ∫_{D×H^{k−r_{0}}}^{⊕} ind_{N}^{G} ⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j})dν^{⊗k}(h) dμ^{⊗k}(ϱ)

= ∫ ⊕ ; Ĝ^{k} ind_{N}^{G} (∫_{D×H^{k−r_{0}}}^{⊕} ⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j})dν^{⊗k}(h))dμ^{⊗k}(ϱ).

This shows the formula.

The strategy is now to prove that the inner integral

∫_{D×H^{k−r_{0}}}^{⊕} ⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j})dν^{⊗k}(h_{1}, …, h_{k})  (4)

defines a square-integrable representation of N for large enough k. Induction in stages then yields the theorem. There are many results on the decomposition of tensor products on nilpotent Lie groups, e.g., in [10], [21], [5]. Here we need only the following qualitative version:

⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j}) ≃^{q} ∫ ⊕_{𝔫^{∗}} κ(f)dη_{1} ∗ dη_{2} ∗ … ∗ dη_{k} ,

where ≃^{q} denotes quasi-equivalence, κ the Kirillov map, dη_{j} a finite quasi N –invariant measure on 𝔫^{∗} , supported by the orbit of h_{j}R(ϱ_{j}) , and ∗ denotes the additive convolution on 𝔫^{∗} . For each 1 ≤ j ≤ k , let n̄_{j} be the equivalence class of n ∈ N in Ad^{∗}(N)h_{j}R(ϱ_{j}) ≃ N/𝔰(j) , where 𝔰(j) is the stabilizer of h_{j}R(ϱ_{j}) and let dη̃_{j} be a quasi N –invariant finite measure on N/𝔰(j) . By uniqueness, dη_{j} is quasi-equivalent to the measure induced by dη̃_{j} and the canonical mapping N/𝔰(j) ∋ n̄_{j} ↦ n_{j}h_{j}R(ϱ_{j}). Whence we may write the above intergral more explicitly as

⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j}) ≃^{q} ∫_{N/𝔰(1)}^{⊕} ∫_{N/𝔰(2)}^{⊕} … ∫_{N/𝔰(k)}^{⊕} κ (∑_{j=1}^{k} n_{j}h_{j}s_{j}R(ϱ_{j})) ×
×dη̃_{1}(n̄_{1})dη̃_{2}(n̄_{2}) … dη̃_{k}(n̄_{k}) .

Now, let dσ_{j} denote a finite measure on 𝔰(j) , equivalent to the Haar measure. We certainly don’t change the quasi-equivalence class if we substitute

⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j}) ≃^{q}

≃^{q} ∫_{N/𝔰(1)}^{⊕} ∫_{𝔰(1)}^{⊕} … ∫_{N/𝔰(k)}^{⊕} ∫_{𝔰(k)}^{⊕} κ (∑_{j=1}^{k} n_{j}s_{j}h_{j}R(ϱ_{j})) ×

×dσ_{1}(s_{1})dη̃_{1}(n̄_{1}) … dσ_{k}(s_{k})dη̃_{k}(n̄_{k}) .

All together, choosing finite measures dg, dn on G and N , equivalent to the respective Haar measures, and using Weil’s formula, we find

∫_{D×H^{k−r_{0}}}^{⊕} ⨂_{j=1}^{k} h_{j}(L ∘ θ)(ϱ_{j})dν^{⊗k}(h) ≃^{q}

≃^{q} ∫_{D×H^{k−r_{0}}}^{⊕} ∫_{N^{k}}^{⊕} κ (∑_{j=1}^{k} n_{j}h_{j}R(ϱ_{j})) dn^{⊗k}(n)dν^{⊗k}(h)

= ∫_{N^{k}×D×H^{k−r_{0}}}^{⊕} κ (∑_{j=1}^{k} g_{j}R(ϱ_{j})) dg^{⊗k}(g) .

Now define for fixed ϱ_{1}, ϱ_{2}, …, ϱ_{k} :

ψ_{ϱ_{1},...,ϱ_{k}} : G^{k} → 𝔫^{∗}, (g_{1}, g_{2}, …, g_{k}) ↦ ∑_{j=1}^{k} g_{j}R(ϱ_{j}) .  (5)

The integral in Eq. (5) is just the integral over 𝔫^{∗} with respect to the measure μ_{ϱ_{1},...,ϱ_{k}} induced by ψ_{ϱ_{1},...,ϱ_{k}} and a finite measure on G^{k} , equivalent to the Haar measure. Since the Plancherel measure of N corresponds to the Lebesgue measure λ_{𝔫^{∗}} on 𝔫^{∗} , it remains to prove the absolute continuity

μ_{ϱ_{1},...,ϱ_{k}} ≪ λ_{𝔫^{∗}} .  (6)

This will follow from an elementary argument from local differential geometry.

Lemma 2.5. If the differential dψ_{ϱ_{1},...,ϱ_{k}} has full rank at one point, the induced measure μ_{ϱ_{1},...,ϱ_{k}} is absolutely continuous with respect to the Lebesgue measure.

Proof. By assumption, there is a point in G^{k} , where ψ_{ϱ_{1},...,ϱ_{k}} has maximal rank. Since this mapping is analytic, the points g ∈ G^{k} satisfying rank dψ_{ϱ_{1},...,ϱ_{k}}(g) ⪇ dim 𝔫^{∗} form a submanifold of lower dimension, thus a nullset. The remainder of the proof is an immediate application of the implicit function theorem.

In order to show that the condition of (2.5) is fullfilled for μ^{⊗k} –almost all (ϱ_{1}, ϱ_{2}, …, ϱ_{n}) and large enough k , we turn now to the more geometrical aspects of our special situation.

Let G be a linear Lie group, acting on a finite dimensional real vector space V and let 𝔤 be its Lie algebra. The derived representation of 𝔤 is defined by

Xy: = lim_{t→0} \frac{exp(tX)y − y}{t} , ∀y ∈ V, X ∈ 𝔤 .

In particular, the tangent space of the orbit G · y at the point y is 𝔤 · y.

If A ⊆ V , we denote by Aff(A) the smallest affine subspace containing A . The following lemma is probably well-known. We include the proof for sake of completeness.

Lemma 2.6. Keep the above notations. Let x ∈ V . Then Aff(Gx) = x + L_{x} , where L_{x} is the vector space

L_{x}: = Span({𝔤gx, g ∈ G}) .

Proof. Define L: = Span(g_{2}x − g_{1}x| g_{1}, g_{2} ∈ G) . Since L is closed, L ⊃ L_{x} . To show the other direction, choose at first g_{1}, g_{2} ∈ G , such that g_{2}g_{1}^{−1} is contained in a Campbell–Hausdorff neighbourhood of the identity. Thus there are t_{0} ≥ 0 and X ∈ 𝔤 with g_{2} = exp(t_{0}X)g_{1} . Hence

g_{2}x − g_{1}x = (exp(t_{0}X) − 1)g_{1}x
= ∫_{0}^{t_{0}} exp(sX)Xg_{1}xds = ∫_{0}^{t_{0}} X \underbrace{exp(sX)g_{1}}_{=:g(s)} x ds

Now, computing the integral as Riemannian sum yields

g_{2}x − g_{1}x = lim_{N→∞} \frac{t_{0}}{N} ∑_{j=0}^{N−1} Xg (\frac{(j + 1)}{N}t_{0}) x

Thus g_{2}x − g_{1}x ∈ L_{x} .

As to the general case, join g_{1} and g_{2} with a compact arc γ and cover γ with finitely many translates U_{1}, U_{2}, …, U_{n+1} of Campbell–Hausdorff neighbourhoods. Then pick

g̃_{1} : = g_{1}, g̃_{j} ∈ U_{j−1} ∩ U_{j} ∩ γ (2 ≤ j ≤ n), g̃_{n+1} : = g_{2} .

By the first part, this yields

g_{2}x − g_{1}x = ∑_{j=1}^{n} \underbrace{(g̃_{j+1} − g̃_{j})x}_{∈L_{x}} ∈ L_{x} .

Lemma 2.7. Keep the above notations and let G = N ⋊ H be as in Theorem 1.1 , acting upon 𝔫^{∗} via the coadjoint action. Then we have for all f ∈ 𝔫^{∗}

Aff(Gf) = L_{f} = Span(Gf) .

Proof. We have Aff(Gf) = f_{1} + L_{f} . Then Span(Gf) = Span(f, L_{f}) . Clearly, L_{f}, f + L_{f} and Span(f, L_{f}) are H –invariant. Since H acts reductively, there is an at most one-dimensional H –invariant complement of L_{f} in Span(f, L_{f}) , say ℝ · y . We have to show that y = 0 . Otherwise y is not fixed under H and, by connectedness, Hy is a ray through y. If necessary, we change the sign of y to obtain Hy ∩ (f + L_{f}) ≠ Ø . By invariance, f + L_{f} contains the whole ray, thus it contains 0 and f ∈ L_{f} , as desired.

The following corollary finishes the proof of Theorem 1.1.

Corollary 2.8. Keep the above notations. There is a k_{0} ∈ ℕ, k_{0} ≤ dim N , such that for all k ≥ k_{0} and for μ^{⊗k} –almost all (ϱ_{1}, ϱ_{2}, …, ϱ_{k}) ∈ Ĝ^{k} there is a (g_{1}, g_{2}, …, g_{k}) ∈ G^{k} such that

Span(𝔤g_{1}R(ϱ_{1}), …, 𝔤g_{k}R(ϱ_{k})) = 𝔫^{∗} .

Proof. Let k be the smallest natural number with:

(∗) There exists a measurable set S ∈ Ĝ^{k} with μ^{⊗k}(S) > 0 and for all (ϱ_{1}, …, ϱ_{k}) ∈ S, (g_{1}, …, g_{k}) ∈ G^{k} :

dim (Span(𝔤g_{1}R(ϱ_{1}), …, 𝔤g_{k}R(ϱ_{k}))) = k − 1 .

By (2.2) and (2.7), we have k > 1 . Using Fubini’s theorem we find

μ^{⊗k}(S) = ∫Ĝ μ^{⊗(k−1)}({(ϱ_{1}, …, ϱ_{k−1})| (ϱ_{1}, … ϱ_{k−1}, ϱ) ∈ S})dμ(ϱ) .

Since μ^{⊗k}(S) > 0 , there exists (ϱ_{1}, …, ϱ_{k−1}) ∈ Ĝ^{k−1} such that

(ϱ_{1}, …, ϱ_{k−1}, ϱ) ∈ S

for all ϱ in a subset S̄ ⊆ Ĝ of positive measure. In particular, for all ϱ ∈ S̄ and for all (g_{1}, …, g) ∈ G^{k} ,

𝔤gR(ϱ) ⊆ Span(𝔤g_{1}R(ϱ_{1}), …, 𝔤g_{k−1}R(ϱ_{k−1})) ,

whence, by (2.7),

L_{R(ϱ)} = Span(G · R(ϱ)) ⊆ Span(𝔤g_{1}R(ϱ_{1}), …, 𝔤g_{k−1}R(ϱ_{k−1})) ,

Thus fix (g_{1}, …, g_{k−1}) ∈ G^{k−1} and observe that

W : = Span(𝔤g_{1}R(ϱ_{1}), …, 𝔤g_{k−1}R(ϱ_{k−1}))

is (k − 1)–dimensional. But the above and (2.7) imply

Span({G · R(ϱ)| ϱ ∈ S̄}) ⊆ W .

But μ(S̄) > 0 , whence by (2.2), k > n + 1 , as desired.

b. The Proof of Theorem 1.2 Let

π = ∫ ⊕ ; Ĝ m(ϱ)ϱdμ

be as in (1.2). By the compactness assumption and ([18],Lemma 4.2), we have for almost all ϱ :

ϱ = ind_{Gχ̃}^{G}χ ≤ ind_{N}^{G}(χ|_{N}) ≃ ind_{N}^{G}g(χ|_{N})

for all g ∈ G . Thus, if dg denotes a finite measure on G , equivalent to the Haar measure,

ϱ ≤^{q} ind_{N}^{G} ∫_{G}^{⊕} gχ|_{N}dg .

Since Gχ|_{N} produces an open orbit in 𝔫^{∗} , we see with (2.5)

∫_{G}^{⊕} gχ|_{N}dg ≤^{q} λ_{N} .

Thus induction in stages shows ϱ ≤^{q} λ_{G}, whence the theorem.

Examples 2.9. (i) Canonical examples for an application of Theorems 1.1 and 1.2 are the generalized affine group, where N = ℝ^{n} and H is the group of diagonal matrices, or the action of ℝ^{+} on the (2n+1)–dimensional Heisenberg group mentioned in the introduction.

(ii) Consider the space ℝ^{3} as row vectors and the three-dimensional skew symmetric matrices Σ . Then 𝔫: = ℝ^{3} × Σ is a two step nilpotent algebra with the bracket

[(u, U), (v, V)] : = (0, u^{t}v − v^{t}u) .

Take H: = SO(3, ℝ) and define

h(u, U) : = (uh^{t}, hUh^{t}) .

Then G: = 𝔫 ⋊ H satisfies the assumptions of 1.1.

(iii) Let G be a linear totally minimal group with abelian nilradical. By [35],2.5 G = V ⋊ H for a vector group V . Then G has a finite extension G̃ satisfying

G̃ = V ⋊ (K × S_{1} … × S_{n}) ,

where K is compact and S_{j} is non-compact simple with finite center for all 1 ≤ j ≤ n . Let Σ_{1} be the product of all the S_{j} , acting trivially on V and Σ_{2} be the product of the remaining factors. Then

G̃ = (V ⋊ Σ_{2}) × Σ_{1} .

Since each S_{i} is a finite extension of a linear algebraic group, Theorem 1.1 is applicable to the first factor and the second factor is handled below. This will yield \overline{A_{r}(G)} = B_{0}(G) .

(iv) The proofs of the Theorems 1.1 and 1.2 carry over without essential changes for connected algebraic groups over arbitrary local fields.

(v) The fact that each C_{0} –representation has a square-integrable tensor power is related to Haagerup’s property (H), as discussed in [6]. A locally compact group is said to have property (H) if there is a C_{0} - representation which weakly contains the trivial representation. Since taking tensor products is continuous [20], the groups occuring in (1.1) have property (H) if and only if they are amenable.

c. Linear Reductive Groups with Compact Center For these groups the solutions of the problems mentioned in the introduction are easily obtained by reformulation of the results of M. Cowling [11, 12, 13] and C.C. Moore [38]. Here, we need

Theorem 2.10. Let G be a simple analytic group with finite center. Then

(i) For every π ∈ Ĝ , there is q = q(π) > 0 and C > 0 , such that all matrix coefficients φ_{ξη} of π are in L^{2q}(G) and satisfy

∥φ_{ξη}∥_{2q} ≤ C∥ξ∥∥η∥ .

Furthermore C is independent of π .

(ii) If every matrix coefficient of a unitary representation (π, H_{π}) is in L^{p}(G), (p ≥ 1) , the same is true for every representation weakly contained in π . In particular, the set

Ĝ_{q}: = {π ∈ Ĝ| all matrix coefficients of π are in L^{2q}(G)}

is closed in Ĝ . (This is true for an arbitrary locally compact group.)

(iii) L^{2}(G) ∗ L^{2}(G) = A(G) ⊆ L^{2+ε}(G) for all ε > 0 . This is called the Kunze-Stein phenomenon.

The first two results carry over without difficulties to linear reductive groups with compact center. In order to prove (iii) one uses Herz’ majorisation principle ([11],7.2). The question on the universality of q as in (i) is solved by results of Cowling and Moore:

Theorem 2.11. A simple group with finite center has Kazhdan’s property (T) if and only if there is a q such that the matrix coefficients of all nontrivial irreducible representations are in L^{2q}(G) .

This implies immediately Corollary 2.12. Let G be a linear reductive group with compact center. Then B_{0}(G) = \overline{A_{r}(G)} , and B_{0}(G) = A_{r}(G) if and only if G is a Kazhdan group.

G is Kazhdan if and only if there is k ∈ ℕ such that π^{⊗k} ≤^{q} λ_{G} for all C_{0} –representations (π, H_{π}) .

Proof. There is a finite central extension G̃ of G satisfying

G̃ = K × S_{1} × … × S_{n} ,

where K is compact and S_{1}, …, S_{n} are non-compact simple groups with finite center. Clearly, it is enough to look at G̃ . The dual of G̃ identifies topologically with K̂ × Ŝ_{1} × … × Ŝ_{n} , since any irreducible representation is an outer tensor product

π = ϱ ⊗ σ_{1} ⊗ … ⊗ σ_{n}, ϱ ∈ K̂, σ_{i} ∈ Ŝ_{i}, 1 ≤ i ≤ n .

Now let ψ be a C_{0} –representation of G̃ which is (without loss of generality) multiplicity free, and

ψ = ∫ ⊕ ; _{G̃} π_{s}dμ(s)

its decomposition in irreducibles. Clearly, almost all π_{s} are nontrivial on every S_{j}, 1 ≤ j ≤ n . By (2.10.ii), the set of these representations is the union of the closed sets ⋃_{q≥2} ˆ G̃_{q} . By regularity of μ , one has for all ξ, η ∈ H_{ψ} :

⟨π(·)ξ, η⟩ = B(G) − lim_{q→∞} ∫_{_{G̃}_{q}}⟨π(·)ξ_{s}, η_{s}⟩dμ(s).

Now if ψ_{q}: = ∫_{_{G̃}_{q}}^{⊕} πdμ , all matrix coefficients of ψ_{q} are in L^{2q}(G̃) and, by

Hölder’s inequality, ψ_{q}^{⊗q} has a dense set of square-integrable coefficients, hence is square-integrable. This shows one direction and the other follows from ([38],3.6). The second statement follows from the same arguments.

This proof is a very special case of the one given in [13].

3. Some Further Groups Satisfying A_{r}(G) = B_{0}(G)

In this section we discuss some groups not covered by Theorem 1.1. We start by continuing Example 2.9.i:

sl(2,R) G: = 𝔥_{n} ⋊ (ℝ^{+} × Sp(n, ℝ)) : Here 𝔥_{n} ⋊ ℝ^{+} is the semidirect product of the Heisenberg group by ℝ^{+} as in (2.9.i). The symplectic group Sp(n, ℝ) acts upon 𝔥_{n} via

A[(x, y), z] : = [(x, y)A^{t}, z] .

This action commutes with that of ℝ^{+} . Recall the metaplectic representations (ω_{r}, L^{2}(ℝ^{n})), r ≠ 0, of the metaplectic group Mp(n, ℝ) , a two-fold covering of Sp(n, ℝ) with covering homomorphism p: Mp(n, ℝ) → Sp(n, ℝ). ω_{r} is defined in the following way: Let π_{r} be an infinite dimensional irreducible representation of 𝔥_{n} whose restriction to the center Z is a multiple of χ_{r} : z ↦ e^{irz} . Since this requirement determines π_{r} up to unitary equivalence, every m ∈ Mp(n, ℝ) defines a unitary operator ω_{r}(m) satisfying:

π_{r}(p(m)n) = ω_{r}(m)π_{r}(n)ω_{r}(m)^{−1}, ∀n ∈ 𝔥_{n} .

The mapping m ↦ ω_{r}(m) is indeed a unitary representation of Mp(n, ℝ) and thus uniquely induces a τ_{r} –representation ω̃_{r} on Sp(n, ℝ) , where τ_{r} is a multiplier of order 2, such that

π′_{r}(n, A) : = π_{r}(n)ω̃_{r}(A), n ∈ 𝔥_{n}, A ∈ Sp(n, ℝ) ,  (7)

extends π_{r} to an irreducible τ_{r} –representation of 𝔥_{n}⋊Sp(n, ℝ) . Furthermore we may take ω_{r} = ω_{s} if rs > 0 . Keeping these notations, we cite a result of Howe and Moore ([27],6.4):

Proposition 3.1. There is a dense set D of vectors in L^{2}(ℝ^{n}) , such that, for all r ≠ 0 , the absolute values of the matrix coefficients of π′_{r} to vectors in D belong to L^{4n+2+ε}((𝔥_{n}⋊Sp(n, ℝ))/Z) for all ε > 0, Z denoting the center of 𝔥_{n} . Now we return to our group G = 𝔥_{n} ⋊ (ℝ^{+} × Sp(n, ℝ)) . The G–orbits in 𝔥_{n}^{∗} are:

{Z^{∗} > 0}, {Z^{∗} < 0}, {Z^{∗} = 0} \ {0}, {0} ,

where (Z^{∗}, X_{1}^{∗}, … X_{n}^{∗}, Y_{1}^{∗}, …, Y_{n}^{∗}) denotes the dual base. Thus C_{0} –representations are supported on

Ĝ_{±}: = {ind_{h_{n}⋊Sp(n,ℝ)}^{G}π′_{±1}⊗σ| σ irreducible τ̄_{±1} –representation of Sp(n, ℝ)}.

So, let π: = ∫ ⊕ ; Ĝ_{+} ind_{h_{n}⋊Sp(n,ℝ)}^{G}π′_{1} ⊗ σdμ (we may assume again that π is multiplicity free). Then, for k ∈ ℕ , the same computation as in (2.4) yields

π^{⊗k} ≤^{q} ∫ ⊕ ; Ĝ_{+}^{k} ∫ ⊕_{ℝ^{+k}} ind_{h_{n}⋊Sp(n,ℝ)}^{G} ⨂_{i=1}^{k} π_{1}^{′t_{i}} ⊗ σ_{i}^{t_{i}} dt^{⊗k}dμ^{⊗k} ,

where dt is a finite measure on ℝ^{+} , equivalent to the Haar measure, and

π_{1}^{′t} ⊗ σ^{t}(n, A) : = π′_{1}(t · n, A)σ(A) == π′_{t^{2}}(n, A)σ(A) .

If k is even, ⨂_{i=1}^{k} π_{1}^{′t_{i}} and ⨂_{i=1}^{k} σ_{i} are ordinary representations. Furthermore, by (3.1), the absolute value of the matrix coefficients of ⨂_{i=1}^{k} π_{1}^{′t_{i}} associated to a dense set of vectors are square-integrable modulo the center Z for k ≥ 2n + 2 , hence

⨂_{i=1}^{k} π_{1}^{′t_{i}} ≤ ind_{Z}^{h_{n}⋊Sp(n,ℝ)}χ_{∑_{i=1}^{k} t_{i}^{2}} .

This yields, by (2.5),

π^{⊗k} ≤^{q} ∫ ⊕ ; Ĝ_{+}^{k} ind_{h_{n}⋊Sp(n,ℝ)}^{G}((ind_{Z}^{h_{n}⋊Sp(n,ℝ)} ∫ ⊕_{ℝ^{+k}} χ_{∑_{i=1}^{k} t_{i}^{2}}dt^{⊗k}) ⨂_{i=1}^{k} σ_{i})dμ^{⊗k}

≤^{q} ∫ ⊕ ; Ĝ_{+}^{k} ind_{h_{n}⋊Sp(n,ℝ)}^{G} ((ind_{Z}^{h_{n}⋊Sp(n,ℝ)}λ_{Z}) ⨂_{i=1}^{k} σ_{i}) dμ^{⊗k}
≤^{q} ∫ ⊕ ; Ĝ_{+}^{k} ind_{h_{n}⋊Sp(n,ℝ)}^{G}λ_{h_{n}⋊Sp(n,ℝ)}dμ^{⊗k} ≤^{q} λ_{G},

since for every locally compact group U and for every representation ϱ of U : λ_{U} ⊗ ϱ ≃^{q} λ_{U} . Decomposing an arbitrary C_{0} –representation π = π_{+} ⊕ π_{−} , where π_{±} is supported on Ĝ_{±} , we see:

Corollary 3.2. Let π be a C_{0} –representation of G = 𝔥_{n}⋊(ℝ^{+}×Sp(n, ℝ)) ,

then for all k > 4n + 4 : π^{⊗k} ≤^{q} λ_{G} .

sl(2,R)Upper Triangular Matrices in SL(3, ℝ) : Let

G: = \{( \begin{matrix}\frac{1}{αβ} & x & z \\ 0 & α & y \\ 0 & 0 & β \end{matrix} ) | x, y, z ∈ ℝ, α, β > 0\} .

The representation theory of this group is a classical example due to Mackey, see ([32],III.B.Ex.6). Clearly, G is a semidirect product of the threedimensional Heisenberg group 𝔥_{1} with ℝ^{+2} , where the action is given by

(α, β) · (x, y, z) : = (\frac{x}{α^{2}β} , \frac{α}{β}y, \frac{z}{αβ^{2}} ) .

The C_{0} –representations are supported by

ind_{h_{1}⋊D′}^{G}π′_{±1} ⊗ χ_{λ} ,

where D′: = {(s^{−2}, s)| s ∈ ℝ^{+}} ≃ ℝ^{+} is the fixed group of π_{±1} , the irreducible representation of 𝔥_{1} which, restricted to the center, is quasiequivalent to z ↦ e^{±iz} , π′_{±1} is an extension to 𝔥_{1} ⋊ D′ and χ_{λ} denotes a character of ℝ^{+} . Here one uses the fact that ℝ^{+} has only trivial cocycle representations. Observe that D′ acts upon 𝔥_{1} via (s^{−2}, s)(x, y, z) = (s^{3}x, \frac{1}{s^{3}} y, z) . This defines an embedding D′ ↪ Sp(1, ℝ) . By the uniqueness of the extension, the representation π′_{+} of 𝔥_{1} ⋊D′ is the restriction of the cocycle representation π_{+}ω̃_{1} of 𝔥_{1} ⋊Sp(1, ℝ) to 𝔥_{1} ⋊D′ , where ω̃_{1} denotes the metaplectic representation. By (3.1), there is a dense set of vectors in H_{π_{+}ω̃_{1}} producing coefficients of π_{+}ω̃_{1} in L^{k}(𝔥_{1} ⋊ Sp(n, ℝ)/Z) for sufficiently large k (k > 6) . Denoting the Haar measures on G_{1} : = (𝔥_{1} ⋊ Sp(1, ℝ))/Z, L_{1} : = (𝔥_{1} ⋊ D′)/Z and G_{1}/L_{1} with dg, dh and dḡ , respectively, we have for ξ, η contained in the above mentioned dense set:

∞ > ∫_{G_{1}} |φ_{ξη}|^{k}(g)dg = ∫_{G_{1}/L_{1}}(∫_{L_{1}} |φ_{ξη}(gh)|^{k}dh)dḡ .

Thus the inner integral ∫_{L_{1}} |φ_{ξη}(hg)|^{k}dh is finite for almost all g ∈ G yielding enough k–integrable vectors. Whence π_{+} and a fortiori π_{+} ⊗ χ_{λ} has a dense subset of L^{k} –vectors for all λ ∈ ℝ . Now, the same arguments used in the above example show:

Corollary 3.3. Let π be a C_{0} –representation of G . Then for all k ≥ 8 :

π^{⊗k} ≤^{q} λ_{G} .

We now turn to nonlinear groups: sl(2,R) G: = H_{n} ⋊ Sp(n, ℝ) : Here H_{n} denotes the (2n + 1)–dimensional Heisenberg group with compact center, that is the image of an infinite dimensional irreducible representation of the simply connected group 𝔥_{n} . We parametrize H_{n} by [(p, q), t], p, q ∈ ℝ^{n}, t ∈ 𝕋 . Then the action of Sp(n, ℝ) on 𝔥_{n} factors to an action on H_{n} . The center of the associated semidirect product G: = H_{n} ⋊ Sp(n, ℝ) is identified with the torus 𝕋 and G/𝕋 ≃ ℝ^{2n} ⋊Sp(n, ℝ) is a linear totally minimal group. The dual of G is determined by Mackey’s theory and decomposes in essentially three measurable parts: Ĝ_{Sp(n,ℝ)} : = {π ∈ Ĝ| ker π ⊃ H_{n}}, Ĝ_{𝕋}: = {π ∈ Ĝ| ker π ⊃ 𝕋} \ Ĝ_{Sp(n,ℝ)} and Ĝ_{faithful} : = Ĝ \ (Ĝ_{Sp(n,ℝ)} ∪ Ĝ_{𝕋}) . Since the center is compact here, the application of (3.1) is more easy than above:

Corollary 3.4. Let π be a C_{0} –representation of G = H_{n} ⋊ Sp(n, ℝ) ,

then for sufficiently large k, π^{⊗k} ≤^{q} λ_{G} .

Proof. We write π as direct integral

π = ∫ ⊕ ; Ĝ_{𝕋} + ∫ ⊕ ; Ĝ_{faithful} ϱdμ(ϱ) =: π_{𝕋} ⊕ π_{faithful} .

The first summand satisfies π⊗m_{𝕋} ≤^{q} λ_{G} for large enough m , by Theorem

1.1, and the second π_{faithful}^{⊗2n+2} ≤^{q} λ_{G} , by (3.1). Thus, for k large enough,

π^{⊗k} ≤^{q} ∑_{ℓ=0}^{k} π⊗k−ℓ_{𝕋} ⊗ π_{faithful}^{⊗ℓ} ≤^{q} λ_{G} .

sl(2,R)Compact Central Extensions of Linear Reductive Groups : At first we consider an example. Let G be a simple analytic group with infinite center Z , e.g., the universal covering group of Sp(n, ℝ) . Then Z is a discrete, finitely generated abelian group, algebraically Z ≃ ℤ^{n} × Tor , where Tor denotes the torsion part. Let c_{1}, c_{2}, …, c_{n} be generators of the free part of Z and t ∈ 𝕋 be of infinite order. Then

L: = {((c_{1}^{k_{1}}, t^{k_{1}}), (c_{2}^{k_{2}}, t^{k_{2}}), … (c_{n}^{k_{n}}, t^{k_{n}})), k_{i} ∈ ℤ}

is a discrete central subgroup of G × 𝕋^{n} . Define the canonical projection

p: G × 𝕋^{n} → \overline{G}: = (G × 𝕋^{n})/L .

By the discreteness of L , we have Z(\overline{G}) = p(Z(G × 𝕋^{n})) = p(Tor × 𝕋^{n}) ([26],III.3.2). Thus the center of \overline{G} is compact and \overline{G}/Z(\overline{G}) ≃ G/Z ≃ AdG is a linear simple group. Hence \overline{G} is totally minimal itself ([35],2.3). Observe that \overline{G} contains a dense subgroup isomorphic to G , whence it is not linear. Using the results of Cowling and Moore presented in the last section, we can show the following

Theorem 3.5. Let G be an analytic group with compact center Z , such that G/Z is a linear reductive Kazhdan group with compact center. Then B_{0}(G) = \overline{A_{r}(G)} .

Proof. Let π be a C_{0} –representation of G . By the compactness of Z , π splits as π = ∑_{γ∈Ẑ} π_{γ} , where π_{γ}|_{Z} ≤^{q} γ for all γ ∈ Ẑ . Thus it remains to consider π_{γ} . Then π_{γ} ⊗ \overline{π_{γ}} is a C_{0} –representation of the Kazhdan group G/Z , whence by (2.12), (π_{γ} ⊗ \overline{π_{γ}})^{⊗k_{0}} ∈ λ_{G/Z} for a finite k_{0} ∈ ℕ . By the Kunze-Stein phenomenon (2.10.iii), all coefficients of π_{γ} ⊗ \overline{π_{γ}} belong to L^{2k}(G/Z) ⊆ L^{2k}(G) for all k > k_{0} . Thus, for all ξ, η ∈ H_{π} : |φ_{ξη}|^{2k} ∈ L^{2}(G) . By Hölder’s inequality, π_{γ}^{⊗2k} is squareintegrable, showing the theorem.

I suppose the result to be true in general.

References

[1] Arsac, G., Sur l’Espace de Banach Engendré par les Coéfficients d’une Représentation Unitaire, Publ. Dépt. Math. Lyon 13 (1976), 1–101.

[2] Baggett, L., Unimodularity and Atomic Plancherel Measure, Math. Ann. 266 (1984), 513–518.

[3] Baggett, L., and K. Taylor, Groups with Completely Reducible Regular Representation, Proc. Amer. Math. Soc. 72 (1978), 593–600.

[4] —, A Sufficient Condition for the Complete Reducibility of the Regular Representation, J. Funct. Anal. 34 (1979), 250–265.

[5] Baklouti, A., and J. Ludwig, La Désintégration des Représentations Unitaires des Groupes de Lie Nilpotents, Preprint.

[6] Bekka, M. E. B., P.-A. Cherix, and A. Valette, Proper Affine Isometric Actions of Amenable Groups, In: London Math. Soc. Lecture Notes Ser. 227, Cambridge University Press, 1995.

[7] Bekka, M. E. B., and M. Mayer, “Ergodic Theory and Topological Dynamics for Group Actions on Homogeneous Spaces,” Preversion.

[8] Berglund, J., H. Jungham, and P. Milnes, “Analysis on Semi groups,” J. Wiley and Sons, 1989.

[9] Brown, I. D., Dual Topology of a Nilpotent Group, Ann. Sci. Ecole Normale Sup.(4)6 (1973), 407–411.

[10] Corwin, L. W., and F. P. Greenleaf, “Representations of Nilpotent Lie Groups and their Applications,” Cambridge, 1990.

[11] Cowling, M., Sur les Coéfficients des Représentations Unitaires des Groupes de Lie Simple, In: Springer Lecture Note 739 (1979), 132– 178.

[12] —, The Kunze-Stein Phenomenon, Ann. Math. 207 (1978), 209–234.

[13] —, The Fourier-Stieltjes Algebra of a Semisimple Group, Colloq. Math. 41 (1979), 84–94.

[14] Dixmier, J., “C^{∗}−Algebras,” North Holland, Amsterdam, 1981.

[15] Duflo, M., and C. C. Moore, On the regular Representation of a Nonunimodular Locally Compact Group, J. Funct. Anal. 21 (1976), 209–243.

[16] Eymard, P., L’Algèbre de Fourier d’un Groupe Localement Compact, Bull. Soc. Math. France 92 (1964), 181–236.

[17] —, A Survey of Fourier Algebras, Contemp. Math. 183 (1995), 111– 128.

[18] Fell, J., Weak Containment and Induced Representations of Groups, Canad. J. Math. 14 (1962), 237–268.

[19] —, A New Proof that Nilpotent Groups are CCR, Proc. Amer. Math. Soc. 13 (1962), 93–99.

[20] —, Weak Containment and Kronecker Products of Group Representations, Pacific J. Math. 13 (1963), 503–510.

[21] Felix, R., Über Integralzerlegungen von Darstellungen nilpotenter Gruppen, Manuscripta math. 27 (1979), 279–290.

[22] Figá-Talamanca, A., Positive Definite Functions that Vanish at Infinity, Pacific J. Math. 69 (1977), 355–363.

[23] Figá-Talamanca, A., and M. Picardello, Functions that Operate on B_{0}(G), Pacific J. Math. 74 (1978), 57–61.

[24] Folland G., “A Course in Abstract Harmonic Analysis,” CRC Press, Boca Raton, 1995.

[25] Führ, H., and M. Mayer, Continuous Wavelet Transforms from Cyclic Representations: A General Approach Using Plancherel Measures, submitted.

[26] Hilgert J. and K. H. Neeb, “Lie Gruppen und Lie Algebren,” Vieweg, 1991.

[27] Howe, R., and C. C. Moore, Asymptotic Properties of Unitary Representations, J. Funct. Anal. 32 (1979), 72–96.

[28] Howe, R., and Eng Che Tan, “Non–Abelian Harmonic Analysis, ” Springer, Berlin, 1992.

[29] Isham, C. J., and J. R. Klauder, Coherent States for n-dimensional Euclidean Groups E(n) and their Application, J. Math. Phys. 32 (1991), 607–620.

[30] Khalil, I., Sur l’Analyse Harmonique du Groupe Affine de la Droite, Studia Math. 51 (1974), 139–167.

[31] Klauder, J. R., and R. F. Streater, A wavelet transform for the Poincaré group, J. Math. Phys. 32 (1991),1609–1611.

[32] Lipsman, R., “Group Representations, ” Springer Lecture Notes in Math. 388, 1974.

[33] Liukkonen, J., and M. Mislove, Symmetry in Fourier-Stieltjes Algebras, Math. Ann. 217 (1975), 97–112.

[34] —, Fourier-Stieltjes Algebras of Compact Extensions of Nilpotent Groups, J. reine angew. Math. 325 (1981), 210–220.

[35] Mayer, M., Asymptotics of Matrix Coefficients and Closures of Fourier-Stieltjes Algebras, J. Funct. Anal. 143 (1997), 42–54.

[36] —, Strongly Mixing Groups, Semigroup Forum 54 (1997), 303–316.
[37] —, Harmonic Analysis on ℝ^{2} ⋊ SL(2, ℝ), Submitted.

[38] Moore, C. C., Exponential Decay of Correlations for Geodesic Flows, in: Group Representations, Ergodic Theory, Operator Algebras and Mathematical Physics, Proc. Conf. in honor of G. W. Mackey, MSRI Publications, Springer 1987, 163–181.

[39] Richardson, R., Principal Orbit Types for Algebraic Transformation Spaces in Characteristic Zero, Invent. Math. 116 (1972), 6–14.

[40] Rieffel, M. A., Square-Integrable Representations of Hilbert Algebras, J. Funct. Anal. 3 (1969), 265–300.

[41] Walter, M., On a Theorem of Figaá-Talamanca, Proc. Amer. Math. Soc. 60 (1976), 72–74.

[42] Zimmer, R., “Ergodic Theory and Semisimple Groups,” Birkhäuser, Boston, 1984.

Matthias Mayer Zentrum Mathematik Technische Universität München D-80290 München mayerm@mathematik.tumuenchen.de

Received June 25, 1998 and in final form January 27, 1999