Journal of Lie Theory Volume 13 (2003) 133–154 \begin{matrix}c \\ ◯ \end{matrix} 2003 Heldermann Verlag

Kazhdan Constants and Matrix Coefficients of Sp (n, R)

Markus Neuhauser^{∗}

Communicated by Alain Valette

Abstract. An infinitesimal Kazhdan constant of Sp (2, R) is computed. The methods used to prove this can also be employed to determine a quantitative estimate of the asymptotics of the matrix coefficients of Sp (n, R) in an elementary manner. An application of the result gives explicit Kazhdan constants for Sp (n, R), n ≥ 2.

1. Introduction

A locally compact group G has Kazhdan’s property T, if for a compact subset Q ⊂ G and an ε > 0, every unitary representation π which has a (Q, ε)-invariant vector, i. e. a vector ξ ∈ H_{π} such that ∥π (g) ξ − ξ∥ < ε ∥ξ∥ for all g ∈ Q, has in fact a nonzero invariant vector. If such (Q, ε) for a group exists it is called a Kazhdan pair. This group theoretic property introduced in [9] has remarkable applications, for an account see [5] and [12].

In this paper, by “representation” we shall always mean “unitary representation”. Let G be a connected Lie group. If π is a representation of G, a vector ξ ∈ H_{π} is called a C^{∞}-vector if g ↦ ⟨π (g) ξ, η⟩ is a C^{∞}-function for all η ∈ H_{π}, cf. for example [14]. The space of C^{∞}-vectors is denoted by H_{π}^{∞}. Let K be a maximal compact subgroup of G. A vector ξ ∈ H_{π} is K -finite if the linear span of π (K) ξ is finite-dimensional. We denote by H_{π,K}^{∞} the space of K -finite, C^{∞}-vectors in H_{π}.

Let X_{1}, …, X_{m} be a basis of the Lie algebra of G, then Δ = − ∑_{k=1}^{m} X_{k}^{2} denotes the Laplacian. If π is a representation of G, let dπ denote the derived representation of the Lie algebra. It can be extended to the universal enveloping algebra.

In [1, Theorem 3.10], it was shown that property T for a connected Lie group G, is equivalent to the existence of an ε > 0 such that

⟨dπ (Δ) ξ, ξ⟩ ≥ ε ∥ξ∥^{2}

for every ξ ∈ H_{π}^{∞} and every π without nonzero fixed vector.

ISSN 0949–5932 / $2.50 \begin{matrix}c \\ ◯ \end{matrix} Heldermann Verlag

∗Supported by grant 20-65060.01 of the Swiss National Fund for Scientific Research.

In [2, page 94], it was shown that restriction to the space H_{π,K}^{∞} is possible, namely:

Theorem 1.1. The connected Lie group G has property T if and only if there exists a constant ε > 0 such that

inf {⟨dπ (Δ) ξ, ξ⟩ : ξ ∈ H_{π,K}^{∞}, ∥ξ∥ = 1} ≥ ε

for any unitary representation π of G without nonzero fixed vector.

We define the infinitesimal Kazhdan constant as

κ_{K} (Δ, G) = inf {⟨dπ (Δ) ξ, ξ⟩ : ξ ∈ H_{π,K}^{∞}, ∥ξ∥ = 1, π ≯ 1}.

The symplectic group Sp (n, R) ⊂ GL (2n, R) is the group of isometries of the skew symmetric bilinear form induced by

J = ( \begin{matrix}0 & −I_{n} \\ I_{n} & 0 \end{matrix} ),

where I_{n} is the n × n identity matrix. So

Sp (n, R) = {g ∈ GL (2n, R) : g^{T} Jg = J}.

In the following G = Sp (n, R) and K = Sp (n, R) ∩ SO (2n) is the standard maximal compact subgroup of G.

Let π be a strongly continuous representation of G on a Hilbert space H_{π}. A vector ξ ∈ H_{π} is called K -finite if the linear span of the set π (K) ξ in H_{π} is finite-dimensional. Denote this dimension by δ (ξ) = dim ⟨π (K) ξ⟩.

Theorem 1.2. For every Sp (2, R)∩SO (4)-finite unit C^{∞}-vector η and every representation π of Sp (2, R) without nonzero invariant vectors

⟨dπ (Δ) η, η⟩ ≥ \frac{1}{4π} sup_{0<ϑ<π/2} \frac{(sin (2ϑ))^{2}}{ϑ} > 0. 11532

for a suitable Laplacian Δ on Sp (2, R), described after Theorem 3.5.

By Theorem 1.1, this implies that Sp (2, R) has Kazhdan’s property T which was shown for any local field in [4] and [13] and with an elementary proof in [3].

The group G can be decomposed as G = KAK, where

A = \{( \begin{matrix}a & 0 \\ 0 & a^{−1} \end{matrix} ) : a = diag (a_{1}, …, a_{n})\},

the subgroup of the diagonal matrices in G. In fact, the matrices in A in the decomposition can be chosen more specially as G = KA^{+}K, where

A^{+} = \{( \begin{matrix}a & 0 \\ 0 & a^{−1} \end{matrix} ) : \begin{matrix}a = diag (a_{1}, …, a_{n}) , \\ a_{1} ≥ a_{2} ≥ … ≥ a_{n} ≥ 1 \end{matrix} \}.

This can be achieved by suitable conjugation of an element of A by permutation matrices contained in K.

The asymptotics of the matrix coefficients will be given for the dense subspace of K -finite vectors of a representation π.

The quantitative estimate of the asymptotic of matrix coefficients will be given in terms of the Harish-Chandra function Ξ defined by

Ξ ( \begin{matrix}a & 0 \\ 0 & a^{−1} \end{matrix} ) = \frac{1}{2π} a^{−1} ∫_{0}^{2π} |a^{−4} (cos ϑ)^{2} + (sin ϑ)^{2}|^{−1/2} dϑ,

cf. for example [7, page 215].

Let g ∈ G with the decomposition g = k_{1}hk_{2}, k_{1}, k_{2} ∈ K,

h = diag (a_{1}, …, a_{n}, a_{1}^{−1}, …, a_{n}^{−1}) ∈ A^{+},

then define

Ψ (g) = Ξ ( \begin{matrix}\sqrt{a_{1}a_{2}} & 0 \\ 0 & \sqrt{a_{1}a_{2}}^{−1} \end{matrix} ) .

The next theorem gives a quantitative estimate for the asymptotics of matrix coefficients.

Theorem 1.3. Let π be a strongly continuous representation of Sp (n, R), n ≥ 2, without nonzero invariant vectors, then

|φ_{ξ,η} (g)| ≤ ∥ξ∥ ∥η∥ \sqrt{δ (ξ) δ (η)}Ψ (g)

for two K -finite vectors ξ, η ∈ H_{π}, where φ_{ξ,η} (g) = ⟨π (g) ξ, η⟩.

Here the main application of this theorem is the proof of Kazhdan’s property T of Sp (n, R), n ≥ 2, with an explicit Kazhdan pair.

Theorem 1.4. Let 0 < δ < 1, ε = 0. 32 × \sqrt{2δ}, and Q = Ψ^{−1} ([1 − δ, 1]), then (Q, ε) is a Kazhdan pair of Sp (n, R).

It is a pleasure to thank M. B. Bekka, H. Führ, G. Schlichting, A. Valette, and the referee for their comments and suggestions. Financial support form the Centre de Coopération Universitaire Franco-Bavarois is acknowledged.

2. Preliminaries

A set in the dual space of S^{2} (R^{2}) is determined, where S^{2} (R^{2}) is identified with the vector space of the symmetric 2 × 2-matrices. This set will be important for the computation of an explicit estimate of the infinitesimal Kazhdan constant of Sp (2, R) in Section 3 and for the determination of an explicit quantitative estimate of the asymptotics of matrix coefficients of Sp (n, R) in Section 5. The asymptotics will be employed in the last section to obtain a Kazhdan pair for Sp (n, R).

For SL (3, R) M. B. Bekka and M. Mayer in [2] have determined a lower bound of the infinitesimal Kazhdan constant associated with a Laplacian.

Let π be a representation of Sp (2, R) on H_{π}. The strategy for establishing the estimates consists in considering the restriction of π to SL (2, R) ⋉ S^{2} (R^{2}).

There is a spectral measure on the dual group N̂ corresponding to π|_{N} where N = S^{2} (R^{2}) is an abelian subgroup. The main problem here will be to find a set W ⊂ N̂ of which the spectral measure can be computed and estimated under the action of a suitably defined one parameter subgroup.

The subgroup

P = \{( \begin{matrix}a & b \\ 0 & a^{T}^{−1} \end{matrix} ) : a ∈ SL (2, R) , ab^{T} = ba^{T} \} ≅ SL (2, R) ⋉ S^{2} (R^{2})

will be considered. If ξ is a vector fixed by the subgroup

N = \{( \begin{matrix}1 & 0 & x & y \\ 0 & 1 & y & z \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix} ) : x, y, z ∈ R\} ≅ R^{3},

then ξ is a fixed vector of Sp (2, R), cf. for example [10, page 88].

So it can be supposed that π has no nonzero N -invariant vector. Let E be the spectral measure of N̂. Then π|_{N} = ∫N̂ χ dE (χ) and E ({0}) = 0. For Borel sets W ⊂ N̂ we have E (a · W) = π (a) E (W) π (a)^{−1} for all a ∈ SL (2, R).

Let ρ denote the action of SL (2, R) on S^{2} (R^{2}) by ρ (a) b = aba^{T}. We have

that a^{T} = ωa^{−1}ω^{−1} for ω = ( \begin{matrix}0 & −1 \\ 1 & 0 \end{matrix} ) and a ∈ SL (2, R). So the dual operation

on N̂ is equivalent to the usual operation ρ since tr (ba^{T} ca) = tr (aba^{T} c). The following basis

s_{1} = ( \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ) , s_{2} = ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ) , s_{3} = ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} )

of S^{2} (R^{2}) is chosen. The isomorphism

( \begin{matrix}x \\ y \\ z \end{matrix} ) ↦ xs_{1} + ys_{2} + zs_{3} = ( \begin{matrix}x + y & z \\ z & x − y \end{matrix} )

yields an identification between N̂ ≅ N ≅ S^{2} (R^{2}) and R^{3}. The spectral measure E is now considered to be defined on R^{3}.

For an angle 0 < ϑ < π and h ∈ R define

S_{h}^{+} (ϑ) = \{( \begin{matrix}x \\ hx + y cos β \\ y sin β \end{matrix} ) : x ∈ R, y > 0, −ϑ < β ≤ ϑ\}.

For 0 < ϑ < \frac{π}{2} one has

S_{h}^{+} (ϑ) = \{( \begin{matrix}x \\ hx + y \\ y tan β \end{matrix} ) : x ∈ R, y > 0, −ϑ < β ≤ ϑ\}.

Let g_{0} (α) = ( \begin{matrix}cos (α/2) & − sin (α/2) \\ sin (α/2) & cos (α/2) \end{matrix} ). Then in the chosen basis g_{0} (α) acts on

R^{3} by

( \begin{matrix}1 & 0 & 0 \\ 0 & cos α & − sin α \\ 0 & sin α & cos α \end{matrix} ).

Hence

g_{0} (α) · S_{0}^{+} (ϑ) = \{( \begin{matrix}x \\ y cos β \\ y sin β \end{matrix} ) : x ∈ R, y > 0, −ϑ + α < β ≤ ϑ + α\}.

This implies that g_{0} (2ϑ) · S_{0}^{+} (ϑ) and S_{0}^{+} (ϑ) are disjoint.

Let ξ be a unit eigenvector of the image π (K), then π (g_{0} (α)) ξ = e^{inα/2}ξ for an n ∈ Z. As g_{0} (α) ∈ SL (2, R),

π (g_{0} (α)) E (S_{0}^{+} (ϑ)) ξ = π (g_{0} (α)) E (S_{0}^{+} (ϑ)) π (g_{0} (α))^{−1} π (g_{0} (α)) ξ
= E (g_{0} (α) · S_{0}^{+} (ϑ)) π (g_{0} (α)) ξ
= e^{inα/2}E (g_{0} (α) · S_{0}^{+} (ϑ)) ξ

and so

∥E (S_{0}^{+} (ϑ)) ξ∥ = ∥π (g_{0} (α)) E (S_{0}^{+} (ϑ)) ξ∥
= ∥E (g_{0} (α) · S_{0}^{+} (ϑ)) ξ∥.

On the other hand S_{0}^{+} (ϑ) and g_{0} (α) · S_{0}^{+} (ϑ) are disjoint for 2ϑ ≤ α ≤ 2π − 2ϑ. Hence E (S_{0}^{+} (ϑ)) E (g_{0} (α) · S_{0}^{+} (ϑ)) = 0 and E (S_{0}^{+} (ϑ)) ξ is orthogonal to E (g_{0} (α) · S_{0}^{+} (ϑ)) ξ.

Let now n ≥ 2, ϑ = π/n, and α_{j} = 2πj/n for 0 ≤ j ≤ n − 1, then

R^{3} \ {0} = ⋃_{j=0}^{n−1} g_{0} (α_{j}) · S_{0}^{+} (ϑ)

where the union is disjoint. So ∑_{j=0}^{n−1} E (g_{0} (α_{j}) · S_{0}^{+} (ϑ)) = id_{H_{π}}. This way ξ can be decomposed into vectors of equal length ξ = ∑_{j=0}^{n−1} E (g_{0} (α_{j}) · S_{0}^{+} (ϑ)) ξ. If ξ is a unit vector

∥E (S_{0}^{+} (ϑ)) ξ∥^{2} = 1/n = ϑ/π.

This equality can be extended first to all ϑ = rπ with r ∈ Q ∩]0, 1[ and then to all r ∈ ]0, 1[. This proves the following.

Lemma 2.1. For 0 < ϑ < π:

∥E (S_{0}^{+} (ϑ)) ξ∥^{2} = ϑ/π.

Let now

S_{h}^{−} (ϑ) = \{( \begin{matrix}x \\ hx + y cos β \\ y sin β \end{matrix} ) : x ∈ R, y < 0, −ϑ < β ≤ ϑ\}
= g_{0} (π) · S_{−h}^{+} (ϑ) ,

then ∥E (S_{0}^{−} (ϑ)) ξ∥^{2} = ∥E (S_{0}^{+} (ϑ)) ξ∥^{2} = ϑ/π.

Let S_{0} (ϑ) = S_{0}^{+} (ϑ) ∪ S_{0}^{−} (ϑ), W^{+} (ϑ) = S_{1}^{+} (ϑ) ∩ S_{0}^{+} (2ϑ), W^{−} (ϑ) = S_{1}^{−} (ϑ) ∩ S_{0}^{−} (2ϑ), and W (ϑ) = W^{−} (ϑ) ∪ W^{+} (ϑ) for 0 < ϑ < π/2.

The determination of the spectral measure of W (ϑ) is a more difficult task. This will be done in the next section.

3. Kazhdan constants associated with a Laplacian

The next proposition is an important step in the determination of the infinitesimal Kazhdan constant associated with Δ.

Proposition 3.1. For 0 < ϑ < π/2 and a unit K -eigenvector ξ the spectral measure is ∥E (W (ϑ)) ξ∥^{2} = 2ϑ/π.

The proof is postponed to Appendix A.

Now it will be investigated how W (ϑ) behaves under the action of the one

parameter group g_{1} (t) = ( \begin{matrix}exp (t/2) & 0 \\ 0 & exp (−t/2) \end{matrix} ). Then g_{1} (t) acts on S^{2} (R^{2})

with the above basis by

( \begin{matrix}cosh t & sinh t & 0 \\ sinh t & cosh t & 0 \\ 0 & 0 & 1 \end{matrix} ).

Hence

g_{1} (t) · S_{1}^{+} (ϑ) = \{( \begin{matrix}x \\ x + y \\ ye^{t} tan β \end{matrix} ) : x ∈ R, y > 0, −ϑ < β ≤ ϑ\}
= S_{1}^{+} (arctan (e^{t} tan ϑ))

and

g_{1} (t) · W^{±} (ϑ) = S_{1}^{±} (arctan (e^{t} tan ϑ)) ∩ (g_{1} (t) · S_{0}^{±} (2ϑ)).

Here

g_{1} (t) · S_{0}^{±} (ϑ)
= \{( \begin{matrix}x cosh t + y cos β sinh t \\ x sinh t + y cos β cosh t \\ y sin β \end{matrix} ) : x ∈ R, ±y > 0, −ϑ < β ≤ ϑ\}.

Since x ∈ R is arbitrary, replace x by \frac{x−y cos β sinh t}{cosh t}. Then the first coordinate becomes x and the second (x − y cos β sinh t) tanh t + y cos β cosh t = x tanh t + \frac{y cos β}{cosh t}. So

g_{1} (t) · S_{0}^{±} (ϑ) = \{( \begin{matrix}x \\ x tanh t + \frac{y cos β}{cosh t} \\ y sin β \end{matrix} ) : x ∈ R, ±y > 0, −ϑ < β ≤ ϑ\}
= S_{tanh t}^{±} (arctan (cosh t tan ϑ)) .

The next proposition determines a ϑ_{t} dependent of t and ϑ such that W (ϑ_{t}) is contained in g_{1} (t) · W (ϑ) giving in the corollary below a lower bound for the spectral measure of W (ϑ_{t}) as an immediate consequence.

Neuhauser 139

Proposition 3.2. For 0 < ϑ < π/2 and t > 0 holds g_{1} (t) · W (ϑ) ⊇ W (arctan (e^{t} tan ϑ)).

The proof is postponed to Appendix B. Corollary 3.3. For 0 < ϑ < π/2 and t > 0,

∥E (g_{1} (t) · W (ϑ)) ξ∥^{2} ≥ \frac{2}{π} arctan (e^{t} tan ϑ).

The purpose of all this is to obtain an estimate of ∥dπ (Y_{1}) ξ∥, where Y_{1} = \frac{1}{2} ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ), for a smooth SO (2)-finite unit vector ξ. Observe that g_{1} (t) = exp (tY_{1}). Proposition 3.4. Let π be a representation of SL (2, R) ⋉ S^{2} (R^{2}) without nonzero S^{2} (R^{2})-invariant vectors, then

∥dπ (Y_{1}) ξ∥ ≥ \frac{1}{2\sqrt{2π}} \begin{matrix}sin & (2ϑ) \\ √ & \overline{ϑ} \end{matrix}

for every smooth SO (2)-eigenvector ξ with ∥ξ∥ = 1. Proof. For ξ smooth of norm 1:

∥E (g_{1} (t) · W (ϑ)) ξ∥ = ∥π (g_{1} (t)) E (W (ϑ)) π (g_{1} (−t)) ξ∥
= ∥E (W (ϑ)) π (g_{1} (−t)) ξ∥.

Differentiating at t = 0 yields

\frac{d}{dt} ∥E (g_{1} (t) · W (ϑ)) ξ∥^{2}|_{t=0}
= \frac{d}{dt} ∥E (W (ϑ)) π (g_{1} (−t)) ξ∥^{2}|_{t=0}
= − ⟨dπ (Y_{1}) ξ, E (W (ϑ)) ξ⟩ − ⟨E (W (ϑ)) ξ, dπ (Y_{1}) ξ⟩.

If f is a real function differentiable at 0 with f (0) = 0 and f (x) ≥ 0 for x ≥ 0, then f′ (0) ≥ 0. Together with Corollary 3.3 this implies

\frac{d}{dt} ∥E (g_{1} (t) · W (ϑ)) ξ∥^{2}|_{t=0}
≥ \frac{2}{π} \frac{d}{dt} arctan (e^{t} tan ϑ)|_{t=0} = \frac{2}{π} \frac{1}{1 + (tan ϑ)^{2}} tan ϑ
= \frac{2}{π} (cos ϑ)^{2} tan ϑ = \frac{2}{π} cos ϑ sin ϑ = \frac{1}{π} sin (2ϑ) .

Hence

2 ∥dπ (Y_{1}) ξ∥ \sqrt{\frac{2ϑ}{π}} ≥ − ⟨dπ (Y_{1}) ξ, E (W (ϑ)) ξ⟩ − ⟨E (W (ϑ)) ξ, dπ (Y_{1}) ξ⟩
= \frac{d}{dt} ∥E (g_{1} (t) · W (ϑ)) ξ∥^{2}|_{t=0}
≥ \frac{1}{π} sin (2ϑ)

and

∥dπ (Y_{1}) ξ∥ ≥ \frac{1}{2\sqrt{2π}} \begin{matrix}sin & (2ϑ) \\ √ & \overline{ϑ} \end{matrix}

for every smooth K -eigenvector ξ of norm 1.

For Y_{2} = \frac{1}{2} ( \begin{matrix}0 & 1 \\ 1 & 0 \end{matrix} ), conjugate to Y_{1}, the same equality holds. Together

with Y_{0} = \frac{1}{2} ( \begin{matrix}0 & −1 \\ 1 & 0 \end{matrix} ), the three elements Y_{0}, Y_{1}, Y_{2} form a basis of the Lie alge-

bra of SL (2, R) orthogonal with respect to the Killing form. The corresponding Casimir operator is C = \frac{1}{2} (Y_{1}^{2} + Y_{2}^{2} − Y_{0}^{2}) and the corresponding Laplacian is Δ = −Y_{1}^{2} − Y_{2}^{2} − Y_{0}^{2} = −2C − 2Y_{0}^{2}.

Theorem 3.5. Let π be a representation of SL (2, R)⋉S^{2} (R^{2}) without nonzero S^{2} (R^{2})-invariant vectors, then

⟨dπ (Δ) η, η⟩ ≥ \frac{1}{4π} sup_{0<ϑ<π/2} \frac{(sin (2ϑ))^{2}}{ϑ}

for every smooth SO (2)-finite unit vector η.

Proof. Let η = ∑_{k=1}^{r} ξ_{k} be the orthogonal decomposition of η into dπ (Y_{0})- eigenvectors, then the observation that C commutes with Y_{0} implies

⟨dπ (Δ) η, η⟩
= ⟨dπ (−2Y_{0}^{2}) η, η⟩ + ⟨dπ (−2C) η, η⟩
= ∑_{k=1}^{r} ⟨dπ (−2Y_{0}^{2}) ξ_{k}, ξ_{k}⟩ + ⟨dπ (−2C) ξ_{k}, ξ_{k}⟩ = ∑_{k=1}^{r} ⟨dπ (Δ) ξ_{k}, ξ_{k}⟩

≥ ∑_{k=1}^{r} 2 (\frac{1}{2\sqrt{2π}} \begin{matrix}sin & (2ϑ) \\ √ & \overline{ϑ} \end{matrix})^{2} ∥ξ_{k}∥^{2} = \frac{1}{4π} \frac{sin^{2} (2ϑ)}{ϑ} ∥η∥^{2} .

The following basis of the Lie algebra sp (2, R) will be considered which contains elements corresponding to Y_{1} and Y_{2}. The Lie algebra sp (2, R) admits a Cartan decomposition into sp (2, R) = k ⊕ p where k = so (4, R) ∩ sp (2, R) and p = S^{2} (R^{2}) ∩ sp (2, R). With

X_{0} = \frac{1}{2} ( \begin{matrix}0 & −1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & −1 \\ 0 & 0 & 1 & 0 \end{matrix} ) , X_{1} = \frac{1}{2} ( \begin{matrix}1 & 0 & 0 & 0 \\ 0 & −1 & 0 & 0 \\ 0 & 0 & −1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix} ) ,

X_{2} = \frac{1}{2} ( \begin{matrix}0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & −1 \\ 0 & 0 & −1 & 0 \end{matrix} ) , X_{3} = \frac{1}{2} ( \begin{matrix}1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & −1 & 0 \\ 0 & 0 & 0 & −1 \end{matrix} )

and

X_{4} = \frac{1}{2} ( \begin{matrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{matrix} ) , X_{5} = \frac{1}{2} ( \begin{matrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & −1 \\ 1 & 0 & 0 & 0 \\ 0 & −1 & 0 & 0 \end{matrix} ) ,

X_{6} = \frac{1}{2} ( \begin{matrix}0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 0 \end{matrix} ) , X_{7} = \frac{1}{2} ( \begin{matrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \\ −1 & 0 & 0 & 0 \\ 0 & −1 & 0 & 0 \end{matrix} ) ,

X_{8} = \frac{1}{2} ( \begin{matrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & −1 \\ −1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \end{matrix} ) , X_{9} = \frac{1}{2} ( \begin{matrix}0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 \\ 0 & −1 & 0 & 0 \\ −1 & 0 & 0 & 0 \end{matrix} )

the Casimir operator satisfies

C = 2 (X_{1}^{2} + X_{2}^{2} + X_{3}^{2} + X_{4}^{2} + X_{5}^{2} + X_{6}^{2} − (X_{7}^{2} + X_{8}^{2} + X_{9}^{2} + X_{0}^{2})).

The elements X_{0}, X_{7}, X_{8}, X_{9} form a basis of k and X_{1}, X_{2}, X_{3}, X_{4}, X_{5}, X_{6} form a basis of p. Let η be a smooth Sp (2, R) ∩ SO (4)-finite unit vector, then

⟨dπ (Δ) η, η⟩ ≥ ⟨dπ (Δ_{1}) η, η⟩,

with Δ_{1} = −X_{0}^{2} − X_{1}^{2} − X_{2}^{2} = −2X_{0}^{2} − 2C_{1} where C_{1} = \frac{1}{2} (−X_{0}^{2} + X_{1}^{2} + X_{2}^{2}) is the Casimir operator of a Lie subalgebra isomorphic to sl (2, R). By Theorem 3.5 this shows that ⟨dπ (Δ) η, η⟩ ≥ (4π)^{−1} (sin (2ϑ))^{2} /ϑ for every smooth Sp (2, R) ∩ SO (4)-finite unit vector η of a representation π without nonzero S^{2} (R^{2})-invariant vectors.

To conclude the proof of Theorem 1.2 let π be a representation of Sp (2, R) without nonzero invariant vector. If the restriction to S^{2} (R^{2}) would have a nonzero invariant vector this would imply the contradiction that Sp (2, R) would have a nonzero invariant vector by an argument similar to the one for SL (2, R) in [10, page 88]. For more details see also the proof of Theorem 4.3. In the notation used there a nonzero S^{2} (R^{2})-invariant vector would imply a nonzero vector invariant under G_{1,1}, G_{1,2}, and G_{2,2} (see next section). But these three subgroups together generate Sp (2, R).

By Theorem 3.5 now only the maximum of the function ϑ ↦ (sin (2ϑ))^{2} /ϑ has to be considered which is obtained at approximately ϑ ≈ 0. 582781 so

\frac{1}{4π} sup_{0<ϑ<π/2} \frac{(sin (2ϑ))^{2}}{ϑ} ≈ 0. 115325 > 0. 11532.

4. Vanishing of matrix coefficients

In this section the qualitative behavior of the matrix coefficients of Sp (n, R) will be analyzed in an elementary manner. The case SL (n, R) was done in [7].

The following notion will be used.

Let X be a Hausdorff topological space. A complex valued function f is said to vanish at infinity if for every ε > 0 there exists a compact set C ⊂ X such that |f (x)| < ε for all x ∈ X \ C.

A sequence goes to ∞ in X if it has no limit point in X. If X is second countable a complex valued function f vanishes at ∞ if lim_{m→∞} f (x) = 0 for every sequence (x_{m})_{m∈N} in X going to ∞. This will be used for Sp (n, R).

The following is easily deduced from the fact that Sp (n, R) = KA^{+}K and π (K) is compact.

Lemma 4.1. Let π be a representation of Sp (n, R) on H_{π} such that the matrix coefficients do not vanish at infinity; then there are ξ, η ∈ H_{π} and a sequence (g_{m})_{m∈N} with g_{m} ∈ A^{+} and g_{m} → ∞ such that (⟨π (g_{m}) ξ, η⟩)_{m∈N} does not converge to 0.

The subgroup

N_{1} = \{( \begin{matrix}1 & 0 & x & y^{T} \\ 0 & I & y & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & I \end{matrix} ) : x ∈ R, y ∈ R^{n−1}\}

of Sp (n, R) will be important. The following proposition shows that a representation of Sp (n, R) which has a matrix coefficient that does not vanish at ∞ has in fact a nonzero vector which is N_{1}-invariant. The next theorem will show this vector is in fact invariant by proving that some specific subgroups generate Sp (n, R).

Proposition 4.2. Let π be a strongly continuous unitary representation of Sp (n, R) on H_{π} and suppose that a matrix coefficient of π does not vanish at ∞, then there is a nonzero N_{1}-invariant vector.

Proof. By Lemma 4.1 there is a sequence (g_{m})_{m∈N} which goes to infinity with g_{m} ∈ A^{+} and a ξ ∈ H_{π} such that the sequence (π (g_{m}) ξ)_{m∈N} does not converge weakly to 0. After passing to a subsequence it can be assumed that (π (g_{m}) ξ)_{m∈N} converges in the weak topology to η ≠ 0 since π (g_{m}) is unitary and the unit ball is compact in the weak topology.

Let g_{m} = ( \begin{matrix}a_{m} & 0 \\ 0 & a_{m}^{−1} \end{matrix} ) with

a_{m} = diag (a_{m,1}, …, a_{m,n}) , a_{m,1} ≥ … ≥ a_{m,n} ≥ 1.

As a_{m} → ∞, we have a_{m,1}^{−1} → 0. The elements g_{m}^{−1}hg_{m} converge to the identity I_{n} for m → ∞ and h ∈ N_{1} as

( \begin{matrix}a^{−1} & 0 \\ 0 & a \end{matrix} ) ( \begin{matrix}I_{n} & b \\ 0 & I_{n} \end{matrix} ) ( \begin{matrix}a & 0 \\ 0 & a^{−1} \end{matrix} ) = ( \begin{matrix}I_{n} & a^{−1}ba^{−1} \\ 0 & I_{n} \end{matrix} )

with a diagonal and b ∈ S^{2} (R^{n}),

( \begin{matrix}a_{m,1}^{−1} & 0 \\ 0 & d_{m}^{−1} \end{matrix} ) ( \begin{matrix}x & y^{T} \\ y & 0 \end{matrix} ) ( \begin{matrix}a_{m,1}^{−1} & 0 \\ 0 & d_{m}^{−1} \end{matrix} ) = ( \begin{matrix}a_{m,1}^{−2}x & a_{m,1}^{−1}y^{T} d_{m}^{−1} \\ a_{m,1}^{−1}d_{m}^{−1}y & 0 \end{matrix} )

with d_{m} = diag (a_{m,2}, …, a_{m,n}) and so a_{m,1}^{−2}x → 0 and a_{m,1}^{−1}d_{m}^{−1}y → 0 because a_{m,j} ≥ 1 for all j.

Next it is proven that η ∈ H_{π} is N_{1}-invariant. Let h ∈ N_{1} with h =

( \begin{matrix}I & b \\ 0 & I \end{matrix} ), then

|⟨π (h) η − η, ζ⟩| = lim_{m→∞} |⟨π (h) π (g_{m}) ξ − π (g_{m}) ξ, ζ⟩|
= lim_{m→∞} |⟨π (g_{m}) (π (g_{m}^{−1}hg_{m}) ξ − ξ) , ζ⟩|
≤ lim_{m→∞} ∥π (g_{m}) (π (g_{m}^{−1}hg_{m}) ξ − ξ)∥ ∥ζ∥
= lim_{m→∞} ∥π (g_{m}^{−1}hg_{m}) ξ − ξ∥ ∥ζ∥ = 0

for all ζ ∈ H_{π} because of the strong continuity of π. So π (h) η = η.

With the help of the last proposition the following yields an elementary proof that the matrix coefficients of Sp (n, R) vanish at infinity.

Let E_{j,k} ∈ R^{n×n} be the matrix which is zero in every entry except for the one at (j, k) which is 1. Let ρ_{j,k} : SL (2, R) → Sp (n, R) be the homomorphisms

ρ_{j,k} ( \begin{matrix}a & b \\ c & d \end{matrix} ) = ( \begin{matrix}I_{n} + (a − 1) (E_{j,j} + E_{k,k}) & b (E_{j,k} + E_{k,j}) \\ c (E_{j,k} + E_{k,j}) & I_{n} + (d − 1) (E_{j,j} + E_{k,k}) \end{matrix} )

for j, k = 1, …, n, j ≠ k,

ρ_{k,k} ( \begin{matrix}a & b \\ c & d \end{matrix} ) = ( \begin{matrix}I_{n} + (a − 1) E_{k,k} & bE_{k,k} \\ cE_{k,k} & I_{n} + (d − 1) E_{k,k} \end{matrix} )

for k = 1, …, n, and ρ̃_{j,k} : SL (2, R) → SL (n, R) the homomorphisms

ρ̃_{j,k} ( \begin{matrix}a & b \\ c & d \end{matrix} ) = I_{n} + (a − 1) E_{j,j} + bE_{j,k} + cE_{k,j} + (d − 1) E_{k,k}

for j, k = 1, …, n. Let

G_{j,k} = ρ_{j,k} (SL (2, R)) ,
G̃_{j,k} = \{( \begin{matrix}ρ̃_{j,k} (g) & 0 \\ 0 & (ρ̃_{j,k} (g)^{T} )^{−1} \end{matrix} ) : g ∈ SL (2, R)\}

for j, k = 1, …, n be the corresponding subgroups.

The first proof of the following was given in [6].

Theorem 4.3. Let π be a unitary representation of Sp (n, R) which does not contain the trivial representation, then the matrix coefficients of π vanish at infinity.

Proof. Assume by contradiction that at least one coefficient of π does not vanish at infinity.

For n = 1 a vector which is N_{1}-invariant is also invariant for Sp (n, R) = SL (2, R), see for example [10, page 88].

Now suppose n ≥ 2, then by Lemma 4.2 there is an N_{1}-invariant ξ. The case n = 1 implies that this vector is also G_{1,k} -invariant for k = 1, …, n. Let G be the subgroup of Sp (n, R) generated by these subgroups. It will be shown that G = Sp (n, R).

Let

ω = ( \begin{matrix}0 & −1 \\ 1 & 0 \end{matrix} ) , ω_{j,k} = ρ_{j,k} (ω) , ω̃_{j,k} = ρ̃_{j,k} (ω)

for j, k = 1, …, n. Then ω_{1,k}ρ_{1,1} (g) ω_{1,k}^{−1} = (ρ_{k,k} (g)^{T} )^{−1} for k = 2, …, n. This

implies G_{k,k} ⊂ G. Since ω_{1,1}ρ_{1,k} (g) ω_{1,1}^{−1} = (ρ̃_{1,k} (g)^{T} )^{−1} for k = 2, …, n, we

have G̃_{1,k} ⊂ G. Also ω̃_{1,j}ρ̃_{1,k} (g) ω̃_{1,j}^{−1} = ρ̃_{j,k} (g) for j, k = 2, …, n, j ≠ k which gives G̃_{j,k} ⊂ G. Finally ω̃_{1,j}ρ_{1,k} (g) ω̃_{1,j}^{−1} = ρ_{j,k} (g) for j, k = 2, …, n, j ≠ k and G_{j,k} ⊂ G.

This implies G = Sp (n, R), see [8, Section 6.9]. So ξ is G-invariant.

5. An estimate for the decay of the matrix coefficients

Before studying the decay of the matrix coefficients of Sp (n, R) the matrix coefficients of the semi-direct product SL (2, R) ⋉ S^{2} (R^{2}) are considered. A set in the unitary dual { \atop S^{2} (R^{2})} of the additive group of S^{2} (R^{2}) will help to determine an estimate for the matrix coefficients of the representations of SL (2, R) ⋉ S^{2} (R^{2}) without nonzero S^{2} (R^{2})-invariant vectors.

Theorem 5.1. Let π be a representation of SL (2, R)⋉S^{2} (R^{2}) on H_{π} without nonzero S^{2} (R^{2})-invariant vectors, then

|φ_{ξ,η} (g_{0} (α) g_{1} (t) g_{0} (β))| = |⟨π (g_{0} (α) g_{1} (t) g_{0} (β)) ξ, η⟩| ≤ c_{ξ,η}e^{−t/2}

for ξ, η ∈ H_{π,K} and c_{ξ,η} is a constant depending only on ξ and η.

Proof. Let Φ : R^{3} → { \atop S^{2} (R^{2})} be the isomorphism

(Φ (x, y, z)) (u) = exp (i tr (( \begin{matrix}x + z & y \\ y & z \end{matrix} ) u))

for u ∈ S^{2} (R^{2}). We identify R^{3} with { \atop S^{2} (R^{2})} via Φ. Let s > 1 and

X_{s} = \{( \begin{matrix}x \\ y \\ z \end{matrix} ) ∈ R^{3} : s^{−2} < y^{2} + z^{2} < s^{2}\},

then ⋃_{s>1} X_{s} = R^{3}\{0}. As π has no nonzero S^{2} (R^{2})-invariant vectors, E (X_{s}) η converges to η for η ∈ H_{π} where E is the spectral measure associated to π|_{S^{2}(R^{2})}. So it is enough to prove the statement for eigenvectors ξ, η ∈ E (X_{s}) H_{π} of π (K) as the matrix coefficients are sesquilinear in ξ and η.

Let t > 2 ln s, then

φ_{ξ,η} (g_{1} (t)) = ⟨π (g_{1} (t)) ξ, η⟩ = ⟨π (g_{1} (t)) E (X_{s}) ξ, E (X_{s}) η⟩
= ⟨E (g_{1} (t) · X_{s}) π (g_{1} (t)) ξ, E (X_{s}) η⟩
= ⟨π (g_{1} (t)) ξ, E ((g_{1} (t) · X_{s}) ∩ X_{s}) η⟩.

By the Cauchy–Schwarz inequality:

|φ_{ξ,η} (g_{1} (t))| ≤ ∥ξ∥ ∥E ((g_{1} (t) · X_{s}) ∩ X_{s}) η∥.

The one-parameter subgroup generated by g_{1} (t) operates in the following way on R^{3}. Then

g_{1} (−t) ( \begin{matrix}x + z & y \\ y & z \end{matrix} ) g_{1} (−t) = ( \begin{matrix}e^{−t} (x + z) & y \\ y & e^{t}z \end{matrix} )
= ( \begin{matrix}e^{−t}x − 2z sinh t + e^{t}z & y \\ y & e^{t}z \end{matrix} )

so by the isomorphism Φ

g_{1} (t) · ( \begin{matrix}x \\ y \\ z \end{matrix} ) = ( \begin{matrix}e^{−t}x − 2z sinh t \\ y \\ e^{t}z \end{matrix} ).

Hence

g_{1} (t) · X_{s} = \{( \begin{matrix}x \\ y \\ z \end{matrix} ) : s^{−2} < y^{2} + e^{−2t}z^{2} < s^{2}\} ⊂ \{( \begin{matrix}x \\ y \\ z \end{matrix} ) : |z| < e^{t}s\}.

As

(g_{1} (t) · X_{s}) ∩ X_{s} ⊂ \{( \begin{matrix}x \\ y \\ z \end{matrix} ) : y^{2} + z^{2} > s^{−2}, |z| < e^{t}s\}

we have |z| (\sqrt{y^{2} + z^{2}})^{−1} < e^{t}s (s^{−1})^{−1} = e^{t}s^{2}. Now z = r cos β with r =

\sqrt{y^{2} + z^{2}} where |cos β| < e^{t}s^{2}. Let ϑ = arccos (e^{t}s^{2}), then −π < β < π if and only if −π < β < −ϑ or ϑ < β < π. By definition of S_{h} (ϑ) and W (ϑ), cf. Section 3,

|φ_{ξ,η} (g_{1} (t))| ≤ \sqrt{1 − ∥E (S_{1} (ϑ)) ξ∥^{2}} ≤ \sqrt{1 − ∥E (W (ϑ)) ξ∥^{2}}
= \sqrt{1 − \frac{2}{π} arccos (e^{−t}s^{2})} = \sqrt{\frac{2}{π} arcsin (e^{−t}s^{2})}
≤ se^{−t/2}.

Finally for t ≤ 2 ln s, |φ_{ξ,η} (g_{1} (t))| ≤ 1 ≤ se^{−t/2} holds.

There is an estimate for the matrix coefficients of the regular representation of SL (2, R) which depends on the Harish-Chandra Ξ function, cf. for example [7, page 217]. For t ∈ R:

Ξ (g_{1} (t)) = (2π)^{−1} e^{−t/2} ∫_{0}^{2π} |e^{−2t} (cos ϑ)^{2} + (sin ϑ)^{2}|^{−1/2} dϑ.

Theorem 5.2. Let π be a representation of SL (2, R)⋉S^{2} (R^{2}) without nonzero S^{2} (R^{2})-invariant vectors, then for the matrix coefficient of any two vectors ξ, η ∈ H_{π} there is the pointwise estimate

|φ_{ξ,η} (g_{1} (t))| ≤ ∥ξ∥ ∥η∥ \sqrt{dim ⟨π (K) ξ⟩ dim ⟨π (K) η⟩}Ξ (g_{1} (t)) ,

where ⟨π (K) ξ⟩ is the subspace spanned by the orbit π (K) ξ.

The proof can be copied word by word form [7, page 226] replacing the corresponding statement by Theorem 5.1.

Hence it is possible to prove Theorem 1.3, which describes the asymptotics of matrix coefficients of Sp (n, R).

Proof of Theorem 1.3. Consider the subgroups

G̃_{1,2} = \{( \begin{matrix}a & 0 & 0 & 0 \\ 0 & I & 0 & 0 \\ 0 & 0 & a^{T}^{−1} & 0 \\ 0 & 0 & 0 & I \end{matrix} ) : a ∈ SL (2, R)\},

P_{1,2} = \{( \begin{matrix}a & 0 & b & 0 \\ 0 & I & 0 & 0 \\ 0 & 0 & a^{T}^{−1} & 0 \\ 0 & 0 & 0 & I \end{matrix} ) : a ∈ SL (2, R) , ab^{T} = ba^{T} \}

isomorphic to SL (2, R) and SL (2, R) ⋉ S^{2} (R^{2}) respectively.

Let π be a representation of Sp (n, R) without nonzero invariant vectors, then the representation π_{1,2} = π|_{P_{1,2}} also has no nonzero S^{2} (R^{2})-invariant vectors, as the matrix coefficients of π and hence the ones of π_{1,2} vanish at ∞, as shown in Theorem 4.3.

To G̃_{1,2} ⊂ Sp (n, R) the estimate of Theorem 5.2 is applied. Let

K_{1,2} = \{( \begin{matrix}a & 0 & 0 & 0 \\ 0 & I & 0 & 0 \\ 0 & 0 & a & 0 \\ 0 & 0 & 0 & I \end{matrix} ) : a ∈ SO (2, R)\}

be a maximal compact subgroup of G̃_{1,2}.

Let ω = ( \begin{matrix}I − E_{2,2} & −E_{2,2} \\ E_{2,2} & I − E_{2,2} \end{matrix} ) ∈ K, then

ωgω^{−1} = diag (a_{1}, a_{2}^{−1}, a_{3}, …, a_{n}, a_{1}^{−1}, a_{2}, a_{3}^{−1}, …, a_{n}^{−1}).

Now

( \begin{matrix}\sqrt{a_{1}a_{2}} & 0 \\ 0 & \sqrt{a_{1}a_{2}}^{−1} \end{matrix} ) ( \begin{matrix}\sqrt{a_{1}/a_{2}} & 0 \\ 0 & \sqrt{a_{1}/a_{2}} \end{matrix} ) = ( \begin{matrix}a_{1} & 0 \\ 0 & a_{2}^{−1} \end{matrix} ).

Let

g̃ = diag (\sqrt{a_{1}a_{2}}, \sqrt{a_{1}a_{2}}^{−1}, 1, …, 1, \sqrt{a_{1}a_{2}}^{−1}, \sqrt{a_{1}a_{2}}, 1, …, 1),
h = diag (\sqrt{a_{1}/a_{2}}, \sqrt{a_{1}/a_{2}}, a_{3}, …, a_{n}, \sqrt{a_{2}/a_{1}}, \sqrt{a_{2}/a_{1}}, a_{3}^{−1}, …, a_{n}^{−1}),

then

|φ_{ξ,η} (g)| = |⟨π (g) ξ, η⟩| = |⟨π \binom{}{g̃} π (hω) ξ, π (ω) η⟩|
≤ ∥ξ∥ ∥η∥ \sqrt{δ (ξ) δ (η)}Ξ ( \begin{matrix}\sqrt{a_{1}a_{2}} & 0 \\ 0 & \sqrt{a_{1}a_{2}}^{−1} \end{matrix} )

by Theorem 5.2, as π is unitary,

dim ⟨π (K_{1,2}) π (hω) ξ⟩ = dim ⟨π (h) π (K_{1,2}) π (ω) ξ⟩
= dim ⟨π (K_{1,2}) π (ω) ξ⟩,

and K_{1,2}ω ⊂ K.

6. Kazhdan pairs

For g ∈ Sp (n, R) there are k_{1}, k_{2} ∈ K and

h = diag (a_{1}, a_{2}, …, a_{n}, a_{1}^{−1}, a_{2}^{−1}, … a_{n}^{−1}) ∈ A^{+}

such that g = k_{1}hk_{2}. This implies

|φ_{ξ,η} (g)| = |⟨π (h) π (k_{2}) ξ, π (k_{1})^{−1} η⟩|
≤ ∥ξ∥ ∥η∥ \sqrt{δ (ξ) δ (η)}Ξ (g_{1} (ln (a_{1}a_{2}))) .

Let Ψ be defined by Ψ (g) = Ξ (g_{1} (ln (a_{1}a_{2}))).

Theorem 6.1. Let 0 < ε < 1 and δ = (4 sin (\frac{arcsin ε}{2}) + ε)^{2} /2 < 1, then (Ψ^{−1} ([1 − δ, 1]) , ε) is a Kazhdan pair of Sp (n, R).

The proof can again be copied word by word from [7, page 230–231] replacing SL (n, R) by Sp (n, R), δ by 1 − δ, and the corresponding statement by Theorem 5.2.

For given δ the ε in the last theorem can be estimated. We can now prove Theorem 1.4.

Proof of Theorem 1.4. Let at first 0 < ε < 1 be arbitrary. The Taylor expansion of x ↦ \sqrt{1 + x} at 0 shows

4 sin ((arcsin ε) /2) = 2\sqrt{2}\sqrt{1 − \sqrt{1 − ε^{2}}}
≥ 2\sqrt{2}\sqrt{1 − (1 − ε^{2}/2)} = 2ε

for 0 < ε < 1, hence 4 sin ((arcsin ε) /2) + ε ≥ 3ε ≥ \sqrt{2} for ε ≥ \sqrt{2}/3. So ε < \sqrt{2}/3 can be assumed. Again with the above mentioned Taylor expansion we have \sqrt{1 + x} ≤ 1 + x/2 for x ≥ −1 and so

\sqrt{1 + x} = \sqrt{1 − \frac{x}{1 + x}}^{−1} ≥ (1 − \frac{x}{2 + 2x})^{−1} = 1 + \frac{x}{2 + x}.

148 Neuhauser By letting x = −ε^{2} this yields

\sqrt{1 − \sqrt{1 − ε^{2}}}
≤ \sqrt{1 − (1 − \frac{ε^{2}}{2 − ε^{2}} )} = \frac{ε}{\sqrt{2}}\sqrt{\frac{1}{1 − ε^{2}/2}} = \frac{ε}{\sqrt{2}}\sqrt{1 + \frac{ε^{2}/2}{1 − ε^{2}/2}}
≤ \frac{ε}{\sqrt{2}} (1 + \frac{ε^{2}/2}{2 − ε^{2}} ) = \frac{ε}{\sqrt{2}} (1 + \frac{1}{4ε^{−2} − 2}) < \frac{ε}{\sqrt{2}} \frac{17}{16}

for 0 < ε < \sqrt{2}/3 and hence 4 sin ((arcsin ε) /2) + ε < (17/8 + 1) ε = (25/8) ε.

Now let 0 < ε = (8/25) \sqrt{2δ} < \sqrt{2}/3, then 4 sin ((arcsin ε) /2) + ε < \sqrt{2δ} and the last theorem shows that (Q, ε) is a Kazhdan pair.

A Proof of Proposition 3.1

The idea is to decompose W (ϑ) suitably such that it can be rearranged to S_{0} (ϑ) using only rotations g_{0} (α).

The union W (ϑ) = W^{−} (ϑ) ∪ W^{+} (ϑ) is disjoint and

W (ϑ) ∩ S_{0} (ϑ) = (W^{+} (ϑ) ∩ S_{0}^{+} (ϑ)) ∪ (W^{−} (ϑ) ∩ S_{0}^{−} (ϑ))

and

W (ϑ) \ S_{0} (ϑ) = (W^{+} (ϑ) ∪ W^{−} (ϑ)) \ S_{0} (ϑ)
= (W^{+} (ϑ) \ S_{0} (ϑ)) ∪ (W^{−} (ϑ) \ S_{0} (ϑ))
= (W^{+} (ϑ) \ S_{0}^{+} (ϑ)) ∪ (W^{−} (ϑ) \ S_{0}^{−} (ϑ)).

Hence ∥E (W (ϑ)) ξ∥^{2} = ∥E (W (ϑ) ∩ S_{0} (ϑ)) ξ∥^{2} + ∥E (W (ϑ) \ S_{0} (ϑ)) ξ∥^{2},

∥E (W (ϑ) ∩ S_{0} (ϑ)) ξ∥^{2}
= ∥E (W^{+} (ϑ) ∩ S_{0}^{+} (ϑ)) ξ∥^{2} + ∥E (W^{−} (ϑ) ∩ S_{0}^{−} (ϑ)) ξ∥^{2} ,

and

∥E (W (ϑ) \ S_{0} (ϑ)) ξ∥^{2}
= ∥E (W^{+} (ϑ) \ S_{0}^{+} (ϑ)) ξ∥^{2} + ∥E (W^{−} (ϑ) \ S_{0}^{−} (ϑ)) ξ∥^{2} .

Then

W^{±} (ϑ) \ S_{0}^{±} (ϑ) = (S_{1}^{±} (ϑ) ∩ S_{0}^{±} (2ϑ)) \ S_{0}^{±} (ϑ)
= S_{1}^{±} (ϑ) ∩ (S_{0}^{±} (2ϑ) \ S_{0}^{±} (ϑ))

where the sign is either everywhere + or everywhere − and

S_{0}^{±} (2ϑ) \ S_{0}^{±} (ϑ) = g_{0} (\frac{3ϑ}{2}) · S_{0}^{±} (\frac{ϑ}{2}) ∪ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{±} (\frac{ϑ}{2})

and the union is again disjoint.

We have

S_{0}^{+} (ϑ) = (W^{+} (ϑ) ∩ S_{0}^{+} (ϑ))

∪ g_{0} (π + 2ϑ) · (S_{1}^{−} (ϑ) ∩ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2}))

∪ g_{0} (π − 2ϑ) · (S_{1}^{−} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2}))

where the union is again disjoint. The validity of this equality for S_{0}^{+} (ϑ) can be deduced from the following equalities for the three sets. It can be shown that

W^{+} (ϑ) ∩ S_{0}^{+} (ϑ) = \{( \begin{matrix}x \\ x + y \\ y tan β \end{matrix} ) : x > 0, y > 0, −ϑ < β ≤ ϑ\}

∪ \{( \begin{matrix}x \\ y \\ y tan β \end{matrix} ) : x ≤ 0, y > 0, −ϑ < β ≤ ϑ\},

g_{0} (π + 2ϑ) · (S_{1}^{−} (ϑ) ∩ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2}))

= \{( \begin{matrix}x \\ hx + y \\ y tan ϑ \end{matrix} ) : x > 0, 0 ≤ h < 1, y > 0\}

and

g_{0} (π − 2ϑ) · (S_{1}^{−} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2}))

= \{( \begin{matrix}x \\ hx + y \\ −y tan ϑ \end{matrix} ) : x > 0, 0 < h ≤ 1, y > 0\}.

An analogous statement holds for S_{0}^{−} (ϑ).

Now W (ϑ) will be decomposed accordingly and put together again from rotated pieces to S_{0} (ϑ). With the above

∥E (W (ϑ)) ξ∥^{2} = ∥E (W^{+} (ϑ) ∩ S_{0}^{+} (ϑ)) ξ∥^{2}

+ ∥E (W^{+} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2})) ξ∥^{2}

+ ∥E (W^{+} (ϑ) ∩ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2})) ξ∥^{2}

+ ∥E (W^{−} (ϑ) ∩ S_{0}^{−} (ϑ)) ξ∥^{2}

+ ∥E (W^{−} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2})) ξ∥^{2}

+ ∥E (W^{−} (ϑ) ∩ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2})) ξ∥^{2}

150 Neuhauser

and by the K -invariance

∥E (W^{+} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2})) ξ∥^{2}
= ∥E (g_{0} (π − 2ϑ) · (S_{1}^{+} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2}))) ξ∥^{2}
= ∥E (g_{0} (π − 2ϑ) · S_{1}^{+} (ϑ) ∩ g_{0} (−\frac{ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2})) ξ∥^{2}

and analogously

∥E (W^{+} (ϑ) ∩ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2})) ξ∥^{2}
= ∥E (g_{0} (π + 2ϑ) · S_{1}^{+} (ϑ) ∩ g_{0} (\frac{ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2})) ξ∥^{2} ,
∥E (W^{−} (ϑ) ∩ g_{0} (\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2})) ξ∥^{2}
= ∥E (g_{0} (π − 2ϑ) · S_{1}^{−} (ϑ) ∩ g_{0} (−\frac{ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2})) ξ∥^{2} ,
∥E (W^{−} (ϑ) ∩ g_{0} (−\frac{3ϑ}{2}) · S_{0}^{−} (\frac{ϑ}{2})) ξ∥^{2}
= ∥E (g_{0} (π + 2ϑ) · S_{1}^{−} (ϑ) ∩ g_{0} (\frac{ϑ}{2}) · S_{0}^{+} (\frac{ϑ}{2})) ξ∥^{2} .

This yields

∥E (W (ϑ)) ξ∥^{2} = ∥E (S_{0}^{+} (ϑ)) ξ∥^{2} + ∥E (S_{0}^{−} (ϑ)) ξ∥^{2}
= 2 ∥E (S_{0}^{+} (ϑ)) ξ∥^{2} = \frac{2ϑ}{π}.

A more detailed proof can be found in [11, page 59–68].

B Proof of Proposition 3.2

It is enough to prove that

W^{±} (ϑ) ⊇ g_{1} (−t) · W^{±} (arctan (e^{t} tan ϑ))

where either both signs are + or both −. Therefore it has to be shown that S_{1}^{±} (ϑ) ∩ S_{0}^{±} (2ϑ) ⊇ S_{1}^{±} (ϑ) ∩ S_{− tanh t}^{±} (arctan (cosh t tan (2 arctan (e^{t} tan ϑ)))). So let

( \begin{matrix}x \\ x + y \\ y tan β \end{matrix} ) ∈ S_{1}^{+} (ϑ) ∩ S_{− tanh t}^{+} (arctan (cosh t tan (2 arctan (e^{t} tan ϑ))))

Neuhauser 151

with x ∈ R, y > 0 and −ϑ < β ≤ ϑ. Then there is z > 0 and α with

− arctan (cosh t tan (2 arctan (e^{t} tan ϑ)))

< α ≤ arctan (cosh t tan (2 arctan (e^{t} tan ϑ)))

such that x + y = −x tanh t + z and y tan β = z tan α. If x ≥ 0, then x + y > 0, since y > 0. Hence 0 < \frac{y tan β}{x+y} ≤ tan β ≤ tan ϑ for 0 < β ≤ ϑ and 0 ≥ \frac{y tan β}{x+y} ≥ tan β > − tan ϑ for −ϑ < β ≤ 0. So ( \begin{matrix}x \\ x + y \\ y tan β \end{matrix} ) ∈ S_{0}^{+} (2ϑ). If x < 0, then x + y = −x tanh t + z > 0. If 0 < β ≤ ϑ and y ≥ −2x (cos ϑ)^{2}, then

0 < \frac{y tan β}{x + y} = (1 + \frac{−x}{x + y} ) tan β
≤ (1 + \frac{1}{−1 + 2 (cos ϑ)^{2}} ) tan β = \frac{2 (cos ϑ)^{2}}{cos (2ϑ)} tan β
≤ \frac{2 (cos ϑ)^{2}}{cos (2ϑ)} tan ϑ = tan (2ϑ) .

If −ϑ < β ≤ 0, holds analogously

0 ≥ \frac{y tan β}{x + y} = (1 + \frac{−x}{x + y} ) tan β
≥ (1 + \frac{1}{−1 + 2 (cos ϑ)^{2}} ) tan β = \frac{2 (cos ϑ)^{2}}{cos (2ϑ)} tan β
> −\frac{2 (cos ϑ)^{2}}{cos (2ϑ)} tan ϑ = − tan (2ϑ) .

For z ≤ −x\frac{tanh t}{cosh t tan(2 arctan(e^{t} tan ϑ))−tan(2ϑ)} tan (2ϑ) and

0 < α ≤ arctan (cosh t tan (2 arctan (e^{t} tan ϑ)))

holds

0 < \frac{z tan α}{−x tanh t + z} = (1 − \frac{−x tanh t}{−x tanh t + z} ) tan α
≤ (1 − \frac{1}{1 + \frac{1}{cosh t tan(2 arctan(e^{t} tan ϑ))−tan(2ϑ)} tan (2ϑ)}) tan α
= \frac{tan (2ϑ)}{cosh t tan (2 arctan (e^{t} tan ϑ))} tan α ≤ tan (2ϑ) .

For − arctan (cosh t tan (2 arctan (e^{t} tan ϑ))) < α ≤ 0 analogously

0 ≥ \frac{z tan α}{−x tanh t + z} = (1 − \frac{−x tanh t}{−x tanh t + z} ) tan α
≥ (1 − \frac{1}{1 + \frac{1}{cosh t tan(2 arctan(e^{t} tan ϑ))−tan(2ϑ)} tan (2ϑ)}) tan α
= \frac{tan (2ϑ)}{cosh t tan (2 arctan (e^{t} tan ϑ))} tan α > − tan (2ϑ) .

152 Neuhauser But,

(cosh t) tan (2 arctan (e^{t} tan ϑ)) − tan (2ϑ)
= (cosh t) \frac{2e^{t} tan ϑ}{1 − (e^{t} tan ϑ)^{2}} − tan (2ϑ)
= (cosh t) \frac{e^{t} tan (2ϑ)}{1 − (e^{t} tan ϑ)^{2}} (1 − (tan ϑ)^{2}) − tan (2ϑ)
= ((cosh t) \frac{e^{t}}{1 − (e^{t} tan ϑ)^{2}} (1 − (tan ϑ)^{2}) − 1) tan (2ϑ)

and hence

\frac{tanh t}{cosh t tan (2 arctan (e^{t} tan ϑ)) − tan (2ϑ)} tan (2ϑ)
= \frac{tanh t}{(cosh t) \frac{e^{t}}{1−(e^{t} tan ϑ)^{2}} (1 − (tan ϑ)^{2}) − 1}
= \frac{(1 − (e^{t} tan ϑ)^{2}) tanh t}{(cosh t) e^{t} (1 − (tan ϑ)^{2}) − 1 + (e^{t} tan ϑ)^{2}} .

Now

(cosh t) e^{t} (1 − (tan ϑ)^{2}) − 1 + (e^{t} tan ϑ)^{2}
= (cosh t) e^{t} − 1 + e^{t} (tan ϑ)^{2} (− (cosh t) + e^{t})
= \frac{e^{2t} − 1}{2} + e^{t} (tan ϑ)^{2} \frac{e^{t} − e^{−t}}{2}
= e^{t} sinh t + e^{t} (tan ϑ)^{2} sinh t = e^{t} \frac{sinh t}{cos^{2} ϑ}.

This implies

\frac{tanh t}{cosh t tan (2 arctan (e^{t} tan ϑ)) − tan (2ϑ)} tan (2ϑ)
= \frac{(1 − (e^{t} tan ϑ)^{2}) tanh t}{e^{t} \frac{sinh t}{cos^{2} ϑ}} = \frac{(1 − (e^{t} tan ϑ)^{2}) cos^{2} ϑ}{e^{t} cosh t}
= \frac{e^{−t} cos^{2} ϑ − e^{t} sin^{2} ϑ}{cosh t}.

Since y ≥ −2x (cos ϑ)^{2}, one has x + y ≥ −x (−1 + 2 (cos ϑ)^{2}) = −x cos (2ϑ) and

z ≤ −x\frac{tanh t}{cosh t tan (2 arctan (e^{t} tan ϑ)) − tan (2ϑ)} tan (2ϑ)
= −x\frac{e^{−t} cos^{2} ϑ − e^{t} sin^{2} ϑ}{cosh t},

so

−x tanh t + z ≤ −x (tanh t + \frac{e^{−t} cos^{2} ϑ − e^{t} sin^{2} ϑ}{cosh t}).

Also

tanh t + \frac{e^{−t} cos^{2} ϑ − e^{t} sin^{2} ϑ}{cosh t}
= \frac{sinh t + e^{−t} cos^{2} ϑ − e^{t} sin^{2} ϑ}{cosh t}
= \frac{(1 − 2 cos^{2} ϑ) sinh t + e^{t} cos^{2} ϑ − e^{t} sin^{2} ϑ}{cosh t}
= \frac{− cos (2ϑ) sinh t + e^{t} cos (2ϑ)}{cosh t} = \frac{cos (2ϑ) (− sinh t + e^{t})}{cosh t}
= cos (2ϑ) .

So −x tanh t + z ≤ −x cos (2ϑ). Hence ( \begin{matrix}x \\ x + y \\ y tan β \end{matrix} ) = ( \begin{matrix}x \\ −x tanh t + z \\ z tan α \end{matrix} ) ∈

S_{0}^{+} (2ϑ).

The inclusion g_{1} (t) · W^{−} (ϑ) ⊇ W^{−} (arctan (e^{t} tan ϑ)) holds analogously. Therefore g_{1} (t) · W (ϑ) ⊇ W (arctan (e^{t} tan ϑ)).

References

[1] Bekka, M. E. B., P.-A. Cherix, and P. Jolissaint, Kazhdan constants associated with a Laplacian on connected Lie groups, Journal of Lie Theory 8 (1998), 95–110.

[2] Bekka, M. E. B., and M. Mayer, On Kazhdan’s property (T) and Kazhdan constants associated to a Laplacian for SL(3,R), Journal of Lie Theory 10 (2000), 93–105.

[3] Bekka, M. B., and M. Neuhauser, On Kazhdan’s property (T) for Sp_{2}(k), Journal of Lie Theory 12 (2002), 31–39.

[4] Delaroche, C., and A. Kirillov, Sur les relations entre l’espace dual d’un groupe et la structure de ses sous-groupes fermés (d’aprés D. A. Kajdan), in “Séminaire Bourbaki, Vol. 10”, Exp. No. 343, 507–528, Soc. Math. France, Paris, 1995.

[5] de la Harpe, P., and A. Valette, “La propriété (T) de Kazhdan pour les groupes localement compacts”, ” Volume 175 of Astérisque, Soc. Math. France, 1989.

[6] Howe, R. E., and C. C. Moore, Asymptotic properties of unitary representations, Journal of Functional Analysis 32 (1979), 72–96.

[7] Howe, R., and E. C. Tan, “Non-Abelian harmonic analysis,” Universitext, Springer-Verlag, 1992.

[8]  Jacobson, N., “Basic algebra I,” Freeman, 1985.

[9] Kazhdan, D. A., Connection of the dual space of a group with the structure of its closed subgroups, Funct. Anal. Appl. 1 (1967), 63–65.

[10] Margulis, G. A., “Discrete subgroups of semisimple Lie groups, ” Volume 17 of Ergebnisse der Mathematik und ihrer Grenzgebiete 3. Folge, Springer-Verlag, 1991.

[11] Neuhauser, M., ,,Neue Methoden zur Bestimmung expliziter Schranken von Kazhdan-Konstanten“, Dissertation, Technische Universität München, 2001 [tumb1.biblio.tu-muenchen.de/publ/diss/ma/2001/neuhauser.html].

[12] Valette, A., Old and new about Kazhdan’s property (T), in: Baldoni, V. and M. A. Picardello, Editors, “Representations of Lie groups and quantum groups,” Volume 311 of Pitman Research Notes in Mathematics Series, 271–333. Longman Scientific & Technical, 1994.

[13] Vasershtein, L. N., On groups possessing property T , Functional Anal. Appl. 2 (1968), 86.

[14] Warner, G., “Harmonic analysis on semi-simple Lie groups I”, Volume 188 of die Grundlehren der mathematischen Wissenschaften in Einzeldarstellungen mit besonderer Berücksichtigung der Anwendungsgebiete, Springer-Verlag, 1972.

M. Neuhauser Technische Universität München Zentrum Mathematik Arcisstr. 21 D-80290 München Germany neuhausm@mathematik.tumuenchen.de Current address: Institut de Mathématiques Université de Neuchâtel Rue Emile Argand 11 CH-2007 Neuchâtel Suisse (Switzerland)

Received October 4, 2001 and in final form January 9, 2002