Journal of Lie Theory Volume 12 (2002) 81–112 \begin{matrix}c \\ ◯ \end{matrix} 2002 Heldermann Verlag

Mixed Models for Reductive Dual Pairs and Siegel Domains for Hermitian Symmetric Spaces

C. S. Leslie^{1}

Communicated by M. Moskowitz

Abstract. Let (G, G′) be the reductive dual pair (Sp(n, ℝ), O(k)) or (U(p, q), U(k)), and let K be a maximal compact subgroup of the noncompact group G. Then for the representations π of G̃ which occur in the Howe duality correspondence for (G, G′), we construct explicit intertwining maps between mixed models of π and spaces of holomorphic sections of vector bundles over the hermitian symmetric space G/K, where G/K is embedded in its holomorphic tangent space as a type III Siegel domain. This result provides a link between the original construction of these representations using tube domain and type II Siegel domain realizations of G/K and more recent constructions using the bounded domain realization of G/K.

1. Introduction

Let (G, G′) be the reductive dual pair (Sp(n, ℝ), O(k)) or (U(p, q), U(k)). Then the noncompact group G is of hermitian type, meaning that G/K is a hermitian symmetric space for K maximal compact in G, and the representations π of G̃ (a suitable covering group for G) that occur in the Howe correspondence are all unitarizable highest weight representations. A natural way to try to geometrically realize these representations of G̃ is to find appropriate invariant subspaces of sections of vector bundles over G/K. In this paper, we obtain such a realization by constructing a set of new and explicit intertwining maps between mixed models [11] of these representations and spaces of holomorphic functions on unbounded realizations of G/K known as type III Siegel domains.

Our results rely on a method due to Davidson and Fabec [3] that produces an intertwining map between an abstract unitary highest weight representation for a hermitian linear group G and a multiplier representation on a function space on the bounded domain realization D of the hermitian symmetric space G/K. In particular, their method produces a G-intertwining map between Fock models for the representations we consider and reproducing kernel spaces of holomorphic

ISSN 0949–5932 / $2.50 \begin{matrix}c \\ ◯ \end{matrix} Heldermann Verlag vector-valued functions on D. By using the geometric framework of Siegel domains and the Davidson-Fabec construction, we obtain a new set of intertwining maps corresponding to mixed models of the oscillator representation. First, we fix partial Cayley transforms c_{b}, 1 ≤ b ≤ r, where r is the real rank of G, and consider the image S_{b} under c_{b} of the bounded domain D via the Harish-Chandra embedding of G/K in its holomorphic tangent space. For b ≠ r, these domains are type III Siegel domains in the sense of Piatetski-Shapiro [17]. Second, we define partial Bargmann transforms B_{b} which intertwine mixed models and Fock models of ω. We show how the action of the inverse partial Bargmann transform on the space of G′-invariant polynomials and the space of G′-harmonic polynomials can be reinterpreted in terms of the geometric action of the partial Cayley transform. As a result, in Theorem 8.8 and Theorem 13.5, we obtain explicit intertwining maps between mixed models of highest weight representations and multiplier representations of holomorphic functions on S_{b}. For the case b = r, we retrieve the intertwining maps found in Kashiwara-Vergne [14]. The structure of the different Siegel domain realizations S_{b} is closely connected to the boundary component theory of the bounded domain D [22]. Thus we obtain a new connection between the polarization of models of the oscillator representation and the boundary component theory of D, which could be of interest in the study of automorphic forms.

1Research partially supported by an NSERC Post-Graduate Fellowship and an NSERC Post-Doctoral Fellowship. The author would like to thank the Department of Mathematics at Columbia University for their hospitality while this paper was being written.

In Sections 2 and 3 we outline a (straightforward) generalization of the Davidson-Fabec construction [3] for arbitrary covers of linear hermitian groups. Sections 4 and 5 describe the dual pair (Sp(n, ℝ), O(k)) and the Fock model for the representations of S̃p(n, ℝ) which occur, while in Section 6 to Section 8, we describe the domains S_{b} and partial Bargmann transforms, and we construct our new S̃p(n, ℝ)-intertwining maps for mixed models. Background on the Fock model for the second dual pair (U(p, q), U(k)) is given in Sections 9 and 10, while in Sections 11 to 13 we develop our intertwining maps for mixed models of the U(p, q)-representations in the Howe correspondence.

This work was done as part of my doctoral dissertation under the supervision of Prof. Joseph A. Wolf. I would like to thank Prof. Wolf for his guidance and support.

2. Lifting Factors of Automorphy

As we will be working with non-linear Lie groups, we need to define various notions in the geometry of hermitian symmetric domains for arbitrary covers of linear groups. In particular, we need to describe how to lift the factor of automorphy for the linear group to a covering group. These ideas are well-known and can be found in a number of references [20, 10, 21]. For convenience, we follow our exposition in [15].

Let 𝔤 be a simple Lie algebra over ℝ with Cartan decomposition 𝔤 = 𝔨⊕𝔭. We assume that [𝔨, 𝔨] ≠ 𝔨. Then 𝔨 has one dimensional center 𝔷(𝔨), and we can find an element H ∈ 𝔷(𝔨) such that ad(H) gives a complex structure on 𝔭. This gives the decomposition 𝔤_{ℂ} = 𝔭_{+} ⊕ 𝔨_{ℂ} ⊕ 𝔭_{−}, where 𝔭_{±} are the ±i eigenspaces of ad(H) on 𝔭_{ℂ} and \overline{𝔭}_{+} = 𝔭_{−} under conjugation with respect to the real form 𝔤 of 𝔤_{ℂ}.

Let G_{ℂ} be the connected, simply connected group with Lie algebra 𝔤_{ℂ}.

Then G_{ℂ} is a complex simple Lie group. Let K_{ℂ} be the connected subgroup of G_{ℂ} corresponding to 𝔨_{ℂ}. Let P_{+} = exp(𝔭_{+}), P_{−} = exp(𝔭_{+}). Then the map 𝔭_{+} × K_{ℂ} × 𝔭_{−} → G_{ℂ} given by (Z, k, W) ↦ exp(Z) k exp(W) is a holomorphic diffeomorphism onto a dense open subset Ω = P_{+}K_{ℂ}P_{−} of G_{ℂ}. We can write g ∈ Ω uniquely as g = (g)_{+}k(g)(g)_{−}, (g)_{±} ∈ P_{±}, k(g) ∈ K_{ℂ}. Then g ↦ k(g) is a holomorphic map Ω → K_{ℂ}.

Let G ⊂ G_{ℂ} be the connected subgroup of G_{ℂ} corresponding to 𝔤 ⊂ 𝔤_{ℂ}. Then G is a linear real Lie group, G ⊂ P_{+}K_{ℂ}P_{−} and G ∩ K_{ℂ}P_{−} = K, where K is the connected subgroup of G corresponding to 𝔨. Finally, G/K is a noncompact hermitian symmetric space, and we say G (or 𝔤) is of hermitian type.

The map ζ : Ω → 𝔭_{+} given by exp(ζ(g)) = (g)_{+} induces a holomorphic diffeomorphism of G/K onto the domain D = ζ(G) ⊂ 𝔭_{+}. The set D is a bounded domain in 𝔭_{+}, and the map G/K → D ⊂ 𝔭_{+} is known as the Harish-Chandra embedding.

For (g, Z) ∈ G_{ℂ} × 𝔭_{+} such that g exp(Z) ∈ Ω, we define g(Z) ∈ 𝔭_{+}, j(g, Z) ∈ K_{ℂ} by the formulas

exp(g(Z)) = (g exp(Z))_{+},
j(g, Z) = k(g exp(Z)).

Now consider a covering map ν : G̃ → G of Lie groups, with K̃ the maximal compact subgroup of G̃ such that ν(K̃) = K. We can define an action of G̃ on D by g̃ · Z = ν(g̃)(Z). Thus D can be identified with G̃/K̃.

The maximal compact subgroup K̃ ⊂ G̃ is a linear group and therefore has a complexification K̃_{ℂ} corresponding to 𝔨_{ℂ}. We can extend the covering map ν : K̃ → K to a covering map \overline{ν} : K̃_{ℂ} → K_{ℂ}. When we have Z ∈ 𝔭_{+} for which j(g, Z) is defined for all g ∈ G, we can define a map which we will also call j by (g̃, Z) ↦ j(ν(g̃), Z) for g̃ ∈ G̃. We need to lift this map to K̃_{ℂ}.

The following statement comes from [15, Proposition 1.1.1]:

Proposition 2.1. Let U be any connected open domain in 𝔭_{+} on which the function j : G̃ × U → K_{ℂ} is defined. Fix a basepoint U_{0} ∈ U. Then there exists a unique holomorphic map j̃: G̃×U → K̃_{ℂ} such that j̃(1G̃, U_{0}) = 1K̃_{ℂ} and \overline{ν} ∘j̃ = j.

In particular, we have a unique holomorphic lift:

j̃: G̃ × D → K̃_{ℂ},  j̃(1G̃, 0) = 1K̃_{ℂ}

Corollary 2.2. The map j̃: G̃ × D → K̃_{ℂ} satisfies the cocyle relation

j̃(g̃_{1}g̃_{2}, Z) = j̃(g̃_{1}, g̃_{2} · Z)j̃(g̃_{2}, Z), g̃_{1}, g̃_{2} ∈ G̃, Z ∈ D  (1)

and the property

j̃(k̃, Z) = k̃.  (2)

Finally, the canonical kernel function q : D × D → K_{ℂ} is defined by

q(Z, W) = j(exp(\overline{W})^{−1}, Z)^{−1}

and satisfies

q(0, Z) = q(Z, 0) = 1, q(W, Z) = \overline{q(Z, W)}^{−1},
q(g(Z), g(W)) = j(g, Z)q(Z, W)\overline{j(g, W)}^{−1}.

Since D is simply connected, we obtain a unique holomorphic lift

q̃: D × D → K̃_{ℂ},  q̃(0, 0) = 1K̃_{ℂ}

such that

q̃(g(Z), g(W)) = j̃(g, Z)q̃(Z, W)\overline{j̃(g, W)}^{−1}.  (3)

3. The Davidson-Fabec Construction

Let G be a linear group of Hermitian type and ω an irreducible unitary highest weight representation of G on a Hilbert space ℍ. In [3], Davidson and Fabec give a geometric construction of an intertwining map between (ω, ℍ) and a space of holomorphic sections of a homogeneous vector bundle over G/K with G acting by translation. The latter space is realized as a space of holomorphic vector-valued functions on the bounded domain D.

We will need to extend the result in [3] to arbitrary covers ν : G̃ → G of the linear group G. We will outline the original argument, making the necessary reformulations to handle our more general setting.

Let ω be a nontrivial irreducible highest weight representation of G̃ on a Hilbert space ℍ with inner product ⟨·|·⟩. Let H ⊂ ℍ be the K̃ -span of the highest weight vector. Then H is an irreducible K̃ -space, and we let τ denote the restriction of ω to V_{τ} = H. Since [𝔨_{ℂ}, 𝔭_{±}] ⊂ 𝔭_{±}, we have T · v = 0 for all T ∈ 𝔭_{+} and v ∈ H.

Extend τ to a holomorphic representation of K̃_{ℂ}, the complexification of K̃. Then we define:

J : G̃ × D → Aut(V_{τ}), J(g̃, T) = τ(j̃(g̃, T)).

Let T ∈ 𝔭_{+}. Define q_{T} : H → ℍ formally by q_{T} v = ∑_{n=0}^{∞} ^{\frac{(\overline{T})^{n}v}{n!}} for v ∈ H. By [3, Theorem 5.1] we have:

Theorem 3.1. Let v be a nonzero vector in H and T ∈ 𝔭_{+}. Then the series which defines q_{T} v converges if and only if T ∈ D. The main result is a reformulation of [3, Theorem 6.1]:

Theorem 3.2. Let v ∈ H and g̃ ∈ G̃. Then ω(g̃)v = q_{ν(g̃)(0)}J(g̃, 0)^{∗}^{−1}v. Theorem 3.2 leads easily to the following Proposition by the proof of [3, Proposition 7.1].

Proposition 3.3. Let g̃ ∈ G̃, T ∈ D, and v ∈ H. Then

ω(g̃)q_{T} v = q_{ν(g̃)(T)}J(g̃, T)^{∗}^{−1}v.

Now the previous Proposition and the irreducibility of ω give:

Proposition 3.4. The span of the vectors q_{T} v, T ∈ D, v ∈ H, is dense in the Hilbert space ℍ. Thus it will be sufficient to determine the intertwining operator on the set of vectors q_{T} v. We first define a kernel function on D:

Theorem 3.5. There is a positive definite operator-valued kernel function Q : D × D → Aut(V_{τ}) such that

⟨q_{T} v|q_{S}w⟩ = ⟨Q(S, T)v|w⟩.  (4)

Following [3], as Q is a positive definite operator-valued kernel function, we use a result of Kunze [13] to construct a Hilbert space H(D, τ) of continuous functions f : D → H such that:

(a) The span of {S ↦ Q(S, T)v : T ∈ D, v ∈ H} is dense in H(D, τ);
(b) For S ∈ D, the map e_{S} : f → f(S) is a continuous map H(D, τ) → H;
(c) (S, T) = e_{S}e_{T}^{∗} for all S, T ∈ D;
(d) The inner product on H(D, τ) is given by
(Q(·, T)v|Q(·, S)w) = ⟨Q(S, T)v|w⟩_{V_{τ}} .

Using property (3) of q̃ and the fact that τ(\overline{k̃}) = τ(k̃)^{∗}^{−1} for k̃ ∈ K̃_{ℂ}, we establish:

Proposition 3.6. Let S, T ∈ D, g̃ ∈ G̃. Then

Q(g̃ · S, g̃ · T) = J(g̃, S)Q(S, T)J(g̃, T)^{∗}^{−1}

It follows, using arguments of Kunze [13] that H(D, τ) is a representation space for G̃:

Proposition 3.7. The formula T (g̃)f(S) = J(g̃^{−1}, S)^{−1}f(g̃^{−1} · S) where f ∈ H(D, τ), defines a strongly continuous unitary representation of G̃ on H(D, τ). Finally, we can write down the intertwining map as in [3]:

Theorem 3.8. The map q_{T} v ↦ Q(·, T)v extends to an intertwining operator Ξ between (ω, ℍ) and (T, H(D, τ)). The intertwining map can be defined globally by Ξf(S) = q_{S}^{∗}f for f ∈ ℍ.

4. The Fock Model for Sp(n, ℝ)

We let G_{ℂ} = Sp(n, ℂ) be the group of automorphisms of ℂ^{2n} preserving a nondegenerate antisymmetric bilinear form J. Fix a basis with respect to which

J = (\begin{matrix}0 & −1_{n} \\ 1_{n} & 0 \end{matrix}), and define an indefinite hermitian form on ℂ^{2n} by ⟨u, v⟩ =

− ∑_{j=1}^{n} u_{j}\overline{v}_{j} + ∑_{j=1}^{n} u_{n+j}\overline{v}_{n+j} with respect to the same basis. We have a real form G of Sp(n, ℂ), isomorphic to Sp(n, ℝ), defined by

G = Sp(n, ℂ) ∩ U(n, n) = \{(\begin{matrix}A & B \\ \overline{B} & \overline{A} \end{matrix}) : AA^{∗} − BB^{∗} = 1_{n}, ^{t}AB = ^{t}BA\}

with maximal compact subgroup K ≅ U(n) = \{(\begin{matrix}A & 0 \\ 0 & \overline{A} \end{matrix}) : A ∈ U(n)\} and

Harish-Chandra decomposition 𝔤_{ℂ} = 𝔭_{+} + 𝔨_{ℂ} + 𝔭_{−}, where

𝔭_{+} = \{(\begin{matrix}0 & Z \\ 0 & 0 \end{matrix}), Z ∈ Sym_{n}(ℂ)\}
𝔨_{ℂ} = \{(\begin{matrix}X & 0 \\ 0 & −^{t}X \end{matrix}) : X ∈ 𝔤𝔩(n, ℂ)\}
𝔭_{−} = \{(\begin{matrix}0 & 0 \\ Y & 0 \end{matrix}), Y ∈ Sym_{n}(ℂ)\} .

Let P_{±} = exp(𝔭_{±}). Then g = (\begin{matrix}A & B \\ C & D \end{matrix}) belongs to the dense open subset

Ω = P_{+}K_{ℂ}P_{−} ⊂ G_{ℂ} if and only if D is nonsingular [19, Lemma 7.3], and for g ∈ Ω we have a unique decomposition g = (g)_{+}k(g)(g)_{−} given by

(\begin{matrix}A & B \\ C & D \end{matrix}) = (\begin{matrix}1 & BD^{−1} \\ 0 & 1 \end{matrix}) (\begin{matrix}^{t}D^{−1} & 0 \\ 0 & D \end{matrix}) (\begin{matrix}1 & 0 \\ D^{−1}C & 0 \end{matrix}).

We identify K_{ℂ} with 𝔤𝔩(n, ℂ) via (\begin{matrix}A & 0 \\ 0 & ^{t}A^{−1} \end{matrix}) ↔ A. Then for (g, Z) ∈

G_{ℂ} × 𝔭_{+} such that g · exp(Z) ∈ Ω, the automorphy factor is given by

j(g, Z) = k(g · exp(Z)) = ^{t}(CZ + D)^{−1}

and the action of g on 𝔭_{+} defined by exp(g(Z)) = (g · exp(Z))_{+} is given by

g(Z) = (AZ + B)(CZ + D)^{−1}.

The image of the Harish-Chandra embedding G/K → D ⊂ 𝔭_{+} is given by the generalized Siegel disk

D = {n-planes p_{Z} : ⟨·, ·⟩|_{p_{Z}} ≫ 0} = {Z ∈ Sym_{n}(ℂ) : 1_{n} − Z^{∗}Z ≫ 0}

where we use the notation that W ≫ 0 if W is positive definite.

Fix k ≥ 1 and let M = M_{n,k}(ℂ) with inner product (w|w′) = Tr(ww′^{∗}). We realize the Heisenberg group H = M × ℝ with group law

(w; t)(w′; t′) = (w + w′; t + t′ + Im(w|w′)).

Let F be the Hilbert space of holomorphic functions on M which are squareintegrable with respect to the Gaussian measure dμ(w) = e^{−π(w|w)}dw. Note that F has a reproducing kernel K given by K(z, w) = e^{π(z|w)}, so that we have

f(z) = ∫_{M} e^{π(z|w)}f(w)e^{−π(w|w)}dw

for f ∈ F and z ∈ M.

As is well known, we obtain an irreducible unitary representation ρ_{0} of H on F with central character χ(t) = e^{−πit} by

ρ_{0}(w; t)f(w′) = e^{−πit}e^{π(w′|w)}e^{− \frac{π}{2} (w|w)}f(w′ − w).

We define an action of G on M by g · w = Aw − iB\overline{w} for g = (\begin{matrix}A & B \\ \overline{B} & \overline{A} \end{matrix}) and

w ∈ M. If we identify w with the 2n × k matrix (\begin{matrix}w \\ −i\overline{w} \end{matrix}), then the action g · w

simply becomes matrix multiplication.

Via the action w ↦ g · w, we can identify G as a group of real-linear automorphisms of M that preserve the real symplectic form Im(· | ·). It follows that G acts as a group of automorphisms of H preserving the center 0 × ℝ of H, and g ∈ G leads to a unitary representation ρ_{0}^{g}(w; t) = ρ_{0}(g · w, t) with the same central character χ. Thus by the Stone-von Neumann theorem, ρ_{0}^{g} and ρ_{0} are unitarily equivalent, and we obtain a projective representation ω : G → U(F)/U(1) defined by

ω(g)ρ_{0}(w; t) = ρ_{0}(g · w; t)ω(g).  (5)

By straightforward calculations as in [6, Proposition 4.31], we obtain the following description of the action of G.

Theorem 4.1. For g = (\begin{matrix}A & B \\ \overline{B} & \overline{A} \end{matrix}) ∈ G, define the operator \overline{ω}(g) modulo ±1

on F by

\overline{ω}(g)f(z) = ∫_{M} K_{g}(z, w)f(w)e^{−π(w|w)}dw,  (6)

K_{g}(z, w) = (det^{− \frac{k}{2}} A)e^{\frac{iπ}{2} Tr(^{t}z\overline{B}A^{−1}z)}e^{π Tr(^{t}\overline{w}A^{−1}z)}e^{\frac{iπ}{2} Tr(^{t}\overline{w}A^{−1}B\overline{w})}

where the sign of det^{− \frac{k}{2}} A is left undetermined. Then \overline{ω}(g) is a unitary operator modulo ± satisfying (5) and moreover \overline{ω}(g_{1})\overline{ω}(g_{2}) = ±\overline{ω}(g_{1}g_{2}). For T ∈ 𝔭_{+} we can define an entire function on M by

q_{T} (z) = e^{\frac{iπ}{2} (z|T \overline{z})} = e^{\frac{iπ}{2} Tr(z^{t}zT^{∗})} = e^{\frac{iπ}{2} Tr(^{t}z\overline{T} z)}.

Now, by a generalization of of the proof for the case k = 1 given in [6, Proposition 4.69], we obtain:

Lemma 4.2. The function q_{T} lies in F if and only if T ∈ D. We use the q_{T} notation and also our expression for the reproducing kernel K on F to obtain:

Lemma 4.3. The operator \overline{ω}(g) given by (6) can be expressed as

\overline{ω}(g)f(z) = (det^{− \frac{k}{2}} A)q_{B\overline{A}^{−1}}(z) ∫_{M} \overline{q}_{−A^{−1}B}(w)K(A^{−1}z, w)f(w)e^{−π(w|w)}dw. (7)

We wish to construct from \overline{ω} a representation of the the double cover G̃ of G. We use the following description of G̃. Consider the function d : G × D → ℂ − {0} given by d(g, Z) = det(j(g, Z)). Since D is simply connected, a branch of the square root function is uniquely determined by its value at a single point.

Given g ∈ G, we identify the choice of branch of d^{\frac{1}{2}} (g, Z) by its value at Z = 0, and we obtain a pair (g, d^{\frac{1}{2}} (g, Z)) ↔ (g, t) where t^{2} = d(g, 0) = det(k(g)). We define G̃ as the set \{(g, d^{\frac{1}{2}} (g, Z))\} with multiplication

(g_{1}, d1^{\frac{1}{2}} (g_{1}, Z))(g_{2}, d2^{\frac{1}{2}} (g_{2}, Z)) = (g_{1}g_{2}, d1^{\frac{1}{2}} (g_{1}, g_{2}(Z))d2^{\frac{1}{2}} (g_{2}, Z)).  (8)

By the cocycle relation d(g_{1}g_{2}, Z) = d(g_{1}, g_{2}(Z))d(g_{2}, Z), we see that the group law is well-defined. The identity element is ˜1 = (1_{G}, 1), that is, we take the root of d(1_{G}, Z) = det(1_{K_{ℂ}}) = 1 whose value is identically 1.

We now define a lift G̃ → U(F) of the map \overline{ω} : G → U(F)/ℤ_{2} given by

(6). We will call this lift ω. For (g, t) ∈ G̃, g = (\begin{matrix}A & B \\ \overline{B} & \overline{A} \end{matrix}) we define an operator

in U(F) by

ω(g, t)f(z) = (\overline{t}^{−1})^{−k}q_{B\overline{A}^{−1}}(z) ∫_{M} \overline{q}_{−A^{−1}B}(w)K(A^{−1}z, w)f(w)e^{−π(w|w)}dw. (9)

Note that for (g, t) ∈ G̃, t^{2} = det(k(g)) = det(^{t}\overline{A}^{−1}), so \overline{t}^{−1} agrees with det^{\frac{1}{2}} A up to sign. More precisely, consider the covering maps ν : G̃ → G and p : U(F) → U(F)/ℤ_{2}. The composition \overline{ω} ∘ ν gives a map G̃ → U(F)/ℤ_{2}. We can see:

Lemma 4.4. The map ω : G̃ → U(F) defines a continuous lift of \overline{ω} ∘ ν with respect to the covering map p : U(F) → U(F)/ℤ_{2}. Now as in [15, Theorem 2.2.5] we have:

Theorem 4.5. The lift ω : G̃ → U(F) defines a unitary representation.

5. The Dual Pair (Sp(n, ℝ), O(k))

Let G′ = O(k) act on F(M) by right multiplication:

c · f(z) = f(zc)

for c ∈ O(k) and f ∈ F. Clearly the action of G′ commutes with ω. In fact, (G, G′) = (Sp(n, ℝ), O(k)) is a dual pair inside Sp(M) = Sp(nk, ℝ), the real symplectic group on the symplectic vector space (M, Im(· | ·)), and ω is the restriction to G̃ of the oscillator representation on S̃p(M).

Following [2, Chapter 7], we define a map

θ : M → 𝔭_{+}, θ(z) = z ·^{t}z.

Let P = P(M) be the subspace of polynomial functions in F. Then P is dense in F. Let I be the space of polynomials invariant under the action of G′. Then I is generated as an algebra by the matrix entries of z → θ(z) and the constant functions [2, Chapter 7]. Let I be the ideal in P generated by the matrix entries of z → θ(z) and let H be the orthogonal complement of I. We call H the space of harmonic polynomials.

We have [2, Chapter 7]:

Proposition 5.1. Let h ∈ P. Then the following are equivalent:

(i) h ∈ H
(ii) g(∂)h = 0 for all matrix entries g of z → θ(z)
(iii) Δ_{i,j}h(z) = 0, 1 ≤ i ≤ j ≤ n where Δ_{i,j} = ∑_{ν=1}^{k} \frac{∂^{2}}{∂z_{iν}∂z_{jν}} .

Let K̃ ⊂ G̃ be the inverse image under ν of K. It is straightforward to check that K̃ is just the linear group {(u, t) ∈ U(n) × ℂ : det(u) = t^{2}}. A simple calculation shows:

Lemma 5.2. Let f ∈ F and (u, t) ∈ K̃. Then

ω(u, t)f(z) = t^{−k}f(u^{−1}z)  (10)

Let { \atop O(k)} denote the unitary dual of O(k). We write (λ, V_{λ}) for an irreducible unitary representation of O(k) on a space V_{λ}, and we let λ′ be the contragredient representation of λ on the dual space V′_{λ}. We now have sufficient background to state the following result of Kashiwara-Vergne [14]:

Theorem 5.3. The space H is K̃ × G′-invariant and decomposes as a multiplicity free orthogonal direct sum of unitary K̃ × G′ representations τ(λ) ⊗ λ′.

For later use, we wish to make the correspondence λ ↦ τ(λ) more concrete. Since G′ = O(k) commutes with the representation ω, we get the decomposition

F(M) = ⊕_{λ∈{ \atop O(k)}}F(M)_{λ}

where F(M)_{λ} denotes the isotypic component of type λ. We write (·, ·) for the canonical bilinear pairing between a vector space V and its dual V′. Now for (λ, V_{λ}) in { \atop O(k)}, we let F(M; λ) be the subspace of functions f from M to V_{λ} such that z ↦ (f(z), ϕ) ∈ F for every ϕ ∈ V′_{λ} and f(zc) = λ(c)^{−1}f(z) for all c ∈ O(k). The group G̃ acts on F(M; λ) by the same formula as ω; we will call this representation ω_{λ}. Then F(M)_{λ} is isomorphic to F(M; λ′) ⊗ V_{λ} by (h ⊗ ψ)(z) = (ψ, h(z)) for h ∈ F(M; λ′), ψ ∈ V_{λ}. Thus we get

F(M) = ⊕_{λ∈{ \atop O(k)}}F(M; λ) ⊗ V′_{λ}

as a representation of G̃ × G′.

Similarly we have the decomposition

H(M) = ⊕_{λ∈{ \atop O(k)}}H(M; λ) ⊗ V′_{λ}

as a representation of K̃ ×G′. Here H(M; λ) ⊂ F(M; λ), and given λ, we identify the representation τ(λ) in the theorem as the irreducible unitary K̃ representation H(M; λ), where K̃ acts as (10). From [2, Proposition 7.12], we obtain:

Proposition 5.4. Let λ ∈ { \atop O(k)} be such that F(M; λ) ≠ 0. Then F(M; λ) is an irreducible G̃ representation.

Fix λ ∈ { \atop O(k)} such that the space H(M; λ) is non-zero. As outlined in Section 3, we will construct a G̃-intertwining map between (ω, F(M; λ)) and sections of the homogeneous vector bundle over D associated to the K̃ -module H(M; λ).

We write τ = τ(λ) for the K̃ -representation given by ω_{λ}|K̃ acting on H(M; λ). We can complexify K̃ as follows. Let

K̃_{ℂ} = {(l, t) ∈ GL(n, ℂ) × ℂ : det(l) = t^{2}} .

Then K̃ is a real form of K̃_{ℂ}, and we can extend τ holomorphically to K̃_{ℂ} by τ(l, t)h(z) = t^{−k}h(l^{−1}z) for (l, t) ∈ K̃_{ℂ}, h ∈ H(M; λ).

Let σ be the real group automorphism of K_{ℂ} whose differential is conjugation with respect to the real form 𝔨 of 𝔨_{ℂ}. If we identify K with U(n) and K_{ℂ} with GL(n, ℂ), then for l ∈ GL(n, ℂ), we have σ(l) = l^{∗}^{−1}, where l^{∗} is the matrix operation of conjugate transpose. We can lift σ to a real group automorphism σ̃ of K̃_{ℂ} fixing K̃ in the obvious way: σ̃(l, t) = (l^{∗}^{−1}, \overline{t}^{−1}). Then τ(σ̃(l, t)) = τ(l, t)^{∗}^{−1}.

The automorphy factor gives a map j : G̃×D → K_{ℂ} defined by j((g, t), Z) = k(g · exp(Z)). For (g, t) ∈ G̃, t corresponds to a choice of branch of the square root of the function d(g, Z) = det(j(g, Z)) via t = d^{\frac{1}{2}} (g, 0). Let t_{Z} = d^{\frac{1}{2}} (g, Z), where we are using the branch of the root function determined by t. Now let j̃((g, t), Z) = (j(g, Z), t_{Z}). Clearly (t_{Z})^{2} = det(j(g, Z)), so we have a well-defined continuous lift of j which satisfies j̃(1_{G}, 0) = 1K̃_{ℂ}.

We write J((g, t), Z) = τ(j̃((g, t), Z)).

Lemma 5.5. Let h ∈ H(M; λ). Then ω_{λ}(g, t)h(z) = q_{g(0)}(z)(J((g, t), 0)^{∗}^{−1}h)(z).

Proof. Recall from (9) that we have for g = (\begin{matrix}A & B \\ \overline{B} & \overline{A} \end{matrix})

ω_{λ}(g, t)h(z) = (\overline{t}^{−1})^{−k}q_{B\overline{A}^{−1}}(z) ∫_{M} \overline{q}_{−A^{−1}B}(w)K(A^{−1}z, w)h(w)e^{−π(w|w)}dw.

Now q_{−A^{−1}B} = 1 + ϕ where ϕ belongs to the closure in F of the subalgebra of I consisting of O(k)-invariants with zero constant coefficient. Thus the function w ↦ ϕ(w)K(w, A^{−1}z) belongs to the closure of the ideal I in F. Since h is harmonic, h is orthogonal to \overline{I}, and we obtain:

ω_{λ}(g, t)h(z) = (\overline{t}^{−1})^{−k}q_{B\overline{A}^{−1}}(z) ∫_{M} K(A^{−1}z, w)h(w)e^{−π(w|w)}dw
= (\overline{t}^{−1})^{−k}q_{B\overline{A}^{−1}}(z)h(A^{−1}z).

Now j̃((g, t), 0) = (j(g, 0), t) = (^{t}\overline{A}^{−1}), J((g, t), 0)^{∗}^{−1} = τ(σ̃(j̃((g, t), 0))) = τ(A, \overline{t}^{−1}), and g(0) = B\overline{A}^{−1}.

Define the operator q_{T} : H → F, T ∈ D, from Section 3 by (q_{T} h)(z) = q_{T} (z)h(z), that is, multiplication by the function q_{T} (z). Then just as in Proposition 3.3, Lemma 5.5 implies:

Corollary 5.6. Let T ∈ D and h ∈ H(M; λ). Then:

ω_{λ}(g, t)q_{T} h = q_{g(T)}J((g, t), T)^{∗}^{−1}h.

Let V_{τ} = H(M; λ), and take v, w ∈ V_{τ}. Exactly as in Section 3, we can construct a positive definite operator-valued kernel function Q : D ×D → Aut(V_{τ}) by: ⟨q_{T} v|q_{S}w⟩ = ⟨Q(S, T)v|w⟩ and a calculation shows that as in Theorem 3.5: Q(S, T) = τ(q̃(S, T)). Q gives the reproducing kernel for a space of V_{τ} -valued holomorphic functions on D, denoted by H(D, τ), on which G̃ acts by translation T_{τ}, where we recall

T_{τ}(g, t)F (T) = J((g, t)^{−1}, T)^{−1}F (g^{−1}(T))

for (g, t) ∈ G̃, T ∈ D, and F ∈ H(D, τ).

Finally, the intertwining map Ξ_{λ} : F(M; λ) → H(D, τ) is given on a dense subset of F(M; λ) by q_{T} v ↦ Q(·, T)v, and is globally defined by Ξ_{λ}f(S) = q_{S}^{∗}f for f ∈ F. In the last statement, q_{S} is regarded as an operator H(M; λ) → F(M; λ) given by multiplication by the function q_{S}.

We want to establish one more form of the intertwining map which will be best suited to our situation:

Proposition 5.7. Let f ∈ F(M; λ) and let w ↦ I_{λ}(w) be the Hom(V_{τ}, V_{λ})- valued polynomial function on M defined by I_{λ}(w)h = h(w) for h ∈ V_{τ} = H(M; λ). Then:

Ξ_{λ}f(T) = ∫_{M} \overline{q_{T} (w)}I_{λ}(w)^{∗}f(w)e^{−π(w|w)}dw.

Proof. The map Ξ_{λ} is defined by ⟨Ξ_{λ}f(T)|h⟩_{H(M;λ)} = ⟨f|q_{T} h⟩_{F(M;λ)} for f ∈ F(M; λ) and h ∈ H(M; λ). Now we calculate:

⟨∫_{M} \overline{q_{T} (w)}I_{λ}(w)^{∗}f(w)e^{−π(w|w)}dw|h⟩_{H(M;λ)}

= ∫_{M} \overline{q_{T} (w)}⟨I_{λ}(w)^{∗}f(w)|h⟩_{H(M;λ)}e^{−π(w|w)}dw
= ∫_{M} \overline{q_{T} (w)}⟨f(w)|h(w)⟩_{V_{λ}}e^{−π(w|w)}dw
= ∫_{M}⟨f(w)|q_{T} (w)h(w)⟩_{V_{λ}}e^{−π(w|w)}dw = ⟨f|q_{T} h⟩_{F(M;λ)}.

6. Cayley Transforms and Unbounded Domains

Let 𝔱_{ℂ} be a compact Cartan subalgebra for 𝔤_{ℂ}, Δ = Δ(𝔤_{ℂ}, 𝔱_{ℂ}) the set of roots, and let Δ_{c} and Δ_{nc} be the subset of compact and noncompact roots, respectively. We can choose an ordering on Δ such that

𝔭_{+} = ∑_{ϕ∈Δ_{nc}^{+}} 𝔤_{ℂ}^{ϕ}, 𝔭_{−} = ∑_{ϕ∈Δ_{nc}^{−}} 𝔤_{ℂ}^{ϕ}.

For ϕ ∈ Δ, define h_{ϕ} ∈ i𝔱 by \frac{2ϕ(h)}{⟨ϕ,ϕ⟩} = ⟨h_{ϕ}, h⟩ for h ∈ 𝔱. We choose root vectors e_{ϕ} ∈ 𝔤_{ℂ}^{ϕ} normalized so that [e_{ϕ}, e_{−ϕ}] = h_{ϕ} and θ(\overline{e}_{ϕ}) = −e_{−ϕ}. Then 𝔭 will have a real basis consisting of x_{ϕ} = e_{ϕ} + e_{−ϕ}, y_{ϕ} = i(e_{ϕ} − e_{−ϕ}), ϕ ∈ Δ_{nc}^{+}.

Let r be the real rank of G̃. We inductively construct a maximal set Ψ of strongly orthogonal noncompact roots by taking ψ_{1} to be the largest element of Δ_{nc}^{+}, ψ_{j} the largest root in Δ_{nc}^{+} strongly orthogonal to ψ_{1}, …, ψ_{j−1}. Then |Ψ| = r. For Γ ⊂ Ψ, we define the partial Cayley transform c_{Γ} [22] by

c_{Γ} = ∏_{Γ} c_{γ}, c_{γ} = exp (\frac{π}{4} y_{γ}) .

Now we have c_{Γ} · exp Z ∈ P_{+}K_{ℂ}P_{−} for every Z ∈ D [19, Chapter III, Section 7], so we can define a domain

S_{Γ} = c_{Γ} (D) = {c_{Γ}(Z) : Z ∈ D} ⊂ 𝔭_{+}.

Then S_{Γ} is the image under the Harish-Chandra embedding ζ of the translated orbit c_{Γ} ·G(x_{0}) ⊂ G_{ℂ}/K_{ℂ}P_{+}, where the basepoint x_{0} = 1K_{ℂ}P_{+}. For Γ nonempty, S_{Γ} gives an unbounded realization of the symmetric domain D [19, 23].

We now specialize to the example G = Sp(n, ℂ)∩U(n, n). We take compact Cartan subalgebra for 𝔤_{ℂ} = 𝔰𝔭(n, ℂ) to be

𝔱_{ℂ} = {diag(b_{1} … b_{n}, −b_{1} ⋯ − b_{n}) : b_{1} … b_{n} ∈ ℂ} .

Write ω_{i} for the linear form on 𝔱_{ℂ} given by ω_{i}(diag(b_{1} … b_{n}, −b_{1} ⋯ − b_{n})) = b_{i}. Then the set of roots is

Δ(𝔤_{ℂ}, 𝔱_{ℂ}) = {±2ω_{i}, 1 ≤ i ≤ n; ±ω_{i} ± ω_{j}, 1 ≤ i ≠ j ≤ n}

with simple system S = {α_{1} … α_{n}} where α_{i} = ω_{i}−ω_{i+1}, 1 ≤ i ≤ n−1, α_{n} = 2ω_{n}. Write Δ^{+} for the system of positive roots determined by S, Δ^{−} = −Δ^{+}. Let ψ_{1} = 2ω_{1}, …, ψ_{n} = 2ω_{n} and let Ψ = {ψ_{1} … ψ_{n}}. Then Ψ ⊂ Δ_{nc}^{+} is a maximal set of strongly orthogonal noncompact positive roots.

For ψ_{i} ∈ Ψ, the element h_{ψ_{i}} defined above is given by h_{ψ_{i}} = E_{i,i}−E_{n+i,n+i} ∈ i𝔱. Taking e_{ψ_{i}} = −iE_{i,n+i} ∈ 𝔤_{ℂ}^{ψ_{i}} and e_{−ψ_{i}} = iE_{n+i,i} ∈ 𝔤_{ℂ}^{−ψ_{i}} gives [e_{ψ_{i}}, e_{−ψ_{i}}] = h_{ψ_{i}} and defines elements x_{ψ_{i}} = e_{ψ_{i}} + e_{−ψ_{i}} and y_{ψ_{i}} = i(e_{ψ_{i}} − e_{−ψ_{i}}) in 𝔭. Now for b = 1 … n, define Γ_{b} = {ψ_{1}, …, ψ_{b}} ⊂ Ψ and let x_{b} = ∑_{ψ∈Γ_{b}} x_{ψ}, y_{b} = ∑_{ψ∈Γ_{b}} y_{ψ} ∈ 𝔭. We have:

x_{b} = (\begin{matrix}0_{b} & 0 & −i · 1_{b} & 0 \\ 0 & 0_{n−b} & 0 & 0 \\ i · 1_{b} & 0 & 0_{b} & 0 \\ 0 & 0 & 0 & 0_{n−b} \end{matrix}), y_{b} = (\begin{matrix}0_{b} & 0 & 1_{b} & 0 \\ 0 & 0_{n−b} & 0 & 0 \\ 1_{b} & 0 & 0_{b} & 0 \\ 0 & 0 & 0 & 0_{n−b} \end{matrix}).

We define the b^{th} partial Cayley transform by c_{b} = c_{Γ_{b}} = ∏_{ψ∈Γ_{b}} exp(\frac{πi}{4}y_{ψ}). Then:

c_{b} = exp (\frac{πi}{4}y_{b}) = (\begin{matrix}\frac{1}{\sqrt{2}} · 1_{b} & 0 & \frac{i}{\sqrt{2}} · 1_{b} & 0 \\ 0 & 1_{n−b} & 0 & 0 \\ \frac{i}{\sqrt{2}} · 1_{b} & 0 & \frac{1}{\sqrt{2}} · 1_{b} & 0 \\ 0 & 0 & 0 & 1_{n−b} \end{matrix}).

For T = (\begin{matrix}T_{11} & T_{12} \\ ^{t}T_{12} & T_{22} \end{matrix}) ∈ D, we calculate

c_{b}(T) = (\begin{matrix}(T_{11} + i)(iT_{11} + 1)^{−1} & \sqrt{2}(iT_{11} + 1)^{−1}T_{12} \\ \sqrt{2}^{t}T_{12}(iT_{11} + 1)^{−1} & T_{22} − i^{t}T_{12}(iT_{11} + 1)^{−1}T_{12} \end{matrix})

with the factor of automorphy given by

j(c_{b}, T) = (\begin{matrix}\sqrt{2}(iT_{11} + 1)^{−1} & 0 \\ −i^{t}T_{12}(iT_{11} + 1)^{−1} & 1_{n−b} \end{matrix}).

Write S_{b} = c_{b}(D) ⊂ 𝔭_{+} for the unbounded realizations of G/K obtained as above. We will write c = c_{n} for the full Cayley transform and S = c(D) ⊂ 𝔭_{+}. As G^{c_{b}} preserves the form

⟨u, v⟩_{b} = u^{∗}c_{b}(\begin{matrix}−1_{n} & 0 \\ 0 & 1_{n} \end{matrix})c_{b}^{∗}v, u, v ∈ ℂ^{2n}

we see that

S_{b} = {n-planes p_{W} : ⟨·, ·⟩_{b}|_{p_{W}} ≫ 0}

= \{W ∈ Sym_{n}(ℂ) : (\begin{matrix}−i(W_{11} − \overline{W}_{11}) − \overline{W}_{12}^{t}W_{12} & −\overline{W}_{12}W_{22} − iW_{12} \\ −\overline{W}_{22}^{t}W_{12} + i^{t}\overline{W}_{12} & 1 − W_{22}^{∗}W_{22} \end{matrix}) ≫ 0\} . (11)

The domains S_{b} are type III Siegel domains [17, 19]. In particular, the domain S is the Siegel upper half-space:

S = {W ∈ 𝔭_{+} ≅ Sym_{n}(ℂ) : Im(W) ≫ 0} .

We will need the following easy observation:

Lemma 6.1. Let W = ( \begin{matrix}W_{11} & W_{12} \\ ^{t}W_{12} & W_{22} \end{matrix} ) ∈ S_{b}. Then Im W_{11} ≫ 0.

Proof. Taking the b × b principal submatrix of the hermitian matrix in (11) defining S_{b} gives

−i(W_{11} − \overline{W}_{11}) − \overline{W}_{12}^{t}W_{12} ≫ 0 ⇒ 2 Im W_{11} ≫ \overline{W}_{12}^{t}W_{12} ⩾ 0.

We will also need formulas for the inverse Cayley transforms. For W ∈ S_{b}, we have c_{b}^{−1} · exp(W) ∈ P_{+}K_{ℂ}P_{−}, and we obtain

c_{b}^{−1}(W) = (\begin{matrix}(W_{11} − i)(−iW_{11} + 1)^{−1} & \sqrt{2}(−iW_{11} + 1)^{−1}W_{12} \\ \sqrt{2}^{t}W_{12}(−iW_{11} + 1)^{−1} & W_{22} + i^{t}W_{12}(−iW_{11} + 1)^{−1}W_{12} \end{matrix})

with factor of automorphy

j(c_{b}^{−1}, W) = (\begin{matrix}\sqrt{2}(−iW_{11} + 1)^{−1} & 0 \\ i^{t}W_{12}(−iW_{11} + 1)^{−1} & 1_{n−b} \end{matrix}).

7. Partial Bargmann Transforms

We begin by introducing the Schrödinger model for irreducible unitary representations of the Heisenberg group. Let M = M_{n,k}(ℝ) with real inner product ⟨x|y⟩ = Tr(x ·^{t}y). We have a real symplectic form on M_{2n,k}(ℝ) = M × M given by:

((\begin{matrix}x \\ y \end{matrix}) , (\begin{matrix}x′ \\ y′ \end{matrix})) ↦ Tr (J (\begin{matrix}x \\ y \end{matrix}) · (^{t}x′ ^{t}y′)) = ⟨x|y′⟩ − ⟨y|x′⟩.  (12)

In this section only, we let H denote the Heisenberg group H = M×M×ℝ with group law

(x, y; t)(x′, y′; t′) = (x + x′, y + y′; t + t′ + \frac{1}{2}(⟨x|y′⟩ − ⟨y|x′⟩)).

The group H is of course isomorphic to the Heisenberg group H = M × ℝ, M = M_{n,k}(ℂ), which we used for the Fock model. We express this isomorphism in

terms of the full Cayley transform c as follows. For w ∈ M, identify w ↔ (\begin{matrix}w \\ −i\overline{w} \end{matrix}).

Define a map M×M → M sending (\begin{matrix}x \\ y \end{matrix}) ↦ w where (\begin{matrix}w \\ −i\overline{w} \end{matrix}) = c^{−1} (\begin{matrix}x \\ y \end{matrix}) ⇔ w =

\frac{1}{\sqrt{2}}(x − iy).

Lemma 7.1. The map H → H defined by (x, y; t) ↦ (\frac{1}{\sqrt{2}}(x − iy); t) is a group homomorphism.

Proof. The result follows from the fact that

Tr (J(\begin{matrix}x \\ y \end{matrix}) · (^{t}x′ ^{t}y′)) = Tr (c^{∗}Jc(\begin{matrix}w \\ −i\overline{w} \end{matrix}) · (w′^{∗} i\overline{w}′^{∗})) = 2 Im(w|w′).

We define an irreducible unitary representation ρ of H on L^{2}(M) with central character χ(t) = e^{−πit} by

ρ(x_{0}, y_{0}; t)f(x) = e^{−πit}e^{−πi⟨x_{0}|x⟩}e^{\frac{πi}{2} ⟨x_{0}|y_{0}⟩}f(x − y_{0}).

The representation (ρ, L^{2}(M)) of H is called the Schrödinger model. We now wish to establish the connection between the representation ρ of H and the Fock model ρ_{0} of H.

Following [6], define the Fourier-Wigner transform V : L^{2}(M) × L^{2}(M) → L^{∞}(M × M) by

V (f, g)(p, q) = ⟨ρ(p, q)f, g⟩_{L^{2}(M)}

for (p, q) ∈ M × M. We think of V (f, g) as a matrix coefficient for ρ where the dependence on the variable t is being ignored. One can easily verify:

V (ρ(x, y)f, g)(p, q) = e^{− \frac{πi}{2} (⟨p|y⟩−⟨q|x⟩)}V (f, g)(p + x, q + y).  (13)

Leslie 95

Now we let ϕ_{0}(x) = e^{− \frac{π}{2} ⟨x|x⟩} ∈ L^{2}(M) and calculate:

V (f, ϕ_{0})(p, q) = ⟨f|ρ(−p, −q)ϕ_{0}⟩
= ∫_{M} f(x)e^{−πi⟨p−iq|x⟩}e^{\frac{π}{4} ⟨p−iq|p−iq⟩}e^{− \frac{π}{4} (⟨p|p⟩+⟨q|q⟩)}e^{− \frac{π}{2} ⟨x|x⟩}
= e^{− \frac{π}{2} |z|^{2}} · e^{\frac{π}{2} ⟨z|z⟩} ∫_{M} f(x)e^{πi\sqrt{2}⟨z|x⟩}e^{− \frac{π}{2} ⟨x|x⟩}dx

where we have made the change of variables z = −\frac{1}{\sqrt{2}} (p − iq) and we write |z|^{2} for Tr(z · z^{∗}). Definition 7.2. The (full) Bargmann transform Bf of a function f ∈ L^{2}(M) is given by

Bf(z) = e^{\frac{π}{2} |z|^{2}}V (f, ϕ_{0})(p, q) = e^{\frac{π}{2} ⟨z|z⟩} ∫_{M} f(x)e^{πi\sqrt{2}⟨z|x⟩}e^{− \frac{π}{2} ⟨x|x⟩}dx

where z = −\frac{1}{\sqrt{2}} (p − iq).

Calculations as in [6] show: Lemma 7.3. The Bargmann transform f ↦ Bf is an isometry from L^{2}(M) to F(M).

By property (13) of the Fourier-Wigner transform, we obtain: Lemma 7.4. Let (x, y) ∈ M × M, f ∈ L^{2}(M). Then if we take w = \frac{1}{\sqrt{2}} (x − iy), we obtain:

ρ_{0}(w)Bf(z) = (Bρ(x, y)f)(z).

We will also need the formula for the inverse Bargmann transform [6]. Lemma 7.5. The (full) inverse Bargmann transform B^{−1} : F(M) → L^{2}(M) is given by

B^{−1}g(x) = e^{− \frac{π}{2} ⟨x|x⟩} ∫_{M} e^{−πi\sqrt{2}⟨\overline{z}|x⟩}e^{\frac{π}{2} ⟨\overline{z}|\overline{z}⟩}e^{−π|z|^{2}}g(z)dz

for g ∈ F(M), provided this integral converges absolutely.

Now fix b ∈ {0 … n} and define the mixed polarization model of the Heisenberg group as H_{b} = M_{b} × M_{n−b} × M_{b} × ℝ, where M_{b} = M_{b×k}(ℝ) and M_{n−b} = M_{(n−b)×k}(ℂ), with group multiplication

(x, w, y; t)(x′, w′, y′; t′) =

(x + x′, w + w′, y + y′; t + t′ + \frac{1}{2}(⟨x|y′⟩ − ⟨y|x′⟩) + Im(w|w′)).

If we make the identification (x, w, y) ∈ M_{b} × M_{n−b} × M_{b} ↔ γ = ( \begin{matrix}x \\ w \\ y \\ −i\overline{w} \end{matrix} ) then we get a real symplectic form on M_{b} × M_{n−b} × M_{b} via

(γ, γ′) ↦ Tr (c_{b}^{∗}Jc_{b}γ · γ′^{∗}) = ⟨x|y′⟩ − ⟨y|x′⟩ + 2 Im(w|w′).  (14)

We let L^{2}(M_{b}, F) be the space of L^{2} functions on M_{b} with values in F(M_{n−b}). Alternatively, L^{2}(M_{b}, F) is the Hilbert space of classes of measurable functions f on M_{b} × M_{n−b} such that

∫_{M_{b}} ∫_{M_{n−b}} f(x, v)e^{−π(v|v)}dv dx < ∞

and such that v ↦ f(x, v) is a holomorphic function on M_{n−b}.

We obtain an irreducible unitary representation ρ of H_{b} on L^{2}(M_{b}, F) with central character χ(t) = e^{−πit} by

ρ(x_{0}, w_{0}, y_{0}; t)f(x, v) =

e^{−πit}e^{π(v|w_{0})}e^{− \frac{π}{2} (w_{0}|w_{0})}e^{−πi⟨x_{0}|x⟩}e^{\frac{πi}{2} ⟨x_{0}|y_{0}⟩}f(x − y_{0}, v − w_{0}).

The case b = n gives the Schrödinger model of ρ as above, b = 0 gives the Fock model, and the values 0 < b < n correspond to mixed models [11].

Definition 7.6. For z ∈ M, write z = (\begin{matrix}z_{1} \\ z_{2} \end{matrix}) where z_{1} ∈ M_{b} and z_{2} ∈ M_{n−b}.

The partial Bargmann transform B_{b}f of a function f ∈ L^{2}(M_{b}, F) is given by

B_{b}f(\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = e^{\frac{π}{2} ⟨z_{1}|z_{1}⟩} ∫_{M_{b}} f(x, z_{2})e^{πi\sqrt{2}⟨z_{1}|x⟩}e^{− \frac{π}{2} ⟨x|x⟩}dx.

The following Lemmas follow immediately from Lemmas 7.3, 7.4 and 7.5:

Lemma 7.7. The partial Bargmann transform f ↦ B_{b}f is an isometry from L^{2}(M_{b}, F) to F(M).

Lemma 7.8. Let (x, y) ∈ M_{b} × M_{b}, f ∈ L^{2}(M_{b}, F). Then if we take w_{1} = \frac{1}{\sqrt{2}} (x − iy), we obtain:

ρ_{0}(\begin{matrix}w_{1} \\ w_{2} \end{matrix})B_{b}f(\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = (B_{b}ρ(x, w_{2}, y)f)(\begin{matrix}z_{1} \\ z_{2} \end{matrix}).

Lemma 7.9. The inverse partial Bargmann transform B_{b}^{−1} mapping F(M) to L^{2}(M_{b}, F) is given by

B^{−1}g(\begin{matrix}x \\ z_{2} \end{matrix}) = e^{− \frac{π}{2} ⟨x|x⟩} ∫_{M_{b}} e^{−πi\sqrt{2}⟨\overline{z}_{1}|x⟩}e^{\frac{π}{2} ⟨\overline{z}_{1}|\overline{z}_{1}⟩}g(\begin{matrix}z_{1} \\ z_{2} \end{matrix})e^{−π|z_{1}|^{2}}dz_{1}

for g ∈ F(M), provided that this integral converges absolutely.

8. Intertwining Maps for Mixed Models

We have set up our partial Bargmann transform so that

ρ_{0}(w)B_{b}f(z) = (B_{b}ρ(c_{b}(w))f)(z)  (15)

where we are identifying c_{b}(w) = c_{b}(\begin{matrix}w \\ −i\overline{w} \end{matrix}) = ( \begin{matrix}x \\ w_{2} \\ y \\ −i\overline{w}_{2} \end{matrix} ) ↔ (x, w_{2}, y). Under this

identification, the group G^{c_{b}} = c_{b}Gc_{b}^{−1} acts on M_{b} × M_{n−b} × M_{b} by matrix multiplication. We define \overline{ω} : G^{c_{b}} → U(L^{2}(M_{b}, F))/ℤ_{2} by \overline{ω}(c_{b}gc_{b}^{−1}) = B_{b}^{−1} ∘ \overline{ω}(g) ∘ B_{b} for g ∈ G and by (15), \overline{ω} will satisfy

\overline{ω}(g′)ρ(x, w, y) = ρ(g′ · (x, w, y))\overline{ω}(g′), where g′ = c_{b}gc_{b}^{−1}.

We define the double cover G̃^{c_{b}} of G^{c_{b}} to be the set \{(g′, d^{\frac{1}{2}} (g′, W))\}, g′ ∈ G^{c_{b}}, where now we take W ∈ S_{b}, so d(g′, W) = det(j(g′, W)) is a function d : G^{c_{b}} × S_{b} → ℂ − {0}. Here we can identify a choice of branch of det(j(g′, W)) with its value at W = e_{b} = c_{b}(0) = ( \begin{matrix}i·1_{b} & 0 \\ 0 & 0_{n−b} \end{matrix} ).

We lift the functions j(c_{b}, ·) : D → K_{ℂ} and j(c_{b}^{−1}, ·) : S_{b} → K_{ℂ}, now

choosing d^{\frac{1}{2}} (c_{b}^{−1}, e_{b}) = (\frac{1}{\sqrt{2}})^{\frac{b}{2}} to be positive. Thus we obtain a map G̃ ⟶^{c_{b}} G̃^{c_{b}}

by (g, t) ⟼^{c_{b}} (g′, t′), where g′ = c_{b}gc_{b}^{−1} and t′ = s_{2} t s_{1} with s_{1}, s_{2} defined by

j̃(c_{b}^{−1}, e_{b}) = (j(c_{b}^{−1}, e_{b}), s_{1}) and j̃(c_{b}, g(0)) = (j(c_{b}, g(0)), s_{2}). Then we get:

ω(g′, t′) = B_{b}^{−1} ∘ ω(g, t) ∘ B_{b}.

Remark 8.1. When b = n, this realization of ω gives the Schrödinger model of the oscillator representation of G̃^{c}, while the cases 0 < b < n correspond to mixed models. For explicit formulas for the action of ω in these models for a generating set of elements of G^{c_{b}}, see [15] or [16].

Finally, we lift j : G̃^{c_{b}} × S_{b} → K_{ℂ} to obtain:
j̃((g′, t′), W) = j̃(c_{b}, g(T))j̃((g, t), T)j̃(c_{b}^{−1}, c_{b}(T))

for T ∈ D, c_{b}(T) ∈ S_{b}.

For λ ∈ { \atop O(k)}, we let L^{2}(M_{b}, F; λ) be the subspace of functions f in L^{2}(M_{b}, F) ⊗ V_{λ} satisfying f(xc, vc) = λ(c)^{−1}f(x, v) for all c ∈ O(k). The partial Bargmann transform B_{b} maps L^{2}(M_{b}, F; λ) → F(M; λ), so once again we can replace ω with ω_{λ} in the above discussion.

We will now construct a G̃^{c_{b}} intertwining map between L^{2}(M_{b}, F; λ) and a space of V_{τ} -valued functions on S_{b}, where V_{τ} = H(M; λ) as before.

The idea is to apply the partial inverse Bargmann transform to the equation:

ω_{λ}(g, t)q_{T} h = q_{g(T)}J((g, t), T)^{∗}^{−1}h, (g, t) ∈ G̃, h ∈ H(M; λ)  (16)

to obtain a similar expression for G̃^{c_{b}}, from which we can determine the intertwining map.

Let W = ( \begin{matrix}W_{11} & W_{12} \\ ^{t}W_{12} & W_{22} \end{matrix} ) ∈ S_{b} with block decomposition as defined in Section 6.

Proposition 8.2. For W ∈ S_{b}, define a function on M_{b} × M_{n−b} by

q′_{W} (x, v) = e^{− \frac{πi}{2} Tr(^{t}x\overline{W}_{11}x)}e^{π Tr(^{t}x\overline{W}_{12}v)}e^{\frac{πi}{2} Tr(^{t}v\overline{W}_{22}v)}.

Then q′_{W} ∈ L^{2}(M_{b}, F).

Proof. Let T = c_{b}^{−1}(W) ∈ D. We will show that q′_{W} = αB_{b}^{−1}q_{T}, α a constant. Since q_{T} ∈ F(M) and |q_{T} (z)| < e^{Cπ|z|^{2}} = e^{Cπ(|z_{1}|^{2}+|z_{2}|^{2})}, the integral defining B_{b}^{−1}q_{T} converges absolutely and gives a function in L^{2}(M_{b}, F).

We will need the following Lemma on Gaussian integrals [6, Appendix A]:

Lemma 8.3. Let A, D ∈ Sym_{N}(ℂ) with ∥A∥ ≤ 1, ∥D∥ ≤ 1, and ∥A∥∥D∥ < 1. Then for any u, v ∈ M_{N,k}(ℂ)

∫_{M_{N,k}(ℂ)} e^{\frac{π}{2} Tr(^{t}wAw)}e^{\frac{π}{2} Tr(^{t}\overline{w}D\overline{w})}e^{π Tr(^{t}uw)}e^{π Tr(^{t}v\overline{w})}e^{−π(w|w)}dw
= det^{− \frac{k}{2}} (1 − AD)e^{\frac{π}{2} Tr(^{t}uD(1−AD)^{−1}u)}e^{π Tr(^{t}v(1−AD)^{−1}u)}e^{\frac{π}{2} Tr(^{t}vA(1−DA)^{−1}v)}.

Now write T = ( \begin{matrix}T_{11} & T_{12} \\ ^{t}T_{12} & T_{22} \end{matrix} ) and calculate:

B_{b}^{−1}q_{T} (\begin{matrix}x \\ z_{2} \end{matrix})

= e^{− \frac{π}{2} Tr(^{t}xx)}e^{\frac{πi}{2} Tr(^{t}z_{2}\overline{T}_{22}z_{2})} ·

∫_{M_{b}} e^{−πi\sqrt{2} Tr(^{t}\overline{z}_{1}x)}e^{\frac{π}{2} Tr(^{t}\overline{z}_{1}\overline{z}_{1})}e^{\frac{πi}{2} Tr(^{t}z_{1}\overline{T}_{11}z_{1})}e^{πi Tr(^{t}z_{1}\overline{T}_{12}z_{2})}e^{−π(z_{1}|z_{1})}dz_{1}

= α e^{− \frac{π}{2} Tr(^{t}xx)}e^{\frac{πi}{2} Tr(^{t}z_{2}\overline{T}_{22}z_{2})}e^{π Tr(^{t}x(−i\overline{T}_{11})(1−i\overline{T}_{11})^{−1}x)}e^{π\sqrt{2} Tr(^{t}x(1−i\overline{T}_{11})^{−1}\overline{T}_{12}z_{2})} ·

e^{− \frac{π}{2} Tr(^{t}z_{2}^{t}\overline{T}_{12}(1−i\overline{T}_{11})^{−1}\overline{T}_{12}z_{2})}, α a constant, by the Lemma

= α e^{− \frac{πi}{2} Tr(^{t}x\overline{W}_{11}x)}e^{π Tr(^{t}x\overline{W}_{12}z_{2})}e^{\frac{πi}{2} Tr(^{t}z_{2}\overline{W}_{22}z_{2})}

where we are using the formulas for c_{b} ( \begin{matrix}T_{11} & T_{12} \\ ^{t}T_{12} & T_{22} \end{matrix} ) = ( \begin{matrix}W_{11} & W_{12} \\ ^{t}W_{12} & W_{22} \end{matrix} ) found in Section 6.

Theorem 8.4. Let h ∈ H(M; λ). Then for T ∈ D, c_{b}(T) ∈ S_{b}, the function f(x, z_{2}) = q′_{c_{b}(T)}(x, z_{2})h(x, iz_{2}) ∈ L^{2}(M_{b}, F; λ). We have:

(B_{b}f)(z) = C q_{T} (z)(J(c_{b}, T)^{∗}h)(iz), C a constant

Proof. We will need the mean value theorem for harmonic polynomials:

Lemma 8.5. Let h be a harmonic polynomial on ℝ^{N} extended holomorphically to ℂ^{N}. Then:

∫_{ℝ^{n}} e^{−2i⟨x|y⟩}e^{−⟨x|x⟩}h(x)dx = π^{\frac{n}{2}} e^{−⟨y|y⟩}h(−iy)

This leads to the following (adapted from [14]):

Lemma 8.6. Let Z ∈ Sym_{N}(ℂ), Im(Z) ≫ 0, and h a polynomial on M_{N} extended holomorphically to M_{N} such that x ↦ h(gx) is harmonic for any g ∈ GL(N, ℝ). Then:

∫_{M} e^{2πi⟨x|y⟩}e^{πi⟨Zx|x⟩}h(x)dx = (det \frac{Z}{i} )^{− \frac{k}{2}} e^{πi⟨−Z^{−1}y|y⟩}h(−Z^{−1}y).

Here, choose the branch of the square root function so that (det \frac{Z}{i} )^{− \frac{1}{2}} is positive

when Z = i · 1_{N}.

Proof. Both sides are holomorphic in Z, so by analytic continuation, it suffices to prove the result for Z = iα^{2}, where α is a real positive definite symmetric matrix. Then

∫_{M} e^{2πi⟨x|y⟩}e^{−π⟨αx|αx⟩}h(x)dx = (det α)^{−k} ∫_{M} e^{2π⟨x|α^{−1}y⟩}e^{−π⟨x|x⟩}h(α^{−1}x)dx

= (det α)^{−k} e^{−π⟨α^{−1}y|α^{−1}y⟩}h(α^{−1}(iα^{−1}y)) = (det \frac{Z}{i} )^{− \frac{k}{2}} e^{πi⟨−Z^{−1}y|y⟩}h(−Z^{−1}y)

by the mean value theorem for harmonic polynomials.

Now we notice:

Lemma 8.7. Let h ∈ H(M; λ) and write (x, v) for coordinates on M_{b}×M_{n−b}. Then the function h_{v} : x ↦ h(x, v) is a polynomial on M_{b} such that x ↦ h_{v}(gx) is harmonic for any g ∈ GL(b, ℝ).

Proof. By Corollary 5.1, h ∈ H(M) ⇔ Δ_{i,j}h(z) = 0, 1 ≤ i ≤ j ≤ n, where Δ_{i,j} = ∑_{ν=1}^{k} \frac{∂^{2}}{∂z_{iν}∂z_{jν}} . But then Δ_{i,j}h(z) = 0 for 1 ≤ i ≤ j ≤ b. Thus writing z = (z_{1}, z_{2}) ∈ M_{b}×M_{n−b}, the polynomial z_{1} ↦ h(z_{1}, z_{2}) is in H(M_{b}). The Lemma follows.

This observation will allow us to use Lemma 8.6 to calculate B_{b}f. Letting W = c_{b}(T) ∈ S_{b}, f(x, z_{2}) = q′_{W} (x, z_{2})h(x, iz_{2}), and recalling that Im W_{11} ≫ 0 by Lemma 6.1, we obtain

B_{b}f(\begin{matrix}z_{1} \\ z_{2} \end{matrix})

= e^{\frac{π}{2} Tr(^{t}z_{1}z_{1})}e^{\frac{πi}{2} Tr(^{t}z_{2}\overline{W}_{22}z_{2})} ∫_{M_{b}}e^{\frac{πi}{2} Tr^{t}x(i−\overline{W}_{11})x}e^{πi Tr^{t}x(\sqrt{2}z_{1}−i\overline{W}_{12}z_{2})}h(x, iz_{2})dx

= C det (\frac{i − \overline{W}_{11}}{\sqrt{2}i} )^{− \frac{k}{2}}q_{T} (\begin{matrix}z_{1} \\ z_{2} \end{matrix})h((\begin{matrix}\sqrt{2}i(i − \overline{W}_{11})^{−1} & (i − \overline{W}_{11})^{−1}\overline{W}_{12} \\ 0 & 1_{n−b} \end{matrix})(\begin{matrix}iz_{1} \\ iz_{2} \end{matrix}))
= C q_{T} (\begin{matrix}z_{1} \\ z_{2} \end{matrix})(J(c_{b}^{−1}, W)^{∗}^{−1}h)(\begin{matrix}iz_{1} \\ iz_{2} \end{matrix})

where C is a constant, using the expression for J(c_{b}^{−1}, W) from Section 6. Now the relation J(c_{b}^{−1}, W)^{−1} = J(c_{b}, T) for T = c_{b}^{−1}(W) gives the result.

Let R_{b} : H(M; λ) → L^{2}(M_{b}, F; λ) be the map given by R_{b}h(x, v) = h(x, iv) for (x, v) ∈ M_{b} × M_{n−b}. By the Theorem, we see that applying the partial inverse Bargmann transform B_{b} to equation (16) and canceling constants on both sides gives:

ω_{λ}(g′, t′)q′_{c_{b}(T)}R_{b}(J(c_{b}, T)^{∗}^{−1}h) = q′_{g′^{−1}(c_{b}(T))}R_{b}(J(c_{b}, g(T))^{∗}^{−1}J((g, t), T)^{∗}^{−1}h)

⇔ ω_{λ}(g′, t′)q′_{c_{b}(T)}R_{b}(h) = q′_{g′^{−1}(c_{b}(T))}R_{b}(J(c_{b}, g(T))^{∗}^{−1}J((g, t), T)^{∗}^{−1}J(c_{b}, T)^{∗}h)

We abuse notation and write q′_{c_{b}(T)} ∘ R_{b} to denote the map H(M; λ) → L^{2}(M_{b}, F; λ) given by h ↦ q′_{c(T)}R_{b}(h). Let h_{1}, h_{2} ∈ H(M; λ) and S, T ∈ D, so c_{b}(S), c_{b}(T) ∈ S_{b}. Then by the Theorem:

⟨q′_{c_{b}(T)} ∘ R_{b}(h_{1})|q′_{c_{b}(S)} ∘ R_{b}(h_{2})⟩_{L^{2}(M_{b},F;λ)}
= ∥β∥^{2}⟨q_{T} J(c_{b}, T)^{∗}h_{1}|q_{S}J(c_{b}, S)^{∗}h_{2}⟩_{F(M;λ)}, β a constant
= ∥β∥^{2}⟨J(c_{b}, S)Q(S, T)J(c_{b}, T)^{∗}h_{1}|h_{2}⟩_{H(M;λ)}

Now define Q′(c_{b}(S), c_{b}(T)) = ∥β∥^{2}J(c_{b}, S)Q(S, T)J(c_{b}, T)^{∗}, so that Q′ : S_{b}×S_{b} → Aut(V_{τ}). Q′ will be a positive definite kernel function because Q is positive definite, and so by [13], Q′ will be the kernel function for a space H(S_{b}, τ) of V_{τ} -valued functions on S_{b}. For (g′, t′) ∈ G̃^{c_{b}} we obtain:

J((g′, t′), c_{b}(S))Q′(c_{b}(S), c_{b}(T))J((g′, t′), c_{b}(T))^{∗}
= Q′(g′(c_{b}(S)), g′(c_{b}(T)))  (17)

where we can express J((g′, t′), c_{b}(S)) = J(c_{b}, g(S))J((g, t), S)J(c_{b}, S)^{−1} for (g, t) ∈ G̃ such that (g, t) ⟼^{c_{b}} (g′, t′).

It follows that G̃^{c_{b}} acts on H(S_{b}, τ) by

T (g′, t′)f(W) = J((g′, t′)^{−1}, W)^{−1}f(g′^{−1}(W))  (18)

for W ∈ S_{b} and that T defines a strongly continuous unitary representation of G̃^{c_{b}}. Finally, the map

q′_{c_{b}(T)}R_{b}(h) ↦ Q′(·, c_{b}(T))h

extends to an intertwining map Ξ_{λ} between (ω_{λ}, L^{2}(M_{b}, F; λ)) and (T, H(S_{b}, V_{τ})) which can be expressed globally by ⟨Ξ_{λ}f(W)|h⟩_{H(M;λ)} = ⟨f|q′_{W} R_{b}(h)⟩_{L^{2}(M_{b},F;λ)}. This map can be written as follows:

Theorem 8.8. Let f ∈ L^{2}(M_{b}, F; λ) and let (x, v) ↦ I_{λ}(x, v) be the polynomial function on M_{b} × M_{n−b} with values in Hom(V_{τ}, V_{λ}) defined by I_{λ}(x, v)h = h(x, iv) for h ∈ V_{τ} = H(M; λ). Then:

Ξ_{λ}f(W) = ∫_{M_{b}} ∫_{M_{n−b}} \overline{q′_{W} (x, v)}I_{λ}(x, v)^{∗}f(x, v)e^{−π(v|v)}dv dx

The proof is as in Proposition 5.7.

Remark 8.9. When b = n, the intertwining map given above is exactly the one found in [14].

9. The Fock Model for U(p, q)

For p ≥ q > 0, define I_{p,q} = (\begin{matrix}−1_{p} & 0 \\ 0 & 1_{q} \end{matrix}), and let

G = U(p, q) = {g ∈ GL(p + q, ℂ) : gI_{p,q}g^{∗} = I_{p,q}} .

Then G is a real form of G_{ℂ} = GL(p + q, ℂ). We write elements g ∈ G_{ℂ} in

block form g = (\begin{matrix}A & B \\ C & D \end{matrix}) where A ∈ M_{p,p}(ℂ), D ∈ M_{q,q}(ℂ). We take the maximal compact subgroup to be K = G ∩ U(p + q) ≅ U(p) × U(q), and we obtain the decomposition 𝔤_{ℂ} = 𝔭_{+} + 𝔨_{ℂ} + 𝔭_{−}, where

𝔭_{+} = \{(\begin{matrix}0 & Z \\ 0 & 0 \end{matrix}) : Z ∈ M_{p,q}(ℂ)\} ,
𝔨_{ℂ} = \{(\begin{matrix}X_{1} & 0 \\ 0 & X_{2} \end{matrix}) : X_{1} ∈ 𝔤𝔩(p, ℂ), X_{2} ∈ 𝔤𝔩(q, ℂ)\} ,
𝔭_{−} = \{(\begin{matrix}0 & 0 \\ Y & 0 \end{matrix}) : Y ∈ M_{q,p}(ℂ)\} .

We identify K_{ℂ} with GL(p, ℂ) × GL(q, ℂ), and for (g, Z) ∈ G_{ℂ} × 𝔭_{+} such that g · exp(Z) ∈ Ω = P_{+}K_{ℂ}P_{−}, we write the automorphy factor

j(g, Z) = k(g · exp(Z)) = (j_{1}(g, Z), j_{2}(g, Z))

with j_{1}(g, Z) ∈ GL(p, ℂ) and j_{2}(g, Z) ∈ GL(q, ℂ). The action of g on 𝔭_{+} defined by exp(g(Z)) = (g · exp(Z))_{+} is given by

g(Z) = (AZ + B)(CZ + D)^{−1}.

Here, the image of the Harish-Chandra embedding G/K → D ⊂ 𝔭_{+} is given by the generalized unit disk

D = {q-planes l_{Z} : (·, ·)|_{l_{Z}} ≫ 0} = {Z ∈ M_{p,q}(ℂ) : 1_{q} − Z^{∗}Z ≫ 0} .

We now follow the exposition in [1] to establish formulas for the Fock model of the oscillator representation of G = U(p, q).

Let M_{p+q} = M_{p+q,k}(ℂ) and define a hermitian inner product h_{0} on M_{p+q} by h_{0}(u, u′) = Tr(I_{p,q}uu′^{∗}). We get an associated Heisenberg group H_{0} = M_{p+q} ×ℝ with group multiplication

(u; t)(u′; t′) = (u + u′; t + t′ − Im h_{0}(u, u′)).

Write u ∈ M_{p+q} as u = (\begin{matrix}u_{1} \\ u_{2} \end{matrix}) with u_{1} ∈ M_{p,k}(ℂ) and u_{2} ∈ M_{q,k}(ℂ). As

before write (u|u′) = Tr(uu′^{∗}), |u|^{2} = (u|u). Let F_{p,q} be the Hilbert space of func-

tions f on M_{p,q} such that u_{1} ↦ f(\begin{matrix}u_{1} \\ u_{2} \end{matrix}) is holomorphic, u_{2} ↦ f(\begin{matrix}u_{1} \\ u_{2} \end{matrix}) is antiholo-

morphic, and f is square-integrable with respect to the Gaussian measure dμ(u) = e^{−π(u|u)}du. Our convention for Lebesgue measure is that du = d(Re u) d(Im u). The reproducing kernel K for F_{p,q} is given by K(z, w) = e^{π(z_{1}|w_{1})}e^{π(w_{2}|z_{2})}, that is,

f(z) = ∫_{M_{p+q}} K(z, w)f(w)e^{−π(w|w)}dw

for f ∈ F_{p,q} and z ∈ M_{p+q}.

We obtain an irreducible unitary representation ρ_{0} of H_{0} on F_{p,q} with central character χ(t) = e^{−πit} by

ρ_{0}(w; t)f(z) = e^{−πit}K(z, w)e^{− \frac{π}{2} (w|w)}f(z − w)

for (w; t) ∈ H_{0} and f ∈ F_{p,q}.

Identify 𝔭_{+} with M_{p,q}(ℂ) as above. For T ∈ 𝔭_{+} we can define a function on M_{p+q} by

q_{T} (z) = e^{π(z_{1}|T z_{2})} = e^{π Tr(z_{1}z_{2}^{∗}T^{∗})}.

Clearly q_{T} is holomorphic in z_{1} and antiholomorphic in z_{2}. Moreover, we have from [1, Proposition 1.4]:

Lemma 9.1. The function q_{T} lies in F_{p,q} if and only if T ∈ D.

Since the left action of G = U(p, q) on M_{p+q} preserves the form h_{0}, the action (w; t) ↦ (g · w; t), where (w; t) ∈ H_{0} and g ∈ G, is an automorphism of H_{0} which fixes the center of H_{0}. By the Stone-von Neumann theorem, we obtain a projective unitary representation ω of G defined by ω(g)ρ_{0}(w; t) = ρ_{0}(g·w; t)ω(g). In fact, we can choose the operators ω(g) such that ω becomes a representation of G as in [1, Theorem 1.12]:

Theorem 9.2. For g = (\begin{matrix}A & B \\ C & D \end{matrix}) ∈ G, define the operator ω(g) on F_{p,q} by

ω(g)f(z) = ∫_{M} K_{g}(z, w)f(w)e^{−π(w|w)}dw,  (19)

K_{g}(z, w) = (det^{−k}A)q_{BD^{−1}}(z)K ((\begin{matrix}A^{−1}z_{1} \\ D^{−1}z_{2} \end{matrix}), w) \overline{q}_{−A^{−1}B}(w).

Then ω : G → U(F_{p,q}) is a continuous unitary representation.

10. The Dual Pair (U(p, q), U(k))

Let G′ = U(k) act on F_{p,q} by right multiplication:

c · f(\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = f(\begin{matrix}z_{1}c \\ z_{2}c^{∗} & −1 \end{matrix})

for c ∈ U(k) and f ∈ F_{p,q}. The action of G′ commutes with ω, and in fact (U(p, q), U(k)) is a dual pair inside Sp(M_{p,q}) = Sp(2(p + q)k, ℝ), where (M_{p,q}, Im h_{0}) is regarded as a real symplectic vector space.

As in Section 5 and [2, Chapter 7], we define a map

θ : M_{p+q} → 𝔭_{+}, θ(z) = z_{1} · z_{2}^{∗}.

Let P be the dense subspace of polynomial functions in F = F_{p,q}, I the ideal in P generated by the matrix entries of z → θ(z), and H be the orthogonal complement of I. We call H = H_{p,q} the space of harmonic polynomials.

Now as in [2, Chapter 7]:

Proposition 10.1. Let h ∈ P. Then the following are equivalent:

(i) h ∈ H_{p,q}
(ii) g(∂)h = 0 for all matrix entries g of z → θ(z)
(iii) Δ_{i,j}h(z) = 0, 1 ≤ i ≤ p, 1 ≤ j ≤ q where Δ_{i,j} = ∑_{ν=1}^{k} \frac{∂^{2}}{∂z_{iν}∂\overline{z}_{jν}} .

One can calculate directly the action of K = U(p) × U(q) on F_{p,q}:

Lemma 10.2. Let f ∈ F and (g_{1}, g_{2}) ∈ K. Then

ω(g_{1}, g_{2})f(z) = (det g_{1})^{−k}f(\begin{matrix}g_{1}^{−1}z_{1} \\ g_{2}^{∗} z_{2} \end{matrix}).

Let { \atop U(k)} denote the unitary dual of U(k), and write λ for a representation in { \atop U(k)}, λ′ for its contragredient. As in [14, Section 6], we get:

Theorem 10.3. The space H is K × G′ invariant and decomposes as a multiplicity free orthogonal direct sum of unitary K × G′ representations τ(λ) ⊗ λ′.

For (λ, V_{λ}) in { \atop U(k)}, let F_{p,q}(λ) be the subspace of functions in F_{p,q} ⊗ V_{λ} satisfying (c·f)(z) = λ(c)^{−1}f(z) for all c ∈ U(k). Let ω_{λ} denote the representation of G on F_{p,q}(λ). Then we have

F_{p,q} = ⊕_{λ∈{ \atop U(k)}}F_{p,q}(λ) ⊗ V′_{λ}

as a representation of G × G′ and

H_{p,q} = ⊕_{λ∈{ \atop U(k)}}H_{p,q}(λ) ⊗ V′_{λ}

as a representation of K × G′.

Finally, analogous to Proposition 5.4, we have:

Proposition 10.4. Let λ ∈ { \atop U(k)} be such that F_{p,q}(λ) ≠ 0. Then F_{p,q}(λ) is an irreducible G representation.

For λ ∈ { \atop U(k)} such that the space H_{p,q}(λ) is non-zero, we construct a Gintertwining map between (ω, F_{p,q}(λ)) and sections of the homogeneous vector bundle over D associated to the K -module H_{p,q}(λ). The intertwining maps for this example were fully worked out in [1] and are similar to the example in Section 5. Here, we merely give an outline and refer to [1] and our previous work for proofs.

We write τ = τ(λ) for the K representation given by ω_{λ}|_{K} acting on H_{p,q}(λ). We extend τ to a holomorphic representation of K_{ℂ} = GL(p, ℂ) × GL(q, ℂ) as follows. Let τ_{0} be the holomorphic representation of K_{ℂ} on H_{p,q}(λ) given by

(τ_{0}(g_{1}, g_{2})h)(z_{1}, z_{2}) = h(g_{1}^{−1}z_{1}, g_{2}^{∗}z_{2})

for h ∈ H_{p,q}(λ), (g_{1}, g_{2}) ∈ GL(p, ℂ)×GL(q, ℂ), and (z_{1}, z_{2}) ∈ M_{p,k}(ℂ)×M_{q,k}(ℂ). Then τ(g_{1}, g_{2}) = (det g_{1})^{−k}τ_{0}(g_{1}, g_{2}) for (g_{1}, g_{2}) ∈ K_{ℂ}.

We write J(g, Z) = τ(j(g, Z)) for (g, Z) ∈ G_{ℂ} × 𝔭_{+} such that j(g, Z) is defined. Using the formula (19) for ω and the orthogonality between H_{p,q} and the ideal I generated by the non-constant U(k)-invariants, we prove just as in Lemma 5.5 [1, Proposition 2.2]:

Lemma 10.5. Let h ∈ H_{p,q}(λ). Then ω_{λ}(g)h(z) = q_{g(0)}(z)(J(g, 0)^{∗}^{−1}h)(z) for g ∈ G.

Corollary 10.6. Let T ∈ D and h ∈ H_{p,q}(λ). Then:

ω_{λ}(g)q_{T} h = q_{g(T)}J(g, T)^{∗}^{−1}h.

Thus we can use the construction outlined in Section 3. Let V_{τ} = H_{p,q}(λ), and take v, w ∈ V_{τ}. We obtain a positive definite operator-valued kernel function Q : D × D → Aut(V_{τ}) by ⟨q_{T} v|q_{S}w⟩ = ⟨Q(S, T)v|w⟩ and calculate that Q(S, T) = τ(q(S, T)). Q gives the reproducing kernel for a space of V_{τ} -valued holomorphic functions on D, denoted by H(D, τ), on which G acts by translation T_{τ}, where

T_{τ}(g)F (T) = J(g^{−1}, T)^{−1}F (g^{−1}(T))

for g ∈ G, T ∈ D, and F ∈ H(D, τ).

The intertwining map Ξ_{λ} : F_{p,q}(λ) → H(D, τ) is given on a dense subset of F_{p,q}(λ) by q_{T} v ↦ Q(·, T)v and is globally defined by Ξ_{λ}f(S) = q_{S}^{∗}f for f ∈ F_{p,q}, regarding q_{S} as an operator H_{p,q}(λ) → F_{p,q}(λ) given by multiplication by the function q_{S}.

Finally, another form of the intertwining map is given by:

Proposition 10.7. Let f ∈ F_{p,q}(λ) and let w ↦ I_{λ}(w) be the Hom(V_{τ}, V_{λ})- valued polynomial function on M defined by I_{λ}(w)h = h(w) for h ∈ V_{τ} = H_{p,q}(λ). Then:

Ξ_{λ}f(T) = ∫_{M} \overline{q_{T} (w)}I_{λ}(w)^{∗}f(w)e^{−π(w|w)}dw.

11. Cayley Transforms for U(p, q)

Retain the notation from Section 6. One can define a maximal set of strongly orthogonal noncompact roots Ψ = {ψ_{1}, …, ψ_{q}} such that h_{ψ_{i}} = E_{i,i}−E_{p+i,p+i} ∈ i𝔱, e_{ψ_{i}} = −iE_{i,p+i} ∈ 𝔤_{ℂ}^{ψ_{i}}, e_{−ψ_{i}} = iE_{p+i,i} ∈ 𝔤_{ℂ}^{−ψ_{i}} and [e_{ψ_{i}}, e_{−ψ_{i}}] = h_{ψ_{i}} for i = 1 … q. Then as before x_{ψ_{i}} = e_{ψ_{i}} + e_{−ψ_{i}} and y_{ψ_{i}} = i(e_{ψ_{i}} − e_{−ψ_{i}}) in 𝔭, and for b = 1 … q, define Γ_{b} = {ψ_{1}, …, ψ_{b}} ⊂ Ψ, x_{b} = ∑_{ψ∈Γ_{b}} x_{ψ}, and y_{b} = ∑_{ψ∈Γ_{b}} y_{ψ} ∈ 𝔭. The matrices for x_{b} and y_{b} are exactly the same are those given in Section 6. Finally, the b^{th} partial Cayley transform c_{b} = c_{Γ_{b}} = ∏_{ψ∈Γ_{b}} exp(\frac{πi}{4}y_{ψ}) is also identical to the one in Section 6.

For T = (\begin{matrix}T_{11} & T_{12} \\ T_{21} & T_{22} \end{matrix}) ∈ D, we have

c_{b}(T) = (\begin{matrix}(T_{11} + i)(iT_{11} + 1)^{−1} & \sqrt{2}(iT_{11} + 1)^{−1}T_{12} \\ \sqrt{2} T_{21}(iT_{11} + 1)^{−1} & T_{22} − i T_{21}(iT_{11} + 1)^{−1}T_{12} \end{matrix})

with the factor of automorphy

j(c_{b}, T) = ((\begin{matrix}\sqrt{2}(iT_{11} + 1)^{−1} & 0 \\ −i T_{21}(iT_{11} + 1)^{−1} & 1_{p−b} \end{matrix}) , (\begin{matrix}\frac{1}{\sqrt{2}}(iT_{11} + 1) & \frac{i}{\sqrt{2}}T_{12} \\ 0 & 1_{q−b} \end{matrix})).

Write S_{b} = c_{b}(D) ⊂ 𝔭_{+} for the unbounded realizations of G/K obtained as above, c = c_{q} for the full Cayley transform and S = c(D) ⊂ 𝔭_{+}. As G^{c_{b}} preserves the form

(u, v)_{b} = u^{∗}c_{b}I_{p,q}c_{b}^{∗}v, u, v ∈ ℂ^{p+q}

we see that

S_{b} = {q-planes l_{W} : (·, ·)_{b}|_{l_{W}} ≫ 0}

= \{W ∈ M_{p,q}(ℂ) : (\begin{matrix}−i(W_{11} − W_{11}^{∗}) − W_{21}^{∗}W_{21} & −W_{21}^{∗}W_{22} − iW_{12} \\ −W_{22}^{∗}W_{21} + iW_{12}^{∗} & 1 − W_{22}^{∗}W_{22} \end{matrix}) ≫ 0\} . (20)

For W ∈ S, we can write W = (\begin{matrix}W_{1} \\ W_{2} \end{matrix}) with W_{1} ∈ M_{q,q}(ℂ), W_{2} ∈ M_{p−q,q}(ℂ).

Then S is the type II Siegel domain [17]:

S = \{(\begin{matrix}W_{1} \\ W_{2} \end{matrix}) ∈ 𝔭_{+} : W_{1} = X + iY, X = X^{∗}, Y = Y^{∗}, Y − \frac{W_{2}^{∗}W_{2}}{2} ≫ 0\}.

The domains S_{b}, b ≠ q have the structure of type III Siegel domains [17]. We observe:

Lemma 11.1. Let W = ( \begin{matrix}W_{11} & W_{12} \\ W_{21} & W_{22} \end{matrix} ) ∈ S_{b}. Write W_{11} = X + iY with X, Y ∈

Herm_{b}(ℂ). Then Y ≫ 0.

Proof. Taking the b × b principal submatrix of the hermitian matrix in (20) defining S_{b} gives

−i(W_{11} − W_{11}^{∗}) − W_{21}^{∗}W_{21} ≫ 0 ⇒ 2Y ≫ W_{21}^{∗}W_{21} ⩾ 0.

For the inverse Cayley transforms, taking W ∈ S_{b}, we obtain:

c_{b}^{−1}(W) = (\begin{matrix}(W_{11} − i)(−iW_{11} + 1)^{−1} & \sqrt{2}(−iW_{11} + 1)^{−1}W_{12} \\ \sqrt{2} W_{21}(−iW_{11} + 1)^{−1} & W_{22} + i W_{21}(−iW_{11} + 1)^{−1}W_{12} \end{matrix}) (21)

with factor of automorphy

j(c_{b}^{−1}, W) = ((\begin{matrix}\sqrt{2}(−iW_{11} + 1)^{−1} & 0 \\ i W_{21}(−iW_{11} + 1)^{−1} & 1_{p−b} \end{matrix}) , (\begin{matrix}\frac{1}{\sqrt{2}}(−iW_{11} + 1) & −\frac{i}{\sqrt{2}}W_{12} \\ 0 & 1_{q−b} \end{matrix})).

12. Generalized Partial Bargmann Transforms

We now define mixed polarization models of the complex Heisenberg group and corresponding models for its irreducible unitary representations.

Fix b ∈ {0 … q} and define a hermitian inner product h_{b} on M_{p+q} = M_{p+q,k}(ℂ) by

h_{b}(u, u′) = Tr(c_{b}I_{p,q}c_{b}^{∗}uu′^{∗}) = h_{0}(c_{b}^{−1}u, c_{b}^{−1}u′).

We get an associated Heisenberg group H_{b} = M_{p+q} × ℝ with group multiplication

(u; t)(u′; t′) = (u + u′; t + t′ − Im h_{b}(u, u′)).

The following Lemma is clear from our choice of group law:

Lemma 12.1. The map H_{0} → H_{b} defined by (w; t) ↦ (c_{b}(w); t) is a group homomorphism.

We let L^{2}(M_{b}, F) = L^{2}(M_{b}, F_{p−b,q−b}) be the space of L^{2} functions on M_{b} with values in F_{p−b,q−b}. Then L^{2}(M_{b}, F) is the Hilbert space of classes of measurable functions f on M_{b} × M_{p−b} × M_{q−b} such that

∫_{M_{b}} ∫_{M_{p−b+q−b}} f(\begin{matrix}a \\ z_{1} \\ z_{2} \end{matrix})e^{−π|z|^{2}}dz da < ∞

and such that (\begin{matrix}z_{1} \\ z_{2} \end{matrix}) ↦ f(\begin{matrix}a \\ z_{1} \\ z_{2} \end{matrix}) is holomorphic in z_{1} and antiholomorphic in z_{2}.

Write H_{b} = M_{b}×M_{p−b}×M_{b}×M_{q−b}×ℝ and (u; t) ∈ H_{b} as u = (( \begin{matrix}a_{1} \\ w_{1} \\ a_{2} \\ w_{2} \end{matrix} ) , t).

We obtain an irreducible unitary representation ρ_{b} of H_{b} on L^{2}(M_{b}, F) with central character χ(t) = e^{−πit} by

ρ_{b}(( \begin{matrix}a_{1} \\ w_{1} \\ a_{2} \\ w_{2} \end{matrix} ) , t)f(\begin{matrix}a \\ z_{1} \\ z_{2} \end{matrix}) =

e^{−πit}e^{−2πi Re(a_{1}|a)}e^{πi Re(a_{1}|a_{2})}e^{π(z_{1}|w_{1})}e^{π(w_{2}|z_{2})}e^{\frac{π}{2} |w|^{2}}f(\begin{matrix}a − a_{1} \\ z_{1} − w_{1} \\ z_{2} − w_{2} \end{matrix}). (22)

Let H′_{b} = M_{b} × M_{b} × ℝ be the subgroup of H_{b} consisting of elements

((\begin{matrix}a_{1} \\ a_{2} \end{matrix}) , t) ↔ (( \begin{matrix}a_{1} \\ 0 \\ a_{2} \\ 0 \end{matrix} ) , t), and let ρ′_{b} = ρ_{b}|_{H′_{b}}. As in Section 7, we define a

Fourier-Wigner transform V_{b} : L^{2}(M_{b}) × L^{2}(M_{b}) → L^{∞}(M_{b} × M_{b}) by

V (f, g) (\begin{matrix}a_{1} \\ a_{2} \end{matrix}) = ⟨ρ′_{b} (\begin{matrix}a_{1} \\ a_{2} \end{matrix}) f, g⟩_{L^{2}(M_{b})}.

Now we let ϕ_{0}(a) = e^{− \frac{π}{2} (a|a)} ∈ L^{2}(M_{b}) and calculate:

V_{b}(f, ϕ_{0})(\begin{matrix}a_{1} \\ a_{2} \end{matrix}) = ⟨f|ρ′_{b} (\begin{matrix}−a_{1} \\ −a_{2} \end{matrix}) ϕ_{0}⟩

=e^{− \frac{π}{2} |z_{1}|^{2}}e^{− \frac{π}{2} |z_{2}|^{2}} · e^{−πi(z_{1}|z_{2})} ∫_{M_{b}} f(a)e^{πi\sqrt{2}(z_{1}|a)}e^{π\sqrt{2}(a|z_{2})}e^{−π(a|a)}da

where we have made the change of variables

(\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = c′_{b}^{−1} (\begin{matrix}−a_{1} \\ −a_{2} \end{matrix}), c′_{b} = \frac{1}{\sqrt{2}} (\begin{matrix}1_{b} & i · 1_{b} \\ i · 1_{b} & 1_{b} \end{matrix}).

Define a map B′_{b} : L^{2}(M_{b}) → F_{b,b} by

B′_{b} f(\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = e^{\frac{π}{2} |z|^{2}}V_{b}(f, ϕ_{0})(c′_{b} (\begin{matrix}−z_{1} \\ −z_{2} \end{matrix}))

where we write |z|^{2} = |z_{1}|^{2}+|z_{2}|^{2}. Then by calculations similar to those in Lemmas 7.3 and 7.4, we obtain:

Proposition 12.2. The transform B′_{b} is an isometry between L^{2}(M_{b}) and F_{b,b} which satisfies

B′_{b} ρ′_{b}(c′_{b} (\begin{matrix}w_{1} \\ w_{2} \end{matrix})) f (\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = e^{− \frac{π}{2} |w|^{2}}e^{π(z_{1}|w_{1})}e^{π(w_{2}|z_{2})}B′_{b} f(\begin{matrix}z_{1} \\ z_{2} \end{matrix}) = ρ′_{0} (\begin{matrix}w_{1} \\ w_{2} \end{matrix}) B′_{b} f (\begin{matrix}z_{1} \\ z_{2} \end{matrix})

where ρ′_{0} (\begin{matrix}w_{1} \\ w_{2} \end{matrix}) = ρ_{0} ( \begin{matrix}w_{1} \\ 0 \\ w_{2} \\ 0 \end{matrix} ).

Definition 12.3. The generalized partial Bargmann transform B_{b}f of a function f ∈ L^{2}(M_{b}, F) is given by

B_{b}f( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} ) = (B′_{b} f( \begin{matrix}· \\ v_{1} \\ · \\ v_{2} \end{matrix} )) (\begin{matrix}z_{1} \\ z_{2} \end{matrix})

= e^{−πi(z_{1}|z_{2})} ∫_{M_{b}} f(\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix})e^{πi\sqrt{2}(z_{1}|a)}e^{π\sqrt{2}(a|z_{2})}e^{−π(a|a)}da.

Corollary 12.4. The transform B_{b} is an isometry between L^{2}(M_{b}, F_{p−b,q−b}) and F_{p,q} which satisfies

B_{b} ρ_{b}(c_{b} ( \begin{matrix}w_{1} \\ v_{1} \\ w_{2} \\ v_{2} \end{matrix} ))f = ρ_{0} ( \begin{matrix}w_{1} \\ v_{1} \\ w_{2} \\ v_{0} \end{matrix} ) B_{b}f.

By a calculation similar to Lemma 7.5, we obtain the inverse partial Bargmann transforms B_{b}^{−1}:

Lemma 12.5. The generalized partial inverse Bargmann transform B_{b}^{−1} mapping F_{p,q} to L^{2}(M_{b}, F_{p−b,q−b}) is given by

B_{b}^{−1}g(\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) = e^{− \frac{π}{2} (a|a)} ∫_{M_{b}} e^{πi(z_{2}|z_{1})}e^{−\sqrt{2}πi(a|z_{1})}e^{\sqrt{2}π(z_{2}|a)}g( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} )e^{−π|z|^{2}}dz

for g ∈ F_{p,q}, dz = dz_{1} dz_{2}, provided that this integral converges absolutely.

13. U(p, q)-Intertwining Maps for Mixed Models

As in Section 8, we have an action of the group G^{c_{b}} = c_{b}Gc_{b}^{−1} on M_{b} × M_{p−b} × M_{b} × M_{q−b} by matrix multiplication and hence an action on H_{b}. We can define a representation ω : G^{c_{b}} → U(L^{2}(M_{b}, F)) by ω(c_{b}gc_{b}^{−1}) = B_{b}^{−1} ∘ω(g)∘B_{b} for g ∈ G. By Corollary 12.4 and the relation ω(g)ρ_{0}(u; t) = ρ_{0}(g · u; t)ω(g) for g ∈ G and (u; t) ∈ H_{0}, we have

ω(g′)ρ_{b}(w; t) = ρ(g′ · w; t)ω(g′)

for g′ = c_{b}gc_{b}^{−1} and (w, t) ∈ H_{b}.

Remark 13.1. This realization of ω acting on L^{2}(M_{b}, F) is a mixed model of the oscillator representation of G^{c_{b}}. For explicit formulas giving the action of ω for a generating set of elements of G^{c_{b}} in this model, see [15] or [16].

For λ ∈ { \atop U(k)}, let L^{2}(M_{b}, F; λ) be the subspace of functions f from

M_{b} × M_{p−b} × M_{q−b} to V_{λ} satisfying (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) ↦ (f (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) , ϕ) ∈ L^{2}(M_{b}, F) for every

ϕ ∈ V′_{λ} and f(\begin{matrix}ac \\ v_{1}c \\ v_{2}c \end{matrix}) = λ(c)^{−1}f (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) for all c ∈ U(k). The partial Bargmann

transform B_{b} maps L^{2}(M_{b}, F; λ) → F(M_{p+q}; λ), so we can replace ω with ω_{λ} above.

We now construct a G^{c_{b}} intertwining map between L^{2}(M_{b}, F; λ) and a space of V_{τ} -valued functions on S_{b}, where V_{τ} = H_{p,q}(λ).

Let W = ( \begin{matrix}W_{11} & W_{12} \\ W_{21} & W_{22} \end{matrix} ) ∈ S_{b}.

Proposition 13.2. For W ∈ S_{b}, define a function on M_{b} × M_{p−b} × M_{q−b} by

q′_{W} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) = e^{−πi Tr(aa^{∗}W_{11}^{∗})}e^{−πi Tr(av_{2}^{∗}W_{12}^{∗})}e^{π Tr(v_{1}a^{∗}W_{21}^{∗})}e^{π Tr(v_{1}v_{2}^{∗}W_{22}^{∗})}.

Then q′_{W} ∈ L^{2}(M_{b}, F).

Proof. By a calculation similar to the one found in the proof of Proposition 8.2, one can show that q′_{W} = αB_{b}^{−1}q_{T}, where T = c_{b}^{−1}(W) ∈ D and α a constant.

Here we note that |q_{T} ( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} )| < e^{Cπ(|z|^{2}+|v|^{2})}, C a constant, so the integral defining

B_{b}^{−1}q_{T} converges absolutely and gives a function in L^{2}(M_{b}, F).

Theorem 13.3. Let h ∈ H_{p,q}(λ). Then for T ∈ D, W = c_{b}(T) ∈ S_{b},

the function f_{W} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) = q′_{W} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) h ((\begin{matrix}a \\ iv_{1} \end{matrix}) , (\begin{matrix}a \\ v_{2} \end{matrix})) ∈ L^{2}(M_{b}, F_{p−b,q−b}; λ). We

have:

(B_{b}f_{W}) ( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} ) = C q_{T} ( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} ) (J(c_{b}, T)^{∗}h) ((\begin{matrix}iz_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}z_{2} \\ v_{2} \end{matrix})), C a constant.

Proof. It will be convenient to work with holomorphic harmonic polynomials, that is, holomorphic V_{λ}-valued polynomials P on M_{p+q,k}(ℂ) such that

Δ_{i,j}h(z) = 0, 1 ≤ i ≤ p, 1 ≤ j ≤ q where Δ_{i,j} = ∑_{ν=1}^{k} \frac{∂^{2}}{∂z_{iν}∂z_{jν}} .

We will call the space of all such polynomials H_{p+q}(λ). Note that we have a map

H_{p,q}(λ) → H_{p+q}(λ), h ↦ P_{h}

given by P_{h}(z_{1}, z_{2}) = h(z_{1}, \overline{z}_{2}), z_{1} ∈ M_{p,k}(ℂ), z_{2} ∈ M_{q,k}(ℂ). The representation τ_{0} of GL(p, ℂ) × GL(q, ℂ) on H_{p,q}(λ) corresponds to a representation on H_{p+q}(λ), which we also call τ_{0}, given by τ_{0}(g_{1}, g_{2})P (z_{1}, z_{2}) = P (g_{1}^{−1}z_{1},^{t}g_{2}z_{2}). The following Lemma is adapted from [14, Lemma 4.2]:

Lemma 13.4. Let Z = X + iY ∈ M_{b,b}(ℂ) with X = X^{∗}, Y = Y^{∗}, Y ≫ 0. Let P ∈ H_{p+q}(λ), and a, a_{1}, a_{2} ∈ M_{b,k}(ℂ). Then:

∫_{M_{b}} e^{−πi Tr a^{∗}a_{1}}e^{−πi Tr aa_{2}^{∗}}e^{−πi Tr aa^{∗}Z^{∗}}P ((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2}))da

= (i)^{bk}e^{πi Tr a_{1}a_{2}^{∗}Z^{∗}^{−1}} det(−Z^{∗})^{−k} ·

τ_{0} ((\begin{matrix}−Z^{∗} & 0 \\ 0 & 1 \end{matrix}) , (\begin{matrix}−Z^{∗}^{−1} & 0 \\ 0 & 1 \end{matrix}))P ((\begin{matrix}a_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}\overline{a} & 2 \\ \overline{v} & 2 \end{matrix})).

Proof. As both sides are holomorphic functions of Z^{∗}, it is enough to verify the equation for Z = iA^{2}, where A is a positive definite hermitian matrix. By the change of variables a ↦ A^{−1}a and the relation

P ((\begin{matrix}Aa \\ iv_{1} \end{matrix}) , ( \overline{Aa} ; \overline{v}_{2} )) = τ_{0} ((\begin{matrix}A^{−1} & 0 \\ 0 & 1 \end{matrix}) , (\begin{matrix}A^{∗} & 0 \\ 0 & 1 \end{matrix})) P ((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2}))

it is enough to prove the result for Z = i. Thus we need to prove:

∫_{M_{b}} e^{−πi Tr a^{∗}a_{1}}e^{−πi Tr aa_{2}^{∗}}e^{−π Tr aa^{∗}}P ((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2}))da  (23)
= e^{−π Tr a_{1}a_{2}^{∗}}τ_{0} ((\begin{matrix}i & 0 \\ 0 & 1 \end{matrix}) , (\begin{matrix}−i & 0 \\ 0 & 1 \end{matrix}))P ((\begin{matrix}a_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}\overline{a} & 2 \\ \overline{v} & 2 \end{matrix})).

We think of M_{b,k}(ℂ) as a real vector space embedded in its complexification M_{b,k}(ℂ) × M_{b,k}(ℂ) by a ↦ (a, \overline{a}). Then it follows from the mean value theorem for harmonic polynomials (Lemma 8.5) that

∫_{M_{b}} e^{−πi Tr a^{∗}a′}e^{−πi Tr aa′^{∗}}e^{−π Tr aa^{∗}}P ((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2}))da  (24)
= e^{−π Tr a′a′^{∗}}P ((\begin{matrix}−ia′ \\ iv_{1} \end{matrix}) , (\begin{matrix}−i\overline{a}′ \\ \overline{v} & 2 \end{matrix})).

Here we are using the fact that a ↦ P ((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2})) is a harmonic polynomial

on M_{b,k}(ℂ) (viewed as a real vector space).

Now we view (24) as an equation between real analytic functions of a′. By analytic continuation, it follows that for any (a_{1}, a_{2}) ∈ M_{b,k}(ℂ) × M_{b,k}(ℂ), we have:

∫_{M_{b}} e^{−πi Tr a^{∗}a_{1}}e^{−πi Tr aa_{2}^{∗}}e^{−π Tr aa^{∗}}P ((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2}))da  (25)
= e^{−π Tr a_{1}a_{2}^{∗}}P ((\begin{matrix}−ia_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}−i\overline{a}_{2} \\ \overline{v} & 2 \end{matrix})).

We observe that (25) is equivalent to (23), which proves the Lemma.

Returning to the theorem, we have

(B_{b}f_{W}) ( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} ) = e^{−πi Tr z_{1}z_{2}^{∗}}e^{π Tr v_{1}v_{2}^{∗}W_{22}^{∗}} ∫_{M_{b}} e^{πi\sqrt{2} Tr z_{1}a^{∗}}e^{π\sqrt{2} Tr az_{2}^{∗}}e^{−π Tr aa^{∗}} ·

e^{−πi Tr(aa^{∗}W_{11}^{∗})}e^{−πi Tr(av_{2}^{∗}W_{12}^{∗})}e^{π Tr(v_{1}a^{∗}W_{21}^{∗})}P_{h}((\begin{matrix}a \\ iv_{1} \end{matrix}) , ( \overline{a} ; \overline{v}_{2}))da

By Lemma 11.1, we have W_{11} = X + iY , with X, Y ∈ Herm_{b}(ℂ) with Y ≫ 0, and thus the previous Lemma applies. Taking a_{1} = −\sqrt{2}z_{1} + iW_{21}^{∗}v_{1}, a_{2}^{∗} = \sqrt{2}iz_{2}^{∗} + v_{2}^{∗}W_{12}^{∗}, Z^{∗} = W_{11}^{∗} − i, the Lemma gives:

(B_{b}f_{W}) ( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} ) = e^{−πi Tr z_{1}z_{2}^{∗}}e^{π Tr v_{1}v_{2}^{∗}W_{22}^{∗}} · (i)^{bk}e^{πi Tr a_{1}a_{2}^{∗}Z^{∗}^{−1}} det(−Z^{∗})^{−k}

τ_{0} ((\begin{matrix}−Z^{∗} & 0 \\ 0 & 1 \end{matrix}) , (\begin{matrix}−Z^{∗}^{−1} & 0 \\ 0 & 1 \end{matrix}))P_{h}((\begin{matrix}a_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}\overline{a} & 2 \\ \overline{v} & 2 \end{matrix}))

where we calculate

e^{−πi Tr z_{1}z_{2}^{∗}}e^{π Tr v_{1}v_{2}^{∗}W_{22}^{∗}} · e^{πi Tr a_{1}a_{2}^{∗}Z^{∗}^{−1}}
= e^{π Tr z_{1}z_{2}^{∗}(−i)(W_{11}^{∗}−i)^{−1}(W_{11}^{∗}+i)}e^{π Tr v_{1}z_{2}^{∗}(−\sqrt{2}i)(W_{11}^{∗}−i)^{−1}W_{21}^{∗}} ·
e^{π Tr z_{1}v_{2}^{∗}(−\sqrt{2}i)W_{12}^{∗}(W_{11}^{∗}−i)^{−1}}e^{π Tr v_{1}v_{2}^{∗}(W_{22}^{∗}−W_{12}^{∗}(W_{11}^{∗}−i)^{−1}W_{21}^{∗})}
= e^{π Tr(\begin{matrix}z_{1} \\ v_{1} \end{matrix} )( z_{2}^{∗} v_{2}^{∗})( \begin{matrix}T_{11}^{∗} & T_{21}^{∗} \\ T_{12}^{∗} & T_{22}^{∗} \end{matrix} )} = q_{T} ( \begin{matrix}z_{1} \\ v_{1} \\ z_{2} \\ v_{2} \end{matrix} )

with ( \begin{matrix}T_{11} & T_{12} \\ T_{21} & T_{22} \end{matrix} ) = c_{b}^{−1} ( \begin{matrix}W_{11} & W_{12} \\ W_{21} & W_{22} \end{matrix} ) given by formula (21) and

(i)^{bk} det(−Z^{∗})^{−k}τ_{0} ((\begin{matrix}−Z^{∗} & 0 \\ 0 & 1 \end{matrix}) , (\begin{matrix}−Z^{∗}^{−1} & 0 \\ 0 & 1 \end{matrix}))P_{h}((\begin{matrix}a_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}\overline{a} & 2 \\ \overline{v} & 2 \end{matrix}))

= (i)^{bk} det(−Z^{∗})^{−k}h((\begin{matrix}−Z^{∗}^{−1} & 0 \\ 0 & 1 \end{matrix}) (\begin{matrix}a_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}−Z^{−1} & 0 \\ 0 & 1 \end{matrix}) (\begin{matrix}a_{2} \\ v_{2} \end{matrix}))
= (i)^{bk} det(−W_{11}^{∗} + i)^{−k} ·

h(( \begin{matrix}\sqrt{2}(1+iW_{11}^{∗})^{−1} & −(W_{11}^{∗}−i)^{−1}W_{21}^{∗} \\ 0 & 1 \end{matrix} )(\begin{matrix}ia_{1} \\ iv_{1} \end{matrix}),( \begin{matrix}\sqrt{2}(1−iW_{11})^{−1}W_{12} & −(W_{11}+i)^{−1}W_{12} \\ 0 & 1 \end{matrix} )(\begin{matrix}a_{2} \\ v_{2} \end{matrix}))

= C J(c_{b}^{−1}, W)^{∗}^{−1}h ((\begin{matrix}ia_{1} \\ iv_{1} \end{matrix}) , (\begin{matrix}a_{2} \\ v_{2} \end{matrix}))

where C is a constant. As J(c_{b}^{−1}, W)^{−1} = J(c_{b}, T) for T = c_{b}^{−1}(W), this proves the theorem.

Now we can proceed exactly as in the (Sp(n, ℝ), O(k)) example in Section 8. First note that the irreducibility of τ and hence of τ_{0} implies that h(iw_{1}, w_{2}) = β_{1} h(w_{1}, w_{2}), (w_{1}, w_{2}) ∈ M_{p,k}(ℂ) × M_{q,k}(ℂ), for some constant β_{1}. Let I_{b} denote the map from H_{p,q}(λ) to the space of harmonic polynomials in L^{2}(M_{b}, F_{p−b,q−b}; λ) given by

I_{b} h (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) = h ((\begin{matrix}a \\ iv_{1} \end{matrix}) , (\begin{matrix}a \\ v_{2} \end{matrix}))

for h ∈ H_{p,q}(λ). By the Theorem, applying the inverse Bargmann transform to (10.6) and canceling constants on both sides leads to:

ω_{λ}(g′)q′_{c_{b}(T)}I_{b}(h) = q′_{g′^{−1}(c(T))}I_{b}(J(c_{b}, g(T))^{∗}^{−1}J(g, T)^{∗}^{−1}J(c_{b}, T)^{∗}h) for T ∈ D, g ∈ G, g′ = c_{b} g c_{b}^{−1} ∈ G^{c_{b}}. Then for h_{1}, h_{2} ∈ H_{p,q}(λ) and S, T ∈ D, we have

⟨q′_{c_{b}(T)} ∘ I_{b}(h_{1})|q′_{c_{b}(S)} ∘ I_{b}(h_{2})⟩_{L^{2}(M_{b},F;λ)}
= ∥β∥^{2}⟨J(c_{b}, S)Q(S, T)J(c_{b}, T)^{∗}h_{1}|h_{2}⟩_{H_{p,q}(λ)}, β a constant

and we can define a positive definite kernel function Q′ : S_{b} × S_{b} → Aut(V_{τ}) by Q′(c_{b}(S), c_{b}(T)) = ∥β∥^{2}J(c_{b}, S)Q(S, T)J(c_{b}, T)^{∗} which satisfies (17). It follows that G^{c_{b}} acts on H(S_{b}, τ) by the strongly continuous unitary representation T defined by (18). Finally, the map q′_{c_{b}(T)}I_{b}(h) ↦ Q′(·, c_{b}(T))h extends to an intertwining map Ξ_{λ} between (ω_{λ}, L^{2}(M_{b}, F; λ)) and (T, H(S_{b}, V_{τ})) such that ⟨Ξ_{λ}f(W)|h⟩_{H_{p,q}(λ)} = ⟨f|q′_{W} I_{b}(h)⟩_{L^{2}(M_{b},F;λ)}. This map can be written as follows:

Theorem 13.5. Let f ∈ L^{2}(M_{b}, F; λ) and let (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) ↦ I_{λ} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) be the poly-

nomial function on M_{b} × M_{p−b} × M_{q−b} with values in Hom(V_{τ}, V_{λ}) defined by

I_{λ} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) h = h ((\begin{matrix}a \\ iv_{1} \end{matrix}) , (\begin{matrix}a \\ v_{2} \end{matrix}))

for h ∈ V_{τ} = H_{p,q}(λ). Then:

Ξ_{λ}f(W) = ∫_{M_{b}} ∫_{M_{p−b+q−b}} \overline{q′_{W}} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) I_{λ} (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix})^{∗} f (\begin{matrix}a \\ v_{1} \\ v_{2} \end{matrix}) e^{−π(v|v)}dv dx

The proof is as in Proposition 5.7.

Remark 13.6. When b = q, the intertwining map given above is exactly the one found in [14].

References

[1] Davidson, M. G., The Harmonic Representation of U(p, q) and its Connection with the Generalized Unit Disk, Pacific J. Math. 129 (1987), 33–55.

[2] Davidson, M. G., Enright, T.J. and Stanke, R., Differential Operators and Highest Weight Representations, Memoirs of the Amer. Math. Soc. 94 (1991).

[3] Davidson, M. G. and Fabec, R. C., Geometric Realizations for Highest Weight Representations, Contemporary Mathematics 191 (1995), 13–31.

[4] Ding, H., Hankel Transforms and Holomorphic Discrete Series, Contemporary Mathematics, 191 (1995), 33–40.

[5] Ding, H. and Gross, K. I., Operator-valued Bessel functions on Jordan Algebras, J. reine angew. Math. 435 (1993), 157–196.

[6] Folland, G. B., “Harmonic Analysis in Phase Space”, Ann. of Math. Stud. 122, Princeton University Press, 1989.

[7] Gelbart, S., Holomorphic Discrete Series for the Real Symplectic Group, Inventiones Math. 19 (1973), 49–58.

[8] Gross, K. I. and Kunze, R. A., Fourier Bessel Transforms and Holomorphic Discrete Series, in “Proceedings of the Maryland Conference on Harmonic Analysis,” 79–122, Lecture Notes in Mathematics 266, Springer, 1977.

[9] Helgason, S., “Differential Geometry, Lie Groups and Symmetric Spaces”, Academic Press, New York, 1978.

[10] Herb, R. and Wolf, J. A., Wave packets for the relative discreteseries. I. The holomorphic case, J. Funct Anal. 73 (1987), 1–37.

[11] Howe, R., Notes on the Oscillator Representation, Unpublished notes.
[12] —, L^{2} duality for stable reductive dual pairs, Unpublished notes.

[13] Kunze, R. A., Positive Definite Operator-valued Kernels and Unitary Representations, in “Proceedings of the Conference on Functional Analysis at Irvine, California”, 235–247,Thomson Book Company, 1966.

[14] Kashiwara, K. and Vergne, M., On the Segal-Shale-Weil Representations and Harmonic Polynomials, Invent. Math. 44 (1978), 1–47.

[15] Leslie, C. S., “Geometric Construction of Intertwining Maps for Mixed Models of Compact Dual Pairs”, Ph.D. Dissertation, University of California at Berkeley, 1998.

[16] Li, J.-S., Singular unitary representations of classical groups, Invent. Math. 97 (1989), 237–255.

[17] Piatetsky-Chapiro (Piatetski-Shapiro), I. I., “Géométrie des domaines classiques et théorie des fonctions automorphes”, Dunod, Paris, 1966.

[18] Prasad, D., Weil Representation, Howe Duality, and the Theta Correspondence, CRM Proceedings and Lecture Notes 1 (1993), 105–127.

[19] Satake, I., “Algebraic Structures of Symmetric Domains”, Iwanami-Shoten and Princeton University Press, Princeton, New Jersey, 1980.

[20] Tirao, J. A., Square integrable representation of semisimple Lie groups, Trans. Amer. Math. Soc. 190 (1974), 57–75.

[21] Wallach, N. R., The Analytic Continuation of the Discrete Series. I, Trans. Amer. Math. Soc 251 (1979), 1–17.

[22] Wolf, J. A., Fine Structure of Hermitian Symmetric Spaces, in “Symmetric Spaces: Short Courses presented at Washington University”, 271–357, Marcel Dekker, 1972.

[23] Wolf, J. A. and Korányi, A, Generalized Cayley Transformations of Bounded Symmetric Domains, Amer. J. Math. 87 (1965), 899–939.

C. S. Leslie Department of Computer Science Columbia University 1214 Amsterdam Avenue, MC: 0401 New York, NY 10027-7003, USA cleslie@cs.columbia.edu

Received September 9, 2000 and in final form January 29, 2001