Journal of Lie Theory Volume 12 (2002) 245–257 \begin{matrix}c \\ ◯ \end{matrix} 2002 Heldermann Verlag

An Invariant Symmetric Non-selfadjoint Differential Operator

Erik G. F. Thomas

Communicated by K.-H. Neeb

Abstract. Let D be a symmetric left invariant differential operator on a unimodular Lie group G of type I. Then we show that D is essentially self– adjoint if and only if for almost all π ∈ Ĝ, with respect to the Plancherel measure, the operator π(D) is essentially self–adjoint. This, in particular, allows one to exhibit a left invariant symmetric differential operator on the Heisenberg group, which is not essentially self–adjoint.

Introduction

Let X = G/H be a homogeneous space, having an invariant measure. If D is an invariant differential operator on X which is symmetric, it is often important to know whether D, with domain the space of test functions D(X) = C_{c}^{∞}(X), is essentially self–adjoint in L^{2}(X).

The simplest positive result in this regard, involving an individual operator, is perhaps the following: if τ is the quasi–regular representation of G on L^{2}(X), and D is a symmetric element in the centre of the universal enveloping algebra, the operator τ(D) is an invariant differential operator on X, which, by a theorem of I.E. Segal [16], is essentially self–adjoint, at least on the Gårding domain. A result of E. Nelson and W.F. Stinespring shows that τ(D) is also essentially self–adjoint on the (smaller) domain D(X) (see [12] or the addendum to §1 below). In general not all invariant differential operators on X are obtained in this way however.

Also, several types of homogeneous space are known with the property that every symmetric invariant differential operator on it is essentially self–adjoint. For instance, every compact homogeneous space (having an invariant measure) has this property. As another example we mention the hyperbolic spaces U(p, q; 𝔽)/U(1; 𝔽)× U(p−1, q; 𝔽), 𝔽 = ℝ, ℂ or ℍ (cf. [6] and [17] theorem C, example b). In particular, this justifies the assertion in Lemma 9 of [14]. E.P. van den Ban has shown that for every semi–simple symmetric pair (G, σ, H), G/H has this property [1]. This also includes cases where (G, H) is not a generalised Gelfand pair [5].

Perhaps because of the abundance of these results no example was known (to the author) of a homogeneous space and a symmetric invariant differential

ISSN 0949–5932 / $2.50 \begin{matrix}c \\ ◯ \end{matrix} Heldermann Verlag operator on it, which is not essentially self–adjoint. On the other hand, several examples are known of a Lie group G, an irreducible representation π of G, and a symmetric element D in the universal enveloping algebra, such that π(D) is not essentially self–adjoint. The best known example of this is probably the one due to J. von Neumann (unpublished cf. [12]) where G is the Heisenberg group.

We have proved the following theorem (corollary to Theorem 1.7):

Selfadjointness Theorem Let G be a unimodular Lie group of type I. Let D be a symmetric element of the universal enveloping algebra. Then D, viewed as a left invariant differential operator on G, with domain D(G), is essentially self–adjoint if and only if π(D) is essentially self–adjoint for almost all irreducible π, with respect to the Plancherel measure.

This then allows one to convert von Neumann’s example into an example of a symmetric left invariant differential operator on the Heisenberg group, which is not essentially self–adjoint.

We have taken the opportunity to state some related results, such as Theorem 1.8, which connects strong commutation of two operators D_{1} and D_{2}, with strong commutation of the operators π(D_{1}) and π(D_{2}), but the reader mainly interested in the counter–example could read as far as half way through the proof of proposition 1.4, and then turn directly to the third example in paragraph 2.

In the addendum to paragraph 1 we show that, for an arbitrary unitary representation U, the operator U(D), on the C^{∞}–vectors, and its restriction to the space of analytic vectors, always have the same closure.

1. Generalities

Let G be a unimodular Lie group which eventually we shall assume to be of type I. We denote D(G) the space of C^{∞} functions with compact support, and D′(G) the space of distributions on G. Having chosen a Haar measure on G we identify the locally integrable functions with distributions as usual. Thus we have the inclusions:

D(G) ⊂→ L^{2}(G) ⊂→ D′(G)

More generally, let

H ⊂→ j D′(G)

be any Hilbert subspace of D′(G), i.e. a linear subspace equipped with a Hilbert space inner–product, such that the inclusion map, j, is continuous. For any f ∈ D′(G) and ϕ ∈ D(G) we use the notation ⟨f, ϕ⟩ = f(\overline{ϕ}). Let j^{∗} : D ⟶ H be the adjoint of j defined by the equation:

(f, j^{∗}ϕ) = ⟨jf, ϕ⟩  (1)

where the left–hand side stands for the inner product in H. (Note that in this equation j is usually omitted from the expression on the right–hand side).

The reproducing operator for H, analogous to the orthogonal projection on a closed subspace of a Hilbert space, is by definition the operator H = jj^{∗}. It is a continuous linear operator from D to D′ which completely characterises H. (see [15] or the summary in [17]).

Every Hilbert subspace H ⊂→ D′ possesses a privileged dense subspace, namely j^{∗}(D), or, somewhat incorrectly, H(D). We shall denote it by H_{0}. (In the particular case where H is a dense subspace of D′, j^{∗} is injective, and one obtains a Gelfand triplet).

Let R denote the right regular representation in D′(G). A Hilbert subspace H is said to be right invariant if R(g)H = H and the restriction of each operator R(g) to H is unitary. This happens if and only if R(g)H = HR(g) for all g, i.e. H intertwines the regular representations on D and D′. Equivalently, H is a convolution operator:

H(ϕ) = K ∗ ϕ  (2)

where K is some positive definite distribution on G. Let us denote R^{H} the restriction of R to the space H. It is a continuous unitary representation in H. To verify the continuity it is sufficient to check weak continuity on H_{0}. We have in fact: (j^{∗}ϕ, R^{H}(g)j^{∗}ψ) = (j^{∗}ϕ, j^{∗}R(g)ψ) = ⟨Hϕ, R(g)ψ⟩ which is continuous with respect to g. Let us note also that the subspace H_{0}, which is invariant under R^{H}, is composed of regular distributions, in fact of functions of class C^{∞}. This is a consequence of formula (2).

Similar considerations apply to left invariant spaces. A space which is both left and right invariant will be called bi–invariant. In the particular case where H = L^{2}(G), we have H(ϕ) = ϕ, and everything we have said is most familiar.

Let 𝔤 denote the Lie algebra of G, and U the universal enveloping algebra of its complexification. We identify U with the algebra of left invariant differential operators on G (U_{L} if any confusion should arise).

These differential operators are viewed as acting on D(G), on E(G), the space of all functions of class C^{∞}, and on D′(G). If D^{∗} denotes the formal adjoint of D, we have, G being unimodular,

⟨D^{∗}f, ϕ⟩ = ⟨f, Dϕ⟩  (3)

for all ϕ ∈ D(G) and f ∈ D′(G).

If U is any unitary representation of G, we denote U_{∞}(D) the operator corresponding to D ∈ U, acting on the space of C^{∞}–vectors for U. If H is a right invariant Hilbert subspace of D′(G), we denote H_{∞} the space of C^{∞}–vectors for R^{H}.

Proposition 1.1. Let H be a right invariant Hilbert subspace of D′(G), and let D ∈ U be a left invariant differential operator.

a. R_{∞}(D) is the restriction to H_{∞} of the operator D : D′(G) → D′(G), i.e.:

R_{∞}^{H}(D)f = Df ∀f ∈ H_{∞}

b. H_{0} ⊂ H_{∞}. If R_{0}^{H}(D) denotes the restriction of R_{∞}^{H}(D) to H_{0′} the operators R_{0}^{H}(D) and R_{∞}^{H}(D) have the same closure.

c. Let T denote either R_{∞}^{H}(D) or R_{0}^{H}(D). Then the domain of T^{∗} is

dom(T^{∗}) = {f ∈ H : D^{∗}f ∈ H}

and T^{∗}f = D^{∗}f for all f in this domain.

Proof. a. It is sufficient to prove this for X ∈ 𝔤, an arbitrary D ∈ U being a linear combination of products of such elements. Now we have R_{∞}^{H}(X)f = \frac{d}{dt}R(exp tX)f_{|t=0} in the space H, and so a fortiori in D′(G). But G being unimodular, we have X^{∗} = −X, and so it is easy to see by transposition, that the above expression yields Xf.

b. Since the reproducing operator intertwines the regular representations in D and D′, we have R^{H}(g)j^{∗}ϕ = j^{∗}R(g)ϕ. On the other hand, the map j^{∗} : D ⟶ H is a continuous linear operator and ϕ is a C^{∞}-vector for the regular representation in D. Thus j^{∗}ϕ is a C^{∞}-vector for R^{H}. Next we need to show that for any f ∈ H_{∞}, there exists f_{n} ∈ H_{0} with f_{n} ⟶ f and Df_{n} ⟶ Df in H. (The fact that R_{∞}^{H}(D) actually has a closure is well known, and besides an immediate consequence of a). Now it is known that H_{∞} is in fact equal to the Gårding domain, i.e. the linear span of the elements of the form R^{H}(ϕ)h, with h ∈ H and ϕ ∈ D(G) ([3] Theorem 3.3). Thus we may assume f = R^{H}(ϕ)h. Let h_{n} ∈ H_{0} tend to h in the space H. Then f_{n} = R^{H}(ϕ)h_{n} = h_{n} ∗ ϕ belongs to H_{0}, f_{n} converges to f, and Df_{n} = R^{H}(Dϕ)h_{n} converges to R^{H}(Dϕ)h = Df.

c. To prove this it will be useful to first note the following

R^{H}(D)j^{∗}ϕ = j^{∗}Dϕ  ∀ϕ ∈ D(G)  (4)

or equivalently

DH(ϕ) = H(Dϕ)  ∀ϕ ∈ D(G)  (5)

which is proved, as before, first for D = X by differentiation. Now, if f = j^{∗}ϕ, h belongs to H, and T denotes R_{0}^{H}(D), we have:

(h, T f) = (h, Df) = (h, j^{∗}(Dϕ)) = ⟨h, Dϕ⟩ = ⟨D^{∗}h, ϕ⟩

If D^{∗}h belongs to H, this equals (D^{∗}h, j^{∗}ϕ), and so we have

(h, T f) = (D^{∗}h, f)

for all f ∈ H_{0}, which implies h ∈ dom(T^{∗}), and T^{∗}h = D^{∗}h. Conversely, if h belongs to the domain of T^{∗}, the above equalities show that there exists a constant M such that,

|⟨D^{∗}h, ϕ⟩| ≤ M||j^{∗}ϕ|| ∀ϕ ∈ D(G)

Thus, by the Riesz–Fréchet representation theorem, there exists an element f ∈ H, such that ⟨D^{∗}h, ϕ⟩ = (f, j^{∗}ϕ) = ⟨f, ϕ⟩ for all ϕ ∈ D(G), which implies D^{∗}h = f ∈ H. Since by b. R_{∞}^{H}(D) and R_{0}^{H}(D) have the same adjoint, the proof is complete.

A particular consequence of proposition 1.1 is that, when D = D^{∗}, the operator R_{0}^{H}(D) is essentially self–adjoint, i.e. has self–adjoint closure, if and only if R_{∞}^{H}(D) is essentially self–adjoint. From now on we shall describe the essential self–adjointness of these operators simply by saying that R^{H}(D) is essentially self–adjoint. Similarly, if U is a unitary representation of G we say that U(D) is essentially self–adjoint if the operator U_{∞}(D), with domain the C^{∞}–vectors has this property. If π is the equivalence class of U we also describe this by saying that π(D) is essentially self–adjoint. Similarly π(ϕ) stands for U(ϕ), etc.

Now let Ĝ_{1} denote the set of equivalence classes π of irreducible unitary representations of G, such that, for each ϕ ∈ D(G), π(ϕ) is an operator of trace class. Then, if for π ∈ Ĝ_{1} we put χ_{π}(ϕ) = trace π(ϕ), χ_{π} is a central positive definite distribution which determines π, the character of π. If we now put

H_{π}(ϕ) = χ_{π} ∗ ϕ = ϕ ∗ χ_{π}  (6)

H_{π} is the reproducing operator of a minimal bi–invariant Hilbert subspace of D′(G), with the property that R^{H_{π}} is the d(π)–fold repetition of π, d(π) being the degree of π. If we topologize Ĝ_{1} by making the map π → χ_{π} a homeomorphism, Ĝ_{1} becomes a Suslin space whose Borel sets are Borel sets of Ĝ in the sense of Mackey [10]. Moreover, if G is a group of type I, which we shall assume from now on, the map π ⟶ χ_{π} is an admissible section for the set of extreme generators of the cone of central positive definite distributions on G, which is a lattice cone. Thus, there exists a unique measure dπ on Ĝ_{1}, the Plancherel measure, such that:

δ = ∫ χ_{π}dπ  (7)

Equivalently, (see [17] Theorem A), one has the direct integral decomposition:

L^{2}(G) = ∫ ⊕ ; Ĝ_{1} H_{π}dπ  (8)

More generally, if H is any bi–invariant Hilbert subspace of D′(G), there exists a unique measure m on Ĝ_{1} such that

H = ∫ ⊕ ; Ĝ_{1} H_{π}dm(π)  (9)

(for details regarding this approach to Plancherel measure see [10] and [17].)

Proposition 1.2. Let D = D^{∗} be a symmetric element in U, and let π belong to Ĝ_{1}. Then π(D) is essentially self–adjoint if and only if R^{H_{π}}(D) is essentially self–adjoint.

Proof. Recall that a densely defined symmetric operator T fails to be essentially self–adjoint if and only if at least one of the equations T^{∗}f = ±if admits a solution f ≠ 0. First assume that π(D) is not essentially self–adjoint. Let K be a closed minimal right invariant subspace of H_{π} (which exists because R^{H_{π}} is a factor representation of type I). Then R^{K} represents π, and so, by proposition 1.1, there exists a non zero solution of the equation Df = if (say) in the space K. But then f belongs to H_{π} and so for the same reason R^{H_{π}}(D) is not essentially self–adjoint. Conversely, assume R^{H_{π}}(D) = T is not essentially self–adjoint. Then there exists an element f ∈ H_{π} such that, for instance T^{∗}f = if, f ≠ 0. Also, H_{π} being the orthogonal direct sum of minimal right invariant closed sub–spaces, there exists such a space, K, such that the orthogonal projection P_{K}f of f on K is not zero. We shall prove that the orthogonal projection operator P_{K} commutes with T^{∗}. Then it will follow that T^{∗}P_{K}f = iP_{K}f, and so, by proposition 1.1, the equation Dk = ik has a non–zero solution in K. Thus R^{K}(D) is not essentially self–adjoint, which means that π(D) is not essentially self–adjoint. To prove that T^{∗} commutes with P_{K}, note that T^{∗} commutes with the operators of left translation in H_{π}. Thus the bounded operators B = (I + T^{∗∗}T^{∗})^{−1} and C = T^{∗}B commute with the left translations. Therefore, by the Godement–Segal commutativity theorem ([8], [10]) the operators B and C belong to the Von Neumann algebra R generated by the operators R^{H_{π}}(g). On the other hand, P_{K} commutes with right translations, and so belongs to the commutant of R. Thus P_{K} commutes with B and C, and so also with the operator T^{∗}, which can be recovered from B and C, i.e. we have P_{K}T^{∗} ⊂ T^{∗}P_{K}. Thus if T^{∗}f = if, k = P_{K}f belongs to the domain of T^{∗}, and T^{∗}k = ik as was to be shown.

Remark 1.3. We have obviously proved something slightly more precise than the statement of Proposition 1.2, namely, that the operators π(D) and R^{H_{π}}(D) have positive (resp. negative) deficiency indices differing from zero, simultaneously.

Proposition 1.4. Let H be any bi–invariant Hilbert subspace of D′(G), and let D ∈ U be any left invariant differential operator. Let K = {f ∈ H : Df = 0} and let K_{π} = {f ∈ H_{π} : Df = 0}. Then

K = ∫ ^{⊕} K_{π}dm(π)  (10)

(m being the measure defined by equation (9))

The proof depends on the following lemma which will be proved, on another occasion.

Lemma 1.5. Let E be a locally convex Hausdorff space such that its dual contains a countable subset separating the points of E. Let F be a closed linear subspace of E. Let Λ be a topological Hausdorff space equipped with a Radon measure m, and let (H_{λ})_{λ∈Λ} be an m-measurable family of Hilbert subspace of E. Also, let K_{λ} = H_{λ} ∩ F, with the Hilbert space structure induced from H_{λ}. Then (K_{λ})_{λ∈Λ} is a m–measurable family of Hilbert subspaces of E.

If we apply this lemma with E = D′(G) and F = {f ∈ D′(G) : Df = 0}, we see that the family (K_{π})_{π∈Ĝ_{1}} is m–measurable. Thus the integral on the right– hand side of (10) exists as Hilbert subspace of D′(G), and it is a closed subspace of H, which we denote as W. Every element f ∈ W has an expansion in D′(G):

f = ∫ f_{π}dm(π)  (11)

where (f_{π})_{π∈Ĝ_{1}} is a square integrable field such that f_{π} ∈ K_{π}. Now, since D : D′(G) ⟶ D′(G) is a continuous linear operator equation (11) yields Df = ∫ Df_{π}dm(π) = 0, which proves the inclusion W ⊂ K. (This will be sufficient for the construction below of a non essentially self-adjoint left invariant differential operator).

To prove the opposite inclusion we need some further notations. By equation (9) every f ∈ H has a unique expansion as in (11), with a square integrable field (f_{π}) where f_{π} ∈ H_{π}. If M ⊂ Ĝ_{1} is a Borel subset, we put:

P_{M}f = ∫_{M} f_{π}dm(π)  (12)

Also, let L (resp. R) denote the Von Neumann algebra of operators in H generated by the left (resp. right) translations. Then it is known that L ∩ R, which by the Godement–Segal commutativity theorem is the centre of L and of R, is generated by the projections P_{M}, in fact (9) is the central decomposition of H. Actually, we shall only need the fact that the projections P_{M} commute with the operators R^{H}(g), which may be easily verified as follows: R(g) being continuous in D′(G) we have from (11): R(g)f = ∫ R(g)f_{π}dm(π), which may also be written:

R^{H}(g)f = ∫ R^{H_{π}}(g)f_{π}dm(π)  (13)

Now, since R^{H_{π}}(g) preserves the norm in H_{π}, the right–hand side of (13) is the integral of a square integrable field, and so equation (13) is the decomposition of R^{H}(g)f corresponding to (9). Thus we have, by definition of P_{M},

P_{M}R^{H}(g)f = ∫_{M} R^{H_{π}}(g)f_{π}dm(π) = R^{H}(g)P_{M}f

which shows that P_{M} commutes with the right translations in H, and so belongs to R′, the commutant of R.

Now let T = R_{0}^{H}(D^{∗}). Then by Proposition 1.1, we have

K = {f ∈ dom(T^{∗}) : T^{∗}f = 0} = Ker(T^{∗})

As in the proof of the previous proposition, we see that, since T^{∗} commutes with left translations, the corresponding operators B and C belong to L′ = R, and so commute with P_{M}, which implies that T^{∗} commutes with P_{M}, i.e. we have P_{M}T^{∗} ⊂ T^{∗}P_{M}. Thus, in particular, if f belongs to K = Ker(T^{∗}), P_{M}f belongs to K. Therefore we have:

DP_{M}f = ∫_{M} Df_{π}dm(π) = 0

for all Borel sets M ⊂ Ĝ_{1}. This implies Df_{π} = 0 m-almost everywhere, i.e. f_{π} ∈ K_{π} m–almost everywhere, which means that f belongs to W. The proof is complete.

Now for any bi–invariant Hilbert subspace H of D′(G) and left differential operator D, let us put:

H^{±} = {f ∈ H : Df = ±if}

Then we have the following corollary of proposition 1.4:

Corollary 1.6. Under the same hypotheses as in proposition 1.4 we have:

H^{±} = ∫ H_{π}^{±}dm(π)  (14)

m being the measure defined by equation (9).

Theorem 1.7. Let H be a bi–invariant Hilbert subspace of D′(G) and let m be the measure on Ĝ_{1} defined by equation (9). Then, if D is a symmetric left invariant differential operator on G, R^{H}(D) is essentially self-adjoint if and only if π(D) is essentially self-adjoint for m–almost all π ∈ Ĝ_{1}.

This is entirely clear from the preceding result once it is recognized that a. the space H^{±} in (14) is equal to the space (0) if and only if almost each space H_{π}^{±} equals (0), and b. π(D) fails to be essentially self–adjoint on a set of positive measure if and only if either H_{π}^{+} ≠ (0) on a set of positive measure, or H_{π}^{−} ≠ (0) on a set of positive measure.

If we let H = L^{2}(G), so that m is the Plancherel measure and H_{0} = D(G), we obtain the theorem stated in the introduction.

Let us mention some related results with only summary indication of proof.

Theorem 1.8. Under the same precondition as in Theorem 1.7, let D_{1} and D_{2} be left invariant symmetric differential operators such that R^{H}(D_{1}) and R^{H}(D_{2}) are essentially self–adjoint. Then R^{H}(D_{1}) and R^{H}(D_{2}) strongly commute if and only if π(D_{1}) and π(D_{2}) strongly commute for m–almost all π ∈ Ĝ_{1} This will be a consequence of the following two propositions:

Proposition 1.9. Let D be a symmetric left invariant differential operator such that R^{H}(D) is essentially self–adjoint. Let M_{0} be the set of elements π ∈ Ĝ_{1} such that π(D) is essentially self-adjoint. Let E be the spectral measure corresponding to the self-adjoint closure of R^{H}(D), and for π ∈ M_{0} let E_{π} be the spectral measure belonging to the closure of R^{H_{π}}(D). Then we have:

E(Δ)f = ∫_{M_{0}} E_{π}(Δ)f_{π}dm(π)  (15)

for every Borel subset Δ ⊂ ℝ and f ∈ H.

Proof. Let T be the closure of R^{H}(D) and E its spectral measure. Then, since T commutes with the projections P_{M}, the E(Δ) also commutes with the P_{M}. Hence there exist spectral measures E_{π} in the spaces H_{π} such that formula (15) is valid (M_{0} = {π : H_{π}^{±} = (0)} is a measurable subset of Ĝ_{1} whose complement has measure 0 by Theorem 1.7). Let T_{π} = ∫ λE_{π}(dλ) be the corresponding self– adjoint operator. It can then be proved that for almost all π, T_{π} is equal to the closure of R^{H_{π}}(D). We only indicate the principle of the proof. Let G_{π} be the graph of T_{π}, G_{M} the graph of T P_{M} = P_{M}T, and let G be the graph in D′(G) × D′(G) of the operator D. Then we show that (G_{π})_{π∈M_{0}} is a measurable family of Hilbert subspaces of D′ × D′, and that G_{M} = ∫_{M}^{⊕} G_{π}dm(π) for all M. Now we know that G_{M} is contained in G, a closed subspace of D′×D′; this implies G_{π} ⊂ G for almost all π. But for those π, T_{π} is a restriction of the adjoint, or closure of R^{H_{π}}(D), and so being maximal symmetric, T_{π} equals this closure.

Remark 1.10. The relation between the graphs mentioned above, by projection on the first space, gives the following relation between the domains of the operator T and T_{π}, viewed, with their graph norms, as Hilbert subspaces of D′(G):

D_{T} = ∫_{M_{0}}^{⊕} D_{T_{π}}dm(π)  (16)

Next consider some abstract Hilbert space K_{π} in which the representation π, or rather a member of π, takes place, and let \overline{K}_{π} and \overline{π} denote respectively the conjugate space and representation. Then there exists an isomorphism:

Φ : \overline{K}_{π} ˆ⊗_{2}K_{π} ⟶ H_{π}

which transforms \overline{π} ⊗ π into the double representation L^{H_{π}}R^{H_{π}}.

Proposition 1.11. With the same conventions as in proposition 1.9, let, for π ∈ M_{0}, F_{π} be the spectral measure in K_{π} corresponding to the closure of π(D), and let I_{π} be the identity in \overline{K}_{π}. Then we have:

E_{π}(Δ) = Φ(I_{π} ⊗ F_{π}(Δ))Φ^{−1}  (17)

for all Borel sets Δ ⊂ ℝ.

Proof. Choose an orthonormal basis in the space \overline{K}_{π}. Then the tensor product becomes a direct sum of copies of K_{π}, which is transformed by Φ into a direct sum:

H_{π} = ∑_{k}^{⊕}H_{π}^{k}

such that each space H_{π}^{k} is minimal right invariant with R^{H_{π}^{k}} ∈ π. Let F^{k} denote the spectral measure in H_{π}^{k} corresponding to F. Then we should show that E_{π}(Δ)f = ∑_{k} F^{k}(Δ)f_{k}, f_{k} being the orthogonal projection of f on the space H_{π}^{k}.

We shall do this, and simplify the notation by dropping the index π throughout the remainder of the proof. Let E′(Δ)f = ∑_{k} F^{k}(Δ)f_{k}. Then E′ is a spectral

measure in the space H = H_{π}. Let T = ∫ λ E′(dλ) and let T_{k} = ∫ λF^{k}(dλ). Then T_{k} is equal to the closure of R^{H_{π}^{k}}(D). The domain D_{T} of T is composed of the elements f ∈ H, such that for all k, f_{k} belongs to the domain of T_{k}, and ∑_{k} ∥T_{k}f_{k}∥^{2} < +∞. Moreover, we then have T f = ∑_{k} T_{k}f_{k}. Let j and j_{k} denote

the inclusions of H, respectively H^{k}, in D′(G). Then, if f = j^{∗}ϕ belongs to H_{0}, f_{k} = j_{k}^{∗}ϕ ∈ H_{0}^{k} ⊂ D_{T_{k}}, and T_{k}f_{k} = j_{k}^{∗}Dϕ is the projection of j^{∗}Dϕ onto H^{k}. Thus f belongs to the domain of T and T f = j^{∗}Dϕ = Df. This means that T is an extension of the operator R^{H}(D), and so T equals the closure of this operator and E′ = E, as was to be shown.

Addendum to §1

The argument in the proof of 1.1, part b, only made use of the fact that H_{0} is invariant under the operators R^{H}(ϕ). In particular, it can be applied to the analytic vectors. Let us state and prove the result explicitly in this case:

Theorem 1.12. Let U be a unitary representation of G. Let D ∈ U, and let U_{ω}(D) be the restriction of U_{∞}(D) to the space of analytic vectors for U. Then U_{ω}(D) and U_{∞}(D) have the same closure.

Proof. Let H, H_{ω} and H_{∞} be the representation space and the subspaces of analytic and C^{∞}-vectors respectively. If f ∈ H_{ω} and v(x) = U(x)f is the corresponding analytic function, we have U(g)U(ϕ)f = ∫ ϕ(x)v(gx)dx, which is an analytic function of g by direct integration of the power series, at least if ϕ has its support in a sufficiently small coordinate patch. Thus H_{ω} is invariant under the operators U(ϕ). Now let f ∈ H_{∞}. By the theorem of Dixmier and Malliavin ([3] Theorem 3.3) we may assume f = U(ϕ)h for some h ∈ H, and ϕ ∈ D(G). Let h_{n} ∈ H_{ω} tend to h (Nelson’s theorem [7]). Then f_{n} = U(ϕ)h_{n} belongs to H_{ω}, tends to f, and U_{ω}(D)f_{n} = U(Dϕ)h_{n} tends to U(Dϕ)h = U_{∞}(D)f. Thus the closure of U_{ω}(D) extends U_{∞}(D), and so these two operators have the same closure.

An analogous assertion and argument is obviously valid for any subspace H_{0} of H_{∞} which is dense in H and invariant under the operators U(ϕ), ϕ ∈ D(G). For example, if, as in the introduction, U = τ is the quasi-regular representation in L^{2}(X), we may take H_{0} = D(X).

Thus the theorem of Dixmier and Malliavin according to which the Gårding domain actually coincides with the space of C^{∞}-vectors, entails some simplification in the situation as described by Nelson and Stinespring ([12] §1).

2. Examples on the Heisenberg group

Let G now be the group of upper-triangular matrices

[ \begin{matrix}1, & x, & z \\ 0, & 1, & y \\ 0, & 0, & 1 \end{matrix} ]

abbreviated (x, y, z). The Lie algebra 𝔤 is identified as usual with the strictly upper triangular matrices, and we put:

X = [ \begin{matrix}0, & 1, & 0 \\ 0, & 0, & 0 \\ 0, & 0, & 0 \end{matrix} ] Y = [ \begin{matrix}0, & 0, & 0 \\ 0, & 0, & 1 \\ 0, & 0, & 0 \end{matrix} ] Z = [ \begin{matrix}0, & 0, & 1 \\ 0, & 0, & 0 \\ 0, & 0, & 0 \end{matrix} ]

Let U^{r} be the unitary representation on L^{2}(ℝ) defined by

[ U^{r}(x, y, z)f ] (t) = exp ir (z + ty)f(t + x)

Then, to summarise the relevant facts, U^{r} is irreducible for every r ∈ ℝ_{∗} = ℝ \ {0}, Ĝ_{1} = Ĝ, the map r ⟶ χ_{r} which associates with r ∈ ℝ_{∗} the character of U^{r}, is a homeomorphism of ℝ_{∗} onto its image in D′(G), and one has the formula:

δ = ∫ χ_{r}|r|dr

Thus ℝ_{∗} may be identified with a (Borel) subset of Ĝ, and |r|dr with the Plancherel measure [2], [9]. It is known moreover, that the space of C^{∞}-vectors for U^{r} is precisely the Schwartz space S(ℝ) ([2] (1.4)). Since we have:

U_{∞}^{r}(X) = \frac{d}{dt} , U_{∞}^{r}(Y) = irt , U_{∞}^{r}(Z) = ir

we see that, as D describes U, U_{∞}^{r}(D) describes precisely the set of linear differential operators with polynomial coefficients. Now D(ℝ) being dense in S(ℝ) for the topology of S(ℝ), the closure of any operator U_{∞}^{r}(D) is equal to the closure of its restriction to D(ℝ). In particular, U_{∞}^{r}(D) and its restriction to D(ℝ) are simultaneously essentially self-adjoint. This self-adjointness will henceforth be described by saying that U^{r}(D) is essentially self-adjoint.

Let us now consider three examples:

Example 2.1. D = −X^{2} + Y^{4}. Then U^{r}(D) = −(\frac{d}{dt})^{2} + r^{4}t^{4}. It is well known that this operator is essentially self-adjoint for all r ∈ ℝ (see [4] XIII.6.15 or [13] X.28). Thus, by Theorem 1.7, D, with domain D(G), is essentially self-adjoint in L^{2}(G).

Example 2.2. (Harmonic oscillator). D = −X^{2}−Y^{2}. Then U^{r}(D) = −(\frac{d}{dt})^{2}+ r^{2}t^{2}. Here again, U^{r}(D) is essentially self-adjoint for all r ∈ ℝ, and so D is essentially self-adjoint.

We mention this example because of the (rather farfetched) possibility to draw the inverse conclusion. For instance, Z being central, D commutes with the elliptic operator X^{2} + Y^{2} + Z^{2}, and so D is essentially self-adjoint by the theorem of Nelson and Stinespring ([12] 2.4). By Theorem 1.7 it follows that U^{r}(D) is essentially self-adjoint for almost all r. The fact that there are no exceptions can be seen directly as follows: For r ≠ 0, let T_{r} be the unitary operator in L^{2}(ℝ) defined by [ T_{r}f ] (t) = |r|^{\frac{1}{4}} f(|r|^{\frac{1}{2}} t). Then we have:

U^{r}(D) = |r|T_{r}U^{1}(D)T_{r}^{−1}  (18)

i.e. up to a factor, the various operators U^{r}(D) are unitarily equivalent. Thus, if one is essentially self-adjoint, so are the others.

Remark The operator D = −X^{2} − Y^{2} has an absolutely continuous spectrum, in spite of the fact that for each r ∈ ℝ_{∗} , U^{r}(D) has a purely discrete spectrum.

Although this is probably known a proof is included to keep this paper selfcontained (an alternative suggested by the referee is to use [13] Thm XIII 85 and 86 and the method of [11]).

First note that, U^{r}(D) being strictly positive, its spectral measure, which we denote F_{r}, is concentrated on ℝ_{∗}^{+} = (0, +∞). Therefore, by Propositions 1.4 and 1.9, the spectral measure E of D is also concentrated on ℝ_{∗}^{+} (this can of course also be seen by checking that D = X^{∗}X + Y^{∗}Y is strictly positive, i.e. positive and injective). Now let Δ be a subset of ℝ_{∗}^{+} which has Lebesque measure equal to 0. Then, to show that E(Δ) = 0, it is sufficient, by proposition 1.4 and 1.9, to show that the set S of all r ∈ ℝ_{∗} such that F_{r}(Δ) ≠ 0 is negligible with respect to the Plancherel measure, i.e. a set of Lebesque measure zero. But by (18) we have

F_{r}(Δ) = T_{r}F_{1}(\frac{1}{|r|}Δ)T_{r}^{−1}  (19)

and so S = \{r ∈ ℝ_{∗} : F_{1}(\frac{1}{|r|}Δ) ≠ 0\}. Now let μ be a positive bounded measure on ℝ_{∗}^{+} having the same sets of measure zero as F_{1}, and let ρ be a strictly positive function on ℝ_{∗}^{+}, regarded as group, integrable with respect to the Haar measure \frac{dr}{r}. Then the convolution product ρ ∗ μ on ℝ_{∗}^{+} is absolutely continuous, and so we have 0 = ρ ∗ μ(Δ) = ∫ ρ(r)μ(\frac{1}{r}Δ)\frac{dr}{r}. Hence μ(\frac{1}{r}Δ) = 0 almost everywhere on ℝ_{∗}^{+}. Consequently, S is negligible as asserted.

A similar remark and argument applies to the operator in example 2.1.

Example 2.3. D = −X^{2} − Y^{4}. Then U^{r}(D) = −(\frac{d}{dt})^{2} − r^{4}t^{4}. The operator U^{r}(D) is not essentially self-adjoint for any r ≠ 0. In fact ‘both’ solutions of the equation U^{r}(D)f = if belong to L^{2}(ℝ).

This is a consequence of Wintner’s theorem ([13] X.9 or [4] XIII.6.20) and of Kodaira’s theorem relating the defect indices of the operator on [0, +∞), (−∞, 0] and (−∞, +∞) (see [4] XIII.2.26). Thus, by Theorem 1.7, the operator D, with domain D(G), is not essentially self-adjoint in L^{2}(G). With respect to the coordinates (x, y, z), −D has the expression:

(\frac{∂}{∂x})^{2} + (\frac{∂}{∂y} + x\frac{∂}{∂z})^{4}  (20)

By exchanging X and Y , or on the line \frac{d}{dt} and irt, one obtains a slightly simpler example of a left invariant differential operator on the Heisenberg group, which is not essentially self-adjoint, namely:

(\frac{∂}{∂x})^{4} + (\frac{∂}{∂y} + x\frac{∂}{∂z})^{2}  (21)

References

[1] van den Ban, E. P., Invariant differential operators on a semi-simple symmetric space, and finite multiplicities in a Plancherel formula., Report, Centre for Mathematics and Computer Science, PM-R8409.

[2] Dixmier, J., Sur les représentations unitaires des groupes de Lie nilpotents, III, Canad. J. Math. 10 (1958), 321–348.

[3] Dixmier, J. and P. Malliavin, Factorisations de fonctions et de vecteurs indéfinément différentiables, Bull. Soc. Math. France, 102 (1978), 305– 330.

[4] Dunford, N. and J. T. Schwartz, “Linear Operators part II”, Interscience Publishers, John Wiley, New York 1963.

[5] van Dijk, G., On generalised Gelfand pairs, Proc. Japan Acad. Sc. 60, A (1984), 30–34.

[6] Faraut, J., Distributions sphériques sur les espaces hyperboliques, J. Math. Pures et Appl. 58 (1979), 369–444.

[7] Gårding, L., Vecteurs analytiques dans les représentations des groupes de Lie, Bull. Soc. Math. France. 88 (1960), 73–93.

[8] Godement, G., Théorie des caractères II: Définitions et propriétés générales des caractères, Ann. of Math., 59 (1954), 63–85.

[9] Kirillov, A. A., Unitary representations of nilpotent groups, Russ. Math. Surveys, 17 (1962), 53–104.

[10] Klamer, F. J. M., Group representations in Hilbert subspaces of a locally convex space, PhD Thesis, University of Groningen, 1979.

[11] Lévy-Bruhl, P., Spectre d’opérateurs de Schrödinger avec potentiel et champs magnétique polynomiaux., Publ. Res. Inst. Math. Sci. 27:3 (1991), 367– 373.

[12] Nelson, E. and W. F. Stinespring, Representation of elliptic operators in an enveloping algebra, Amer. J. Math. 81 (1959), 547–560.

[13] Reed, M. and B. Simon, “Methods of Modern Mathematical Physics ”, Academic Press, New York, 1975, 1978.

[14] Rossmann, W., Analysis on real hyperbolic spaces, J. Func. Anal. 30 (1978), 448–477.

[15] Schwartz, L., Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés, J. Anal. Math. 13 (1964), 115–256.

[16] Segal, I. E., Hypermaximality of certain operators on Lie groups, Proc. Amer. Math. Soc. 3 (1952), 13–15.

[17] Thomas, E. G. F., The theorem of Bochner-Schwartz-Godement for generalised Gelfand pairs, Functional Analysis: Surveys and recent results III, K.D. Bierstedt and B. Fuchsteiner (eds.) Elsevier Science Publishers B.V. (North Holland), 1984.

E. G. F. Thomas Universiteit Groningen Mathematisch Instituut Postbus 800, 9700 AV The Netherlands E.G.F.Thomas@math.rug.nl

Received December 20, 2000 and in final form May 17, 2001