The following paper is available by anonymous ftp at triples.math.mcgill.ca in the directory pub/blute as nuclear.ps.gz. It is also on Prakash Panangaden's homepage at www-acaps.cs.mcgill.ca. Feel free to contact me if there are any problems. Cheers, Rick Blute Nuclear and Trace Ideals in Tensored *-Categories ================================================= Samson Abramsky Richard Blute Department of Computer Science Department of Mathematics University of Edinburgh and Statistics Edinburgh, Scotland University of Ottawa Ottawa, Ontario, Canada Prakash Panangaden Department of Computer Science McGill University Montreal, Quebec, Canada Presented to Mike Barr on the occasion of his 60th birthday. Abstract ======== We generalize the notion of nuclear maps from functional analysis by defining nuclear ideals in tensored *-categories. The motivation for this study came from attempts to generalize the structure of the category of relations to handle what might be called ``probabilistic relations''. The compact closed structure associated with the category of relations does not generalize directly, instead one obtains nuclear ideals. Most tensored *-categories have a large class of morphisms which behave as if they were part of a compact closed category, i.e. they allow one to transfer variables between the domain and the codomain. We introduce the notion of nuclear ideals to analyze these classes of morphisms. In compact closed categories, we see that all morphisms are nuclear, and in the category of Hilbert spaces, the nuclear morphisms are the Hilbert-Schmidt maps. We also introduce two new examples of tensored *-categories, in which integration plays the role of composition. In the first, morphisms are a special class of distributions, which we call tame distributions. We also introduce a category of probabilistic relations which was the original motivating example. Finally, we extend the recent work of Joyal, Street and Verity on traced monoidal categories to this setting by introducing the notion of a trace ideal. For a given symmetric monoidal category, it is not generally the case that arbitrary endomorphisms can be assigned a trace. However, we can find ideals in the category on which a trace can be defined satisfying equations analogous to those of Joyal, Street and Verity. We establish a close correspondence between nuclear ideals and trace ideals in a tensored *-category, suggested by the correspondence between Hilbert-Schmidt operators and trace operators on a Hilbert space. When we apply our notion of trace ideal to the category of Hilbert spaces, we obtain the usual trace of an endomorphism in the trace class.