Dear Thorsten, I'm not familiar with the notation that you are using, although I can guess what is meant in some cases
I am not sure I completely understand your comments. I guess it may be helpful to be more precise:
F : FinSet -> Set F A = Real -> A
I assume you mean A->Real. It's true that the monad for vector spaces sends a finite set A to R^A, which can be seen as the set of functions from A to R. For a general set A (not necessarily finite) FA is the set of functions from A to R of finite support. Equivalently, FA is the set of formal finite linear combinations of elements of A.
I suspect my eta and >>= give then rise to a monad on Set? However, I don't see how to do this if the vector spaces are not finite.
Yes, this gives a monad on Set whose algebras are vector spaces, not necessarily finite dimensional. I'm not sure what it is you claim to be doing when you "do this". In any case there is a monad on Set whose algebras are vector spaces; there is not a monad on Set whose algebras are finite dimensional vector spaces. You can see this last statement by noting that the category of algebras for a monad on Set is always cocomplete.
Btw, I only used this as an example. My question was rather wether people have studied monoids in categories of functors which are not endofunctors. I believe this notion is useful in functional programming and Type Theory as a natural generalisation of the notion of a monad.
Yes, monoids in categories of functors are useful concepts. Of course to define a monoid you need a monoidal structure on the ambient category. There may be many possibilities, and for some of them the corresponding notion of monoid looks more like a monad than for others. For some monoidal structures one should really think of the monoids as not generalizations of monads, but special cases of monads. Your example of finitary monads is a good example. So are operads. There are more examples in the paper "notions of Lawvere theory" available from my home page or as arXiv:0810.2578. Regards, Steve Lack.