I finally managed to use STARTTLS together with the Courier Mail Server. SSL connections (imapd-ssl and esmptd-ssl) already worked with the generated certificate. But although SSL worked trouble free and TLS options had nearly the same settings I could not get STARTTLS to work.
The overall reason was that the certificate file was owned 600 by root. With SSL this is not a big problem as the daemon which listens on port 465 and 993 does all the crypto handshake itself and runs under root. But with STARTTLS the connection is already forwarded to a non-privileged process before TLS starts its handshake (in fact it is initiated by the client over a cleartext connection).
So the solution was quite easy: Just give the certificate file the same owner the non-privileged courier processes run under – user “daemon” in case of CentOS 4/5.
Update: Yes, the solution appears to be quite obvious. But in fact courier doesn’t give any usefull information about the missing permissions. Instead it is handled as there would no no certificate file at all and SSL is disabled silently.