The server I tested dovecot on is plain simple ispCP 1.0.5 on Lenny, the only mod used is dovecot instead of courier (mostly after the howto here and some help from the official dovecot/courier howto). Filesystem is ext3, but the only slow application is dovecot (database / apache are pretty much fast as ever), so I ruled out the hard drive / filesystem as the main problem. Ram is ok, cpu usage pretty normal, no "standard" explanation why dovecot should break down like that.
If there's no easy fix I'll try to switch back to courier and test the speed with the "old one", just thought I'd ask first
Markus