Hi Henrik!
I'd look on the IDM database end.You don't mention what database you're using or version of IDM. But I'd look at the following:
1. Download the Configuration Analyzer. And take a look at its results. It will refer you to queries that can be optimized and "choke points"
2. Also check the admin console if you're running 7.2. Later SPs also have some configuration analyzer functionality to check for queries that are taking a long time (configurable) to run.
Matt