I was searching the forum some more and it sounds like this guy has a similar problem as mine: http://www.peachparts.com/shopforum/tech-help/126958-m104-misfire.html

Conclusion from that thread was a bad ECU. I'll have to try a new ECU and see if the problem is fixed.