You called it dirt, but in reality what was it? Was it gasket material? Was it grime from the surrounding areas?

The oil pump is quite robust and the filter will catch whatever makes it through.

If you are really particular, drain the oil out of the pan and then rapidly pour a gallon of diesel fuel through the front if the timing cover is still off while the drain plug is out and a catch pan underneath. This will flush out most any loose material and the slight amount of diesel residue won't hurt anything.