[NMLUG] Python tight loop causing massive CPU barfage

[paul.tietjens at moriarty.k12.nm.us (Paul Tietjens)]

I have a python script that essentially opens a few thousand (between 
70,000 and 230,000 or so) files, reads the first 1024 bytes and looks 
for a string match.

The goal is to search an entire partition full of Maildirs for specific 
emails.

I want the process to happen as fast as possible.  So far, it takes 
around 21 minutes - but there's a snag.  While this script is running, 
every other process on the machine becomes sluggish to the point of 
nonresponsiveness.

No amount of playing with nice and priority levels seems to help.

What has helped, is a small sleep() in the loop - but that raises the 
amount of time taken to complete the tasks fairly rapidly (from 21 
minutes to over an hour).

In the end, I set up a goofy sort of throttling that alters the amount 
of time sleep()ing by the average load.

Is there a better way to do this?  I'm not much of a coder, and I know 
there are a couple on this list - so any tips offered, no matter how 
nebulous, would be great.

Thanks in advance!