|
[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
[NMLUG] Python tight loop causing massive CPU barfage
On Tue, 2005-02-08 at 20:44, Paul Tietjens wrote:
> I have a python script that essentially opens a few thousand (between
> 70,000 and 230,000 or so) files, reads the first 1024 bytes and looks
> for a string match.
>
> The goal is to search an entire partition full of Maildirs for specific
> emails.
>
> I want the process to happen as fast as possible. So far, it takes
> around 21 minutes - but there's a snag. While this script is running,
> every other process on the machine becomes sluggish to the point of
> nonresponsiveness.
>
> No amount of playing with nice and priority levels seems to help.
This should work. Perhaps python is spawning a thread or process
in the search, which has higher priority.
Look for where this MIGHT happen, and see if you
can reduce the priority on the spawned thread or process.
Another possibility... perhaps its just spawning a LOT of
threads or processes. Think about re-coding to do search
in python instead of doing something like spawning grep, head or cat...
(Sorry I can't help more, I don't know python)
> What has helped, is a small sleep() in the loop...
see if they have something like usleep, which allows sleeps
shorter than 1 second. Some systems support a sleep(0)
which does not actually sleep, but does release the processor
to service the next process in the queue.
--
Aaron Birenboim \ I have an inferiority complex,
Albuquerque, NM, USA \ but its not a very good one.
aaron at birenboim.com \
http://aaron.birenboim.com \
|
|
