[NMLUG] Python tight loop causing massive CPU barfage

[paul.tietjens at moriarty.k12.nm.us (Paul Tietjens)]

Sarang wrote:

>My thought: start w/ something like:
>
>generate-list-of-files | xargs head -c 1024 > tempfile.txt
>
>and then work w/ tempfile.txt (not trivial, still some work to be done
>there). "head" is probably optizimed and so probably faster than using
>python to grab the first 1K of each file.
>
>  
>
Good thought, and my original script wasn't much more than that - and in 
bash.

It was very slow.  Probably related all the instances of head spawned, 
but I'm just taking a wild guess, there.

Find does seem to be the fastest way to get a list of files sorted out 
of a tree, and I did end up keeping the find commands (using 
commands.getoutput(find) from within python).

My instinct is to avoid writing files  - as I would assume every extra 
IO op would crate a new bottleneck - but I could be wrong!