|
[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
[NMLUG] Python tight loop causing massive CPU barfage
My thought: start w/ something like:
generate-list-of-files | xargs head -c 1024 > tempfile.txt
and then work w/ tempfile.txt (not trivial, still some work to be done
there). "head" is probably optizimed and so probably faster than using
python to grab the first 1K of each file.
--
Sincerely, Sarang (sarang@sarangworld.com)
AOL Instant Messenger ID: JavaMath
Backup Email: saranghome@softhome.net
<html><img src=3D'http://noproxy.sarangworld.com/EXP/rr.php?20050209.130928=
'></html>
On Wed, 9 Feb 2005, Moses Gold wrote:
> Date: Wed, 9 Feb 2005 07:51:44 -0800 (PST)
> From: Moses Gold <moegold2001@yahoo.com>
> Reply-To: New Mexico Linux Users Group Mail List <nmlug@nmlug.org>
> To: New Mexico Linux Users Group Mail List <nmlug@nmlug.org>
> Subject: Re: [NMLUG] Python tight loop causing massive CPU barfage
>
>
>
> could it be worth building a flat database containing the first 1k of eac=
h
> file?
>
> --- Paul Tietjens <paul.tietjens@moriarty.k12.nm.us> wrote:
>
> > Tim Emerick wrote:
> > > have you checked out fgrep? I've not used it but looking at the man =
seemed
> > > to do what you are looking to do.
> > >
> > > Tim
> > >
> >
> > fgrep would probably work, as would a combination of find/grep/rm - and
> > we've used these.
> >
> > It takes around two hours for these tools to walk through 230k files.
> > Probably related to the fact that grep/fgrep looks at the entire file -
> > where my script looks only at the first 1024 bytes.
> >
> > I suppose we could make a fancier script (output first few lines, grep
> > THAT, act on matches) and that's how we ended up with this python thing=
=2E :)
> > _______________________________________________
> > NMLUG mailing list
> > NMLUG@nmlug.org
> > http://www.nmlug.org/mailman/listinfo/nmlug
> >
>
>
>
>
> __________________________________
> Do you Yahoo!?
> All your favorites on one personal page =96 Try My Yahoo!
> http://my.yahoo.com
> _______________________________________________
> NMLUG mailing list
> NMLUG@nmlug.org
> http://www.nmlug.org/mailman/listinfo/nmlug
>
|
|
