Re: 1.25 RC

On Fri, Jul 22, 2005 at 09:05:31PM +0100, Minty wrote:
> > This makes me think that it might be useful to have something (on
> > the wiki?) that details the low level process for what we do with the
> > information in each file (reading + writing). I'm not sure if there's
> > an equivalent document for Lucene that we could use as a starting
> > point. Otherwise we can just build it up bit by bit ...
> This seems to be fairly comprehensive
> http://lucene.apache.org/java/docs/fileformats.html

This isn't really at the correct level. 

I'm thinking of something that goes through the steps of what happens
when we're indexing / merging / searching etc, rather than a
byte-by-byte guide to each file. I'm more for what each file and how it
relates to the process .

> I am increasingly of the view that slurping the whole file, and using
> pack isn't going to scale (for index size > available ram).

For the big files, yes, but at the minute all we've done is streamline a
process that tended to read the whole file anyway.

> How about we try using file handles, seeking, but then reading in an
> entire record where this is possible and decoding with pack.  Rather
> than read_vint, read_vint, read_string, read_vint, etc. blocks?

That's what I did with the stuff I posted last night.

Tony