[NTLUG:Discuss] Disk I/O timings.

[Steve Baker]

merlin wrote:
> 
> On Wed, 5 Dec 2001, Steve Baker wrote:
> 
> > Using a Ramdisk in this application is impractical since there are many tens of
> > gigabytes of data involved. The files vary in size from perhaps 10Kb to a few
> > megabytes - but there could easily be 40,000 of them and I don't really know
> > in advance which files I'll need to read.
> >
> > I read the files at a rate of at most one every few seconds - the overall
> > performance isn't at issue here.
> 
> How far ahead of time do you know you're going to read a file?

A second or two - if it was important to know.

> Could a
> spawn a parallel thread, say a second or two before hand, that did a
> open/read small chunk/close type operation to load the file into cache
> before your main routine needed to hit the file?

Yes - that's exactly what I assume I'll need to do - but I'm not
using the 'read()' system call to read the file - but instead using this
'mmap()' trick to map it into a chunk of memory. (That happens to be a very
convenient thing for the way my program works).

So *IF* the kernel physically reads the file when I 'open()' or 'mmap()' it,
all I need to put into the parallel thread is the open and mmap calls with
some kind of a signal to tell the main thread that the data is ready to be read.

However, *IF* the kernel only actually reads the file when I access the memory
locations mapped onto the file (which really has to be done in the main thread)
then that won't help.  In that case, the secondary 'file reading' thread would
have to cycle through those memory locations that the main program wants to
access in order to 'pre-fetch' them from the disk.  That's a considerable
inconvenience and actually means that I might as well just 'read()/fseek()'
the file - which I don't really want to have to do.

But understanding the system is the first step to program design - and right
now, I don't understand the minute details of how all this happens.  Once I
do, I presume that some kind of an elegant solution will leap out at me... :-)

----------------------------- Steve Baker -------------------------------
Mail : <sjbaker1 at airmail.net>   WorkMail: <sjbaker at link.com>
URLs : http://www.sjbaker.org
       http://plib.sf.net http://tuxaqfh.sf.net http://tuxkart.sf.net
       http://prettypoly.sf.net http://freeglut.sf.net
       http://toobular.sf.net   http://lodestone.sf.net