[NTLUG:Discuss] Disk I/O timings.

[Steve Baker]

George Lass wrote:
> 
> Not knowing anything about the data you are dealing with, this
> suggestion
> may be way off base, but have you thought about placing your files into
> a ramdisk on system startup?

Using a Ramdisk in this application is impractical since there are many tens of
gigabytes of data involved. The files vary in size from perhaps 10Kb to a few
megabytes - but there could easily be 40,000 of them and I don't really know
in advance which files I'll need to read.

I read the files at a rate of at most one every few seconds - the overall
performance isn't at issue here.

Several other people suggested optimisation techniques to make things
faster overall - that's all useful information - but that's not exactly
my problem.

I'm writing an embedded realtime system here - I can tolerate or work-around
delays *if* I know when they'll happen.

I know that the disk accesses (even if utterly perfectly optimised) are
going to take enough time to be very significant to my application
- but I need to understand *WHEN* that hit will occour so that I can take
precautions to avoid it's effects (eg by running that step in a parallel
thread).

That's why I want to know at what point in my open/fstat/mmap/read-memory/munmap/close
cycle the delay will hit my program and cause it to block.

As I said before, experiments suggest that sometimes it's in the 'open'
call - and sometimes when I attempt to extract data from the array that
I mmap'ed the file into.  It's hard to figure this out though because
the kernel (and perhaps disk drive) caches make experiments unrepeatable.

Knowing exactly when the kernel reads the file data into it's cache
(or into my address space) would help - and figuring a way to disable
(or at least flush) the kernel cache would make it easier to do experiments
to find out what's happening.

----------------------------- Steve Baker -------------------------------
Mail : <sjbaker1 at airmail.net>   WorkMail: <sjbaker at link.com>
URLs : http://www.sjbaker.org
       http://plib.sf.net http://tuxaqfh.sf.net http://tuxkart.sf.net
       http://prettypoly.sf.net http://freeglut.sf.net
       http://toobular.sf.net   http://lodestone.sf.net