[NTLUG:Discuss] NFS Synchronization

[George Lass]

--- On Fri, 5/21/10, Robert Pearson <e2eiod at gmail.com> wrote:

> From: Robert Pearson <e2eiod at gmail.com>
> Subject: Re: [NTLUG:Discuss] NFS Synchronization
> To: "NTLUG Discussion List" <discuss at ntlug.org>
> Date: Friday, May 21, 2010, 3:43 AM
> On Thu, May 20, 2010 at 4:08 PM,
> George Lass <george.lass at sbcglobal.net>
> wrote:
> > I need to set up a pair of NFS servers that need to be
> synchronized and am wondering if rsync is the right way to
> do it?  The primary use of the pair of servers is for
> recording and playback of phone messages stored in .wav
> files.  The only "gotcha" is that any particular recording
> will originally be written (recorded) randomly to one of the
> servers or the other, while read (playback) requests can be
> made to either server.  Thus within just a minute or 2, the
> file must be propagated from one server to the other.  It
> should be assumed that the directory structures where the
> .wav files reside are identical on both machines.
> >
> > Thanks in advance for any advice on this,
> >
> > George Lass
> >
> 
> This is an intriguing design requirement. It may be as
> simple as it
> looks or it may not be.
> Chris's solution is right on and a good working one for
> this type of scenario.
> The exact design depends on whether you are going for a
> complete DIY,
> homegrown, lowest possible cost or something more advanced
> and
> expensive.
> The scenario I interpret from your information is two
> asynchronous NFS
> servers in an Active/Active mirror.
> This is a caching nightmare to be avoided.
> There might be a better way to do this depending on your
> true requirements.
> It depends on the write bandwidth requirement for the NFS
> servers and
> the read requirement.
> If you could describe a typical operation it would be
> helpful. Or
> worst case operation would be even better.
> My worst case would be something like this.
> These .wav file are logs of telephone calls. Two servers
> are needed
> for bandwidth due to call volume. This means the same
> telephone number
> could be accessing both servers separately simultaneously
> (call
> waiting) or without much time lag between accesses. Now
> there are two
> .wav files on two different servers with the same name but
> different
> time stamps containing different information. How does the
> read know
> which one it wants or does it want all the most recent
> ones?
> It would be good to know if writes are imperative while
> reads are
> deferrable within a very short delay. You did mention a one
> or two
> minute sync delay is ok.
> One solution to do this is to sync all the files on a
> separate "read
> only" device using deduplication software (DIY solution).
> This leaves
> the NFS front end completely asynchronous and free to just
> handle
> writes. The "read only" device is the mirror. Some of the
> Web front
> ends do this.
> There are other considerations depending on the application
> and Security needs.
> 
> As per Chris's suggested solution the more done in the
> storage devices
> the better.
> 
> _______________________________________________
> http://www.ntlug.org/mailman/listinfo/discuss
> 

Fortunately the worst case is not quite that bad, and in retrospect, I should have mentioned that the .wav files will all have unique names regardless of which server they are written to.  That should eliminate many of the previously mentioned issues.  That said, here are some more details concerning what it is that I need to do....

The phone switch that I am dealing with can read and write .wav files in real time via the NFS protocol.  During switch setup I can give it 2 different IP addresses and mount points for NFS servers.  When I tell the switch to play a .wav file, I specify which of the files is "primary" and which is "secondary".  It will request the file from the primary first, and if it's not there, or the primary server is down, it will make the same request of the secondary server.

Record operations are essentially the same.  I specify which of the 2 pre configured servers is primary and which is secondary.  Again, it will try to create the .wav file on the primary server, and if that fails it will try to create it on the secondary.

Given the above flexibility, I could normally specify server "A" as primary and "B" as secondary, and for the most part I would just need to sync from A to B.  But as previously mentioned, I would like the phone switch to load share the 2 servers.  So I will likely set up toggle functions that will tell the phone switch to alternate A primary, B secondary / B primary, A secondary for every other read, and for every other write.

Hope this helps and thanks!

George