[NTLUG:Discuss] backing up in linux....

[Robert Pearson]

On 9/15/06, Eric Waguespack <ewaguespack at gmail.com> wrote:
> the problem with this is, a)  i want to be able to easily see what is
> on volume # $RANDOM, and I do not want to have a situation where if I
> lose volume 1 I am s.o.l.

Could you define "easily see" more?
Is that disk space usage or by Content or both?
How do you do the baseline Information Integrity now?
Backup is the common process. The backup tool varies.

I have been asking for something like this for a long time.
Several times I have written my own solution.
Why several times? Each situation is unique. Like yours.
The tool may be the same and a large part of the code is the
same but the people, processes and required results are different.

The closest tool is "Snapshotting". This solves the Information
Integrity problem, provided the snapshot gets committed before
failure, and provided the destination does not fail at the same time
as the source. This can happen if everything was in one building
or site and all power was lost.

"File syncing" tools are the second choice. You want streaming so
as a file is changed it is "sync'd". Streaming puts a load on the
resources so most people opt for every 15, 30 or 60 minutes.
Streaming is the only choice for  "if I lose volume 1 I am s.o.l."
These tools are subject to the same "power caveat" above.

Can you afford the $$$ to do this? The inexpensive way, in $$$
terms, is to write it yourself. Depends on what you value time at.
$$$ is why there are so many "home-grown" solutions and people
love Linux, rsync and tar.

The inexpensive "Snapshotting" and FileSyncing tools are for
Personal Computing and the SOHO. For large, 100's of GBs or
TBs, sites you need "industrial strength" and the price goes up.

>
> On 9/15/06, Robert Pearson <e2eiod at gmail.com> wrote:
> > On 9/15/06, Chris Cox <cjcox at acm.org> wrote:
> > > Eric Waguespack wrote:
> > > > thanks for all of the advise... but nothing suggested is really what I
> > > > was looking for.
> > > >
> > > > maybe I'll submit an enhancement request:
> > > >
> > > > man cp
> > > > ..
> > > > --pause-and-ask-user-for-another-destination-if-target-device-is-full
> > > > ..
> > >
> > > GNU tar will do this... take a look at the man page.
> > >
> > > You can call a script when you reach end of medium and do
> > > whatever you want.
> >
> >
> > This is not a trivial problem.
> > What you may be looking for is "multi-volume" capability.
> > The key phrase is "multi-volume".
> > Anything that wrote to tape in a serious fashion, backups are serious,
> > will do multi-volume to most media. You would need to check with the
> > product vendor to see if it has been tested with USB thumb drives.
> >
> > The challenge here is not in the software but whether the USB thumb
> > drives report, or signal, being "full" in a way that the multi-volume
> > software can get the signal. The early 8 mm cartridge tapes did not
> > have an EOT (end of tape) sensor, so no EOT signal came from the
> > drive. You had to calculate bytes written and tape used.
> >
> > Floppies had the same problem. How does the floppy tell you it is full?
> > I can remember sitting there with a handful of floppies wanting for the
> > screen to tell me to insert the next one.
> > I think I was using the Norton System software.
> >
> > GNU tar options from "info tar":
> >
> > `--multi-volume'
> > `-M'
> >      Informs `tar' that it should create or otherwise operate on a
> >      multi-volume `tar' archive.
> >
> > `--new-volume-script'
> >      (see -info-script)
> >
> > `--info-script=SCRIPT-FILE'
> > `--new-volume-script=SCRIPT-FILE'
> > `-F SCRIPT-FILE'
> >      When `tar' is performing multi-tape backups, SCRIPT-FILE is run at
> >      the end of each tape.  If SCRIPT-FILE exits with nonzero status,
> >      `tar' fails immediately.