[NTLUG:Discuss] systemd, mbr boot record recovery, but now can't mount anything (?)

[Chris Cox]

so... here is what happened.

The host has been around for a bit.  Survived many upgrades, even fairly
radical ones.

So... while I wanted to blame systemd for "the problem" all I can really
do is blame systemd for making diagnosis and finding the answer *very*
difficult (I'm sort of hoping Linus gets into a "bad spot" and has to
troubleshoot a systemd system.... I imagine a flurry of "colorful
metaphors").

Anyhow, what tripped me up is something actually quite old.  It is the
move away from "ata" to "scsi".  Again, this system has been around for
awhile and it was running openSUSE 13.2 before the accident, but
apparently there was something (maybe) held in place udev wise to present
the compatibility links as we moved from "ata" to "scsi" especially in
/dev/disk/by-id.

Anyhow, long story short, once I change things, I was able to mount again.

Now... when the problem was occurring, I had *one* erroneous line in
/etc/fstab still referring to an device by "ata"..... and that caused
complete chaos on the system (things like this didn't used to happen btw).


So... that fixing that *one* line made things normal.  With the bad line,
nothing would mount (and I mean nothing... which is very very very bad). 
So if you ever find youself with a "booted system" but in a systemd
emergency target, make sure your /etc/fstab is sane.  To troubleshoot,
just whittle everything down to just the / being in there... that will at
least get rid of the "cannot mount/busy" problem for everything.  Again,
no hints, and certainly *not* behavior like we would have pre-systemd.

You may seen my exact problem if using an old host that has been through
many upgrades including one that moved us from "ata" to "scsi"... and
yes... this system has been through it many times... just reared its ugly
head on this recovery.   Sigh....

As for systemd... essentially "how to hide everything you knew and make it
extremely hard to diagnose and troubleshoot anything".  There's more to my
adventure than stated here.... and hours were spent on something that
would have normally taken me mere minutes to determine the problem.  And
this isn't solved by becoming a Lennart worshipper..... systemd has a long
long long way to go before it's something I would like enterprise wide,
yet here we are.... sigh....

As for systemd fan boyz... I hope you get into a hard spot and spend days
troubleshooting something simple.... but then again, I doubt even that
will open your eyes.... sheesh....  (Linus... we need your help again).