[NTLUG:Discuss] Challenge....

[asenec@senechalle.net]

Write a script which will write the process table to 
a /tmp file and run it once a minute out of the rootcron.
Look at the content of the last file written before the reboot,
or shutdown, and you might get a clue.  I'd give each file a 
unique name so you won't overwrite the file.  Something
like /tmp/dbug-$hour-$minute would work.  Also, if you're running
Apache, add the pid to the debug log, like:

LogFormat "%h %l %u %t \"%r\" %>s %b %P" common

Annette
> 
> Douglas King wrote:
> 
> > OK, I've got a problem.  This past week.....I've had a RedHat 7.3 
> > webserver shut down every night somewhere between the hours of 1:30 AM 
> > and 3:00 AM.  We have checked ALL the scheduled crons, etc. and find 
> > nothing that would be shutting it down "naturally".  The power light 
> > on the case remains on, but you cannot SSH into it, nor is the machine 
> > functional.  Log files don't indicate a lot to me...although, I did 
> > catch a potential hacker 2 nights ago...but he's since been dealt with.
> >
> > Where do I look now? 
> >
> > _______________________________________________
> > https://ntlug.org/mailman/listinfo/discuss
> >
> Douglas,
> 
> Could be nearly anything, tough to tell from the outside looking in. 
> Here are a couple suggestions, for what they're worth, followed by some 
> questions to ask (ranging from obvious to simply logical, may be of no 
> use to you at all or they might trigger an epiphany, hope for the later).
> 
> 1. check the logs to see who's logged in when the machine shuts down 
> every night and look for a pattern.
> 2. check the logs to see what processes are logging just prior to the 
> shut down every night and look for a pattern.
> 
> Here are some things to consider:
> If you have a monitor attached to the machine - is the led green, yellow 
> or what (ie is the monitor on with signal, on standby, or off)?
> Is the screen blank or does the machine only appear to be 'hung' not off?
> Where is the machine located? Is it at home or in a lab? Is it 
> physically secure?
> Is it possible that their was an external power event that caused the 
> machine to shutdown?
> Is the machine equipped with any external controller such as wake on 
> lan, remote shutdown, etc?
> Is it possible that their was an internal hardware event that caused the 
> machine to shutdown (power supply issue, CPU overheat, etc)?
> What kind of machine is it and does it have hardware monitoring 
> capabilities - cpu sensors, power sensors, etc?
> Is the machine overclocked or insufficiently ventilated (could cause any 
> number of chaotic problems)?
> Is APM turned off (no standbye or hibernate features enabled for the 
> system or monitor)?
> Is this a new box or has this box had recent software/hardware/use 
> pattern changes?
> Is tripwire installed and have you looked into it?
> 
> Some possible causes:
> Security breach.
> Someone shutting the machine down through software (something along the 
> lines of 'shutdown -h now' or 'halt').
> External power event (brownout - power spike).
> Internal power event (powersupply, motherboard, harddrive).
> Internal motherboard failure.
> Hard disk failure.
> Power management failure
> 
> Later, hope this helps - surely some guru'll tell you how to turn on 
> 'uberlog' and it'll just spit out the answer. I eagerly await the 
> findings...
> 
> Will
> 
> 
> _______________________________________________
> https://ntlug.org/mailman/listinfo/discuss
>