[Cialug] Guarantee SSH availability

[Nicolai]

On Mon, Jun 27, 2011 at 04:39:20PM -0500, Kenneth Younger wrote:
> I had a sever crush under load the other day, so much so that it made SSHing
> into the box impossible. There's got to be a way to keep that admin tool
> available until the end, right?

Hi Kenny,

Indeed there is: it's called daemontools.  Combine it with resource
limits and you've got a reasonably stable system.

 http://cr.yp.to/daemontools.html

daemontools is from the same guy who wrote qmail, djbdns, and a long
list of other quality software.  I've been using it on almost every
system I've had in the last 11 years, and it has never crashed or
failed.  It's solid.

Basically, it's a collection of tiny tools, one of which runs quietly in
the background, monitoring whatever daemons you want, and restarting
them if they happen to exit for some reason.

IF you:

 1. Monitor critical daemons using daemontools and
 2. Have resource limits properly configured,

Then you're unlikely to see this problem re-occur.

What exactly happened, though?  Knowing this will help guide you in your
tuning of resource limits.  And I'm pretty curious!

Nicolai