[Cialug] Guarantee SSH availability

[Kenneth Younger]

I'm not sure what exactly occurred. It's a basic LAMP server with an nginx
proxy doing some caching there.

Running this server has been humbling, because I used to think I knew what I
was doing with regard to running a server - but this thing has given me a
new respect for the subtle configuring that an expert can make.

I know this isn't HUGE traffic, but it was getting 3000-4000 pageviews an
hour at peak, which is the most traffic I've ever seen on a site I run. I
think my problem is I just don't know how to tell what went wrong after the
fact...

How would tell if it ran out of memory, or the CPU was just fully pegged, or
the disk started thrashing?

-Kenny

On Tue, Jun 28, 2011 at 11:10 AM, Nicolai <nicolai-cialug at chocolatine.org>wrote:

> On Mon, Jun 27, 2011 at 04:39:20PM -0500, Kenneth Younger wrote:
> > I had a sever crush under load the other day, so much so that it made
> SSHing
> > into the box impossible. There's got to be a way to keep that admin tool
> > available until the end, right?
>
> Hi Kenny,
>
> Indeed there is: it's called daemontools.  Combine it with resource
> limits and you've got a reasonably stable system.
>
>  http://cr.yp.to/daemontools.html
>
> daemontools is from the same guy who wrote qmail, djbdns, and a long
> list of other quality software.  I've been using it on almost every
> system I've had in the last 11 years, and it has never crashed or
> failed.  It's solid.
>
> Basically, it's a collection of tiny tools, one of which runs quietly in
> the background, monitoring whatever daemons you want, and restarting
> them if they happen to exit for some reason.
>
> IF you:
>
>  1. Monitor critical daemons using daemontools and
>  2. Have resource limits properly configured,
>
> Then you're unlikely to see this problem re-occur.
>
> What exactly happened, though?  Knowing this will help guide you in your
> tuning of resource limits.  And I'm pretty curious!
>
> Nicolai
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug
>



-- 
Kenneth Younger III
Founder, Sheer Focus Inc.
e: kenny at sheerfocus.com
p: (515) 367-0001
t: @kenny <http://twitter.com/kenny>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cialug.org/pipermail/cialug/attachments/20110628/12033f61/attachment-0001.html>