[Cialug] Guarantee SSH availability

[Charles Dunbar]

Sar would be your tool of choice to figure out "what happened back
then."  Usually installed through sysstat in your package manager.

http://linux.die.net/man/1/sar

By default when you install it, it runs every 10 minutes via a cron,
but if you need more fine control, just change that to what you need.

Need to see the CPU utilization at the time? sar -u, Queue length
(load)? sar -q, I/O? -b, Memory? -R.  Combine them all together to get
one big report at once, can even specify time frames with -s and -e to
not get the whole output of the day.

I use sar on a daily basis to see "what was going on at that time"
when it comes to hosting websites.

Charles

On Tue, Jun 28, 2011 at 12:06 PM, Kenneth Younger <kenny at sheerfocus.com> wrote:
> I'm not sure what exactly occurred. It's a basic LAMP server with an nginx
> proxy doing some caching there.
>
> Running this server has been humbling, because I used to think I knew what I
> was doing with regard to running a server - but this thing has given me a
> new respect for the subtle configuring that an expert can make.
>
> I know this isn't HUGE traffic, but it was getting 3000-4000 pageviews an
> hour at peak, which is the most traffic I've ever seen on a site I run. I
> think my problem is I just don't know how to tell what went wrong after the
> fact...
>
> How would tell if it ran out of memory, or the CPU was just fully pegged, or
> the disk started thrashing?
>
> -Kenny
>
> On Tue, Jun 28, 2011 at 11:10 AM, Nicolai <nicolai-cialug at chocolatine.org>
> wrote:
>>
>> On Mon, Jun 27, 2011 at 04:39:20PM -0500, Kenneth Younger wrote:
>> > I had a sever crush under load the other day, so much so that it made
>> > SSHing
>> > into the box impossible. There's got to be a way to keep that admin tool
>> > available until the end, right?
>>
>> Hi Kenny,
>>
>> Indeed there is: it's called daemontools.  Combine it with resource
>> limits and you've got a reasonably stable system.
>>
>>  http://cr.yp.to/daemontools.html
>>
>> daemontools is from the same guy who wrote qmail, djbdns, and a long
>> list of other quality software.  I've been using it on almost every
>> system I've had in the last 11 years, and it has never crashed or
>> failed.  It's solid.
>>
>> Basically, it's a collection of tiny tools, one of which runs quietly in
>> the background, monitoring whatever daemons you want, and restarting
>> them if they happen to exit for some reason.
>>
>> IF you:
>>
>>  1. Monitor critical daemons using daemontools and
>>  2. Have resource limits properly configured,
>>
>> Then you're unlikely to see this problem re-occur.
>>
>> What exactly happened, though?  Knowing this will help guide you in your
>> tuning of resource limits.  And I'm pretty curious!
>>
>> Nicolai
>> _______________________________________________
>> Cialug mailing list
>> Cialug at cialug.org
>> http://cialug.org/mailman/listinfo/cialug
>
>
>
> --
> Kenneth Younger III
> Founder, Sheer Focus Inc.
> e: kenny at sheerfocus.com
> p: (515) 367-0001
> t: @kenny
>
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug
>
>