[Cialug] Guarantee SSH availability

[David Champion]

You probably want to do some standard LAMP tuning - look at your Apache
setup, see how many processes it's running etc.

Is this a home-grown web app, or a canned package?

You may need to spend some time doing some fine-tuning on the application
and database. Just doing some simple SQL optimization (i.e. make sure there
are indexes on the join conditions), I've seen queries go from 15 seconds to
under a second.

-dc

On Tue, Jun 28, 2011 at 12:06 PM, Kenneth Younger <kenny at sheerfocus.com>wrote:

> I'm not sure what exactly occurred. It's a basic LAMP server with an nginx
> proxy doing some caching there.
>
> Running this server has been humbling, because I used to think I knew what
> I was doing with regard to running a server - but this thing has given me a
> new respect for the subtle configuring that an expert can make.
>
> I know this isn't HUGE traffic, but it was getting 3000-4000 pageviews an
> hour at peak, which is the most traffic I've ever seen on a site I run. I
> think my problem is I just don't know how to tell what went wrong after the
> fact...
>
> How would tell if it ran out of memory, or the CPU was just fully pegged,
> or the disk started thrashing?
>
> -Kenny
>
>
> On Tue, Jun 28, 2011 at 11:10 AM, Nicolai <nicolai-cialug at chocolatine.org>wrote:
>
>> On Mon, Jun 27, 2011 at 04:39:20PM -0500, Kenneth Younger wrote:
>> > I had a sever crush under load the other day, so much so that it made
>> SSHing
>> > into the box impossible. There's got to be a way to keep that admin tool
>> > available until the end, right?
>>
>> Hi Kenny,
>>
>> Indeed there is: it's called daemontools.  Combine it with resource
>> limits and you've got a reasonably stable system.
>>
>>  http://cr.yp.to/daemontools.html
>>
>> daemontools is from the same guy who wrote qmail, djbdns, and a long
>> list of other quality software.  I've been using it on almost every
>> system I've had in the last 11 years, and it has never crashed or
>> failed.  It's solid.
>>
>> Basically, it's a collection of tiny tools, one of which runs quietly in
>> the background, monitoring whatever daemons you want, and restarting
>> them if they happen to exit for some reason.
>>
>> IF you:
>>
>>  1. Monitor critical daemons using daemontools and
>>  2. Have resource limits properly configured,
>>
>> Then you're unlikely to see this problem re-occur.
>>
>> What exactly happened, though?  Knowing this will help guide you in your
>> tuning of resource limits.  And I'm pretty curious!
>>
>> Nicolai
>> _______________________________________________
>> Cialug mailing list
>> Cialug at cialug.org
>> http://cialug.org/mailman/listinfo/cialug
>>
>
>
>
> --
> Kenneth Younger III
> Founder, Sheer Focus Inc.
> e: kenny at sheerfocus.com
> p: (515) 367-0001
> t: @kenny <http://twitter.com/kenny>
>
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cialug.org/pipermail/cialug/attachments/20110628/aa7623be/attachment.html>