[Cialug] Guarantee SSH availability

[Matthew Nuzum]

On Tue, Jun 28, 2011 at 12:06 PM, Kenneth Younger <kenny at sheerfocus.com>wrote:

> I'm not sure what exactly occurred. It's a basic LAMP server with an nginx
> proxy doing some caching there.
>
> Running this server has been humbling, because I used to think I knew what
> I was doing with regard to running a server - but this thing has given me a
> new respect for the subtle configuring that an expert can make.
>
> I know this isn't HUGE traffic, but it was getting 3000-4000 pageviews an
> hour at peak, which is the most traffic I've ever seen on a site I run. I
> think my problem is I just don't know how to tell what went wrong after the
> fact...
>
> How would tell if it ran out of memory, or the CPU was just fully pegged,
> or the disk started thrashing?
>
> -Kenny
>
>
I know the feeling.

PHP can be a bit harder to tune because the interpreter is part of Apache.
Therefore it's hard to estimate how many child processes your system can
support. If I was on a 512MB system I'd probably limit it to 20 and double
that for a 1G or bigger system. If you find you've got a lot of spare memory
under load then you can decide where to put it, often MySQL is a good
candidate.

I've seen situations like yours happen a few times in the last 18 months. In
each case it was out of file handles. If that happens you're toast. You
can't log in remotely to do anything about it.

What I've done, and has paid off in spades, is run Xen to create my own VPS
on a dedicated server. It means you need an extra IP address but it also
means you've got a guaranteed bit of a safety net. You can ssh into your xen
host (aka dom0) and then take over the console of your vm that's gone out of
control and try to fix it or at least hold its hand in its final hour. It
also means that a physical server can be divided up into smaller servers
which might be convenient. LXC may be a good alternative if you aren't able
to run Xen.


-- 
Matthew Nuzum
newz2000 on freenode, skype, linkedin and twitter

"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -Thomas Edison
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cialug.org/pipermail/cialug/attachments/20110628/20cab2e4/attachment.html>