[Cialug] Guarantee SSH availability

[Matthew Nuzum]

It's an interesting question. I asked around at work and here are the
responses:

David Owen:
One method requiring a lot of work would be to set resource limits on
everything else on your system to guarantee that SSH always has
resources available.  This would also significantly under-commit or
under-utilize your system.

I've never actually had problems with SSH except it being pushed into
swap by a memory hog, and the system becoming generally unresponsive as
a result (theoretically, resource limits *would* help with that).

It seems like there ought to be a way to tackle it from the opposite
direction though, similar to locking pages in RAM.

-----------

David Owen:
> I've never actually had problems with SSH except it being pushed into
> swap by a memory hog, and the system becoming generally unresponsive
> as a result (theoretically, resource limits *would* help with that).
>
> It seems like there ought to be a way to tackle it from the opposite
> direction though, similar to locking pages in RAM.

In this scenario you have to consider that getting an interactive shell
is often the hard part.  Once you get past SSH auth on a loaded system
(reading in pubkeys off disk, etc) you then have to page in bash.  Once
you're at bash on a thrashing system, now you have to get a program like
/bin/ls in core (or at least page in the appropriate bash code to do
something like "echo *").

One approach (of supremely dubious general utility) is to make sure the
system has no swap.  In an out-of-memory situation, sometimes the si/so
load turns what should be an immediate kill into an hour-long pagefest.

Nick Moffitt

 -----------

On 28 June 2011 10:25, Nick Moffitt wrote:
> One approach (of supremely dubious general utility) is to make sure the
> system has no swap.  In an out-of-memory situation, sometimes the si/so
> load turns what should be an immediate kill into an hour-long pagefest.

When I first had an SSD laptop, I tried using no swap, and I found
that when it ran low on memory, it quite abruptly became totally
unusable, whereas with swap you get a bit of a softer landing where
the machine is slowing down but still responsive enough that you can
kill something.  I guess this is because it's having to kick out
program text that's still relatively hot, because it has nowhere to
put even relatively cool heap memory.  Of course swapping to SSD will
be a bit different to magnetic disks.

It seemed like Ubuntu out of the box does not have an OOM killer that
intervenes before the machine bogs down.  So I think that old advice
is not very good today.

Martin Pool

-----------

On Tue, 2011-06-28 at 09:25 +0000, Nick Moffitt wrote:
> David Owen:
> > It seems like there ought to be a way to tackle it from the opposite
> > direction though, similar to locking pages in RAM.
>
> In this scenario you have to consider that getting an interactive shell
> is often the hard part.  Once you get past SSH auth on a loaded system
> (reading in pubkeys off disk, etc) you then have to page in bash.  Once
> you're at bash on a thrashing system, now you have to get a program like
> /bin/ls in core (or at least page in the appropriate bash code to do
> something like "echo *").

Could you mark sshd as a realtime process?  Would bash then inherit the
priority? (and down the stack)

Ted Gould

-----------

I'll let you know if anything else comes up, but in essence, I went to some
experts and got a lot of head scratching. The implication is that there is
no easy solution to this.


On Mon, Jun 27, 2011 at 4:39 PM, Kenneth Younger <kenny at sheerfocus.com>wrote:

> I had a sever crush under load the other day, so much so that it made
> SSHing into the box impossible. There's got to be a way to keep that admin
> tool available until the end, right?
>
> A cursory search of google found nothing - of course I might be using the
> wrong keywords...
>
> -Kenny
>
> --
> Kenneth Younger III
> Founder, Sheer Focus Inc.
> e: kenny at sheerfocus.com
> p: (515) 367-0001
> t: @kenny <http://twitter.com/kenny>
>
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug
>
>


-- 
Matthew Nuzum
newz2000 on freenode, skype, linkedin and twitter

"Opportunity is missed by most people because it is dressed in overalls and
looks like work." -Thomas Edison
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cialug.org/pipermail/cialug/attachments/20110628/9e135ffc/attachment-0001.html>