[Cialug] Guarantee SSH availability

[Jeff Chapin]

I'm going to stop lurking on this thread long enough to thank everyone,
both for the great question (obviously it has no trivial answer) and all
the in-depth answers people are putting effort into posting.

Jeff

On 06/28/2011 09:32 AM, Matthew Nuzum wrote:
> It's an interesting question. I asked around at work and here are the
> responses:
>
> David Owen:
> One method requiring a lot of work would be to set resource limits on
> everything else on your system to guarantee that SSH always has
> resources available.  This would also significantly under-commit or
> under-utilize your system.
>
> I've never actually had problems with SSH except it being pushed into
> swap by a memory hog, and the system becoming generally unresponsive as
> a result (theoretically, resource limits *would* help with that).
>
> It seems like there ought to be a way to tackle it from the opposite
> direction though, similar to locking pages in RAM.
>
> -----------
>
> David Owen:
> > I've never actually had problems with SSH except it being pushed into
> > swap by a memory hog, and the system becoming generally unresponsive
> > as a result (theoretically, resource limits *would* help with that).
> >
> > It seems like there ought to be a way to tackle it from the opposite
> > direction though, similar to locking pages in RAM.
>
> In this scenario you have to consider that getting an interactive shell
> is often the hard part.  Once you get past SSH auth on a loaded system
> (reading in pubkeys off disk, etc) you then have to page in bash.  Once
> you're at bash on a thrashing system, now you have to get a program like
> /bin/ls in core (or at least page in the appropriate bash code to do
> something like "echo *").
>
> One approach (of supremely dubious general utility) is to make sure the
> system has no swap.  In an out-of-memory situation, sometimes the si/so
> load turns what should be an immediate kill into an hour-long pagefest.
>
> Nick Moffitt
>
> -----------
>
> On 28 June 2011 10:25, Nick Moffitt wrote:
> > One approach (of supremely dubious general utility) is to make sure the
> > system has no swap.  In an out-of-memory situation, sometimes the si/so
> > load turns what should be an immediate kill into an hour-long pagefest.
>
> When I first had an SSD laptop, I tried using no swap, and I found
> that when it ran low on memory, it quite abruptly became totally
> unusable, whereas with swap you get a bit of a softer landing where
> the machine is slowing down but still responsive enough that you can
> kill something.  I guess this is because it's having to kick out
> program text that's still relatively hot, because it has nowhere to
> put even relatively cool heap memory.  Of course swapping to SSD will
> be a bit different to magnetic disks.
>
> It seemed like Ubuntu out of the box does not have an OOM killer that
> intervenes before the machine bogs down.  So I think that old advice
> is not very good today.
>
> Martin Pool
>
> -----------
>
> On Tue, 2011-06-28 at 09:25 +0000, Nick Moffitt wrote:
> > David Owen:
> > > It seems like there ought to be a way to tackle it from the opposite
> > > direction though, similar to locking pages in RAM.
> >
> > In this scenario you have to consider that getting an interactive shell
> > is often the hard part.  Once you get past SSH auth on a loaded system
> > (reading in pubkeys off disk, etc) you then have to page in bash.  Once
> > you're at bash on a thrashing system, now you have to get a program like
> > /bin/ls in core (or at least page in the appropriate bash code to do
> > something like "echo *").
>
> Could you mark sshd as a realtime process?  Would bash then inherit the
> priority? (and down the stack)
>
> Ted Gould
>
> -----------
>
> I'll let you know if anything else comes up, but in essence, I went to
> some experts and got a lot of head scratching. The implication is that
> there is no easy solution to this.
>
>
> On Mon, Jun 27, 2011 at 4:39 PM, Kenneth Younger <kenny at sheerfocus.com
> <mailto:kenny at sheerfocus.com>> wrote:
>
>     I had a sever crush under load the other day, so much so that it
>     made SSHing into the box impossible. There's got to be a way to
>     keep that admin tool available until the end, right?
>
>     A cursory search of google found nothing - of course I might be
>     using the wrong keywords...
>
>     -Kenny
>
>     -- 
>     Kenneth Younger III
>     Founder, Sheer Focus Inc.
>     e: kenny at sheerfocus.com <mailto:kenny at sheerfocus.com>
>     p: (515) 367-0001
>     t: @kenny <http://twitter.com/kenny>
>
>     _______________________________________________
>     Cialug mailing list
>     Cialug at cialug.org <mailto:Cialug at cialug.org>
>     http://cialug.org/mailman/listinfo/cialug
>
>
>
>
> -- 
> Matthew Nuzum
> newz2000 on freenode, skype, linkedin and twitter
>
> "Opportunity is missed by most people because it is dressed in
> overalls and looks like work." -Thomas Edison
>
>
>
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> http://cialug.org/mailman/listinfo/cialug
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cialug.org/pipermail/cialug/attachments/20110628/0175c72b/attachment.html>