[Cialug] How to determine the cause of a Linux performance problem?

[Pixel // pinterface]

Greetings, Linux Experts!

I have a problem I am hoping you will be able to help me figure out.

I've got a VPS.  It runs a fair few things (LAMP stack; mail server;
etc.), but has always performed admirably.  A couple of days ago,
performance took a nosedive and I have been unable to recover.

Now, normally I'd undo whatever it was I did, but what I did between
performance being great and it sucking was: nothing.  I was sleeping.
Woke up, and everything was slow.  No software installs, no updates.
Just Zs.  (There was an ongoing IMAP login attack due to a
misconfiguration in fail2ban which prevented it from being stopped
automagically, but fixing that did not bring me down to my normal
levels.  Neither did rebooting or upgrading my kernel, FYI. :P)

Sadly, I can't attribute this to a sudden influx of HTTP traffic (it
hasn't changed), nor a sudden influx of spam^H^H^H^He-mail (that hasn't
changed either).

But my load average has jumped from being normally <0.1, to >0.6.  Which
wouldn't bother me so much except it's a very /slow/ 0.6.  My VMstat
graphs show a rather sizeable jump in processes under "I/O sleep", but
my fork rate and number of processes are both normal.

It appears (via top) that the processes bumping up the load average by
being in wait states are often "kjournald" and "flush-###", which makes
me suspect maybe there's a drive issue or excessive I/O contention from
neighboring VPSes.  Unfortunately, I'm not sure how I'd go about
figuring that out (well, short of migrating to another piece of hardware
and seeing if that fixes things), versus Some Other Thing which might be
wrong.

I'm hoping you guys can offer some pointers as to what might be going on
with my system, and how I would determine that.

Thanks!

-- 
pix