[Cialug] Disk Performance and Measurement

[Nathan C. Smith]

I wanted to see what disk performance was like on an AoE SAN I have so I compared to it a Compaq DL 360 machine I have with (hardware) mirrored drives.  I thought the Compaq looked pretty good.

ServerA:~# hdparm -t --direct /dev/cciss/c0d0p1
/dev/cciss/c0d0p1:
 Timing O_DIRECT disk reads:  198 MB in  3.01 seconds =  65.73 MB/sec

With no basis of comparison, I assumed this was probably pretty respectable.
I then measured the AoE San performance.  (The Drives are RAID 5)

ServerB:~# hdparm -t --direct /dev/etherd/e200.0p1
/dev/etherd/e200.0p1:
 Timing O_DIRECT disk reads:  548 MB in  3.01 seconds = 182.26 MB/sec

"Wow, that pretty good," I thought in my best Strongbad voice..  But then I checked a machine running a 3Ware Card in a RAID (can't remember what level) array and got this:

ServerC:~# hdparm -t --direct /dev/sdb1
/dev/sdb1:
 Timing O_DIRECT disk reads:  1592 MB in  3.01 seconds = 529.53 MB/sec

Whoa! Can that be right? pretty amazing.  All these machines seem to perform acceptably to me for their jobs, but I realize that is a very subjective measurement, and there may be things like caches involved to help some of the other systems perform better when they are not doing disk tests.  Do any of you have access to a truly enterprise class SAN that have run similar tests to get an idea of what awesome performance should look like?  This is strictly read too, I don't want to completely open this can of worms.  ;-)

What is acceptable performance?  I realize that is a really open-ended question because it depends on the application.  None of my machines are serving big databases, mostly just shuffling files.  I did a straight PATA disk and got 132 MB in  3.00 seconds =  43.99 MB/sec so I'm thinking anything better than that is an improvement.  But I'd like to learn about the experiences of others.

-Nate