[Cialug] Attack troubleshooting?

[Josh More]

I really wish I had more time to respond to this, but events are conspiring against me.  Here's a scattershot answer:

---pre-analysis---


1) What Ken said is correct.  If the system holds data that, if leaked, could result in legal action, stop and get a professional.  A shockingly large number of cases are blown because untrained people breach chain of custody procedure.

2) Following on what Ken said, if the system is a client's system that you were responsible for, stop and get professional help.  Having a trained third party in the mix can make the difference between a client suing you and their continuing to be a client of yours.

3) If neither #1 nor #2 apply, determine if the system can handle downtime.  If it can, power it down and take an image of the system with dd.  Do NOT use clonezilla.  You need a full binary copy including unused hard drive space.  Then clone the COPY with CloneZilla so you have something you can work on.

4) If the system cannot handle downtime, check and make sure that neither 1 nor 2 apply.  If you get here, you probably have a false assumption somewhere.


---analysis---

Take notes.  Take lots and lots of notes.  If you are wrong about 1 or 2 you'll need them.

On either the copy of a copy or the live system (shudder), look at the logs first... both the logs of the system and the logs of the firewall it was (hopefully) behind.  Be sure to match date/time stamps.  If you're using NTP, this will be easier.  If you're not using NTP, this is a good lesson for next time..  See what jumps out at you.  

If nothing does, load chkrootkit and rkhunter and see what they tell you.  They'll be a lot of false positives, but each one should be looked for.  Scan the system from the outside using NMAP.  Iterate through all ports.  Do *not* trust lsof or netstat.  They are likely trojaned (as I see Dave has responded).  They will lie to you.  The tools that chkrootkit and rkhunter will use will probably lie too, but they're better.

Verify the entire system against the RPM database.  This may lie to you too and will almost certainly be full of false positives, but it's gotta be done.

Lastly, analyze the disk offline.  Compute MD5 sums and compare them to the RPM database (takes a bit of scripting).  Or mount the disks and chroot them to do the same thing and hope that the rootkit was kernel-level only.

There's a LOT more to it, but this should at least get you started.  I cannot stress enough, though, the importance of getting someone else involved if there's a chance that this is more than you can handle.  Hard stops include credit card numbers, banking details, social security numbers, general financial information and general health information. 

Extremely hard stops involve child pornography or terrorist materials.  If you encounter these, step back from the system and call the FBI immediately.  I am not exaggerating or overly stressing this point.  Immediately means immediately.  Do not touch a single key on the system if you encounter this sort of data.  Yes I have stories.  No I cannot share them.  Trust me, please.

If you need help, let me know off list.  I have a lot of engagements stacking up this week, but if it's an emergency I can slide things and help a bit.  (I trust you'll understand that in cases like this, it wouldn't be free, but like Dave and Dan do with their businesses, I can probably swing some sort of LUG discount.)


Josh More | Senior Security Consultant - CISSP, GIAC-GSLC Gold, GIAC-GCIH
Alliance Technologies | www.AllianceTechnologies.net
400 Locust St., Suite 840 | Des Moines, IA 50309
515.245.7701 | 888.387.5670 x7701

Blog: Don't just blame the bad guys, it's your fault too
http://www.alliancetechnologies.net/blogs/morej

How are we doing? Let us know here:
http://www.alliancetechnologies.net/forms/alliance-technologies-feedback-survey

________________________________________
From: cialug-bounces at cialug.org [cialug-bounces at cialug.org] on behalf of L. V. Lammert [lvl at omnitec.net]
Sent: Monday, February 28, 2011 22:23
To: Central Iowa Linux Users Group
Subject: [Cialug] Attack troubleshooting?

Just had a Centos 5.5 box come under some sort of attack, .. it appeqrs
that there is something ON the box that was trying to connect to an
outside IP.

Any thoughts on how to isolate the cause? I finally got into the box by
playing with the firewall, but don't see any logins or anything untoward
in ps.

        Lee
_______________________________________________
Cialug mailing list
Cialug at cialug.org
http://cialug.org/mailman/listinfo/cialug