To be prepared to respond to rootkit infections, you need several items. Different rootkits require different response techniques and warrant different response tools. The following are some standard items that should be in every toolkit:
• Statically compiled binaries
• Packet capture software
• Port scanning software
Chapter 5: Data Networks Security
117
• Incident response/forensics boot disk
• Enterprise forensic software
• Rootkit detection software
Obviously not all of these items are required, but they will all help. All of the data from your gold image baseline, as specified previously, should also be in the toolkit.
Ensure ahead of time that the gold image data is maintained, uncompromised, and contains all of the latest data from the system.
The most difficult part is having a current gold image baseline. It generally takes a significant investment of time to keep these maintained and requires a commensurate amount of management to ensure that it happens. If a current image is not available, it is too late to prepare. At this point, to track down the rootkit, you may have to rely on one of the various downloadable rootkit detection tools. These tend to have limitations, however.
Statically Compiled Binaries Since user-mode Linux rootkits usually function through some kind of file substitution or library modification, it is a very good idea to have a set of precompiled binaries contained on a CD-ROM. Having these will help you gather information from a live, infected system.
It is not always advisable to pull the plug on an infected system. Sometimes, however, you need to collect volatile data from the system before shutting it down for further investigation. If the system has suffered some kind of file substitution, none of the output from any of the native utilities, or those that depend on the system libraries, can be trusted.
Having statically compiled binaries available allows for successful and accurate data extraction from a machine that has been compromised by a user-mode rootkit. Table 5-2 lists statically compiled binaries in a well-stocked toolkit. Depending on the situation, you may need more.
Packet Capture Software For more information about the network activity of a compromised box, starting a packet capture of the machine’s activity at the very beginning of an investigation can be very enlightening. A simple TCPDump packet capture is adequate and can be imported into nearly all traffic analysis tools. Assuming the traffic was not encrypted, information can be acquired about both sides of the traffic and possibly what occurred during their connection.
Also, performing a packet capture may enable you to see or reconstruct traffic that is not displayed well in netstat on a nonrooted system, such as ICMP tunnels. While not used in the majority of compromises, ICMP tunnels are used quite frequently for firewall evasion and can be hard to detect.
Regardless, capturing network traffic going to or coming from the box can provide direction on where to look next. This traffic could lead back to the attackers or to other compromised systems on the network.
In addition, file transfers may be occurring that give you some idea what the attackers want. The packet captures may be able to serve in some sort of evidentiary capacity. This assumes that it is acceptable to allow the traffic to proceed in an effort to collect evidence for later purposes.
In most situations, companies just want to stop the bleeding and are uninterested in gathering data purely for evidentiary purposes. Law enforcement agencies are often not very quick to investigate Internet-related crimes, so gathering evidence may be a futile pursuit.
Port Scanning Software Rootkits usually contain some kind of backdoor, enabling access back into the system. Subsequently, you cannot trust the netstat output when it reveals the listening port. Exceptions to this limitation include having a compiled version of netstat if the rootkit was a file substitution rootkit or if the attackers never hid the backdoor port.
It is surprising how often attackers overlook basics and forget to enable the network traffic concealment options available in a rootkit. Forgetting to enable and configure all of the features of rootkits happens in a significant number of cases.
Regardless of the circumstances, the extent of the rootkit at the beginning of the investigation, when you have the best opportunity to gather volatile data, is generally
arp dd free logname pgrep sdiff sum unexpand
basename df gawk ls pinky sed sync uniq
bash diff gcc md5sum pkill seq sysctl unlink
cat diff3 ginstall mkdir plipconfi g setuidgid t uptime
chgrp dir grep mkfi fo pmap sha1sum tac users
chmod dircolors groups mknod pr shred tail v
chown dirname head mv printenv skill tar vdir
chroot du hostid nameif printf slabtop tee vmstat
cksum echo hostname netcat ps slattach tload w
cmp env id netstat ptx sleep top watch
comm expand ifconfi g nice pwd snice touch wc
cp expr join nl rarp sort tr who
csplit factor kill nohup readlink split true whoami
cut false less od rm stat tsort yes
date fmt link paste rmdir stty tty
dcgen fold ln pathchk route su uname
Table 5-2 Useful Statically Compiled Binaries
Chapter 5: Data Networks Security
119
not known. Therefore, it is a good practice to gather as much data as possible to alleviate fears that something might be missed that could be used later.
So, if you suspect a rootkit on the system, use an external port scanner (such as Nmap) and enumerate the listening ports, both TCP and UDP, from an external view. Compare the scanner’s results with the output of the netstat command with the -na switches.
If the port scanner identifies ports that are not shown in netstat output, those are likely the ports being used and hidden by the rootkit.
Finding a hidden port on the system may help provide more information about the rootkit, particularly if you can connect to it, interact with it, and possibly deactivate it.
However, most rootkits have the ability to protect the connection with a password. If you encounter this, try various default rootkit passwords in case the attackers never changed it. Also, like any other network logon, rootkit logons are subject to dictionary and brute-force attacks.
Incident Response/Forensics Boot Disk A fantastic number of tools and options for incident response and computer forensic boot disks are available, many of which are free and designed to analyze Linux systems. These utilities can be used for anything from performing password recovery to a full-fledged forensic analysis. Helix is a very robust example of a boot disk providing these feature sets.
Once you’ve made the decision to shut down a machine, a bootable Linux IR or Forensic distro can be useful for digging into the system and determining the level of compromise that occurred. These utilities determine what files were altered and provide a way to correct any issues.
While it is true that given enough time, you can locate and fix the modifications to a system, the process will be greatly simplified by preparing well ahead of time. All of the nonvolatile data pertaining to physical files and configurations, users, groups, user-group associations, and so on, assembled in the gold image baseline, will turn out to be very handy when performing an analysis.
Going through this process instead of re-imaging the drive on the affected machine can be useful for four reasons.
• First, sometimes it is better to repair the system than to restore it from a backup.
This is especially true in a situation where a signifi cant amount of data loss could be avoided.
• Second, backup images are often quite old and substantial system changes have been made between the time the image was created and the time the incident happened. While this is indicative of poor planning and is something that should be audited and tested, recovering the system manually is a way to fi x the problem and leave the system in a better state than if it had been recovered from backup.
• Third, a backup image may not be available for the system and recovering it manually is your only hope. Sadly, this situation is very common and one of the main reasons for using this kind of software.
• Fourth, and by far probably the most common use of IR and forensic software, is a desire to understand what happened, how it can be mitigated, and how it can be prevented. You may also be curious to fi nd any cool tools left behind that you can then use to be beef up your toolkit.
Enterprise Forensic Software Enterprise forensic software can be of tremendous value when responding to an incident, especially a rootkit. It can search the drives and files on the compromised machine, as if the machine were physically present and being analyzed in person. Good enterprise forensic software, such as EnCase or FTK, can allow access to volatile data and even provide an ability to modify and remediate the infected system.
This simplifies the process of gathering and analyzing data (especially from multiple machines) and determining the present state of the machine(s), plus it provides the option to fix the problem remotely. If the gold image baseline(s) of the compromised machine(s) were created beforehand, enterprise forensic software will enable a comparison of the current state of the machine with the data in the gold image and determine which files are compromised, which processes should not be running, which ports should not be listening, which files should not be in use, which modules should not be loaded, and so on.
If any tool is going to allow surgical recovery of a machine, without performing a reinstallation or recovering from an older image, an enterprise forensic software package with remediation capabilities will do the trick. It is very important to mention that not all enterprise forensic software packages are created equally. Determine which packages have the greatest benefit for a particular distribution and server application. Each package has strengths in certain key areas and weaknesses in others.
Just like any tool, bench testing should be done to see how the software performs, how it operates, and how or if it will be useful against a rootkit. The key area here is that the software should implement its own kernel module and should access all data through its own module and from a physical perspective, when possible.