Chapter 5: Systems and subversion

5.1 Introduction

In the last two chapters we looked at information that can be found in file systems. Such information is static in nature, and is typically examined after the fact. In this chapter we turn our attention from static data to the more dynamic world of running code, and look at system state in real time.

After an overview of the basic elements of computer system architecture, we walk through the system life cycle from start-up to shutdown, and present some measurements of the complexity of today's operating systems. We cannot fail to observe how this complexity introduces opportunities for subversion.

Before we turn to system subversion, we recapitulate the essential kernel and process concepts that are often involved with subversion. There is a lot more to kernels and processes than we can describe in the context of this book, and the interested reader is referred to any good book on UNIX and UNIX system administration [Nemeth, 2001].

Subversion is the subject of the second half of this chapter. We present the mechanisms behind several generations of so-called rootkit software, and show examples of their detection. This is an ongoing cat and mouse game of making observations, subversion of observations, and detection of subversion. And since none of the popular systems today can offer strong security guarantees, this cat and mouse game is likely to continue until a radical change is made to system architecture.

5.2 The standard computer system architecture

Over the past 30 years the basic architecture of computer systems has not changed significantly, despite major advances in hardware and in operating systems. The main principle that underlies this architecture is separation of concerns. We distinguish two main levels: process and system. At the process level, software runs in an environment that is relatively independent of hardware and operating system details. At the system level, hardware and software provide the environment in which processes execute. Within each level we distinguish some additional structure as shown in figure 5.1.

Executable program
System library
Operating system kernel

Figure 5.1. General relationship between hardware, operating system kernel, system libraries and executable program files.

Going from top to bottom we encounter the following layers:

The benefits of this architecture are portability and simplicity. Portability means that the exact same application software can be used on multiple versions of similar operating systems, and on multiple configurations of similar hardware. Simplicity means that processes do not have to be aware that they share one machine with other processes. The operating system deals with all the complexities of resource management, and the hardware does the work.

5.3 UNIX System life cycle from startup to shutdown

Having introduced the basic layers of the computer system architecture we now take a bottom-up approach from hardware to executable programs, and watch how different layers take control of the machine at different points in time. Figure 5.2 gives a simplified picture of the entire system start-up procedure.

When a computer system is powered up, resident firmware (often called BIOS, EEPROM, etc.) performs a hardware self test and some low-level configuration. This includes finding out the type and capacity of installed random-access memory, locating additional resident firmware in, for example, disk or network interface controllers, configuring plug-and-play devices, and so on. Meanwhile, intelligent peripherals may execute their own power-up self test and configuration sequences.

After completion of the power-up self test and low-level configuration, the resident firmware loads a boot program from disk or from the network, and gives it control over the machine. The boot program loads either the operating system kernel or the next stage boot program. Boot programs and operating system kernels often have their own configuration parameters as discussed in a later section.

The kernel probes the hardware for devices such as disk or network interface controllers and may load kernel modules to access and initialize those devices. Once the kernel completes its initialization, it creates the init process, and passes control to it. From this point onwards the kernel becomes dormant. Kernel code executes only in response to hardware interrupt events and in response to system call requests from processes.

Figure 5.2: simplified process genealogy of a typical UNIX system.

The init process executes the system startup and shutdown procedures. These procedures are implemented by a multitude of "rc" files (shell command files) that are usually located under the /etc or /sbin directory. The startup shell command files mount file systems, configure kernel modules and kernel parameters (more about those in a later section), configure network interfaces, and start daemon processes, including the window system that presents the graphical console login window. The init process also controls non-graphical logins via hard-wired and dial-up terminal ports. The shutdown shell command files stop specific processes in a controlled manner, before all the left-over processes are forcibly terminated.

An important function of init is to implement the so-called run levels. With BSD-like systems these are called single-user (administrative access only) and multi-user. Solaris and Linux run levels are called S (single user) and 0-6, where 0 and 6 are for system shutdown and 1-5 are for increasing levels of service. For example, different run levels may enable multi-user logins, network logins, graphical console logins, or file sharing; the exact details of run level functionality differ per system.

5.4 Case study: system startup complexity

In the previous section we talked about multitudes of files that are involved with startup and shutdown sequences. To find out how complex these sequences are, we measured the number of different files that systems attempt to accesses during boot procedures. The measurements were done with generic i386 Solaris and RedHat Linux systems, and include booting up system configurations with and without graphical user interface. For system configurations with a graphical user interface, the file counts include one login/logout with the default user interface. The results are shown in table 5.1.

In order to get these numbers we recorded all file access requests by all init child processes in real time, including file accesses that happened while file systems were still mounted read-only. Tools that can monitor file access and other system call requests are discussed in chapter 6. Playing with system startup sequences is fraught with peril, especially when playing games with a critical process such as init. It was very easy to make a mistake that rendered the entire system unusable. Our measurements were possible thanks to VMware's virtualization software (see chapter 6). Whenever we made a mistake, we simply discarded all our changes to the file system and tried again.

System type and configurationProgram files Other files Non-existent files

RedHat 4.1 boot, no GUI81 290 217
Redhat 5.2 boot, no GUI86 494 289
Redhat 6.1 boot, no GUI76 639 262
Redhat 6.1 boot and default GUI login/logout107 1667 1090

Solaris 2.5.1 boot, no GUI65 250 229
Solaris 7.0 boot, no GUI77 344 273
Solaris 7.0 boot and default GUI login/logout150 1153 1986

Table 5.1: Number of different file names accessed while booting generic Linux and Solaris systems, with and without a graphical user interface (GUI). The RedHat and Solaris counts include all accesses after init startup. Program files include executable files as well as run-time libraries and other files with executable machine code.

Two things stand out in the table. First is that the graphical user interface surpasses in complexity the entire system that it runs on. The second thing that stands out is the large number of requests involving non-existent pathnames. One source of non-existent pathnames is sloppiness in the form of default command PATH searches. A second source is a different form of sloppiness: many startup scripts probe the file system in order to find out what is installed. Finally, a major source of non-existent pathname lookups is the backwards compatibility support for legacy pathnames that are built into program and configuration files.

What the table does not show is the large numbers of attempts to access the same pathname repeatedly, regardless of whether the pathname exists. This is a symptom of inefficiency, and all the little inefficiencies together ensure that system startup times do not improve despite continual improvements in hardware performance.

Our little study shows that there exist literally hundreds if not thousands of opportunities to subvert the integrity of a system without changing any file: it is sufficient to add one extra file in an innocuous place so that it is accessed during, for example, system start-up. This kind of subversion can be prevented to some extent by requiring that files have valid digital signatures. At the time of writing, such features are still in the form of experimental add-ons for UNIX systems [van Doorn, 2001; Williams, 2002].

5.5 Kernel configuration mechanisms

After the overview of architecture and system life cycle we now take a closer look at individual software layers of the system architecture. First we'll take a look at the kernel level. The purpose of the operating system kernel is to make computer hardware more useful to application programs, just like the purpose of application programs is to make computer systems more useful to human beings. While the application program interfaces provided by UNIX kernels and system libraries are relatively standardized by organizations such as ISO, IEEE and XOPEN, there can be large differences between kernel internals. We will focus on the common elements in the architecture of typical UNIX kernels as shown in figure 5.3.

process layer
system calls
low-level code and device drivers
hardware layer

Figure 5.3: Major subsystems of a typical UNIX kernel. The adjacent hardware and process architecture layers are shown for context.

When a machine is turned on, built-in firmware configures the hardware, and loads a boot program from disk, or from a network server. Depending on how powerful this boot program is, one or more additional boot programs may be needed to get the kernel up and running. The general sequence of events is described in the system manual pages: Solaris boot(1M), Linux boot(7), or FreeBSD boot(8). Boot programs are controlled by configuration parameters, and sometimes they configure initial kernel configuration parameter values. Although the bootstrapping stage represents only a minuscule portion of the system life cycle, its integrity is critical for the integrity of the entire system. For an implementation of secure booting in the IBM PC environment, see the work by Arbaugh and others [Arbaugh, 1997].

System Boot loader configuration Kernel file name Kernel configuration

Solaris /etc/bootrc (x86 platform)
firmware (sparc platform)
/kernel/genunix /etc/system
Linux /boot/grub/grub.conf
/boot/vmlinuz /etc/sysctl.conf
FreeBSD /boot.config

Table 5.2: Typical boot loader and initial kernel configuration information.

As is to be expected, there is great variation in the way kernels are configured. Linux and FreeBSD systems have a sysctl command that gives read/write access to kernel configuration parameters and other data; the Linux /proc pseudo file system also gives read/write access to a subset of those parameters and other data. Solaris has the ndd command for reading or writing parameters and other data that lives in the drivers that implement the IP protocol family, while other Solaris kernel parameters may be set at run time with the mdb or adb commands. Listing 5.1 shows only a few of the more than 700 kernel configuration parameters of a FreeBSD kernel.

freebsd50% sysctl -a 
kern.ostype: FreeBSD
kern.osrelease: 5.0-RELEASE
kern.osrevision: 199506
kern.version: FreeBSD 5.0-RELEASE #0: Thu Jan 16 22:16:53 GMT 2003

kern.maxvnodes: 4182
kern.maxproc: 532
kern.maxfiles: 1064
kern.argmax: 65536
kern.securelevel: 1
. . . 728 more lines omitted . . .

Listing 5.1: Examples of FreeBSD kernel parameters.

Loadable kernel modules are chunks of executable code and data that can be assimilated into a running kernel at any point during its life cycle. Once a module is loaded, its code runs as part of the kernel. With monolithic kernels, this means the module's code has access to everything inside and outside the kernel. Kernel modules are used to implement functionality within all of the major kernel subsystems shown in figure 5.3: from device drivers, file systems, and network protocols, up to system calls that provide new features to processes. Modules are loaded while the kernel initializes, and under the control of init startup procedures. Table 5.3 shows typical commands to manipulate the status of loadable kernel modules.

System Commands
Solaris modload modunload modinfo
Linux insmod rmmod lsmod
FreeBSD kldload kldunload kldstat

Table 5.3: Typical commands to load, unload and query the status of loadable kernel modules.

Many UNIX systems have relatively monolithic kernels: most of the code is permanently linked into the kernel file, and only a handful modules are loaded dynamically. Solaris takes the opposite approach: more than a hundred modules are loaded dynamically. Even the process scheduler is loaded as a kernel module, as shown in listing 5.2.

solaris9% modinfo
 Id Loadaddr   Size Info Rev Module Name
  5 fe925000   3e92   1   1  specfs (filesystem for specfs)
  7 fe92a275   2fda   1   1  TS (time sharing sched class)
  8 fe92cdcb    888   -   1  TS_DPTBL (Time sharing dispatch table)
 10 fe92ce43    208   -   1  pci_autoconfig (PCI BIOS interface 1.38)
 11 fe92cfc7  27d6e   2   1  ufs (filesystem for ufs)
 . . . 100 lines omitted . . .
131 deeae12e    ae4  23   1  ptm (Master streams driver 'ptm' 1.4)
132 fe9610f1    be4  24   1  pts (Slave Stream Pseudo Terminal dr)
133 feaa37b7   12ae  17   1  ptem (pty hardware emulator)

Listing 5.2: Examples of Solaris 9 kernel modules (ix86 platform).

The convenience of loadable kernel modules also has a darker side. Because their code has access to everything inside and outside a monolithic kernel, they are also powerful tools in the hands of intruders. As we will see later, kernel modules exist for hiding traces of intrusion (including the intruder's kernel modules), and for controlling nearly invisible back doors that give privileged access to intruders. We will return to the issue of corruption and detection at the end of this chapter.

5.6 Protecting forensic information with kernel security levels

Many kernel configuration parameter settings affect the security of a system. One kernel configuration parameter of particular interest is the kernel security level. This is a standard feature on 4.4 BSD descendant systems, and is available as add-on feature for Linux. As the security level increases, functionality is reduced. The security level can be raised at any time with, for example, the sysctl command, but only the init process can lower the security level while the system is in single user mode. Table 5.4 summarizes the semantics as defined with 4.4BSD. Some systems support additional security levels and/or additional restrictions. For precise details, see the securelevel(7) manual page, or its equivalent.

Level Restrictions

-1 Permanently insecure mode. Always run the system in level 0 mode, and do not raise the security level when changing from single-user mode to multi-user mode.
0 Insecure mode, normally used while booting the system. There are no additional restrictions beyond the usual file and system call access controls.
1 Secure mode, normally used after the system switches to multi-user mode. Immutable and append-only file attributes may no longer be turned off, open for writing is no longer allowed with disk devices that contain mounted file systems, as well as memory devices, and kernel modules may no longer be loaded or unloaded.
2 Highly secure mode. Even unmounted disks may no longer be opened for writing.

Table 5.4: Typical 4.4BSD security level settings and restrictions.

These security features can be valuable for the protection of forensic information. An append-only flag protects logfiles against change of already written content, and an immutable flag protects a file against any change including renaming (although such protection means little when a parent directory can still be renamed). Disallowing open-to-write of disk devices protects file systems against tampering that would otherwise be hard to detect. And none of these measures would be effective unless write access is revoked to the kernel memory or main memory devices, because it would be trivial to turn off the protection offered by security levels.

The security level mechanism should not be confused with the discretionary or mandatory access control mechanisms that are built into many operating systems. The purpose of access control is to enforce a policy. The problem with policies is that they may be changed at any point in time by a suitably authorized process. The purpose of security levels, on the other hand, is to revoke access unconditionally; once a system enters security level 1, there is no mechanism that grants permission to load kernel modules or to write to raw memory or to mounted disk devices, except for returning the system to single-user mode or rebooting the system into a less secure configuration. Thus, security levels are not impenetrable, but they add a barrier against compromise that cannot be implemented with access control alone.

5.7 Typical process and system status tools

Each UNIX system comes with its own assortment of process and system status monitoring tools. Most tools look at one or two particular aspects of the system: process status, network status, input/output, and so on. Because of the large variation between tools and between systems we will introduce only a few representative tools and refer the reader to the system documentation for information about individual platforms. Process and system status tools can reveal signs of intruder activity such as files, processes or network connections. This makes them a prime target for subversion by intruders. We will discuss subversion of status tools later in the chapter.

The ps command is the basic process status tool. Many systems provide multiple implementations and/or user interfaces; the Linux version even has more than a dozen personalities. In our examples we use the BSD user interface because it provides information that is not available via some of the other user interfaces. Solaris has a BSD compatible ps command in /usr/ucb/ps.

By default, the ps command tries to produce nice output, but that means a lot of information is suppressed or truncated. For example, in order to see the entire command line of a process we need to specify one or more w options, and we need to specify the e option to display information in a process's environment. This is a list of name=value pairs that is inherited from the parent process. Linux and FreeBSD require super-user privileges in order to view environment information from other user's processes. Solaris currently does not impose this restriction.

A process environment may reveal whether the process was started in the regular manner or not. For example, system processes started at boot time tend to have few if any environment settings. If a system process such as inetd (the process that manages incoming connections for many common network services) was restarted by hand, then the environment could give away useful information such as the remote user's origin, working directory information, etc., as shown with the following command:

$ ps -aewww
. . .
 6597  ??  Ss     0:00.01 PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/game
s:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin:/root/bin MAIL=/var/ma
il/wietse BLOCKSIZE=K USER=wietse LOGNAME=wietse HOME=/root SHELL=/bin
/csh SSH_CLIENT= 841 22 SSH_TTY=/dev/ttyp0 TERM=xterm PWD
=/home/wietse XNLSPATH=/usr/X11R6/lib/X11/nls XKEYSYMDB=/usr/X11R6/lib
/X11/XKeysymDB XAUTHORITY=/home/wietse/.Xauthority /usr/sbin/inetd -wW
. . .

The lsof (list open files) tool lists processes with all their open files, network ports, current directories, and other file system related information [Abell, 2004]. This very useful program brings together information that is often scattered across several different tools. Because the output can reveal a lot about what a process is doing, many lsof implementations give little or no information about processes that are owned by other users. To examine processes owned by other users, lsof needs to be invoked by the super-user.

Below is a typical sample of output for an OpenSSH server process that is waiting for connection requests. OpenSSH is an implementation of the SSH protocols for encrypted remote logins [OpenSSH, 2004].

# ps ax | grep sshd
  186  ??  Is     0:01.17 /usr/sbin/sshd
39288  pb  R+     0:00.00 grep sshd
# lsof -p 186
sshd    186 root cwd VDIR  13,131072      512     2 /
sshd    186 root rtd VDIR  13,131072      512     2 /
sshd    186 root txt VREG  13,131072   198112 15795 /usr/sbin/sshd
. . .10 run-time library object files omitted. . .
sshd    186 root  0u VCHR        2,2      0t0  7955 /dev/null
sshd    186 root  1u VCHR        2,2      0t0  7955 /dev/null
sshd    186 root  2u VCHR        2,2      0t0  7955 /dev/null
sshd    186 root  3u IPv6 0xd988e720      0t0   TCP *:ssh (LISTEN)
sshd    186 root  4u IPv4 0xd988e500      0t0   TCP *:ssh (LISTEN)

In the output, cwd is the current directory, rtd the root directory which in this case is the regular file system root, txt is the executable file, and 0..4 are open files and sockets. For each entry, lsof gives additional information such as the type (directory, regular file, character special device, socket for IP version 4 or 6), and other identifying information such as the device and inode number or the address of the socket control block.

Of particular interest is the lsof -i option which shows all processes with active network ports. The output can reveal processes that aren't supposed to have open network connections, and that could be a sign of trouble. The example below shows a shell (command interpreter) process that is attached to TCP port 21. Normally, this port is used by FTP server processes to receive commands and to report status results. In the example, is the address of the local machine, and is the address of an attacking machine. Note: the -> arrow does not imply that the connection was initiated by the local machine; the lsof command has no information about which end of a connection is the client or server.

# lsof -ni 
. . .
sh      39748 root 0u IPv4 0xd9892b60 TCP>
sh      39748 root 1u IPv4 0xd9892b60 TCP>
sh      39748 root 2u IPv4 0xd9892b60 TCP>
. . .

Output like this is a sure sign that someone exploited an FTP server vulnerability in order to spawn a shell process. These exploits are popular because the shell process inherits full system privileges from the FTP server process.

The findings of lsof can be checked against the findings of other tools that also look at process and/or file/network information, such as the netstat command. The following command shows the status of all network ports and of all network connections. We omit all but the connection that corresponds with the FTP server exploit.

# netstat -nf inet
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address (state)
. . .
tcp4       0      0  ESTABLISHED
. . .

With Linux systems one would specify netstat -n --inet for an equivalent result. Talking of Linux, its netstat command has the very useful -p option to display process ID and process name information.

As we see, there is significant overlap between tools. When only an individual tool is compromised, the output from other tools may reveal the inconsistency. This is why intruders often replace multiple system utilities as part of their cover-up operation. However, system status tools are not really independent. Ultimately, all tools rely on a common source of information, the kernel. When the kernel is compromised, all tools may fail, and other techniques are needed to expose the compromise. We will return to this at the end of the chapter.

5.8 How process and system status tools work

All process and system status tools get their information from the running kernel. Historical UNIX systems made only a limited amount of process and system status information available via well-documented system calls. Most information was obtained by directly accessing poorly documented data structures in kernel memory via the /dev/kmem pseudo device.

Modern UNIX systems make some process and system information accessible via the /proc pseudo file system. Each process has a subdirectory /proc/pid that is named after the numerical process ID. Each subdirectory contains an assortment of pseudo files for different process attributes; table 5.5 gives a few examples. Specific details of the proc file system are described in the system manual: FreeBSD procfs(4), Linux proc(5), and Solaris proc(4).

With the /proc pseudo file system, process attributes are accessed by opening the corresponding pseudo file. Process status reporting tools read status information, and programs such as debuggers manipulate processes by writing and reading control information (debuggers are discussed in chapter 6). Linux and FreeBSD /proc files are mostly text based and can be accessed in meaningful ways with the cat or echo commands, while Solaris uses binary data structures that require specialized tools as described in the proc(1) manual page.

Process attribute Solaris FreeBSD Linux

program file /proc/pid/object/a.out /proc/pid/file /proc/pid/exe
process memory /proc/pid/as /proc/pid/mem /proc/pid/mem
memory map /proc/pid/map /proc/pid/map /proc/pid/maps
command line /proc/pid/psinfo /proc/pid/cmdline /proc/pid/cmdline

Table 5.5: Examples of per-process entries in the /proc file system. The Solaris psinfo pseudo file contains most of the information that is needed by the ps (list processes) command.

Besides processes, Linux and FreeBSD make a limited amount of kernel status information available under /proc, while Solaris uses additional pseudo devices such as /dev/ip for network status and /dev/kstat for kernel statistics. This apparent chaos of lookup mechanisms is not a problem as long as you can depend on tools such as ps, netstat or lsof whose output is relatively system independent. It complicates life, however, when you need to write tools that attempt to bypass potentially compromised utilities.

5.9 Limitations of process and system status tools

Unfortunately, the accuracy of information from system status tools is limited. Some limitations are unavoidable because the tools look at information that is changing, such as memory usage or open files. Other limitations are less obvious, often accidental, and can complicate the interpretation of results. To illustrate this we will show some limitations in the way process status tools report process command line information.

As a first example, process status tools may produce incomplete command-line information. Although UNIX systems have generous limits on the length of command lines, how much of this can be recovered depends on the UNIX system implementation and on the tool being used. Table 5.6 shows typical limits for Solaris, RedHat Linux and FreeBSD systems on i386 type hardware. The NCARGS constant in the file <sys/param.h> defines the combined upper bound on the command line and process environment.

System NCARGS ps command line length limit
FreeBSD 5.0 64 kbyte length truncated to 0 if > 10000 bytes
Redhat 8.0 128 kbyte length truncated to 0 if > 4 kbytes
Solaris 9 1 Mbyte /usr/ucb/ps: all information recoverable
/usr/bin/ps: length truncated to 80

Table 5.6: Command line limitations in typical process status tools.

Another oddity is that a process may modify its own command line as displayed by ps. This is possible on systems such as FreeBSD, Linux, and Solaris, but with Solaris the changes are visible only if you invoke the BSD style /usr/ucb/ps command with the w option; this is probably more an artefact of implementation than the result of a deliberate decision. As an example, the Sendmail mail transfer agent routinely changes the command line in order to display the process state. Here is an example for RedHat 8.0 Linux that displays all sendmail related processes:

redhat80% ps ax | egrep 'COMMAND|sendmail'
  604 ?    S     0:00 sendmail: accepting connections
  614 ?    S     0:00 sendmail: Queue runner@01:00:00 for /var/spool/client

Finally, the process name (the first component of the command line array) can be different from the executable file name. Some ps implementations will display the executable file name in parentheses when it differs from the process name. And even if the process name equals the file name, one UNIX file can have any number of names, as described in chapter 3. A fine example of this is (again) the Sendmail program. This command is installed such that different hard links or symbolic links to the same program file have a different names. Each name serves a different purpose, even though all names ultimately refer to the same executable file.

Having discussed how process and system status tools work, where they get their information, and the origins of some of their limitations, we now turn to popular methods to subvert the findings of these tools.

5.10 Subversion with rootkit software

As mentioned in the preceding sections, the results from process and system status tools are subject to subversion. In the sections that follow we discuss how systems are subverted, how subversion is hidden, and how subversion may be detected. We limit the discussion to the upper three layers of the system architecture: executable file, library and kernel. Detection of subversion at the hardware level is beyond the scope of this book.

In our discussion of software subversion we will look at a category of malware that is known by the name of "rootkit", and that became popular first by the middle 1990s. The name stands for a combination of malicious software (such as a network sniffer or an attack tool), back door software that gives access without having to break into the machine, together with a collection of modifications to system software that hide the rootkit and other traces of the intrusion.

What makes rootkits popular is that they automate the process of installation and hiding, making it quick and painless. Typically, a rootkit is installed after system security is breached with a procedure that is also highly automated. When a rootkit is found on a compromised machine then it is very well possible that the whole incident didn't involve human control at all. While the intruder was doing something else, the rootkit installed itself and announced the compromised machine as another victim machine via some IRC channel.

5.11 Command-level subversion

Command-level rootkits hide the presence of malware by making changes to system commands. This approach is based on a very simple principle: in order to suppress bad news, they silence the messenger of that news. Table 5.7 shows a list of typical command-level rootkit modifications. Depending on the specific type of malware involved, many rootkits make modifications in addition to those listed in the table.

Replaced commands Hidden information
du, find, ls Malware configuration files and network sniffer logs
pidof, ps, top Network sniffer and/or back door process
netstat Network ports associated with malware
ifconfig Network sniffing "enabled" status

Table 5.7: Typical system utilities that are replaced by command-level rootkits, and the information that the replacements attempt to hide.

As intrusion technology has evolved over time, so have rootkits. The first rootkits came with network sniffers to collect user names and passwords (such as the esniff program), while later versions came with remotely controlled agents for distributed denial of service attacks (such as the T0rn rootkit).

Typical back door software is in the form of a modified login program, or a non-standard or modified network server for the finger or ssh service, or an inetd server that listens on a secret network port. The back door is typically enabled by entering a specific password, or by connecting from a specific network source port or IP address. These are, however, not the only types of back door in existence as we will see later.

5.12 Command-level evasion and detection

In order to evade detection, early rootkits not only replaced system utility software but also erased records in system logfiles. Some rootkits even went to the effort of giving modified system utilities the same file time stamps and CRC (cyclic redundancy check) value as the original files. Later command-level rootkits don't bother and simply install modified programs that hide the presence of malware.

Regardless of these details, none of the changes compromise integrity of the kernel. Detection of command-level rootkit modifications is therefore relatively easy, as long as one uses a trusted copy of the system utilities. Here is a number of ways in which command-level rootkits can be discovered:

As an example of exposing a command-level root kit, we examine two utilities that are part of the T0rn rootkit for Linux, which was in widespread use in 2001. First we search the /bin/ls executable file with the strings and grep commands for strings that look like file names:
$ strings /bin/ls | grep /
. . .five more lines omitted. . .

The file name /usr/src/.puta/.1file looks very suspicious. If we try to list the /usr/src/.puta directory, the ls command hides the name as we would expect:

$ cd /usr/src
$ ls -a
.             ..            linux         linux-2.2.14  redhat

However, the directory name still shows up when we use the echo command, together with the .* wild-card expansion feature that is built into the command shell:

$ echo .* *
. .. .puta linux linux-2.2.14 redhat

In the .puta/.1file rootkit configuration file we find a lengthy list of file and directory names that must remain hidden, because these contain the malware program files, configuration files and data files:

$ cat .puta/.1file
. . .29 more lines omitted. . .

Just as we can detect modified file utilities by comparing their results against output from an unmodified tool, we can detect modified process status utilities by comparing their output against information from the /proc file system. Table 5.8 shows that the ps command is hiding a process with ID 153 (it also shows that /proc and ps disagree on whether "2" corresponds to a process, but that is a different issue).

Entries in /proc Output from "ps ax"
    1  ?  S    0:06 init [3] 

    3  ?  SW   0:00 (kupdate)
    4  ?  SW   0:00 (kpiod)
    5  ?  SW   0:00 (kswapd)
    6  ?  SW<  0:00 (mdrecoveryd)

  271  ?  S    0:00 /sbin/pump -i eth0 
  341  ?  S    0:00 portmap 
  356  ?  SW   0:00 (lockd)

Table 5.8: Comparison of process information from the /proc file system and from the ps command.

The system utilities that were replaced by the rootkit do a good job of hiding process 153. It is not only censored by process status tools such as ps, it also does not show up with network status tools such as netstat. However, for reasons that we may never know, this rootkit does not replace the lsof command and therefore it can help to reveal the purpose of process 153:

# lsof -p 153
nscd    153 root  cwd    DIR    3,5   4096     2 /
nscd    153 root  rtd    DIR    3,5   4096     2 /
nscd    153 root  txt    REG    3,5 201552 35646 /usr/sbin/nscd
. . .
nscd    153 root    7u  IPv4    177          TCP *:47017 (LISTEN)
. . .

The file name /usr/sbin/nscd suggests that this is a system program, but comparison with uncompromised systems shows that this program is present only in later Linux versions. Connecting with telnet to TCP port 47017 on the local machine confirms that we are looking at a back door process. In this case, we are welcomed by the opening banner of what appears to be an SSH server. SSH is popular with legitimate and illegitimate users because it encrypts and protects network, making it immune to inspection and manipulation.

# telnet localhost 47017
Connected to rh62.
Escape character is '^]'.

Each rootkit differs slightly in its approach to hide the presence of malware, and therefore each rootkit needs a slightly different approach for detection. An example of software that automates the search for known rootkits is the chkrootkit toolkit [Murilo, 2003]. It runs on a dozen different UNIX platforms, and at the time of writing recognizes more than 50 different rootkits. Chkrootkit looks for deleted login/logout records, signatures of replaced system utilities, rootkit configuration files and directories, missing processes, and for signs of kernel level subversion. But that will be the topic of a later section.

5.13 Library-level subversion

Instead of replacing system utilities, rootkits can hide their existence by making changes at the next level down in the system architecture, the system run-time library. A good example of this is redirecting the open() and stat() calls. The purpose of these modifications is to fool file integrity checking software that examines executable file contents and attributes. By redirecting the open() and stat() calls to the original file, it will appear as if the file is still intact, while the execve() call executes the subverted file. For example, listing 5.3 shows how one could redirect the open() call in a typical Linux run-time library:

#include <errno.h>
#include <syscall.h>
#include <real_syscall.h>

  * Define a real_open() function to invoke the SYS_open system call.
static  real_syscall3(int, open, const char *, path,
		              int, flags, int, mode)

  * Intercept the open() library call and redirect attempts to open
  * the file /bin/ls to the unmodified file /dev/.hide/ls.
int open(const char *path, int flags, int mode)
    if (strcmp(path, "/bin/ls") == 0)
        path = "/dev/.hide/ls";
    return (real_open(path, flags, mode));

Listing 5.3: Library-level backdoor to redirect specific open() system calls. The real_syscall3() macro, whose definition is too ugly to be shown here, is a slightly modified copy of the standard Linux _syscall3() macro. We use it to define our own real_open() function that invokes the SYS_open system call.

Would an MD5 hash reveal the library modification? Not necessarily. While the run-time linker uses the low-level open() system call when it accesses the modified library file, the md5sum command uses the fopen() library routine and can be still redirected to the unmodified library file.

To work around modifications at this level, rootkit detection tools need to carry their own trusted copy of the system library routines. However, such measures and countermeasures have become less relevant with the arrival of rootkits that make changes to the running kernel, and that are therefore much harder to circumvent. or detect.

5.14 Kernel-level subversion

As we have seen in the previous section, rootkit modifications to system utilities are easy to circumvent. As long as we have a copy of the unmodified utilities we can still find the malware files, processes, and network ports. In a similar manner, rootkit modifications to system library routines are easy to circumvent.

The game changes dramatically when modifications are moved from the process layer into the kernel layer. Compromised kernel code cannot be circumvented easily, because hardware memory protection prevents user processes from doing so. All accesses to kernel memory must be mediated by the kernel, whether it is compromised or not. And since the running kernel is the source of information for all file, process and network status tools, those tools may produce inaccurate results when the kernel is compromised. Despite all these handicaps, kernel-level modifications may still be detectable, as we will show at the end of the chapter.

5.15 Kernel rootkit installation

Just like command-level rootkits, kernel-level rootkits are installed after the security of a system is compromised. Over time, different methods have been developed to inject rootkit code into a kernel.

The exact details of these methods are highly system dependent. Even the methods that use officially documented interfaces are likely to break with different versions of the same operating system. We refer the interested reader to the references.

5.16 Kernel rootkit operation

The purpose of many kernel rootkits is to hide malware processes, files and network ports, and of course to hide itself. There are two sides to information hiding: the output side and the input side. On the output side, the kernel must censor the output from system calls that produce a list of processes, files, network ports, etc. On the input side, any attempt to manipulate a hidden process, file, network port, etc., must fail as if the object does not exist. In addition, rootkits may redirect system calls such as open() in order to subvert the operation of software that verifies the integrity of executable file content and attributes. Figure 5.4 shows the typical architecture of early kernel rootkit implementations.

Figure 5.4: Rootkits based on system call interposition.

Early kernel rootkits subvert system calls close to the process-kernel boundary. In order to prevent access to a hidden file, process, etc., they redirect specific system calls to wrapper code that inspects the parameters and that decides whether the system call is allowed to happen. For example, code to subvert the open() system call goes like this:

evil_open(pathname, flags, mode)
    if (some_magical test succeeds)
        call real_open(pathname, flags, mode)
        error: No such file or directory

To prevent rootkit disclosure, system calls that produce lists of files, processes, network ports or kernel modules are intercepted in order to suppress information that must remain hidden. For example, the code that subverts the getdents() system call (list directory entries) goes like this:

evil_getdents(handle, result)
    call real_getdents(handle, result)
    if (some_magical test fails)
        remove hidden objects from result

The advantage of system call interposition is that the code is relatively easy to understand: the change is made at a point that is close to the user of those system calls. One drawback of this approach is that many system calls need to be intercepted. For example, in order to hide the existence of a file one would have to intercept all system calls that have a file name argument: open(), chdir(), unlink()and many others. That alone is some 40 system calls on Linux, FreeBSD and Solaris.

This drawback is addressed by subverting UNIX kernels at a level that is closer to the information that is being hidden. In the next example we will show how this can be used to hide files. Figure 5.5 shows the typical architecture of such subversion.

UNIX systems support a variety of file system types. Besides file systems with a UNIX origin such as UFS, EXT2FS, and EXT3FS, many systems support non-UNIX file systems such as FAT16, FAT32, NTFS, and others. Typically, each file system implements a common virtual file system (VFS) interface with operations to look up, open, or close a file, to read directory entries, and a dozen or so other operations [Kleiman, 1986].

Figure 5.5: Rootkits based on interposition at object interfaces, showing subversion of multiple UFS file system methods.

Of all these virtual file system interface operations, the lookup operation is of particular interest. System calls that access an existing file or directory, etc., by name use the lookup operation to translate the pathname into the underlying file system object. By redirecting the lookup call at the virtual file system layer it is possible to hide a file from all system calls that access an existing file by name:

evil_lookup(parent_directory, pathname, ...)
    if (some_magical test succeeds)
        call real_lookup(parent_directory, pathname, ...)
        error: No such file or directory

This modification is sufficient to hide an existing file from system calls that attempt to access it, such as chdir() or open(). However, it does not hide the file's existence from system calls that create a new directory entry, such as link(), mkdir(), socket(), mkfifo() and other system calls. Unless the malware is prepared to redirect names of new files that collide with names of hidden files, the system calls will fail with a "File exists" error. A rootkit detector that knows specific rootkit file names can exploit this property.

The kernel changes that we discussed sofar are relatively easy to detect, because they change code addresses in kernel tables that normally never change. We can examine these tables from outside the kernel by accessing /dev/kmem, or from inside the kernel with a forensic kernel module. An example follows in the next section.

Detection methods that look for changes in kernel tables can be defeated by leaving the tables alone, and by patching an existing kernel function so that it calls the malware. For an example of patching live kernel code, see [Cesare, 1999]. Such modifications can be found by inspecting all the running kernel code and by verifying all instructions against all kernel and kernel module files. This a non-trivial task, as the contents of modules change while they are linked into the kernel, in order to update references to external functions and data.

5.17 Kernel rootkit detection and evasion

Kernel rootkits, like their non-kernel predecessors, may be exposed because they introduce little inconsistencies into a system. Some inconsistencies may show up externally, in the results from system calls that manipulate process, files, kernel modules, and other objects. Other inconsistencies show up only internally, in the contents of kernel data structures. Internal inconsistency is unavoidable, because every hidden process, file, or kernel module occupies some storage. That storage has to be claimed as "in use", and has to be referenced by something in the kernel's path of execution, even though the storage does not appear in kernel symbol tables. We will present examples of both types of inconsistency below.

As an example of externally visible inconsistency we present a simple hidden process detection technique. The idea is to create a long sequence of processes. On UNIX systems that allocate PID (process ID) values sequentially, the sequence will show a hole where a PID is in use, or where a PID falls within a reserved range. As examples of the former, UNIX systems consider a PID as "in use" when it belongs to a running process, or when it belongs to a group of processes (either as a process group ID, or as its politically correct version, a POSIX session ID). As examples of reserved ranges, FreeBSD and Linux skip over the first 100 and 300 PID values, respectively, when the PID counter wraps around at its maximal value.

The hidden process search can be implemented with the somewhat slow Perl program shown in listing 5.4 that creates a sequence of child processes with the fork() system call. Whenever a hole appears in the sequence of child PID numbers, the program attempts to fill the hole with information about existing processes in /proc. In order to avoid false alarms due to short-lived processes that terminate before /proc can be searched, the program cycles twice through the entire PID range, which makes the program even slower.


# checkpid - find hidden processes 

$last_pid = $PROCESS_ID;
$first_pid = $last_pid + 1;

for (;;) {

    # Fork a child processes and wait until the child terminates.
    if (($pid = fork()) < 0) { die "cannot fork: $!\n"; }
    if ($pid == 0) { exit; }

    # Update the visible process list. Fill gaps in the PID sequence
    # with information from the /proc file system.
    $proc_seen[$pid] = 1;
    if ($last_pid + 1 < $pid - 1) { check_existing_processes(); }

    # Update the maximal process ID, and how many times we reached it.
    if ($pid > $max_pid) { $max_pid = $pid; } 
    if ($last_pid > $pid) { $wrapped_around++; }
    $last_pid = $pid;

    # Report findings when we complete two passes over the PID range.
    if ($wrapped_around > 1 && $pid >= $first_pid) {
	print "Maximal PID: $max_pid\n";
	for $pid (0..$max_pid) { 
	    report_missing($pid) if !$proc_seen[$pid]; }
	exit 0;

Listing 5.4: A program that cycles through the process ID space and that recognizes hidden processes by the holes they leave behind.

The implementation of the check_existing_processes() function is system dependent and is omitted to save space; the complete script will be made available at the book website. When this program is run on a rootkit-infested Solaris system, it produces as output:

Maximal PID: 29999
Missing PID: 10257

The first line reports the highest process ID observed, and the second line reports a PID that was never observed as a process ID or as the ID of a process group or of a POSIX session. Indeed, ID 10257 corresponds to a hidden process that was set up in a lab environment in order to test this tool.

Due to the way the hidden process finding tool works, a positive signal should definitely be taken seriously. A negative result, on the other hand, would not mean that a system is clean. The rootkit might be too sophisticated for detection in this manner. Another limitation of the tool is that it cannot explore the reserved PID range (0-99 on BSD, 0-299 on Linux) and consequently will report those ranges as possibly hidden processes.

The findrootkit tool works in a very different manner. It examines kernel memory and can therefore give specific answers [Dik, 2004]. findrootkit is written in Perl, and uses the mdb low-level debugger to examine the running Solaris kernel via the /dev/kmem interface. The tool checks the consistency of information from multiple sources within and outside the kernel:

These consistency checks can reveal the presence of hidden kernel modules and other code that hides in the kernel. Additionally, the tool knows that specific kernel modules implement specific functions. For example, findrootkit knows the kernel modules that implement specific file systems or specific system calls. Tables 5.9a and 5.9b show an example of a kernel modification report.

Interposed vnode

specfs:ioctl 0xfe9f23c8
procfs:lookup 0xfe9f2080
procfs:readdir 0xfe9f22fc
ufs:setattr 0xfe9f1420
ufs:getattr 0xfe9f174c
ufs:lookup 0xfe9f1a08
ufs:readdir 0xfe9f1d50
ufs:remove 0xfe9f1e30
ufs:rename 0xfe9f1eec

system call

fork 0xfe9f2fb4
fork1 0xfe9f3058
kill 0xfe9f30fc
sigqueue 0xfe9f31a4
exec 0xfe9f324c
exece 0xfe9f3264

Tables 5.9a, 5.9b: Solaris rootkit kernel modification report, showing changes to the file system operations table, and to the system call jump table.

The report in tables 9a and 9b shows the findrootkit results for a Solaris kernel with a hidden kernel module. A number of file system operations and system calls is interposed, and is shown with the hexadecimal address of each interposing function. The kernel module name and interposing function names are unavailable, because those names were removed by the kernel module. The replacements in the proc file system are what one would expect for process hiding: the procfs:lookup() operation reporting that a hidden process does not exist, and the procfs:readdir() operation removing hidden processes from process listings. Numerous UFS (the default Solaris file system) operations have been modified for presumably nefarious purposes, as well as the specfs:ioctl() operation for the SPECFS file system, which provides access to device special files, to network sockets, and to other objects that exist outside the Solaris file system.

As with the previous tool, a sufficiently sophisticated rootkit can evade detection. In particular, tools like the last one that examine a kernel from outside can be fooled by subverting the /dev/ksyms and/or /dev/kmem drivers so that they lie about the contents of kernel memory. Even running the consistency checker inside the kernel would not make it immune to such tampering.

5.18 Conclusion

Writing this chapter has produced at least one good result: it has convinced its author to raise the BSD security level on critical machines. Although such protection can be subverted, it buys additional time, and forces an intruder to raise alarms.

The rootkits discussed in this chapter go through a lot of effort to hide processes or files, but there is no good reason why a rootkit should need to use processes or files in the first place. With some loss of convenience, backdoor software can be running entirely within the kernel, or at least the part of the backdoor that is memory resident. If storage space is needed there is plenty available in the mostly unused swap space, and the backdoor can be controlled via any number of local or networked covert channels. A rootkit that makes no persistent changes to the machine can be practically undetectable by software that runs within or above a compromised kernel. Finding it requires direct hardware access, or software that runs between the kernel and the hardware, such as a virtual machine monitor or hypervisor. At the time of writing, monitors and hypervisors are rarely used. And they are not the ultimate solution, either. They will have bugs, and therefore will be prone to subversion.

Although this chapter did not cover the possibilities for subversion at the hardware level, that does not mean that the authors are ignorant of its potential. Any writable storage presents an opportunity for subversion, especially when that storage is associated with, or is even part of, a processor of some kind. Hoglund and McGraw discuss this topic in a PC hardware specific context [Hoglund, 2004].


[Abell, 2004] The lsof (list open files) tool by Victor A. Abell, 2004.

[Arbaugh, 1997] W. A. Arbaugh, D. J. Farber, J. M. Smith, "A Secure and Reliable Bootstrap Architecture", Proceedings of the 1997 IEEE Symposium on Security and Privacy, pp. 6571, May 1997.

[ASR, 1996] Avalon Security Research, amodload kernel loader for SunOS 4, 1996.

[Cesare, 1999] Silvio Cesare, "Runtime kernel kmem patching", 1999.

[Dik, 2004] Casper H.S. Dik, private communication.

[van Doorn, 2001] L. van Doorn, G. Ballintijn, and W. A. Arbaugh, "Design and implementation of signed executables for linux," University of Maryland, Tech. Rep. CS-TR-4259, June 2001.

[Hoglund, 2004] Greg Hoglund, Gary McGraw, chapter 8 in "Exploiting Software". Addison Wesley, 2004.

[Jbtzhm, 2002] Jbtzhm, "Static Kernel Patching", Phrack 60, 2002.

[Kleiman, 1986] S.R. Kleiman, Vnodes: An Architecture for Multiple File System Types in Sun UNIX", Summer 1986 Usenix Conference Proceedings.

[Mandia, 2001] The Carbonite forensic software by Kevin Mandia and Keith J. Jones, 2001.

[Murilo, 2003], The chkrootkit rootkit detection tool by Nelson Murilo and Klaus Steding-Jessen, 2003.

[Nemeth, 2001] Evi Nemeth, Garth Snyder, Scott Seebass, Trent R. Hein, "UNIX System Adminstration Handbook", third edition. Prentice Hall PTR, 2001.

[OpenSSH, 2004] The OpenSSH remote connectivity software.

[Plaguez, 1998] Plaguez, "Weakening the Linux Kernel", Phrack 52, 1998.

[Plasmoid, 1999] Plasmoid, "Solaris Loadable Kernel Modules", 1999.

[Pragmatic, 1999] Pragmatic, "Attacking FreeBSD with Kernel Modules", 1999.

[Sd, 2001] Sd, Dev, "Linux on-the-fly kernel patching without LKM", Phrack 57, 2001.

[Simpson, 2001] The checkps rootkit detector by Duncan Simpson.

[Stevens, 1997] W. Richard Stevens, as cited in the Raw IP Networking FAQ.

[StJude, 2002] The Saint Jude Kernel-Level IDS project.

[Truff, 2003] Truff, "Infecting loadable kernel modules", Phrack 61, 2003.

[Williams, 2002] Michael A. Williams, "Anti-Trojan and Trojan Detection with In-Kernel Digital Signature testing of Executables".