DIY Linux Kernel Rootkit Detection
Today I will teach you how to roll your own detection of a classic Linux kernel rootkit using shell scripting (for explanation) and osquery (for production).
Your chance of encountering a Linux kernel rootkit in the wild is slim due to compatibility and distribution challenges, so this explanation is geared toward the most paranoid folks in the audience. For this demonstration, we'll use reveng_rtkit – one of the more modern examples of a Linux rootkit.
As of February 2023, this reveng_rtkit runs well on Debian, so I used lima with limactl start template://debian to create my test environment.
How does one detect a rootkit anyways?
The trick to detecting a rootkit is to find cracks in the illusion they present. Most rootkits only hide their presence when probed with a specific syscall, but Linux has hundreds of syscalls to choose from. Most commonly, Linux rootkits will hide processes or directory entries by overriding getdents(2), but neglect stat(2). Some rootkits hide file contents by overriding read(2), but forget about mmap(2).
Collecting Evidence
Collecting the state of a system before and after a malware installation is an easy way to determine what attributes to alert on.
If you are only concerned about filesystem changes, many methods are available, such as Sleuthkit or hashdeep. Fewer choices exist if you are concerned about the larger overall system state.
Since I work on the [osquery-detection-kit,]((https://github.com/chainguard-dev/osquery-defense-kit project) I’ll use it to collect a subset of system information to a file. osquery-detection-kit does require extremely recent versions of Go and osquery to be installed, so here's how I installed the dependencies:
arch=$(uname -m | sed -e s/x86_64/amd64/g -e s/aarch64/arm64/g)
curl -L https://go.dev/dl/go1.20.1.linux-${arch}.tar.gz | sudo tar -C /usr/local -zxvf -
curl -LO https://pkg.osquery.io/deb/osquery_5.7.0-1.linux_${arch}.deb
sudo dpkg -i osquery_5.7.0-1.linux_${arch}.deb
export PATH=/usr/local/go/bin:$PATH
Here's how I collected the data:
git clone https://github.com/chainguard-dev/osquery-defense-kit
cd osquery-defense-kit
git pull
make collection
This will populate a subfolder named out/ with evidence, so that we may compare against it later.
Installing The Rootkit
I followed the instructions on https://github.com/reveng007/reveng_rtkit to install the rootkit. Here's how I loaded it:
sudo insmod reveng_rtkit.ko
Afterward, sudo dmesg shows:
[ 382.584069] [+] reveng_rtkit: Created by @reveng007(Soumyanil)
[ 382.584070] [+] reveng_rtkit: Loaded
[ 382.584071] [*] reveng_rtkit: Hiding our rootkit LKM from `lsmod` cmd, `/proc/modules` file path and `/proc/kallsyms` file path
Finding System State Differences
Now I'm able to collect system state once again and diff the two:
cd $HOME/osquery-defense-kit
make collect
diff -ubR ./out/<old> ./out/<new>
The first thing that stood out was that one of the sysctl's changed:
-config_value: current_value:0 name:kernel.tainted oid: subsystem:kernel type:string
+config_value: current_value:12288 name:kernel.tainted oid: subsystem:kernel type:string
The rootkit uses kill -31 to hide processes, so I used kill -31 $$ to hide my shell process, ran the collection, and ... found no further differences. This means I'm going to have to improve our detection scripts.
For testing, I also started a sleep 7200 & process in the background from the hidden shell. This rootkit also hid this subprocess – not all are so consistent.
Detecting hidden pids
Kernel rootkits tend to do two things with varying levels of success: hide their kernel module, and hide a process ID. revenge_rtkit does this by hiding getdents() calls to /proc, but if you know the hidden process ID, you can stat it directly. Ironically, this is also how Linux hides lightweight threads from users.
For most of Linux's life, the maximum number of pids on a system was 32768, a relatively small area to stat. Most modern Linux distros have bumped this number up, so this process is going to be slow. To check your system's maximum pid number, run cat /proc/sys/kernel/pid_max.
Here's a shell script that iterates through all possible pid numbers, revealing any that were not found when listing /proc: [full source]
[[ $EUID != 0 ]] && echo "* WARNING: For accurate output, run $0 as uid 0"
declare -A visible
cd /proc || exit
start=$(date +%s)
for pid in *; do
visible[$pid]=1
done
for i in $(seq 2 "$(cat /proc/sys/kernel/pid_max)"); do
[[ ${visible[$i]} = 1 ]] && continue
[[ ! -e /proc/$i/status ]] && continue
[[ $(stat -c %Z /proc/$i) -ge $start ]] && continue
# pid is a kernel thread
[[ $(awk '/Tgid/{ print $2 }' "/proc/${i}/status") != "${i}" ]] && continue
exe=$(readlink "/proc/$i/exe")
cmdline=$(tr '\000' ' ' <"/proc/$i/cmdline")
echo "- hidden $(cat /proc/$i/comm)[${i}] is running ${exe}: ${cmdline}"
done
Here's the output of this script:
- hidden bash[1924] is running /usr/bin/bash: /bin/bash --login
- hidden sleep[18518] is running /usr/bin/sleep: sleep 7200
To use roughly the same logic with osquery, use the following within sudo osqueryi [full source]:
WITH RECURSIVE cnt(x) AS (
SELECT 1
UNION ALL
SELECT x + 1
FROM cnt
LIMIT 4194304
)
SELECT p.*
FROM cnt
JOIN processes p ON x = p.pid
WHERE x NOT IN (
SELECT pid
FROM processes
)
AND p.start_time < (strftime('%s', 'now') - 1)
AND (
p.pgroup = p.pid
OR (
p.pid = p.parent
AND p.threads = 1
)
)
Detecting unusual kernel taints
Earlier we talked about the value of sysctl kernel.tainted changing from 0 to 12288. Let's use the following script to diagnose it [full source]:
declare -A table=(
[0]="proprietary module was loaded"
[1]="module was force loaded"
[2]="kernel running on an out of specification system"
[3]="module was force unloaded"
[4]="processor reported a Machine Check Exception (MCE)"
[5]="bad page referenced or some unexpected page flags"
[6]="taint requested by userspace application"
[7]="kernel died recently, i.e. there was an OOPS or BUG"
[8]="ACPI table overridden by user"
[9]="kernel issued warning"
[10]="staging driver was loaded"
[11]="workaround for bug in platform firmware applied"
[12]="externally-built (out-of-tree) module was loaded"
[13]="unsigned module was loaded"
[14]="soft lockup occurred"
[15]="kernel has been live patched"
[16]="auxiliary taint, defined for and used by distros"
[17]="kernel was built with the struct randomization plugin"
[18]="an in-kernel test has been run"
)
taint=$(cat /proc/sys/kernel/tainted)
[[ $taint == 0 ]] && exit
echo "kernel taint value: ${taint}"
for i in $(seq 18); do
bit=$(($i-1))
match=$(($taint >> $bit &1))
[[ $match == 0 ]] && continue
echo "* matches bit $bit: ${table[$bit]}"
done
echo ""
echo "dmesg:"
dmesg | grep taint
Here's the output of that script when this rootkit is loaded:
kernel taint value: 12288
* matches bit 12: externally-built (out-of-tree) module was loaded
* matches bit 13: unsigned module was loaded
dmesg:
[ 368.765518] reveng_rtkit: loading out-of-tree module taints kernel.
[ 368.777600] reveng_rtkit: module verification failed: signature and/or required key missing - tainting kernel
Here's the osquery query I generated for alerting on this kind of taint [full source]:
SELECT current_value AS value,
current_value & 65536 AS is_aux,
current_value & 8192 is_unsigned,
current_value & 4096 AS out_of_tree,
current_value & 512 AS kernel_warning,
current_value & 614 AS requested_by_userspace,
current_value & 8 AS force_unloaded,
current_value & 4 AS out_of_spec,
current_value & 2 AS force_loaded,
current_value & 1 AS proprietary
FROM system_controls
WHERE name = "kernel.tainted"
AND current_value NOT IN (0, 512, 12289, 4097)
Detecting unusual /dev entries
We noted an unusual device earlier, which is used to communicate to the rootkit:
crw------- 1 root root 247, 0 Feb 23 15:53 etx_device
The “247” in the output there is the major device number, which generally maps to a kernel module. To find unexpected devices like this, you can either whitelist expected device names, or expected major numbers. We'll do it both ways here.
To use major number logic, you'll want to refer to https://www.kernel.org/doc/Documentation/admin-guide/devices.txt. Armed with this information, there are two data sources to check, /proc/devices, and /dev. The first one is interesting, as it's a map of major numbers to drivers:
Character devices:
1 mem
4 /dev/vc/0
4 tty
…
189 usb_device
226 drm
247 etx_Dev
248 aux
One major flaw in my plan is that there are sections of dynamically generated major numbers, for instance, our suspicious 247 major device lands squarely in this section:
240-254 block LOCAL/EXPERIMENTAL USE
So, I went with a hybrid approach to discover devices that are commonly found on UNIX systems, by major number when possible, and the device name when it's dynamic [full source]:
declare -A expected_major=(
[1]="memory"
[2]="pty master"
[3]="pty slave"
[4]="tty"
[5]="alt tty"
[6]="parallel"
[7]="vcs"
# [8]="scsi tape"
[9]="md"
[10]="misc"
[13]="input"
[21]="scsi"
[29]="fb"
...
)
declare -A expected_low=(
["bsg/"]=1
["dma_heap/system"]=1
...
)
declare -A expected_high=(
["drm_dp_aux"]=1
["iiodevice"]=1
)
for path in $(find /dev -type c); do
hex=$(stat -c '%t' $path)
major=$(( 16#${hex} ))
pattern=$(echo $path | cut -d/ -f3- | tr -d '[:0-9]')
# Unix98 PTY Slaves
(( major >= 136 && major <= 143 )) && continue
[[ ${expected_major[$major]} != "" ]] && continue
class="UNKNOWN"
(( major >= 60 && major <= 63 )) && class="LOCAL/EXPERIMENTAL"
(( major >= 120 && major <= 127 )) && class="LOCAL/EXPERIMENTAL"
if (( major >= 234 && major <= 254 )); then
class="low dynamic"
[[ ${expected_low[$pattern]} == 1 ]] && continue
fi
if (( major >= 384 && major <= 511 )); then
class="high dynamic"
[[ ${expected_high[$pattern]} == 1 ]] && continue
fi
echo "${class} major device ${pattern}[${major}]"
echo "* $(ls -lad $path)"
echo "* /proc/devices: $(sed -n '/Block devices:/q;p' /proc/devices | grep -e "^ *${major}")"
echo ""
done
Here is the output of this script on a system with reveng_rtkit installed:
low dynamic major device etx_device[247]
* crw------- 1 root root 247, 0 Mar 3 19:48 /dev/etx_device
* /proc/devices: 247 etx_Dev
In osquery, there is no reliable way to determine a major number, as it depends on a local magic database. There we'll rely on a simple whitelist of device names [full source]:
SELECT *
FROM
file
WHERE
(
path LIKE '/dev/%'
OR directory LIKE '/dev/%'
)
AND path_expr NOT IN (
'/dev/acpi_thermal_rel',
'/dev/autofs',
'/dev/block/',
'/dev/block/:',
'/dev/bsg/',
'/dev/bsg/:::',
…
)
In Closing
In a future episode, we'll explore what it takes to detect an eBPF rootkit. The general philosophy is the same: know what to expect from your system and detect half-hearted illusions. I hope you enjoyed this article!
If you have questions, find me at @tstromberg.