unfinished.bike

This is a small technical article to save other adventurous people some pain.

The Elastic Agent installer fails to install on immutable Linux distributions such as Fedora SilverBlue or Ubuntu Core, because /usr is read-only. These instructions modify the elastic-agent 8.13.4 installer and the SELinux configuration to allow daemons to start from /opt/bin:

sed -i s/"\/usr\/bin\/elastic-agent"/"\/opt\/bin\/elastic-agent"/ elastic-agent
sudo semanage fcontext -a -t bin_t "/var/opt/bin(/.*)?"                                                            sudo restorecon -R -v /var/opt/bin/

With those modifications, the Elastic 8.13.4 installer works properly, but let's dig into how I got there ...

The Original Error

Here's what an install session normally looks like on Fedora SilverBlue (BlueFin):

sudo ./elastic-agent install --force --url=https://endpoint

[    ] Successfully copied files  [1s] Error uninstalling.  Printing logs
Error: error installing package: failed to write shell wrapper (/usr/bin/elastic-agent): open /usr/bin/elastic-agent: read-only file system
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.13/fleet-troubleshooting.html

You can workaround this first hurdle by modifying the agent to use /usr/bin to /opt/bin:

sed -i s/"\/usr\/bin\/elastic-agent"/"\/opt\/bin\/elastic-agent"/ elastic-agent

The reason sed works well to modify a binary is that “/usr/bin/elastic-agent” and “/opt/bin/elastic-agent” are the same-sized string.

SELinux

If you re-run the script now, it will get a little further, but hang here:

[=   ] Service Started  [2s] Elastic Agent successfully installed, starting enrollment.
[=   ] Waiting For Enroll...  [2s] Error initializing version information: reading package version from file "/var/opt/Elastic/Agent/package.version": open /var/opt/Elastic/Agent/package.version: no such file or directory

If so, hit Ctrl-Z to explore, as the installer will timeout. Let's see what's going on with the Elastic agent:

sudo systemctl status --full --lines=100 elastic-agent                                       

The problem is that ExecStart is failing with 203/EXEC.

● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
     Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2024-05-15 10:45:54 EDT; 1min 30s ago
    Process: 15454 ExecStart=/opt/bin/elastic-agent (code=exited, status=203/EXEC)
   Main PID: 15454 (code=exited, status=203/EXEC)
        CPU: 713us

203/EXEC is usually invoked when the target of ExecStart does not exist. You'll find that it does exist and even executes properly. It turns out that any error in execution, including SELinux, will also result in the same error code. If you run journalctl -f, you may see this innocuous-looking message:

May 15 10:54:45 fedora audit[16492]: AVC avc:  denied  { execute } for  pid=16492 comm="(ic-agent)" name="elastic-agent" dev="dm-0" ino=505894 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0

Apparently, SELinux does not allow systemd to start programs from /opt. The AVC here stands for “Access Vector Cache”. The first workaround I arrived at was modifying ExecStart to /bin/sh /opt/bin/elastic-agent, but the real fix is changing our SELinux configuration to allow binaries to start from this directory:

sudo semanage fcontext -a -t bin_t "/var/opt/bin(/.*)?"
sudo restorecon -R -v /var/opt/bin/

Now that you have SELinux configured properly, the agent should start up:

sudo systemctl restart elastic-agent

If you suspended the original agent installer as I did, you can resume it with fg and see that it's properly enrolled:

Send job 1 (sudo ./elastic-agent install --force --url=https://wherever.fleet.us-central1.gcp.cloud.es.io:443 --enrollment-token=xyz== --tag=t) to foreground
[=== ] Waiting For Enroll...  [21m53s] {"log.level":"info","@timestamp":"2024-05-15T11:07:45.132-0400","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":482},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-05-15T11:07:45.132-0400","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":300},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=== ] Done  [21m53s]                           
Elastic Agent has been successfully installed.

Success!