incus: incus-agent static builds with interpreter unusable on non-nix Linux VMs

[psychomario]

Nixpkgs version

• Unstable (26.05)

Describe the bug

incus-agent.linux.* as built from the script in #446389 causes the static incus-agent binaries to be built with an interpreter (ld-linux-x86-64.so.2) which is in the nix store, and is therefore causes the agent to not run on non-Nix Linux VMs (and assumes an identical host/VM store for Nix linux VMs).

This breaks tooling which requires the agent, such as incus shell

Steps to reproduce

nix shell nixpkgs#patchelf nixpkgs#incus -c /bin/sh -c 'patchelf --print-interpreter $(dirname $(dirname $(which incus)))/share/agent/incus-agent.linux.x86_64'

outputs something like

/nix/store/rcp9sdrrq8sfxkm5zdykglx7hd2gzbfy-glibc-2.40-66/lib/ld-linux-x86-64.so.2

Launching a VM, and using incus console [vm] --show-log will show incus-agent.service fails to start. Mounting the root filesystem to set a password, logging in with incus console, and trying to manually run the /run/incus_agent/incus-agent binary gives an error due to the missing interpreter.

Expected behaviour

The interpreter in the built ELFs should probably not be a Nix store path

Screenshots

No response

Relevant log output

Additional context

The following override can be used to hardcode a different interpreter which allows the incus-agent to work on e.g Ubuntu VMs.

  incuspatched = pkgs.incus.overrideAttrs (
    final: prev: {
      postBuild =
        builtins.replaceStrings [ ''-ldflags="-s"'' ] [ ''-ldflags="-s -I /lib64/ld-linux-x86-64.so.2"'' ] prev.postBuild;
    }
  );

This is probably not a proper fix (it won't then work on Nix VMs) but it works.

System metadata

• system: "x86_64-linux"
• host os: Linux 6.12.58, NixOS, 25.11 (Xantusia), 25.11.20251117.89c2b23
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.31.2
• channels(root): "nixos"
• nixpkgs: /nix/store/f640ps0hcp7w5jzg18djf8gdhl6r2rnl-source

Notify maintainers

@aanderse @adamcstephens @megheaiulian @mkg20001

---

Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)

I assert that this issue is relevant for Nixpkgs

• I assert that this is a bug and not a support request.
• I assert that this is not a duplicate of an existing issue.
• I assert that I have read the NixOS Code of Conduct and agree to abide by it.

Is this issue important to you?

Add a 👍 reaction to issues you find important.

[psychomario]

It seems the static builds are not static enough, as the binaries are still ET_DYN. The binaries in IncusOS are ET_EXEC and therefore don't have an .interp section at all.

I've been able to (manually) compile the agent to be ET_EXEC with the following, although i've not been able to hotpatch it into the derivation yet (seems like some glibc overlap causing undefined references)

CGO_LDFLAGS="-L$(nix eval --raw nixpkgs#musl.outPath)/lib" GOARCH=amd64 go build -ldflags '-linkmode external -extldflags "-static"' -o incus-agent.linux.x86_64 -tags=agent,netgo github.com/lxc/incus/v6/cmd/incus-agent

This brings in a musl build-time dependency.

[psychomario]

Upon further investigation I've got to the following, which uses musl-gcc for native Linux targets, while leaving Windows targets unchanged. I've confirmed this outputs ET_EXEC binaries which work fine in a variety of Incus VM types (success on Alpine, Ubuntu & NixOS).

However, this method does not work for compiling i686 Linux targets, I think this would need proper cross compiling.

  incuspatched = pkgs.incus.overrideAttrs (
    final: prev: {
      nativeBuildInputs = prev.nativeBuildInputs ++ [ pkgs.musl.dev ];
      postBuild =
        builtins.replaceStrings
          [
            ''
              build_incus_agent() {
                GOOS="$1" GOARCH="$2" CGO_ENABLED=0 \
                go build -ldflags="-s" -tags=agent,netgo \
                  -o "$out/share/agent/incus-agent.$1.$3" ./cmd/incus-agent
              }
            ''
            ''build_incus_agent linux   386    i686''
          ]
          [
            ''
              build_incus_agent_windows() {
                GOOS="$1" GOARCH="$2" CGO_ENABLED=0 \
                go build -ldflags="-s" -tags=agent,netgo \
                  -o "$out/share/agent/incus-agent.$1.$3" ./cmd/incus-agent
              }
              build_incus_agent_linux() {
                CC=musl-gcc \
                GOOS="$1" GOARCH="$2" \
                go build -tags=agent,netgo \
                  -ldflags '-s -linkmode external -extldflags "-static"' \
                  -o "$out/share/agent/incus-agent.$1.$3" ./cmd/incus-agent
              }
              build_incus_agent() {
                build_incus_agent_$1 $@
              }
            ''
            ''''
          ]
          prev.postBuild;
    }
  );

[adamcstephens]

Why are you running patchelf on the binary? These are not dynamically linked binaries and thus do not need patchelf run against them.

𑁱 ldd /nix/store/4cxqnq9ayidrdlcrgmbbvjzxqpg13y8i-incus-6.19.1/share/agent/*
/nix/store/4cxqnq9ayidrdlcrgmbbvjzxqpg13y8i-incus-6.19.1/share/agent/incus-agent.linux.i686:
        not a dynamic executable
/nix/store/4cxqnq9ayidrdlcrgmbbvjzxqpg13y8i-incus-6.19.1/share/agent/incus-agent.linux.x86_64:
        not a dynamic executable
/nix/store/4cxqnq9ayidrdlcrgmbbvjzxqpg13y8i-incus-6.19.1/share/agent/incus-agent.windows.i686:
        not a dynamic executable
/nix/store/4cxqnq9ayidrdlcrgmbbvjzxqpg13y8i-incus-6.19.1/share/agent/incus-agent.windows.x86_64:
        not a dynamic executable

Launching another distro, e.g. ubuntu, successfully connects to the agent.

𑁱 incus launch images:ubuntu/24.04 --ephemeral --vm -c limits.cpu=8 -c limits.memory=8GB
Launching the instance
Instance name is: wired-impala

𑁱 incus list wired-impala
+--------------+---------+------------------------+-------------------------------------------------+-----------------------------+-----------+
|     NAME     |  STATE  |          IPV4          |                      IPV6                       |            TYPE             | SNAPSHOTS |
+--------------+---------+------------------------+-------------------------------------------------+-----------------------------+-----------+
| wired-impala | RUNNING | 10.143.96.143 (enp5s0) | fd42:31fa:a167:90b:1266:6aff:fe50:67a5 (enp5s0) | VIRTUAL-MACHINE (EPHEMERAL) | 0         |
+--------------+---------+------------------------+-------------------------------------------------+-----------------------------+-----------+

𑁱 incus shell wired-impala
root@wired-impala:~# ps -ef | grep incus
root         430       1  0 13:56 ?        00:00:00 /run/incus_agent/incus-agent
root         585     575  0 13:57 pts/0    00:00:00 grep --color=auto incus
root@wired-impala:~# ldd /run/incus_agent/incus-agent
        not a dynamic executable
root@wired-impala:~# logout

[psychomario]

I'm just using patchelf as a tool to print out the interpreter, it's not doing any patching. You can see the same with readelf:

❯ readelf -l /nix/store/hhj75am4p5pwk2iw35k84alhmjkfpkir-incus-6.18.0/share/agent/incus-agent.linux.x86_64

Elf file type is DYN (Position-Independent Executable file)                                                                                                                               Entry point 0x485300                                                                                                                                                                      There are 11 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x0000000000000268 0x0000000000000268  R      0x1000
  INTERP         0x0000000000000fad 0x0000000000400fad 0x0000000000400fad
                 0x0000000000000053 0x0000000000000053  R      0x1
      [Requesting program interpreter: /nix/store/rcp9sdrrq8sfxkm5zdykglx7hd2gzbfy-glibc-2.40-66/lib/ld-linux-x86-64.so.2]
[snipped for brevity]
 Section to Segment mapping:
  Segment Sections...
   00
   01     .interp
[snipped for brevity]

❯ ldd /nix/store/hhj75am4p5pwk2iw35k84alhmjkfpkir-incus-6.18.0/share/agent/incus-agent.linux.x86_64
        statically linked

[adamcstephens]

What is your problem though? Incus wouldn't be able to use incus shell or display the IP information if the agent wasn't working inside the VM.

[adamcstephens]

Here's readelf from inside the Ubuntu VM.

root@wired-impala:~# readelf -l /run/incus_agent/incus-agent

Elf file type is EXEC (Executable file)
Entry point 0x484f60
There are 6 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x0000000000000150 0x0000000000000150  R      0x1000
  NOTE           0x0000000000000f78 0x0000000000400f78 0x0000000000400f78
                 0x0000000000000064 0x0000000000000064  R      0x4
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x00000000007ff211 0x00000000007ff211  R E    0x1000
  LOAD           0x0000000000800000 0x0000000000c00000 0x0000000000c00000
                 0x00000000008f06a8 0x00000000008f06a8  R      0x1000
  LOAD           0x00000000010f1000 0x00000000014f1000 0x00000000014f1000
                 0x0000000000086460 0x00000000000d0a60  RW     0x1000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x8

 Section to Segment mapping:
  Segment Sections...
   00
   01     .note.go.buildid
   02     .text .note.gnu.build-id .note.go.buildid
   03     .rodata .typelink .itablink .gosymtab .gopclntab
   04     .go.buildinfo .go.fipsinfo .noptrdata .data .bss .noptrbss
   05

[adamcstephens]

And proof that the binary is running:

root@wired-impala:~# systemctl status incus-agent
WARNING: terminal is not fully functional
Press RETURN to continue
● incus-agent.service - Incus - agent
     Loaded: loaded (/usr/lib/systemd/system/incus-agent.service; static)
     Active: active (running) since Sun 2025-12-07 13:56:57 UTC; 8min ago
       Docs: https://linuxcontainers.org/incus/docs/main/
    Process: 376 ExecStartPre=/lib/systemd/incus-agent-setup (code=exited, status=0/SUCCESS)
   Main PID: 430 (incus-agent)
      Tasks: 13 (limit: 8797)
     Memory: 28.4M (peak: 29.7M)
        CPU: 479ms
     CGroup: /system.slice/incus-agent.service
             └─430 /run/incus_agent/incus-agent

Dec 07 13:56:56 distrobuilder-345b76d9-29f4-4ecd-aada-0c89df9f453b systemd[1]: Starting incus-agent.service - Incus - agent...
Dec 07 13:56:57 wired-impala systemd[1]: Started incus-agent.service - Incus - agent.
Dec 07 13:57:06 wired-impala su[562]: (to root) root on pts/0
Dec 07 13:57:06 wired-impala su[562]: pam_unix(su-l:session): session opened for user root(uid=0) by (uid=0)

This seems to be working correctly to me. Please provide a reproducer of what's broken, not the suspected cause.

[psychomario]

Seems this was fixed between 6.18.0 and 6.19.1 and I wasn't on the version I thought I was.

My apologies.

[adamcstephens]

Glad you fixed it. :)