Compile static gdbserver for MIPS with Nix

I’ve tested the following instructions using GDB 15.2 and NixOS 24.11.

About gdbserver

When you run GDB to debug a process, two things happen:

1. GDB starts or attaches to a process that you want to debug.
2. GDB provides you a user interface so that you can control and inspect that process.

The GDB user interface itself typically runs on your own machine. You can start or attach to processes even outside of your own machine. GDB provides a program called gdbserver for this task. gdbserver manages process execution without providing a user interface itself. GDB then attaches to gdbserver over the network using a TCP socket.

gdbserver isn’t the only program you can attach GDB to. Programs that are compatible to the gdbserver protocol are called GDB stubs. Almost any platform has a GDB stub: even QEMU has its own gdbserver compatible implementation. This lets you attach GDB to a virtual machine without having to run gdbserver on it.

Can’t find the GDB stub to solve your stub problems? No problem: you can make your own GDB stub using gdbstub.

To add another level of complexity, reverse engineering tools like Ghidra offer another level on top of GDB. When you have a working GDB stub, you can reap the benefits even when you are using Ghidra. Here’s a guide on how to talk to gdbserver from Ghidra.

Running gdbserver on embedded devices

Back to gdbserver: this program can run on any platform that supports GDB. If your target isn’t powerful enough to run a full GDB, you can just run gdbserver. Many times, debug targets don’t offer any user interface. gdbserver itself is small and can be copied over to your debug target, for example with TFTP. This is useful when your target device has only limited storage capacity.

This happens a lot when working with embedded Linux. Or, what the cool kids call “IoT” nowadays.

Most distributions of GDB compile it using the GNU C library. Yet, some targets may not have the GNU C library available and use a different C library instead. No problem: you can compile gdbserver as a static binary. When gdbserver is a static binary, it doesn’t need to load external shared libraries, including the GNU C library.

The target that I was recently working with doesn’t use the GNU C library. The GNU C library is also called glibc. Instead, this target’s programs are all built with uClibc.

It’s difficult to run dynamically linked programs on platforms with different C libraries. Even if they use the same dynamic library, you can encounter version compatibility issues.

Using static linking gives you more control over how your programs are executed. This sometimes means that your program needs more storage. Functions that are normally shared are now included in your program’s binary data.

If you can live with a slight increase in required storage, static linking makes your life easier when working with embedded devices.

Process memory issues

Another issue with my target is that it uses an old Linux kernel 2.6.36 version. Modern GDB changed the way it inspects and modifies running processes. Old Linux kernels around version 2.4.0 didn’t let processes write to the process memory at /proc/PID/mem. With newer Linux kernels, this is now possible.

Recently, GDB started using /proc/PID/mem instead of ptrace to change the target process memory. This also applies to setting breakpoints. Because of this, old Linux and new GDB don’t play along well without some changes.

Without fixing this issue, I managed to run gdbserver on my target, but not do any useful debugging. gdbserver wasn’t able to set any breakpoints. This is the error message that GDB connected to gdbserver shows when trying to set a breakpoint:

(gdb) break $function_name
Breakpoint 1 at 0xXXXXXX
(gdb) si
Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0xXXXXXX
Cannot insert breakpoint -12.
Temporarily disabling shared library breakpoints:
breakpoint #-12

I wasn’t familiar with how GDB sets breakpoints and manipulates process memory in the first place.

I found this discussion on the Sourceware Bugzilla helpful. According to the participants there, using ptrace to change program memory was always more of a short-term solution. Instead, the “proper” way of writing to process memory in Linux is by writing to /proc/pid/mem. Enabling this behavior in the kernel was in the pipeline for a long time. See this LWN article right here.

The reason older Linux kernel versions turned it off is that, according to the LWN article, writes to /proc/pid/mem were seen as a “security hazard.”

The LWN article mentions that the patch set enabling this new feature is based on Linux version 2.6.38-rc8. My debug target’s kernel version is 2.6.36. Since that comes after my target’s kernel version, it becomes clear why my target doesn’t support writing to process memory. With a later Linux kernel version, this wouldn’t have been an issue.

gdbserver as part of GDB 15.2 writes to the process memory to set breakpoints using /proc/pid/mem. My target’s kernel won’t let gdbserver write to the process memory this way and it fails. GDB’s error message doesn’t tell you that it failed to write to the process memory and gives you another cryptic error message.

You can reproduce this problem more directly by writing to the process memory instead of settings breakpoints.

In the following snippet, I’ve connected used the set command in GDB to overwrite the memory at a specific address:

(gdb) set *0x42a000 = 1
Cannot access memory at address 0x42a000
(gdb)

You can instruct gdbserver to give you a detailed log of what it’s doing using the --debug=all flag. Refer to the gdbserver documentation to learn more about its command line arguments. When gdbserver receives GDB’s set command, it prints the following output to the console on the target:

[event-loop] handle_file_event: invoking fd file handler `remote-net`
[threads] handle_serial_event: handling possible serial event
[remote] getpkt: getpkt ("X42a000,4:");  [no ack sent]
[threads] write_memory: Writing 01000000 to 0x0042a000 in process 331
[remote] putpkt_binary_1: putpkt ("$E01#a6"); [noack mode]

The $E01#a6 string is the raw message that gdbserver receives over TCP. You can read more about the wire format used by GDB here

Does this error mean that GDB has a regression? Or is my target’s kernel to blame? As always, it’s a combination of two pieces of software relying on features that the other one doesn’t want you to use this or that way. And then calcification sets it and people start relying on this new behavior.

The maintainers of GDB didn’t expect someone to try running it on an old Linux kernel running on MIPS and I don’t consider this anyone’s fault.

The Sourceware Bugzilla discussion links to a patch that you can use to make modern GDB use ptrace again.

This patch fixes the problem. With this patch, gdbserver first tries to write to the process memory directly and checks the result of that write. If it fails, it reverts back to using ptrace to write values to the process memory instead.

Download the patch file from the preceding link and name it 210-use-ptrace.patch.

Missing header file

Another issue was that GDB wouldn’t compile correctly because of a missing C header file. The header file that GCC claims is missing is sgidefs.h. This error doesn’t happen when compiling GDB as a dynamically linked package. When compiling GDB as a static package instead, Nix uses the Musl C library instead of the GNU C library. I don’t understand the exact relation between those two errors. Either way, when compiling against the Musl C library, GCC complains about a missing sgidefs.h header file.

I found a workaround for this issue in a merge request in the crosstool-ng’s GitLab repository here. By applying the path file 200-use-asm-sgidefs.h.patch in the merge request, you can fix this error.

Nix flake file

With the two patches 200-use-asm-sgidefs.h.patch and 210-use-ptrace.patch, create the following Nix flake file:

# Save the following contents as `flake.nix`
{
  description = "MIPS GDB cross compile";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          crossSystem = {
            config = "mipsel-linux-gnu";
          };
        };
      in
      {
        packages = rec {
          # pkgsStatic since GDB should be statically compiled
          # (no GNU C standard library)
          gdb = pkgs.pkgsStatic.gdbHostCpuOnly.overrideAttrs
            (final: super: {
              patches = super.patches ++ [
                ./200-use-asm-sgidefs.h.patch
                ./210-use-ptrace.patch
              ];
            });
        };
      }
    );
}

Build GDB by running the following command:

nix flake build .#gdb

When the build finishes, Nix places a symbolic link called result in the current folder. The gdbserver executable binary is contained in result/bin/gdbserver. Use TFTP or a file transfer tool to copy gdbserver to your debug target.

Connecting GDB and gdbserver

On the target, attach gdbserver to a running process using the following command:

# Assuming you have pidof available and are
# connected to a Bash-like console
gdbserver --attach 0.0.0.0:4444 $(pidof process_name)

On your own machine, run GDB by executing gdb in the terminal. Inside GDB, attach to the gdbserver by running the following:

# Assuming that the target's IP address is 10.0.0.1
target remote 10.0.0.1:4444

You can also start GDB and connect immediately by starting gdb like so:

gdb -ex 'target remote 10.0.0.1:4444'