I'm building a handheld games console. Not a Raspberry Pi jammed into a case running RetroPie, not another Game Boy Emulator. A fully custom device with its own microcontroller, its own firmware, and its own SDK for third-party developers to write original games.

The hardware runs bare metal, meaning no operating system, no dynamic linker, and no runtime sitting between the firmware and the code running on the device. Games are separate binaries that get loaded onto the device independently of the firmware, which means that the firmware and any given game might have been compiled months or even years apart by completely different people.

Early on in the project I decided to keep the firmware closed source. Partly to protect the device from cloning, partly so I could iterate on the platform without exposing the internals. Turns out this decision has some interesting engineering consequences.

Closed source firmware means that game developers cannot compile directly against it, so separate compilation is a firm requirement. Separate compilation on bare metal, with nothing to mediate between the two binaries, raises the immediate question: how do you maintain a stable contract between them?

The chain of consequences looks like this:

Closed source firmware
    ↓
Separate compilation is mandatory
    ↓
A stable binary interface is the only contract
    ↓
The wrong primitive breaks old games silently
    ↓
Indexed vtable is the right primitive
    ↓
Games compiled today run on firmware shipped years from now

A note on Rust and ABIs

I chose Rust as the primary language for this project. It fits bare metal well: memory safety without a runtime, zero-cost abstractions, and excellent tooling. But Rust has no stable ABI of it's own, this isn't an oversight, this is by design. Two Rust functions can own reliably call each other when they are compiled at the same time, together. Once you introduce separate compilation, you have to opt into a known stable ABI explicitly.

This means falling back to the lingua franca of the compiled world: C. The extern "C" calling convention is not special because C is somehow better. It's special because everything agrees on it. It's the pin standard that all sockets accept.

The approaches I considered

The simplest idea:

Place the functions at a known address in memory, say for example 0x1000_0000. Both the firmware and the SDK know where to find the functions, no negotiation required.

… Add diagram here?

The problem with the direct flash layout? It becomes part of the public contract, permanently. Any changes to the firmware that shifts the functions position in memory (adding a feature, reorganising the code, updating a dependancy) slightly moves the functions away from their expected address. A game compiled against the old layout would call whatever now lives at 0x1000_0000. On bare metal with nothing to catch it would either causes a crash or silent corruption.

Fixed address tie the entire internal implementation to a public promise that can never safely be changed.

Struct of function pointers

This was my first serious attempt at tackling the ABI problem, and it wasn't too far from the right solution. Rather than using fixed addresses, it passes a struct of function pointers to the game at runtime. The firmware populates the struct and hands it over to the game at boot. The structure can live wherever the linker decides to put it.

// Firmware passes this to the game at startup
struct Abi {
    draw_pixel: fn(x: u16, y: u16, color: u16),
    draw_rect:  fn(x: u16, y: u16, w: u16, h: u16),
    play_sfx:   fn(sample: &[u8], channel: u8),
}

This looked promsing. No fixed addresses, no flash layout coupling. But the compiled game doesn't look up draw_rect by name or by any explicitly defined identifier. It reads whatever is located at a fixed byte offset into the struct, an offset determined by the struct layout at compile time and baked permanently into the game binary. If anything in the struct changes, those offset shift.

// v1 firmware
struct Abi {
    draw_pixel: fn(x: u16, y: u16, color: u16),      // offset 0
    draw_rect:  fn(x: u16, y: u16, w: u16, h: u16),  // offset 8
}

// v2 firmware, new function added
struct Abi {
    draw_pixel:  fn(x: u16, y: u16, color: u16),        // offset 0
    draw_sprite: fn(data: &[u8], x: i16, y: i16),       // offset 8  <- NEW
    draw_rect:   fn(x: u16, y: u16, w: u16, h: u16),    // offset 16 <- SHIFTED
}

// A game compiled against v1 reads offset 8 expecting draw_rect.
// It gets draw_sprite instead. Silent corruption.

You could make the structure approach work if you only ever appended new functions at the end, never inserting, removing, or reordering. But that constraint only exists as a convention. Nothing enforces it, and one mistake breaks every compiled game silently.

Bytecode VM

A different approach to the problem entirely: shiping the games as bytecode rather than native binaries, and running an interpreter on device. This sidesteps the ABI problem completely because we would control the instruction set and there is no binary contract to break. It would also enable developers to use a very friendly scripting language such as Lua to develop games, lowering the barrier to entry significantly.

Sounds like an ideal solution! But the cost is performance. The hardware is a microcontroller, not an application processor, and a bytecode interpreter carries substantial overhead on top of an already constrained device.

We might revisit this as a secondary option later on to create a beginner friendly sdk.

Indexed vtable

An indexed vtable is an array of function pointers where each array slot has a permanent numeric index that never changes, regardless of what else changes in the firmware.

// The vtable: a flat array of function pointers
static VTABLE: Abi = Abi {
    functions: [
        draw_pixel   as usize,  // index 0, permanent
        draw_sprite  as usize,  // index 1, permanent
        draw_rect    as usize,  // index 2, permanent
        clear_screen as usize,  // index 3, permanent
        play_sfx     as usize,  // index 4, permanent
        // new functions appended here, never inserted
    ]
};

Index 0 is always draw_pixel. Index 4 is always play_sfx. New functions added in future versions are always appended at the end of the array, and every existing index stays exactly where it was.

This approach feels similar to the structure of pointers at first inspection, but it differs in one critical way. It makes the same constraints explicit and permanent. Each function has a named, numbered identity that is part of its public definition rather than an anonymous position derived from struct layout. The game binary contains a reference to index 1, not offset 8. Index 1 means draw_sprite because that is what was explicitly defined and committed to, not because of some arbitrary position in memory that it sits in today.

The failure mode if we break either contract is identical: old games call the wrong memory address with no warning. The difference is how easy it is to break accidentally. For these reasons we decided to proceed with the indexed vtable approach.

How the vtable reaches the game

The firmware doesn't place the vtable at a fixed address. It passes a pointer directly to the game at boot time, a pattern called runtime injection.

// game_init is the game's entry point, called by the firmware at boot
// The firmware passes the vtable pointer as an argument
#[no_mangle]
pub extern "C" fn game_init(abi: *const Abi) {
    // The SDK stores this pointer internally
    // The game developer never sees it
}

This method decouples the vtable from the flash layout entirely, making for a more robust implementation. The firmware is able to place the vtable wherever it chooses and the game receives that address at runtime rather than assuming it at compile time.

I later discovered that Panic's Playdate uses the runtime injection pattern as well, having arrived at it independently though the system's constraints, and discovering the convergence later was a good sign.

Things I can never do

Two things are off limits once the vtable is stable:

  • Change what an index points to. Index 4 is play_sfx, if the function signature needs to change in a breaking way, it's get's a new index and the old one stays in place for compatibility.
  • Remove an index. If an index is removed then every game that calls it would break. Deprecation is fine, but removal is not.

The indexed vtable is a public contract, and a contract is only useful if it's unconditional.

Where Rust pays off

Crossing the vtable boundary is an inherently unsafe act, it involves: Raw Pointers, transmute, and C ABI calling conventions. This is unavoidable at the boundary between two separately compiled binaries.

It is not however, the game developer's problem.

I decided early on that unsafe parts of the code belong in one place, and that place is not the game developer's mental model. The SDK includes a macro that generates all of the unsafe boundary-crossing code at compile time. Game developers have to write none of it.

// What the game developer writes
console.draw_sprite(&sprite, x, y);

// What the macro generates, entirely out of sight
unsafe {
    let f: fn(&[u8], i16, i16) = core::mem::transmute(
        (*ABI_PTR).functions[1]
    );
    f(sprite, x, y);
}

The implementation crosses an unsafe boundary, but the API does not. Game developers should be thinking about game logic, not ABI mechanics, and the macro is where that separation is enforced.

console.draw_sprite()         <- developer writes this, pure safe Rust
    ↓
safe Rust SDK function
    ↓
macro-generated unsafe block  <- all unsafe lives here, generated at compile time
    ↓
vtable lookup by index
    ↓
firmware implementation       <- closed source, free to change
    ↓
pixels on screen

What this gets you

At the machine level none of this is complicated. An indexed vtable is a contiguous block of memory containing function addresses. The runtime injection is a single pointer to the start of this memory passed to the game at boot time. The macro generation is a one-time cost at compile time.

What you get in return is a platform that can actually evolve. Firmware updates, bug fixes, new features, performance improvements: none of which touches the compiled game binaries. A game built today will still run on firmware released years from now because the indices are permanent and the contract will not have exceptions.

One business decision (keeping the firmware closed source) produced a chain of constraints that gave me a platform built to last.

No OS. No linker. No broken games.

This is the first in a series of articles about building a handheld games console from scratch: custom hardware, bare-metal Rust, and an SDK for built for everyone. More soon.