qstr Generation¶

MicroPython's qstr (interned string) mechanism is central to the interpreter's identity-based attribute lookup. Every method name, attribute name, module name, and error message is registered as a qstr at build time and gets a small-integer ID.

tools/zig_qstr_shadow.py extends MicroPython's qstr/module scanner to recognize Zig source files. Without it, Zig modules couldn't fully participate in qstr-based dispatch.

What qstrs are¶

A qstr is an interned, immutable string with a known integer ID. The MicroPython runtime stores qstrs as small u16 / u32 values; comparing two qstrs is an integer compare (much cheaper than a string compare). That makes attribute access fast: looking up obj.method translates to "find the dict slot whose key is qstr-id 1234", one integer compare per key, no string traversal.

Examples:

MP_QSTR_print          // qstr ID for "print"
MP_QSTR___init__       // qstr ID for "__init__"
MP_QSTR_StopIteration  // qstr ID for "StopIteration"

These macros expand to integer constants at compile time. The integer values come from qstrdefs.generated.h, which is auto-generated by sandbox/micropython/py/makeqstrdefs.py based on which qstrs the source code references.

The build-time scanner¶

makeqstrdefs.py runs as part of make port. Its job:

Preprocess every .c source through cpp -E -DNANOPYTHON_CONFIG_FULL=1 (the build runs with full to ensure the qstr set covers both arms).
Scan the preprocessed output for MP_QSTR_<name> references.
Emit qstrdefs.collected.h listing every unique <name>.
Sort + dedupe + assign integer IDs → qstrdefs.generated.h.

So if any .c file mentions MP_QSTR_threshold, the qstr threshold gets an ID and the dispatch code can use it. If no .c file mentions it, the dispatch code that tries to use the qstr will produce a compile error: 'MP_QSTR_threshold' undeclared.

The Zig problem¶

Zig sources are NOT scanned by makeqstrdefs.py. It only looks at .c extensions. So if a Zig module references mp.MP_QSTR_threshold, that reference is invisible to the qstr generator, and MP_QSTR_threshold never gets assigned an ID.

The early workaround was the tombstone .c file pattern: keep a small .c file for each Zig module solely to hold the MP_QSTR_* and MP_REGISTER_MODULE markers the scanner needs. Example pre-NX-2:

// src/modules/modgc_shim.c
#include "py/runtime.h"
#include "py/obj.h"

extern const mp_obj_module_t mp_module_gc;
MP_REGISTER_MODULE(MP_QSTR_gc, mp_module_gc);

// (qstr anchor for threshold; the real fn lives in modgc.zig)
__attribute__((used)) static const qstr threshold = MP_QSTR_threshold;

The Zig version of the module had the actual logic; the .c shim existed solely so qstr-gen would emit MP_QSTR_threshold into the generated header.

This was the "qstr-gen floor": the irreducible C surface that even an otherwise-pure-Zig module needed.

NX-2: scanning Zig¶

NX-2 added tools/zig_qstr_shadow.py. The trick: it generates a synthetic .c file from .zig sources, then the existing scanner picks it up.

The .zig file declares a special comment block:

//! nanopython — Zig port of py/modgc.c
//!
//! __qstr_shadow_begin__
//! #if MICROPY_GC_ALLOC_THRESHOLD
//! MP_QSTR_threshold
//! #endif
//! __qstr_shadow_end__

const std = @import("std");
// ... rest of module

The shadow generator reads src/**/*.zig, finds these blocks, and emits a synthetic .c file:

// Auto-generated by tools/zig_qstr_shadow.py.
#include "py/obj.h"
#include "py/qstr.h"

/* from src/modules/modgc.zig */
#if MICROPY_GC_ALLOC_THRESHOLD
__attribute__((used)) static const qstr npy_zig_qstr_anchor_1 = MP_QSTR_threshold;
#endif

This synthetic .c lands at port/build-embed/zig_qstr_shadows/zig_qstr_shadow.c. It's added to SRC_QSTR in port/Makefile, which makes it part of the input that makeqstrdefs.py scans. The scanner sees MP_QSTR_threshold (gated by the #if); the generated header gets the entry; the dispatch code compiles.

Setup in port/Makefile¶

ZIG_QSTR_SHADOW := build-embed/zig_qstr_shadow.c
ZIG_SOURCES := $(shell find $(abspath ../src) -name '*.zig')
QSTR_GLOBAL_REQUIREMENTS += $(ZIG_QSTR_SHADOW)
SRC_QSTR += $(ZIG_QSTR_SHADOW)

$(ZIG_QSTR_SHADOW): $(ZIG_SOURCES) $(abspath ../tools/zig_qstr_shadow.py)
    @mkdir -p $(dir $@)
    $(Q)python3 $(abspath ../tools/zig_qstr_shadow.py) $@ $(ZIG_SOURCES)

$(shell find ... -name '*.zig') is evaluated at Makefile-parse time rather than build time. That means new .zig files added between makes aren't automatically re-scanned. The known-minor issue: make port re-runs picks them up; incremental builds don't.

Block syntax¶

A shadow block lives between __qstr_shadow_begin__ and __qstr_shadow_end__ markers. Lines inside the block:

MP_QSTR_<name>: emitted as __attribute__((used)) static const qstr ... = MP_QSTR_<name>;
#if ... / #endif: emitted verbatim (so config-gated qstrs work)
Verbatim C (e.g., MP_REGISTER_MODULE(...), MP_REGISTER_ROOT_POINTER(...)): emitted as-is

The shadow file is one .c per build, with all blocks concatenated. The generator deduplicates by ordering paths consistently (sorted to keep the output byte-stable across rebuilds; caught and fixed in commit 9ea9a4b after the verifier flagged non-determinism).

Examples in the tree¶

// src/modules/file_io.zig
//! __qstr_shadow_begin__
//! #if MICROPY_PY_IO
//! MP_QSTR_FileIO
//! MP_QSTR_TextIOWrapper
//! MP_QSTR_buffer
//! MP_QSTR_readlines
//! #endif
//! __qstr_shadow_end__

// src/modules/modgc.zig
//! __qstr_shadow_begin__
//! #if MICROPY_GC_ALLOC_THRESHOLD
//! MP_QSTR_threshold
//! #endif
//! __qstr_shadow_end__

// src/modules/modmain.zig (NX-3 N4)
//! __qstr_shadow_begin__
//! extern const mp_obj_module_t mp_module___main__;
//! MP_REGISTER_MODULE(MP_QSTR___main__, mp_module___main__);
//! __qstr_shadow_end__

The third example illustrates MP_REGISTER_MODULE: that's how Zig modules participate in the module-registration table. Without it, the runtime's mp_module_get couldn't find the module.

What the shadow can't do¶

The synthetic .c file emits one symbol per qstr. It doesn't:

Initialize the MICROPY_REGISTERED_MODULES table. That table is generated by makemoduledefs.py as a struct initializer, listing every module's MP_REGISTER_MODULE invocation. The initializer expands into the table-consuming code, which is the mp_builtin_module_table definition in src/objects/objtype/objmodule_shim.c. That file can't move to Zig without porting the table-consuming side too, which requires running makemoduledefs.py output through a step that produces Zig source instead of C source. Not done; not in strangler scope.

This is why objmodule_shim.c is the 86-LOC architectural floor of compiled C. The shadow generator handles the qstr side; the module-table consumption side stays in C.

The conversion arc that NX-2 unlocked¶

Before NX-2: every Zig module that wanted to register qstrs OR a module name needed a tiny tombstone .c file. After NX-2: any Zig module can declare a shadow block and skip the tombstone entirely.

Net deletions in NX-2: - src/modules/modarray.c (8 LOC) - src/modules/modcollections.c (5 LOC) - src/modules/moderrno.c (8 LOC) - src/modules/modmath.c (7 LOC) - src/modules/modmicropython.c (9 LOC) - src/modules/modstruct.c (8 LOC) - src/modules/modsys.c (30 LOC, the most substantial, since it also held ROOT_POINTER and MODULE_DELEGATION macros) - src/modules/modbuiltins_shim.c (23 LOC) - src/modules/modgc_shim.c (9 LOC)

Total: 107 LOC of C eliminated. Cost: 114 LOC of Python (the shadow generator) + ~30 LOC of Makefile additions. Net negative in code line-count, but the structural win was bigger: any future Zig module port can now stay 100% Zig.

Limitations + extensions¶

Whole-block conditional gates¶

The shadow generator emits all blocks unconditionally and lets the C preprocessor (via #if) gate them. This means the #if MICROPY_X in your shadow block has to match the gate vendor C uses for the corresponding qstr. If you write #if FOO_BAR and the actual config macro is MICROPY_FOO_BAR, qstr-gen silently won't emit the qstr.

Comment markers¶

The block delimiters are //! prefixed (Zig doc-comment marker). The generator strips that prefix; any other comment syntax won't be recognized.

Path quoting¶

The generator outputs absolute paths in /* from /Users/.../src/modules/foo.zig */ comments. These paths leak into the shadow file. Bit-stable across rebuilds (sorted), but workspace-specific. For now, acceptable.

Future: scanning Zig for `MP_QSTR_*` directly¶

A more ambitious extension: skip the comment-block annotations and scan Zig source for MP_QSTR_<name> references inline. Problem: Zig uses mp.MP_QSTR_X (member access on the @cImport namespace), and @hasDecl(mp, "MP_QSTR_X") is a config-gated runtime check. Blindly anchoring every reference would over-include qstrs that some configs don't have.

The current explicit-block design avoids this; the cost is one annotation per Zig module that wants qstr-table participation. Tractable.