bruce/switch_gateway_firmware
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

core1-mqtt #1

Merged
bruce merged 18 commits from core1-mqtt into main 2026-06-09 01:46:25 +00:00
Conversation 0 Commits 18 Files changed 41 +3702 -3911
bruce commented 2026-06-09 01:46:19 +00:00
Owner
Copy link
No description provided. 
First commit of the core1-mqtt work (moving the Python MQTT bridge on-board
onto core 1 over the W5100S). Build-system only; no behaviour change yet.

- Switch PICO_BOARD to wiznet_w5100s_evb_pico: pins PICO_PLATFORM=rp2040,
  the W5100S SPI0 pins, 2 MB flash + W25Q080 stage2. (Was bare Pico 2/rp2350.)
- Add submodules lib/ioLibrary_Driver (W5100S socket/DHCP/DNS) and lib/coreMQTT
  (pinned v2.3.1, static / malloc-free / float-free). Vendor lib/jsmn/jsmn.h.
- Link pico_multicore; add the jsmn include dir. Libraries are in-tree but not
  yet compiled — wired into the build by the later net commits.
- button_actions.c: replace the hand-rolled PIO->IRQ ternary (whose fallback
  PIO2_IRQ_0 only exists on RP2350's 3rd PIO block) with the platform-agnostic
  pio_get_irq_num(); RP2040 has 2 PIO blocks. Required for an RP2040 build.

Builds clean for RP2040 (text 214 KB, bss 38 KB of 264 KB SRAM), no warnings.
Full design: docs/core1-mqtt-plan.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the struct contract between core 0 (domain) and core 1 (the upcoming
network bridge). No JSON crosses cores — events and commands are typed C
structs over two pico_util SPSC queues, leaving the SIO FIFO free for the
flash_safe_execute() lockout handshake. The largest message, scan_complete,
is 9 bytes here vs ~4.5 KB as JSON.

This commit is purely additive: the existing USB-CDC JSON emit/parse path is
untouched and still works (removed in a later commit). Core 0 now ALSO:
  - forwards every emitted state event to core 1 as an ic_event_t, and
  - drains an inbound ic_cmd_t queue, dispatching through the SAME domain
    entry points the JSON/CLI use (ic_dispatch_cmd).

Details:
  - intercore.h/.c: ic_event_t (11 variants) / ic_cmd_t (9 variants) + two
    queues. Event queue depth 512 so a full synchronous resync burst
    (~386 events incl. all mappings) never drops. Provision msg is not
    carried (core 1 synthesises it from the state enum) to keep the event
    element ~32 B. NB queue_init() calloc's the rings once at boot (~16 KB);
    a permanent allocation, not runtime alloc/free.
  - serial_protocol.c: tee in serial_poll (emit_event JSON to USB + push
    struct), inbound command drain, levt_to_ic converter, ic_dispatch_cmd.
    emit_state_dali() also pushes — it is the only path a config-only
    set_dali (which produces no dirty-poll event) reaches core 1.
  - button_fsm_emit.c / serial_protocol_provision.c: push button / provision
    events alongside their existing printf.
  - intercore_init() runs in serial_protocol_init(), before the main loop.

Builds clean for RP2040 (text 217 KB, bss 39 KB), no warnings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring up core 1 as a stub network bridge and make the single hardware
watchdog tolerant of a network outage while still catching a real core-1 wedge.

  - core1.c/.h: multicore_launch_core1_with_stack() with an explicit static
    8 KB stack. core1_entry() calls flash_safe_execute_core_init() first
    (registers core 1 as the flash lockout victim — the one app-side action
    needed for multicore flash safety; pico-vfs already wraps the writes),
    then loops beating a heartbeat and draining/discarding the event queue
    (the net stack replaces the discard later). core1_launch() blocks
    (bounded 1 s) until that init is done, so a subsequent core-0 littlefs
    write is safe.
  - main.c: launch core 1 after intercore_init(); replace the two
    unconditional watchdog_update() calls with wd_feed(), which feeds only
    while core 1's heartbeat is fresh (< 2 s) or within a 15 s post-launch
    grace window. A network outage keeps core 1 looping/beating so the chip
    never reboots; only a genuine wedge stops the beat and lets the 5 s
    watchdog fire.

BSS +8 KB (the core-1 stack); builds clean for RP2040 (text 213 KB, bss 47 KB).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revert a premature optimisation. The previous commit dropped the gateway's
free-form provisioning status message from the event struct (to keep the
element ~32 B) and planned to synthesise it on core 1 from the state enum.
That loses real detail HA shows in its "Commission Status" sensor — error
specifics, per-step progress, prose counts — and adds a synthesiser core 1
would not otherwise need.

RAM is plentiful (264 KB), so carry the 80-byte msg verbatim, faithfully
reproducing the Python bridge. The event queue is ~50 KB of heap now; fine.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Get the board onto the network. This is the netif half of the original
"commit 4"; the TCP socket state machine + DNS move to the coreMQTT commit,
where the connection manager that drives them lives.

  - net/wizchip_glue.c: SPI0 + CS/RST/INT glue and the ioLibrary callbacks.
    The cris callback disables interrupts around each W5100S access so a
    register/burst transfer can't be interrupted mid-transaction by core 1's
    flash-lockout (SIO) IRQ — which would park core 1 with CS asserted while
    core 0 writes flash. Each access is microseconds, so it only briefly
    defers a flash write, never blocks it.
  - net/netif.c: non-blocking bring-up/DHCP/link/recovery state machine.
      * reliability #9: getVER()==0x51 sanity; full RSTn reset + wizchip_init
        re-init with exponential backoff on repeated failure (SPI desync
        leaves the chip unrecoverable at the socket level).
      * reliability #10: CW_GET_PHYLINK — parks in LINK_WAIT with the cable
        out instead of churning.
      * RTR=200 ms / RCR=3 so a dead peer surfaces in ~0.8 s (tunable).
      * stable per-board locally-administered MAC from the flash uid.
      * DHCP lease + renewal (default); static addressing supported (the
        /config.json net settings are wired in a later commit).
  - core1.c: netif_init(NULL) + netif_tick() each loop; alive log now reports
    link/ready. (Still discards events — MQTT publish replaces that.)
  - CMake: build ioLibrary (wizchip_conf/socket/w5100s/dhcp) with _WIZCHIP_=
    W5100S, link hardware_spi + pico_unique_id; -w on the vendored sources so
    our -Wall -Wextra output stays clean.

Builds clean for RP2040 (text 223 KB, bss 48 KB), zero warnings in our code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stand up the broker connection on core 1 and make it self-healing. Treats the
connection as disposable: any fault tears down to a known state and rebuilds.

  - net/mqtt_app.c: a 5-state connection manager (OFFLINE -> TCP_CONNECTING ->
    MQTT_CONNECTING -> ONLINE -> BACKOFF) over a single non-blocking W5100S
    socket. Implements the socket-level reliability requirements:
      #1 SF_IO_NONBLOCK + getSn_SR() state machine; every op has a wall-clock
         timeout (TCP connect 5 s) so connect() can't wedge the loop.
      #2 explicit SOCK_CLOSE_WAIT -> disconnect()+close() recycle.
      #3 act on Sn_IR (CON / DISCON / TIMEOUT) then clear via setSn_IR().
      #5 setSn_KPALVTR (6 * 5 s = 30 s) once ESTABLISHED.
    coreMQTT on top via a TransportInterface (SOCK_BUSY -> 0, error -> -1):
      #6 MQTT_ProcessLoop tracks PINGRESP natively; a keepalive timeout returns
         an error and we reconnect.
      #7 Last Will <prefix>/availability = "offline" (retained) at CONNECT.
      #8 exponential backoff (1 -> 30 s) with integer-LCG jitter at the TCP+MQTT
         layer; faults close the TCP socket abruptly so the LWT fires and HA
         sees us offline at once.
    Connected/message hooks are exposed for the translation layer; on connect we
    publish availability=online. MQTT_Connect timeout (1200 ms) stays under the
    core-1 heartbeat-staleness threshold.
  - core_mqtt_config.h: state-array/buffer sizing; coreMQTT logging routed to
    the '#'-prefixed CDC log (errors/warnings only).
  - Broker addressed by IP (recommended fixed-LAN setup). Compile defaults are
    placeholders (0.0.0.0:1883, prefix "lighting", no creds); real values wired
    from /config.json later. Hostname/DNS is a noted follow-up.
  - core1.c: mqtt_app_init/tick; alive log now reports mqtt state.

Builds clean for RP2040 (text 242 KB, bss 50 KB), zero warnings in our code.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port mqtt_bridge.py's auto-discovery to core 1. On each MQTT (re)connect the
full retained entity set is (re)published so HA repopulates from scratch.

Published INCREMENTALLY — one entity per disc_step(), a bounded batch per tick,
gated on free socket-TX space (TX_HEADROOM). Blasting ~600 retained configs at
once would overrun the 2 KB socket TX buffer and freeze the core-1 heartbeat;
the tx-free check means a ~650 B publish never meets a full buffer, so it flows
at network speed and never blocks. Payloads are built one at a time into a
single reused 1.5 KB buffer (no full-set-in-RAM).

Entities (shapes mirror mqtt_bridge.py exactly — topics, device blocks,
availability arrays, value templates, patterns):
  - 24x7 switch-plate event + binding-text entities (+ binding state seeds)
  - 32 relay switches (gated on system + gateway availability)
  - 64 ballast lights per DALI slave (+ per-ballast availability seeded offline;
    scan_complete brings present ones online), availability_mode "all" over
    system + gateway + per-ballast
  - 16 group lights + 16 members-text entities per slave (+ state seeds)
  - per gateway: 3 diagnostic sensors, 4 commission buttons, 1 target number
    (+ target_state / status seeds)
  - gateway/relay-board reachability availability seeded online

Enablers:
  - intercore: a core0->core1 config snapshot (slave lists) filled in main()
    before core1 launch — discovery needs to know the slaves.
  - mqtt_app: getSn_TX_FSR accessor for discovery flow control.
  - ha_bridge.c: the discovery cursor + builders + connected hook.

Controller-config and per-ballast config-number entities are published lazily
on first state_cfg / state_dali (next commit, with the event translation).

Builds clean for RP2040 (text 250 KB, bss 52 KB), zero warnings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the mqtt_bridge.py port. The board now publishes live state to HA and
acts on HA commands directly — no Python bridge in the path.

Outbound (ic_event_t -> retained MQTT state), mirroring the bridge:
  relay/dali(state+config)/group(state+members)/gateway-availability/button/
  mapping(+binding)/provision-status(with verbatim msg + state-enum->string)/
  config-state. Per-ballast and controller config-number entities moved into
  the incremental discovery cursor (TX-safe) rather than lazy bursts.

scan_complete's 64-address availability sweep is its own incremental cursor:
  popping a scan_complete starts a sweep that publishes a bounded, TX-gated
  batch per tick and pauses event draining until done — a single event would
  otherwise blast >2 KB and stall.

Inbound (MQTT -> ic_cmd_t -> core 0) via jsmn + direct parsing, covering every
subscribed topic: relay/dali/dali-config/group/group-members/provision(all/
unaddressed/address/target/abort)/mapping set+delete/binding/config set+field.
Target-address is latched per slave like the bridge.

Reconnect recovery: on connect -> incremental discovery -> subscribe to the 15
command filters -> push IC_CMD_RESYNC. Core 0 re-emits hello + state_cfg + all
cached state + scan_complete + mappings, repopulating every retained topic.
While disconnected, queued events are dropped (resync repopulates).

core-0 change: ic_dispatch_cmd's resync also emit_state_cfg() so core 1 can
build the controller config device.

Builds clean for RP2040 (text 260 KB, bss 52 KB), zero warnings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidate the four persisted files to two. config.json now also holds the
modbus baud + DE mode and the MQTT + network settings; mappings.json is
untouched (kept separate for its tolerant, frequently-rewritten format).

  - config.c: parse with jsmn (malloc/float-free; JSMN_STATIC so config.c and
    ha_bridge.c each get a private copy). Flat keys (mqtt_host, net_ip, ...) for
    simple, robust, human-editable JSON. One-time migration imports
    /modbus_baud.txt + /modbus_config.txt then deletes them.
  - modbus.c: read baud + DE mode from g_config (set by config_load before
    modbus_init); modbus_set_baud / set_modbus_de_mode now write g_config +
    config_save instead of their own files. Removed the now-dead text-file
    parse helpers. (The one called-out core-0 domain edit.)
  - intercore snapshot extended with the resolved broker IP/port/creds/prefix
    and the net settings; main() fills it (parsing the broker dotted-quad to an
    IP) before launching core 1.
  - core1: build netif_config_t + mqtt_config_t from the snapshot — the bridge
    now connects to the configured broker instead of the placeholder default.
  - CLI: `mqtt set host=.. port=.. user=.. pass=.. prefix=..` / `mqtt show`,
    `net set dhcp=on|off ip=.. mask=.. gw=.. dns=..` / `net show`. Changes save
    to config.json and apply on next boot (snapshot is read at core-1 launch).

Compile defaults are placeholders only (broker unset, prefix "lighting", DHCP);
no secrets committed. Builds clean for RP2040 (text 273 KB, bss 52 KB).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The batch wipe-and-replace transaction existed only for bridge/sync_bindings.py,
which is now obsolete — Home Assistant (with the mapper tool) owns binding via
the per-binding set_mapping / del_mapping path that core 1 exposes.

  - delete bridge/sync_bindings.py
  - remove the replace_mappings_begin/add/commit/abort JSON handlers and
    diff_emit_cb from serial_protocol.c
  - remove the mapping_txn_* backend + the s_txn_scratch table from
    mapping.c/.h (frees ~3.8 KB BSS)

Builds clean for RP2040 (text 272 KB, bss 48 KB).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
USB CDC is now a human text CLI only — all JSON emit/parse is gone. JSON lives
solely at core 1's MQTT edge. The state emitters still feed core 1 (typed
structs); they just no longer print JSON to USB.

  - emit_state_dali/relay/group/cfg, emit_config_mapping, serial_provision_poll:
    converted from "printf JSON + push struct" to push-only.
  - removed emit_event (the levt->JSON formatter); serial_poll forwards events
    via levt_to_ic only.
  - removed handle_json_line (inbound JSON command parser) — handle_line ignores
    any stray '{' line; the text CLI is the only USB command surface.
  - removed serial_tx_ack + the dead inbound provision JSON handlers
    (provisioning is driven via MQTT -> ic_dispatch_cmd) and the prov_json
    helpers; serial_tx_hello is now a human text log.
  - dali_xlsx_sync.py (pure text CLI) and manual screen/picocom are unaffected;
    `mqtt set`/`net set` configure the broker/network over the same CLI.
  - README: on-board (no Python bridge) topology, RP2040/W5100S pinout, build
    submodules, and a protocol-section banner noting the inter-core struct
    contract + text-only USB. mqtt_bridge.py kept as the behavioural reference.

Final RP2040 footprint: text 262 KB / 2 MB flash (13%); 48 KB static SRAM +
~50 KB heap (inter-core queues) + 8 KB core-1 stack of 264 KB — ~150 KB free.
Builds clean, zero warnings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First on-hardware test: getVER() read 0x00 (SPI desync) at the initial 30 MHz
SPI clock. Dropping to 10 MHz and lengthening the reset (RSTn low 10 ms, 150 ms
PLL-settle before the first access) brings the chip up cleanly — VER=0x51, PHY
link up, DHCP lease acquired. 10 MHz is ample for MQTT and conservative for the
board's signal integrity.

Also adds a write/read-back diagnostic in the VER-mismatch path (re-read VER +
RTR write/read) to distinguish a dead SPI link from an anomalous read on any
future failure.

Validated on a WIZnet W5100S-EVB-Pico: "net: DHCP lease ip=192.168.1.234" /
"core1: alive: link=1 ready=1".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On-hardware: config/mappings wouldn't persist on a fresh board. Root cause —
fs_init() ran in main() BEFORE core1_launch(), so the first-boot littlefs
format (a flash write via pico-vfs) happened before core 1 had registered as
the flash lockout victim. flash_safe_execute() then returns NOT_PERMITTED (no
lockout helper; PICO_FLASH_ASSUME_CORE1_SAFE defaults to 0), the format fails,
and the filesystem never mounts — so every fopen() fails. (The old single-core
board only ever *mounted* an already-formatted fs, so it never hit this.)

Fix: launch core 1 first (it calls flash_safe_execute_core_init), THEN fs_init().
Core 1 registers as the victim, signals ready, then parks on a new config-ready
gate until core 0 has loaded config + filled the snapshot, after which it brings
up the network. intercore_init() moves to main() (out of serial_protocol_init).
Also log fs_init failure.

Validated on hardware: cat /config.json now returns persisted settings; broker
config survives reboot.

Also adds an MQTT_Connect-failure diagnostic (Sn_SR + unread RX bytes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On-hardware: MQTT connected (CONNECT accepted by the broker) but the Pico never
received the CONNACK — MQTT_Connect failed with MQTTNoDataAvailable while the
diagnostic showed sr=ESTABLISHED, rx_rsr=4 (the 4-byte CONNACK sat unread).

Root cause is a bug in the vendored ioLibrary_Driver socket.c recv(): on the
non-IPv6 path the SF_IO_NONBLOCK check returns SOCK_BUSY *before* the
"if (recvsize != 0) break;" check, so a non-block socket's recv() never delivers
received data (the IPv6 path has the checks in the correct order).

Work around it in our transport_recv without patching the vendored lib: read
Sn_RX_RSR robustly (twice, until stable — W5100S volatile-register quirk); if
data is present, flip the socket to blocking just for that read (it cannot block
— data is already there), then restore non-block.

Validated on hardware against a native broker: full HA discovery (3219 PUBLISHes,
~510 KB) streams correctly; connect + discovery + resync all work. (A separate
finding: Docker Desktop for Mac does not forward the sustained PUBLISH stream
from an external LAN client to a published port — the broker must run natively
or otherwise be reachable; the firmware itself is correct.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The non-block recv() bug is tracked upstream (Wiznet/ioLibrary_Driver#173,
open since 2026-03-23, multiple reporters). Reference it in the workaround.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Keep the vendored ioLibrary_Driver submodule pristine and fix its
non-block recv() ordering bug (upstream issue #173 — returns SOCK_BUSY
before checking recvsize, so the MQTT CONNACK sat unread) via a patch
applied idempotently at CMake configure time.

- patches/0001-ioLibrary-recv-nonblock-issue-173.patch: the one-line fix
  (gate the SOCK_BUSY early-return on recvsize==0, matching the IPv6 path).
- CMakeLists: reverse-check to skip an already-patched tree; forward-apply
  otherwise; a failed apply only WARNs (an upstream fix obsoletes it).
- mqtt_app transport_recv: reverted the in-code BLOCK/NONBLOCK flip
  workaround back to the simple recv()/SOCK_BUSY path — the patch fixes
  recv() at the source, so the transport stays a thin wrapper.

Repeatable and trivially removable once upstream fixes #173. Flashed and
verified on the W5100S-EVB-Pico: discovery completes, MQTT stable
(link=1 mqtt=1, no churn).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
coreMQTT serialises each packet as several iovec fragments (fixed header,
topic-len, topic, [packet-id], payload). With only a scalar `send` in the
TransportInterface, sendMessageVector() called send() once per fragment,
and each ioLibrary send() issues a W5100S SEND immediately (no Nagle) —
so one PUBLISH left the wire as 3-4 tiny back-to-back TCP segments.

Implement the optional `writev` callback: gather all fragments into one
scratch buffer (MQTT_NET_BUF_SIZE; we have ample RAM) and send once.
Partial gather/send is handled — return the accepted count and coreMQTT
re-calls for the remainder — so large payloads and a near-full TX FIFO
still work.

Verified on the W5100S-EVB-Pico with en0 tcpdump: every PUBLISH now goes
as a single [P.] segment, immediately ACKed (state OFF=42B, ON=84B,
config=106B), instead of fragmenting per field.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MQTT bridge now runs on-board on core 1, so the external Python host
and its leftovers are gone. Also enable CLI echo by default and bring the
docs in line with the actual firmware.

Removed:
- bridge/ — mqtt_bridge.py + env/requirements (replaced by core 1; the
  rotating controller.log* and .env secrets went with the directory).
- docs/{core1-mqtt-plan,issue-raw-cmd-async,state-sync-review}.md — design
  notes for completed work; git history preserves them.
- src/bindings.txt, src/lights.png — unreferenced scratch data committed
  into src/.
- mapping_serialise_all() — dead since get_mappings switched to incremental
  per-binding emission.
- The stray JSON `button` printf to USB in button_fsm_emit.c — a leftover
  from the old USB-JSON protocol; the typed core-1 event is the only path.

Changed:
- CLI echo + backspace editing now default ON (serial_protocol.c): the CLI
  is a human-only interface now that machine clients are gone.
- `help` now lists the `mqtt` / `net` config commands (were dispatched but
  undocumented).
- README.md: rewritten against the code — text-CLI-only USB, real MQTT
  topic structure / discovery / availability from ha_bridge.c, key=value
  CLI grammar, /mappings.json + consolidated /config.json, editable DALI
  group membership. Removed the Python-bridge run/Docker sections and all
  references to removed commands.
- CLAUDE.md: documented the on-board core-1 architecture, the flash-lockout
  launch order, the four hardware-only bugs, the ioLibrary recv() #173
  build-time patch, the writev publish coalescing, and the echo default.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bruce merged commit a53f729825 into main 2026-06-09 01:46:25 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
bruce/switch_gateway_firmware!1
No description provided.