DWM3001CDK surgery#

Hard-won, non-obvious findings from putting a hand-written Matter node next to the credential UWB reader on a DWM3001CDK (nRF52833 + DW3110, NCS v3.3.0 / Zephyr). The goal was a board that goes from factory to walk-up unlock and a working Home tile with nothing but the Home app — no build-time key, no donor ESP32, nothing typed in.

The sibling document for the other port is esp32-gotchas.md. This one is narrower and deeper: the nRF52833 has 128 KB of RAM, and almost every trap below is either that constraint or a consequence of writing the Matter node by hand rather than linking CHIP.

Code lives under firmware, the portable Matter node in modules/ultrawidelock_matter, and the reader in modules/ultrawidelock_cred (shared byte-for-byte with the ESP32 port — see §9.2).

---

1. RAM is the binding constraint, and it bites as stack overflows#

1.1 The budget, measured#

MEASURED. Reader + hand-written Matter node + Thread MTD/MED, in the RELEASE=1 SMP=1 image after the five-fabric transaction work: 120,740 B of 131,072 B (92.12%), 10,332 B free. Application flash is 397,360 B of 433,664 B (91.63%). The like-for-like increase against clean main is 4,992 B of RAM and 8,560 B of flash; both builds used NCS v3.3.0, Zephyr 4.3.99, LTO, release logging, and SMP.

That is no longer the shape of the problem. After the speed/robustness work the same image measures 118,312 B RAM (90.3%) and 417,684 B of the 433,664 B app partition (96.3%) — roughly 12.8 KB of RAM spare against 15.6 KB of flash. RAM was reclaimed and flash was spent, so flash is now the binding constraint, and it is the one to check first. Everything below about how to measure RAM still applies; only which number is closest to the wall has changed.

127,352 B has been recorded as unrunnable on this board. That is an observation, not a spec, but treat it as a ceiling: builds above it have failed to come up. Every RAM change needs a boot check (§8.4), and the ECDH self-test line is the cheap proof the image is alive:

*** Booting nRF Connect SDK v3.3.0 ***
ECDH self-test: PASS (NIST CAVP P-256 CDH count 0)

1.2 Three different stacks overflowed, all from adding one feature#

VERIFIED. Adding persistence to the Matter node produced three MPU stack-guard faults on three different threads, over a few hours. None was diagnosable from the symptom, and two presented as something else entirely:

The lesson is not "these numbers": it is that a default is a starting point, not a size, and that two of the three were within a few hundred bytes of fitting. A margin that thin fails only when something else happens to run at the same moment, which is why identical firmware worked for weeks and then failed twice in an evening.

1.3 Read the paint; never estimate#

VERIFIED. CONFIG_INIT_STACKS=y fills stacks with 0xAA and costs nothing at runtime. A debugger reads the high-water mark straight out of RAM, so no analyzer thread is needed:

# addresses from build-*/app/zephyr/zephyr.map
JLinkExe … -CommandFile <(printf 'savebin s.bin, 0x20015900, 0x1040\nq\n')
# skip the 64 B MPU guard at the low end, count the leading 0xAA run:
#   peak = (size - guard) - run

CONFIG_THREAD_ANALYZER_AUTO measures the same paint and costs a thread plus its stack — enabling it took an image to 96.6% RAM and past the unrunnable figure. Use the paint, not the analyzer, on this part.

Two traps when reading it:

• The MPU guard occupies the first 64 B and is never painted. Counting 0xAA from offset 0 reports "100% used" on a healthy stack.
• Symbol addresses move with every BSS change. A stale address reads as garbage, and a stale last_count (§8.3) reads as a hung board. Re-read from the map after every build.

1.4 Where the bytes went, and the cheap reclaims#

MEASURED. Two changes cost far more than they looked:

• Raising ULTRAWIDELOCK_TRUST_MAX 4 → 8 grew ULTRAWIDELOCK_PROV_BLOB_MAX 476 → 864 B and struct ultrawidelock_trust_store 390 → 778 B. A function holding both had a 1,642 B frame, which went through the bottom of the 4 KB main stack. Fix: the blob is static, and the cap settled at 6.
• A struct matter_im_read made static to describe one attribute costs ~264 B, because it carries MATTER_IM_MAX_PATHS. On the stack instead, on a work queue with measured headroom.

General rule on this part: large single-use objects belong on a stack with known headroom, not in BSS — but only after that headroom has been measured.

---

2. Matter state must not be written from the OpenThread thread#

VERIFIED. Matter datagrams arrive through the OpenThread UDP callback, so the entire Interaction Model — decrypt, decode, cluster command, response encode, framing — runs on ot_work_q. Two separate failures came from that:

1. An NVS write there overflows it. CommissioningComplete → store fabrics → NVS faulted after both fabrics were accepted, both CASE sessions established and the subscription primed. The pairing succeeded on the wire and failed anyway.
2. A write anywhere in a handshake stalls it. Storing at AddNOC wrote ~1.7 KB across several settings keys; an NVS sector erase on this part runs to tens of milliseconds. The commissioner retransmitted Sigma1 and the second fabric's CASE died with Sigma3 REJECTED (-6) ×5, then RemoveFabric.

Current rule: prepare only provisional state before CommissioningComplete, then cross a durability boundary on the system work queue before returning success. The OpenThread callback waits on a bounded semaphore; it never writes NVS itself. A failed persistence operation therefore fails the command instead of acknowledging an identity that will disappear after reset.

Each attempt owns only the slots and staged Thread data it created. Completion promotes those slots to committed state; fail-safe expiry clears only those provisional slots. An established Apple fabric is no longer collateral damage when a later Home Assistant commissioner aborts.

---

3. A subscription the node can actually serve needs four things#

VERIFIED. "Matter Accessory / No Response", and a Home tile stuck spinning on Unlocking, were four independent bugs. Each looked like the whole problem on its own.

3.1 The priming report must fit the IPv6 MTU#

MATTER_MAX_MESSAGE_LEN (1232) is the ceiling for the whole message — 1280 less the IPv6 and UDP headers — so the exchange headers (36) and the AEAD tag (16) come out of it, not on top. Spending all 1232 on the payload builds a datagram up to 52 bytes over the MTU.

Nothing logs. The framing succeeds, the send returns fine, and the datagram is simply never delivered, so the subscriber re-subscribes forever and the CASE table churns.

BLE hides this. BTP re-fragments, so the identical report crosses a commissioning session intact and only subscriptions carried over Thread fail — which presents as an accessory that works while pairing and dies immediately after. Tell it apart by which transport the established subscription sat on, not by whether one established at all.

3.2 The node must be able to initiate an exchange#

Everything a responder-only node sends answers something. After the priming report the server has to speak unprompted, and a controller that gets no report calls the accessory unresponsive however healthy the session is.

The session role is unchanged, which makes this cheap: keys stay role-relative to CASE (still encrypt with r2i) and the message counter is per-session, not per-exchange. Only the exchange role differs — set I, use an id of your own, and do not write it back over the peer's live exchange id.

Three subtleties, all found by tests rather than by reading the code:

• Never piggyback the peer's pending ack onto an exchange you just opened. An ack names a counter within an exchange.
• Do not clear ack_pending on that path. Clearing an ack that was never encoded makes the peer retransmit a message already handled, and the exchange that owes the ack stalls.
• Acknowledge a subscription report's StatusResponse on the report's OWN exchange, with I set. CHIP matches an inbound message to an exchange by id and by the initiator flag being the opposite of its own (ExchangeContext::MatchExchange), so an ack framed with I clear on the peer's exchange id matches nothing and is dropped as unsolicited -- and the peer retransmits the very message being acknowledged. Every LockState report drew the full MRP schedule until matter_exchange_ack_initiator() existed.

3.3 Report on change#

The tile reads LockState, not the InvokeResponse. A controller takes the SUCCESS and then waits for the attribute to be reported before the UI moves — so answering the command and stopping there is a lock that opens and a UI that spins forever.

3.4 Report on a timer, not only on change#

Matter's contract is a report at least every max_interval whether or not anything changed (600 s is what Apple asks for here). Reporting only on change means a lock nobody touches for ten minutes stops existing.

One timer for all subscriptions, not one each: they carry the same attribute, the interval is a floor rather than a schedule, and six timers are still unnecessary at 92.12% RAM. 120 s is deliberately early — a report is ~67 B on a link whose round trip measured 1.4 s, so being early is nearly free and being late is the entire failure. Stop re-arming when nothing is subscribed, so a node nobody watches is not waking its radio.

3.5 Bridge the reader's own state into LockState#

VERIFIED. A walk-up unlock never touches the Door Lock cluster — it is the reader's own Aliro transaction — so the tile keeps showing whatever the last tile tap set. The Wallet animates unlocked while the app says locked, and the app is not wrong so much as uninformed.

Hook the point that sends the reader-status notification, because it carries both transitions (the grant that fires the animation and the walk-away relock). Do not use a credential-verdict callback: it fires on the unlock and never on the relock, so the tile would show a lock that opens and never closes.

---

4. Persistence, and what must not be persisted#

4.1 Nothing Matter-side was persisted at all#

VERIFIED. The fabric table was plain RAM, so every reset silently un-commissioned the node: it came back advertising commissionable, Thread never started because nothing replayed the dataset, and the controller showed an accessory that was simply gone. Every flash cost a full re-pair, which is also why iterating on this port was so expensive.

The current image persists the fabrics, per-fabric ACLs, Thread dataset, xPAN id and the ICAC slot. Then restore is not enough on its own: a restored identity is commissioned but not reachable until the dataset is handed to the stack and one SRP instance per fabric is registered. That pair is what commissioning does as a side effect; the boot path has no commissioner to trigger it.

4.2 Versioned per-slot records, not a table rewrite#

The mf2 namespace stores metadata, network state, one record per fabric, one ACL record per fabric, and the shared ICAC. Each record is versioned, sealed, and written through the backend's atomic replacement primitive. A removal writes a valid tombstone before returning success, so a power cut cannot expose an older deleted fabric. One corrupt fabric or ACL record is discarded without destroying its neighbours.

The fabric record also carries the fabric's UpdateFabricLabel string, which is why an image from before that field cannot be restored: record_read() rejects any record whose stored length is not the length it expects, so a pre-label identity is dropped at load and the node comes back uncommissioned. One re-pair, once.

The serializer is a bounded 528 B static union, not an object on the OpenThread stack. The settings region is 16 KB at 0x7c000; Zephyr NVS and the FreeRTOS log use different media formats but implement this same transaction contract.

4.3 Completion belongs to an attempt, not the node#

The old global commissioning_complete boolean made fail-safe rollback a no-op as soon as any administrator had completed. Two bitsets now distinguish committed slots from the slots owned by the active attempt. The persistent records contain only committed fabrics, so a reboot cannot promote a half-added controller and a later PASE session cannot erase a working one.

4.4 Erase SRP identity as one unit#

The original failure remains instructive: keeping a stable SRP host name while destroying its client key can make a border router reject the new owner as OT_ERROR_DUPLICATED. The current port persists the SRP key and service-name identity together. Its registration slots have live, removing, and free lifetimes, and their OpenThread service structures are not reused until the asynchronous removal callback returns them. A duplicate registration is retried with a fresh service name instead of being logged as false success.

make flash-erase still intentionally clears commissioning and reader state. Use controller RemoveFabric for one administrator and SW2 only for an unreachable last-resort reset.

4.5 An erase that cannot fail visibly#

VERIFIED. Discarding every settings_delete() return behind (void) and logging "erased" unconditionally made a wipe that removed nothing indistinguishable from one that worked. Several hours were spent re-clearing a board that kept coming back with the same fabrics. Report each key's rc.

---

5. Advertising: one payload, one gate, and three ways to lose it#

5.1 Only one payload fits a legacy advert#

VERIFIED. Flags (3) + Matter service data (12) + Aliro service data (26) = 41 bytes of the 31 available. A second advertising set costs ~24.8 KB of RAM, which this part does not have. So the node advertises commissionable while it holds no fabric, and Aliro only once it does.

5.2 Restarting advertising from the disconnect callback fails#

VERIFIED. bt_le_adv_start() returns -12 (-ENOMEM) when called from inside the disconnected callback — Zephyr has not released the connection object yet. Defer it to a work item and retry.

Worse than the failure was the silence: the advert logged only on success, so a failed restart left the reader invisible with nothing in the log but the "re-advertising" line before it. The board had unlocked once and then ignored every approach, while the Home tile kept working, because Matter runs over Thread and this is BLE. An advertising failure must never be quiet — it presents as dead hardware and points the investigation everywhere except at advertising.

5.3 Matter provisioning must refresh the advertisement#

VERIFIED, and it hid for weeks. A phone resolves a reader by a dynamic tag derived from the GRK. The reader starts advertising long before SetAliroReaderConfig arrives, since the controller sends it as a post-commissioning operational command — so at start the GRK is the dev default's all zeros and only the bare 0xFFF2 UUID goes out.

Without an explicit refresh the board ends up provisioned, holding both credentials, reachable over Matter, tile working — and invisible to every walk-up.

A reboot hides it, because the boot path applies the stored GRK before it advertises at all, and every earlier test power-cycled after pairing. It only surfaced when a reboot happened before a pairing instead.

5.4 The advert gate must be re-run after a restore#

The advert is chosen at BLE start, before stored fabrics are loaded. Without re-running it, a restored reader keeps advertising commissionable and never offers 0xFFF2 again — a node that unlocks until its first reboot and silently stops after it.

---

6. The trust store, and the issuer/endpoint key trap#

6.1 A full store must evict, not refuse#

VERIFIED. An Apple home installs two endpoint keys per pairing, they accumulate, and nothing removed them. With a cap of 4, the store filled on the second pairing and trust_add() answered a full store with a permanent refusal — so the key the phone actually presents could never be added, and pairing again only made it worse by adding more stale anchors. There is no recovery from that on a board with no console.

Observed as 13 consecutive walk-ups reaching device signature OK and then credential key NOT trusted. The two unlocks that appeared to work in the same session had gone through the expedited-fast path, which skips the trust check entirely — so the store had not matched a presented key once.

Evict a slot that has never completed a standard phase first (no Kpersistent means no phone ever authenticated with it), and only then the oldest.

6.2 Log the operands, not just the verdict#

"not in trust store" names the comparison but not what was compared. The two candidate explanations — a credential that was never delivered, versus stale anchors crowding out the current one — are told apart only by the bytes. Printing the presented key beside every anchor turned an evening of guessing into one attempt.

6.3 Credential types, and a conclusion that was wrong twice#

VERIFIED. SetCredential delivers type 6 (issuer key — correctly refused as an anchor) and type 7 (evictable endpoint key — stored). A pairing observed 3 calls: one type 6, two type 7.

This project recorded, then withdrew, then partly re-recorded a claim about these keys. The lesson that survived: an absence in a capture is evidence about the capture, not about the protocol. Captures that only ever showed type 6 were captures of pairings that never got far enough to send type 7 — the subscription bug (§3.1) was stopping them.

---

7. A failed pairing used to be a brick#

VERIFIED on the old design, hit four times in one evening. A commissioning could install and persist a fabric, time out, and leave the advert gate offering Aliro 0xFFF2 to a controller that had already forgotten the accessory.

The current design has two recovery levels. Fail-safe expiry rolls back only the active attempt, including its staged Thread data, ACL, ICAC ownership, SRP service, sessions, and subscriptions. A fabric that did reach CommissioningComplete can be removed by an authenticated surviving administrator; the targeted tombstone is durable before the success response.

Factory reset on SW2 held through reset remains the last resort when no administrator can reach the node. led0 blinks to confirm the hold, the reader identity, every trust anchor, and every Matter fabric are erased, and the boot continues commissionable. It is no longer the normal response to a failed share.

If you are recovering a board without that button (older images), the equivalent is a one-boot clear flag, flashed once and then flashed away.

---

8. Debugging on this board#

8.1 Poll the RTT ring with JLinkExe#

VERIFIED, after two wrong turns.

• JLinkGDBServer is wrong here: it disturbs the target enough to stop BLE, and pairing then fails with the phone stuck on "connecting". Flowing RTT does not prove the core is running — that argument was made and was wrong.
• JLinkRTTLogger only works in the foreground, so any keystroke kills the capture, and it degrades to "RTT Control Block not found" on every invocation until the board's USB is replugged (nothing in software cures it).
• JLinkExe never failed, including while the logger was broken, and is non-interactive so it backgrounds cleanly.

Read WrOff/RdOff, savebin the ring, slice host-side for the wrap, then write RdOff back. Without the write-back the ring fills and NO_BLOCK_SKIP discards everything new — which looks exactly like a board that stopped logging.

Control block layout from the map: aUp[0] = _SEGGER_RTT + 24, then sName/pBuffer/SizeOfBuffer/WrOff/RdOff at +0/+4/+8/+12/+16.

8.2 Make a failed probe read loud#

Silently continuing on a read error makes a broken J-Link indistinguishable from a quiet board. That ambiguity cost an hour.

8.3 Prove liveness independently#

Read nrf_rtc_timer's last_count twice: it advances at 32,768 Hz iff the kernel runs. Re-read its address from the map after every build — a stale address returns a constant and reads as a hung board (this produced exactly one false "STILL HUNG").

For a fault, halt and read IPSR: 004 is MemManage. Compare the faulting SP against the stack regions in the map — an SP below a stack's base is that stack overflowing.

8.4 Prove reachability independently of the controller#

dns-sd -B _matter._tcp local shows the operational services; dns-sd -Gv6 <host>.local gives the address; ping6 settles it. 0% loss while the controller reports "No Response" means the problem is above basic Thread reachability — that single check redirected the investigation more than once. The current DWM image is an rx-on MED, not a sleepy end device.

Note the mDNS records are cached by the border router, so stale instances from earlier failed pairings linger and are not evidence of anything.

---

9. Controller behaviour, and how to test against it#

9.1 Every reboot costs ~10 minutes of controller sulk#

VERIFIED, and it is not a bug to fix. Matter subscriptions are RAM on both sides. After a reset the controller keeps retransmitting into sessions the node lost — visible as encrypted for session 0xNNNN, which is not ours — and re-subscribes only when its own max_interval expires.

The node cannot force it. The exchange id needed to answer is inside ciphertext it has no key for. (An earlier claim here that a real node replies with a StatusReport was withdrawn; CHIP drops these too.)

The practical consequence dominates everything else in this document:

The other half of the same trap: the controller provisions late. Credentials have landed ~40 minutes after the pairing UI finished. A capture shorter than the thing being waited for will say it never happened.

9.2 modules/ is shared — run both suites#

VERIFIED the hard way. A change under modules/ultrawidelock_cred was validated against the host suite only. The ESP32 suite — which is the only one that builds ultrawidelock_reader.c — had been broken by it two commits earlier and nobody noticed:

make test                  # host suite
./tests/ports/esp32/run.sh  # host-compiled, no ESP-IDF, no hardware, seconds

When behaviour changes deliberately, update the stale assertions to the new behaviour rather than silencing them — a test asserting that a full trust store refuses is asserting the bug §6.1 exists.

9.3 A test proves nothing until you watch it fail#

House rule, and it earned its keep repeatedly here: revert the fix, confirm the exact checks fail, restore. Two real bugs in §3.2 were found this way and not by reading.

9.4 Check exit codes, not output#

A build reported success because the pipeline ended in tail, and the failure was invisible. Write to a file and test $?.

---

10. What is still open#

• CONFIG_ULTRAWIDELOCK_PROV_CLEAR_ON_BOOT does not apply. Set in prj.conf or an overlay it still leaves # CONFIG_… is not set in .config, with no Kconfig warning, while a symbol added 22 lines later in the same file applies fine. The symbol has no depends on and no enclosing if/menu. Unexplained. Workaround: force the #if IS_ENABLED(...) to #if 1 for one boot, then revert.
• Apple Home plus Home Assistant hardware fault injection. Five-fabric, rollback, removal, response-replay, persistence, Thread-staging, and SRP behavior are host-tested and the DWM image builds and fits. The live CDK-19 through CDK-26 rows in hardware-validation.md have not run.
• One ICAC owner. Five fabrics fit, but the RAM-bounded portable table has one shared ICAC buffer. A second fabric that requires its own intermediate certificate is rejected without mutating the existing owner.
• RAM at 90.3%, flash at 96.4%. Flash is now the tighter of the two: about 15 KB spare in the app partition against roughly 13 KB of RAM. Any further work starts by measuring, not by adding, and a new static allocation is now a smaller decision than a new code path.

11. Three bugs the host suite could not see#

All three shipped in the speed/robustness work, all three passed make check with thousands of assertions green, and all three stopped the walk-up dead on a real DWM3001CDK. They share one root cause, so they are worth reading together.

The pattern: a fixture that agrees with itself proves nothing. Each bug was a comparison between a value derived by a helper and the same value as it arrives on air. The host tests built both sides from that same helper, so the fixture agreed with itself no matter which byte order the code chose. The radio does not have that luxury.

1. KeySource compared in the wrong byte order. ccc_uad_addresses() emits KeySourceHigh || KeySourceLow; ccc_parse_mhr() copies the Aux Security Header field in transmission order, which is the reverse. A straight memcmp rejected every Pre-POLL. Observed: uad ks=0f3795ed against hdr ks=ed95370f.
2. DestShort assembled little-endian instead of MSB-first. Same function, same root cause, and it survived the first fix because the diagnostic printed the two halves in different formats: uad dest=02ff | hdr dest=02ff is the byte pair 02 ff against the u16 0x02ff, which read the other way is 0xff02. They were never equal. If you print two things to compare them, print them the same way.
3. ranging_block compared against a stale snapshot. The block number reaches the receiver through the deferred Pre-POLL decode, while the Final evidence is snapshotted synchronously in the Final RFRAME callback. The two are one block apart whenever the stash has not drained: blk=2 want=1, with session id and STS index both matching exactly. The check was dropped rather than repaired -- final_sts_index derives from g_armed_index, increments once per block, and binds the round more tightly than a 16-bit counter does.

Symptom to remember. Bugs 1 and 2 present as tx0 in the ⟐ heartbeat with sts●: frames arrive, none are answered, no · NNcm ever appears, and the session dies on credential phase deadline expired. tx0 means the Response was never armed, which puts the fault at or before POLL. Bug 3 presents as healthy tx and a healthy error ratio with still no · NNcm.

Why none of it was visible. Every reject path in ccc_shim_rx.c reports through DIAGK, which a pretty-shell build compiles to nothing. A rejected Pre-POLL had no observable at all. The Pre-POLL accepted: URSK proven on air line exists for exactly this reason and is the first thing to look for; if it is absent, the gate ahead of it is rejecting and you will need to promote the reject paths to ultrawidelock_printf temporarily to find out which.