krane-fb-stub/RESEARCH.md

# Research log — krane display pipeline & stub bring-up

This file is the running lab notebook for the krane-fb-stub project: every
investigation done against the actual sources (coreboot, Linux, depthcharge —
both mainline trees and the *device-era* code as recovered from the SPI flash
version strings), what was found, and how each finding changed the payload.
Ordered chronologically; conclusions marked **[REVERSED]** were later
contradicted by deeper evidence.

---

## Phase 0 — Task framing

Goal: smallest freestanding arm64 binary that depthcharge boots as a Linux
kernel, locates the boot-splash framebuffer, and paints checkpoint colors
(red → yellow → green → blue, ~2 s each, hold blue), to validate the
depthcharge → custom-payload pipeline before any U-Boot bring-up.

Device ground truth (verified live):

- Lenovo IdeaPad Duet, MT8183, board `google,krane`, **sku176**
  (`sku-id = 0xb0` in `/firmware/coreboot` of the live FDT).
- coreboot table (LBIO) at `0xffed9000`, size `0x380`; CBMEM region follows.
  Confirmed both by `dtc -I dtb -O dts /sys/firmware/fdt` (kernel's copy of
  depthcharge's fixed-up tree) and by sysfs
  (`/sys/devices/platform/ffed9000.coreboot/coreboot7` carries
  `MODALIAS=coreboot:t00000012` = LB_TAG_FRAMEBUFFER present).
- Boot chain: BootROM → coreboot → TF-A BL31 → depthcharge → our payload.
- Partition layout: `mmcblk0p1` ChromeOS-kernel (what depthcharge boots
  internally), `sda1` (USB stick, pmOS) is the currently-running system.
- Device firmware strings extracted from the 8 MB SPI (`flashrom -p internal`):
  - `v1.9308_26_0.0.22-10476-g8c7859a3c9` (RO, built ~Feb 2021)
  - `v1.9308_26_0.0.22-10566-g8417b00957`, built `Mon May 30 03:26:46 UTC 2022`
    (RW — the slot that actually runs)
  Both are ChromeOS depthcharge ≈ v0.0.22 (R93-era, early 2021/2022).

## Phase 1 — Format verification (all checked against source, not the prompt)

### coreboot tables (`src/commonlib/include/commonlib/coreboot_tables.h`)

- `physical_address` is `lb_uint64_t` = `__aligned(4) uint64_t`
  (`LB_ENTRY_ALIGN == 4`). On LE AArch64 that is a plain 8-byte LE value at
  offset 8 of `struct lb_framebuffer` — the "hi/lo split" is an alignment
  trick, not separate fields.
- `sizeof(struct lb_framebuffer) == 40` (host-compiled probe against the real
  header). Offsets: tag 0, size 4, physical_address 8, xres 16, yres 20,
  bytes_per_line 24, bpp 28, red 29/30, green 31/32, blue 33/34,
  reserved 35/36, orientation 37, flags 38, pad 39.
- `lb_header` / `lb_record` match the prompt. Records start at
  `header_bytes`, walked `table_entries` times.
- Live cross-check: parsing the live `/sys/firmware/fdt` yields LBIO at
  `0xffed9000`, size `0x380` — matches the values confirmed by hand with dtc.

### arm64 Image header (Linux `booting.rst` + depthcharge `boot64.c`)

- 64-byte header, magic `ARM\x64` (0x644d5241) at 0x38, code0 must branch
  past the header; entry with x0 = DTB, x1–x3 = 0, MMU off, DAIF masked.
- `image_size` must be exact and little-endian (depthcharge checks the magic
  and uses `image_size` for KASLR slot math).

### depthcharge mainline (as of this repo's clone)

- `boot64.c`: kernel image is decompressed to a random 2 MiB-aligned slot;
  the Image magic is verified; entry = first byte; x0 = flat DTB, x1–x3 = 0.
  → stub must be **position-independent** (build enforces zero dynamic
  relocations) and carry its own stack (16 KiB in-image).
- **Contradicts the prompt**: `fit.c` *skips* FIT configs without an FDT
  subimage ("config %s has no FDT, skipping") and gives up with no match.
  So the packed FIT must contain a DTB. We embed the genuine
  `mt8183-kukui-krane-sku176` DTB (config conf-27 of the running p1 FIT)
  purely as a config-selection key; depthcharge injects
  `/firmware/coreboot` at boot via `src/boot/coreboot.c::install_coreboot_data`
  (adds `#address-cells`, `#size-cells`, `ranges`, `compatible`, `reg`),
  so the runtime DTB at x0 always has the node we parse.
- `mkdepthcharge` builds the FIT itself (`mkimage -f auto -T kernel
  -O linux -C none -d <vmlinuz> -b <dtb>`, then patches the kernel subimage
  to `kernel_noload`), so a raw arm64 Image goes in directly — no manual
  uImage wrapping. Verified: the packed image has the same shape as the
  working pmOS FIT (`kernel_noload` + `flat_dt`, compression none).

## Round 1 (eMMC flash, first attempt) — BLACK SCREEN, menu vanished

Observed: depthcharge dev-menu exits → black screen, nothing ever painted.
Had to force power-cycle; recovered via USB boot (backup `~/mmcblk0p1-pmos-
backup.img`, sha256 `20951a54…`, taken *before* the first flash).

### Investigation

The black screen was initially read as "stub crashed". The decisive evidence
came from identifying the **exact firmware versions in the SPI dump**:

- depthcharge `v1.9308_26_0.0.22-10566-g8417b00957` (RW slot, 2022-05-30)
  — i.e. ChromeOS R93-era (2021/2022), not mainline 2026.
- coreboot `4.14`-era (built ~Feb 2021 / May 2022).

Checking the chromiumos/depthcharge git history (`git log -S`) for the code
that ran *on the device*:

1. `69725330` (2020-09-22, "display: Black out screen in display_cleanup()",
   in R93): `display_cleanup()` at `CleanupOnHandoff` **before jumping to
   the payload** does:
   1. `clear_screen(black)` — erases the menu from the LBIO framebuffer;
   2. `backlight_update(0)` — drives `DISP_PWM` (GPIO 43) and
      `EN_LCD_BL` (GPIO 176) low;
   3. calls `display_ops->stop` → 2021-era `mtk_display_stop()`:
      `OVL_EN=0`, `OVL0_2L_EN=0` — the overlay fetch engine is stopped.
2. The DSI link, panel power and display MTCMOS remain up — panel/DSI
   poweroff only exists in 2025+ code (`e9f50b81`, `9be6dc52`, Dec 2025).
   Also `LB_TAG_PANEL_POWEROFF` is emitted only by skywalker boards.

**Conclusion:** the stub *did* run, painted red→yellow→green→blue and halted
on blue — into a framebuffer nothing was scanning, with the backlight off.
"Painted in the dark." The prompt's premise ("framebuffer is live and DMA'd
by the time any payload runs") is false for this firmware generation.

### Key register facts established (all from device-era sources)

- OVL0 base `0x14008000`; `DISP_REG_OVL_EN = 0x000C`, `DISP_REG_OVL0_2L_EN =
  0x100C` (2021 `mtk_ddp.c` — **[REVERSED]** this log first claimed 0x0F00
  for OVL_EN; see Round 2). The 0xF40 "L0_ADDR" register still holds the
  menu's scanout address after stop — it is not reset.
- GPIO controller at `0x10005000`; `dout` block at +0x140, 16 bytes per
  32-pin group, `set` at +4 / `rst` at +8. Backlight pins:
  `DISP_PWM = 43` (group 1, bit 11 → set reg `0x10005154`),
  `EN_LCD_BL = PERIPHERAL_EN13 = 176` (group 5, bit 16 → `0x10005194`).
- Panel: sku176 → `(sku_id >> 4) & 0xF = 11` → **BOE TV101WUM_NL6**
  (1200×1920, RGB888, 4 lanes, pixel clock 159.425 MHz → DSI data rate
  ≈ 956 Mbps). Not needed for the fix (panel stays initialized).

### The fix (`main.c::display_revive()`)

Minimal inverse of the three teardown steps, executed before painting:

1. `OVL_EN = 1`, `OVL0_2L_EN = 1` (offsets from device-era mtk_ddp.c);
2. GPIO `dout.set` writes for pins 43 and 176 (GPIO base `0x10005000`,
   dout block +0x140, 16 B per 32-pin group — layout from depthcharge's
   `mtk_gpio.c`/`mt8183.h`);
3. repaint — `OVL_L0_ADDR` still holds the address depthcharge used for the
   menu, which is the same LBIO-record address our parser extracts.

No DSI/panel re-init, no MTCMOS, no clock touching.

## Round 2 (post-revival reboot) — STILL BLACK: wrong OVL_EN offset

Observed: reboot into internal drive → pitch black after depthcharge boots
`mmcblk0p1`. No colors. p1 verified intact afterwards (first 159,744 bytes
hash to the payload; `vbutil_kernel --verify` on-device OK), so the stub ran.

### Live-environment re-checks

- `/dev/mem` LBIO read still blocked: pmOS kernel has `CONFIG_STRICT_DEVMEM`
  (confirmed in `/boot/config`), `0xffed9000` sits in
  `fae00000-ffdfffff : reserved` per `/proc/iomem` → `dd if=/dev/mem` gives
  "Bad address". kexec remains the only live-read route; not needed — see
  below.
- **Open item resolved without hardware access**: the runtime LBIO
  `physical_address` is provably non-zero. Depthcharge's device-era
  `mtk_display_init()` (`src/drivers/video/mtk_ddp.c`) programs
  `OVL_L0_ADDR` from `lib_sysinfo.framebuffer.physical_address` (i.e. the
  LBIO record) and the dev menu was visibly rendered through it;
  `src/drivers/video/display.c` also gates display handling on
  `physical_address != 0`. Mainline coreboot passes `fb_addr=0` for kukui
  (`fb_new_framebuffer_info_from_edid(edid, 0)`), but the device-era
  coreboot published a real address.

### Root cause

The stub's `display_revive()` re-enabled the overlay at the wrong register:
it wrote `OVL_EN` at `0x14008000 + 0x0F00`, but the actual offset in
depthcharge's `mtk_ddp.c` — confirmed in the 2021 revision (`497450b4`,
Jan 2021, inside the RO build window), the current tree, the merge commit
that unified mt8173/mt8183 (`74376061`, 2019), and Linux
`drivers/gpu/drm/mediatek/mtk_disp_ovl.c` — is `0x000C`.
The Round-1 "0x0F00 from device-era mtk_ddp.c" note was a misread
(0x0F00-era MT8173 offset / confusion with the 0x0F40 L0_ADDR neighborhood).
Consequence: depthcharge's stop wrote `OVL_EN=0` at `0x000C`, the stub
revived at `0x0F00` (a different, wrong register), the overlay engine stayed
disabled, and every fill landed in a buffer nothing scanned out. Painting in
the dark, again.

### Fix

`main.c::display_revive()`: `DISP_REG_OVL_EN` corrected to `0x000C`.
`OVL0_2L_EN = 0x100C` was already correct. Backlight GPIO writes were
correct (match `kukui_backlight_update`).

### Verification

payload `krane-fb-stub-payload.bin` sha256 `9e7cf29d…` (an intermediate
`41270e53…` rebuild was flashed once but built from the stale WIP tree
before the fix was synced into it — same wrong code as Round 1; caught by
disassembly before reboot), flashed to `mmcblk0p1`, `cmp` +
`vbutil_kernel --verify` OK. Awaiting reboot.

## Flash state

- `mmcblk0p1` reflashed with the Round-3 diagnostic payload:
  sha256 `9d7859dc04dd9c445143f6b45600bb972cf5fc1497c06a4f9db549e3d2971e73`,
  verified on-device (`cmp` first 159,744 bytes + `vbutil_kernel --verify`
  → body verification succeeded). Earlier: Round-1 `edb65b29…`, Round-2
  `9e7cf29d…` (both functionally identical wrong-offset code; `41270e53…`
  was a stale-tree rebuild caught by disassembly).
- USB stick `sda1` untouched throughout (sha256 watched: `c4ee2044…`).
- eMMC backup: `~/mmcblk0p1-pmos-backup.img` (sha256 `20951a54…`).
  **Copy it onto the USB stick before rebooting** so recovery never depends
  on eMMC.


## Round 3 — still black with corrected offset; diagnostic stub

Reboot with payload `9e7cf29d…` (correct `OVL_EN=0x000C`): still pitch
black. Fact check: all four teardown steps of device-era `display_cleanup`
are now provably undone — `clear_screen(black)` (repainted), the backlight
GPIOs (matches `kukui_backlight_update`), `OVL_EN`/`OVL0_2L_EN` (correct
offsets), and `disable_graphics_buffer()` (a libpayload double-buffer
helper, no hardware effect). Device-era `init_screen()` sequencing
(`ui/display.c`) proves the dev menu was drawn into the LBIO
`physical_address` and scanned out via `OVL_L0_ADDR` = same address, so the
stub's fills go to the right buffer. Revival logic is correct on paper —
yet black.

**The unexcluded branch: the stub may never execute.** "Menu vanished" only
proves depthcharge reached handoff — cleanup runs unconditionally. And the
internal-boot path was never baseline-proven: pmOS always ran from USB; no
custom image has ever been observed booting from p1 on this device. Every
black screen so far is equally consistent with "payload never ran".

### Diagnostic design (payload `9d7859dc…`)

Payload now paints/blinks BEFORE trusting any parsing:

1. First instructions: `OVL_EN=1`, `OVL0_2L_EN=1`, backlight GPIOs on.
2. Parse (<1 ms). Failure → 5 slow backlight blinks, spin.
3. Success → red → yellow → green → blue, blue held.

Decision tree: colors = pipeline validated; 5 blinks on black = parse
failed at runtime; nothing at all = payload never handed off (boot path).

### Abandoned: OVL_L0_ADDR stage-0 read

An earlier design read `OVL_L0_ADDR` (0x14008F40) pre-parse to paint the
menu buffer without knowing `pa`. Rejected: on qemu `-M virt`, READS of
unassigned MMIO data-abort (writes are silently dropped) — it killed the
test instantly; the same risk exists on real hardware if a display clock
gate is off, and a fault there is indistinguishable from "never ran". The
stub now contains no MMIO reads at all.

## Round 4 — still black, no backlight: GPIO dout offset was wrong too

Reboot with the Round-3 diagnostic payload (`9d7859dc…`): pitch black, no
blinks, **no backlight at all**. Stage-0 turns the backlight on as its
first act, and the backlight demonstrably works via these same GPIOs (the
dev menu had it) — so either the payload never ran, or the GPIO writes
missed. Re-derived the register layout from the device-era header
(`src/drivers/gpio/mt8183.h`) instead of the Round-1 notes:

    GpioRegs: dir[6]@0x000, rsv00[160], dout[6]@0x100, rsv01[160],
              din[6]@0x200; GpioValRegs = 16 B (val@0, set@4, rst@8)

**The Round-1 note's `dout @ +0x140` was wrong** — actual offset `+0x100`.
Consequences of the old addresses: the "pin 43 set" write to `0x10005154`
actually set bit 11 of group 5 = **pin 172 (I2S1_MCK)**; the "pin 176 set"
write to `0x10005194` hit a reserved region. Correct set/rst addresses:
pin 43 → `0x10005114`/`0x10005118`, pin 176 → `0x10005154`/`0x10005158`.
The Round-3 blink was additionally broken: it wrote 0 to a *set* register
(a no-op) instead of writing the bit to the *rst* register. Both fixed.

This also reframes Rounds 2–4: the OVL revival (correct offsets) and the
fills may have been **working the whole time** — with the backlight never
enabled, an LCD shows nothing regardless of what is scanned out. The only
proven failures were the two register-offset misreads in the Round-1
research notes (`OVL_EN 0x0F00`, `dout 0x140`), both now [REVERSED] and
fixed; the stub itself has never been disproven.

Payload `55ccc1bd…` flashed (cmp + vbutil OK). host/qemu tests pass.


## Diagnostic decision tree (after this fix)

| observation | meaning |
|---|---|
| black, no colors, no blink | payload never handed off — boot-path problem (vboot/mkdepthcharge), investigate p1 boot |
| black + 5 slow backlight blinks (NOW FUNCTIONAL — see Round 4) | stub ran, parse failed at runtime |
| colors appear but sequence stops early | fill/mask problem — stuck color identifies stage |
| red → yellow → green → blue, blue held | pipeline fully validated |

## Round 5 — blinks decoded: LBIO physical_address really IS 0

The Round-4 payload's diagnostic worked: 3–5 backlight blinks then steady
backlit black = `parse_fail()`. The stub runs, stage-0 works, the GPIO fix
works — the parse rejected the LBIO framebuffer record. Root cause found
in the actual coreboot **4.14** source (fetched from the 4.14 tag):

- `src/mainboard/google/kukui/mainboard.c` (4.14) is identical to
  mainline: `fb_new_framebuffer_info_from_edid(edid, 0)` — and 4.14's
  `edid_fill_fb.c` passes `fb_addr` through verbatim, no carveout, no
  allocation. **The LBIO record on this device genuinely has
  physical_address = 0.**

How the menu still renders: libpayload cbgfx draws into
`phys_to_virt(pa)` = DRAM address 0, and coreboot's display pipeline scans
out address 0. With `pa = 0`, depthcharge's `display_init_required()` is
false, so board.c never registers display ops — meaning at handoff
`display_cleanup` runs but `backlight_update` and `stop` are no-ops:
**the OVL was never stopped and the backlight never disabled by
depthcharge.** The "menu vanishing" was just the 9.2 MB black fill at
address 0 (safe for the payload: depthcharge itself does it after the
kernel slot is chosen, and the stub demonstrably survived every round).

**Round-2's "open item resolved" reasoning was wrong** — menu rendering
does NOT prove pa != 0, because address 0 works as a framebuffer region.
The original Phase-1 note ("mainline coreboot passes fb_addr=0 for kukui")
was correct all along. All five black screens so far trace to the stub's
`!fb->pa` rejection of a legitimate record (plus the two Round-1 register
offset misreads, which additionally kept the backlight dark).

Fix: `find_framebuffer` accepts `pa == 0` and the stub paints at address
0 — the buffer the still-running OVL scans. OVL revival and backlight
writes are kept but are now believed to be redundant on this firmware.

Payload `b25d9132…` flashed (cmp + vbutil OK). host_test gained a pa==0
regression test; qemu_test unchanged (its synthetic record uses pa!=0).

## Round 6 — pa==0 accepted but parse STILL fails: bit-band diagnostic dump

Reboot with `b25d9132…` (pa==0 accepted): **5 blinks again** — parse_fail
persists, and with pa==0 accepted the failure is somewhere else in
find_coreboot_reg or find_framebuffer. Every local reproduction passes:
host_test parses the live fdt (21/21 checks incl. the new pa==0 test),
qemu end-to-end passes, and the runtime tree shape is provably identical
to the live fdt (same fixup code; krane DTB has no pre-existing /firmware
node; root cells 2/2). The remaining unknowns are the actual runtime
values — what is really at x0 and what the real LBIO bytes are.

Key enabler: on failure the scanout address is KNOWN without any parse —
DRAM address 0 (Round 5; the visible menu erase proves the OVL scans it).
So the stub can render diagnostic data directly on the panel.

### New failure path (payload `e619166a…`)

`fail_dump()`: 5 blinks (execution proof, same signature), then paint a
bit-band dump into address 0: one row per u32, 32 cells of 32×32 px, MSB
first, white=1 / black=0, rows every 64 px. Word 0 = marker:

- `0xC0DE0001` — find_coreboot_reg failed; words 1-2 = x0 pointer,
  words 3-6 = raw DTB header (magic, totalsize, off_struct, off_strings,
  off_mem_rsvmap).
- `0xC0DE0002 | rc` — find_framebuffer failed; words 1-3 = LBIO
  addr/size, words 4-6 = raw table magic/header_bytes/entries, words 7-12
  = raw words at table+24..+68 (first record headers).

User photographs the panel; values are decoded offline.

## Round 7 — diag at address 0 invisible: cbgfx rejects pa==0

Reboot with `e619166a…`: 5 blinks, then **no bit-bands** — the render
into DRAM address 0 never reached the panel. That kills the "OVL scans
address 0" corollary of Round 5, and with it the whole address-0 theory:
the device-era libpayload `cbgfx_init()` explicitly REJECTS
`physical_address == 0` (`CBGFX_ERROR_FRAMEBUFFER_ADDR`). The menu could
never have been drawn through a zero pa — so the runtime framebuffer
address is NON-zero, and the LBIO record's pa (whatever it holds) is not
necessarily what the panel scans.

The authoritative scanout address is `OVL_L0_ADDR` (0x14008F40): it is
programmed by depthcharge's `mtk_display_init()` for the menu and left
in place by `stop()`. Round 5's "OVL never stopped / backlight never
disabled" reasoning is also [REVERSED] — with display ops registered,
`display_cleanup` really did disable the OVL and kill the backlight
(which is why Round 4's "no backlight at all" was observed).

### Payload `cebb9b1b…` (full file rewrite)

- `read_scanout_addr()` reads `OVL_L0_ADDR`; guarded as plausible DRAM
  (`>= 0x40000000`).
- Success path: if the LBIO record's pa is 0, substitute the scanout
  address, then paint checkpoints into the buffer the OVL actually
  scans (after stage0 re-enables the engines and backlight).
- Failure path: `fail_dump` blinks 5× then renders the bit-band dump AT
  THE SCANOUT ADDRESS; word 15 additionally carries the raw L0_ADDR
  value. Markers unchanged (`0xC0DE0001` DTB stage, `0xC0DE0002|rc`
  LBIO stage).

The OVL register READ is device-only (qemu -M virt aborts on reads to
unassigned MMIO; qemu never takes the failure/substitution path, so the
test is unaffected).

## Round 8 — hex-digit diagnostic dump

The bit-band dump WAS visible at the L0_ADDR scanout (proving both the
scanout substitution and the render path), but 32 coarse squares per row
proved unreadable from a photo, and the apparent row/column confusion
made transcription unreliable. The dump rendering is replaced with
**giant hex digits**: one u32 per line, 8 digits of a 3x5 cell font
scaled x10 (30x50 px), MSB nibble first, white on depthcharge's black
background. Readable in any panel orientation, transcribable as text.
Same data, same markers (`0xC0DE0001` DTB stage, `0xC0DE0002|rc` LBIO
stage, word 15 = raw OVL_L0_ADDR value).

Payload `10760e3c…` flashed (cmp + vbutil OK). host_test and qemu_test
pass.

## Round 9 — dump decoded: x0's FDT buffer is empty; LBIO fallback

The hex dump worked. Transcription (`C0DE0001` = DTB stage failure):

    x0            = 0x5F800000
    dt[0..3]      = 0, 0, 0, 0        (no FDT magic — buffer is zeros)
    OVL_L0_ADDR   = 0xFD536000        (real scanout, top-of-DRAM
                                       reserved region, matches
                                       /proc/iomem fae00000-ffdfffff)

x0 = `_fit_fdt_start` (device-era src/arch/arm/fit.c: the flattened tree
is placed in depthcharge's own `_fit_fdt_start.._fit_fdt_end` buffer and
that pointer is handed off). The kernel later reserves exactly
`5f800000-5f815fff` for its FDT (size ≈ the live fdt's 86738 bytes), so
the pointer is right — but for the payload boot the buffer content reads
as zeros. Why the kernel sees a valid FDT there while the payload sees
zeros is still OPEN (cache/flush difference vs the pmOS kernel boot is
the leading suspicion; the pmOS kernel is ~30 MB vs our 18 KB stub —
decompression footprint differs). For the stub this does not matter:

**Fallback added**: if `find_coreboot_reg` fails, use the fixed coreboot
table address `0xffed9000` (coreboot memlayout constant on this board;
confirmed by the kernel's own coreboot driver, sysfs tags, and
`/sys/firmware/fdt`). The stub no longer depends on the DTB at all.
`find_framebuffer` failure still dumps the raw LBIO bytes.

Payload `adbea06f…` flashed (cmp + vbutil OK). host_test and qemu_test
pass.

## Round 10 — SUCCESS: full pipeline validated

Payload `adbea06f…`: **red → yellow → green → blue, blue held.** The
depthcharge → custom-payload → display pipeline works end to end:

1. depthcharge boots the dev-signed FIT from `mmcblk0p1` and hands off
   (x0 = FDT buffer, x1-x3 = 0, MMU off);
2. stage0 re-enables OVL (+0x000C / +0x100C) and backlight (GPIO dout
   0x10005114 / 0x10005154);
3. the LBIO table at the fixed 0xffed9000 parses (record: pa handled via
   OVL_L0_ADDR substitution = 0xFD536000);
4. fills at the live scanout are visible.

Effective gotchas that cost nine rounds, all [REVERSED] or resolved:
OVL_EN offset (0x0F00→0x000C), GPIO dout offset (0x140→0x100), LBIO pa=0
legitimacy (cbgfx rejects it; scanout comes from OVL_L0_ADDR), and the
empty x0 FDT buffer (bypassed with the fixed table address; root cause
still open and RELEVANT FOR U-BOOT).

Next phase: U-Boot as the FIT kernel payload. Handoff prompt for the next
agent: `/home/vhaudiquet/u-boot-krane-prompt.md` (milestones: krane
defconfig/board, Image-header wrapper + flash flow, framebuffer vidconsole
reusing the validated revival/scanout code, then eMMC + bootflow; all work
upstreamable; x0-FDT-zeros trap documented).

Recovery: power-cycle, boot USB (unchanged), `dd if=mmcblk0p1-pmos-backup.img
of=/dev/mmcblk0p1 bs=4M conv=fsync`.

## Open items

- `physical_address` in the LBIO record — **resolved in Round 2**: provably
  non-zero at runtime (depthcharge's device-era `mtk_display_init` programs
  `OVL_L0_ADDR` from the LBIO record and the menu rendered through it; the
  `/dev/mem` read attempt failed on `CONFIG_STRICT_DEVMEM` + reserved-RAM,
  "Bad address", not EPERM as first assumed).
- qemu test updated to the same expectation (colors + held blue); it
  exercises parser + fill logic, not the MMIO revival writes.

## Round 11 — U-Boot as payload: target, driver, first flash (result pending)

First mainline-U-Boot flash after the Round-10 pipeline validation.

### Build (milestone 1)

- Branch `krane` in `/home/vhaudiquet/u-boot` (mainline main @
  `527115ef`); commits `29aec558` (board/defconfig/MAINTAINERS),
  `923cc754` (dtc include order), `f0470c9b` (video driver).
- `configs/mt8183_kukui_krane_defconfig` forked from pumpkin: keeps
  `CONFIG_POSITION_INDEPENDENT=y`, DEBUG_UART (0x11002000 @ 26 MHz,
  goes nowhere — absence of colors is not failure), MTK serial/MMC/WDT;
  drops fastboot/TPHY/mtu3/USB. `CONFIG_BOOTDELAY=-1` so the first boot
  stops at the prompt instead of autobooting pmOS from p3.
- DT: `CONFIG_OF_UPSTREAM=y`, `CONFIG_DEFAULT_DEVICE_TREE="mediatek/
  mt8183-kukui-krane-sku176"` — **with OF_UPSTREAM the name must include
  the vendor dir**; no arch/arm/dts registration needed.
- **New trap found while building**: U-Boot's legacy
  `include/dt-bindings/clock/mt8183-clk.h` (BSP numbering,
  `CLK_TOP_CLK26M=0`) shadows the upstream
  `dts/upstream/include/dt-bindings/clock/mt8183-clk.h`
  (`CLK_TOP_CLK26M=112`) for OF_UPSTREAM builds — the krane tree failed
  with undefined `CLK_CAM_LARB6`, and even fixing those macros would
  have silently produced wrong clock numbers everywhere. Fix:
  `scripts/Makefile.lib` prefers `dts/upstream/include` when
  `CONFIG_OF_UPSTREAM=y`. Pumpkin's in-tree dts (different binding
  generation) keeps working because its config does not set OF_UPSTREAM.

### Video driver (milestone 3, flashed before first reboot)

`drivers/video/mt8183_scanout.c` (`CONFIG_VIDEO_MT8183_SCANOUT`,
checkpatch-clean), binds the upstream `ovl0@14008000` node
(`mediatek,mt8183-disp-ovl`; the two 2L overlays match a different
compatible, so exactly one probe). probe():

1. revival identical to the stub's stage0: `OVL_EN=1` (+0x000C),
   `OVL0_2L_EN=1` (+0x100C), GPIO dout set for pins 43/176;
2. LBIO parse at the fixed 0xffed9000 (ported from the stub's
   Round-10-validated `find_framebuffer`; host regression test
   compiled the function standalone against `lbio.bin`: record parsed,
   bad signature rejected);
3. scanout = LBIO pa if >= 0x40000000, else `OVL_L0_ADDR` (+0x0F40)
   (Round-9 ground truth: 0xFD536000); else probe fails gracefully;
4. geometry from the record (xRGB 32bpp required; U-Boot's BPP32
   console default composes pixels exactly as r<<16|g<<8|b — checked in
   video-uclass `video_index_to_colour`);
5. both the coreboot table and the fb sit above the DTB's 2 GiB DRAM
   window (memory@40000000 is 0x80000000 long), so both regions are
   identity-mapped with `mmu_map_region()` first (MMIO < 0x20000000 is
   already mapped Device by `arch/arm/mach-mediatek/armv8-mem-map.c`);
   `video_set_flush_dcache(true)` for the console writes.

Revival happens BEFORE parsing, so "backlit black" = ran but parse/
probe failed, "dark" = never reached the driver — same decision tree
as the stub. No autoboot: U-Boot banner + prompt should stay on panel.

### Wrap + flash (milestone 2)

`krane-fb-stub/build-uboot-payload.sh`: 64-byte arm64 Image header
(code0 `b +0x40` = 0x14000010, image_size = 64 + len(u-boot.bin),
flags bit3, magic at 0x38) prepended to `u-boot.bin`, mkdepthcharge
with the krane DTB, `vbutil_kernel --verify`. (Header-bytes were
checked with `od` after an edit mishap dropped the flags word — always
re-verify the raw header bytes.) depthcharge decompresses the FIT and
jumps to the first byte — U-Boot's `_start` lands at +0x40, PC-relative
(PIC) so the arbitrary 2 MiB slot is fine.

- Payload `krane-uboot-payload.bin` sha256 `1fc74a0a…`, flashed to
  `mmcblk0p1` (dd 4M conv=fsync), `cmp` OK (589824 bytes), on-device
  `vbutil_kernel --verify` → body verification succeeded.
- Backup `mmcblk0p1-pmos-backup.img` (sha256 `20951a54…`) copied onto
  the USB stick (sda3) — recovery no longer depends on eMMC.

### Expected observation

Panel shows the U-Boot banner (white/light-gray text on black, portrait
1200×1920) and the prompt, frozen (BOOTDELAY=-1). Nothing on panel = boot
path or early crash; backlit black = reached handoff but driver probe
failed. The x0-FDT-zeros trap is bypassed: U-Boot uses its embedded DTB
and never reads the handoff FDT.

## Round 12 — first U-Boot boot: pitch black; diagnostic wrapper

Reboot with Round-11 payload `1fc74a0a…`: **pitch black, no backlight**.
Per the Round-11 decision tree the video driver's probe never ran (the
OVL/backlight revival is its first act). So either the image never
executed (but Round 10 proved the handoff path, and only the payload
contents changed) or U-Boot died between entry and the video probe — a
wide window (PIE fixup, relocation, DM scan, and notably `initr_env`
(MMC/clock probe) runs before `stdio_add_devices` in board_r.c, i.e.
before video probe and the banner).

### Localization flash (payload `9955943c…`)

The wrapper itself now carries the life sign, independent of U-Boot:
`uboot-wrapper.S` (linked at +0x40 inside the Image, immediate-encoded
PIC, no relocations) runs before U-Boot proper:

1. OVL revival + backlight (identical writes to the stub's stage0);
2. reads OVL_L0_ADDR, fills the scanout (guarded >= 0x40000000) with
   full-screen green (0x8ca000 bytes = 1200*1920*4);
3. branches (imm26 patched at build time) to U-Boot's entry at
   +0x40+wrapper_len (116 bytes).

Layout verified by disassembly before flashing: header
(code0 b +0x40, image_size 0x6d9ac, flags bit3, magic), wrapper
instruction sequence, patched tail branch, U-Boot intact at +0xB4.
Payload flashed, cmp + on-device vbutil verify OK.

### Decision tree

| observation | meaning |
|---|---|
| green screen + backlight | wrapper ran; U-Boot crashed before video probe |
| dark, no backlight | wrapper never executed — handoff/boot-path problem with THIS image |
| U-Boot banner | everything works (banner replaces the green) |

## Round 13 — green screen: U-Boot crashes pre-video-probe; init checkpoints

Reboot with `9955943c…`: **full-screen green, stable** — the wrapper ran
and U-Boot proper died between entry and the video probe (the video
uclass would have `video_clear`ed to black for the banner). The crash
window is wide: PIE fixup, relocation, DM scan, and notably
`initr_env`/`initr_mmc` run BEFORE `stdio_add_devices` in board_r.c.

### Checkpoint flash (payload `fc48d83b…`)

U-Boot itself now paints checkpoints into the live scanout from board
code (`mt8183.c`, TEMPORARY, gated on `CONFIG_BOARD_EARLY_INIT_F/R`,
`dcache_status()`-guarded with `mmu_map_region`+flush for the
post-relocation one):

| band | checkpoint | position |
|---|---|---|
| full green | wrapper | before U-Boot (Round 12) |
| orange top | `board_early_init_f` | after initf_dm, before get_clocks/timer/serial/dram_init |
| blue bottom | `dram_init_banksize` | after dram_init, right before relocation |
| red top | `board_early_init_r` | post-relocation, before dm/mmc/env/console/video |

Read: last paint reached = code survived through that checkpoint; the
crash/hang is in the next window. Success = banner on black.

## Round 14 — all green decoded: U-Boot's 4K-alignment requirement

Reboot with `fc48d83b…`: **all green, no bands** — U-Boot hung before
`board_early_init_f`. Root cause found in start.S: with
CONFIG_POSITION_INDEPENDENT, U-Boot verifies `adr _start` & 0xfff == 0
(it uses ADRP+ADD with lo12 relocations during the PIE fixup) and
otherwise loops in `wfi` forever — a silent hang, no exception. Our
wrapper pushed U-Boot's entry to load_base+0xB4: misaligned by
construction.

### Fix (payload `faee130f…`)

`build-uboot-payload.sh` now pads the wrapper so U-Boot starts at the
next 4 KiB boundary (0x1000) inside the 2 MiB-aligned image; wrapper
tail branch patched accordingly (0x140003d4 → 0x1000, verified by
disassembly). Keep this invariant for every future layout change.

## Round 15 — all black again (Round 14 anomaly); blink-train wrapper

Reboot with `faee130f…` (4K-aligned U-Boot): **all black, no backlight**.
Unexpected: the wrapper is byte-identical to Rounds 12/13 except its
tail-branch immediate, and it demonstrably painted green in both. Black
without any wrapper trace was only ever explained by "payload never
ran". Candidates for a genuine regression: none found in the wrapper
path; no U-Boot driver can touch the backlight GPIOs (no MTK GPIO
driver in-tree, PINCTRL off); nothing before the video probe touches
display HW. Working hypothesis: the Round-14 boot transiently did not
hand off (or the report reflects a state we cannot yet distinguish).
Note: a build slip was caught before flashing — the diagnostic hooks
were gc-section'ed out because `.config` was stale; defconfig was
re-run and the hook symbols verified with `nm` (board_early_init_f,
board_early_init_r, dram_init_banksize, misc_init_r all present).

### Round-15 payload (`b1c5c380…`, flashed, cmp + vbutil OK)

Maximally distinguishable phases:

- wrapper: **magenta** full screen + backlight, then **3 slow backlight
  blinks** (~250 ms each via the arch timer), then branch to U-Boot.
  Repeating blink trains = WDT reset loop; a single train then colors =
  wrapper ran; nothing at all = never ran.
- U-Boot checkpoints unchanged: orange top (`board_early_init_f`), blue
  bottom (`dram_init_banksize`), red top (`board_early_init_r`).
- new: **yellow band** + backlight re-assert in `misc_init_r`
  (CONFIG_MISC_INIT_R) — proves full init past the banner stage.

### Decision tree

| observation | meaning |
|---|---|
| nothing, no blink | payload not handed off (depthcharge-side) |
| blink train once, then magenta persists | U-Boot crashed before `board_early_init_f` (PIE fixup / early board_init_f / initf_dm) |
| blink train repeating periodically | reset loop (WDT) — identify the phase by the colors that flicker |
| magenta + orange/blue/red/yellow bands | crash in the next window after the last band |
| banner on black | success |

## Round 16 — bisecting the R14/R15 handoff failure

R15 (`b1c5c380…`): nothing at all — no magenta, no blink train. Since
the wrapper + header bytes at 0x40 are functionally identical across
R13 (ran) and R14/R15 (did not run), the failure is depthcharge-side.
Only depthcharge-visible structural change between R13 and R14: the
mkdepthcharge body grew from 0x80000 to 0x81000 when the 4 KiB
alignment padding landed (R15: 0x82000). Verified the device-era
depthcharge (R93, `0.0.22-10566-g8417b00957`) sources are not locally
available (clone is mainline-only; googlesource fetch failed); mainline
KERNEL_SIZE is 0x20000000, so a 512 KiB constraint has no known
mechanism yet — but the correlation is 2-for-2.

### Discriminating flash (payload `d418252a…`)

R13 layout exactly (wrapper v1 = green fill, no blinks, U-Boot @0xB4)
but with R14's body size (0x81000, via the larger R15 U-Boot):

| observation | conclusion |
|---|---|
| green (persistent) | body size innocent; the 0x1000 offset somehow guilty — re-bisect |
| black | FIT body > 0x80000 breaks this depthcharge — fix = keep FIT under 512 KiB (minimal FDT subimage) with U-Boot @0x1000 |

## Round 17 — R16 green decodes body-size theory; total-size bisect

R16 (R13 layout, body 0x81000): **green** — body size ruled out (R16
and R14 share body 0x81000). Updated correlation: total wrapped-image
size / image_size field: ran ≤ 449772, dead ≥ 453432. Device-era
sources recovered far enough to rule out boot64/fit_decompress size
checks (kukui r93/12573-B code identical modulo cosmetics; bounds
0x5f800000/0x60000000/0x80000000 — kend≈0x4006xxxx passes; the loader
is DETERMINISTIC: first CB_MEM_RAM range, kstart=0x40000000 — the
"random 2 MiB slot" note from Phase 1 was mainline behavior, [REVERSED]
for the device).

### Test A (payload a27f0354…, flashed, cmp+vbutil OK)

R16's exact known-good content (wrapper v1 green, U-Boot @0xB4,
contiguous — wfi hang expected) + 8 KiB zero tail padding → total
457964, image_size=457964, body 0x83000. Isolates TOTAL SIZE from the
0x1000-offset/padding variable:

| observation | conclusion |
|---|---|
| green | total size innocent → the 0x1000 alignment padding/placement is the killer (re-test padding content) |
| black | total size (image_size value) alone gates the handoff |

## Round 18 — total size ruled out; wrapper-only isolator

R17 (R16 content + 8 KiB tail pad, total 457964 > dead 453432):
**green** — total size ruled out. The only remaining structural
difference between ran (R12/13/16/17: U-Boot contiguous at 0xB4) and
dead (R14/15: U-Boot at 0x1000 behind interior zero pad): the interior
pad + shifted placement. Also discovered: the U-Boot binary does NOT
start at _start — `arch/arm/cpu/u-boot.lds` places 0x40 bytes of
`.vectors` first, so link _start = 0x4C000140 (all prior alignment math
shifts by 0x40; the PIE requirement is runtime _start ≡ link _start
mod 4K AND start.S's wfi check demands runtime _start ≡ 0 mod 4K, so
link _start must be 4K-aligned too).

### Test B (payload 9485c59e…, flashed, cmp+vbutil OK, body 0x14000)

4 KiB image: header + wrapper v3 (magenta + 3 blinks + ~5 s hold) +
zeros to 0x1000 + NO U-Boot, tail branch patched to self (hang):

| observation | conclusion |
|---|---|
| magenta + blinks + hang | interior pad innocent; the trigger is U-Boot content/placement at 0x1000 |
| black, nothing | depthcharge rejects the padded image before any execution |

## Round 19 — Test B decoded: wrapper runs in padded layout; correct PIE alignment + post-clear checkpoint

Test B (`9485c59e…`): **wrapper color + 3 blinks + color held** — the
wrapper runs fine even in the padded/0x1000 layout, with NO U-Boot
present. Depthcharge is fully exonerated: it copies and jumps every
payload. Combined with R14/R15 (black), the story closes: U-Boot (at
0x1000, aligned) RAN in R14/R15, reached the video probe and its
`video_clear()` (black over the wrapper color, backlight back on), and
died somewhere between the clear and a visible banner. The user's
"no backlight" was a backlit-black misread (matches R5's physics).
(The R15 blink train was evidently too brief to register.)

### Alignment math corrected ([REVERSED] earlier notes)

The U-Boot file does NOT start at _start: a 4-byte stub precedes it
(nm: link _start = CONFIG_TEXT_BASE + 4). Two constraints must hold
simultaneously:
1. start.S wfi check: runtime _start ≡ 0 (mod 4K);
2. PIE fixup (adrp+add lo12): runtime _start ≡ link _start (mod 4K).
⇒ link _start must be 4K-aligned: CONFIG_TEXT_BASE=0x4C000FFC → link
_start=0x4C001000; U-Boot file placed at image offset 0xEC0 → runtime
_start=0x40001000. Verified: nm shows 0x4c001000; the payload script
now asserts it against u-boot.sym. (Earlier R14/R15 layouts had runtime
_start page offset 0x40 → wfi-hang — but green should have persisted;
the observed black is only explained by the video_clear path, which
means R14's U-Boot PASSED the wfi check... [OPEN] the R14/R15 images
had link _start=0x4C000140 (page offset 0x140) and runtime 0x1040
(0x40) — mismatched mod 4K by 0x100, so even past the wfi check the PIE
fixup would corrupt pointers; either way the video_clear observation
stands as the only black-producing mechanism.)

### Round 19 payload (`1f598c7a…`, flashed, cmp+vbutil OK)

- CONFIG_TEXT_BASE=0x4C000FFC, U-Boot file at image 0xEC0 (correct
  alignment for both constraints);
- new checkpoint: WHITE band painted at the END of video_post_probe
  (after the vidconsole child is bound+probed) — post-clear deaths are
  now distinguishable from banner-stage deaths;
- all earlier checkpoints retained (orange/blue/red bands, misc_init_r
  yellow).

### Decision tree

| observation | meaning |
|---|---|
| magenta + blinks + magenta persists | U-Boot died BEFORE the video probe (bands show how far) |
| black + white band | video probe completed; died between post_probe and banner |
| black, no white band | died inside video_post_probe after video_clear (bind/probe of vidconsole) |
| banner (white on black) | works |

## Round 20 — post-probe black decoded: U-Boot reaches video_clear; two-band instrumentation

Round 19 (`1f598c7a…`): wrapper color + blinks, then **black, no white
band**. Decoded: U-Boot runs (correct alignment), reaches the video
uclass `video_clear()` (the only black-painter — that's the blue→black
transition), and dies/ends before the white band at the end of
video_post_probe. The window contains only: colours setup, vidconsole
bind ("vidconsole0"), device_probe of the console child, cyclic
registration. A GRACEFUL failure there (bind/probe error → video probe
fails) leaves exactly the observed state: cleared black, no banner
(goes nowhere — no console), backlight on. Also noted: CONFIG_CYCLIC=y
makes video-uclass line 732's `gd->flags && GD_FLG_RELOC` typo live
(harmless-looking cyclic_register every 10 ms).

### Round 20 payload (`90f0de99…`, flashed, cmp+vbutil OK)

Two-band instrumentation inside video_post_probe:
- **cyan** band right after `video_clear()` — visible ⇒ the clear and
  the fb mapping work, death is later in this function or after;
- **white** band after the vidconsole child is bound+probed (existing).

### Decision tree

| observation | meaning |
|---|---|
| black only (no cyan) | died between clear and cyan — video_set_default_colors/... (near-impossible) |
| cyan only | death/graceful-failure in bind or probe of vidconsole0 |
| cyan + white | died between post_probe end and the banner (jumptable/console_init_r/puts) |
| banner | works |

## Round 21 — [REVERSED R19/R20] root cause: build script patched delay's `ret`, not the tail branch

User report R20 ("blue fill, one blink, black, backlight off, ~1 s,
NO 5 s hold") exposed a build bug that invalidates the R19/R20
interpretation — U-Boot never executed at all in R19/R20:

- The wrapper's tail `b .` is NOT the last instruction: v3 puts the
  `delay` subroutine (8 insns incl. `ret`) after it. The script patched
  `wrapper[-4:]` = delay's `ret`, overwriting it with the branch.
- Effect: the FIRST blink's `bl delay` never returns — it branches to
  the U-Boot file's first word (0x00000000 = UDF) with the backlight in
  the blink-OFF phase → screen black, backlight off, hang. Blue fill +
  one dark blink = exactly the R19/R20 observations. The "reached
  video_clear" decode of R19/R20 is [REVERSED]; the cyan/white bands
  were never reached (kept in the tree — still useful).
- Test B (R18) was immune because its wrapper-only image left `b .` as
  the self-branch (patching `ret` was harmless there). R12–R17 ran
  because wrapper v1/v2 had the tail branch as the last instruction.

### Fixes (payload `cff0e0e9…`, flashed, cmp+vbutil OK)

1. Script now SEARCHES the wrapper for the single `b .` (0x14000000)
   and asserts exactly one occurrence; patch target = the real branch.
2. Placement recomputed from the ELF: `_start` is NOT at file offset 0
   — the image leads with 0xFFC bytes (vectors/stub) so link
   _start = 0x4C001000 = file offset 0x1000 (TEXT_BASE 0x4C000FFC).
   U-Boot file placed at image 0x1000 → runtime _start = 0x40002000
   (4K-aligned); PIE delta = -0xBFFFF000 (4K-aligned). Both alignment
   constraints hold for the first time in one layout.
3. Verified in the packed image: `b .`@0x118 → 0x140007ba → target
   image 0x2000 = runtime 0x40002000 ✓; `ret`@0x138 intact ✓.

### Expected behavior

Blue fill → 3 blinks (~1.5 s) → 5 s blue hold → U-Boot takes over.
If U-Boot boots: banner (white on black). If it dies early, the
orange/blue/red board-init bands and the post-clear cyan / post-probe
white bands localize it (Round 19/20 instrumentation still in place).

## Round 22 — wrapper fully healthy; U-Boot dies before board_init_f's first band

Round 21 (`cff0e0e9…`): **blue, 3 blinks, blue held forever** — the
wrapper now runs its complete sequence (fill, blink train, 5 s hold)
and jumps to 0x40002000. Blue persisting = nothing in U-Boot painted
anything: death (or non-execution) BEFORE board_early_init_f's orange
band. start.S review: the 4K wfi check passes (runtime _start
0x40002000); the PIE fixup applies the full delta to relative relocs
and our delta (-0xBFFFF000) is 4K-aligned, so adrp+lo12 stays valid;
crt0's SP (INIT_SP_RELATIVE, adrp __bss_start + 0x80000) is also
4K-delta-safe. So the hang is somewhere in [PIE fixup .. initcalls
before board_early_init_f] — a path no payload has ever reached on
this board (R12–R17 all died at the 4K wfi check).

### Round 22 payload (`21c3f792…`, flashed, cmp+vbutil OK)

New checkpoint: **green band (y=400)** painted at the very top of
board_init_f (common/board_f.c, temporary) — before the entire
initcall sequence.

### Decision tree

| observation | death localized to |
|---|---|
| blue hold only (no green) | start.S / crt0 / PIE fixup (pure asm window) |
| green only | initcalls before early_init_f: fdtdec_setup (embedded DTB!), log, bootstage, bloblist, arch/mach_cpu_init, initf_dm (clock driver probes) |
| green + orange | after early_init_f, before dram_init_banksize |
| green + orange + blue | after dram_init_banksize, before relocation |

## Round 23 — [REVERSED R22 analysis] true root cause: 4-byte linker fill before _start skews the PIE fixup

R22 (`21c3f792…`): blue hold only — board_init_f's green band never
painted. Root cause found in the disassembly, not guessed:

- `start.o`'s input `.text` section is 8-byte aligned, but
  CONFIG_TEXT_BASE=0x4C000FFC is only 4-aligned → ld inserts a 4-byte
  FILL at the start of .text → `_start` lands at TEXT_BASE+4, while
  `_TEXT_BASE` (the word start.S's PIE fixup loads as the link base)
  and `__image_copy_start` remain at TEXT_BASE.
- start.S: `pie_fixup: adr x0,_start; ldr x1,_TEXT_BASE; subs x9,x0,x1`
  → x9 = true_delta + 4 → EVERY relative relocation skewed by 4 →
  corrupted gd/function/fdt pointers → death before board_init_f.
  (The `adrp + #:lo12:` pairs used for the rela bounds also assume
  delta ≡ 0 mod 4K — with the skew, both mechanisms break.)
- This retroactively explains the R12–R22 alignment confusion: the
  "+4" was never a stub/vectors mystery — it was a linker fill that
  appears iff CONFIG_TEXT_BASE is not 8-aligned.
- [REVERSED] the "vectors/stub bytes precede _start" note from R18/19;
  the 0xFFC leading bytes in old u-boot.bin dumps were the fill plus
  ELF file-offset artifacts, and the R22 "dies in pure asm" conclusion
  was wrong in mechanism (it died IN the fixup's corrupted pointers).

### Fix (payload `18cf3e93…`, flashed, cmp+vbutil OK)

- CONFIG_TEXT_BASE=0x4C001000 (8-aligned) → no fill:
  `_start == __image_copy_start == _TEXT_BASE == 0x4C001000`, file
  offset 0 = `b reset` (verified in u-boot-nodtb.bin and in the packed
  image at 0x1000).
- File placed at image offset 0x1000 → runtime _start = 0x40001000,
  start.S 4K check passes; PIE delta = -0xC000000 (4K-aligned).
- Payload script now derives the file offset from __image_copy_start
  (not ELF section offsets) and ASSERTS `_start == __image_copy_start`
  so a fill regression fails at build time instead of on device.

### Expected

blue → 3 blinks → 5 s hold → green band (board_init_f) → orange/blue/
red bands → cyan/white (video probe) → banner.

## Round 24 — C environment alive; video probe is the failing stage

Round 23 (`18cf3e93…`): blue → blinks → hold → bands. The user
reported (top to bottom): two red-ish stripes, yellow, green — i.e.
board_init_f (green), board_early_init_f (orange, reads red-ish),
board_early_init_r (red), misc_init_r (yellow) ALL ran; dram_init
blue is at the very bottom (y=1820, easy to miss). CRITICAL: the
screen was NOT cleared and no cyan/white band appeared → the video
uclass post_probe (whose FIRST action after colour setup is
video_clear) never ran → the video driver's probe FAILED or the
device never bound. Everything else (banner needs the vidconsole)
follows from that. Note: the "yellow" seen may be either misc_init_r
(y=300) or the post-clear cyan band (0x00ffff00 = yellow in xRGB,
y=260) — irrelevant since no clear happened.

### Round 24 payload (`03fc7b4d…`, flashed, cmp+vbutil OK)

New checkpoints (mt8183_scanout.c + video-uclass.c video_post_bind):
- **purple** (y=470): a video device got BOUND (driver matched DT)
- **magenta** (y=500): driver probe entered
- **gray** (y=620): geometry resolved (coreboot LBIO parsed + scanout
  addr valid)
- **red bands** at failure exits: y=780 (ovl reg read failed), y=800
  (coreboot LBIO not found), y=820 (pixel format mismatch), y=840 (no
  plausible scanout address)

### Decision tree

| observation | meaning |
|---|---|
| no purple | video driver never bound (DT compatible / DM scan issue) |
| purple, no magenta | bound but probe not called (uclass sequencing) |
| magenta + red@800 | coreboot table/LBIO not found (map/parse) |
| magenta + red@820 | LBIO found but format mismatch |
| magenta + red@840 | no plausible scanout address |
| magenta + gray | geometry OK; failure later (clear/probe tail) |
| cyan/white appear | video_clear + post_probe reached — back on the R19/20 tree |

## Round 25 — [REVERSED] not a probe failure: the video device was never probed (console env wiring)

Round 24 (`03fc7b4d…`): **purple, no magenta, no red error bands** —
the driver matched the DT node and bound, but device_probe was never
called, and none of the probe's error paths ran. Root cause is in
common code, not the driver:

- stdio_add_devices()'s probe-all loop over UCLASS_VIDEO is compiled
  OUT when CONFIG_SYS_CONSOLE_IS_IN_ENV=y — and that defaults to y via
  CONSOLE_MUX (inherited from the pumpkin defconfig fork).
- With SYS_CONSOLE_IS_IN_ENV, console_init_r() reads env stdout/stderr
  and muxes via iomux_doenv(). There was NO environment (and the env
  backend was MMC — inherited, wrong for a depthcharge payload), so
  the names were NULL, console_search_dev() never saw the string
  "vidconsole", the lazy stdio_probe_device(name, UCLASS_VIDEO) never
  fired, and the console fell back to serial only. The video device
  was never probed → no clear, no cyan/white, no banner. Screen keeps
  the init bands.

### Fix (payload `e4bdd875…`, flashed, cmp+vbutil OK)

- CONFIG_ENV_IS_NOWHERE=y (drops the MMC env dependency inherited
  from pumpkin — correct for a payload stage, deterministic default
  env);
- new CONFIG_ENV_SOURCE_FILE="krane" +
  board/mediatek/mt8183/krane.env:
    stdin=serial
    stdout=serial,vidconsole
    stderr=serial,vidconsole
  so console_init_r() → iomux_doenv() → console_search_dev
  ("vidconsole") → stdio_probe_device → probes the video device
  (driver probe → clear → vidconsole registered) → banner muxed to
  serial + screen.
- R24 instrumentation kept for this round.

### Expected

blue → blinks → hold → green/orange/blue/red/yellow bands → magenta
(probe entered) → screen CLEARS black → gray (geometry) → cyan/white
(video clear/probe) → **white-on-black banner** (or a red failure
band: y=800 lbio, y=820 fmt, y=840 scanout).

## Round 26 — video probe SUCCEEDED (clear ran); post-console hang reset via WDT; WDT disabled for observability

Round 25 (`e4bdd875…`): bands → **screen cleared black** → reset back
to the depthcharge menu. Decode:
- The clear proves console env wiring now works: krane.env's
  stdout=serial,vidconsole → console_init_r → iomux_doenv →
  console_search_dev("vidconsole") → stdio_probe_device → video
  driver probed (magenta/gray painted, then erased by the full-screen
  video_clear) → post-probe cyan/white should have painted too (thin,
  y=200/260 — possibly missed in the crash window).
- The reset is the MTK watchdog: CONFIG_WDT_MTK autostarts a 60 s
  watchdog at initr_watchdog (early board_r); it is serviced by the
  cyclic scheduler only once main_loop() runs. A hang anywhere between
  the video probe (still in board_r console init) and the first
  schedule() → 60 s → WDT reset → depthcharge menu. User's
  "immediately" = unattended reset.

### Round 26 payload (`08ee5c1f…`, flashed, cmp+vbutil OK)

- **# CONFIG_WATCHDOG_AUTOSTART is not set** — the WDT is probed but
  not started, so a hang now freezes the screen with bands visible
  instead of resetting. (Temporary bring-up measure; upstream will
  want the WDT on.)
- New checkpoint: **light-gray band (y=900)** at main_loop entry
  (common/main.c) — everything before it completed.

### Decision tree

| observation | meaning |
|---|---|
| black screen stays black, no reset, no gray | hang between video probe and main_loop (console tail / late board_r / initr_mmc...) |
| black + gray band, frozen | main_loop reached, hang in preboot/cli_init/cli_loop — inspect further |
| banner + prompt (white on black) | WORKING |

## Round 27 — pink band in board_late_init; WDT-off reset anomaly needs timing

Round 26 (`08ee5c1f…`): user reports "same as before": bands → black
(clear) → menu. Ambiguity: the report may describe the R25 payload
(WDT autostart ON, 60 s → reset explains it) or the R26 payload (WDT
autostart OFF — then the reset source is NOT U-Boot's watchdog:
coreboot's mtk_wdt_init DISABLES the WDT, depthcharge doesn't arm it,
so a reset would need another mechanism). The black→menu GAP will
discriminate: ~55-65 s ⇒ watchdog; a few seconds ⇒ something else.

The missing light-gray band (main_loop entry, y=900) keeps the death
window at [misc_init_r (yellow, AFTER console_init_r!) .. main_loop].
Refined understanding of the observed order: the video probe runs
INSIDE console_init_r (lazy stdio_probe_device via iomux), so the
screen clears BEFORE misc_init_r's yellow paints — yellow on black is
expected, and the pre-clear bands (green/orange/red + probe's
magenta/gray/cyan/white) are erased by the clear. Board_r tail:
console_init_r → console_announce_r (first vidconsole TEXT!) →
arch/misc_init_r → interrupt/timer init → board_late_init →
EVT_LAST_STAGE_INIT → initr_mem → run_main_loop. No text has ever
been reported — the first vidconsole puts (console_announce banner or
"In:/Out:/Err:") may itself be the crash point.

### Round 27 payload (`948d5e2b…`, flashed, cmp+vbutil OK)

- New **pink band (y=860)** in board_late_init (CONFIG_BOARD_LATE_INIT
  enabled). Sequence of late checkpoints: yellow (misc_init_r) → pink
  (board_late_init) → light-gray (main_loop).
- WDT still off (autostart=n).

## Round 28 — reset mechanism identified: sync abort → panic → PSCI reset; PANIC_HANG to capture the dump

Round 26/27 analysis (user: reset in <1 s after the black screen,
WDT autostart already off). The reset mechanism is now proven from
source, not guessed:

- coreboot's mtk_wdt_init() DISABLES the MTK WDT (clears WDT_ENABLE);
  depthcharge never touches it; U-Boot's mtk_wdt_probe() even stops
  it. Hardware watchdog ruled out.
- arch/arm/mach-mediatek/mt8183/init.c: reset_cpu() →
  psci_system_reset() (SMC into BL31) — instant warm reset.
- lib/panic.c: CONFIG_PANIC_HANG unset → panic() → do_reset() →
  reset_cpu() → PSCI.
- arch/arm/lib/interrupts_64.c: do_sync() → show_regs →
  panic("Resetting CPU ...").

⇒ ANY synchronous abort after the video probe = instant reset to the
depthcharge menu, <1 s. Matches the observation exactly. (Also
retroactively plausible for earlier "resets".)

### Round 28 payload (`d1e78b0f…`, flashed, cmp+vbutil OK)

CONFIG_PANIC_HANG=y — on an abort, U-Boot now prints the full
exception dump ("Synchronous Abort" handler, esr, FAR, registers, PC)
through the vidconsole and HANGS with the text visible instead of
resetting. Plus R27's pink (board_late_init, y=860) and R26's
light-gray (main_loop, y=900) bands.

### Ask to the user

Reboot, select Internal storage. Expect: bands → black → then either
frozen black with WHITE TEXT (transcribe it! esr/FAR/ELR tell us the
exact fault), or a hang at some band stage. Report the band stack
(yellow/pink/gray) and any text.

## Round 29 — crash localized inside the driver probe; step-by-step bands

Round 28 (`d1e78b0f…`, PANIC_HANG): **purple (bind) + magenta (probe
entered), frozen; no gray, no pink, no text.** Decode:
- The crash is inside mt8183_scanout_probe, between entry (magenta,
  y=500) and geometry resolution (gray, y=620) — and NOT in any of the
  handled error paths (no red bands).
- Why no abort text despite PANIC_HANG: at this point the vidconsole
  doesn't exist yet, so printf output goes to the pre-console buffer
  and the dump is invisible. Expected — not a contradiction.
- Note the magenta band itself proves post-reloc mmu_map_region + MMIO
  read of 0x14008f40 + fb write work (dcache on). Remaining suspects
  between magenta and gray: revival MMIO writes (same 4K page as the
  working read — unlikely), mmu_map_region(0xffed9000, 4K) (first
  non-fb, non-DRAM mapping post-reloc — prime suspect),
  find_framebuffer() reads of the coreboot table, format check.

### Round 29 payload (`35376f63…`, flashed, cmp+vbutil OK)

Sub-checkpoints inside the probe (all 20-row bands below magenta):
- **mint** y=520: revival writes done
- **violet** y=540: coreboot table mapped
- **pale yellow** y=560: LBIO found
- **pale cyan** y=580: format OK
- red bands on the handled failures unchanged (y=780/800/820/840)

### Ask

Reboot and report which of mint/violet/pale-yellow/pale-cyan appear
below the magenta band. The first missing one identifies the aborting
statement.

## Round 30 — real text logging: fb font logger + abort dump on panel

Round 29 result: probe entered (magenta), then frozen — but the user's
color report ("light green") is ambiguous between mint (y=520) and
pale yellow (y=560), and band combinatorics are getting unreadable.
User proposed real logs — agreed; implemented:

- mt8183_scanout.c now has krane_fb_log() / krane_fb_log_hex(): a
  minimal framebuffer text logger using the built-in 8x16 font
  (video_font.h), drawing white text into the live scanout (bottom
  rows, y>=1700), with a static cursor and per-call dcache flush.
- Probe steps now emit text: PROBE → REVIVE → MAPTBL → LBIO → FMT →
  GEO (bands kept for redundancy).
- krane_fb_log_abort(esr, elr): reads FAR (CurrentEL-aware mrs) and
  draws "ABORT! ESR=0x… ELR=0x… FAR=0x…" on the panel; wired into
  arch/arm/lib/interrupts_64.c do_sync() (TEMPORARY). With
  CONFIG_PANIC_HANG=y the screen freezes with the dump visible —
  even though the pre-console printf buffer stays invisible.

### Round 30 payload (`60f5dba8…`, flashed, cmp+vbutil OK)

### Ask

Reboot and report: (a) the text line(s) at the bottom (PROBE REVIVE
MAPTBL LBIO FMT GEO — where does the sequence stop?), and (b) if an
"ABORT! ESR=0x… ELR=0x… FAR=0x…" appears, transcribe the three hex
values.

## Round 31 — [ROOT CAUSE] page-table pool exhausted: mmu_map_region(coreboot table) → panic

Round 30 result: "PROBE REVIVE" only — the hang is exactly at
mmu_map_region(COREBOOT_TABLE_ADDR=0xffed9000, SZ_4K). No ABORT! text
drawn ⇒ not a fault ⇒ a HANG. The only hang inside mmu_map_region:
create_table() → panic("Insufficient RAM for page table") → hang().

Why: get_page_table_size() budgets only for the STATIC memory map
(count_ranges × 2 + 4 spare tables). Post-reloc dynamic mappings
(the fb 0xFD536000/0x8ca000 → several L3 tables, then the 4K table
region at 0xffed9000 → yet another L3) exhausted the pool. This
retroactively explains the R25/R26 "<1 s reset to menu": with
PANIC_HANG off, that same panic() → do_reset() → reset_cpu() →
psci_system_reset() — the reset the user saw. Root cause found for
the whole "dead/black/reset" saga since Round 12.

### Fix (payload `34b9660f…`, flashed, cmp+vbutil OK)

mt8183.c: board override of the __weak get_page_table_size() →
0x40000 (64 tables), sized for the driver's dynamic mappings. This
override is the intended extension mechanism and is upstreamable.

### Expected

PROBE REVIVE MAPTBL LBIO FMT GEO → screen clears → banner (white on
black) + U-Boot prompt. If anything still fails, the ABORT!/panic
text now lands on the panel.

## Round 32 — MILESTONE 3 ACHIEVED: banner + prompt live on the panel

Round 31 (`34b9660f…`) works: the user reports the U-Boot banner and
the prompt rendered white-on-black on the krane panel, via the
standard video uclass + vidconsole on the depthcharge scanout. The
full boot chain to a working framebuffer console is:

depthcharge → payload (arm64 Image header + wrapper v3 @0x40 +
U-Boot @0x1000, runtime _start 0x40001000) → wrapper (revive display,
3 blinks, 5 s hold) → U-Boot start.S (4K check ✓, PIE fixup, delta
-0xC000000) → board_init_f → relocation → board_r → console_init_r →
krane.env stdout=serial,vidconsole → iomux → stdio_probe_device →
mt8183_scanout probe (revive OVL + backlight, LBIO parse, fallback
OVL_L0_ADDR=0xFD536000) → video_clear → vidconsole0 → banner.

Key fixes that made it work (final state):
1. TEXT_BASE=0x4C001000 (8-aligned ⇒ no linker fill ⇒ _start ==
   __image_copy_start == _TEXT_BASE; PIE delta 4K-aligned).
2. Payload layout: U-Boot file at image offset 0x1000 ⇒ runtime
   _start 0x40001000 (both 4K constraints).
3. ENV_IS_NOWHERE + krane.env: stdout=serial,vidconsole (the video
   device only probes via the iomux lazy path).
4. get_page_table_size() board override (0x40000) — dynamic
   mmu_map_region calls (fb + coreboot table) exhausted the default
   budget ⇒ panic (reset without PANIC_HANG, hang with it).

Still WIP/temporary in the u-boot tree (to revert before upstreaming):
- all diag band/text instrumentation (mt8183_scanout.c krane_*,
  video-uclass.c, board_f.c, main.c, interrupts_64.c hook),
- CONFIG_PANIC_HANG=y, CONFIG_BOARD_LATE_INIT=y,
- CONFIG_WATCHDOG_AUTOSTART=n (decide: re-enable for production).

Next: milestone 4 — distro_bootcmd booting pmOS from USB (sda), then
cleanup/upstreaming pass (single-purpose commits, MAINTAINERS,
checkpatch, revert diagnostics).

## Round 33 — USB stack enabled (xHCI + T-PHY + keyboard); payload `35345ec7…`

Milestone 4 step 1: make the pogo keyboard (USB HID behind the GL610
hub on `usb_host`) work at the prompt. Krane defconfig had been forked
from pumpkin minus all USB/PHY configs.

### Source survey (all mainline drivers, no new driver needed)

- U-Boot `drivers/usb/host/xhci-mtk.c` binds `mediatek,mtk-xhci`
  (matches the `usb@11200000` child of `ssusb@11201000`) and expects
  reg-names `mac` + `ippc` and a `phys` list ON THE XHCI NODE. The
  upstream mt8183.dtsi models the block for the Linux mtu3 driver:
  `ippc` (0x11203e00) and the `phys` live on the PARENT mtu3 node,
  the xhci child has only `mac`. U-Boot has no mtu3 driver.
  → new `arch/arm/dts/mt8183-kukui-krane-sku176-u-boot.dtsi`
  (auto-included at the END of the upstream dts by
  scripts/Makefile.lib `dtsi_include_list`, so `&label` refs work)
  adds `ippc` to `reg`/`reg-names` and
  `phys = <&u2port0 PHY_TYPE_USB2>, <&u3port0 PHY_TYPE_USB3>` to
  `usb_host`. Not a DT fork; upstream file untouched. Verified in the
  compiled dt.dtb.
- `drivers/phy/phy-mtk-tphy.c` binds `mediatek,generic-tphy-v2`
  (mt8183 u3phy node is already status-okay with both ports).
- Regulators: kukui DT gives `vusb33-supply = <&mt6358_vusb_reg>`, but
  U-Boot has no MT6358 PMIC/regulator driver. Safe: on missing
  regulator, `device_get_supply_regulator` fails (debug only) and
  `regulator_set_enable(NULL)` returns -ENOSYS, which
  `xhci_mtk_ldos_enable` tolerates (vusb33 is a 3.3 V LDO the firmware
  already keeps on; no VBUS regulator exists in the DT for usb_host).
  No PMIC work needed.
- MMU: MTK armv8-mem-map pre-maps 0x0–0x20000000 as device; xhci
  (0x11200000), ippc (0x11203e00) and phy (0x11f40000) need NO dynamic
  mappings — the R31 get_page_table_size headroom is untouched.
- Enumeration→keyboard path: `U_BOOT_USB_DEVICE(usb_kbd,
  kbd_id_table)` in common/usb_kbd.c matches HID boot-protocol
  keyboards during `usb_scan_device`; the bound UCLASS_KEYBOARD probe
  registers stdio "usbkbd" and re-runs `iomux_doenv(stdin)` when the
  env `stdin` contains the name.
- Timing: no `initr_usb` exists in board_r.c anymore — USB is only
  enumerated on `usb start`. CONFIG_PREBOOT has a built-in default of
  "usb start" when USB_KEYBOARD=y (boot/Kconfig), so
  `CONFIG_USE_PREBOOT=y` suffices: preboot runs at main_loop entry
  (after console_init_r, so probe errors print to the panel/vidconsole;
  keyboard not yet in the stdin mux at console_init_r is fine —
  iomux_err falls back to serial, then usb_kbd probe re-muxes stdin).
  SYS_CONSOLE_ENV_OVERWRITE defaults n, so krane.env's
  `stdin=serial,usbkbd` survives until the re-mux.

### Changes (u-boot `krane` branch)

- configs/mt8183_kukui_krane_defconfig: USE_PREBOOT, CMD_USB, USB,
  DM_USB, USB_XHCI_HCD, USB_XHCI_MTK, USB_KEYBOARD, USB_STORAGE (for
  the sda distro boot), PHY, PHY_MTK_TPHY. Defconfig re-run done.
- arch/arm/dts/mt8183-kukui-krane-sku176-u-boot.dtsi: new overlay
  (ippc + phys onto usb_host).
- board/mediatek/mt8183/krane.env: stdin=serial,usbkbd.
- Build OK; `_start` still 0x4C001000 == __image_copy_start; dtb
  verified (mac+ippc+phys on usb@11200000). Payload `35345ec7…`
  flashed, cmp OK, vbutil_kernel body verification succeeded.

### Decision tree for the reboot

| observation | meaning |
|---|---|
| banner → `usb start` scanning lines → prompt; typing works | WORKING |
| banner → "Scanning bus … failed, error N" or xhci/phy error text, then prompt | enumeration failed at stage N (transcribe the error line) |
| banner → prompt, no usb lines at all | preboot never ran |
| reset to menu <1 s after banner | sync abort in USB probe (PANIC_HANG should freeze instead — check it is still set) |
| prompt but keys do nothing | keyboard not enumerated (pogo not seated?) or stdin mux — re-seat keyboard and power-cycle |

Awaiting user reboot report.

## Round 34 — [ROOT CAUSE] mtu3 node never bound; "No USB controllers found"

Round 33 report: "no usb controller detected" — matches
usb-uclass.c:467 `No USB controllers found`, printed when
`controllers_initialized == 0`, i.e. NO UCLASS_USB bus device was even
probed (a probed-but-failed controller would have printed
"Bus usb@11200000: probe failed, error N"). So the xHCI node was never
BOUND. Not a driver/pin problem: xhci-mtk.o was linked, the dtb had
mac+ippc+phys, and pinctrl is irrelevant to the controller.

### Root cause (found in source, drivers/core/root.c)

`dm_scan_fdt_node()` walks ONLY the immediate subnodes of a bound node.
Recursion into a node's children happens solely when some driver binds
that node (bus drivers recurse via dm_scan_fdt_dev, e.g. the generic
simple-bus driver's uclass post_bind, drivers/core/simple-bus.c).
`usb@11200000` is a child of `ssusb@11201000` ("mediatek,mtu3"), which
has NO U-Boot driver ⇒ its subtree was never scanned ⇒ usb_host never
bound ⇒ zero UCLASS_USB devices. MMC/video bind because they are
direct children of /soc (simple-bus).

### Fix (u-boot `75fe5bc9`, payload `9c55f520…`)

The R33 overlay now also sets
`compatible = "mediatek,mt8183-mtu3", "mediatek,mtu3", "simple-mfd"`
on ssusb: the generic simple-bus driver binds the node (CONFIG_SIMPLE_BUS=y)
and its post_bind scans the xhci child. No MMIO touched by that driver
(empty `ranges` → plat read skipped harmlessly).

Build OK, `_start` still 0x4C001000, dtb verified (simple-mfd present).
Flashed `9c55f520…`, cmp OK, vbutil body verification succeeded.

### Decision tree for the reboot

| observation | meaning |
|---|---|
| banner → `Bus usb@11200000: ...` lines → `N USB Device(s) found` → prompt; typing works | WORKING |
| banner → `Bus usb@11200000: probe failed, error -N` | xhci/phy/clk probe failed at stage N — transcribe |
| banner → `Scanning bus ... failed, error N` | root-hub enumeration failed — transcribe |
| `Bus usb@11200000: No USB Device found` then prompt | controller OK, nothing on the bus — re-seat the keyboard on the pogo pins, power-cycle |
| `No USB controllers found` again | binding still failing — next step: krane_fb_log in lists_bind_fdt path |

Awaiting user reboot report.

## Round 35 — Round 34 VERIFIED (keyboard works); console rotated to landscape

User confirmed: USB scans properly and the pogo keyboard works at the
prompt — Milestone 4 step 1 done (payload `9c55f520…` verified live).
Milestone 3+4a: banner + prompt + keyboard input on the panel.

### Rotation request: portrait console → landscape

Panel is the BOE TV101WUM-NL6, natively 1200×1920 portrait. Upstream
DT marks the panel node `rotation = <270>` (mounted rotated in the
chassis); pmOS/ChromeOS force landscape with the keyboard attached.

Mainline already supports this without touching the scanout path:
`CONFIG_CONSOLE_ROTATION` builds the rotated text drivers
vidconsole1/2/3 (drivers/video/console_rotate.c); when the video
driver sets `priv->rot`, video-uclass binds `vidconsole<rot>` and the
rotated console swaps geometry itself (console_core.c: rot%2 ⇒
cols = ysize/8 = 240, rows = xsize/16 = 75). The first vidconsole's
stdio name is still "vidconsole", so krane.env needs no change.

### Change (u-boot `69787732`, payload `4fcd863d…`)

- mt8183_scanout probe: `uc_priv->rot = 1` (90° clockwise) with a
  comment explaining the 270° panel mount. Guess between rot=1 and
  rot=3 (both landscape-upright; they differ by which side the
  keyboard hinge ends up on): first try rot=1, flip to 3 if the user
  reports upside-down text.
- defconfig: CONFIG_CONSOLE_ROTATION=y (CONSOLE_NORMAL stays; the
  uclass picks the driver from rot).
- fb geometry, LBIO parse, revive path untouched. Build OK, `_start`
  still 0x4C001000. Flashed `4fcd863d…`, cmp OK, vbutil body
  verification succeeded.

### Decision tree for the reboot

| observation | meaning |
|---|---|
| banner + prompt readable in landscape with keyboard; keys work | DONE |
| landscape but text upside down | switch rot=1 → rot=3 in mt8183_scanout.c |
| still portrait | vidconsole1 not bound — check CONSOLE_ROTATION made it into the build |

Awaiting user reboot report.

## Round 36 — [ROOT CAUSE] R35 defconfig edit dropped CONFIG_USB; rot 1→3

Round 35 report: landscape text upside down; USB/keyboard completely
dead — not even "No USB controllers found"; banner position unchanged.

### USB regression: my R35 defconfig edit overwrote CONFIG_USB=y

The R35 `PUT 77.=77` used a line number from a view where lines 77-80
were elided, assuming 77 was CONFIG_USB=y. It actually was (R33 had
placed the USB block at 77-82), so the edit replaced CONFIG_USB=y with
a duplicate CONFIG_VIDEO/CONFIG_VIDEO_MT8183_SCANOUT pair. Result:
`menuconfig USB` unset after the defconfig re-run → no UCLASS_USB, no
`usb` command (CMD_USB depends on USB_HOST→USB), no preboot default
(PREBOOT="usb start" defaults only with USB_KEYBOARD). USE_PREBOOT=y
survived (explicit line), so preboot ran "usb start" → unknown-command
error, easily missed. "Not even No USB controllers found" is the
tell: that message lives inside usb_init(), which was no longer built.
The banner stayed put because rot=1 vs rot=3 both start at a corner;
the console HAD rotated (upside-down landscape).

Lesson: after ANY defconfig edit + re-run, grep the FULL set of
load-bearing options (USB/PHY/CONSOLE/PREBOOT), not just the new one.

### Fix (u-boot `055563bd`, payload `4b3ce2a2…`)

- defconfig: restore CONFIG_USB=y, drop the duplicated VIDEO lines.
  .config verified for the complete set: USB, DM_USB, USB_XHCI_HCD,
  USB_XHCI_MTK, USB_KEYBOARD, USB_STORAGE, CMD_USB, USE_PREBOOT,
  PREBOOT="usb start", PHY_MTK_TPHY, CONSOLE_ROTATION.
- mt8183_scanout.c: rot=1 → rot=3 (270° CW) per the upside-down
  report; comment updated.
- Build OK, `_start` 0x4C001000, vidconsole3 symbols linked. Flashed
  `4b3ce2a2…`, cmp OK, vbutil body verification succeeded.

### Decision tree for the reboot

| observation | meaning |
|---|---|
| landscape text upright (keyboard hinge side correct), `Bus usb@11200000: N USB Device(s) found`, keys work | DONE — rotation settled |
| landscape text STILL upside down | rot semantics flipped vs assumption — try rot=1 again (would contradict R35, re-check) |
| portrait again | vidconsole3 not bound — transcribe the banner line |
| USB lines present but no keys | keyboard enumeration issue — transcribe the `Bus ...` line |

Awaiting user reboot report.

## Round 37 — Round 36 VERIFIED (landscape + keyboard work); stub-free payload

User confirmed Round 36 fully works (landscape upright, keyboard
functional). User took over the config: `.config` hand-tuned with
VIDEO_FONT_16X32 (bigger console font), CONFIG_CHROMEOS=y,
VIDEO_ANSI off — the defconfig re-run must therefore NOT be executed
blindly anymore (documented in U-BOOT.md).

### Stub removal (payload `cc014efa…`)

The entry shim kept only the load-bearing part: OVL_EN/OVL0_2L_EN=1,
backlight GPIOs 43/176 on, patched branch to U-Boot. Removed the
diagnostic phase (magenta fill, 3-blink train, 5 s hold, delay
subroutine). The shim itself cannot be removed: depthcharge jumps to
payload+0x40 and U-Boot's PIE fixup needs runtime _start 4K-aligned
so U-Boot stays at image offset 0x1000. The build script's single
`b .` search-and-patch still applies (the patched branch is now the
only instruction after the revival writes).

### Verification

U-Boot built with the user's .config untouched; `_start` still
0x4C001000 == __image_copy_start; wrapper now 64 bytes (0x40..0x1000),
runtime _start 0x40001000. Payload `cc014efa…` flashed, cmp OK,
vbutil body verification succeeded.

### Handoff

New `U-BOOT.md` in this repo: boot chain, payload layout constants,
build/flash/verify command sequences, recovery paths, U-Boot tree
state (kept features vs WIP/temporary), rotation and keyboard notes.
The user owns the U-Boot config and further development.

## Round 38 — real display pipeline: DSI host + MIPI TX PHY + BOE panel drivers

Goal: replace the firmware-handoff revival with a real cold bring-up of
the MT8183 display pipeline (MMSYS → OVL0 → OVL0_2L → RDMA0 → COLOR0 →
CCORR0 → AAL0 → GAMMA0 → DITHER0 → DSI0 → panel), upstreamable, on
krane-updates (commits on top of 1ae771d9f90).

### Sources ported (numbered findings)

1. **MMSYS clock gates** (`clk-mt8183.c`): the in-tree clock driver had
   NO mmsys provider (grep CLK_MM/mmsys empty). Added CG_CON0 (0x100)
   / CG_CON1 (0x110) gate groups with set/clr at +4/+8 and the full
   CLK_MM_* gate list, ported from Linux `drivers/clk/mediatek/
   clk-mt8183-mm.c` (v6.6). Legacy vs upstream binding headers number
   CLK_MM_* identically (OVL0=19, DSI0_MM=31…), so the C-side gate ids
   match the DTB cells. Parents: mm_sel → legacy CLK_TOP_MUX_MM(85),
   dpi0_sel → CLK_TOP_MUX_DPI0(110), f26m → CLK_TOP_F26M_CK_D2(4).
2. **GPIO**: no MT8183 pinctrl/gpio driver in-tree (pinctrl-mtk-common
   has no mt8183 table). Wrote a minimal `drivers/gpio/mt8183_gpio.c`
   (dir/dout/din only, pinmux left to firmware) from the device-era
   depthcharge `src/drivers/gpio/mt8183.h` GpioRegs layout: dir[6],
   dout[6], din[6] as GpioValRegs (val@0, set@4, rst@8, 16 B/group),
   blocks at 0x000/0x100/0x200. Pin 43 = DISP_PWM, 45 = LCM_RST,
   66/166/36 = the three panel rail enables, 176 = PERIPHERAL_EN13.
   The set/rst semantics for pins 43/176 were already proven on device
   (Round 4).
3. **MIPI TX PHY** (`drivers/phy/phy-mtk-mipi-tx.c`): UCLASS_PHY,
   PLL programming + analog lane bring-up ported from Linux
   `drivers/phy/mediatek/phy-mtk-mipi-dsi-mt8183.c`: txdiv/txdiv0 by
   rate (956.55 Mbps → txdiv0=2, VCO≈3.83 GHz), SDM_PWR_ON → ISO_EN
   clear → pcw = rate*txdiv<<24/26MHz → POSDIV → PLL_EN; lane_con
   BG_CORE_EN/BG_LPF_EN sequence; lanes switched off until the host
   enables them; CK_CKMODE_EN set. Data rate passed via
   generic_phy_set_mode(PHY_MODE_MIPI_DPHY, bps). The Linux efuse lane
   calibration is NOT ported — coreboot 4.14 runs uncalibrated lanes
   on this device (its mtk_mipi_dphy.c programs fixed values), so
   calibration is an optional refinement. drive-strength-microamp
   defaults to 4600 uA (Linux default).
4. **DSI host** (`drivers/video/mtk_dsi.c`): UCLASS_DSI_HOST.
   Register map from Linux `mtk_dsi.c` = coreboot `dsi_common.h`
   (identical offsets). D-PHY timing formulas ported from both (same
   math). Video timing (VSA/VBP/VFP/VACT, HSA/HBP/HFP word counts,
   PSCTRL custom header 0xb<<26, SIZE_CON, D-PHY turnaround taken out
   of HFP/HBP) ported from coreboot `dsi.c` — the code proven on this
   device. Command FIFO programming (short/long packets, BTA for
   reads, HSTX bit for non-LPM) from Linux mtk_dsi_cmdq(). Data rate =
   pixelclock*bpp/lanes (no mipi ratio; coreboot uses 100/100,
   mainline Linux dropped the ratio entirely). Flow: clocks
   (mmsys gates via clk uclass) → phy set_mode/init/power_on → DSI
   reset (FORCE_COMMIT USE_MMSYS|ALWAYS + CON_CTRL pulse) → phy
   timing → rxtx → 1 ms → DPHY reset pulse → HS clk off → video
   timing → HS clk on; [panel init commands in command mode]; enable()
   → video mode + start. NOTE: coreboot never sets DSI_EN (CON_CTRL
   bit 1) and works on this hardware; followed coreboot.
5. **Panel** (`drivers/video/panel_boe_tv101wum.c`): UCLASS_PANEL
   for "boe,tv101wum-nl6". Timing = Linux boe_tv101wum_nl6_default_mode
   (159.425 MHz, 100/40/24 / 10/14/4). Init command stream ported
   VERBATIM from coreboot 4.14 `panel_params/panel-BOE_TV101WUM_NL6.c`
   (packed {cmd,len,data…} stream, ends with 150 ms delay; NO explicit
   sleep-out/display-on — the panel comes up in video mode, as proven
   by every coreboot boot on this device). Power order from Linux
   boe_panel_prepare(): avdd/avee/pp1800 (fixed regulators, GPIO
   enables now real through the new GPIO driver), 10 ms, reset pulse
   high 1.5 ms / low 8 ms, init DCS, then backlight phandle
   (pwm-backlight) if compiled in. On this board the DCS stream is the
   coreboot one, so no 0x11/0x29 are sent.
6. **Display driver** (`drivers/video/mt8183_display.c`): UCLASS_VIDEO
   on the ovl0 node. Stage machine [dsi] phase prints: clocks (CG_CON0
   ALL + CG_CON1 DSI0 clears + SMI_LARB0 +0x380 = 0) → panel (uclass
   probe) → dsi-init → dsi-enable → pipeline (MMSYS routing/mutex,
   OVL ROI/bgclr, RDMA0 size+FIFO 5 KiB, COLOR0 bypass, PQ relay,
   OVL L0 con/src_size/pitch/addr, engines on, backlight GPIOs). All
   routing bits/mutex/registers from coreboot ddp.h/ddp_common.h.
   **Naming trap**: coreboot's `DISP_OVL1_BASE` (0x14009000) is the
   upstream DT's `ovl_2l0` (OVL0_2L) — OVL0_2L is the second engine of
   the main path, which is why coreboot programs ROI on "disp_ovl[0]"
   and "[1]". Framebuffer at 0xFC000000 (above the 2 GiB DTB DRAM
   window; same region class as the firmware scanout at 0xFD536000),
   FDT memreserve + LMB (commit 1ae771d9f90 pattern). Fallback: any
   failing stage → mt8183_disp_setup_handoff() (the old revival path,
   now shared code in mt8183_disp.c) with a log line naming the stage.
7. **Kconfig**: choice VIDEO_MT8183_SCANOUT (old behavior) vs
   VIDEO_MT8183_DISPLAY (default; selects VIDEO_MTK_DSI,
   PHY_MTK_MIPI_TX, PANEL_BOE_TV101WUM; needs VIDEO_MIPI_DSI, PANEL,
   PHY). Defconfig adds MT8183_GPIO, POWER, DM_REGULATOR,
   DM_REGULATOR_FIXED (POWER was explicitly off in the krane
   defconfig; without it the fixed regulators cannot bind).

### Known gaps / decisions

- SCPSYS display power domain: no MT8183 power-domain driver in-tree
  (mtk-power-domain.c has no mt8183 compatible). The bring-up relies
  on the display MTCMOS being on (true on every path that reaches
  U-Boot here). Documented in the driver.
- Backlight is still the two GPIOs (DISP_PWM 43 + EN_LCD_BL 176)
  driven as board glue, not the pwm-backlight node: driving a real PWM
  on pin 43 would need the MT8183 pinctrl mux (no pinctrl driver
  in-tree), and the backlight node's power-supply chain (reg_vsys ←
  mt6358) has no PMIC driver. The pwm-backlight phandle is wired and
  used when BACKLIGHT_PWM is available.
- The panel node in the upstream DT has no pp3300-supply; Linux enables
  a dummy there. Only avdd/avee/pp1800 are handled.
- OVL layer input format = coreboot's RGBA8888 value for the 32-bit
  XRGB surface (proven on device); naming is MTK-internal.

### Payload

`krane-uboot-payload.bin` sha256
`d29192c06266663f6b2bb2fa683a8acdb180a9d9049358d1c7fb6f17b28bb39c`
(the hash differs between rebuilds — U-Boot embeds a build timestamp;
verify via vbutil instead), vbutil body verification succeeded, `_start == 0x4C001000 ==
__image_copy_start` verified. Flash recipe unchanged (U-BOOT.md).
Serial now shows `[dsi] phase 0/9/F` lines describing which path ran.

## Round 39 — cold bring-up: stream dies at DSI re-init and never returns

User observations, three flashes:

- Flash A (initial): sub-ms white band top (portrait), then black lit.
- Flash B (reset polarity fixed: pulse ends released, of_to_plat
  releases reset at panel probe): sub-ms wide dark-gray artifact while
  backlight ramps, then black, backlight on, NO reset (PANIC_HANG=y
  works — no abort → no magenta).
- Flash C (PHY analog → coreboot LANE_CON 0x3fff0180/0x00c0, timing →
  coreboot HSA24/HBP40/VSA4/VBP14): "blinking band then black".

Decoded so far:

- The brief artifact = the firmware scanout still running while we
  paint/mid-fill; the stream then dies for good.
- Black + backlight + no magenta = U-Boot console runs blind: bring-up
  "succeeds", banner drawn into the new 0xFC000000 fb, but the DSI link
  never re-transmits after our re-init. The failure is inside the
  DSI/PHY re-init itself (link dead), not an abort.
- The DSI re-init kills the firmware stream the moment we stop/start
  the host (mtk_dsi_reset → command mode → video restart), so after
  that point ALL panel-side diagnostics are invisible: our bands paint
  into the old fb, which nothing scans anymore. Instrumentation blind
  spot: any post-first-DSI-touch failure looks like "black, lit".

Audit findings during Round 39 (both fixed in flash C):

1. My PHY used Linux-style analog init (per-lane RTCODE + HSTX LDO
   ref). Linux's per-lane RTCODE regs are written from efuse
   calibration data; uncalibrated Linux path != coreboot path.
   Replaced with coreboot's proven LANE_CON sequence (embeds bandgap +
   lane impedance defaults). drive-strength property dropped.
2. Sync/back-porch split: Linux HSA40/HBP24/VSA14/VBP4 vs coreboot
   HSA24/HBP40/VSA4/VBP14 (same totals). Sync-pulse video mode is
   sensitive to this split; aligned to coreboot.

Open questions for flash D (not yet ruled out):

- mm_sel (TOP mux 0x40[10:8]) must be set to a source ≥ 956.55 Mbps ×
  lanes/4 ... actually DSI0_IF digital clock comes from mm_sel; the
  DT "hs" clock is mipi_tx0 PLL (a clock OUTPUT of the PHY). We never
  program the mm_sel mux parent — we rely on firmware's setting.
  coreboot never touches muxes either (relies on defaults), so
  probably fine.
- CG_CON1 bit 7 (DISP_26M / CLK_MM_26M): coreboot does NOT clear it
  (CG_CON1_DISP_DSI0 = bits 0|1 only); we match.
- MIPID0_26M: an apmixedsys 26M gate feed consumed ONLY by mipi_tx0.
  The U-Boot clk driver has NO MIPID0 26M gate (apmixed_plls only).
  On this firmware generation it is on at handoff. NOT a suspect for
  the dead link (panel/DSI stays alive until WE touch it).

## Round 40 — cold bring-up WORKING, diagnostics removed, series cleaned

Final boot log on device: "[dsi] phase 9: full bring-up done", U-Boot
console on panel (landscape, rot=3 from DT), kernel boots after
bootefi bootmgr with logs visible. Serial console via Suzy-Q works.

Root causes found this round (in order):

1. **Panel driver NULL deref (the big one)**: boe_panel_send_init_
   sequence() reads plat->device, but nothing ever set mplat->device.
   EVERY previous build aborted (PANIC_HANG) at the panel init stage,
   right after the reset pulse. Fixed: mt8183_display.c publishes
   mplat->device = &priv->device in STAGE_PANEL.
2. **CONFIG_BAUDRATE=921600**: payload reprogrammed the console to
   921600 (8x) while BootROM/terminal run 115200. minicom -b does not
   reliably switch speeds; stty/picocom do. Fixed to 115200.
3. **mtk_serial HS0 sample regs**: _mtk_serial_setbrg wrote
   sample_count=0/sample_point=0xffffffff in low speed mode; coreboot
   leaves them untouched. Fixed upstreamable (serial: mtk commit).
4. **bpix line eaten by diag edits**: a temporary-diag edit removed
   uc_priv->bpix = VIDEO_BPP32 from mt8183_display_bringup(); with
   bpix=0 the console glyph path returns -ENOSYS ("Error: 1 bit/pixel
   mode, but BMP has 256"), no text renders, video_clear mis-fills.
   Restored.
5. DSI re-init kills the firmware scanout mid-boot: all panel-side
   diagnostics after the engine reset are invisible. Workaround during
   bring-up: minimal-touch DSI path (no engine/DPHY resets, no PLL
   reprogram; firmware state + panel init + video restart). The final
   cold path resets everything and works (panel reset + full init).

Diagnostic techniques that worked: color bands into the live firmware
scanout (visible only until the DSI reset), backlight blink post-codes
(panel-independent), Suzy-Q serial (decisive).

Final series on krane-updates (base 1ae771d9f90, checkpatch 0 errors):
- fab1110ff35 clk: mt8183 MMSYS display gates
- 7de03c4fa39 gpio: mt8183 GPIO driver
- 58d57350b43 phy: MT8183 MIPI TX D-PHY
- 0c6a4de7ce2 video: MT8183 MIPI DSI host
- c91f8f93d5a video: BOE TV101WUM-NL6 panel
- eaab63c84cb video: mt8183 display pipeline bring-up
- 36c0a9919eb krane: select full display pipeline bring-up
- 68b992ad2c2 serial: mtk sample-registers fix
- 357608ac481 arm: mediatek: krane: 115200 serial console

Known issues / follow-ups:
- infra_clks array order vs legacy header IDs mismatch after index 51
  (pre-existing upstream): needs id_offs_map; bites CLK_INFRA_USB,
  AP_MSDC0, I2C4/6/7/8 gate addressing.
- efi_add_memory_map rejects 0xfc000000 (above DTB DRAM window):
  "reserving fdt memory region failed ... -22"; -17 for payload region
  is benign. Matters for kernel GOP handoff quality.
- Kernel (pmOS) takes over the panel via fbcon/GOP; panel node is
  status=disabled in the kernel DT, so no kernel DSI driver probe.
- Kernel "stuck at Running /init as init process" on serial: pmOS-side
  init issue, not display.
- MAINTAINERS warnings from checkpatch: add entries when submitting.

## Round 41 — [ROOT CAUSE] serial stall = nonexistent console=ttyMT0; fixed to ttyS0,115200

Symptom recap: Ubuntu kernel (7.0.0-30-generic, grub/U-Boot path) printed
early logs on Suzy-Q serial until a fixed point mid-log-line, then silence.
Looked like a hang; was a console handoff, not a hang.

Root cause: `console=ttyMT0,921600` in GRUB_CMDLINE_LINUX_DEFAULT names a
device that does not exist on this kernel stack. Mainline 8250_mtk
(CONFIG_SERIAL_8250_MT6577) registers its ports on the universal 8250
driver, device/console name "ttyS" — NOT "ttyMT" (ttyMT0 is the MTK
*vendor* driver, drivers/tty/serial/mtk-serial.c, not in mainline).
Ground truth from this same machine's pmOS kernel (6.12.87-mt81, same
driver, same uart node):
  11002000.serial: ttyS0 at MMIO 0x11002000 ... is a ST16650V2
  printk: legacy console [ttyS0] enabled

Failure sequence on the Ubuntu boot:
1. earlycon=uart8250,mmio32,0x11002000 prints at firmware rate (115200) —
   early logs readable.
2. console_init(): vt console enables (console=tty0 matched, CON_CONSDEV);
   printk unregisters all boot consoles → earlycon dies, cutting output
   mid-line (the "stall point", deterministic).
3. The pending ttyMT0 entry is retried at every later console
   registration and never matches (univ8250_console.match only accepts
   console=uart8250,... earlycon-style entries). No serial console is
   ever enabled → silence for the rest of boot. Kernel keeps running on
   tty0 only.

Doc's open question answered: `console=ttyS0` WITHOUT a baud argument is
NOT firmware-rate. serial8250_console_setup (8250_port.c) defaults
`int baud = 9600` when options are absent and probing is off; only the
console=uart8250,mmio32,... match path probes the hardware divisor
(probe_baud) to keep the firmware rate. So the fix must state 115200
explicitly. (mtk8250_set_termios handles 115200 fine: uartclk 26 MHz,
quot=14, highspeed register 0.)

Fix applied (on this rootfs, U-Boot/grub boot path only — the running
depthcharge boot is unaffected):
- /etc/default/grub: GRUB_CMDLINE_LINUX_DEFAULT changed from
  "console=tty0 console=ttyMT0,921600 earlycon=uart8250,mmio32,0x11002000"
  to "console=tty0 console=ttyS0,115200 earlycon=uart8250,mmio32,0x11002000"
- sudo grub-mkconfig -o /boot/grub/grub.cfg (backup:
  /etc/default/grub.bak-round41). Verified regenerated grub.cfg carries
  console=ttyS0,115200 and the devicetree /boot/dtb line (10_linux keeps
  it as long as /boot/dtb exists).

Boot mechanics after fix: earlycon (115200) → dies when vt enables at
console_init() → ttyS0 console matched/enabled at console_initcall
(generic "ttyS" name match + .setup → uart_set_options 115200 on the
legacy placeholder port, harmless on arm64; Rockchip/rk3399
console=ttyS2,1500000 uses the same path) → real port registers via
mtk8250_probe ~0.7 s later, console follows serial8250_ports[0]
automatically; hardware stays at U-Boot's 115200. Small window (~0.7 s)
between console_init and 8250_mtk probe where serial messages are lost
to the placeholder port; add keep_bootcon if that gap matters (duplicates
output; also useful as a diagnostic: with keep_bootcon the whole log
stays on serial regardless of handoff).

Verification checklist (user, needs Ctrl+U in depthcharge):
1. Boot to grub → Ubuntu with terminal at 115200 8N1.
2. Expect: early logs (earlycon), brief gap, then full kernel log through
   userspace at 115200; /dev/console = ttyS0 (last console= wins).
3. If silence still occurs: add keep_bootcon to the cmdline and compare —
   if output continues, it is purely a console handoff issue; if not,
   capture the last ~50 lines and triage initcalls (initcall_debug).
4. Panel check (separate bug, believed fixed): frozen U-Boot console
   should show no noise blocks; kernel output on tty0 via the EFI
   framebuffer may appear on the panel once vt/simpledrm come up — that
   is expected, not a regression.

No U-Boot-side change needed; no payload reflash required. krane-updates
untouched.

## Round 42 — stall survives console fix: real hang; debug cmdline staged

Round 41 fix (ttyS0,115200) did NOT change the symptom: output still cuts
at the same point (~2.3s, mid-line, right after the "evm: security.evm
attributes" message). Conclusion: the ttyS0 console WAS working from
~0.8s (mtk8250 probe registers port line 0, which is the same struct the
console attached to, so the console follows the real port automatically;
output between 0.8s and 2.3s already went through the working console at
115200). The stall is a genuine hang, not a console handoff artifact.

Cut-point analysis: "evm: security.evm attributes reinitialized" is a
late_initcall (evm_init_xattrs). What runs next, in order:
1. remaining late(_sync/_rosync) initcalls,
2. "Freeing unused kernel memory", free_initmem,
3. smp_init() — secondary CPU bring-up via PSCI (BL31) — AFTER all
   initcalls, immediately before "Run /init",
4. "Run /init as init process".
The mid-line cut is consistent with either a late-initcall hang or a
hang inside smp_init/PSCI cpu_on (nbcon printk kthread can be preempted
mid-line by a hard hang). U-Boot-handoff state is a candidate for both
(U-Boot payload leaves devices in non-reset state; secondary CPUs were
parked by BL31 either way).

Debug boot staged (2026-09-02):
- /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT now:
  "console=tty0 console=ttyS0,115200 earlycon=uart8250,mmio32,0x11002000
   keep_bootcon ignore_loglevel initcall_debug maxcpus=1"
  (backup: /etc/default/grub.bak-round42; grub.cfg regenerated).
- keep_bootcon: earlycon survives the whole boot -> any output loss after
  this point is hardware/hang, never handoff.
- initcall_debug + ignore_loglevel: last "calling <fn>" without a matching
  "initcall <fn> ... returned" names the hang.
- maxcpus=1: tests the PSCI/secondary-CPU bring-up path.

User checklist (needs Ctrl+U, terminal 115200 8N1, CAPTURE TO FILE):
1. Boot grub -> Ubuntu. WARNING: boot is now slow and verbose (initcall
   trace over 115200 serial, ~30-60s extra; earlycon + ttyS0 duplicate
   every line — expected, same UART).
2. Capture the FULL serial log to a file (e.g. picocom -b 115200
   /dev/ttyUSB0 | tee boot42.log) from before U-Boot output starts.
3. Report the last ~50 lines. Key reads:
   - Last "calling ..." with no "returned" -> hung initcall (name it).
   - Last line = "smp: Bringing up secondary CPUs ..." -> PSCI/secondary
     CPU hang; next test drops maxcpus=1 and we go after the U-Boot
     handoff / TF-A PSCI state (compare with depthcharge boot of the
     same kernel; MT8183 is 4xA73+4xA53, all 8 boot fine via depthcharge
     on the pmOS kernel with the same BL31).
   - Output now survives past 2.3s to login -> the single-core change
     fixed it; then bisect smp/hotplug vs initcall.
4. If it STILL cuts mid-line at the same spot with earlycon kept alive:
   hang is in whatever the last complete "calling" line names, or in
   free_initmem/smp_init if no initcall_debug lines trail. A hard hang
   with a fully drained console that cuts mid-line would point at the
   printing path itself dying with the machine (clock/powerdomain kill
   during the print) — then we check whether the system is actually
   alive (add a delayed "reboot" via kernel cmdline? no — check via
   serial ping input: type a key; if the tty echoes, userspace is alive
   and only output died).

No U-Boot changes made. krane-updates untouched.

## Round 43 — debug boot #2 result "nothing changed"; aliveness/panic probes staged

User reports debug boot (keep_bootcon ignore_loglevel initcall_debug
maxcpus=1) = "nothing changed", still cut mid-line at ~2.3s. Suspicion:
initcall_debug would visibly flood the log from ~0.5s — an IDENTICAL log
suggests the cmdline may not have been applied. Need the "Kernel command
line:" line from the user's capture (kernel prints it early under
ignore_loglevel) plus the last ~100 lines.

New cmdline (added this round; backup /etc/default/grub.bak-round43):
... keep_bootcon ignore_loglevel initcall_debug maxcpus=1 panic=10
softlockup_panic=1 hung_task_panic=1
Rationale: CONFIG_LOCKUP_DETECTOR / CONFIG_DETECT_HUNG_TASK are on;
hung task default timeout 120s. If the kernel is alive-but-stuck, a
panic + 10s reboot follows within ~2min; the reboot brings back the
U-Boot banner on serial — visible aliveness proof even if the UART dies
mid-boot. Nothing after many minutes = hard hang or dead UART path.
(Type a key on the terminal: tty echo = userspace alive.)

ramoops dead end (for now): CONFIG_PSTORE_RAM=m (module, loads too late
to catch an early panic) and the Ubuntu /boot/dtb has no ramoops node;
pmOS kernel's own DT reserves 1 MiB ramoops at 0xffedb000. Could add a
ramoops node to /boot/dtb + load ramoops.ko from initramfs as a later
panic-capture path.

Candidate explanations for an identical cut across 3 cmdlines:
A. cmdline never applied (verify via "Kernel command line:" in capture).
B. hard hang independent of cmdline content at a fixed early point
   (U-Boot handoff state: xHCI/eMMC/DSI left active; or a driver probing
   a device in non-reset state — initcall_debug will name it).
C. system alive, UART dies at a fixed point (clock/mux/pinctrl) —
   indistinguishable from hang on serial alone; panic probes + key-echo
   test split this.

## Round 44 — fb-log.txt analyzed: boot never had the new cmdline; true root cause

fb-log.txt (user capture, mtime 2026-09-02 15:05) shows the OLD cmdline
boot: 0 "calling" initcall_debug lines (would be thousands), no
"printk: legacy console [ttyS0] enabled" at the 1.787s ttyS0 port
registration, no earlycon disable message. The Kernel command line
printk itself is truncated mid-line ("root=0") — the capture pipeline
drops bytes (other lines spliced: "[0x410fd034]28f3628b...",
"0x...bbc00000ce(s) found"). So all Round 42/43 debug params were never
exercised; and the Round 41 fix has plausibly never been tested either.

The log instead pins the REAL stall mechanism:
- 1.787s: 8250_mtk probes, registers ttyS0. Old cmdline has console= only
  tty0+ttyMT0 → port is NOT a console → uart_configure_port() powers it
  OFF (serial_core: "power down all ports by default, except the console",
  uart_change_pm(UART_PM_STATE_OFF)) → 8250_mtk runtime suspend gates the
  UART clock.
- earlycon keeps printing raw MMIO into a now-unpowered UART → output
  dies silently at a fixed point (~2.38s, probe+autosuspend delta),
  mid-line. Kernel continues fine on tty0.
This explains: identical cut across all previous boots, why it starts
exactly at 8250_mtk probe + delay, and why pmOS (no console= → all
consoles default-enabled → ttyS0 becomes console → port stays powered)
never stalls. The Round 41 fix (console=ttyS0,115200) attaches the
console at probe → port stays powered → serial should survive. It just
has never actually been booted.

Next boot protocol (Round 45):
1. Lossless capture, no terminal in the path:
     stty -F /dev/ttyACMx 115200 raw -echo
     cat /dev/ttyACMx > boot45.log
   (Ctrl+C after). Verify: grep -c "calling " boot45.log  (expect
   thousands); grep "console \[ttyS0\] enabled" (expect present right
   after the 11002000.serial line).
2. If the cmdline STILL doesn't apply (no "calling" lines): grub is
   serving a stale config — at the grub menu press "e" on Ubuntu and
   boot the edited entry (Ctrl-X), or move the params directly into
   /boot/efi/EFI/BOOT/grub.cfg.
3. If params verified and output still dies: check key-echo + panic
   reboot probes (Round 43) — then it is genuinely the UART path/hang,
   not console power-off.

## Round 45 — [ROOT CAUSE #2] standalone grub image with hardcoded cmdline; grub-install redone

picocom capture (fb-log.txt, 15:23) finally delivered a CLEAN "Kernel
command line:" line: it read "console=ttyMT0,921600" — the ORIGINAL
cmdline, no Round 41-43 params ever reached the kernel in any boot.

Root cause of the delivery failure: /boot/efi/EFI/BOOT/BOOTAA64.EFI was
a grub-mkstandalone image (905 KB, built Aug 31) with a memdisk-embedded
grub.cfg containing hardcoded menuentries ("linux (hd0,gpt3)/boot/
vmlinuz-7.0.0-30-generic ... console=ttyMT0,921600 earlycon=..."). It
never read the ESP stub nor /boot/grub/grub.cfg — every grub-mkconfig
since was a no-op. The 209-byte ESP stub existed but was dead code
(standalone image prefix = (memdisk)/boot/grub). This also explains the
BOOT_IMAGE=(hd0,gpt3)/... form in the kernel log (matches the embedded
entry verbatim).

Fix: sudo grub-install --target=arm64-efi --efi-directory=/boot/efi
--boot-directory=/boot --bootloader-id=BOOT --no-nvram (grub 2.14),
then cp grubaa64.efi over BOOTAA64.EFI (fallback path). New image:
2.9 MB monolithic, zero embedded cmdline occurrences, plus grub-install
wrote a fresh EFI/BOOT/grub.cfg stub (search.fs_uuid e362f850 ->
configfile /boot/grub/grub.cfg). Chain now: U-Boot bootmgr ->
BOOTAA64.EFI -> /boot/grub/grub.cfg (ext4) -> Ubuntu entry with
console=ttyS0,115200 keep_bootcon ignore_loglevel initcall_debug
maxcpus=1 panic=10 softlockup_panic=1 hung_task_panic=1.
Backup of the standalone image + old stub:
/boot/efi/EFI/BOOT-standalone-bak45 (restore by copying back if ever
needed).

Round 44's serial-path analysis stands as the expected outcome: with
console=ttyS0,115200 the port is a console at 8250_mtk probe time, so
uart_configure_port keeps it powered and the "power down non-console
ports" path that killed earlycon at ~2.4s never runs.

Round 46 test protocol (user, picocom OK — capture was lossless):
1. Ctrl+U -> U-Boot -> grub -> Ubuntu.
2. First marker: "Kernel command line:" line must contain console=ttyS0,
   115200 keep_bootcon initcall_debug.
3. Thousands of "calling ..." lines; "printk: legacy console [ttyS0]
   enabled" right after the 11002000.serial ttyS0 line.
4. Boot will be slow/verbose (115200 flood, earlycon+ttyS0 duplicate
   lines). If it reaches login: serial console fixed; then trim cmdline
   back (drop debug params) and re-verify a clean boot.
5. If output still dies: key-echo test + wait for panic-reboot probes
   (~2 min, hung_task 120s + panic=10).

## Round 46 — serial stall FIXED (kernel reaches initrd); initrd missing mmc devices

Round 45 grub-install fixed the delivery: kernel boots past the old
2.4s stall all the way to the dracut initrd. Serial stall ROOT CAUSE
confirmed as the console power-off path (Round 44): ttyS0 console
attached at 8250_mtk probe keeps the port powered.

Cmdline trimmed per user request — now:
  console=tty0 console=ttyS0,115200 earlycon=uart8250,mmio32,0x11002000
(backup /etc/default/grub.bak-round46). earlycon kept temporarily while
the initrd issue is debugged; drop it at the end.

NEW ISSUE: dracut initrd cannot find root by UUID; /dev has no mmc*.
Static analysis says the initrd is complete:
- usr/lib/modules/.../drivers/mmc/host/mtk-sd.ko.zst PRESENT (note: file
  is mtk-sd.ko, module name mtk_sd — earlier grep with "mtk_sd" missed it)
- mmc_block, cqhci, mmc_hsq present; modules.alias has
  of:N*T*Cmediatek,mt8183-mmc -> mtk_sd; modules.dep lists deps
- PMIC chain present: mtk-pmic-wrap, mt6397 (MFD), mt6358-regulator,
  mt6397-regulator, rtc-mt6397
- pinctrl-mt8183 / clk-mt8183 / infracfg are built-in (=y)
- vermagic matches kernel image (same Jul 31 build), module signed
- /boot/dtb (custom krane-fb-stub DTB) is byte-identical in structure to
  upstream /boot/efi/mt8183-kukui-krane-sku176.dtb (full-file diff
  EMPTY): mmc0 @11230000 okay, compatible mediatek,mt8183-mmc, clocks
  topckgen+infracfg phandles valid
So the failure is runtime: either mtk_sd never got loaded by udev, or
its probe fails/defers. Diagnostics for the dracut emergency shell:
  cat /proc/modules | grep -Ei 'mtk|mmc|pmic'
  modprobe mtk_sd && ls /dev/mmc*
  dmesg | grep -iE 'mtk-sd|msdc|mmc|pmic|regulator'
  ls /sys/bus/platform/devices | grep mmc
If modprobe succeeds and /dev/mmcblk0 appears: just "exit" — dracut
resumes, mounts root, boot completes; fetch dmesg/journal from the
booted system afterwards to pin the root cause (probe defer vs error).
Round 46 backup: /etc/default/grub.bak-round46.

## Round 47 — mt6358_regulator was the missing initrd load; native display path completed + plymouth

User confirmed: modprobe mt6358_regulator in the dracut shell unblocked
the initrd (regulators registered, mmc0 deferred probe resolved, root
mounted). Boot then proceeded to systemd (Ubuntu 26.04.1 userspace on
this rootfs) and stopped after "Starting wpa_supplicant.service" — open
issue, suspected mt7663s/mt76 SDIO path (initrd dmesg showed msdc cmd52
errors on mmc1). Discriminator: press Enter on serial — login prompt =
system alive, wpa-supplier-only stuck.

Panel goes blank when the kernel takes over ("graphical console
disappear"): NOT a reason to blacklist the display stack (rejected —
native display is the goal). Real cause found: the initrd contained
mediatek-drm/mtk_mmsys/mtk_mutex/DSI-phy (which reset the DSI link U-Boot
left running -> panel dark) but NOT the panel/backlight/PWM modules, so
nothing could re-light it. The DTB (/boot/dtb) has the full native path
ENABLED: panel@0 boe,tv101wum-nl6 (avdd/avee/pp1800 fixed GPIO
regulators), pwm-backlight on SoC pwm@11005000. The old "DSI/panel
disabled in distro DTB" note does not apply to this DTB.

Fixes applied:
- /etc/dracut.conf.d/50-display.conf:
    add_drivers+=" panel-boe-tv101wum-nl6 pwm-mediatek pwm_bl "
    force_load="mt6358_regulator"
  (force_load because udev failed to load the already-present module at
  runtime in the previous initrd — root cause unknown, worked around.)
- plymouth + plymouth-theme-spinner + plymouth-label installed via apt;
  dracut now embeds plymouthd (50plymouth) — initrd rebuilt
  (53 MB, 16:43). NOTE: apt's dracut trigger also runs update-initramfs,
  so future kernel/apt operations keep the config.
- /etc/default/grub: cmdline now
    console=tty0 console=ttyS0,115200 earlycon=uart8250,mmio32,0x11002000 splash
  ("splash" only, no "quiet" — serial stays verbose for the wpa debug).
  grub.cfg regenerated.

Expected next boot: panel lights during initrd (native DSI panel takes
over from U-Boot firmware scanout — brief flicker), plymouth splash on
the panel, serial stays verbose. If panel still dark: capture
dmesg | grep -iE 'panel|dsi|drm|backlight' from serial login and check
panel bind/defer.

## Round 48 — regulators still missing with force_load; deterministic pre-udev hook

fb-log.txt (16:52 boot): plymouth-start ran, but the deferred tree was
back — mmc0 (ldo_vio18), usb (ldo_vusb), gpu (buck_vgpu), i2c (vcn18/
vcamio), AND the whole MT8183 power-controller:
  mtk-power-controller: power-domain@2 failed to get power supply
  (domain-supply = MT6358 buck, coupled vproc pair)
  -> iommu, all larbs, ovl/rdma/dsi/mutex/aal/ccorr/color/gamma, pwm,
     backlight_lcd0 all defer on "supplier 10006000.syscon:power-
     controller not ready"
So the ENTIRE deferred forest (eMMC + display + iommu + backlight) has a
single root: MT6358 regulators not registering. force_load="mt6358_regulator"
did NOT load it (no evidence of any generated load mechanism in the
initrd).

Fix (deterministic): dracut pre-udev hook. Gotchas found:
- dracut 110-11 does NOT copy host /usr/lib/dracut/hooks into the image.
- Runtime hookdir = /var/lib/dracut/hooks (dracut-lib.sh:367); stage dir
  is pre-udev (dash), per source_hook pre-udev in usr/bin/dracut-pre-udev.
- Placed /var/lib/dracut/hooks/pre-udev/01-mtk-pmic.sh (host) with:
    modprobe mtk_pmic_wrap; modprobe mt6397; modprobe mt6358_regulator
  and /etc/dracut.conf.d/50-display.conf: install_items+=" <that path> "
  (install_items preserves the path; survives apt-triggered
  update-initramfs).
Verified in rebuilt initrd: var/lib/dracut/hooks/pre-udev/01-mtk-pmic.sh
present, executable. Manual escape if a boot still lands in dracut
shell: modprobe mtk_pmic_wrap mt6397 mt6358_regulator, then exit.
Expected: regulators register ~4s into initrd; mmc0, power-controller,
display/iommu/backlight all unblock; panel lights; plymouth splash.
Still open: wpa_supplicant hang (previous boot; suspect mt7663s/mt76
SDIO after msdc cmd52 errors on mmc1). Check with Enter-on-serial for
login prompt, then journalctl.

## Round 49 — [ROOT CAUSE] soft lockup = live scanout faulting through re-enabled M4U; U-Boot quiesce committed

fb-log.txt (17:13): eMMC fixed (regulators registered via pre-udev hook),
boot went further than ever: initrd pivot, real-root systemd,
wpa_supplicant [OK] (previous hang gone). Two issues surfaced:

1. UBSAN shift-out-of-bounds mt6358-regulator.c:384 — ffs(0)-1 = -1 in
   mt6358_get_buck_voltage_sel. Root: mt6358_volt_fixed_ops routes
   get_voltage_sel through the buck helper, but the fixed LDOs
   (vio18, vrf12, ...) never initialize da_vsel_reg/da_vsel_mask
   (v7.0 mainline has the same code). Non-fatal: read-only path, selector
   0 == nominal for these LDOs. Upstreamable fix: use
   regulator_get_voltage_sel_regmap for fixed ops (they have valid
   vsel_reg/vsel_mask). NOT the lockup cause.
2. mtk-iommu fault storm: reads at iova 0xbe000xxx (the U-Boot
   framebuffer at 0xBE000000!) from master larb0/port0 — the display
   engine kept scanning out the U-Boot console while the kernel's M4U
   enabled translation; the region has no IOMMU mapping. Interrupt storm
   starved timer handling: CPU#1 rcu_exp_gp_kthr soft lockups (26/52/89s),
   rcu_preempt GP kthread starved on CPU4 (first A73, "timer wakeup
   didn't happen"). Boot wedged around 40s, right at
   NetworkManager/ModemManager startup.

Fix (commit d3d3502de0a on krane-new-panel-driver, checkpatch 0/0):
"video: mt8183: quiesce the display pipeline at ExitBootServices".
- drivers/video/mt8183_display.c: board_quiesce_devices() — stops the DSI
  video stream + powers the D-PHY down (mtk_dsi_disable), stops the
  pipeline engines (OVL0/OVL0_2L/RDMA0/COLOR/PQ blocks/mutex), turns the
  backlight off and gates the MMSYS display clock domains. The kernel
  display driver does a full cold bring-up (Round 40), so nothing of the
  handoff state needs preserving.
- drivers/video/mt8183_disp.c: mt8183_disp_disable_backlight() (inverse
  of enable; DOUT clear registers at +8 in the GPIO dout block).
- drivers/video/mt8183_disp.h: DOUT_CLEAR macro + prototypes.
U-Boot rebuilt; payload rebuilt and vbutil_kernel-verified:
krane-uboot-payload.bin sha256 771a0fc6dfda12af9d6779b7637787dc5177b6db3e3f7cba7443be9911a09bb5.
PENDING: dd to /dev/mmcblk0p1 (user confirmation per protocol), then
boot via Ctrl+U.

Expected next boot: panel goes dark after the kernel's EFI stub calls
ExitBootServices (U-Boot hands over with the pipeline quiesced — no more
frozen console, no fault storm), kernel brings the panel up natively
(~5-10s), plymouth splash, full boot. The UBSAN warning remains (harmless;
module fix is a follow-up).

Round 49 addendum: payload flashed to /dev/mmcblk0p1 (dd verified with
cmp against the source file, 860160 bytes). Ready for Ctrl+U boot test.

## Round 50 — [ROOT CAUSE] U-Boot quiesce works; new oops = mtk_smi larb runtime-resume before iommu bind; patched modules installed

fb-log.txt (17:54): the IOMMU fault storm + RCU soft lockup are GONE (the
U-Boot quiesce commit d3d3502de0a works). Boot got to coldplug, then:

  Internal error: Oops 0000000096000004, FAR=0x0, pc
  mtk_smi_larb_config_port_gen2_general+0xf0 [mtk_smi], lr
  mtk_smi_larb_resume+0xb8, via pm_runtime_get_suppliers from
  mtk_drm_init (mediatek_drm module load, udev-worker PID 266).
  Code bytes match mainline v7.0 drivers/memory/mtk-smi.c exactly:
  `ldr x1,[x28,#144]` (= larb->mmu, offset 144) then `ldr x1,[x1]` at
  +0xf0 -> NULL because larb->mmu is only set by mtk_smi_larb_bind(),
  the IOMMU component bind, which ran at 8.168s — AFTER the oops at
  8.155s. mediatek_drm's probe runtime-resumes the larb through the
  device link/genpd before the IOMMU binds it. Unfixed in upstream
  master (checked mtk-smi.c master == v7.0).

Also confirmed this boot: mt6358 UBSAN fires from
mt6358_regulator_probe->regulator_register->machine_constraints_voltage
(ops->get_voltage_sel on register: mt6358_get_buck_voltage_sel derefs
da_vsel_mask which MT6358_REG_FIXED never sets). Fixed LDOs DO have
valid vsel_reg/vsel_mask (MT6358_*_ANA_CON0 / GENMASK(3,0)), so the
correct ops is regulator_get_voltage_sel_regmap (as
mt6358_volt_range_ops uses for regmap reads elsewhere).

Fix: rebuilt both modules out-of-tree against the Ubuntu headers
(/usr/src/linux-headers-7.0.0-30-generic, Module.symvers, MODVERSIONS
OK, vermagic matches, unsigned load = taint only, MODULE_SIG not
forced). Source validated against the shipped modules before patching:
rebuilt unpatched mtk-smi.ko reproduces the oops Code bytes at +0xf0.

- mtk-smi.ko: guard in mtk_smi_larb_resume: if (!larb->mmu) return 0;
  after enabling clocks (no IOMMU master attached yet -> nothing to
  configure). Upstreamable: "memory: mtk-smi: skip MMU port config on
  larb runtime-resume before the IOMMU binds".
- mt6358-regulator.ko: mt6358_volt_fixed_ops.get_voltage_sel ->
  regulator_get_voltage_sel_regmap (line 495; vproc/vsram buck ops
  untouched).

Installed to /lib/modules/7.0.0-30-generic/kernel/drivers/{memory/
mtk-smi.ko.zst,regulator/mt6358-regulator.ko.zst}; originals kept as
*.orig-round49 (NOTE: named round49, stamped before analysis); depmod
run. BTF skipped (no vmlinux) — same as many Ubuntu modules.

Expected next boot: no oops; mediatek_drm probes; panel lights
natively; plymouth; boot to login.

## Round 51 — false alarm: same oops; initrd ships stale module copies; initrd rebuilt

fb-log.txt (next boot): IDENTICAL oops (same pc +0xf0, same Code bytes,
same UBSAN from mt6358_get_buck_voltage_sel). Cause: dracut initrd
contains its own module copies (usr/lib/modules/7.0.0-30-generic/...)
built Jul 31 — the modules we replaced under /lib/modules never load;
initrd modules are what run during coldplug (before root pivot). The
"more errors" = the oops printed twice (a second udev worker retried
the mediatek_drm finit_module and hit the same fault) + dracut
initqueue hang, collateral of the oops killing the worker handling the
mmcblk uevent chain (root node never settled).

Fix: dracut -f rebuild — initrd now carries the patched mtk-smi.ko.zst
(114115 bytes, mine keeps DWARF that Ubuntu strips to dbgsym; loads
fine) and patched mt6358-regulator.ko.zst. Stale .orig-round49 backup
files also got copied into the initrd by dracut (harmless, never
loaded).

Next boot expectation: no oops, no UBSAN, initqueue completes, root
mounts, mediatek_drm probes, panel lights natively.

## Round 52 — [WEDGE] boot reaches real root; CPU2 kworker spin + multi-CPU timer death; evidence-led prep for next boot

fb-log.txt (post-Round-51 initrd): no oops, no UBSAN, no IOMMU storm —
the module fixes hold. Boot reaches systemd, NetworkManager,
wpa_supplicant. Display: backlight comes back (pwm_bl) but screen stays
BLACK and mediatek_drm never registers an fbdev. Then CPU#2 soft lockups
(26/52/119s, kworker/2:3), RCU stalls on 4/5/7, rcu_preempt kthread
(cpu3) "timer wakeup didn't happen". Start of wedge ~13.8s.

Key observations:
- The soft-lockup STACK DUMPS never appear on the serial console (only
  the header lines) — dumps are lost somewhere in printk/console path.
  Rely on ramoops next boot instead.
- The current initrd was MISSING mediatek_drm (my Round-51 rebuild
  dropped it vs the Jul 31 build). So this boot loaded mediatek_drm from
  the real root at ~13s — exactly when the wedge started. Both full-boot
  wedges (49, 52) time-correlate with mediatek_drm activity; the boot
  where its probe oopsed early (51) never wedged the CPUs.
- PSCI CPUidle EXonerated: the running pmOS kernel uses psci_idle with
  the SAME WFI/cpu-sleep/cluster-sleep-0 states and the same stock ATF,
  up 17min+ fine. Not the cause despite first suspicion.
- No unbounded loops found statically in mtk_crtc/mtk_dsi/mtk-mutex/
  cmdq-mailbox/cpufreq/mtk-coupler; only the DSI IRQ handler
  do{}while(tmp & DSI_BUSY) (unbounded, irqs-off) — but a CPU stuck
  there could never report its own soft lockup, so it is not the
  reported kworker spin. U-Boot quiesce now clears DSI INTEN/INTSTA
  anyway (commit c112424b983, checkpatch clean; payload sha256
  7ed2de1e0ab3ff80cb6f461889d06bbf3d9d773f555308d02107dcb124cea92b,
  flash PENDING user confirmation).
- grub2-common/grub-initrd-fallback FAILED at exactly the wedge moment
  (collateral; recordfail cleared via grub-editenv).

Prepared for the next boot (all in place):
1. cmdline: timer_migration=off (targets the timer-migration/tick
   failure class matching "timer wakeup didn't happen"; nohz/timer
   rework landed 6.13..7.0 while pmOS runs 6.12.87 stable with the
   backported fixes), sysrq_always_enabled, softlockup_panic=1
   panic=10 (auto-evidence: wedge -> panic -> stacks -> warm reboot).
2. ramoops via DTB: /boot/dtb-krane-ramoops.dtb adds a reserved-memory
   region at 0xBFF00000 (1 MiB) + ramoops node (console 512K, dmesg
   128K, pmsg 128K). grub entries use it via devicetree (10_linux picks
   /boot/dtb-7.0.0-30-generic first). pmOS kernel has PSTORE=n and a
   different DTB so it ignores the region.
3. initrd rebuilt: mediatek_drm + full display stack (mtk_mutex,
   mtk_mmsys, mtk_smi, mtk_iommu, dsi phy, cmdq) + ramoops (force_load)
   + scp.img.zst restored/added. scp remoteproc should now bind at
   initrd coldplug instead of failing with -2.
4. grub experiment entries: "Ubuntu 7.0 EXP-B: cpuidle.off=1" and
   "pmOS kernel via U-Boot (wedge bisect)" (pmOS kernel + pmOS DTB +
   pmOS initrd from the ESP, under our U-Boot). Ladder: default entry
   (timer_migration=off) -> if wedged+panicked, second boot archives
   /sys/fs/pstore via systemd-pstore (enabled) to
   /var/lib/systemd/pstore; read stacks from there. If still wedging
   without evidence, try EXP-B, then the pmOS-under-U-Boot entry to
   separate bootloader state from kernel regression.

Black display analysis: pipeline is quiesced at ExitBootServices (by
design), simpledrm fb0 exists but nothing scans it out; mediatek_drm
did not complete bind in this boot (late load + wedge). With the display
stack back in the initrd and the wedge fixed, the kernel should bring
the panel up natively. If the wedge turns out to be INSIDE mediatek_drm
probe, the ramoops stacks will show it.

Round 52 addendum (evidence path locked in): CONFIG_PSTORE_CONSOLE and
PSTORE_PMSG are NOT set in the Ubuntu kernel, so ramoops only produces
a dmesg-ramoops record on PANIC. That is exactly what the new cmdline
gives: softlockup_panic=1 -> full ring buffer (incl. the lockup stacks
that never reached the serial console) -> dmesg-ramoops -> panic=10 ->
warm reboot. Each boot, systemd-pstore (enabled, runs ~12.5s, before
the 13.8s wedge point) archives the previous panic to
/var/lib/systemd/pstore. Read results from there (or /sys/fs/pstore on
a boot that completes) after the test.

## Round 53 — ramoops region collided with U-Boot's runtime data at DRAM top; moved to 0x60000000

fb-log.txt (early crash, 1.31s): efi_call_rts oops — "Unable to handle
kernel paging request at 0xbff29f30", x0=0xbff29ee0. 0xbff2xxxx is
U-Boot's EFI runtime services data: U-Boot relocates to the TOP of DRAM
(0xBFF00000..0xC0000000 for 2 GiB), exactly where I placed the ramoops
no-map region. The no-map carve-out removed those pages from the
kernel's linear map; the first EFI runtime call after boot (efi_rts_wq,
rtc-efi probe) dereferenced U-Boot's runtime data and the whole runtime
services path died with it. Boot never reached the wedge test.

Fix: ramoops moved to 0x60000000 (mid-DRAM; clear of the 0x50000000
shared-dma-pool at 0x50000000-0x52900000, the low kernel image, and the
top-of-RAM U-Boot runtime area). Both /boot/dtb-krane-ramoops.dtb and
/boot/dtb-7.0.0-30-generic rebuilt. Everything else (cmdline, initrd,
grub entries) unchanged. Lesson: never reserve anything at the top of
DRAM on this platform — that is U-Boot's relocation + EFI runtime +
variable-store area.

Round 53 addendum: U-Boot payload 7ed2de1e0ab3ff80cb6f461889d06bbf3d9d773f555308d02107dcb124cea92b
(quiesce + DSI INTEN/INTSTA clearing, commits d3d3502de0a + c112424b983)
flashed to /dev/mmcblk0p1, cmp-verified.

## Round 54 — no-watchdog silent lock both boots; new prime suspect: mt7663s wifi fw download deadlocking mtk-sd/eMMC I/O; hung_task_panic wired

Two boots (default + EXP-B cpuidle.off=1) both lock at the same point:
last kernel line = sbs uevent at ~12.8/13.2s, services continue to
ModemManager start, then TOTAL silence — no softlockup, no RCU stall,
no panic. A silent (sleeping) deadlock, not a spin: cpuidle is
exonerated, and timer_migration=off turned out to be an UNKNOWN param
on 7.0 ("will be passed to user space") so it never applied anyway.

New leading theory: NetworkManager brings wlan0 up right there ->
mt76 mt7663s firmware download over the SDIO link that shows CRC
errors from boot (msdc cmd52 host->error=0x2) -> mtk-sd driver wedges
-> eMMC I/O hangs (grub2-common/grub-initrd-fallback grubenv writes on
eMMC FAIL in every wedged boot!) -> system sleeps forever. Round 52's
spinning kworker/2:3 = mt76 fw download busy-wait; the current
silent shape = same trigger, deeper sleep. Not yet proven.

Prepared:
- hung_task_panic=1 added to all Ubuntu entries (CONFIG_DETECT_HUNG_TASK
  + HUNG_TASK_BLOCKER are on): a 120s-stuck D-state task now panics with
  full stacks AND the blocker name into the ring buffer -> ramoops
  dmesg-ramoops -> auto-reboot -> systemd-pstore archives it.
- EXP-C entry: module_blacklist=mt76,mt76_sdio,mt7663s,
  mt7663_usb_sdio,mt76_connac_lib,mt7615_common (wifi off) + same
  panic params. If EXP-C boots past the wedge point, wifi/SDIO is
  the trigger.
- pmOS bisect entry fixed: the pmOS system moved to the USB drive
  (sda1 kernel FIT, sda2 /boot, sda3 root); the running kernel is
  6.12.87-mt81, gzipped Image decompressed and staged as
  /boot/vmlinuz-pmos-6.12.87 (PE/EFI stub verified) + initramfs +
  dtb on eMMC, entry boots it via U-Boot.
- NOTE: our shell session is a chroot into eMMC p3; the real running
  pmOS boots from the USB drive (pmos_root_uuid=ddc5b150 = sda3).

## Round 55 — EXP-C panicked with FULL STACKS: the wedge is CPUs going dead to IPIs at coldplug settle

EXP-C (wifi blacklisted) locked like the others, but this time the
softlockup detector FIRED and we finally have stacks:
- watchdog: CPU#3 soft lockup 26s, udev-worker (PID 257), stack:
  smp_call_function_many_cond <- kick_all_cpus_sync <-
  flush_module_icache <- load_module <- finit_module. A udev module
  load broadcast an IPI and got no answer for 26s.
- panic path: "SMP: failed to stop secondary CPUs 0-2,5-7" — SIX of
  eight CPUs were already unreachable when it panicked; only CPU3
  (the loader) and CPU4 responded to the stop IPI.
- Timeline: last normal log 8.69s (ccifreq deferral spam ending =
  coldplug settling); CPUs died in the ~8.7-10.3s window; report at
  36.3s. The dead CPUs never softlockup-report themselves (no
  watchdog ticks at all — deeper than an IRQs-off spin: no timer
  interrupts / IPIs reaching them).
- wifi-blacklist did NOT prevent the wedge -> mt76/SDIO is NOT the
  trigger. The wedge family across all boots = multi-CPU death at
  initrd coldplug settle; the visible symptom (spin vs silent sleep)
  depends on which task notices first.
- ramoops was broken all along (-22 "failed to locate DT
  /reserved-memory resource"): v7.0 of_device_alloc creates MEM
  resources only from `reg`; a root-level ramoops node with
  memory-region never gets one. FIXED: moved the node into
  /reserved-memory with compatible="ramoops" + reg (upstream exynos
  pattern), record/console/pmsg sizes inside the node. Recompiled
  /boot/dtb-krane-ramoops.dtb + /boot/dtb-7.0.0-30-generic. Next
  panic will be archived to /var/lib/systemd/pstore by systemd-pstore.
- New EXP-D entry: maxcpus=1 (if it boots fully, the death is in the
  per-CPU idle/PSCI/PM layer, not in drivers).
- Available next: pmOS-kernel-via-U-Boot bisect entry (env vs kernel
  split), EXP-B cpuidle.off=1 (already shown insufficient alone).

Interpretation candidate for the dead-CPU signature: CPUs stopped
servicing IPIs AND their own timer ticks — PSCI/ATF-level CPU state
(suspend that never returns) or clock/power gated out from under
running CPUs around sync_state/coldplug settle. No proof yet.

## Round 56 — pmOS-6.12-via-U-Boot bisect made actually runnable without the USB drive

Constraint discovered: the USB-C port is shared between the serial
cable and the pmOS USB drive — both cannot be attached at once, so the
pmOS rootfs (sda3) is unavailable for the bisect boot. Workaround:
the running pmOS rootfs IS reachable via /proc/1/root, so:
- Copied /proc/1/root/lib/modules/6.12.87-mt81 (18 MiB) to
  /lib/modules/ on the eMMC Ubuntu root. In the pmOS kernel mtk-sd,
  mtk-smi and mediatek-drm are BUILT-IN (its initramfs has only 24
  modules), so the pmOS initramfs can mount eMMC p3 with no modules;
  wifi (mt7663s) etc. load from the copied tree at full coldplug.
- Rewrote the 'pmOS 6.12 kernel via U-Boot (wedge bisect)' grub entry:
  /vmlinuz-pmos-6.12.87 (decompressed Image, PE/EFI stub verified)
  + /initramfs-pmos-6.12.87 + /dtb-pmos-krane.dtb, all on eMMC,
  with pmos_root_uuid=e362f850 (Ubuntu eMMC root) so the FULL coldplug
  window runs under the 6.12 kernel + Ubuntu userspace + U-Boot
  handoff. Dropped pmos_boot_uuid (this initramfs was rebuilt for the
  USB layout; FAT ESP mount could stall it). Added
  softlockup_panic=1 hung_task_panic=1 panic=10 so a 6.12 wedge
  panics with stacks on serial.
- Interpretation: bisect boots fine -> 7.0 kernel bug. Bisect wedges
  the same way -> U-Boot handoff / ATF / DTB environment issue.
- NOTE: the bisect boots UBUNTU userspace under a pmOS kernel — it is
  NOT the real pmOS; do not confuse the two after boot. depmod of the
  copied tree was done on pmOS originally; modules.dep present (935).

## Round 57 — EXP-D (maxcpus=1) BOOTS FULLY: wedge requires SMP; pmOS bisect entry had /boot path bug (fixed)

- EXP-D maxcpus=1 boots through coldplug to serial login (panel still
  black — display issue is separate). Wedge does not occur with one
  CPU. Combined with EXP-B (cpuidle.off=1 wedged): the death is tied
  to multi-CPU bring-up/coupling, not to the idle framework itself.
- Unexplained: sudo hang at the EXP-D login prompt (no kernel output,
  log ends at "[sudo: authenticate]"). Ask user to retry and WAIT
  >=2-3 min: hung_task_panic should panic with the D-state stack +
  blocker on serial (single CPU means nobody reports a CPU0 death,
  but a sleeping task is still catchable).
- ramoops DID NOT register in the EXP-D boot (no probe message at
  all; module ramoops.ko.zst IS in the initrd; v7.0 has OF match
  table + reserved_mem_matches entry so the /reserved-memory node
  should get a device). Unresolved — have user check
  `ls /sys/fs/pstore` and `modprobe -v ramoops; dmesg|grep -i ramoops`
  on the next EXP-D boot.
- pmOS-6.12-via-U-Boot entry: grub spammed file-not-found then fell
  through — ROOT CAUSE: my rewritten entry used root-level paths
  (/vmlinuz-pmos-...) but on eMMC the files are in /boot/. Fixed:
  /boot/vmlinuz-pmos-6.12.87, /boot/initramfs-pmos-6.12.87,
  /boot/dtb-pmos-krane.dtb.
- Added EXP-E maxcpus=4 (big A73 cluster only, no LITTLE cpus):
  discriminates LITTLE-cluster involvement (cpufreq policy4, CCI,
  cpus 4-7) from big-cluster SMP.
- Current best theory family: something in the multi-CPU bring-up/
  cluster-coupling path (cpufreq/CCI/SVS/power) kills CPUs dead to
  IPIs at coldplug settle on 7.0; absent with maxcpus=1; not cpuidle
  (EXP-B); not wifi (EXP-C); not ramoops region (existed in wedging
  boots only since round 53, wedge predates it).

## Round 58 — pmOS 6.12 via U-Boot DIES too (env confirmed!); EXP-E hung later; cleanup-trio suspicion

- pmOS 6.12 via U-Boot (all 8 CPUs, own DTB, eMMC root): all 8 CPUs
  boot (0.088s), eMMC enumerates (HS400 2.28s), mediatek-drm binds,
  fb1 created — then SILENT at ~3.02s: last prints clk banner (2.994)
  / genpd banner (3.002) / ALSA list (3.017); "Freeing unused kernel
  image (initmem) memory" never printed. THE SAME KERNEL BOOTS FINE
  VIA DEPTHCHARGE. => U-Boot handoff is a necessary condition.
  Environment, not (only) 7.0 kernel.
- Window analysis: death sits in the late_initcall_sync tail, right
  where clk_disable_unused -> genpd_poweroff_unused ->
  regulator_init_complete run. regulator_init_complete silently
  force-disables boot-on-but-unclaimed regulators — U-Boot (display
  bring-up) leaves regulators/clocks/domains ON that depthcharge
  does not; the kernel then tears down something CPUs depend on.
  CPUs 1-7 die, CPU0 freezes shortly after (no free_initmem print).
  Unifying with 7.0: EXP-C full-SMP "CPUs dead to IPIs" and the
  ~10s module-load IPI spin = same teardown, different notice time;
  maxcpus=1 survives (nothing to tear down under other CPUs);
  EXP-E (maxcpus=4) PASSED the 2.49s cleanup (Freeing initmem +
  Run /init seen) and hung later at NM/ModemManager (~15s, wifi not
  blacklisted — mt76 fw download is back as a candidate for THAT
  hang, possibly a second, separate deadlock).
- ALSO: U-Boot hands off at EL2 ("All CPU(s) started at EL2"),
  depthcharge at EL1 — another handoff delta to keep in mind.
- Prepared: pmOS bisect entry now has initcall_debug +
  clk_ignore_unused + pd_ignore_unused + regulator_ignore_unused.
  Boot it: if it reaches login, the teardown trio is the killer and
  we bisect which of the three; the initcall_debug tail pins the
  exact hung function if it still dies.
- Note: EXP-D ramoops still silent — check `ls /sys/fs/pstore` +
  `modprobe -v ramoops` on a working boot sometime.

## Round 59 — ignore-params FIXED the 3s death; two separate bugs now cleanly separated

pmOS 6.12 via U-Boot with clk_ignore_unused + pd_ignore_unused +
regulator_ignore_unused: SAILS through the 3s teardown death into
full userspace (systemd starting Ubuntu services at 15s), then hangs
at ~16.1s at NetworkManager/ModemManager start WITH wifi active —
the classic point. initcall_debug lines did not appear on serial
(KERN_DEBUG vs console loglevel mystery — unresolved, moot now).

Bug matrix across experiments (wifi = mt76 bring-up):
- teardown death (clk/genpd/regulator cleanup under U-Boot handoff,
  CPUs 1-7 killed, CPU0 freezes): pmOS 6.12 @3s pre-params. Fixed by
  the three ignore params.
- Bug B (mt76 fw download deadlocks with >1 CPU): EXP-B (cpuidle.off,
  wifi on) ~14s; EXP-E (maxcpus=4, wifi on) ~15s; this boot (8 CPUs,
  wifi on) ~16.1s. Absent with wifi blacklisted (EXP-C reached the
  OTHER bug at 9-10s) and with 1 CPU (EXP-D wifi came up fine).
- EXP-C (8 CPUs, wifi off): died 9-10s = first all-8-idle window
  => deep idle/domain-sleep death on 7.0 (cpuidle.off=1 should fix).
So: Bug A' on 7.0 = deep idle (cluster/domain sleep) kills CPUs;
Bug A on 6.12 = the teardown kills CPUs (only seen on clang-built
pmOS kernel — Ubuntu gcc 7.0 passed teardown in EXP-D/E).
maxcpus=1 avoids both (no domain idle states, no other CPUs).

New entries (this round):
- EXP-F (7.0): cpuidle.off=1 + mt76 module_blacklist, full 8 CPUs.
  If it boots fully -> both bugs confirmed, working 8-CPU system.
- pmOS bisect entry: same three ignore params + cpuidle.off=1 +
  mt76 blacklist. If it boots fully -> 6.12 also working via U-Boot.
Next after confirmation: live root-causing on the working system
(disable cpuidle states one by one via sysfs to find the killer
state; bisect mt76 with 8 CPUs), plus decide the real fix (DTB
always-on marks? U-Boot handoff cleanup? mt76 fix?).

## Round 60 — second full panic nails the shape: individual CPUs die silently in hardirq context

EXP-F reboot: same panic shape as EXP-C — udev module load spinning
in kick_all_cpus_sync (CPU#5, started ~10.3s), but this time
"SMP: failed to stop secondary CPUs 4,7": only cpus 4 and 7 were
dead; 0-3,5,6 answered. Cp 4,7 = LITTLE cluster members, but 5,6
(same cluster) alive => NOT a cluster-wide clock/regulator kill.
Individual random CPUs go silent at ~8-11s (initrd coldplug window).

Interpretation: CPUs stuck in HARDIRQ context (explains: no IPI
service, no timer ticks, no self softlockup report, no panic; a
spinning hardirq handler never returns so hrtimers never fire).
Trigger candidate: an IRQ handler with an unbounded wait loop
(mtk_dsi irq do{}while-DSI_BUSY, cmdq mailbox, cros-ec rpmsg/spi)
arming at coldplug under the U-Boot handoff hardware state.
Downstream effects now unified: module-load IPI spins -> panics;
mt76 fw-download work queued on a dead CPU's kworker -> the
NM/ModemManager-era hangs (EXP-E, pmOS round 58 boot); silent
freeze when no spinner reports.
maxcpus=1 survival remains consistent (no IRQ spreading).

Prepared EXP-G: EXP-F + irqaffinity=0 -> all external IRQs on CPU0;
if a handler spins, CPU0 dies first/visibly. Ask user for sysrq
(BREAK + w/t) during any wedge: 'l' backtrace of all CPUs would show
the stuck hardirq handler directly.

## Round 61 — irqaffinity=0 did NOT protect: 7 of 8 CPUs died (0-2,4-7); cpuidle confirmed OFF; cascade model

EXP-G (EXP-F + irqaffinity=0): CPU#3 spun in kick_all_cpus_sync
(module load, started ~14.3s, further than before — real-root modules
loading), "SMP: failed to stop secondary CPUs 0-2,4-7": SEVEN CPUs
dead including CPU0 — but with all device IRQs pinned to CPU0 a
spinning device handler would have killed only CPU0. =>
- cpuidle.off=1 IS effective ("failed to register cpuidle driver",
  "CPUidle PSCI: Failed to create psci-cpuidle device") — no PSCI
  suspend path exists in these boots at all.
- Simple device-IRQ-storm-as-primary is dead too (CPU0 died anyway).
Working model now: PRIMARY = CPU(s) stuck in a hardirq handler
(any CPU; can hit several — EXP-F had 4,7); CASCADE = a stop_machine
(jump-label/text patch during module probes) parks every other CPU's
stopper thread in multi_cpu_stop with IRQs masked, waiting forever
for the stuck one -> whole-machine silent death; the innocent
module-load CPU then spins in kick_all_cpus_sync and softlockups.
Consistent with 7-dead (EXP-G) and 2-dead (EXP-F) variants.

picocom correction: C-a C-b = "set baudrate" (that was the prompt!).
Serial BREAK in picocom = C-a C-j (pulse BREAK), then the sysrq
letter (l = all-CPU backtrace, t = task dump) quickly after.

Prepared EXP-H: EXP-F + threadirqs -> handlers run as kernel threads;
a spinning handler becomes schedulable and the softlockup/hung-task
detector NAMES it (stack + handler identity) instead of silently
killing CPUs. This is the experiment that should finally reveal the
killer function.

## Round 62 — EXP-H (threadirqs): same crash, new victim; pseudo-NMI prepared as the stack-revealing tool

EXP-H: CPU#2 kworker/2:2 stuck 26s in smp_call_function_single <-
rcu_barrier <- fqdir_free_fn (netns frag teardown work — another
ALL-CPU barrier wait, not the cause). "failed to stop 0-1,3-7" — 7
CPUs dead again. threadirqs didn't change the class => the stuck
CPUs are NOT in a plain device-IRQ handler (those would have become
visible as threaded tasks). All panics share: reporting CPU waits in
an smp_call/rcu_barrier on other CPUs that never service IPIs.
Death window ~2.5-14s (coldplug storm), every full-SMP U-Boot boot,
both kernels. maxcpus=1 immune. cpuidle confirmed off. irqaffinity=0
confirmed ineffective (CPU0 died). Display/iommu never bind on 7.0
under U-Boot (deferred), so the DSI-IRQ-loop theory is weakened.
Ubuntu 7.0 has CONFIG_ARM64_PSEUDO_NMI=y but disabled by default
("watchdog: NMI not fully supported").
Prepared EXP-I: + irqchip.gicv3_pseudo_nmi=1 nmi_watchdog=1 ->
hard lockup detector becomes live; CPUs stuck with IRQs masked
(multi_cpu_stop, hardirq, anything) will SELF-REPORT their stacks
via pseudo-NMI on serial. This should finally show where the dead
CPUs are. Removed softlockup_panic from EXP-I so hardlockup reports
print repeatedly instead of one soft-lockup panic cutting the dump
short. sysrq during the ~26s wedge window also works: picocom
C-a C-j (pulse BREAK) then 'l' (all-CPU backtrace).

## Round 63 — EXP-I null test (quirk), ftrace dump-on-panic prepared

EXP-I: pseudo-NMI refused by an UPSTREAM QUIRK: the krane DTB's GIC
node carries "mediatek,broken-save-restore-fw" ("broken MediaTek
firmware that doesn't properly save and restore GIC priorities") and
cpufeature.c disables pseudo-NMI on it — printed at 0.000000.
NOT our wedge cause (pmOS idles/suspends constantly on the same DT
without pseudo-NMI and never dies; the breakage only matters for
priority-programmed NMI). No stacks obtained.

Prepared EXP-J: ftrace=function + ftrace_dump_on_oops (both =y in
Ubuntu kernel). Function tracing records every CPU's executed
functions into per-CPU ring buffers; at the softlockup panic the
kernel dumps ALL CPUs' buffers to serial — INCLUDING the frozen
CPUs' last executed functions before they died. This should name
the code the dead CPUs were running, no timing luck needed.
Boot is slower (function tracing on); panic dump is LARGE (serial
@115200 — let it run, could take minutes; do not interrupt).

## Round 64 — EXP-J dump partially captured: only CPU 6 (alive); logfile capture next

ftrace dump-on-panic WORKS (trace lines after "SMP: stopping secondary
CPUs"). User's terminal-buffer paste contained ONLY CPU 6's section
("6" in "6d.h3." = CPU 6 hex): alive & normal (timer, mmc, idle)
through trace-ts 75720287-75726786us (~26s window before panic).
Dump order = CPU 0 first -> dead CPUs' sections (0-5,7) were the
EARLIEST output, lost to terminal scrollback while surviving CPUs
kept dumping for 5+ min at 115200.
This wedge: CPU#3 rcu_exp_gp_kthr stuck 26s (started ~78s, after
login) — later + less deterministic than coldplug wedges (ftrace
overhead shifts timing); same all-CPU-IPI-dead mechanism.
NEXT: rerun EXP-J with picocom --logfile /tmp/fb-full.log (or
| tee). Dead CPUs' final functions = first sections of dump.

## Round 65 — EXP-J full analysis: CPUs die ENTERING WFI idle; nohlt test prepared

Full ftrace dump (131k lines captured, timestamp-merged across CPUs,
covering trace-ts 34.82-34.925s — ~100ms before the mass freeze):
Panic 60.7s CPU#1 rcu_exp_gp_kthr waiting on CPU6. Dead: 0,4-7.
CONFIRMED DEATH POINTS (last trace event before silence):
- CPU4: 34.880s check_and_switch_context <-__schedule (entering idle)
- CPU7: 34.924s timer_base_try_to_set_idle <-tick_nohz_stop_tick
  (programming wake timer, entering NOHZ idle)
- CPU5, CPU6: still running at capture cutoff (death later, uncaptured)
CPUs 0-3 sections were beyond the cutoff. NOT killed mid-execution:
cores go into WFI idle and NEVER WAKE — no trace, no IPI response
(failed-to-stop SGIs), no timer wake, no watchdog. The GIC stops
delivering to a WFI'd core.
WHY cpuidle.off=1 DOESN'T HELP: it only removes the cpuidle
framework; default idle is still cpu_do_idle (WFI).
KEY HANDOFF DELTA (arm_arch_timer.c arch_timer_select_ppi):
- depthcharge: EL1 entry -> hyp unavailable -> VIRT timer (CNTV)
- U-Boot: EL2 entry -> hyp available -> PHYS NONSECURE timer (CNTP)
  (log: "cp15 timer running at 13.00MHz (phys)") — wake PPI path
  never exercised by depthcharge-launched kernels on this board.
Firmware context: DT GIC node carries mediatek,broken-save-restore-fw
(upstream quirk; kernel only uses it to disable pseudo-NMI) — known
broken firmware save/restore of GIC state on this SoC family.
858921 note: workaround (= Cortex-A73 counter read) active ONLY on
CPUs 4-7; dead sets always include big cores but also little cores
(CPU0 has no workaround and dies too) — 858921 not the trigger.
maxcpus=1/4 survive: big cluster offline AND fewer idle cores.
NEXT: EXP-K = + nohlt (cpu_idle_force_poll=1; do_idle busy-polls,
NEVER executes WFI). If boot reaches userspace with 8 CPUs and no
wedge => WFI confirmed as trigger. Then: make U-Boot hand off at
EL1 (kernel would pick CNTV like depthcharge) or find GIC/SPM wake
fix. nohlt is power-hungry — diagnostic/permanent stopgap only.

## Round 66 — EXP-K: BUG A CONFIRMED (WFI trigger); silent hang at network.target = Bug C

EXP-K (nohlt = cpu_idle_force_poll=1, CPUs busy-poll in idle, never
WFI): the ~8-14s coldplug wedge DID NOT HAPPEN — boot sailed through
to full userspace: NM up, network.target reached (~20s+), ccifreq
spam ended normally ~8-9s. => CPUs die while EXECUTING WFI. Bug A
root cause class: core enters WFI and its GIC redistributor/timer
wake never fires again (U-Boot EL2 handoff -> kernel uses PHYS
nonsecure timer PPI as wake source; depthcharge EL1 -> CNTV virt;
broken-save-restore-fw firmware context).
REMAINING silent hang: log stops right after "Reached target
network.target" (no systemd-user-sessions line). Same point as
EXP-F's silent hang (13.5s) and pmOS-6.12-via-U-Boot 16.1s —
BUT mt76 is blacklisted in EXP-K => NOT Bug B (wifi-independent).
Called it Bug C: SMP-dependent silent deadlock after NM start;
silent because hung_task_panic was NOT set (softlockup needs a
spinning CPU; this is a blocked/deadlock state) and sysrq BREAK+l
on ttyS0 got no response in that state (serial IRQ possibly dead
too, or full freeze).
EXP-K2 prepared: same + hung_task_panic=1 hung_task_timeout_secs=10
-> 10s after a task hangs, panic prints ALL CPU stacks = names the
deadlock. Boot EXP-K2 next; when it stops, WAIT ~15s for the
auto-panic dump (no sysrq needed).

## Round 67 — VFS root panic was a grub entry mistake (mine), fixed

Both reboot attempts: "Cannot open root device ... unknown-block(0,0),
available partitions: (EMPTY)" + prepare_namespace in the panic
trace = kernel got NO initramfs (prepare_namespace never runs when
an initrd is present). Without initrd there are no modules -> no
mtk_sd -> no block devices. EFI banner in both boots lacks the
INITRD=0x... line. Cause: my EXP-K edit accidentally replaced the
entry's initrd line with a second linux line and then deleted the
original linux line -> entry booted with NO initrd at all. Fixed:
initrd line restored, grub regenerated and verified (entry now has
linux+initrd+devicetree). NOT a kernel regression. Also: the entry
label stayed "EXP-K" (I edited in place; "EXP-K2" never existed as
a label — the hung_task params were active in both boots).
Next boot: EXP-K again (nohlt + hung_task_panic) — when output
stops, wait ~15s for the auto hung-task panic dump.

## Round 68 — Bug C named: QCA Bluetooth firmware download (hci_uart); bt blacklist test ready

eMMC journal (hostname duet) preserved the EXP-K death context that
serial never showed: ath10k_sdio wifi loaded, ModemManager +
wpa_supplicant started, then:
  Bluetooth: hci0: QCA Downloading qca/rampatch_00440302.bin
  kernel: ------------[ cut here ]------------   <- freeze, no more
Bug C = firmware-download deadlock, same class as Bug B (mt76 SDIO
fw download at NM time) but via QCA BT UART. With wifi blacklisted,
the BT path (hci_uart) hits the equivalent bug at user-sessions
time. Re-explains EXP-E hang and pmOS 16.1s hang.
Also: hung_task_timeout_secs=10 is NOT a valid boot param (only
sysctl kernel.hung_task_timeout_secs; default 120s in Ubuntu) —
hung_task_panic WAS active but needs 120s to fire; reboots were
too early. Boot param valid: hung_task_panic only.
NEXT: EXP-K entry now blacklists hci_uart,btqca,bluetooth on top of
mt76 + nohlt. If it reaches serial login with 8 CPUs => both
remaining boot bugs are firmware-download deadlocks under U-Boot
handoff (wifi SDIO + BT UART). Then: bisect WHICH SMP interaction
breaks fw download (candidates: SDIO/UART DMA + per-CPU IRQ wake
marginality, GIC-to-SPM wake path, or sg_table/DMA vs IOMMU-off).

## Round 69 — BT blacklist did NOT fix the user-space hang; sysctl dump prepared

EXP-K with hci_uart/btqca/bluetooth blacklisted: same silent hang,
stops around NM/hostnamed/ModemManager start (slightly earlier than
the network.target stop of the previous EXP-K boot — placement
varies). So Bug C is NOT (only) the QCA BT download — the journal's
cut-here during rampatch download was likely collateral, not the
cause. Firmware-download-class theory: NOT yet confirmed for C.
hung_task_timeout_secs boot param is invalid (only sysctl exists;
default 120s — user reboots too early to ever see the dump).
Prepared: /etc/sysctl.d/99-krane-hungtask.conf
(kernel.hung_task_timeout_secs=10, hung_task_all_cpu_backtrace=1,
hung_task_warnings=100) — applies to every boot of this rootfs;
10s after a task hangs, panic prints ALL CPUs' stacks and names
the blocker (v7.0 has debug_show_blocker = mutex owner).
NEXT: reboot EXP-K (nohlt, wifi+bt blacklisted). At the hang WAIT
AT LEAST 3 MINUTES. The panic dump is the deliverable.

## Round 70 — WFI theory DEAD; new unified suspect: mtk-cci-devfreq first rate switch

EXP-K (nohlt) definitive panic: CPU#2 kworker/2:0 stuck 22s in
rcu_barrier <- fqdir_free_fn, "failed to stop secondary CPUs
0-1,3-7" — ALL SEVEN OTHER CPUS DEAD WITH NO WFI EVER EXECUTED
(deaths at ~14.3s). The "die at idle entry" ftrace reading was an
artifact: the round-64 capture ended 3s BEFORE the deaths (boot 1
panicked at 104.7 = death ~78s; round-65 capture = boot 2, death
~34.6s). CPUs freeze mid-execution, idle mode irrelevant.
CORRECTION of the record: round 65's "CPU4/7 died entering idle"
was premature — last-traced-event != death point for CPUs 5/6/0-3.
NEW UNIFIED SUSPECT: drivers/devfreq/mtk-cci-devfreq.c probe:
each deferred retry raises VPROC to the highest CCI OPP voltage
(mtk_ccifreq_set_voltage BEFORE devfreq registration!), then
devm_devfreq_add_device defers -517 on CPUFREQ_PARENT_DEV
(mtk-cpufreq module) = the x71 spam. When mtk-cpufreq finally
registers (real-root module storm ~9-15s), ccifreq attaches and
the passive governor immediately syncs CCI rate+voltage on the
live system. A wedged CCI/PLL/VPROC switch freezes ALL cores
mid-instruction (shared resource) — matches every panic signature.
Spam-end -> deaths correlation holds in every boot (8.7s spam end;
ftrace boot spam 34.6, deaths 34.88+; nohlt boot spam 9.5, deaths
14.3). maxcpus=1/4 survive = big-cluster policy/transition absent.
Bug A and Bug C are probably THE SAME BUG.
NEXT: EXP-K now also blacklists mtk_cci_devfreq + mtk_svs (cpufreq
still allowed). If the boot reaches serial login with 8 CPUs =>
CCI DVFS stack = killer. Then bisect: cpufreq vs ccifreq, and WHY
the switch wedges under U-Boot (clock state left by payload?).

## Round 71 — CCI/SVS exonerated (still dies, NEW victim set {3,5,6,7}); cpufreq is the last DVFS suspect

EXP-K + mtk_cci_devfreq/mtk_svs blacklisted: STILL dies at ~14.3s
(same rcu_barrier/fqdir_free_fn victim class) BUT the dead set
changed for the first time: 3,5-7 — big cores 5,6,7 + little core
3, with CPU4 ALIVE (first time ever with 8 CPUs online). CCI/SVS
exonerated as the trigger.
Remaining DVFS piece in the 14s module storm: mediatek-cpufreq
(module mediatek-cpufreq.ko; big-cluster policy init = switch to
intermediate clock, reprogram main PLL, change shared VPROC/VSRAM
rails). Explains: deaths spanning both clusters (shared rail),
survivor variation, maxcpus=1/4 immunity (big policy never inits —
EXP-E with 4 CPUs survived cpufreq and hit mt76 Bug B at NM
instead), depthcharge immunity (clock tree left in expected state;
Jul 28 journal: "CPU4: Running at unlisted initial frequency:
1199999 KHz, changing to 1248000" — U-Boot may leave a
non-OPP-listed rate -> fatal big PLL jump). No cpufreq messages at
all appear on serial in U-Boot boots before the freeze.
NEXT: EXP-K + mediatek-cpufreq blacklisted. Serial login with 8
CPUs => big-cluster cpufreq policy init = the killer under U-Boot
handoff. Then compare with U-Boot's leftover MPU rate (bootefi
"CPU: ..." or /proc/cpuinfo) and design the real fix (U-Boot clock
cutover or cpufreq driver quirk).

## Round 72 — cpufreq EXONERATED too; round-55 signature; EXP-L definitive trace prepared

EXP-K + mediatek-cpufreq blacklisted: STILL dies — and the panic is
the round-55 signature EXACTLY: udev-worker CPU#1 stuck 23s in
smp_call_function_many_cond <- kick_all_cpus_sync <-
flush_module_icache <- load_module <- finit_module. Dead set {4,5,6}
(3 big cores; CPU7 alive for the first time). DVFS fully exonerated
(cci, svs, cpufreq all blacklisted — still dies). nohlt also ruled
out (round 70: dies without any WFI). The module-load IPIs reveal
the freeze, they don't cause it; several modules were mid-load in
parallel (cros_ec_keyb(+), hid_multitouch(+), hid_google_hammer(+),
extcon(+), cros_ec_dev(+)).
Invariants now: deaths at ~9-15s in the udev storm; victim sets
vary but always include big cores; U-Boot handoff required;
depthcharge immune; maxcpus=1/4 immune; idle mode irrelevant;
DVFS irrelevant; irqaffinity/threadirqs irrelevant.
EXP-L prepared: round-63 config EXACTLY (cpuidle.off + mt76
blacklist + ftrace=function + ftrace_dump_on_oops, NO nohlt) +
trace_buf_size=128 (bounds each CPU ring -> dump ~30k lines,
completes in minutes; frozen CPUs' tails = their true death points;
normal WFI idle so idle CPUs still trace their entry).
Boot with picocom --logfile; capture EVERYTHING until reboot.