# Screen mirror

Watch the glass of a Kode Dot live in a browser and drive it with clicks: the frame stream, the credit window, the two transports and the viewer.

## What it is

The mirror sends the picture of the screen to a computer, live, and lets the computer send touches and button presses back. You watch and control a Kode Dot that is not in front of you.

It turns the shadow copy that AO_Display already keeps for `screenshot` into a stream. The shadow is a copy of the panel in PSRAM. The stream is JPEG frames of whatever changed, each sent as soon as it exists. A still screen sends nothing: when the changed band is empty, no frame leaves and no byte is spent. The mirror exists only because `KODEOS_SCREENSHOT` keeps the shadow, and `KODEOS_MIRROR` (default on) depends on it.

Input needs no special path. The verbs `touch`, `tap`, `swipe` and `btn` publish the same events AO_Touch and AO_Buttons publish, so no receiver can tell an injected sample from a finger.

## How it is connected

```mermaid
flowchart LR
  subgraph board["On the Kode Dot: AO_Display, every 10 ms"]
    shadow["shadow band"] --> ppa["kdrv_ppa"]
    ppa --> jpeg["kdrv_jpeg"]
    jpeg --> frame["frame buffer"]
    frame -- "slices" --> wire["console or<br/>BLE ring"]
  end
  subgraph computer["On the computer"]
    viewer["kmirror.py"]
  end
  wire --> viewer
  viewer -- "mirror_ack, tap, btn" --> con["AO_Console"]
  class shadow,frame,wire store
  class viewer ext
  class con ao
```

*One mirror frame*

**AO_Display owns the pipeline.** It claims `KBSP_RES_JPEG` and `KBSP_RES_PPA` only when the build has the mirror, and runs one pass per tick of `DISPLAY_MIRROR_TICK_SIG` every 10 ms (`KMIRROR_TICK_MS`). AO_Console only parses the `mirror_*` verbs and passes each on as an event. The codec and scaler are documented on `sdk/kdrv/mirror`.

A pass has five stages:

1. **Shadow band.** `kdrv_display_dirty_take()` returns the rows that changed since the last call. The mirror keeps its own accumulator (`s_mirror_y1` and `s_mirror_y2` in `kdrv_display.c`), widened by every flush, so a mirror slower than the panel does not miss frames. A band the pass cannot use is kept in `pend_y1` and `pend_y2` for the next tick.
2. **Scale.** `kmirror_band()` turns the rows into a scaled rectangle inside the panel. It uses the first and last row, since `ceil((y2-y1)/scale)` counts too few rows for a band that does not start on a scale boundary.
3. **PPA.** The shadow is stored bottom-up, like a BMP, so every band goes through the pixel processing accelerator, even at scale 1, to flip and scale in one hardware pass. Its output buffer is one full panel of RGB565, rounded up to a whole number of cache lines.
4. **JPEG.** `kdrv_jpeg_encode()` takes the RGB565 output. A full 410 by 502 frame (402 KB) becomes about 19.7 KB at quality 70 in 3 ms, measured on a Kode Dot.
5. **Copy out, then slice.** The whole frame, a 14-byte header plus the JPEG, is copied into a PSRAM buffer before any record leaves, because the PPA and the codec keep their output pointer valid only until their next call. Each tick then sends up to 12 slices (`KMIRROR_SLICE`, the constant `help` and `screenshot` use), so one step stays short.

| File | What it holds |
| --- | --- |
| `kaos/display/src/display_ao.c` | The state machine: `mirror_pass()`, `mirror_try_start()`, `mirror_send_slice()`, `mirror_abort()`, `mirror_pause()` |
| `kaos/console/src/console_ao.c` | The `mirror_*`, `touch`, `tap`, `swipe` and `btn` verbs |
| `kaos/core/src/kaos_wire.h` | The base64 encoder and the slice sizes |
| `kdrv/mirror/include/kmirror_proto.h` | The header, the credit window and the band arithmetic. No hardware, host-tested |
| `kdrv/radio/src/kdrv_ble.c` | The binary characteristic and its ring |
| `tools/kmirror.py` | The reference viewer |

## Specifications

### The frame

Every frame is a header and `len` JPEG bytes. The stream is one continuous buffer: the next header follows the last JPEG byte. Fields are little-endian.

| Bytes | Field | Type | Meaning |
| --- | --- | --- | --- |
| 0 to 1 | `seq` | `uint16_t` | Wraps at 65535 |
| 2 to 3 | `x` | `uint16_t` | Left edge, already scaled |
| 4 to 5 | `y` | `uint16_t` | Top edge, already scaled |
| 6 to 7 | `w` | `uint16_t` | Width, already scaled |
| 8 to 9 | `h` | `uint16_t` | Height, already scaled |
| 10 to 13 | `len` | `uint32_t` | JPEG bytes that follow (`KMIRROR_HDR_LEN` is 14) |

Only the first frame after `mirror_start` or a pause covers the whole panel. Later frames are the rectangle that changed, so a viewer draws each part at its own offset on a canvas and cannot treat the stream as motion-JPEG.

### Verbs

| Verb | Arguments | Answer |
| --- | --- | --- |
| `mirror_start` | `scale` 1 or 2, default 2. `q` 1 to 100, default 70 | `>ok`, then the stream. The board lowers `q` above 70 to 70 (`KMIRROR_QUALITY_MAX`). Not over the Kode channel |
| `mirror_stop` | none | `>ok`. A frame in progress is cut with `mabort` |
| `mirror_ack` | `seq` 0 to 65535, required | `>ok`. Cumulative |
| `mirror_info` | none | `mirror paused scale q window in_flight sent stalls`, then `>ok` |

### Two transports

The board picks the transport once, when a frame begins, so a central that attaches or leaves mid-frame cannot split a frame across two.

| Transport | Format | Slice |
| --- | --- | --- |
| Cable, base, and the text pipe of the Bluetooth console | Base64 inside ordinary `m=` records, sent through `kode_log_reply()`. Decode only whole groups of 4 characters and never pad a short tail | 162 source bytes per 216-character record |
| Bluetooth binary characteristic, UUID index `0x0004` | The same header and raw JPEG bytes, no base64. Notify-only, with its own 8 KB ring, in PSRAM, and retry callout | 214 bytes (`KMIRROR_BLE_PAY_BYTES`) |

The 214 comes from the NimBLE mbuf: a 256-byte block spends 16 bytes on `struct os_mbuf`, 8 on the packet header and 18 on leading space for the ATT, L2CAP and HCI headers. A longer slice chains a second mbuf out of a pool of 24 and empties it in about a dozen notifications.

A 30 s stream with the screen changing continuously ran at these rates:

| Slice | Frames per second |
| --- | --- |
| 214 bytes | 3.23 |
| 162 bytes | 2.77 |
| 244 bytes | 1.40 |
| 250 bytes (the MTU limit) | 1.13 |

### Flow control

| Mechanism | Value |
| --- | --- |
| Credit window | 4 frames outstanding (`KMIRROR_WINDOW`). Memory is not the limit, since there is one frame buffer. The window bounds how far `seq` runs ahead of the host's acks. A host that never acks gets exactly four frames |
| Acknowledgement | `mirror_ack seq=` is cumulative, like TCP. The board refuses a `seq` that is old or was never sent |
| Slice pacing | 12 records per 10 ms tick, independent of credit. A frame already being sent needs no credit |
| Stall | 5 s (`KMIRROR_STALL_MS`) with no progress pauses the session |
| Frame limit | 14 bytes plus 32 KB (`KMIRROR_JPEG_CAP`). A larger frame is dropped, and the full-frame request is cleared so the same oversized encode does not repeat at 100 Hz |

A stall has three forms, and all of them feed one counter:

- No credit, because the host stopped acknowledging.
- No progress inside a frame, because the cable is full or the Bluetooth ring never empties.
- Nothing can be encoded, because the shadow, the PPA or the codec refuses the rows. `mirror_try_start()` reports this as `MIRROR_BLOCKED`.

An idle screen is `MIRROR_IDLE` and never feeds the counter.

A pause stops the tick timer and frees the PSRAM frame buffer, but the session stays open. `mirror_start` or any `mirror_ack` wakes it, resets the credit window and forces a full frame, since a host that missed frames has an out-of-date canvas. A session ends when the transport is gone, for example when a Bluetooth central disconnects mid-frame. A slow transport only pauses it.

On the cable, control and frames share one connection. A viewer that sends quick `btn` presses queues its `mirror_ack` behind them and the board thinks it stopped. A window of 4 gives slack and the pause keeps the session alive.

### Control records

| Record | Meaning |
| --- | --- |
| `mabort` | The board abandoned the frame in progress. The next `m=` record starts a new header. Sent only when at least one byte of the frame has left |
| `mpause` | The session sleeps. Send `mirror_start` to wake it |

Both have no `=`, so a parser tells them from data without decoding, and both travel on the console line on both transports. A marker in the binary stream would fail: an abort comes only mid-frame, so a header with `len=0` would land right behind a partial body and a decoder that follows `len` would read its 14 bytes as JPEG and misread every later frame.

### Measured

| Figure | Value |
| --- | --- |
| Hardware encode | 402 KB frame to about 19.7 KB at quality 70 in 3 ms. About 29 KB at quality 90 |
| Cable, scale 2, q 70, full-panel frame | 8,380 bytes in 81 ms, three trials at 81, 84 and 81 ms. About 12 fps of full-panel frames. The cost is about 142 bytes per millisecond plus 22 ms per frame |
| Bluetooth, tested with another slice size | 8,336 bytes in a median 2,477 ms, 20,074 bytes in 5,795 ms. About 3.5 bytes per millisecond plus 120 ms per frame, roughly forty times slower than the cable |
| Idle screen | Zero frames. A 30 s run on a still launcher over Bluetooth sent one frame with `tx_drop=0` and `log_dropped=0` |
| RAM | Without `KODEOS_MIRROR`, about 530 KB of PSRAM that would be held from boot is saved, and the codec, the scaler, the self-test `jpeg` and the characteristic go |

### The viewer, kmirror.py

`tools/kmirror.py` serves one HTTP page. The page draws each frame on a canvas at the offset in its header, turns clicks into `tap`, and has a pad of buttons that drives `btn`.

| Option | Meaning |
| --- | --- |
| `-p PORT` | The cable or base port. Omitted, the board is found automatically |
| `--ble` | Stream over Bluetooth with its own bridge. Builds a second signed app under the cache and asks once for Bluetooth permission |
| `--scale 1 or 2`, `--q 1-100` | Passed to `mirror_start`. Defaults 2 and 70 |
| `--host`, `--http-port` | The bind address, default `127.0.0.1`, and the port, default 8420 |
| `--open` | Opens the page in the default browser |
| `--steal` | Ends whoever holds the serial port (cable only) |
| `-v` | Shows board log lines while streaming |

The server answers:

- `GET /` for the page.
- `GET /stream` for the frames re-framed as header plus JPEG without base64.
- `POST /tap`, `/touch` and `/btn` with coordinates in canvas pixels, multiplied back by the scale. `/btn` takes `press` and `release`.

The viewer sends `mirror_ack` for each completed frame, answers `mpause` with `mirror_start`, and restarts the stream from its own side 2 s after an input with no picture back.

`MirrorDecoder` decodes the cable text and `BleFrameDecoder` the binary characteristic, and both share `control_line()` for the two control records. `tools/tests/test_kmirror.py` checks what the two sides share and needs no board.

## Gotchas

> **WARNING**
>
> **A viewer that never acknowledges is a screenshot.** It gets four frames and then nothing, until the 5 s limit pauses the session. Send `mirror_ack seq=` for every completed frame.

> **WARNING**
>
> **Never pad a short tail of the base64 stream.** Padding with `=` produces different bytes without an error, and a header decoded from them is wrong. Decode only whole groups of 4 characters and keep the 0 to 3 leftover for the next call (`decode_whole_groups()` in `tools/kpayload.py`).

> **NOTE**
>
> **`q=100` is valid on the wire, and the board caps it at 70.** A full panel at a higher quality does not fit the 32 KB buffer.

> **NOTE**
>
> **Only one Bluetooth central at a time.** `kmirror.py --ble` subscribes to the mirror characteristic itself, so a `kble.py` bridge must be stopped first (`pkill -f 'kble.*--hold'`). Opening the cable port resets the board, so the viewer waits for `ping` before it sends `mirror_start`. The charging base does not reset it.

## Example code

Watch and drive the glass from a browser.

1. Connect the Kode Dot with the USB-C cable. Opening the port restarts it.
2. Run `python3 tools/kmirror.py -p PORT --scale 1 --q 85 --open`.
3. The browser opens `http://127.0.0.1:8420` and shows the face. Click the picture: the Kode Dot receives a `tap` at that point.
4. Press the buttons on the page. The board receives `btn` presses, and the pad moves the selection of the launcher.
5. Leave the screen still. The stream sends nothing until something changes.
6. Press Ctrl-C in the terminal to stop the viewer.

## In the SDK

- [`components/kaos/display`](https://github.com/kodediy/kode-sdk/tree/main/components/kaos/display): `display_ao.c` holds the pipeline and the credit window.
- [`components/kdrv/mirror`](https://github.com/kodediy/kode-sdk/tree/main/components/kdrv/mirror): the JPEG encoder, the pixel accelerator and `kmirror_proto.h`. See `sdk/kdrv/mirror`.
- [`tools/kmirror.py`](https://github.com/kodediy/KodeOS/blob/main/tools/kmirror.py): the reference viewer.