# Mirror codec and protocol

The three files behind the live screen mirror of a Kode Dot: the P4's JPEG encoder, its pixel accelerator and the frame header with its credit window.

## What it is

The screen mirror streams the glass of a Kode Dot to a computer as a series of JPEG frames. The module `kdrv/mirror` holds the three files it is made of:

| File | Job |
| --- | --- |
| `kdrv_jpeg.c` | The ESP32-P4's hardware JPEG encoder: RGB565 in, JPEG out |
| `kdrv_ppa.c` | The P4's Pixel Processing Accelerator: halves a band of the screen in both directions in one hardware pass |
| `kmirror_proto.c` | The 14-byte frame header, the flow-control window and the arithmetic that turns changed rows into a rectangle. No hardware in it |

An app does not call them. AO_Display owns the codec and the scaler (`sdk/kaos/display`), and the viewer side of the stream is on `internals/mirror`.

The encoder is never the slow part. A full 410 by 502 frame is 402 KiB of RGB565. The encoder turns it into **19.7 KB at quality 70 in 3 ms**, measured on a Kode Dot, reading the pixels where they already are. The transport to the computer is always the bottleneck.

The three files are compiled only with `CONFIG_KODEOS_MIRROR`, which depends on `KODEOS_SCREENSHOT` and is on by default. Between them the codec and the scaler hold about 530 KB of PSRAM from boot, 128 KB of encoder output and one full panel of RGB565 for the scaler. A build that turns the mirror off does not pay it. The JPEG encoder also serves the `screenshot dst=http` upload, which encodes the whole glass once per picture.

## How it is connected

```mermaid
flowchart LR
  subgraph board["On the Kode Dot"]
    ao["AO_Display"] --> fb["front<br/>framebuffer"]
    fb -- "kdrv_display_band" --> ppa["kdrv_ppa"]
    fb -- "scale 1, in place" --> jpeg["kdrv_jpeg"]
    ppa --> jpeg
    jpeg --> hdr["kmirror_proto<br/>header"]
    cred["credit window"]
  end
  hdr --> wire["console wire<br/>or BLE"]
  wire --> host["viewer on<br/>the computer"]
  host -- "mirror_ack seq" --> cred
  class ao ao
  class fb store
  class host ext
```

*One mirror frame*

### The owner and one pass

**AO_Display owns the two peripherals.** It claims `KBSP_RES_JPEG` and `KBSP_RES_PPA`, and calls `kdrv_jpeg_init()` and `kdrv_ppa_init()` while it brings the panel up. Neither init is fatal. A board whose encoder or scaler will not allocate still has a panel and still boots, and the mirror reports that it is blocked.

Each pass of the mirror is one AO step in `mirror_try_start()`:

1. Take the rows that changed since the last frame from the display driver, or the whole panel for the first frame after a host attaches.
2. `kmirror_band()` turns that range into a scaled rectangle that stays inside the panel. An empty range is not an error: an idle screen produces no bytes.
3. Read the band from the front framebuffer. `kdrv_display_band()` returns it top-down, at full size.
4. At scale 2, `kdrv_ppa_prepare()` halves it into the scaler's own buffer. At scale 1 the band is encoded where it lies, since the encoder accepts any input alignment.
5. `kdrv_jpeg_encode()` compresses it. AO_Display clamps the requested quality to 70 (`KMIRROR_QUALITY_MAX`).
6. A frame larger than the 14-byte header plus 32 KB is dropped and logged.
7. The header and the JPEG are copied into the AO's own frame buffer in the **same step**, because both drivers hand back a pointer that is valid only until their next call. The frame then leaves in slices.

A refusal at step 3, 4 or 5 keeps the rows for the next pass. Five seconds without progress, or without credit from the host, log one warning and pause the stream (`KMIRROR_STALL_MS`).

### The try-lock

**One engine and one buffer, so a try-lock.** Each driver has one hardware engine and one output buffer. A second caller inside would read the buffer while the first still writes it. The guard is a mutex used only as a **try-lock**: `xSemaphoreTake()` with a zero timeout. The loser is refused with `KODE_DENIED` and not blocked, because `khal_mutex_t` offers only a blocking take, and blocking would put AO_Display (priority 15) behind someone else's job inside what must stay a short AO step.

The lock exists because `test component=jpeg` also calls the encoder, inline on AO_Autotest's task, which is not AO_Display's.

The two drivers differ in how long they hold it.

- `kdrv_jpeg_selftest()` keeps the lock across the encode **and** its read of the result. If it did not, AO_Display could take the lock for a live frame between the two and overwrite the buffer under it, and a healthy codec would report itself broken.
- `kdrv_ppa_prepare()` releases the lock before it sets its output pointer. Its only caller reads that pointer in the same AO step, and QP's run-to-completion rule keeps that safe.

### Alignment

**The scaler's buffer has two alignment rules.** `kdrv_ppa` allocates its output with `MALLOC_CAP_SPIRAM | MALLOC_CAP_CACHE_ALIGNED`, which makes the **address** line up with the caches. The PPA driver also checks the **size**: it must be a whole number of cache lines.

A full panel is 410 x 502 x 2 = 411,640 bytes, which is not a multiple of the chip's 64-byte line.

So `PPA_OUT_CAP` is written as arithmetic that rounds the panel size up to a whole number of the larger of the L1 and L2 line sizes. That gives 411,648 bytes here, and it follows a change of the panel or of the cache setting.

The encoder's output must also start on a cache line, so `kdrv_jpeg` allocates it with `jpeg_alloc_encoder_mem()` and not `heap_caps_malloc()`. ESP-IDF refuses any other output buffer. The input has no alignment rule.

### The protocol

**The protocol has no hardware.** `kmirror_proto.c` includes no ESP-IDF header and keeps no state of its own, so the host tests check all of it on a computer. It holds three things that are hard to get right and impossible to notice on a bench: a header's byte order, a sequence number across its wrap, and a band's edges after scaling. Each shows up as a fault only hours into a stream.

- **The header is serialised byte by byte.** Sending the struct as it sits in memory would put this compiler's padding and this CPU's byte order on the wire. A host test pins literal byte positions, not only the round trip.
- **The window counts with unsigned subtraction.** `in_flight` is `next - acked - 1` in 16 bits, correct across the wrap at 65535. A comparison like `next > acked` works for an afternoon, then stalls the stream with nothing in any log to say why: at 10 frames a second the sequence wraps every 109 minutes.
- **The ack is cumulative, like TCP's.** Acknowledging the newest frame retires every older one. `kmirror_ack()` refuses a sequence that is not in flight, so a stale or invented ack cannot open the window without bound. The refusal is silent on the wire: `mirror_ack` always answers `>ok`.
- **A band's row count depends on where it sits.** Source rows `[y1, y2)` land on destination rows `[y1/s, (y2-1)/s]`, and the code counts `bottom - top + 1`. A formula on the length alone, `ceil((y2-y1)/s)`, loses a row for a misaligned band. Rows 1 to 3 at half scale touch destination rows 0 and 1, two rows, and the length-only formula says one.

## Specifications

### The frame header

Every frame is a header followed by `len` bytes of JPEG. All numbers are little-endian. `x`, `y`, `w` and `h` count pixels of the **scaled** picture. At present the mirror always sends full-width bands, so `x` is 0.

| Offset | Size | Field | Meaning |
| --- | --- | --- | --- |
| 0 | 2 | `seq` | Frame number; wraps at 65535; `mirror_ack seq=` names it |
| 2 | 2 | `x` | Left edge of the rectangle |
| 4 | 2 | `y` | Top edge |
| 6 | 2 | `w` | Width |
| 8 | 2 | `h` | Height |
| 10 | 4 | `len` | JPEG bytes that follow the header |

`KMIRROR_HDR_LEN` is 14.

### The credit window

`kmirror_credit_t` is the sender's half of the flow control: `next` (the sequence of the next frame), `acked` (the last one the host confirmed) and `window` (frames that may be outstanding, a `uint8_t`). AO_Display starts it with a window of 4 (`KMIRROR_WINDOW`).

| Call | What it does |
| --- | --- |
| `kmirror_credit_init(c, window)` | `next` 0, `acked` `0xFFFF`, which reads as "nothing sent". A window of 0 becomes 1, since 0 would stall the stream for ever |
| `kmirror_may_send(c)` | True when fewer frames are in flight than the window allows |
| `kmirror_in_flight(c)` | Frames sent and not acknowledged. Public for `mirror_info` |
| `kmirror_sent(c)` | Takes the next sequence number and counts the frame as out. Returns 0 for a `NULL` `c`, since no value could signal the error |
| `kmirror_ack(c, seq)` | Records an ack. False when `seq` is not in flight: the distance from the last ack must be at least 1 and at most the frames in flight |

### The band

`kmirror_band(y1, y2, scale, v_res, &out_y, &out_h)` clamps `[y1, y2)` into the panel (`v_res` is 502 rows) and scales it. Scale 1 is full size, 2 is half. A scale of 0 becomes 1. `y1 >= y2` after clamping gives height 0 and is not an error. The result never runs past `v_res / scale`.

### The encoder and the scaler

| Thing | Value |
| --- | --- |
| Encoder input | RGB565, any alignment, any width and height (need not be a multiple of 16) |
| Sub-sampling | YUV420. The 19.7 KB figure is at this setting |
| Quality | 1 to 100 at the driver. AO_Display clamps the mirror to 70 (`KMIRROR_QUALITY_MAX`) |
| Encoder output buffer | 128 KB (`JPEG_OUT_CAP`), four times the 29 KB a full frame took at quality 90. Allocated by `jpeg_alloc_encoder_mem()` |
| Encoder deadline | 2000 ms (`JPEG_TIMEOUT_MS`). A full frame takes 3 ms, so past this the peripheral is stuck and not slow |
| Scaler operation | Scale, rotate and mirror in blocking mode; one client, one pending transaction |
| Scaler scale | 1 or 2 only. Scale 2 needs a source of at least 2 by 2 pixels. The result must fit the output buffer |
| Scaler output buffer | 411,648 bytes (`PPA_OUT_CAP`), in PSRAM, cache-line aligned in address and size |
| Scaler transform | None besides the scale: no rotation, no flip. `kdrv_display_band()` already returns the band top-down |
| Mirror frame ceiling | 14 bytes of header plus 32 KB (`KMIRROR_JPEG_CAP`) |
| Mirror pacing | 10 ms tick (`KMIRROR_TICK_MS`); 5 s without progress pauses (`KMIRROR_STALL_MS`) |

### The calls

| Call | What it does |
| --- | --- |
| `kdrv_jpeg_init()` | Creates the guard, the engine and the output buffer. Idempotent. `KODE_NO_RESOURCES` if any will not allocate |
| `kdrv_jpeg_encode(rgb565, w, h, quality, &out, &len)` | One encode. `out` is borrowed until the next call. `KODE_DENIED`: not initialised, a bad argument (`NULL`, `w` or `h` 0, quality outside 1 to 100) or another call is running. `KODE_ERROR`: the engine refused. `KODE_NO_RESOURCES`: the result did not fit |
| `kdrv_jpeg_selftest()` | Encodes a 16 by 16 mid-grey square, one whole MCU at YUV420, at quality 80, and checks it starts with `FFD8` and ends with `FFD9`. The body of `test component=jpeg`. `KODE_DENIED` means the encoder was busy and not broken. `KODE_NO_RESOURCES` means it never initialised |
| `kdrv_ppa_init()` | Registers the client, allocates the output buffer and the guard. Idempotent |
| `kdrv_ppa_prepare(src, w, h, scale, &dst, &dw, &dh)` | One blocking pass. `dst` is borrowed until the next call. `KODE_DENIED`: not initialised, a bad argument, a scale other than 1 or 2, a result too large, or another pass is running. `KODE_ERROR`: the hardware refused |
| `kmirror_hdr_put(dst, h)`, `kmirror_hdr_get(src, h)` | Serialise and parse the header. `put` ignores a `NULL`; `get` returns false for a `NULL` |
| `kmirror_credit_init()`, `_may_send()`, `_in_flight()`, `kmirror_sent()`, `kmirror_ack()` | The window, above |
| `kmirror_band()` | The rectangle, above |

### The files

| File | What it does |
| --- | --- |
| `include/kdrv_jpeg.h`, `src/kdrv_jpeg.c` | The JPEG encoder: engine, output buffer, try-lock, self-test |
| `include/kdrv_ppa.h`, `src/kdrv_ppa.c` | The scaler: client, output buffer sized in whole cache lines, try-lock |
| `include/kmirror_proto.h`, `src/kmirror_proto.c` | Header serialisation, the credit window, the band arithmetic. Pure C |

## Gotchas

> **WARNING**
>
> **The output pointer is valid until the next call, not until you have finished reading it.** Both `kdrv_jpeg_encode()` and `kdrv_ppa_prepare()` release their guard the moment the hardware returns. Copy the frame before your AO step ends, as `mirror_try_start()` does.

> **WARNING**
>
> **An output buffer whose size is not a whole number of cache lines fails every pass.** The PPA driver logs `out.buffer addr or out.buffer_size not aligned to cache line size` and the mirror sends no frame and gives no other explanation. `PPA_OUT_CAP` is arithmetic on purpose. Do not replace it with the bare 411,640.

> **NOTE**
>
> **`KODE_DENIED` is an ordinary answer.** From outside, it is how "the codec is busy" looks. Retry. Five seconds of nothing but retries is a stall, and the mirror pauses.

> **NOTE**
>
> **Four smaller points.**
>
> - **The driver does not limit quality to 70.** `mirror_start` does. At quality 100 a full-panel JPEG is several times the 32 KB frame ceiling, and every frame would be dropped.
> - **Acks may skip frames.** A host that acknowledges only the newest frame it has received behaves correctly. A host that waits for one ack per frame misreads the window.
> - **A `NULL` is tolerated everywhere in `kmirror_proto`, but do not pass one.** The tolerance only stops a broken contract from becoming a crash somewhere else.
> - **Without `CONFIG_KODEOS_MIRROR` the symbols are gone.** That includes `kdrv_jpeg_selftest()` and with it `test component=jpeg`, since the codec has no other caller.

## Example code

The protocol half runs without the screen. This walks the credit window and the band arithmetic by hand and logs the results.

```c
/*
 * mirror_credit.c - 2026-10-28 - from Kode with love
 *
 * Fills the mirror's credit window of 4, retires all four frames with one
 * cumulative ack, and maps rows 1 to 3 to a half-scale rectangle. The log
 * shows: in flight 4, may send 0; in flight 0, may send 1; band y 0 h 2.
 */
#include "kode.h"
#include "kmirror_proto.h"

#define TAG  "credit"

/* Exercise the window and the band, then idle so the console stays up. */
void app_main(void)
{
    kmirror_credit_t credit;
    uint16_t         seq = 0U;
    uint32_t         y = 0U;
    uint32_t         h = 0U;

    kode_init();
    kmirror_credit_init(&credit, 4U);
    for (uint8_t i = 0U; i < 4U; ++i) {
        seq = kmirror_sent(&credit);
    }
    KLOG_N(TAG, "in flight %u, may send %u",
           (unsigned)kmirror_in_flight(&credit), (unsigned)kmirror_may_send(&credit));

    (void)kmirror_ack(&credit, seq);   /* the newest frame retires the older ones */
    KLOG_N(TAG, "in flight %u, may send %u",
           (unsigned)kmirror_in_flight(&credit), (unsigned)kmirror_may_send(&credit));

    kmirror_band(1, 3, 2U, 502U, &y, &h);
    KLOG_N(TAG, "band y %lu h %lu", (unsigned long)y, (unsigned long)h);

    while (1) {
        kode_delay_ms(1000);
    }
}
```

Compiled against the SDK with ESP-IDF v6.1 on 2026-10-28. **TBD**: a run on a Kode Dot.

## In the SDK

- [`components/kdrv/mirror`](https://github.com/kodediy/kode-sdk/tree/main/components/kdrv/mirror): the encoder, the scaler and the protocol.
- [`kmirror_proto.h`](https://github.com/kodediy/kode-sdk/blob/main/components/kdrv/mirror/include/kmirror_proto.h): the header, the window and the band, with the host test `tests/host/cases/test_kmirror_proto.c` beside them.
- [`components/kaos/display`](https://github.com/kodediy/kode-sdk/tree/main/components/kaos/display): AO_Display, the owner, with the mirror's state machine (`sdk/kaos/display`).