SDK

Mirror codec and protocol

The three files behind the live screen mirror of a Kode Dot: the P4's JPEG encoder, its pixel accelerator and the frame header with its credit window.

What it is

The screen mirror streams the glass of a Kode Dot to a computer as a series of JPEG frames. The module kdrv/mirror holds the three files it is made of:

An app does not call them. AO_Display owns the codec and the scaler (sdk/kaos/display), and the viewer side of the stream is on internals/mirror.

The encoder is never the slow part. A full 410 by 502 frame is 402 KiB of RGB565. The encoder turns it into 19.7 KB at quality 70 in 3 ms, measured on a Kode Dot, reading the pixels where they already are. The transport to the computer is always the bottleneck.

The three files are compiled only with CONFIG_KODEOS_MIRROR, which depends on KODEOS_SCREENSHOT and is on by default. Between them the codec and the scaler hold about 530 KB of PSRAM from boot, 128 KB of encoder output and one full panel of RGB565 for the scaler. A build that turns the mirror off does not pay it. The JPEG encoder also serves the screenshot dst=http upload, which encodes the whole glass once per picture.

How it is connected

flowchart LR
subgraph board["On the Kode Dot"]
  ao["AO_Display"] --> fb["front<br/>framebuffer"]
  fb -- "kdrv_display_band" --> ppa["kdrv_ppa"]
  fb -- "scale 1, in place" --> jpeg["kdrv_jpeg"]
  ppa --> jpeg
  jpeg --> hdr["kmirror_proto<br/>header"]
  cred["credit window"]
end
hdr --> wire["console wire<br/>or BLE"]
wire --> host["viewer on<br/>the computer"]
host -- "mirror_ack seq" --> cred
class ao ao
class fb store
class host ext

The owner and one pass

AO_Display owns the two peripherals. It claims KBSP_RES_JPEG and KBSP_RES_PPA, and calls kdrv_jpeg_init() and kdrv_ppa_init() while it brings the panel up. Neither init is fatal. A board whose encoder or scaler will not allocate still has a panel and still boots, and the mirror reports that it is blocked.

Each pass of the mirror is one AO step in mirror_try_start():

1. Take the rows that changed since the last frame from the display driver, or the whole panel for the first frame after a host attaches.
2. kmirror_band() turns that range into a scaled rectangle that stays inside the panel. An empty range is not an error: an idle screen produces no bytes.
3. Read the band from the front framebuffer. kdrv_display_band() returns it top-down, at full size.
4. At scale 2, kdrv_ppa_prepare() halves it into the scaler's own buffer. At scale 1 the band is encoded where it lies, since the encoder accepts any input alignment.
5. kdrv_jpeg_encode() compresses it. AO_Display clamps the requested quality to 70 (KMIRROR_QUALITY_MAX).
6. A frame larger than the 14-byte header plus 32 KB is dropped and logged.
7. The header and the JPEG are copied into the AO's own frame buffer in the same step, because both drivers hand back a pointer that is valid only until their next call. The frame then leaves in slices.

A refusal at step 3, 4 or 5 keeps the rows for the next pass. Five seconds without progress, or without credit from the host, log one warning and pause the stream (KMIRROR_STALL_MS).

The try-lock

One engine and one buffer, so a try-lock. Each driver has one hardware engine and one output buffer. A second caller inside would read the buffer while the first still writes it. The guard is a mutex used only as a try-lock: xSemaphoreTake() with a zero timeout. The loser is refused with KODE_DENIED and not blocked, because khal_mutex_t offers only a blocking take, and blocking would put AO_Display (priority 15) behind someone else's job inside what must stay a short AO step.

The lock exists because test component=jpeg also calls the encoder, inline on AO_Autotest's task, which is not AO_Display's.

The two drivers differ in how long they hold it.

• kdrv_jpeg_selftest() keeps the lock across the encode and its read of the result. If it did not, AO_Display could take the lock for a live frame between the two and overwrite the buffer under it, and a healthy codec would report itself broken.
• kdrv_ppa_prepare() releases the lock before it sets its output pointer. Its only caller reads that pointer in the same AO step, and QP's run-to-completion rule keeps that safe.

Alignment

The scaler's buffer has two alignment rules. kdrv_ppa allocates its output with MALLOC_CAP_SPIRAM | MALLOC_CAP_CACHE_ALIGNED, which makes the address line up with the caches. The PPA driver also checks the size: it must be a whole number of cache lines.

A full panel is 410 x 502 x 2 = 411,640 bytes, which is not a multiple of the chip's 64-byte line.

So PPA_OUT_CAP is written as arithmetic that rounds the panel size up to a whole number of the larger of the L1 and L2 line sizes. That gives 411,648 bytes here, and it follows a change of the panel or of the cache setting.

The encoder's output must also start on a cache line, so kdrv_jpeg allocates it with jpeg_alloc_encoder_mem() and not heap_caps_malloc(). ESP-IDF refuses any other output buffer. The input has no alignment rule.

The protocol

The protocol has no hardware. kmirror_proto.c includes no ESP-IDF header and keeps no state of its own, so the host tests check all of it on a computer. It holds three things that are hard to get right and impossible to notice on a bench: a header's byte order, a sequence number across its wrap, and a band's edges after scaling. Each shows up as a fault only hours into a stream.

• The header is serialised byte by byte. Sending the struct as it sits in memory would put this compiler's padding and this CPU's byte order on the wire. A host test pins literal byte positions, not only the round trip.
• The window counts with unsigned subtraction. in_flight is next - acked - 1 in 16 bits, correct across the wrap at 65535. A comparison like next > acked works for an afternoon, then stalls the stream with nothing in any log to say why: at 10 frames a second the sequence wraps every 109 minutes.
• The ack is cumulative, like TCP's. Acknowledging the newest frame retires every older one. kmirror_ack() refuses a sequence that is not in flight, so a stale or invented ack cannot open the window without bound. The refusal is silent on the wire: mirror_ack always answers >ok.
• A band's row count depends on where it sits. Source rows [y1, y2) land on destination rows [y1/s, (y2-1)/s], and the code counts bottom - top + 1. A formula on the length alone, ceil((y2-y1)/s), loses a row for a misaligned band. Rows 1 to 3 at half scale touch destination rows 0 and 1, two rows, and the length-only formula says one.

Specifications

The frame header

Every frame is a header followed by len bytes of JPEG. All numbers are little-endian. x, y, w and h count pixels of the scaled picture. At present the mirror always sends full-width bands, so x is 0.

KMIRROR_HDR_LEN is 14.

The credit window

kmirror_credit_t is the sender's half of the flow control: next (the sequence of the next frame), acked (the last one the host confirmed) and window (frames that may be outstanding, a uint8_t). AO_Display starts it with a window of 4 (KMIRROR_WINDOW).

The band

kmirror_band(y1, y2, scale, v_res, &out_y, &out_h) clamps [y1, y2) into the panel (v_res is 502 rows) and scales it. Scale 1 is full size, 2 is half. A scale of 0 becomes 1. y1 >= y2 after clamping gives height 0 and is not an error. The result never runs past v_res / scale.

The encoder and the scaler

The calls

The files

Gotchas

Example code

The protocol half runs without the screen. This walks the credit window and the band arithmetic by hand and logs the results.

/*
 * mirror_credit.c - 2026-10-28 - from Kode with love
 *
 * Fills the mirror's credit window of 4, retires all four frames with one
 * cumulative ack, and maps rows 1 to 3 to a half-scale rectangle. The log
 * shows: in flight 4, may send 0; in flight 0, may send 1; band y 0 h 2.
 */
#include "kode.h"
#include "kmirror_proto.h"
 
#define TAG  "credit"
 
/* Exercise the window and the band, then idle so the console stays up. */
void app_main(void)
{
    kmirror_credit_t credit;
    uint16_t         seq = 0U;
    uint32_t         y = 0U;
    uint32_t         h = 0U;
 
    kode_init();
    kmirror_credit_init(&credit, 4U);
    for (uint8_t i = 0U; i < 4U; ++i) {
        seq = kmirror_sent(&credit);
    }
    KLOG_N(TAG, "in flight %u, may send %u",
           (unsigned)kmirror_in_flight(&credit), (unsigned)kmirror_may_send(&credit));
 
    (void)kmirror_ack(&credit, seq);   /* the newest frame retires the older ones */
    KLOG_N(TAG, "in flight %u, may send %u",
           (unsigned)kmirror_in_flight(&credit), (unsigned)kmirror_may_send(&credit));
 
    kmirror_band(1, 3, 2U, 502U, &y, &h);
    KLOG_N(TAG, "band y %lu h %lu", (unsigned long)y, (unsigned long)h);
 
    while (1) {
        kode_delay_ms(1000);
    }
}

Compiled against the SDK with ESP-IDF v6.1 on 2026-10-28. TBD: a run on a Kode Dot.

In the SDK

• components/kdrv/mirror: the encoder, the scaler and the protocol.
• kmirror_proto.h: the header, the window and the band, with the host test tests/host/cases/test_kmirror_proto.c beside them.
• components/kaos/display: AO_Display, the owner, with the mirror's state machine (sdk/kaos/display).