kodeOS internals

Screen mirror

Watch the glass of a Kode Dot live in a browser and drive it with clicks: the frame stream, the credit window, the two transports and the viewer.

What it is

The mirror sends the picture of the screen to a computer, live, and lets the computer send touches and button presses back. You watch and control a Kode Dot that is not in front of you.

It turns the shadow copy that AO_Display already keeps for screenshot into a stream. The shadow is a copy of the panel in PSRAM. The stream is JPEG frames of whatever changed, each sent as soon as it exists. A still screen sends nothing: when the changed band is empty, no frame leaves and no byte is spent. The mirror exists only because KODEOS_SCREENSHOT keeps the shadow, and KODEOS_MIRROR (default on) depends on it.

Input needs no special path. The verbs touch, tap, swipe and btn publish the same events AO_Touch and AO_Buttons publish, so no receiver can tell an injected sample from a finger.

How it is connected

flowchart LR
subgraph board["On the Kode Dot: AO_Display, every 10 ms"]
  shadow["shadow band"] --> ppa["kdrv_ppa"]
  ppa --> jpeg["kdrv_jpeg"]
  jpeg --> frame["frame buffer"]
  frame -- "slices" --> wire["console or<br/>BLE ring"]
end
subgraph computer["On the computer"]
  viewer["kmirror.py"]
end
wire --> viewer
viewer -- "mirror_ack, tap, btn" --> con["AO_Console"]
class shadow,frame,wire store
class viewer ext
class con ao

AO_Display owns the pipeline. It claims KBSP_RES_JPEG and KBSP_RES_PPA only when the build has the mirror, and runs one pass per tick of DISPLAY_MIRROR_TICK_SIG every 10 ms (KMIRROR_TICK_MS). AO_Console only parses the mirror_* verbs and passes each on as an event. The codec and scaler are documented on sdk/kdrv/mirror.

A pass has five stages:

1. Shadow band. kdrv_display_dirty_take() returns the rows that changed since the last call. The mirror keeps its own accumulator (s_mirror_y1 and s_mirror_y2 in kdrv_display.c), widened by every flush, so a mirror slower than the panel does not miss frames. A band the pass cannot use is kept in pend_y1 and pend_y2 for the next tick.
2. Scale. kmirror_band() turns the rows into a scaled rectangle inside the panel. It uses the first and last row, since ceil((y2-y1)/scale) counts too few rows for a band that does not start on a scale boundary.
3. PPA. The shadow is stored bottom-up, like a BMP, so every band goes through the pixel processing accelerator, even at scale 1, to flip and scale in one hardware pass. Its output buffer is one full panel of RGB565, rounded up to a whole number of cache lines.
4. JPEG. kdrv_jpeg_encode() takes the RGB565 output. A full 410 by 502 frame (402 KB) becomes about 19.7 KB at quality 70 in 3 ms, measured on a Kode Dot.
5. Copy out, then slice. The whole frame, a 14-byte header plus the JPEG, is copied into a PSRAM buffer before any record leaves, because the PPA and the codec keep their output pointer valid only until their next call. Each tick then sends up to 12 slices (KMIRROR_SLICE, the constant help and screenshot use), so one step stays short.

Specifications

The frame

Every frame is a header and len JPEG bytes. The stream is one continuous buffer: the next header follows the last JPEG byte. Fields are little-endian.

Only the first frame after mirror_start or a pause covers the whole panel. Later frames are the rectangle that changed, so a viewer draws each part at its own offset on a canvas and cannot treat the stream as motion-JPEG.

Verbs

Two transports

The board picks the transport once, when a frame begins, so a central that attaches or leaves mid-frame cannot split a frame across two.

The 214 comes from the NimBLE mbuf: a 256-byte block spends 16 bytes on struct os_mbuf, 8 on the packet header and 18 on leading space for the ATT, L2CAP and HCI headers. A longer slice chains a second mbuf out of a pool of 24 and empties it in about a dozen notifications.

A 30 s stream with the screen changing continuously ran at these rates:

Flow control

A stall has three forms, and all of them feed one counter:

• No credit, because the host stopped acknowledging.
• No progress inside a frame, because the cable is full or the Bluetooth ring never empties.
• Nothing can be encoded, because the shadow, the PPA or the codec refuses the rows. mirror_try_start() reports this as MIRROR_BLOCKED.

An idle screen is MIRROR_IDLE and never feeds the counter.

A pause stops the tick timer and frees the PSRAM frame buffer, but the session stays open. mirror_start or any mirror_ack wakes it, resets the credit window and forces a full frame, since a host that missed frames has an out-of-date canvas. A session ends when the transport is gone, for example when a Bluetooth central disconnects mid-frame. A slow transport only pauses it.

On the cable, control and frames share one connection. A viewer that sends quick btn presses queues its mirror_ack behind them and the board thinks it stopped. A window of 4 gives slack and the pause keeps the session alive.

Control records

Both have no =, so a parser tells them from data without decoding, and both travel on the console line on both transports. A marker in the binary stream would fail: an abort comes only mid-frame, so a header with len=0 would land right behind a partial body and a decoder that follows len would read its 14 bytes as JPEG and misread every later frame.

Measured

The viewer, kmirror.py

tools/kmirror.py serves one HTTP page. The page draws each frame on a canvas at the offset in its header, turns clicks into tap, and has a pad of buttons that drives btn.

The server answers:

• GET / for the page.
• GET /stream for the frames re-framed as header plus JPEG without base64.
• POST /tap, /touch and /btn with coordinates in canvas pixels, multiplied back by the scale. /btn takes press and release.

The viewer sends mirror_ack for each completed frame, answers mpause with mirror_start, and restarts the stream from its own side 2 s after an input with no picture back.

MirrorDecoder decodes the cable text and BleFrameDecoder the binary characteristic, and both share control_line() for the two control records. tools/tests/test_kmirror.py checks what the two sides share and needs no board.

Gotchas

Example code

Watch and drive the glass from a browser.

1. Connect the Kode Dot with the USB-C cable. Opening the port restarts it.
2. Run python3 tools/kmirror.py -p PORT --scale 1 --q 85 --open.
3. The browser opens http://127.0.0.1:8420 and shows the face. Click the picture: the Kode Dot receives a tap at that point.
4. Press the buttons on the page. The board receives btn presses, and the pad moves the selection of the launcher.
5. Leave the screen still. The stream sends nothing until something changes.
6. Press Ctrl-C in the terminal to stop the viewer.

In the SDK

• components/kaos/display: display_ao.c holds the pipeline and the credit window.
• components/kdrv/mirror: the JPEG encoder, the pixel accelerator and kmirror_proto.h. See sdk/kdrv/mirror.
• tools/kmirror.py: the reference viewer.