kodeOS internals

Kode channel

How the owner of a Kode Dot sends console commands over the internet, how the board joins, how commands sent while it is away wait, and what a command may do.

What it is

The Kode channel lets the account that owns a Kode Dot send console commands to it over the internet and read the answers. The board joins the channel by itself whenever it has Wi-Fi. A command arrives as one console line in an MQTT message, and each line of the answer leaves as one message on another topic.

The channel is a fourth way into the console, next to the cable, the charging base and Bluetooth. A line from the channel goes through the same parser and the same command table, tagged with the source KODE_CONSOLE_SRC_NET. It shows nothing on the screen and changes nothing on the LED. The cloud side is run by Kode, and this page describes what the board does and what it asks of the cloud.

The session is persistent. A command sent while the board is off, asleep without a network or out of range waits at the broker, and the board runs it, in the order it was sent, when it connects again. How long the broker keeps it is the setting chan_keep.

The channel is built into any firmware that has Wi-Fi, MQTT and the console (CONFIG_KODEOS_CHANNEL, default on), kodeOS and SDK apps alike. The presence message names the running firmware.

How it is connected

sequenceDiagram
autonumber
box "On the Kode Dot"
  participant N as AO_Net
  participant C as C5 identity
end
box "Kode cloud"
  participant A as api.kode.diy
  participant B as MQTT broker
end
N->>A: POST /kodes/challenge
A-->>N: nonce
N->>C: sign kode-auth/v1 and nonce
C-->>N: ECDSA signature
N->>A: POST /kodes/connect, cert, nonce, signature
A-->>N: broker, topics, token
N->>B: MQTT 5 over TLS

AO_Net owns the channel. In order, it:

1. reads the certificate from the C5;
2. asks the api for a challenge;
3. has the C5 sign it with the key of the unit;
4. posts the certificate, the nonce and the signature to /kodes/connect;
5. opens an MQTT 5 session to the broker the answer names, asking the broker to keep it while the board is away.

The key never leaves the C5.

The Wi-Fi is the door and the stored setting chan is the switch: the board joins when it has an address and the one MQTT client is free, and leaves when it loses the address. With chan off, channel_info reads off. When a client of your own holds the MQTT client, the state is waiting with the reason mqtt_busy, and the board joins at the next address.

AO_Log copies replies into a ring, and AO_Net drains it. The console tags each reply with the source of its command through kode_log_reply_src(). AO_Log writes every line to the cable and Bluetooth, and a line tagged for the channel also goes, without its newline, into a ring in PSRAM (32 KB). AO_Net publishes one message per line on the out topic. A ! log line has no tag and never goes out.

An app that uses kode_mqtt_* itself sets CONFIG_KODEOS_CHANNEL=n. Otherwise mqtt_up, mqtt_down, mqtt_pub, mqtt_sub and mqtt_unsub answer busy while the channel holds the client. Launching an app ends the session of kodeOS, and before a deliberate restart the board leaves cleanly, so the Kode Dot reads offline at once.

Specifications

What the board asks of the cloud

The board needs an identity: the C5 holds the key of the unit and the certificate that the factory CA signed. A unit whose C5 has no certificate reports it once and stays off the channel.

• The cert is the unit's certificate only: the first unit_len bytes of what the C5 returns.
• The token is a JWT valid for 24 h, at most 511 characters. The board renews the session 82800 s (23 h) after the connect (CONFIG_KODEOS_CHANNEL_RENEW_S) and does not read the expiry. The 23 h and the waits between tries run in real time, so they keep counting while the board sleeps. A renewal or a retry that falls due wakes the radio side for a moment, the channel does its work, and the board rests again with the screen dark (sdk/kaos/net).
• The firmware hard-codes one address, the root of the api. The broker, the client id and the topics come from the answer and are used as given. An answer is refused whole unless all eight fields are present and fit, the broker is mqtts://, the serial is this unit's and no topic holds an MQTT wildcard.
• shots is optional: a full https:// prefix ending in /, at most 128 bytes, with no space or control character. A session without a valid one is a good session and uploads are refused.
• Back-off after a failure starts at 5 s and doubles to 10 min with about 25 % jitter. The terminal states are no_identity and refused.

The session keepalive is 60 s. After connecting, the board follows the in topic at QoS 1 and publishes {"online":true,"app":"<project name>","version":"<version>"} retained on the status topic. The project name of kodeOS is kodeos.

The last will is the retained {"online":false}. A clean stop fires no will, so the board publishes the same message itself before any stop that is on purpose.

A publisher must not retain a message on the in topic. The broker replays it at each subscription, so a retained app_run would reset the board into the same command for ever. The board refuses a retained payload.

The line that arrives

The board refuses a payload that is retained, cut by the driver, 240 bytes or longer (KODE_CONSOLE_LINE_MAX), empty, or that holds a CR, an LF or a NUL. A line feed at the end counts. A refusal runs nothing, logs a warning and answers >err code=3 msg=line on the out topic, or msg=retained for a retained payload. A good line joins the queue of the channel, and the queue gives it to the console in its turn. When the queue is full, the answer is >err code=4 msg=busy. Strings are quoted as everywhere: app_run name="snake".

Commands sent while the Kode Dot is away

A board that is off, asleep without a network or out of range misses what is sent to the in topic. The session of the channel is persistent, so the broker keeps those messages and delivers them when the board connects again. A power-on after OFF, a reboot, the return of Wi-Fi, a reconnect to the broker and the renewal of the session are all such a connect.

config key="chan_keep" value=<age> sets how long the broker keeps the session:

• The age counts the absence, not each message. A board away for less than the age gets everything the broker kept. A board away for longer finds the session gone, with every message in it.
• A change is stored at once (kset/chan_keep, owner AO_Net) and used at the next connect, a renewal included. The channel is not reconnected for it. The log writes one notice: channel: keep the session <age> (stored); the next connect asks for it.
• When the store refuses the write, the new age applies until the next reset. The log writes a warning and the answer reads stored=no. After a reset the stored age applies.
• channel_info shows keep= (the age the connect of the live session asked for, - while the state is not up) and session=. The board shows what it asked. It does not read the age the broker grants, so a broker with a lower maximum keeps the session for that maximum while keep= still shows the age asked.
• session=resumed means the broker kept the session, so the commands sent meanwhile are on their way. session=new means it held nothing for this Kode Dot. The log says the same once per connect: channel: session resumed or channel: new session (nothing kept for this Kode).

Order and pace

The commands the broker kept arrive in a burst, and the console takes one command at a time. The channel holds the burst and hands the console one line at a time.

The messages in the driver

esp-mqtt acknowledges a QoS 1 message itself, when its data callback returns, and calls the callback inside its API lock. The callback therefore copies the message into a ring in PSRAM and returns at once. It never waits for AO_Net, which can be blocked in a publish under the same lock. AO_Net empties the ring on a notice.

Lost on a reset, and run twice

• The queue and the ring are RAM. A reset in the middle of a burst drops the lines still in them, and the broker does not deliver them again because it has their acknowledgement. A queued app_run, an install or a panic resets the board. A queued power_off ends the run. Every line behind them is lost, and the board logs nothing about it at the next boot.
• Delivery is at least once. If the connection drops after the board received a message and before the acknowledgement reached the broker, the broker delivers it again and the command can run twice. The board has no key to tell the second delivery from a new command. Send only commands that are safe to repeat, or accept the risk.

What the broker must do

These are asks of the cloud side.

• Accept MQTT 5 from the Kode Dot. The channel does not fall back to 3.1.1.
• Honour the session expiry that the board asks for. The broker's maximum must be at least 7 days (604800 s) for 7d, and never asks for 4294967295 s.
• Publish to the in topic at QoS 1. A broker may or may not queue QoS 0 for an offline session, depending on its configuration.
• Return the same clientId for a Kode Dot each time. A different id starts a session with nothing in it, and the commands waiting under the earlier id are never delivered.
• Keep a burst small. The queue holds 32 lines and the ring 48 messages. Lines past 32 that fit in the ring are answered busy, so the sender sees them. Messages past what the ring holds before AO_Net empties it are dropped on the board, acknowledged and silent.
• Set the limit of the broker's own queue for a session (max_mqueue_len in EMQX) and say what a sender can expect when a Kode Dot has been away past it. The board cannot tell that a message was dropped, or which.
• Never retain a command.
• Keep a command well under 1 KB. The board does not change esp-mqtt's receive buffer, which defaults to 1024 bytes. TBD: what the broker does with a larger packet for this client is not measured.

While the board is ASLEEP its session stays up, so a command arrives as it always did and wakes the board when the console takes the line (sdk/kaos/lowpower). The persistent session matters across OFF, a reboot and a lost connection.

Three sources, one console

src_allows() holds the whole policy. Every other command runs from every port.

The channel carries lines, not bytes. screenshot dst=file and screenshot dst=http answer in a few lines and run from every port. Commands with many ordinary records (help, ps, tasks, test component=all) run from the channel.

A fourth source, SCHED, is not a port. It is a line that AO_Scheduler runs at a cron time (sdk/kaos/scheduler). The policy treats it as it treats the channel, and also refuses cron to it (>err code=5 msg=sched). Its replies go to the log as notices.

The channel has no allow-list. A command the cable runs runs here, with the exceptions above. Whoever holds the account of the owner, or one of its session tokens of 15 minutes, can:

• reflash kodeOS from a URL;
• erase apps;
• close the identity for good with ident_lock confirm=on;
• forget the networks and BLE keys with c5_wipe confirm=on or wifi_forget.

After that the board stays off the network until it is given Wi-Fi again by cable or Bluetooth.

cron is open, and what it leaves outlasts the session. The verb cron (internals/command-set) is accepted from the channel like every other verb, and the entry it leaves is stored. The entry runs later, with no session open and nobody at the board, and it can run every verb a SCHED line may run. Whoever can publish on the in topic can leave work for the future. List the entries with cron and remove one with cron name="<name>" when=off.

The board trusts the topic: a line on the in topic of its live session runs, and what may publish there is the rule of the cloud.

A command can end its own session: wifi_down, wifi_leave, radio_down and config key="chan" value=off. A command that resets the board answers >ok before the reset and the board tries to publish it, but it may not arrive. The lines queued behind it are lost with the reset.

How the answers go out

Replies keep going out on the old client while a renewal asks the api, and wait in the ring from the moment the old client is told to stop.

The fast mode

The Wi-Fi radio of the C5 sleeps between beacons by default (WIFI_PS_MIN_MODEM). A frame sent to a sleeping station is lost at first and the broker's TCP retransmits it, so a command reaching an idle Kode Dot arrives 3 to 6 s late. With power save off a round trip takes about 0.1 s and the board draws about 27 mA more (an average of 107 mA became 134 mA on the bench).

• The first good line while the mode is off asks AO_Radio for no power save. That first line of a burst is still slow.
• Each good line, and each frame of a screen upload stream, restarts a hold of 60 s (CHAN_FAST_HOLD_MS). A refused payload does not count.
• The mode ends 60 s after the last line, or at once when the channel is left. wifi_ps set to off keeps the station out of power save at all times.
• The log writes one notice when it turns on and one when it turns off.

The screen upload

The channel cannot carry a picture, but it carries the order for one. screenshot dst=http makes the board encode the glass as a JPEG (quality 80, KSHOT_QUALITY) and upload it with one HTTPS PUT. It needs KODEOS_SHOT_HTTP (default on), which needs the channel and the mirror's codec. The buffer holds 64 KB (KSHOT_JPEG_CAP) and a larger frame is not sent.

1. The owner's session asks the api for a link: POST /kodes/<serial>/shots. The body {} asks for one picture, and {"mode":"stream","seconds":60} for a stream. The answer holds the picture id, a token and the console line. A link for one picture lives 2 minutes, for a stream the seconds chosen, at most 5 minutes.
2. The owner sends that line to the Kode Dot on the in topic: screenshot dst=http token="<token>". The api issues tokens of 66 characters and the board accepts up to 128.
3. The board uploads to the shots prefix plus the token, with the JWT of the session as bearer. The api accepts the upload only with the link and the token of this Kode Dot.
4. The owner reads the picture from the api: GET /kodes/<serial>/shots/<id>.

The board never takes an address from a line. It takes the token, and the prefix comes from the connect answer, which arrived over TLS after the C5 signed the challenge.

screenshot dst=http token="<token>" [fps=<1..10>] [secs=<1..300>]
screenshot dst=http stop=on

A picture waits at most 20 s for its answer, and a late answer is dropped. A stream is not waited for. fps and secs make a stream and the missing one takes its default of 2 frames a second or 60 s.

A stream behaves like this:

• It runs in AO_Net, so the console is free and the owner can drive the board with btn, tap and touch while it shows the glass.
• A frame starts when its time has come and the last PUT is done, so a slow link lowers the rate and frames never queue.
• The api refuses frames closer than 100 ms. One TLS connection stays open between frames.
• Three network errors in a row end the stream, a 429 drops that frame, and a new valid order replaces a running stream.
• It ends with one of these words in a notice: time, expired (404), refused, channel, network, replaced, display or stopped.

What the upload keeps safe and what it costs:

• The board never follows a redirect and sends only to https://.
• It never logs the token, the link or the JWT (the HTTP client's log is held at info during a PUT, whatever config key="log" says).
• It wipes its copies of the token, the URL and the bearer when the upload ends. The connection of a stream keeps its URL until the stream closes, and the last picture stays in a PSRAM block until the next PUT.
• The PUT takes about 64 KB of PSRAM on first use and a second TLS session beside the MQTT one. Nothing new is taken from internal RAM.

channel_info

channel_info answers one record and >ok.

channel=up serial="KODE<serial>" broker="mqtts://..." since_s=312 why=- next_s=- lost=0 fast=on shot=stream sent=118 fps_got=3.9 left_s=30 keep=86400 session=resumed

Gotchas

Example code

Check the channel from the console, then switch it off and on.

1. Join Wi-Fi: python3 tools/kcli.py -p PORT cmd --gap 10 'wifi_auto'.
2. Run python3 tools/kcli.py -p PORT cmd 'channel_info'. A joined board reads channel=up, and since_s counts seconds in the state.
3. Run python3 tools/kcli.py -p PORT cmd 'config key="chan" value=off'. A second channel_info reads channel=off.
4. Run config key="chan" value=on. The board joins again as soon as the MQTT client is free.
5. Run config key="chan_keep" value=7d. The log says that the next connect asks for it. With the channel up, channel_info keeps keep=86400 until the channel connects again, and then reads keep=604800.

TBD: the cloud side of the channel (which accounts and tokens may publish on the in topic) is not described here; Kode's ecosystem documentation settles it.

In the SDK

• components/kaos/net: net_ao.c runs the channel and net_channel.c reads the api's answers.
• components/kaos/console: src_allows() and the screen upload order.
• components/kdrv/radio: kdrv_net.c holds the HTTPS and MQTT client the channel uses.