# Test pipeline

The checks each repository carries as scripts, run on a computer before a change is merged: what each one runs, which risk it covers, and the rule that merges a change.

## What it is

The checks of kodeOS, the SDK and the C5's firmware are scripts in each
repository. Nothing runs them for you: whoever opens a pull request runs them
on a computer before asking for the merge, and the reviewer runs the ones the
change touches again. A green result is a terminal that ends in `ok` or
`all passing`, not a mark on GitHub.

This page lists the scripts per repository, which risk each one covers and
the rule that merges a change. The release, built by hand from a clean clone of
its tag, is on `internals/release-pipeline`.

## How it is connected

```mermaid
flowchart LR
  change["a change on a branch"] --> fast["the fast checks<br/>no ESP-IDF"]
  fast --> host["the host suites<br/>ESP-IDF v6.1"]
  host --> fw["the firmware build<br/>and its four checks"]
  fw --> pr["pull request"]
  pr --> review["one approval"]
  review --> main["main"]
  class change ext
  class main store
```

*What a change goes through before it is merged*

The checks run from the order of their cost: the fast ones first, so a broken
rule is found in seconds and nobody waits for a firmware build to hear about it.

kodeOS runs its checks on the SDK **as pinned** by the `sdk` submodule. A
suite that passed in the SDK says nothing about the commit kodeOS builds from,
so the submodule must be where the change leaves it:
`git submodule update --init --recursive` before you trust a result.

## Specifications

### kodeOS

From the root of the repository, with ESP-IDF v6.1 loaded
(`. ~/.espressif/v6.1/esp-idf/export.sh`).

| Step | Command | Needs |
| --- | --- | --- |
| The kit's tokens | `python3 sdk/components/kui/tools/gen_tokens.py --check`, when `tokens.json` changed | Python only |
| The tool tests | `tools/tool_tests.sh`: the bench tools, the release scripts, the factory tools and the SDK's gate tests, against fakes | Python and bash |
| The C5's host tests | `sh sdk/c5/test/run.sh` | A C compiler |
| The wire | `KODE_REQUIRE_C5=1 sdk/tools/check_wire_twins.sh`: the P4's and the C5's copies of each wire header are the same | The C5 submodule |
| The layers | `tools/check_layers.sh` | Nothing |
| The to-do markers | `tools/check_todo_refs.sh BASE`, where `BASE` is the commit the branch started from | git |
| The host suites | `tools/host_tests.sh`: the six SDK suites | ESP-IDF v6.1, Ruby |
| The build | `idf.py build` | ESP-IDF v6.1 |
| The size | `tools/check_os_size.sh build/kodeos.bin` | The build |
| The RAM | `sdk/tools/check_ram.sh build/kodeos.elf tools/ram_budget.txt` | The build |
| The starts | `sdk/tools/check_starts.sh build --all` | The build |
| MISRA | `sdk/tools/check_misra.sh build/compile_commands.json tools/misra_baseline.txt sdk/components components/kgui` | The build, and cppcheck at the version on the baseline's first line |
| A clean tree | `git status`: the build changed no file that is in the repository | The build |

Details of the build's checks:

| Check | Behaviour |
| --- | --- |
| Size | Against the smaller of two ceilings, both read from the tree: the 4 MB slot ESP-IDF checks, and the mailbox's image space, `KAPPS_PENDING_INTENT_OFF` in `kapps_fmt.h`. It prints `name n of ceiling bytes, left (pct%)` and warns once under 10 % is left |
| RAM | The image's static use of internal RAM against `static_max` (417000 bytes) in `tools/ram_budget.txt`, and every data symbol of 2048 bytes or more in internal RAM must have its own line there |
| Starts | Every Active Object compiled is in the image, each one's start sits in the object that holds its queue, and the OS role names every start strongly, so kodeOS parks none of its own owners |
| MISRA | `cppcheck` and its MISRA addon over the first-party C in `sdk/components/` and `components/kgui/`. The baseline lists, per file and rule id, a count the file must not exceed. `sdk/tools/build_cppcheck.sh` builds the version the baseline names; `--update` rewrites the baseline after a change that lowers a count |
| Clean tree | A file the build rewrote and nobody committed is a generated file that belongs in the change |

`tools/check_todo_refs.sh` reads only the lines a change adds. A `TODO`,
`FIXME`, `HACK` or `XXX` there must name its issue right after it, as in
`TODO(#42)` or `FIXME kodediy/kode-sdk#7`.

`tools/check_starts.sh` reads the archives a build made. It fails when an
Active Object's start is not in the object that holds its queue: in an app,
`kode_init()` names each optional start weakly, so a door linked without its
start would post to an AO with no queue and loop the boot. `--all` also asks
that every AO is in the image; `--absent` asks that the named ones are not.

### The SDK

From the root of the SDK repository. The first four steps need no ESP-IDF.

| Step | Command |
| --- | --- |
| The kit's tokens | `python3 components/kui/tools/gen_tokens.py --check`, when `tokens.json` changed |
| The gates' own tests | `bash tools/test_check_ram.sh`, `test_check_misra.sh`, `test_check_wire_twins.sh`, `test_check_starts.sh` and `test_check_app_size.sh`, each in `tools/` |
| The examples' pin | `tools/check_example_pins.sh` |
| The layers | `tools/check_layers.sh` |
| The host suites | `tools/host_tests.sh`, with ESP-IDF v6.1 loaded. Delete `tests/host*/build` first when a header changed: stale mocks fake compile errors |
| The target build | `tests/target` in each of five variants, below, when `components/` changed |

A variant is the file `tests/target/sdkconfig.<variant>` on top of the
defaults, named by `KODE_TARGET_VARIANT`. A build directory holds one variant
and refuses another, so `rm -rf build` between two. In `tests/target`, with
ESP-IDF v6.1 loaded:

| Variant | Build | RAM budget | Start check |
| --- | --- | --- | --- |
| The full build, in the OS role as kodeOS is | `idf.py build` | `ram_budget.txt` | `--all` |
| `nowifi`: radio on, Wi-Fi off, no mirror. The only build with the display linked, the console on and no stream | `KODE_TARGET_VARIANT=nowifi idf.py build` | `ram_budget.txt` | `--all` |
| `minimal`: the LED and the buttons on an app's own defaults (`main/minimal.c`) | `KODE_TARGET_VARIANT=minimal idf.py build` | `ram_budget.minimal.txt` | `--absent` the twelve owners the app never uses |
| `minimal_noradio`: the same with `KODEOS_RADIO=n` and `KODEOS_CONSOLE=n`, the build that compiles `kdrv_c5_hold.c` | `KODE_TARGET_VARIANT=minimal_noradio idf.py build` | `ram_budget.minimal_noradio.txt` | `--absent` the same twelve, AO_Console and AO_Autotest |
| `testdot`: the full build with AO_TestDot, the only one that compiles the final test, the OV5640 and the station's chips | `KODE_TARGET_VARIANT=testdot idf.py build` | `ram_budget.testdot.txt` read before `ram_budget.txt` | `--all` |

After each build, `../../tools/check_ram.sh build/kode_target.elf <budget>` and
`../../tools/check_starts.sh build <rule>`. The twelve owners a minimal app
never uses are AO_Radio, AO_Net, AO_Display, AO_Storage, AO_AppMgr, AO_Ir,
AO_Nfc, AO_Rfid, AO_Audio, AO_Imu, AO_Haptic and AO_Touch.

`tests/target` links every SDK component with every option on, except the
camera, so its static use of internal RAM is what the SDK costs an app before
the app adds a line. `check_example_pins.sh` checks that every example's
`main/idf_component.yml` and the SDK's README name the same SDK version, and
that it is not older than the newest `v*` tag. An example builds against the
tag it pins, not against the tree it sits in. In a shallow clone with no tags
it checks only the first rule and says so.

### The C5's firmware

From the root of `kode_slave_c5` (in kodeOS, `sdk/c5/`).

| Step | Command | ESP-IDF |
| --- | --- | --- |
| The host tests | `KODE_C5_CFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=undefined -fno-omit-frame-pointer -g" sh test/run.sh`: the pure parts, pointer and length arithmetic over wire frames and records | None |
| The image kodeOS ships | `idf.py -B build_kode set-target esp32c5 build`, then `tools/check_image.sh build_kode` | v6.0.2 |
| The production line's image | `idf.py -B build_closed -DSDKCONFIG=build_closed/sdkconfig -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.defaults.esp32c5;sdkconfig.defaults.closed" set-target esp32c5 build`, then `tools/check_image.sh build_closed closed`: what it must hold, and that its bootloader fits once signed | v6.1 |

The closed image needs ESP-IDF v6.1: the bootloader of 6.0.2 revokes the free
key digest slots on its first boot, whatever the configuration says.

### What is tested where

| What can go wrong | Where it is tested | How |
| --- | --- | --- |
| The bootloader jumps into a bad app or keeps a sketch armed after the wrong reset | `test_kapps_boot.c`, `test_kapps_req.c` | The decisions are pure functions in `kapps_fmt.h`; the bootloader asserts at compile time that its reset numbers are ESP-IDF's |
| The apps region believes a broken directory copy, or loses a write to a power cut | `test_kapps_dir.c`, `test_kdrv_apps_region.c` | Emulated flash with the board's table, failing at every step of the write |
| An install, an eviction or an erase leaves the region wrong | `test_appmgr_ao.c`, `test_kdrv_apps_plan.c` | AO_AppMgr over the emulated flash |
| An update is never confirmed | `test_boot_ao.c`, the `host_slot` suite | AO_Boot's first transition; `kdrv_slot` over `esp_ota_ops` |
| The console misreads a line | `test_console_grammar.c`, `test_console_parse.c`, `test_log_ao.c` | The grammar functions, then whole lines through AO_Log and AO_Console |
| A stored setting answers wrong | `test_console_config.c` and each owner's file | `config_support.h` plays the console against each owner |
| The Kode channel accepts a wrong answer | `test_net_channel.c`, `test_net_ao.c` | The parser alone, then AO_Net through a session |
| A bench tool sends a file wrong | `tools/tests/test_kcli.py` and the other tool tests | A fake serial port plays the board |
| The P4 and the C5 drift apart on the wire | `sdk/tools/check_wire_twins.sh` | Compares `kode_wire.h` and `kir_wire.h` with their twins in the C5 firmware, ignoring the SPDX line. In kodeOS it runs against the real headers (`KODE_REQUIRE_C5=1`); in the SDK alone the C5's copies are not checked out, so the SDK runs only the script's own test |
| A door links without its owner's start | `sdk/tools/check_starts.sh` | Over the archives of kodeOS's build and of each `tests/target` variant |
| The hardware, or the boot of a real board | `test component=...` on a Kode Dot, and the checks of a release on a flashed unit | `internals/release-pipeline`, `internals/production-test` |

### What no script covers

| Gap | Why |
| --- | --- |
| The bootloader on a board | Host tests cover its decisions, not the flash reads, the image check or the jump |
| The logic of kgui | Entangled with LVGL objects. `components/kgui/sim` renders the screens only |
| What the screen shows | Checked on a board with a screenshot and the mirror: `internals/bench-tools` |
| The rows a person judges | `buttons`, `led` and `display`: `internals/production-test` |
| The C5's closed configuration on a chip | `tools/check_image.sh` checks the unsigned build; the signing, the flash encryption and the first boot happen on the production line |
| Radios on the air, power-off on battery | Only a person with a Kode Dot on the bench |

### How a change reaches main

| Rule | kodeOS | SDK | C5 firmware |
| --- | --- | --- | --- |
| Ruleset on `main` | "Protect main" | "Protect main" | None |
| `main` cannot be deleted or force-pushed | Yes | Yes | No |
| A change arrives by pull request only | Yes | Yes | No |
| Approvals | One, from the `main-approvers` team; a new push dismisses it | The same | None |
| Bypass | The `main-approvers` team, at merge time only | The same | |
| Tags | | | "Release tags: admins only": only an admin creates, moves or deletes one |

The approval is where the checks are vouched for: the pull request says which
of the steps above ran and how they ended, and the reviewer asks for any the
change touches that it does not name. The rulesets of kodeOS and the SDK also
list required status checks (`static`, `host`, `firmware` in kodeOS;
`host-tests`, `esp32p4`, `esp32p4 (nowifi)` in the SDK). No workflow reports
them, so every merge goes through the bypass of the `main-approvers` team.

The workflow files under `.github/workflows/` stay in the three repositories,
and GitHub Actions does not run them. They hold the same commands as the tables
above, so they are also a written record of each check.

## Gotchas

> **WARNING**
>
> **`check_layers.sh` and `host_tests.sh` say `skipped` without the SDK.**
> Both kodeOS wrappers skip when the `sdk` submodule is missing, and exit 0. If
> you see `skipped`, run `git submodule update --init --recursive` before you
> trust the result. With `CI=true` in the environment a skip fails instead.

> **WARNING**
>
> **A target build of the SDK refuses the variant in its directory.** The
> `sdkconfig` lives in `build/` and remembers the variant it was made for, so a
> second variant in the same directory stops at configure time with `rm -rf
> build` in the message. That is the fix.

> **NOTE**
>
> **A case that fails only on someone else's computer is real.** The host suite
> is order-coupled, so a failure that depends on the order of the cases is a
> finding, not noise. It also runs as root in some containers, and root writes
> into a read-only directory: `internals/host-tests`.

## Example code

Run kodeOS's checks before you open a pull request, from the root of the
repository.

1. Run `tools/check_layers.sh`. Every line reads `ok` and the exit status is 0.
2. Run `tools/tool_tests.sh`. It ends with `tool tests, all passing`.
3. Run `KODE_REQUIRE_C5=1 sdk/tools/check_wire_twins.sh` and `sh sdk/c5/test/run.sh`.
4. Load ESP-IDF v6.1 and run `tools/host_tests.sh`. It ends with `cases across 6 suites, all passing`.
5. Build with `idf.py build`, then run `tools/check_os_size.sh build/kodeos.bin`. It prints the image size against its ceiling.
6. Run `sdk/tools/check_ram.sh build/kodeos.elf tools/ram_budget.txt` and `sdk/tools/check_starts.sh build --all`.
7. Run `sdk/tools/check_misra.sh build/compile_commands.json tools/misra_baseline.txt sdk/components components/kgui` with the baseline's cppcheck version on your path.
8. If you added a `TODO`, run `tools/check_todo_refs.sh BASE`, where `BASE` is the commit your branch started from.
9. Run `git status`. The build left no change in a tracked file.

## In the SDK

- [`tools`](https://github.com/kodediy/KodeOS/tree/main/tools): kodeOS's checks, with their tests in `tools/tests`.
- [`tools`](https://github.com/kodediy/kode-sdk/tree/main/tools): the SDK's checks and the gates kodeOS calls from `sdk/tools/`.
- [`tests/target`](https://github.com/kodediy/kode-sdk/tree/main/tests/target): the project that builds every component for the chip, with its variants and RAM budgets.