# Release pipeline

How a version tag becomes a kodeOS release, built by hand from a clean clone of the tag: the order of the three repositories, the checks, the C5 image, the welcome app, the factory pack and the publish.

## What it is

A release is a numbered kodeOS that people can download. A maintainer makes
one by hand. They push a git tag named `vMAJOR.MINOR.PATCH` on `main`, clone
that tag into an empty folder, run the checks, build the C5 image, kodeOS and
the welcome app, and assemble the release files. They try the factory pack on
a Kode Dot, and publish the files as a GitHub release with `gh release create`.

Four files make a release:

- a factory pack: a zip with the six images and a flashing script;
- the standalone `kodeos-vX.bin` for updates;
- `SHA256SUMS.txt`;
- `BUILD_INFO.txt`, the record of what the release was built from.

The clean clone is the point. A build in a working tree can pick up a file
nobody committed, a submodule left on another commit or a `dependencies.lock`
from last week. A clone of the tag holds only what the tag names.

The checks a change goes through before it is merged are on
`internals/test-pipeline`.

## How it is connected

```mermaid
flowchart LR
  c5["C5 firmware<br/>tag, when it changed"] --> sdk["SDK<br/>tag"]
  sdk --> pins["kodeOS: kgui pins,<br/>PROJECT_VER, submodule"]
  pins --> tag["kodeOS tag"]
  tag --> clone["clean clone<br/>of the tag"]
  clone --> build["checks, build_c5.sh,<br/>idf.py, build_welcome.sh"]
  build --> dist["assemble_release.sh<br/>BUILD_INFO.txt"]
  dist --> try["flash a Kode Dot"]
  try --> pub["gh release create"]
  class c5,sdk,tag ext
  class dist store
  class try chip
```

*From three tags to a published kodeOS*

Three repositories release, always in this order, because each pins the one
before it: the C5's firmware (when it changed), the SDK, which pins the C5 as
its `c5` submodule, and kodeOS, which pins the SDK as its `sdk` submodule and
by tag in kgui's manifest. The SDK tag and the kodeOS tag are two separate
numbers.

An annotated tag is its own git object and is not the commit. Resolve a tag to
its commit before you trust it: `git rev-parse TAG^{commit}`. A pin resolved
against the wrong object builds code nobody reviewed.

`dependencies.lock` is not in the repository, so the tag named in kgui's
manifest alone decides what a clean clone builds. If that tag does not exist,
the build fails.

## Specifications

### The order of a kodeOS release

1. If the C5's firmware changed, release it first (below). Then move the SDK's `c5` submodule to that commit.
2. Release the SDK (below): tag `vAA.BB.CC` on its `main`.
3. In kodeOS, on a branch, set the four `version:` pins of `components/kgui/idf_component.yml` (`kodediy/kui`, `kodediy/kode`, `kodediy/kbsp`, `kodediy/kdrv`) to that tag. They are the only per-release manifest edit in the repository.
4. Set `PROJECT_VER` in the root `CMakeLists.txt` to the new kodeOS number, and move the `sdk` submodule to the commit the SDK tag names.
5. Run the checks (`internals/test-pipeline`), open the pull request and merge it.
6. Tag `vXX.YY.ZZ` on the merged commit of `main` and push the tag.
7. Build the release from a clean clone of the tag, try it and publish it (below).

### The build from a clean clone

From an empty folder. Each command stops the release when it fails.

| Step | Command | What it proves |
| --- | --- | --- |
| Clone | `git clone --recurse-submodules --branch vXX.YY.ZZ git@github.com:kodediy/KodeOS.git` | The tree is exactly the tag's |
| Version | `tools/validate_release_tag.sh vXX.YY.ZZ` prints `XX.YY.ZZ` | The tag is well formed. The printed string is `PROJECT_VER` and names the zip and the bin |
| On `main` | `git fetch origin main` then `tools/check_on_main.sh HEAD origin/main` | The tag's commit was merged |
| The SDK pin | kgui's four `version:` lines name one tag, and `git -C sdk rev-parse HEAD` equals `git -C sdk rev-parse vAA.BB.CC^{commit}` | kgui and the submodule name the same tree |
| The C5 on its `main` | In `sdk/c5/`: `git fetch origin main` then `../../tools/check_on_main.sh <the commit the SDK pins> origin/main`, the commit from `git -C sdk ls-tree HEAD c5` | The C5 image comes from a merged commit |
| The checks | kodeOS's steps on `internals/test-pipeline` | The tag passes what every merge passes |
| The C5 image | With ESP-IDF v6.0.2 loaded: `tools/build_c5.sh` | Writes `storage_image/c5_slave.bin` and its stamp |
| kodeOS | With ESP-IDF v6.1 loaded: `idf.py -DPROJECT_VER=XX.YY.ZZ build` | The OS, with the C5 image in its FAT |
| The welcome app | In the same shell: `tools/build_welcome.sh` | Writes `apps_image/<project>.bin` and its stamp |
| Assemble | `tools/assemble_release.sh XX.YY.ZZ` | Writes the zip, the bin and `SHA256SUMS.txt` in `dist/` |
| Record | `BUILD_INFO.txt` in `dist/`, below | What the release was built from |

The C5 image comes before kodeOS's build, because kodeOS's build puts it in
the FAT image.

### The scripts

| Script | What it does | Exit |
| --- | --- | --- |
| `tools/validate_release_tag.sh TAG` | Accepts `^v[0-9]+\.[0-9]+\.[0-9]+$` and prints the tag without the `v`. It is the only place the tag format is defined | 0 valid, 1 malformed |
| `tools/check_on_main.sh COMMIT REF` | Succeeds when `COMMIT` is an ancestor of `REF`. The caller fetches `REF` first | 0 yes, 1 no |
| `tools/check_os_size.sh kodeos.bin` | Prints `name n of ceiling bytes, left (pct%)` against the smaller of the 4 MB slot and the mailbox's image space (`KAPPS_PENDING_INTENT_OFF` in `kapps_fmt.h`), and warns once under 10 % is left | 0 under, 1 past the ceiling or unreadable |
| `tools/build_c5.sh` | `set-target esp32c5` and a build of `sdk/c5/` into `build_c5/`, copies `kode_c5.bin` to `storage_image/c5_slave.bin`, and writes the stamp `storage_image/c5_slave.commit` as `<commit> <sha256>`, with `-dirty` after the commit when the C5 tree has local changes. Warns when the commit is not the one the SDK pins | 0 built, 1 submodule missing, `IDF_PATH` unset or no image |
| `tools/build_welcome.sh` | Reads `apps_image/welcome.pin` (`<repository> <app folder> <commit>`), clones that repository into `build_welcome/src` once, builds the folder at the pinned commit, and writes `apps_image/<project>.bin` and its stamp `<project>.commit` | 0 built, 1 no pin, the source not at the pin, or a failed build |
| `tools/assemble_release.sh VERSION [BUILD [OUT [APPS]]]` | Turns a finished build into the release files. Defaults: `build`, `dist`, `apps_image` | 0 done, 1 an input is missing or a check failed |

`build_c5.sh` needs ESP-IDF v6.0.2, because the C5 firmware builds with it.
Host and slave must run the same esp-hosted version, so building from the
pinned commit is what keeps the pair true. If `idf.py` is a shell function
(some installers define it so), both build scripts run `$IDF_PATH/tools/idf.py`.

The welcome app lives in `kode-app-onboarding`, a private repository, so
`build_welcome.sh` needs a key with read access to it. A fork points
`welcome.pin` at a welcome app of its own.

### What assemble_release.sh checks and writes

| Step | Detail |
| --- | --- |
| Inputs | `bootloader/bootloader.bin`, `partition_table/partition-table.bin`, `ota_data_initial.bin`, `kodeos.bin`, `storage.bin` from the build, and `flash.sh`, `FACTORY_README.txt`, `THIRD_PARTY_NOTICES.txt` from `tools/release/`. The first missing one stops the script with one message |
| Size | `check_os_size.sh` on `kodeos.bin` |
| FAT contents | `build/storage_stage` must hold exactly `kode/` and `c5_slave.bin` |
| C5 stamp | The commit in `build/c5_slave.commit` must equal what `git -C sdk ls-tree HEAD c5` pins, and the SHA-256 must match the `c5_slave.bin` in the FAT |
| Welcome app | The image named by `CONFIG_KODEOS_WELCOME_APP` must be in `apps_image/` with its stamp, built from the commit `welcome.pin` names, and match its stamp's SHA-256 |
| `apps.bin` | `tools/mkapps.py` builds the apps region with the welcome app installed and no boot request. `flash.sh` must write it at the region's base in `partitions.csv` |
| `kodeos-factory-vX.zip` | The six images, `flash.sh`, `README.txt` and `THIRD_PARTY_NOTICES.txt`: nine entries. `flash.sh` keeps its executable bit |
| `kodeos-vX.bin` | `kodeos.bin` byte for byte, for `install ... dst=os` or the mailbox |
| `SHA256SUMS.txt` | `sha256sum` over the zip and the bin |

Files are copied by name from an explicit list, so a key or a stray file in
the folder cannot reach a release file.

### BUILD_INFO.txt

Written in the clone's root after the assembly:

```sh
{
  echo "kodeOS XX.YY.ZZ"
  echo "kodeos commit  $(git rev-parse HEAD)"
  echo "sdk commit     $(git -C sdk rev-parse HEAD)"
  echo "c5 stamp       $(cat build/c5_slave.commit)"
  echo "welcome pin    $(cat apps_image/welcome.pin)"
  echo "welcome stamp  $(cat apps_image/*.commit)"
  echo "ESP-IDF        $(python3 -c 'import json; print(json.load(open("build/project_description.json"))["git_revision"])')"
  echo
  echo "--- dependencies.lock"
  cat dependencies.lock
} > dist/BUILD_INFO.txt
```

The resolved `dependencies.lock` is the only record of which component
versions the release was built with, because the repository keeps none.

### The factory pack

`flash.sh PORT` needs Python 3 with esptool 5 or newer and checks the version
before it touches the board. It makes one call:
`esptool --chip esp32p4 -p PORT write-flash --erase-all` with these images.

| Offset | Image |
| --- | --- |
| `0x2000` | `bootloader.bin` |
| `0x8000` | `partition-table.bin` |
| `0x1B000` | `ota_data_initial.bin` (blank otadata) |
| `0x20000` | `kodeos.bin` |
| `0xC40000` | `apps.bin` (the apps region with the welcome app) |
| `0x1800000` | `storage.bin` (Kody's pack and the C5 image) |

The erase removes everything on the chip: settings, remembered networks, BLE
bonds, installed apps, the mailbox and every file on the internal volume.
Afterwards otadata is blank, so the board boots `factory`, and the apps region
holds the welcome app with no boot request. `boot_info` answers
`running=factory` and `otadata=blank`. See `internals/boot-model`.

### Trying it on a Kode Dot

Before publishing, flash a unit with the pack the release will carry and read
it back over the cable.

| Block | Checks |
| --- | --- |
| Files | `cd dist && sha256sum -c SHA256SUMS.txt` reports `OK` twice |
| Flash | Unzip the factory pack and run its own `./flash.sh PORT` |
| Boot | `info` names a running slot on the tag's version and `coredump=none`; `boot_info`'s reason is not `unexpected` |
| Self-test | `test component=main`: `pmic`, `gauge`, `ioexp`, `clocks`, `imu`, `mag` all `pass` |
| Memory | `ps`: `int_low` of at least 64 KB, `alloc_fail=0`, `log_dropped=0` |

### Publishing

| Repository | Command | Title | Files |
| --- | --- | --- | --- |
| kodeOS | `gh release create vXX.YY.ZZ --repo kodediy/KodeOS --verify-tag --title "kodeOS vXX.YY.ZZ" --generate-notes dist/kodeos-factory-vXX.YY.ZZ.zip dist/kodeos-vXX.YY.ZZ.bin dist/SHA256SUMS.txt dist/BUILD_INFO.txt` | `kodeOS vXX.YY.ZZ` | The four release files |
| SDK | `gh release create vAA.BB.CC --repo kodediy/kode-sdk --verify-tag --title "kode-sdk vAA.BB.CC" --notes-file notes.md --latest` | `kode-sdk vAA.BB.CC` | None. The notes are the tag's first message line, then the notes GitHub generates (`gh api repos/kodediy/kode-sdk/releases/generate-notes -f tag_name=vAA.BB.CC --jq .body`) |
| C5 firmware | `gh release create vC --repo kodediy/kode_slave_c5 --verify-tag --title "kode_c5 vC" --generate-notes dist/*` | `kode_c5 vC` | Below |

`--verify-tag` refuses a tag that is not on GitHub, so a release never names a
tag that exists only on one computer.

### The C5 firmware's release

Only an admin can push a `v*` tag on `kode_slave_c5` (the ruleset "Release
tags: admins only"). From a clean clone of the tag, on its `main`:

1. With ESP-IDF v6.0.2: `idf.py -B build_kode set-target esp32c5 build` and `tools/check_image.sh build_kode`. Copy `kode_c5.bin`, the bootloader, the partition table, `ota_data_initial.bin` and `flasher_args.json` into `dist/`, and write `dist/c5_slave.commit` as `<commit> <sha256 of kode_c5.bin>`, the format `build_c5.sh` writes in kodeOS.
2. With ESP-IDF v6.1: the closed build into `build_closed` and `tools/check_image.sh build_closed closed` (`internals/test-pipeline`).
3. `SIGN_TOKEN=<token> tools/sign_closed.py build_closed kode_c5-closed`. The factory's signing service holds the two Secure Boot keys and signs the bootloader and the app; the script checks both signatures against the keys in `pki/` before it writes anything. It needs `espsecure` on the path.
4. Zip the folder as `dist/kode_c5-closed-vC.zip`, copy its `kode_c5.bin` to `dist/kode_c5-closed.bin`, and write `SHA256SUMS.txt` over `dist/`.
5. Publish with the command above.

kodeOS builds the C5 image itself, from the commit the SDK pins, and does not
read this release. The release is the record of what that commit produces, and
the production line takes the closed image from it: the line's station writes
it to a virgin C5, and its first boot closes the C5 for good.

## Gotchas

> **DANGER**
>
> **`flash.sh` erases the whole chip and does not ask.** Everything on the Kode
> Dot goes, including installed apps and files on the internal volume. Copy what
> you keep off first, for example with `file_get`.

> **WARNING**
>
> **The pin check fails with "the sdk submodule is not at vX".** kgui's four
> `version:` pins and the `sdk` submodule must name the same tree, or the
> component manager compiles the submodule for some names and the tag's own copy
> for others, a mixture built without a word. Bump the four together, move the
> submodule to the SDK tag's commit, and check the tag's commit with
> `git rev-parse TAG^{commit}`.

> **WARNING**
>
> **`assemble_release.sh` refuses a FAT with anything but `kode/` and
> `c5_slave.bin`.** A bench file left in `storage_image/` ends up in
> `storage.bin`. A clean clone has none, which is one reason the release is built
> from one.

> **WARNING**
>
> **`SIGN_TOKEN` signs firmware every closed C5 accepts.** Keep it out of shell
> history and out of any file in the clone. `sign_closed.py` never prints it.

## Example code

A dry run of a release on your own computer, from a clean clone of `main`,
with no tag. It builds version `00.00.00`, and the pin check may find the
submodule ahead of the last tag, which is normal between two releases.

1. Check the tag format: `tools/validate_release_tag.sh v0.1.0` prints `0.1.0`. `v1.2.3-rc1` prints an error and exits 1.
2. Load ESP-IDF v6.0.2 and build the C5 image: `tools/build_c5.sh`. It prints the commit and size, and writes `storage_image/c5_slave.commit`.
3. Load ESP-IDF v6.1 and build kodeOS: `idf.py -DPROJECT_VER=00.00.00 build`.
4. Build the welcome app: `tools/build_welcome.sh`.
5. Assemble: `tools/assemble_release.sh 00.00.00`. It lists the zip, the bin and `SHA256SUMS.txt` in `dist/`.
6. Check the sums: `cd dist && sha256sum -c SHA256SUMS.txt` reports `OK` twice.
7. Flash a test unit from the pack: unzip it and run `./flash.sh PORT`. The last lines tell you to run `boot_info` and look for `running=factory`.

## In the SDK

- [`tools/assemble_release.sh`](https://github.com/kodediy/KodeOS/blob/main/tools/assemble_release.sh): the release files, with `tools/release` holding the pack's templates.
- [`tools/build_welcome.sh`](https://github.com/kodediy/KodeOS/blob/main/tools/build_welcome.sh) and [`apps_image`](https://github.com/kodediy/KodeOS/tree/main/apps_image): the welcome app's pin and build.
- [`components/kdrv/apps`](https://github.com/kodediy/kode-sdk/tree/main/components/kdrv/apps): `kapps_fmt.h`, where the mailbox ceiling the size check reads is defined.