Rolling out across a fleet

Upgrading one device from the dashboard is straightforward. Upgrading four thousand is a different task, and the difference is not volume — it is that you need to know how it is going while it is happening, and be able to stop.

The two version fields

Everything about fleet rollout follows from one idea: a device has a version you asked for and a version it is running, and they are separate fields.

Setting the first is the whole of “starting an upgrade”. The device notices, at its next check-in, that the two disagree, downloads the version it has been asked for, switches, and reboots. Afterwards it reports the new version and the fields agree again.

This means a rollout is not a job the platform runs. There is no batch to monitor, no operation to poll. The desired state is written on each device, and each device converges on it independently, whenever it happens to be reachable. That is what makes it safe on a flaky network — and it is also why you need to measure progress by comparing the two fields.

Upgrade progress is also reported as a condition named OSReady in the device’s status, which is the more precise signal when you want to know whether a particular device is mid-upgrade, not simply behind.

Measuring a rollout

Both fields are filterable, so you can count any population you care about without enumerating devices.

# Devices asked for 2.1.2 but not yet running it: still in flight
cuttle devices list devices \
  --parent projects/$PROJECT/regions/- \
  --filter 'spec.osVersion="2.1.2" AND status.deviceInfo.osVersion!="2.1.2"' \
  --field-mask status.deviceInfo.osVersion \
  -o json

# Devices already there
cuttle devices list devices \
  --parent projects/$PROJECT/regions/- \
  --filter 'status.deviceInfo.osVersion="2.1.2"' -o json

# Devices never asked: not yet in the rollout
cuttle devices list devices \
  --parent projects/$PROJECT/regions/- \
  --filter 'spec.osVersion!="2.1.2"' -o json

Those three numbers are your rollout dashboard. The first should trend to zero; if it plateaus, the devices remaining in it are the ones to investigate.

Stage it with labels

Do not set the target version on the whole fleet at once. Use labels to define waves and promote one at a time.

Label devices according to how much you are willing to break:

Then promote a wave by setting the version on that label’s devices only. For a handful of devices the dashboard is quickest — set the OS version on each as described in OS upgrades. For a wave of any size, drive it through the API:

DEVICES=$(curl -s -H "Authorization: Bearer $API_KEY" --get \
  --data-urlencode 'filter=metadata.labels CONTAINS "wave:canary"' \
  --data-urlencode 'view=NAME' \
  "https://devices.apis.edgelq.com/v1/projects/$PROJECT/regions/-/devices" \
  | jq -r '.devices[].name')

for DEVICE in $DEVICES; do
  curl -s -X PUT -H "Authorization: Bearer $API_KEY" \
    -H 'Content-Type: application/json' \
    --data '{"spec":{"osVersion":"2.1.2"}}' \
    "https://devices.apis.edgelq.com/v1/${DEVICE}?updateMask=spec.osVersion"
done

Between waves, wait long enough for the devices to have actually checked in and converged — and confirm they did. A canary that has not reported the new version is not a passing canary; it is an unknown.

Choose canaries you can recover by hand. The point of a canary is that its failure is cheap, and a device you would have to send an engineer to is not cheap.

Devices that are offline

A device that is switched off or disconnected when you set the version does not miss the upgrade. It converges when it comes back, however much later that is, because the desired version is stored on its record rather than pushed to it.

This is usually what you want, with one consequence to plan for: a device offline for two months will upgrade the moment it reconnects, possibly skipping several versions and at an unpredictable time. If some of your fleet is intermittent, expect a long tail of stragglers instead of a rollout that finishes cleanly, and check for an upgrade path constraint if the version gap is large.

What to check before starting

• Confirm the version is available for every device type in the wave. OS versions are published per device type, and a mixed fleet may not have the same version available everywhere.
• Check upgrade path constraints. Some versions require a minimum current version, which matters most for the stragglers above.
• Confirm connectivity, including through any proxy — devices download the image themselves.
• Check that the devices manage their own OS. Devices running the droplet agent on your own Linux rather than SPEKTRA Edge OS do not self-upgrade; OS management is disabled for them, and their OS is your responsibility. They will not participate in a rollout, so exclude them from your counts; they are not stuck.

Stopping and rolling back

There is no “cancel rollout” button, because there is no rollout object — so stopping one means changing the desired state.

To stop it spreading, stop promoting waves. Devices you have not yet retargeted are unaffected.

To reverse devices already upgraded, set spec.osVersion back to the previous version. Because each device retains the version it was running, the rollback does not require a download and is quick.

To reverse devices mid-upgrade, set the version back and let them converge. A device that fails to start a new version can also fall back on its own, as described in OS upgrades.

Watching instead of polling

For anything beyond a few hundred devices, do not poll the counts on a loop. Open a watch on the device collection with a field mask covering just the two version fields and the OSReady condition, and update your view as changes arrive. Rollout progress is exactly the case watch exists for: a large collection where a small, unpredictable subset changes at a time.

Alert on the rollout, not just the fleet

The failure that costs you is not a device that fails to upgrade — you will see that in the counts. It is a device that upgrades successfully and then misbehaves.

Before starting, make sure you would notice: connectivity alerts so a device that upgrades and never returns raises something, and application health checks so a device that comes back with its workload broken is visible. Rolling out without those means the canary phase cannot tell you anything, because there is nothing watching it.

Related

• OS upgrades — the single-device flow and what survives an upgrade.
• Monitoring and alerts — noticing when an upgraded device goes wrong.