Operations

• 1: OS upgrades
• 2: Rolling out across a fleet
• 3: Reboot, shutdown & recovery
• 4: Health checks
• 5: Status & indicators
• 6: Boot & shutdown hooks
• 7: Running workloads
• 8: Transfer files

1 - OS upgrades

You upgrade a device’s operating system by setting the OS version you want; the device downloads the new version, switches to it, and reboots. Because the system image is separate from your configuration and data, your setup is preserved across the change, and you can roll back to a previous version.

How upgrades work

1. You set the target OS version for a device (or a group of devices) from the dashboard.
2. The device downloads the requested version and prepares it alongside the current one.
3. The device switches to the new version and reboots.
4. After reboot, the device reports its new version, and your configuration and data are intact.

A device keeps more than one OS version on disk — the one it is running and the one it can fall back to — which is what makes switching versions and rolling back fast and safe.

flowchart TB
    subgraph BEFORE["Before"]
        direction TB
        A1["System image slot A<br/><b>running 2.1.0</b>"]
        B1["System image slot B<br/><i>previous version</i>"]
    end

    subgraph AFTER["After upgrade and reboot"]
        direction TB
        A2["System image slot A<br/><i>kept for rollback</i>"]
        B2["System image slot B<br/><b>running 2.1.2</b>"]
    end

    KEEP["Configuration + data<br/><i>untouched throughout</i>"]

    BEFORE -->|"1. download 2.1.2 into the spare slot<br/>2. switch<br/>3. reboot"| AFTER
    KEEP -.-> BEFORE
    KEEP -.-> AFTER

Rolling back is the same operation in reverse: the previous version is still in the other slot, so the device does not need to download anything.

Release channels and compatibility

• OS versions are published per device type, so a device is only offered versions compatible with its hardware.
• Some upgrades may require a minimum current version before you can move to the target, to ensure a supported upgrade path.

Upgrade a device from the dashboard

To go through this section, you need:

• access to the SPEKTRA Edge dashboard
• an active project
• a device provisioned under the project

Go to the Device overview page of the target device under your project by clicking the name of the device on the Project overview page.

Open the Update device details page by clicking the Edit detail option of the device menu option.

Select the OS version, 2.1.2 in this example, and click Save to trigger the OS upgrade process.

The upgrade operation will start automatically and you will see the OS version transition information on the Device overview page.

After some time, you will see the device Offline, which indicates the device is restarting with the new OS version.

You will see the device up and running with the new OS version on the Device overview page once it boots up.

Rolling back

If a new version misbehaves, you can roll a device back to the version it was running before, or to its original factory version. Because the previous version is retained on the device, rollback does not require re-downloading it.

Set the desired (earlier) version the same way you set an upgrade; the device switches back and reboots.

Select the downgrade OS version, 2.1.0 in this example, and click Save to trigger the OS downgrade process.

The OS downgrade process automatically starts once you click the Save button of the Update device details page.

After a minute or so, you will see the device status Offline with the OS version transition information on the Device overview page.

You will see the device up and running with the new OS version once it boots up.

Encrypted devices

Managed upgrades preserve automatic unlock on encrypted devices — no manual action is required to keep an encrypted device unlocking across an upgrade.

If an upgrade does not complete

• Confirm the device has connectivity to download the new version (including through any proxy).
• A device that fails to start a new version can fall back to the retained previous version.
• See the troubleshooting playbooks for upgrade issues.

Related

• SPEKTRA Edge OS release notes — what changed in each OS version, so you can decide which version to move a device to.

2 - Rolling out across a fleet

Upgrading one device from the dashboard is straightforward. Upgrading four thousand is a different task, and the difference is not volume — it is that you need to know how it is going while it is happening, and be able to stop.

The two version fields

Everything about fleet rollout follows from one idea: a device has a version you asked for and a version it is running, and they are separate fields.

Setting the first is the whole of “starting an upgrade”. The device notices, at its next check-in, that the two disagree, downloads the version it has been asked for, switches, and reboots. Afterwards it reports the new version and the fields agree again.

This means a rollout is not a job the platform runs. There is no batch to monitor, no operation to poll. The desired state is written on each device, and each device converges on it independently, whenever it happens to be reachable. That is what makes it safe on a flaky network — and it is also why you need to measure progress by comparing the two fields.

Upgrade progress is also reported as a condition named OSReady in the device’s status, which is the more precise signal when you want to know whether a particular device is mid-upgrade, not simply behind.

Measuring a rollout

Both fields are filterable, so you can count any population you care about without enumerating devices.

# Devices asked for 2.1.2 but not yet running it: still in flight
cuttle devices list devices \
  --parent projects/$PROJECT/regions/- \
  --filter 'spec.osVersion="2.1.2" AND status.deviceInfo.osVersion!="2.1.2"' \
  --field-mask status.deviceInfo.osVersion \
  -o json

# Devices already there
cuttle devices list devices \
  --parent projects/$PROJECT/regions/- \
  --filter 'status.deviceInfo.osVersion="2.1.2"' -o json

# Devices never asked: not yet in the rollout
cuttle devices list devices \
  --parent projects/$PROJECT/regions/- \
  --filter 'spec.osVersion!="2.1.2"' -o json

Those three numbers are your rollout dashboard. The first should trend to zero; if it plateaus, the devices remaining in it are the ones to investigate.

Stage it with labels

Do not set the target version on the whole fleet at once. Use labels to define waves and promote one at a time.

Label devices according to how much you are willing to break:

Then promote a wave by setting the version on that label’s devices only. For a handful of devices the dashboard is quickest — set the OS version on each as described in OS upgrades. For a wave of any size, drive it through the API:

DEVICES=$(curl -s -H "Authorization: Bearer $API_KEY" --get \
  --data-urlencode 'filter=metadata.labels CONTAINS "wave:canary"' \
  --data-urlencode 'view=NAME' \
  "https://devices.apis.edgelq.com/v1/projects/$PROJECT/regions/-/devices" \
  | jq -r '.devices[].name')

for DEVICE in $DEVICES; do
  curl -s -X PUT -H "Authorization: Bearer $API_KEY" \
    -H 'Content-Type: application/json' \
    --data '{"spec":{"osVersion":"2.1.2"}}' \
    "https://devices.apis.edgelq.com/v1/${DEVICE}?updateMask=spec.osVersion"
done

Between waves, wait long enough for the devices to have actually checked in and converged — and confirm they did. A canary that has not reported the new version is not a passing canary; it is an unknown.

Choose canaries you can recover by hand. The point of a canary is that its failure is cheap, and a device you would have to send an engineer to is not cheap.

Devices that are offline

A device that is switched off or disconnected when you set the version does not miss the upgrade. It converges when it comes back, however much later that is, because the desired version is stored on its record rather than pushed to it.

This is usually what you want, with one consequence to plan for: a device offline for two months will upgrade the moment it reconnects, possibly skipping several versions and at an unpredictable time. If some of your fleet is intermittent, expect a long tail of stragglers instead of a rollout that finishes cleanly, and check for an upgrade path constraint if the version gap is large.

What to check before starting

• Confirm the version is available for every device type in the wave. OS versions are published per device type, and a mixed fleet may not have the same version available everywhere.
• Check upgrade path constraints. Some versions require a minimum current version, which matters most for the stragglers above.
• Confirm connectivity, including through any proxy — devices download the image themselves.
• Check that the devices manage their own OS. Devices running the droplet agent on your own Linux rather than SPEKTRA Edge OS do not self-upgrade; OS management is disabled for them, and their OS is your responsibility. They will not participate in a rollout, so exclude them from your counts; they are not stuck.

Stopping and rolling back

There is no “cancel rollout” button, because there is no rollout object — so stopping one means changing the desired state.

To stop it spreading, stop promoting waves. Devices you have not yet retargeted are unaffected.

To reverse devices already upgraded, set spec.osVersion back to the previous version. Because each device retains the version it was running, the rollback does not require a download and is quick.

To reverse devices mid-upgrade, set the version back and let them converge. A device that fails to start a new version can also fall back on its own, as described in OS upgrades.

Watching instead of polling

For anything beyond a few hundred devices, do not poll the counts on a loop. Open a watch on the device collection with a field mask covering just the two version fields and the OSReady condition, and update your view as changes arrive. Rollout progress is exactly the case watch exists for: a large collection where a small, unpredictable subset changes at a time.

Alert on the rollout, not just the fleet

The failure that costs you is not a device that fails to upgrade — you will see that in the counts. It is a device that upgrades successfully and then misbehaves.

Before starting, make sure you would notice: connectivity alerts so a device that upgrades and never returns raises something, and application health checks so a device that comes back with its workload broken is visible. Rolling out without those means the canary phase cannot tell you anything, because there is nothing watching it.

Related

• OS upgrades — the single-device flow and what survives an upgrade.
• Monitoring and alerts — noticing when an upgraded device goes wrong.

3 - Reboot, shutdown & recovery

This page covers power actions you can take remotely and the options for recovering a device that is not behaving as expected.

Reboot and shutdown

You can reboot or shut down a device remotely from the dashboard or with the cuttle command-line tool. These actions are carried over the device’s connection to the platform and require no inbound access to the device.

In the dashboard, open the device’s Device overview page, click the vertical triple dots next to the page title, and choose Reboot or Shutdown.

From the command line:

cuttle devices reboot projects/your-project/regions/us-west2/devices/pi03
cuttle devices shutdown projects/your-project/regions/us-west2/devices/pi03

Both commands warn you and wait for you to type yes, because workloads on the device — and anything depending on them — go down with it. Pass --force to skip the prompt in scripts.

A device that loses power or is reset returns to its configured state on next boot, because configuration and data are persistent and the system image is read-only.

Automatic recovery

The OS is designed to keep itself running unattended:

• It restarts automatically after an unexpected fault rather than waiting at an error.
• Workloads configured to run are brought back up after a reboot.
• If a network configuration change does not produce a working connection, the device can fall back to known-good network settings — see connection fallback.

Recovery options

When a device needs hands-on recovery, the options in increasing order of impact:

See User accounts for password reset and Disk encryption for encrypted-device recovery.

When a device is offline

If a device is offline and unreachable through the platform, work through the troubleshooting playbooks — most often the cause is connectivity, time, or a network configuration change.

Related

• Status indicators — reading what state a device is in.
• OS upgrades — how upgrades roll back if a device fails to boot.
• Disk encryption — recovery on an encrypted device.

4 - Health checks

Health checks let a device continuously verify that it — or something running on it — is healthy, and report the result to the platform. You define health checks from the dashboard (or the API).

Types of checks

You can define host-level checks of several types:

How results are used

Each check’s result contributes to the device’s reported conditions, which you can see on the device overview page. Use health checks to detect problems such as a dependent service being unreachable, a required peripheral missing, or a local endpoint failing — and to surface them centrally instead of discovering them per device.

Recommendations

• Check the things that matter for the device’s role (for example, the local application endpoint, or a sensor’s device file).
• Keep checks lightweight and give them sensible intervals so they reflect real health without adding load.

Related

• Status & indicators — how health surfaces on the device and dashboard.
• Alerts — get told when a check starts failing.
• Operations — the rest of this section.

5 - Status & indicators

A device reports its state both to the platform and, on supported hardware, through a physical LED. This page explains how to read each.

Connection status

On the platform, a device is shown as:

Device information and metrics

The device reports details and resource usage you can view on the dashboard, including:

• OS version, hardware details, disks, and network interfaces
• CPU, memory, and storage usage
• temperature

These help you spot devices that are low on storage, running hot, or pinned at high resource usage.

LED indicator

On hardware with a status LED, the device signals its lifecycle and health through the LED so an on-site technician can tell at a glance what state a device is in — for example, still starting up, working normally, or needing attention. Use the LED for quick on-site triage, and the dashboard for detail.

The LED is driven by SPEKTRA Edge OS, so it is not available on a device running the droplet agent installed from a .deb package. Everything else on this page — connection status, device information, metrics, and health check conditions — is reported by the agent and works on either installation. See Install the droplet agent.

Conditions from health checks

Health checks you define contribute to the device’s reported conditions, giving you application- and peripheral-level health alongside the built-in status.

Related

• Health checks — defining the checks that feed device conditions.
• Reboot and recovery — what to do when a device is stuck or offline.
• Troubleshooting — playbooks for the common failures.

6 - Boot & shutdown hooks

Hooks let you run your own scripts at defined points in the device lifecycle — for example to prepare hardware, mount a special device, or clean up on shutdown. They are the supported way to extend device behavior without modifying the read-only system image.

Available hooks

You can provide a script for each of these points:

How to provide a hook

Place your script at the path for the point you want to act on, and make it executable — a script that is not executable is skipped silently:

sudo install -m 755 my-script /isodevice/config/hooks/on-boot

Because /isodevice/config is configuration storage, the script persists across reboots and OS upgrades, and runs automatically at the corresponding point.

Guidelines

• Keep hooks fast and resilient. The on-boot and after-agent hooks are allowed five minutes each before they time out, and a slow boot hook delays startup. Make hooks idempotent and have them exit cleanly.
• Do not rely on modifying the system image. Use hooks to act on configuration and data, not to patch the OS — image changes do not persist.
• Log from your hook so you can see what it did via logging.
• Test on one device first before distributing a hook to a fleet.

Related

• How persistence works — the other settings supplied as files, and what survives an upgrade.
• Reboot, shutdown & recovery — when hooks run.
• Operations — the rest of this section.

7 - Running workloads

Devices run your applications as containers. You deploy and manage them from the SPEKTRA Edge platform; this page covers only how workloads interact with the OS. For the full application deployment workflow, see the Deploy guide.

How workloads use device storage

• Container images and workload data live on the device’s data area, which uses most of the disk.
• A workload can mount paths from the device into its containers when it needs access to host data or hardware.
• Credentials a workload needs (such as registry pull secrets) are delivered through the platform rather than stored in the image.

Networking for workloads

• Any network ports a workload exposes are determined by the workload itself, not by the OS — open them in your site firewall as needed. See Connectivity & ports.
• Workload traffic honors the device’s proxy configuration.

When storage runs low

Because container images and data share the data area, a device that fills its data area can fail to start new workloads. Watch storage usage in device metrics, and keep logging levels reasonable so logs do not consume the space your workloads need.

Start, stop, and restart a workload

You can restart a workload from the dashboard, using the Restart option on the application’s menu — see Deploy an application. From the cuttle command line you can do the same, and also act on a single service inside a pod:

# restart a whole pod
cuttle applications restart projects/your-project/regions/us-west2/pods/nginx

# start, stop, or restart one service in a pod
cuttle applications pods stop \
  projects/your-project/regions/us-west2/pods/nginx web

The command reaches the workload through the device the pod runs on, so the device has to be online. Stopping a pod this way stops its containers on the device; it does not remove the deployment, so the workload is brought back up when the device reboots.

Lifecycle across reboots and upgrades

Workloads configured to run are brought back up after a reboot, and persist across OS upgrades because their images and data are on persistent storage.

Related

• Deploy an application — how workloads get onto a device.
• OS upgrades — how workloads survive a version change.
• Operations — the rest of this section.

8 - Transfer files

You can copy files to and from a device through the platform, over the outbound connection the device already keeps open. Nothing listens on the device for this, so file transfer works on devices behind NAT or a restrictive firewall, exactly like a remote shell does.

Both the dashboard and the cuttle command-line tool use the same transfer service on the device, so it does not matter which one you use.

Before you copy files onto a device

A device’s system image is read-only, and anything you write outside the persistent areas is gone after the next reboot. Choose the destination accordingly:

See how persistence works for the full storage model.

From the dashboard

Open the device’s Device overview page and click the vertical triple dots next to the page title.

• Upload files — enter the destination directory in Path on device, then select or drag in one or more files. The files keep their own names under that directory.
• Download file — enter the full path of a single file in Path on device. The file downloads through your browser when the transfer completes.

Both actions need the connect to device permission (services/proxies.edgelq.com/permissions/projects.connect), the same permission that governs the remote terminal.

From the command line

cuttle devices scp takes the same two arguments as ordinary scp, with the device side written as <device-name>:<path>. Whichever argument carries the device name decides the direction — you cannot copy between two devices.

Upload a file:

cuttle devices scp ./hook.sh \
  projects/your-project/regions/us-west2/devices/pi03:/isodevice/config/hooks/

Download a file:

cuttle devices scp \
  projects/your-project/regions/us-west2/devices/pi03:/isodevice/data/report.csv \
  ./report.csv

Add -r to transfer a whole directory, and -v to print per-file progress:

cuttle devices scp -r -v ./bundle \
  projects/your-project/regions/us-west2/devices/pi03:/isodevice/data/

How paths are interpreted

On upload, a trailing slash on the remote path means “into this directory”:

• …:/isodevice/data/ keeps the local file name.
• …:/isodevice/data/report.csv renames the file on the device.
• With -r, a trailing slash creates the local directory itself on the device; without one, only its contents are copied into the remote path.

The remote path of an upload must be absolute — it has to begin with /.

On download, what matters is whether the local path already exists as a directory:

• If it does, the file is written into it under its remote name.
• If it does not, the local path is used as the output file name.
• With -r, an existing local directory receives the remote directory beneath it; a local path that does not exist yet is created and becomes the root of the downloaded tree.

Collecting files for support

To gather the logs and system state an investigation usually needs, use cuttle devices collect-debug-bundle rather than copying files one at a time.

Related

• Remote access — administering a device without inbound ports.
• Boot & shutdown hooks — the most common reason to upload a file.
• Storage — what lives where on the device’s disk.