AI Alerting

Let’s learn how to use AI Alerting on SPEKTRA Edge to detect problems without hand-tuned thresholds, and to have an AI agent investigate and remediate alerts for you.

Managing remote devices means you can’t watch every dashboard all the time. AI Alerting is designed for exactly this: it notices unusual behavior on its own, figures out what is likely wrong, and either fixes it or escalates to a human with a written explanation.

This page has two parts:

1. Concepts — what AI Alerting is, its architecture, and how a fired alert is handled end to end. Read this first to build a mental model.
2. Setup — a hands-on walkthrough that builds one policy, field by field. Our example detects abnormal CPU behavior on devices and lets the AI agent restart the offending pod after your approval.

Concepts: what AI Alerting is and how it works

This part explains the moving pieces and how they fit together. No configuration yet — that’s Setup.

AI Alerting vs. Alerts

SPEKTRA Edge has two complementary alerting areas in the dashboard sidebar:

This page covers AI Alerting. You’ll find it under the AI Alerting entry in the sidebar (the AI Alerting overview page).

Architecture and end-to-end flow

AI Alerting reuses the same building blocks as classic Alerts — an alerting policy that groups conditions and notification channels — and adds two things on top:

1. Smarter conditions. In addition to fixed thresholds, a condition can use adaptive thresholds or anomaly detection. These are two different things — see Adaptive thresholds vs. anomaly detection below.

2. An AI agent that handles fired alerts. When an alert fires, the agent investigates, decides what is going on, and takes an action — see How the AI agent handles alerts below.

Putting those together, this is the end-to-end flow. However the alert is triggered — a fixed threshold, an adaptive threshold, or an anomaly detector — it enters the same AI handling pipeline: the agent gathers context (including your supporting docs and queries), decides what is going on, and then the outcome flows into whatever you configured — a notification, an automatic or approved remediation, or an escalation to a human.

flowchart TD
    %% --- Detection ---
    FT["Fixed threshold"]
    AT["Adaptive threshold"]
    AD["Anomaly detection"]
    FT --> FIRE
    AT --> FIRE
    AD --> FIRE
    FIRE(["Alert fires<br/>(after 'Raise after')"])

    %% --- Investigation ---
    FIRE --> INV["AI agent investigates"]
    LIVE["Metrics, logs,<br/>resource state"] --> INV
    DOCS["Supporting docs"] --> INV
    SQ["Supporting queries"] --> INV
    SSH["Read-only checks over SSH<br/>(if connectivity enabled)"] --> INV

    %% --- Decision ---
    INV --> DEC{"AI agent decides"}
    DEC -->|"transient blip"| IGN["Ignore"]
    DEC -->|"over-sensitive"| ADJ["Adjust condition"]
    DEC -->|"fixable"| REM["Remediation:<br/>Fix in SSH or Reboot"]
    DEC -->|"cannot resolve"| ESC["Escalate to operator"]

    %% --- Remediation approval gate ---
    REM --> AA{"Auto-accept<br/>enabled?"}
    AA -->|"no"| WAIT["Await operator approval"]
    AA -->|"yes"| APP["Apply automatically"]
    WAIT -->|"approved"| APP
    APP --> RECHECK{"Still firing?"}
    RECHECK -->|"yes"| ESC

    %% --- Outcomes notify channels ---
    IGN --> NOT
    ADJ --> NOT
    APP --> NOT
    ESC --> NOT
    NOT[["Notify channels:<br/>Slack / email / webhook"]]

Adaptive thresholds vs. anomaly detection

Both go beyond a fixed threshold, but they answer different questions.

Adaptive thresholds are still thresholds — an upper and/or lower bound. The only difference from a fixed threshold is that the platform computes the bound for you from recent history (by default, the last week) instead of you typing a fixed number. It looks at the range the metric actually reached, adds some buffer, and alerts when the value crosses that band. The question it answers is “is the value too high or too low right now?” — it just keeps the “too high / too low” line up to date as the device’s normal level drifts.

Anomaly detection is a learned model of behavior, not a bound. A small AI model (an LSTM autoencoder) is trained on the metric’s history and learns its normal shape over time — including how it rises and falls through the day. It then flags readings that don’t match that learned pattern. The question it answers is “does this look like how this device normally behaves?” — so it can catch a value that is unusual even though it never leaves the normal band (for example, CPU sitting flat at 40% overnight when it normally idles near 5%).

They also complement each other: while an anomaly detector is still training (or if it isn’t defined), the adaptive thresholds keep watch, so you have coverage from day one and sharper detection once the model is ready. A common setup is to enable adaptive thresholds and one or more anomaly detectors on the same condition.

How the AI agent handles alerts

When an alert fires — from any of the detection methods above — the AI agent runs a short loop: investigate → decide → act.

What the agent looks at

To diagnose an alert, the agent pulls together several sources of context:

• Live signals — the device’s metrics, logs, and resource state around the time of the alert.
• Supporting queries — predefined metric, log, or resource lookups attached to the policy. They run automatically when an alert fires and their results are handed to the agent, so it always starts with the data you consider relevant (see Add supporting queries).
• Supporting documents — natural-language references the agent reads while diagnosing (see the next section).
• Read-only device checks — only if you enabled agent connectivity, the agent may run safe, read-only commands on the device over the secure SSH tunnel.

How supporting documents are used

Supporting documents are natural-language references — runbooks, operational notes, known-issue write-ups — that you attach to a policy or condition. The agent reads them as context while diagnosing an alert; they are not code and they don’t run. They shape the agent’s understanding and its decision.

For example, a short runbook that says:

If the processor service spikes CPU right after a config push, it’s a known issue — restarting the pod is safe and resolves it.

lets the agent recognize the situation, explain it in its diagnosis, and confidently choose the Reboot remediation instead of escalating. Without that note, the same symptom might be escalated to a human because the agent has no way to know the restart is safe.

Good supporting documents are specific and action-oriented: they describe a symptom, what it usually means, and what a safe response is. They’re the main way you transfer your team’s operational knowledge to the agent. You attach them during setup (see Attach supporting documents).

What the agent decides and does

After investigating, the agent chooses one outcome:

1. Ignore — the cause looks transient and harmless.
2. Adjust — the condition was too sensitive; the platform retunes it.
3. Remediate — apply a Fix in SSH command or Reboot the affected pod. If auto-accept is off, the agent only proposes the fix and waits for an operator to approve it; if it’s on, the fix is applied automatically. If a remediation is applied and the alert keeps firing, the agent escalates.
4. Escalate — it can’t resolve the issue and hands off to a human.

Every step is recorded as an AI diagnosis note on the alert, so you can read how the agent reached its conclusion.

Guardrails: keeping the agent safe

A few guardrails are built in and worth knowing:

• The agent only connects to a device when you enable agent connectivity; otherwise it works from collected metrics and logs alone.
• You can restrict which command-line tools the agent may use during investigation, limiting it to a known-safe set.
• Remediations are limited to the kinds you allow (Fix in SSH and/or Reboot), and — unless you enable auto-accept — require your approval before anything runs.

Setup: create and configure a policy

Now the hands-on part. We’ll build the CPU example from the top of the page: detect abnormal device CPU, and let the agent restart the offending pod after your approval.

What you need

• an access to the SPEKTRA Edge dashboard
• an active project
• a device provisioned under the project (and, for the reboot example, an application running as a pod on that device)
• optionally, one or more notification channels already created

Create the policy

Go to AI Alerting → Policies in the sidebar to open the AI Alerting overview page, then click Create policy in the top-right corner.

The Create policy page is a single form. We’ll walk through it top to bottom.

1. Choose a template and name the policy

• Template — start from a predefined set of conditions (for example, a base device or pod template). Templates come with sensible conditions already filled in, which you can tune later. For our example, pick a device template. Note that the template cannot be changed after the policy is created.
• Display name — a human-readable name, for example Device CPU anomaly.
• Description — optional free text describing what the policy is for.

2. Attach supporting documents and notification channels

• Supporting docs — attach the runbooks or notes the agent should read while diagnosing an alert (this is the feature described in How supporting documents are used). It’s optional but makes the agent noticeably better.
• Notification channels — select the same Slack, email, or webhook notification channels you use elsewhere. You can leave this empty and add channels later.

3. Enable the AI agent and pick remediations

This is the heart of AI Alerting. Turn on AI agent enabled to switch on AI-powered analysis and remediation for alerts raised by this policy. Two more controls appear once it is on:

For our example, enable the AI agent, enable connectivity, leave auto-accept off, and select both Fix in SSH and Reboot as allowed remediations.

4. Set the processing location and resource

• Remediations and Processing location sit side by side on the form. For Processing location, choose Backend (in the cloud) or Edge (on the device itself). Edge processing keeps detection working locally even when connectivity to the cloud is intermittent; Backend is the right choice for metrics that only make sense centrally (such as connectivity). This field cannot be changed after creation. For our anomaly-detection example, keep it on Backend — on-device anomaly detection is coming soon, but today anomaly models run in the backend (adaptive and fixed thresholds work in either location).
• Resource — the resource type this policy monitors, for example Device or Pod. The Reboot remediation is only available when the policy can identify a pod.

5. (Optional) Add supporting queries

Supporting queries are the predefined lookups described in What the agent looks at. They are gathered automatically and handed to the agent as context whenever an alert fires. Click Add query and choose a Type:

• Time series query — a metric lookup with a filter and aggregation (alignment period, per-series aligner, cross-series reducer, group-by fields).
• Log query — a log lookup with a filter.
• REST get / REST list query — fetch a resource (or a list of resources) from a SPEKTRA Edge service, using an endpoint, path, view, and field mask.

Filters use templates, so you can reference the alert’s labels with <label_key> placeholders (for example resource.labels.device_id="<device_id>"). The built-in <project_id> and <region_id> placeholders are always available.

When the form is complete, click Create to save the policy.

Add conditions

A policy without conditions never fires. Open your new policy from the AI Alerting overview page and add a time series condition (the template you picked may have already added some, which you can edit or delete).

Each condition defines one or more queries (metric filter + aligner + reducer) and how they turn into alerts. You can use fixed thresholds, adaptive thresholds, and anomaly detection — together if you like. For the difference, see Adaptive thresholds vs. anomaly detection.

Threshold and adaptive thresholds

Set a Threshold to alert when a value crosses a fixed bound. To make that bound adapt to the device’s own history instead of being a fixed number, turn on:

• Auto adapt upper — the platform maintains the upper bound for you.
• Auto adapt lower — the platform maintains the lower bound for you.

With adaptive thresholds you can still set hard guard rails (a maximum upper and a minimum lower bound) so the adaptive value never becomes too tolerant or too sensitive. The platform derives the adaptive bound from historic data (by default the last week) plus a configurable buffer.

Anomaly alerting

Add an entry to the Anomaly alerting list to catch deviations from the metric’s learned pattern. This trains a small AI model on the metric’s history, so it needs a training period before it starts detecting. The key fields are:

• Analysis window — the sliding window of data the model looks at each time.
• Training period — how much history the model learns from before it starts flagging deviations (at least one day).

A good starting pattern is one wide, coarse detector (a long analysis window with a larger step) to catch slow drifts, plus one narrow, fine detector to catch sudden spikes.

Raise after / silence after

For both methods you can set:

• Raise after (sec) — how long the condition must hold before an alert fires (noise reduction).
• Silence after (sec) — how long after violations stop before the alert clears.

Save the condition when you’re done.

Enable the policy

Back on the AI Alerting overview page, slide the Enabled switch on for your policy. Detection (and, once alerts fire, the AI agent) is now live.

Reading alert state

On the AI Alerting overview page, the alerts tables show the handling state:

Open an alert to see the full AI diagnosis, the proposed remediation, and the controls to Approve a remediation or Acknowledge the alert.

Choosing which events notify you

For AI Alerting you can pick exactly which lifecycle events send a notification on each channel, including:

• New firing — a new alert started.
• AI escalated to operator — the agent needs a human.
• AI remediation awaiting approval — a fix is proposed and waiting for you.
• AI remediation applied / Operator remediation applied — a fix was applied by the agent or by a person.
• Stopped firing — the alert cleared.

Next step

Combine AI Alerting with well-chosen alert metrics for broad coverage, and route everything through your notification channels so you hear about remote devices the moment something looks wrong.