SPEKTRA Edge Logging Service API

Understanding the Logging service API.

SPEKTRA Edge Logging service is a structured data store that provides the ability to store arbitrary data in the form of logs series with timestamps and user-defined metadata. To achieve that, we are using google.protobuf.Struct OR google.protobuf.Any, a well-known type used to store any JSON-like (or protobuf) data. It allows users not to worry about defining any proto message. We can store any data type such as string, int, float, bytes, map, etc.

Use cases for the logging service include:

system or application logs
event logs
tracing
core / debug dump
blob data

The stored logs can be queried based on timestamps and also filtered based on user-defined metadata fields.

Full API Specifications (with resources):

Version v1alpha2
Version v1

Resources

Log Descriptors

A log descriptor is the context of all logs created. It describes log, its metadata, service, API version, region, and the service-defined labels, for example process name can be an example.

A log descriptor also privdes indexing patterns for its labels. Each promoted index set is defined with a log descriptor is combined with the following fields:

a scope (project, organization, or service parent)
a service
a log descriptor

When querying, it is required that users specify parent, service, and log descriptor fields, plus any promoted fields as defined by the log descriptor (field promotedLabelKeySets). If there is an empty set among available lists, it means that logs can be queried by scope, service and log descriptor fields only.

Log Entries

A log entry is a combination of timestamps with underlying data items. Log entries sharing the same scope, service, version, log descriptor and labels are considered to form a single Log instance.

Storing and querying logs

As an example, we will define and create a log descriptor for storing IoT device logs:

logDescriptor:
  name: projects/<projectID>/logDescriptors/iot.app.org/syslog
  displayName: IoT Device syslog
  description: IoT Device syslog
  labels:
  - key: module
    description: process/module/component name
  - key: device_id
    description: device ID
  - key: log_level
    description: Log level (debug, info, warn, error etc)
  promotedLabelKeySets:
  - labelKeys: ["log_level"]
  - labelKeys: ["device_id"]
  - labelKeys: ["module", "device_id"]

Create the log descriptor using cli:

cuttle logging create log-descriptor -f logdescriptor.yaml

Sample code for storing logs:

import (
   "context"
   "fmt"
   "time"


   "google.golang.org/protobuf/types/known/structpb"
   "google.golang.org/protobuf/types/known/timestamppb"


   clog "github.com/cloudwan/edgelq-sdk/logging/client/v1/log"
   log_common "github.com/cloudwan/edgelq-sdk/logging/common/v1"
   rlog "github.com/cloudwan/edgelq-sdk/logging/resources/v1/log"
   rlog_descriptor "github.com/cloudwan/edgelq-sdk/logging/resources/v1/log_descriptor"


   "github.com/cloudwan/edgelq-sdk/examples/utils"
)

const (
   logDescriptorName  = "iot.app.org/syslog"
   projectID          = "development"
   demoDeviceID       = "temperature-sensor-101"
   sampleModule       = "sensor-phy"
   sampleLogLevel     = "info"
   controllerEndpoint = "logging.stg01b.edgelq.com:443"
   credsFile          = "/etc/conf/edgelq-credentials.json"
)

func create_log_client(
   ctx context.Context,
) clog.LogServiceClient {
   grpcConn := utils.Dial(ctx, controllerEndpoint, "", credsFile) // Panics
   return clog.NewLogServiceClient(grpcConn)
}

func storeLogs(
  ctx context.Context,
  logCli clog.LogServiceClient,
  payloadString string,
  logTime time.Time,
) error {
   logData := map[string]interface{}{
       "sensor-dev": "/dev/zz",
       "process":    "sensor-agent",
       "message":    "Initialised sensor device",
   }

   payload, err := structpb.NewStruct(logData) // Check to rules below to avoid errors
   if err != nil {
       return err
   }

   logEntry := &rlog.Log{
       Service:       "iot.app.org",
       Version:       "v2",
       LogDescriptor: rlog_descriptor.NewNameBuilder().SetProjectId(projectID).SetId(logDescriptorName).Reference(),
       Labels: map[string]string{
           "device_id": demoDeviceID,
           "module":    sampleModule,
           "log_level": sampleLogLevel,
       },
       Time:    timestamppb.New(logTime),
       Payload: payload,
   }

   _, err = logCli.CreateLogs(ctx, &clog.CreateLogsRequest{
       Parent: rlog.NewNameBuilder().SetProjectId(projectID).ParentReference(),
       Logs:   []*rlog.Log{logEntry},
   })
   if err != nil {
       return fmt.Errorf("Failed to upload logs to the server: %w", err)
   }
   return nil
}

Note however, that it is also possible to provide the “Name” field in the rlog.Log value, but the ID must be equal to the value provided by the earlier CreateLogs request. Labels, LogDescriptor, Service, and Version can be skipped however then, reducing request size. It is still required to send at least one CreateLogs request with full meta information and labels, to have Name allocated.

With some logs submitted, we can try to query.

Log query takes three inputs:

Parents

Project ID or Organization ID
Interval

Start time and end time for the logs request
Filter

Filter can combine multiple conditions based on the labels defined in the log descriptor. Log Descriptor and service name are mandatory filters. User-defined labels are optional filters.

In this example from Quickstart though, at least one user label is mandatory with a filter, because we have promoted labels:

promotedLabelKeySets:
- labelKeys: ["log_level"]
- labelKeys: ["device_id"]
- labelKeys: ["module", "device_id"]

According to these sets, we must specify log_level or device_id at least. But if we specify module and log_level at the same time, then logging service will use the last index as the most optimal.

Query logs without filtering on label values:

cuttle logging query logs --parents projects/<projectID> \
  --interval '{"startTime": "2022-08-15T00:00:00Z", \
               "endTime": "2022-08-16T00:00:00Z"}' \
  --filter 'logDescriptor="projects/<projectID>/logDescriptors/iot.app.org/syslog" \
            labels.device_id="temperature-sensor" AND \
            service="iot.app.org"' \
  -o json

Query logs based on device ID as a filter:

cuttle logging query logs --parents projects/<projectID> \
  --interval '{"startTime": "2022-08-15T00:00:00Z", \
               "endTime": "2022-08-16T00:00:00Z"}' \
  --filter 'logDescriptor="projects/<projectID>/logDescriptors/iot.app.org/syslog" \
            labels.device_id="temperature-sensor" AND \
            service="iot.app.org"' \
  -o json

Query logs with device ID and module as a filter:

cuttle logging query logs --parents projects/<projectID> \
  --interval '{"startTime": "2022-08-15T00:00:00Z", \
               "endTime": "2022-08-16T00:00:00Z"}' \
  --filter 'logDescriptor="projects/<projectID>/logDescriptors/iot.app.org/syslog" \
            labels.device_id="temperature-sensor" AND \
            labels.module=”sensor-phy” AND service="iot.app.org"' \
  -o json

Querying Logs programmatically using SDK:

func fetchLogs(
  ctx context.Context,
  logCli clog.LogServiceClient,
  deviceID, module, logLevel, logDescriptorName string,
  startTime, endTime time.Time,
) ([]*rlog.Log, error) {
	filterBuilder := rlog.NewFilterBuilder().Where().
                          Service().Eq("iot.app.org").
		                  Where().LogDescriptor().
                          Eq(rlog_descriptor.NewNameBuilder().
                             SetProjectId(projectId).
                             SetId(logDescriptorName).Reference())
    if deviceID != "" {
        filterBuilder = filterBuilder.Where().Labels().
                                      WithKey("device_id").Eq(deviceID)
    }
    if module != "" {
        filterBuilder = filterBuilder.Where().Labels().
                                      WithKey("module").Eq(module)
    }

    resp, err := logCli.ListLogs(ctx, &clog.ListLogsRequest{
        Parents: []*rlog.ParentName{
            rlog.NewNameBuilder().SetProjectId(projectID).Parent(),
        },
        Filter:  filterBuilder.Filter(),
        Interval: &log_common.TimeInterval{
            StartTime: timestamppb.New(startTime),
            EndTime:   timestamppb.New(endTime),
        },
    })
    if err != nil {
        return nil, err
    }
    return resp.GetLogs(), nil
}

Buckets

It is possible to restrict log creation/queries to a specific subset of logs within scope (service, organization, or project).

For example, suppose we have a device agent, and we want to ensure it can read/write only from/to specific owned logs. We can create the following bucket:

cuttle logging create bucket <bucketId> --project <projectId> \
  --region <regionId> \
  --logs '{
    "descriptors":["projects/<projectId>/logDescriptors/devices.edgelq.com/syslog"],
    "labels": {"project_id":{"strings": ["<projectId>"]}, \
               "region_id":{"strings": ["<regionId>"]}, \
               "device_id":{"strings": ["<deviceId>"]}}
  }'

We can now create a Role for Device (Yaml):

- name: services/devices.edgelq.com/roles/restricted-device-agent
  scopeParams:
  - name: region
    type: STRING
  - name: bucket
    type: STRING
  grants:
  - subScope: regions/{region}/buckets/{bucket}
    permissions:
    - services/logging.edgelq.com/permissions/logs.create
    - services/logging.edgelq.com/permissions/logs.query

The project can be specified in RoleBinding. When we assign the Role to the Device, then the device agent will be only able to create/query logs for a specific bucket - and this bucket will guarantee that:

Device can read/submit logs only for the projects/<projectId>/logDescriptors/devices.edgelq.com/syslog descriptor.
All logs for the syslog descriptor will have to specify the “project_id” label equal to the specified project, “region_id” equal to the specified region, and “device_id” equal to the specified device. When querying, the filter will have to specify all those fields.

Buckets ensure also correctness even if the client is submitting binary log keys (Log name with binary data key is provided, but labels are empty).

Provided example above is for information - Service devices.edgelq.com already provides Buckets for all Devices!

Understanding the logging.edgelq.com service APIv1, in proto package ntt.logging.v1.

Understanding the logging.edgelq.com service APIv1alpha2, in proto package ntt.logging.v1alpha2.