This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

SPEKTRA Edge Logging Service API

Understanding the Logging service API.
  • 1:
  • 2:

SPEKTRA Edge Logging service is a structured data store that provides the ability to store arbitrary data in the form of logs series with timestamps and user-defined metadata. To achieve that, we are using google.protobuf.Struct OR google.protobuf.Any, a well-known type used to store any JSON-like (or protobuf) data. It allows users not to worry about defining any proto message. We can store any data type such as string, int, float, bytes, map, etc.

Use cases for the logging service include:

  • system or application logs
  • event logs
  • tracing
  • core / debug dump
  • blob data

The stored logs can be queried based on timestamps and also filtered based on user-defined metadata fields.

Full API Specifications (with resources):

Resources

Log Descriptors

A log descriptor is the context of all logs created. It describes log, its metadata, service, API version, region, and the service-defined labels, for example process name can be an example.

A log descriptor also privdes indexing patterns for its labels. Each promoted index set is defined with a log descriptor is combined with the following fields:

When querying, it is required that users specify parent, service, and log descriptor fields, plus any promoted fields as defined by the log descriptor (field promotedLabelKeySets). If there is an empty set among available lists, it means that logs can be queried by scope, service and log descriptor fields only.

Log Entries

A log entry is a combination of timestamps with underlying data items. Log entries sharing the same scope, service, version, log descriptor and labels are considered to form a single Log instance.

Storing and querying logs

As an example, we will define and create a log descriptor for storing IoT device logs:

logDescriptor:
  name: projects/<projectID>/logDescriptors/iot.app.org/syslog
  displayName: IoT Device syslog
  description: IoT Device syslog
  labels:
  - key: module
    description: process/module/component name
  - key: device_id
    description: device ID
  - key: log_level
    description: Log level (debug, info, warn, error etc)
  promotedLabelKeySets:
  - labelKeys: ["log_level"]
  - labelKeys: ["device_id"]
  - labelKeys: ["module", "device_id"]

Create the log descriptor using cli:

cuttle logging create log-descriptor -f logdescriptor.yaml

Sample code for storing logs:

import (
   "context"
   "fmt"
   "time"


   "google.golang.org/protobuf/types/known/structpb"
   "google.golang.org/protobuf/types/known/timestamppb"


   clog "github.com/cloudwan/edgelq-sdk/logging/client/v1/log"
   log_common "github.com/cloudwan/edgelq-sdk/logging/common/v1"
   rlog "github.com/cloudwan/edgelq-sdk/logging/resources/v1/log"
   rlog_descriptor "github.com/cloudwan/edgelq-sdk/logging/resources/v1/log_descriptor"


   "github.com/cloudwan/edgelq-sdk/examples/utils"
)

const (
   logDescriptorName  = "iot.app.org/syslog"
   projectID          = "development"
   demoDeviceID       = "temperature-sensor-101"
   sampleModule       = "sensor-phy"
   sampleLogLevel     = "info"
   controllerEndpoint = "logging.stg01b.edgelq.com:443"
   credsFile          = "/etc/conf/edgelq-credentials.json"
)

func create_log_client(
   ctx context.Context,
) clog.LogServiceClient {
   grpcConn := utils.Dial(ctx, controllerEndpoint, "", credsFile) // Panics
   return clog.NewLogServiceClient(grpcConn)
}

func storeLogs(
  ctx context.Context,
  logCli clog.LogServiceClient,
  payloadString string,
  logTime time.Time,
) error {
   logData := map[string]interface{}{
       "sensor-dev": "/dev/zz",
       "process":    "sensor-agent",
       "message":    "Initialised sensor device",
   }

   payload, err := structpb.NewStruct(logData) // Check to rules below to avoid errors
   if err != nil {
       return err
   }

   logEntry := &rlog.Log{
       Service:       "iot.app.org",
       Version:       "v2",
       LogDescriptor: rlog_descriptor.NewNameBuilder().SetProjectId(projectID).SetId(logDescriptorName).Reference(),
       Labels: map[string]string{
           "device_id": demoDeviceID,
           "module":    sampleModule,
           "log_level": sampleLogLevel,
       },
       Time:    timestamppb.New(logTime),
       Payload: payload,
   }

   _, err = logCli.CreateLogs(ctx, &clog.CreateLogsRequest{
       Parent: rlog.NewNameBuilder().SetProjectId(projectID).ParentReference(),
       Logs:   []*rlog.Log{logEntry},
   })
   if err != nil {
       return fmt.Errorf("Failed to upload logs to the server: %w", err)
   }
   return nil
}

Note however, that it is also possible to provide the “Name” field in the rlog.Log value, but the ID must be equal to the value provided by the earlier CreateLogs request. Labels, LogDescriptor, Service, and Version can be skipped however then, reducing request size. It is still required to send at least one CreateLogs request with full meta information and labels, to have Name allocated.

With some logs submitted, we can try to query.

Log query takes three inputs:

  • Parents

    Project ID or Organization ID

  • Interval

    Start time and end time for the logs request

  • Filter

    Filter can combine multiple conditions based on the labels defined in the log descriptor. Log Descriptor and service name are mandatory filters. User-defined labels are optional filters.

In this example from Quickstart though, at least one user label is mandatory with a filter, because we have promoted labels:

promotedLabelKeySets:
- labelKeys: ["log_level"]
- labelKeys: ["device_id"]
- labelKeys: ["module", "device_id"]

According to these sets, we must specify log_level or device_id at least. But if we specify module and log_level at the same time, then logging service will use the last index as the most optimal.

Query logs without filtering on label values:

cuttle logging query logs --parents projects/<projectID> \
  --interval '{"startTime": "2022-08-15T00:00:00Z", \
               "endTime": "2022-08-16T00:00:00Z"}' \
  --filter 'logDescriptor="projects/<projectID>/logDescriptors/iot.app.org/syslog" \
            labels.device_id="temperature-sensor" AND \
            service="iot.app.org"' \
  -o json

Query logs based on device ID as a filter:

cuttle logging query logs --parents projects/<projectID> \
  --interval '{"startTime": "2022-08-15T00:00:00Z", \
               "endTime": "2022-08-16T00:00:00Z"}' \
  --filter 'logDescriptor="projects/<projectID>/logDescriptors/iot.app.org/syslog" \
            labels.device_id="temperature-sensor" AND \
            service="iot.app.org"' \
  -o json

Query logs with device ID and module as a filter:

cuttle logging query logs --parents projects/<projectID> \
  --interval '{"startTime": "2022-08-15T00:00:00Z", \
               "endTime": "2022-08-16T00:00:00Z"}' \
  --filter 'logDescriptor="projects/<projectID>/logDescriptors/iot.app.org/syslog" \
            labels.device_id="temperature-sensor" AND \
            labels.module=”sensor-phy” AND service="iot.app.org"' \
  -o json

Querying Logs programmatically using SDK:

func fetchLogs(
  ctx context.Context,
  logCli clog.LogServiceClient,
  deviceID, module, logLevel, logDescriptorName string,
  startTime, endTime time.Time,
) ([]*rlog.Log, error) {
	filterBuilder := rlog.NewFilterBuilder().Where().
                          Service().Eq("iot.app.org").
		                  Where().LogDescriptor().
                          Eq(rlog_descriptor.NewNameBuilder().
                             SetProjectId(projectId).
                             SetId(logDescriptorName).Reference())
    if deviceID != "" {
        filterBuilder = filterBuilder.Where().Labels().
                                      WithKey("device_id").Eq(deviceID)
    }
    if module != "" {
        filterBuilder = filterBuilder.Where().Labels().
                                      WithKey("module").Eq(module)
    }

    resp, err := logCli.ListLogs(ctx, &clog.ListLogsRequest{
        Parents: []*rlog.ParentName{
            rlog.NewNameBuilder().SetProjectId(projectID).Parent(),
        },
        Filter:  filterBuilder.Filter(),
        Interval: &log_common.TimeInterval{
            StartTime: timestamppb.New(startTime),
            EndTime:   timestamppb.New(endTime),
        },
    })
    if err != nil {
        return nil, err
    }
    return resp.GetLogs(), nil
}

Buckets

It is possible to restrict log creation/queries to a specific subset of logs within scope (service, organization, or project).

For example, suppose we have a device agent, and we want to ensure it can read/write only from/to specific owned logs. We can create the following bucket:

cuttle logging create bucket <bucketId> --project <projectId> \
  --region <regionId> \
  --logs '{
    "descriptors":["projects/<projectId>/logDescriptors/devices.edgelq.com/syslog"],
    "labels": {"project_id":{"strings": ["<projectId>"]}, \
               "region_id":{"strings": ["<regionId>"]}, \
               "device_id":{"strings": ["<deviceId>"]}}
  }'

We can now create a Role for Device (Yaml):

- name: services/devices.edgelq.com/roles/restricted-device-agent
  scopeParams:
  - name: region
    type: STRING
  - name: bucket
    type: STRING
  grants:
  - subScope: regions/{region}/buckets/{bucket}
    permissions:
    - services/logging.edgelq.com/permissions/logs.create
    - services/logging.edgelq.com/permissions/logs.query

The project can be specified in RoleBinding. When we assign the Role to the Device, then the device agent will be only able to create/query logs for a specific bucket - and this bucket will guarantee that:

  • Device can read/submit logs only for the projects/<projectId>/logDescriptors/devices.edgelq.com/syslog descriptor.
  • All logs for the syslog descriptor will have to specify the “project_id” label equal to the specified project, “region_id” equal to the specified region, and “device_id” equal to the specified device. When querying, the filter will have to specify all those fields.

Buckets ensure also correctness even if the client is submitting binary log keys (Log name with binary data key is provided, but labels are empty).

Provided example above is for information - Service devices.edgelq.com already provides Buckets for all Devices!

1 -

Understanding the logging.edgelq.com service APIv1, in proto package ntt.logging.v1.

Service logging.edgelq.com in version v1, proto package ntt.logging.v1

Here is the list of resources supported in Logging service APIv1:

Bucket Resource

Bucket Resource

Name patterns:

  • projects/{project}/regions/{region}/buckets/{bucket}
  • organizations/{organization}/regions/{region}/buckets/{bucket}
  • services/{service}/regions/{region}/buckets/{bucket}

Parent resources:

This section covers the methods and messages to interact with Bucket resource.

Bucket Methods

Here is the list of Bucket resource methods:

GetBucket Method

GetBucket

rpc GetBucket(GetBucketRequest) returns (Bucket)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.get

The equivalent REST API is:

GET /v1/{name=projects/*/regions/*/buckets/*} 

GET /v1/{name=organizations/*/regions/*/buckets/*} 

GET /v1/{name=services/*/regions/*/buckets/*}

BatchGetBuckets Method

BatchGetBuckets

rpc BatchGetBuckets(BatchGetBucketsRequest) returns (BatchGetBucketsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.batchGet

The equivalent REST API is:

GET /v1/buckets:batchGet

ListBuckets Method

ListBuckets

rpc ListBuckets(ListBucketsRequest) returns (ListBucketsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.list

The equivalent REST API is:

GET /v1/{parent=projects/*/regions/*}/buckets 

GET /v1/{parent=organizations/*/regions/*}/buckets 

GET /v1/{parent=services/*/regions/*}/buckets

WatchBucket Method

WatchBucket

rpc WatchBucket(WatchBucketRequest) returns (WatchBucketResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.watch

The equivalent REST API is:

POST /v1/{name=projects/*/regions/*/buckets/*}:watch 

POST /v1/{name=organizations/*/regions/*/buckets/*}:watch 

POST /v1/{name=services/*/regions/*/buckets/*}:watch

WatchBuckets Method

WatchBuckets

rpc WatchBuckets(WatchBucketsRequest) returns (WatchBucketsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.watch

The equivalent REST API is:

POST /v1/{parent=projects/*/regions/*}/buckets:watch 

POST /v1/{parent=organizations/*/regions/*}/buckets:watch 

POST /v1/{parent=services/*/regions/*}/buckets:watch

CreateBucket Method

CreateBucket

rpc CreateBucket(CreateBucketRequest) returns (Bucket)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.create

The equivalent REST API is:

POST /v1/{parent=projects/*/regions/*}/buckets (BODY: bucket)

POST /v1/{parent=organizations/*/regions/*}/buckets 

POST /v1/{parent=services/*/regions/*}/buckets

UpdateBucket Method

UpdateBucket

rpc UpdateBucket(UpdateBucketRequest) returns (Bucket)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.update

The equivalent REST API is:

PUT /v1/{bucket.name=projects/*/regions/*/buckets/*} (BODY: bucket)

PUT /v1/{bucket.name=organizations/*/regions/*/buckets/*} 

PUT /v1/{bucket.name=services/*/regions/*/buckets/*}

DeleteBucket Method

DeleteBucket

rpc DeleteBucket(DeleteBucketRequest) returns (Empty)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/buckets.delete

The equivalent REST API is:

DELETE /v1/{name=projects/*/regions/*/buckets/*} 

DELETE /v1/{name=organizations/*/regions/*/buckets/*} 

DELETE /v1/{name=services/*/regions/*/buckets/*}

Bucket Messages

Here is the list of Bucket resource messages:

Bucket Message

Name Type Description
name string (name of Bucket) Name of Bucket When creating a new instance, this field is optional and if not provided, it will be generated automatically. Last ID segment must conform to the following regex: [\w./-]{2,128}
metadata Meta Metadata is an object with information like create, update and delete time (for async deleted resources), has user labels/annotations, sharding information, multi-region syncing information and may have non-schema owners (useful for taking ownership of resources belonging to lower level services by higher ones).
services repeated string Optional service names (if required), for example “devices.edgelq.com”
versions repeated string Oprional service versions (if required), for example “v1”
logs repeated Bucket.RequiredTypedLabels Allowed log combinations (OR). If empty, all log types are allowed.
required_alt_kvs repeated Bucket.ResolvedKeysWithValues All combinations of key-values (in integer forms) - one of them must be passed by every Log object for given bucket. Its computed by server side and for internal use.

Bucket.ResolvedValues Message

ResolvedValues contains binary representation of types and labels and possible values. Due to limitations in some db backends (looking at firestore), we use int64 instead of uint64.

Name Type Description
key int64
values repeated int64

Bucket.ResolvedKeysWithValues Message

ResolvedKeysWithValues binds multiple keys with possible values.

Name Type Description
resolved_kvs repeated Bucket.ResolvedValues

Bucket.RequiredTypedLabels Message

RequiredTypedLabels describes required label values for specified log descriptors. All logs in Create operation must contain at least one allowed type and then labels must match all the labels. For list queries, filter must contain at least one type and all labels must be present in condition containing all or subset of allowed label values.

Name Type Description
descriptors repeated string (reference to LogDescriptor)
labels map<string, Bucket.RequiredTypedLabels.Strings>

Bucket.RequiredTypedLabels.Strings Message

Name Type Description
strings repeated string

GetBucketRequest Message

A request message of the GetBucket method.

Name Type Description
name string (name of Bucket) Name of ntt.logging.v1.Bucket
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetBucketsRequest Message

A request message of the BatchGetBuckets method.

Name Type Description
names repeated string (name of Bucket) Names of Buckets
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetBucketsResponse Message

A response message of the BatchGetBuckets method.

Name Type Description
buckets repeated Bucket found Buckets
missing repeated string (name of Bucket) list of not found Buckets

ListBucketsRequest Message

A request message of the ListBuckets method.

Name Type Description
parent string (parent name of Bucket) Parent name of ntt.logging.v1.Bucket
page_size int32 Requested page size. Server may return fewer Buckets than requested. If unspecified, server will pick an appropriate default.
page_token string (cursor of Bucket) A token identifying a page of results the server should return. Typically, this is the value of ListBucketsResponse.next_page_token.
order_by string (orderBy of Bucket) Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination list of field path with order directive, either ‘asc’ or ‘desc’. If direction is not provided, ‘asc’ is assumed. e.g. “state.nested_field asc, state.something.else desc, theme”
filter string (filter of Bucket) Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask
include_paging_info bool Indicates if list response should contain total count and offset (fields current_offset and total_results_count).

ListBucketsResponse Message

A response message of the ListBuckets method.

Name Type Description
buckets repeated Bucket The list of Buckets
prev_page_token string (cursor of Bucket) A token to retrieve previous page of results. Pass this value in the ListBucketsRequest.page_token.
next_page_token string (cursor of Bucket) A token to retrieve next page of results. Pass this value in the ListBucketsRequest.page_token.
current_offset int32 Current offset from the first page or 0 if no page tokens were given, paging info was not requested or there was an error while trying to get it). Page index can be computed from offset and limit provided in a request.
total_results_count int32 Number of total Buckets across all pages or 0, if there are no items, paging info was not requested or there was an error while trying to get it.

WatchBucketRequest Message

A request message of the WatchBucket method.

Name Type Description
name string (name of Bucket) Name of ntt.logging.v1.Bucket
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

WatchBucketResponse Message

A response message of the WatchBucket method.

Name Type Description
change BucketChange

WatchBucketsRequest Message

A request message of the WatchBuckets method.

Name Type Description
type WatchType Type of a watch. Identifies how server stream data to a client, which fields in a request are allowed and which fields in response are relevant.
parent string (parent name of Bucket) Parent name of ntt.logging.v1.Bucket
page_size int32 Requested page size. Server may return fewer Buckets than requested. If unspecified, server will pick an appropriate default. Can be populated only for stateful watch type.
page_token string (cursor of Bucket) A token identifying a page of results the server should return. Can be populated only for stateful watch type.
order_by string (orderBy of Bucket) Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination Can be populated only for stateful watch type.
resume_token string A token identifying watch resume point from previous session. Can be populated only for stateless watch type.
starting_time .google.protobuf.Timestamp Point in the time from which we want to start getting updates. This field can be populated only for stateless watch type and if resume token is not known yet. If specified, initial snapshot will NOT be provided. It is assumed client can obtain it using separate means. Watch responses will contain resume tokens which should be used to resume broken connection.
filter string (filter of Bucket) Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view Changes to Bucket that don’t affect any of masked fields won’t be sent back.
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask Changes to Bucket that don’t affect any of masked fields won’t be sent back.
max_chunk_size int32 Maximum amount of changes in each response message. Query result response is divided on the server side into chunks with size of a specified amount to limit memory footprint of each message. Responses will hold information whether more elements will continue for the actual change. If unspecified, server will pick an appropriate default.

WatchBucketsResponse Message

A response message of the WatchBuckets method.

Name Type Description
bucket_changes repeated BucketChange Changes of Buckets
is_current bool If request specified max_chunk_size (or this limit was enforced if stateless watch has been chosen), then responses with “full changeset” will be divided into chunks. Client should keep receiving messages and, once is_current has value true, combine this recent message with all previous ones where is_current is false. If this is the first is_current in a whole watch stream, then it means that client should have, at this moment, contain snapshot of the current situation (or more accurately, snapshot of situation at the moment of request). All Buckets will be of type Added/Current (depending on watch_type specified in the request). Further responses will be incremental - however messages may still be chunked and is_current logic still applies. is_current is always true for stateful watch if max_chunk_size was left to 0.
page_token_change WatchBucketsResponse.PageTokenChange When present, PageTokens used for page navigation should be updated. Present only if is_current is true (last chunk).
resume_token string Token that can be used if current connection drops and client needs to reconnect. Populated only for stateless watch type. Present only if is_current is true (last chunk).
snapshot_size int64 Server may occasionally send information how many resources should client have in its state so far (response message without any changes, but with snapshot_size field specified). If client has different value than the one sent by the server, then it should be treated by a client as an error and should reconnect. If value is smaller then 0, then client should ignore this field as unpopulated. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will be never sent together with is_current, is_soft_reset and is_hard_reset flags.
is_soft_reset bool In case of internal issue server may send response message with this flag. It indicates that client should drop all changes from recent responses where is_current is false only! If last message had is_current set to true, client should do nothing and process normally. Resume token received before is still valid. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_hard_reset or snapshot_size.
is_hard_reset bool In case of internal issue server may send response message with this flag. After receiving, client should clear whole state (drop all changes received so far) as server will send new snapshot (Buckets will contains changes of type Current only). Any resume tokens should be discarded as well. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_soft_reset or snapshot_size.

WatchBucketsResponse.PageTokenChange Message

Name Type Description
prev_page_token string (cursor of Bucket) New token to retrieve previous page of results.
next_page_token string (cursor of Bucket) New token to retrieve next page of results.

CreateBucketRequest Message

A request message of the CreateBucket method.

Name Type Description
parent string (parent name of Bucket) Parent name of ntt.logging.v1.Bucket
bucket Bucket Bucket resource body
response_mask CreateBucketRequest.ResponseMask Optional masking applied to response object to reduce message response size.

CreateBucketRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields.

UpdateBucketRequest Message

A request message of the UpdateBucket method.

Name Type Description
bucket Bucket Bucket resource body
update_mask .google.protobuf.FieldMask FieldMask applied to request - change will be applied only for fields in the mask
cas UpdateBucketRequest.CAS Conditional update applied to request if update should be executed only for specific resource state. If this field is populated, then server will fetch existing resource, compare with the one stored in the cas field (after applying field mask) and proceed with update only and only if they match. Otherwise RPC error Aborted will be returned.
allow_missing bool If set to true, and the resource is not found, a new resource will be created. In this situation, ‘field_mask’ is ignored. https://google.aip.dev/134#create-or-update
response_mask UpdateBucketRequest.ResponseMask reduce message response size.

UpdateBucketRequest.CAS Message

CAS - Compare and Swap. This object is used if user wants to make update conditional based upon previous resource version.

Name Type Description
conditional_state Bucket Conditional desired state of a resource before update.
field_mask .google.protobuf.FieldMask Field paths from conditional state of resource server should check and compare.

UpdateBucketRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated. Field body_mask is ignored if set.
updated_fields_only bool Include all fields that were actually updated during processing. Note this may be larger than update mask if some fields were computed additionally. Name is added as well.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields. If skip_entire_response_body is true, this field is ignored.

DeleteBucketRequest Message

A request message of the DeleteBucket method.

Name Type Description
name string (name of Bucket) Name of ntt.logging.v1.Bucket

Log Resource

Log Resource

Name patterns:

  • projects/{project}/logs/{log}
  • organizations/{organization}/logs/{log}
  • services/{service}/logs/{log}
  • projects/{project}/regions/{region}/buckets/{bucket}/logs/{log}
  • organizations/{organization}/regions/{region}/buckets/{bucket}/logs/{log}
  • services/{service}/regions/{region}/buckets/{bucket}/logs/{log}

Parent resources:

This section covers the methods and messages to interact with Log resource.

Log Methods

Here is the list of Log resource methods:

ListLogs Method

ListLogs

rpc ListLogs(ListLogsRequest) returns (ListLogsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logs.list

The equivalent REST API is:

GET /v1/logs

CreateLogs Method

CreateLogs

rpc CreateLogs(CreateLogsRequest) returns (CreateLogsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logs.create

The equivalent REST API is:

POST /v1/logs (BODY: logs)

StreamingCreateLogs Method

StreamingCreateLogs

rpc StreamingCreateLogs(StreamingCreateLogsRequest) returns (StreamingCreateLogsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logs.streamingCreate

The equivalent REST API is:

POST /v1/logs:streamingCreate

Log Messages

Here is the list of Log resource messages:

Log Message

Name Type Description
name string (name of Log) Name of Log When creating a new instance, this field is optional and if not provided, it will be generated automatically. Last ID segment must conform to the following regex: [\w./-=+]{1,128}
scope string Contains scope from name field without resource ID. Used for internal purpose for filtering (logs are using custom store). Example formats are: - organization/umbrella - projects/mars_exploration - services/devices.edgelq.com
service string Service name, for example “devices.edgelq.com”
region string Region of the service to which log is assigned, for example “us-west2”
version string Associated service version, for example “v1”
log_descriptor string (reference to LogDescriptor) Associated log descriptor
labels map<string, string> List of query-able labels
time .google.protobuf.Timestamp Log timestamp
json_payload .google.protobuf.Struct Log JSON payload
pb_payload .google.protobuf.Any Log Protobuf payload
string_payload string Log string type payload
bytes_payload bytes Log bytes type payload
bin_key string Log bin_key identifies unique Log tuple: <scope, service, region, version, labels, log_descriptor Key is not to be decoded outside of service, but treated as opaque string. This field is equivalent to ID part of the log name and therefore any one of them can be used. This is intented for streaming log create to conserve some bandwidth

ListLogsRequest Message

Request message for method [ListLogs][ntt.logging.v1.ListLogs]

Name Type Description
parents repeated string (parent name of Log) Parent references of ntt.logging.v1.Log
filter string (filter of Log) Filter that specifies which logs should be returned
interval TimeInterval The time interval for which results should be returned. Only logs that contain data points in the specified interval are included in the response.
page_size int32 Cap on a number of log entries to be included in a response. Number of logs in an actual response can be higher, since logs are read in bulk with second precision - exceed logs above the limit will share same timestamp as the logs below the limit. Results will be adjusted to the “end time” taken from interval field (adjusted also by page_token if provided).
page_token string Token which identifies next page with further results. Token should be taken from [ListLogsResponse.next_page_token][ntt.logging.v1.ListLogsResponse.next_page_token].

ListLogsResponse Message

Response message for method [ListLogs][ntt.logging.v1.ListLogs]

Name Type Description
logs repeated Log Logs that match the filter included in the request.
next_page_token string If there are more results than have been returned, then this field is set to a non-empty value. To see the additional results, use that value as pageToken in the next call to this method.
execution_errors repeated Status Query execution errors that may have caused the logs data returned to be incomplete.

ListLogsResponse.ErrorDetails Message

ErrorDetails is used when one of the queried regions fails to produce results. It is used in execution_errors field (see subfield ntt.rpc.Status.details).

Name Type Description
region_id string region id which failed to give results.

CreateLogsRequest Message

Request message for method [CreateLogs][ntt.logging.v1.CreateLogs]

Name Type Description
parent string (parent name of Log) Parent reference of ntt.logging.v1.Log
logs repeated Log List of logs to create/append. If they have specified name field, it must match provided parent field.

CreateLogsResponse Message

Response message for method [CreateLogs][ntt.logging.v1.CreateLogs]

Name Type Description
log_names map<uint32s, string (name of Log)> Log names indexed by Create position. All logs, except failed ones will be present. If all logs were written successfully, then map will have keys all from 0 to N-1.
failed_logs repeated CreateLogsResponse.CreateError Logs that failed to be created

CreateLogsResponse.CreateError Message

Describes the result of a failed request to write logs.

Name Type Description
logs repeated Log All logs that failed to be written. This field provides all of the context that would be needed to retry the operation.
status Status The status of the requested write operation.

StreamingCreateLogsRequest Message

A request message of the StreamingCreateLogs method. This method only supports creating logs for the same project and region for the entire duration of the stream. Any client that intends to create logs in multiple projects or regions should not use this API. If the client uses the same scope, service, version and log descriptor, these fields need not be sent after the first message even if binary key is not used in the log entry. The server will set the values observed in the first message in case these are empty and binary key is also not present in the log

Name Type Description
parent string (parent name of Log) Parent reference of ntt.logging.v1.Log only required in the first request message
region_id string only required in the first request in order to route the request to the correct region
logs repeated Log List of logs to create/append. If they have specified name field, it must match provided parent field

StreamingCreateLogsResponse Message

A response message of the StreamingCreateLogs method.

Name Type Description
bin_keys map<uint32s, string> Binary key will be set in the map key corresponding to the array index in the request, but only if the particular request entry did not use binary key. Indices for log entries in request having binary key set will be ignored in the response
failed_logs repeated CreateLogsResponse.CreateError Logs that failed to be created

LogDescriptor Resource

LogDescriptor Resource

Name patterns:

  • projects/{project}/logDescriptors/{log_descriptor}
  • organizations/{organization}/logDescriptors/{log_descriptor}
  • services/{service}/logDescriptors/{log_descriptor}

Parent resources:

This section covers the methods and messages to interact with LogDescriptor resource.

LogDescriptor Methods

Here is the list of LogDescriptor resource methods:

GetLogDescriptor Method

GetLogDescriptor

rpc GetLogDescriptor(GetLogDescriptorRequest) returns (LogDescriptor)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.get

The equivalent REST API is:

GET /v1/{name=projects/*/logDescriptors/*} 

GET /v1/{name=organizations/*/logDescriptors/*} 

GET /v1/{name=services/*/logDescriptors/*}

BatchGetLogDescriptors Method

BatchGetLogDescriptors

rpc BatchGetLogDescriptors(BatchGetLogDescriptorsRequest) returns (BatchGetLogDescriptorsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.batchGet

The equivalent REST API is:

GET /v1/logDescriptors:batchGet

ListLogDescriptors Method

ListLogDescriptors

rpc ListLogDescriptors(ListLogDescriptorsRequest) returns (ListLogDescriptorsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.list

The equivalent REST API is:

GET /v1/{parent=projects/*}/logDescriptors 

GET /v1/{parent=organizations/*}/logDescriptors 

GET /v1/{parent=services/*}/logDescriptors

WatchLogDescriptor Method

WatchLogDescriptor

rpc WatchLogDescriptor(WatchLogDescriptorRequest) returns (WatchLogDescriptorResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.watch

The equivalent REST API is:

POST /v1/{name=projects/*/logDescriptors/*}:watch 

POST /v1/{name=organizations/*/logDescriptors/*}:watch 

POST /v1/{name=services/*/logDescriptors/*}:watch

WatchLogDescriptors Method

WatchLogDescriptors

rpc WatchLogDescriptors(WatchLogDescriptorsRequest) returns (WatchLogDescriptorsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.watch

The equivalent REST API is:

POST /v1/{parent=projects/*}/logDescriptors:watch 

POST /v1/{parent=organizations/*}/logDescriptors:watch 

POST /v1/{parent=services/*}/logDescriptors:watch

CreateLogDescriptor Method

CreateLogDescriptor

rpc CreateLogDescriptor(CreateLogDescriptorRequest) returns (LogDescriptor)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.create

The equivalent REST API is:

POST /v1/{parent=projects/*}/logDescriptors (BODY: log_descriptor)

POST /v1/{parent=organizations/*}/logDescriptors 

POST /v1/{parent=services/*}/logDescriptors

UpdateLogDescriptor Method

UpdateLogDescriptor

rpc UpdateLogDescriptor(UpdateLogDescriptorRequest) returns (LogDescriptor)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.update

The equivalent REST API is:

PUT /v1/{log_descriptor.name=projects/*/logDescriptors/*} (BODY: log_descriptor)

PUT /v1/{log_descriptor.name=organizations/*/logDescriptors/*} 

PUT /v1/{log_descriptor.name=services/*/logDescriptors/*}

DeleteLogDescriptor Method

DeleteLogDescriptor

rpc DeleteLogDescriptor(DeleteLogDescriptorRequest) returns (Empty)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.delete

The equivalent REST API is:

DELETE /v1/{name=projects/*/logDescriptors/*} 

DELETE /v1/{name=organizations/*/logDescriptors/*} 

DELETE /v1/{name=services/*/logDescriptors/*}

LogDescriptor Messages

Here is the list of LogDescriptor resource messages:

LogDescriptor Message

Name Type Description
name string (name of LogDescriptor) Name of LogDescriptor When creating a new instance, this field is optional and if not provided, it will be generated automatically. Last ID segment must conform to the following regex: [\w./-]{1,128}/[\w./-]{1,128}
metadata Meta Metadata is an object with information like create, update and delete time (for async deleted resources), has user labels/annotations, sharding information, multi-region syncing information and may have non-schema owners (useful for taking ownership of resources belonging to lower level services by higher ones).
display_name string Optional. A concise name for the log type that might be displayed in user interfaces. It should be a Title Cased Noun Phrase, without any article or other determiners.
description string Optional. A detailed description of the log type that might be used in documentation.
labels repeated LabelDescriptor A set of labels used to describe instances of this log type.
promoted_label_key_sets repeated LabelKeySet Promoted Label Key Sets allow defining multiple indexing rules for underlying backend enabling query optimizations.

GetLogDescriptorRequest Message

A request message of the GetLogDescriptor method.

Name Type Description
name string (name of LogDescriptor) Name of ntt.logging.v1.LogDescriptor
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetLogDescriptorsRequest Message

A request message of the BatchGetLogDescriptors method.

Name Type Description
names repeated string (name of LogDescriptor) Names of LogDescriptors
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetLogDescriptorsResponse Message

A response message of the BatchGetLogDescriptors method.

Name Type Description
log_descriptors repeated LogDescriptor found LogDescriptors
missing repeated string (name of LogDescriptor) list of not found LogDescriptors

ListLogDescriptorsRequest Message

A request message of the ListLogDescriptors method.

Name Type Description
parent string (parent name of LogDescriptor) Parent name of ntt.logging.v1.LogDescriptor
page_size int32 Requested page size. Server may return fewer LogDescriptors than requested. If unspecified, server will pick an appropriate default.
page_token string (cursor of LogDescriptor) A token identifying a page of results the server should return. Typically, this is the value of ListLogDescriptorsResponse.next_page_token.
order_by string (orderBy of LogDescriptor) Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination list of field path with order directive, either ‘asc’ or ‘desc’. If direction is not provided, ‘asc’ is assumed. e.g. “state.nested_field asc, state.something.else desc, theme”
filter string (filter of LogDescriptor) Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask
include_paging_info bool Indicates if list response should contain total count and offset (fields current_offset and total_results_count).

ListLogDescriptorsResponse Message

A response message of the ListLogDescriptors method.

Name Type Description
log_descriptors repeated LogDescriptor The list of LogDescriptors
prev_page_token string (cursor of LogDescriptor) A token to retrieve previous page of results. Pass this value in the ListLogDescriptorsRequest.page_token.
next_page_token string (cursor of LogDescriptor) A token to retrieve next page of results. Pass this value in the ListLogDescriptorsRequest.page_token.
current_offset int32 Current offset from the first page or 0 if no page tokens were given, paging info was not requested or there was an error while trying to get it). Page index can be computed from offset and limit provided in a request.
total_results_count int32 Number of total LogDescriptors across all pages or 0, if there are no items, paging info was not requested or there was an error while trying to get it.

WatchLogDescriptorRequest Message

A request message of the WatchLogDescriptor method.

Name Type Description
name string (name of LogDescriptor) Name of ntt.logging.v1.LogDescriptor
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

WatchLogDescriptorResponse Message

A response message of the WatchLogDescriptor method.

Name Type Description
change LogDescriptorChange

WatchLogDescriptorsRequest Message

A request message of the WatchLogDescriptors method.

Name Type Description
type WatchType Type of a watch. Identifies how server stream data to a client, which fields in a request are allowed and which fields in response are relevant.
parent string (parent name of LogDescriptor) Parent name of ntt.logging.v1.LogDescriptor
page_size int32 Requested page size. Server may return fewer LogDescriptors than requested. If unspecified, server will pick an appropriate default. Can be populated only for stateful watch type.
page_token string (cursor of LogDescriptor) A token identifying a page of results the server should return. Can be populated only for stateful watch type.
order_by string (orderBy of LogDescriptor) Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination Can be populated only for stateful watch type.
resume_token string A token identifying watch resume point from previous session. Can be populated only for stateless watch type.
starting_time .google.protobuf.Timestamp Point in the time from which we want to start getting updates. This field can be populated only for stateless watch type and if resume token is not known yet. If specified, initial snapshot will NOT be provided. It is assumed client can obtain it using separate means. Watch responses will contain resume tokens which should be used to resume broken connection.
filter string (filter of LogDescriptor) Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view Changes to LogDescriptor that don’t affect any of masked fields won’t be sent back.
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask Changes to LogDescriptor that don’t affect any of masked fields won’t be sent back.
max_chunk_size int32 Maximum amount of changes in each response message. Query result response is divided on the server side into chunks with size of a specified amount to limit memory footprint of each message. Responses will hold information whether more elements will continue for the actual change. If unspecified, server will pick an appropriate default.

WatchLogDescriptorsResponse Message

A response message of the WatchLogDescriptors method.

Name Type Description
log_descriptor_changes repeated LogDescriptorChange Changes of LogDescriptors
is_current bool If request specified max_chunk_size (or this limit was enforced if stateless watch has been chosen), then responses with “full changeset” will be divided into chunks. Client should keep receiving messages and, once is_current has value true, combine this recent message with all previous ones where is_current is false. If this is the first is_current in a whole watch stream, then it means that client should have, at this moment, contain snapshot of the current situation (or more accurately, snapshot of situation at the moment of request). All LogDescriptors will be of type Added/Current (depending on watch_type specified in the request). Further responses will be incremental - however messages may still be chunked and is_current logic still applies. is_current is always true for stateful watch if max_chunk_size was left to 0.
page_token_change WatchLogDescriptorsResponse.PageTokenChange When present, PageTokens used for page navigation should be updated. Present only if is_current is true (last chunk).
resume_token string Token that can be used if current connection drops and client needs to reconnect. Populated only for stateless watch type. Present only if is_current is true (last chunk).
snapshot_size int64 Server may occasionally send information how many resources should client have in its state so far (response message without any changes, but with snapshot_size field specified). If client has different value than the one sent by the server, then it should be treated by a client as an error and should reconnect. If value is smaller then 0, then client should ignore this field as unpopulated. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will be never sent together with is_current, is_soft_reset and is_hard_reset flags.
is_soft_reset bool In case of internal issue server may send response message with this flag. It indicates that client should drop all changes from recent responses where is_current is false only! If last message had is_current set to true, client should do nothing and process normally. Resume token received before is still valid. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_hard_reset or snapshot_size.
is_hard_reset bool In case of internal issue server may send response message with this flag. After receiving, client should clear whole state (drop all changes received so far) as server will send new snapshot (LogDescriptors will contains changes of type Current only). Any resume tokens should be discarded as well. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_soft_reset or snapshot_size.

WatchLogDescriptorsResponse.PageTokenChange Message

Name Type Description
prev_page_token string (cursor of LogDescriptor) New token to retrieve previous page of results.
next_page_token string (cursor of LogDescriptor) New token to retrieve next page of results.

CreateLogDescriptorRequest Message

A request message of the CreateLogDescriptor method.

Name Type Description
parent string (parent name of LogDescriptor) Parent name of ntt.logging.v1.LogDescriptor
log_descriptor LogDescriptor LogDescriptor resource body
response_mask CreateLogDescriptorRequest.ResponseMask Optional masking applied to response object to reduce message response size.

CreateLogDescriptorRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields.

UpdateLogDescriptorRequest Message

A request message of the UpdateLogDescriptor method.

Name Type Description
log_descriptor LogDescriptor LogDescriptor resource body
update_mask .google.protobuf.FieldMask FieldMask applied to request - change will be applied only for fields in the mask
cas UpdateLogDescriptorRequest.CAS Conditional update applied to request if update should be executed only for specific resource state. If this field is populated, then server will fetch existing resource, compare with the one stored in the cas field (after applying field mask) and proceed with update only and only if they match. Otherwise RPC error Aborted will be returned.
allow_missing bool If set to true, and the resource is not found, a new resource will be created. In this situation, ‘field_mask’ is ignored. https://google.aip.dev/134#create-or-update
response_mask UpdateLogDescriptorRequest.ResponseMask reduce message response size.

UpdateLogDescriptorRequest.CAS Message

CAS - Compare and Swap. This object is used if user wants to make update conditional based upon previous resource version.

Name Type Description
conditional_state LogDescriptor Conditional desired state of a resource before update.
field_mask .google.protobuf.FieldMask Field paths from conditional state of resource server should check and compare.

UpdateLogDescriptorRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated. Field body_mask is ignored if set.
updated_fields_only bool Include all fields that were actually updated during processing. Note this may be larger than update mask if some fields were computed additionally. Name is added as well.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields. If skip_entire_response_body is true, this field is ignored.

DeleteLogDescriptorRequest Message

A request message of the DeleteLogDescriptor method.

Name Type Description
name string (name of LogDescriptor) Name of ntt.logging.v1.LogDescriptor

Logging Service Shared Methods and Messages

Logging Service Shared Messages

Here is the list of Logging service shared messages:

BucketChange Message

BucketChange is used by Watch notifications Responses to describe change of single Bucket One of Added, Modified, Removed

Name Type Description
added BucketChange.Added Added is returned when watched document is added, either created or enters Query view
modified BucketChange.Modified Modified is returned when watched document is modified
current BucketChange.Current Current is returned in stateless watch when document enters query view or is modified within.
removed BucketChange.Removed Removed is returned when Bucket is deleted or leaves Query view

BucketChange.Added Message

Bucket has been added to query view

Name Type Description
bucket Bucket
view_index int32 Integer describing index of added Bucket in resulting query view.

BucketChange.Current Message

Bucket has been added or modified in a query view. Version used for stateless watching

Name Type Description
bucket Bucket

BucketChange.Modified Message

Bucket changed some of it’s fields - contains either full document or masked change

Name Type Description
name string (name of Bucket) Name of modified Bucket
bucket Bucket New version of Bucket or masked difference, depending on mask_changes instrumentation of issued [WatchBucketRequest] or [WatchBucketsRequest]
field_mask .google.protobuf.FieldMask Used when mask_changes is set, contains field paths of modified properties.
previous_view_index int32 Previous view index specifies previous position of modified Bucket. When modification doesn’t affect sorted order, value will remain identical to [view_index].
view_index int32 Integer specifying Bucket new index in resulting query view.

BucketChange.Removed Message

Removed is returned when Bucket is deleted or leaves Query view

Name Type Description
name string (name of Bucket)
view_index int32 Integer specifying removed Bucket index. Not populated in stateless watch type.

LabelDescriptor Message

Name Type Description
key string The label key.
description string Label description

LabelKeySet Message

LabelKeySet is used for defining PromotedLabelKeySets

Name Type Description
label_keys repeated string

LogDescriptorChange Message

LogDescriptorChange is used by Watch notifications Responses to describe change of single LogDescriptor One of Added, Modified, Removed

Name Type Description
added LogDescriptorChange.Added Added is returned when watched document is added, either created or enters Query view
modified LogDescriptorChange.Modified Modified is returned when watched document is modified
current LogDescriptorChange.Current Current is returned in stateless watch when document enters query view or is modified within.
removed LogDescriptorChange.Removed Removed is returned when LogDescriptor is deleted or leaves Query view

LogDescriptorChange.Added Message

LogDescriptor has been added to query view

Name Type Description
log_descriptor LogDescriptor
view_index int32 Integer describing index of added LogDescriptor in resulting query view.

LogDescriptorChange.Current Message

LogDescriptor has been added or modified in a query view. Version used for stateless watching

Name Type Description
log_descriptor LogDescriptor

LogDescriptorChange.Modified Message

LogDescriptor changed some of it’s fields - contains either full document or masked change

Name Type Description
name string (name of LogDescriptor) Name of modified LogDescriptor
log_descriptor LogDescriptor New version of LogDescriptor or masked difference, depending on mask_changes instrumentation of issued [WatchLogDescriptorRequest] or [WatchLogDescriptorsRequest]
field_mask .google.protobuf.FieldMask Used when mask_changes is set, contains field paths of modified properties.
previous_view_index int32 Previous view index specifies previous position of modified LogDescriptor. When modification doesn’t affect sorted order, value will remain identical to [view_index].
view_index int32 Integer specifying LogDescriptor new index in resulting query view.

LogDescriptorChange.Removed Message

Removed is returned when LogDescriptor is deleted or leaves Query view

Name Type Description
name string (name of LogDescriptor)
view_index int32 Integer specifying removed LogDescriptor index. Not populated in stateless watch type.

TimeInterval Message

A time interval extending just after a start time through an end time. If the start time is the same as the end time, then the interval represents a single point in time.

Name Type Description
end_time .google.protobuf.Timestamp Optional - end of the time interval. If not provided, current time will be assumed.
start_time .google.protobuf.Timestamp Required. The beginning of the time interval. The start time must not be later than the end time.

2 -

Understanding the logging.edgelq.com service APIv1alpha2, in proto package ntt.logging.v1alpha2.

Service logging.edgelq.com in version v1alpha2, proto package ntt.logging.v1alpha2

Here is the list of resources supported in Logging service APIv1alpha2:

Log Resource

Log Resource

Name patterns:

  • logs/{log}
  • projects/{project}/logs/{log}
  • organizations/{organization}/logs/{log}

Parent resources:

This section covers the methods and messages to interact with Log resource.

Log Methods

Here is the list of Log resource methods:

ListLogs Method

ListLogs

rpc ListLogs(ListLogsRequest) returns (ListLogsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logs.list

The equivalent REST API is:

GET /v1alpha2/logs

CreateLogs Method

CreateLogs

rpc CreateLogs(CreateLogsRequest) returns (CreateLogsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logs.create

The equivalent REST API is:

POST /v1alpha2/logs (BODY: logs)

Log Messages

Here is the list of Log resource messages:

Log Message

Name Type Description
name string (name of Log) Name of Log. It contains scope + ID of the log. ID is a base64 encoded unique key that identifies tuple: scope service region_id version log_descriptor labels Key is not to be decoded outside of service, but treated as opaque string
scope string Contains scope from name field without resource ID. Used for internal purpose for filtering (logs are using custom store). Example formats are: - organization/umbrella - projects/mars_exploration - <system>
service string Service name, for example “devices.edgelq.com”
region string Region of the service to which log is assigned, for example “us-west2”
version string Associated service version, for example “v1alpha2”
log_descriptor string (reference to LogDescriptor) Associated log descriptor
labels map<string, string> List of query-able labels
time .google.protobuf.Timestamp Log timestamp
payload .google.protobuf.Struct Log payload

ListLogsRequest Message

Request message for method [ListLogs][ntt.logging.v1alpha2.ListLogs]

Name Type Description
parents repeated string (parent name of Log) Parent references of ntt.logging.v1alpha2.Log
filter string (filter of Log) Filter that specifies which logs should be returned
interval TimeInterval The time interval for which results should be returned. Only logs that contain data points in the specified interval are included in the response.
page_size int32 Cap on a number of log entries to be included in a response. Number of logs in an actual response can be higher, since logs are read in bulk with second precision - exceed logs above the limit will share same timestamp as the logs below the limit. Results will be adjusted to the “end time” taken from interval field (adjusted also by page_token if provided).
page_token string Token which identifies next page with further results. Token should be taken from [ListLogsResponse.next_page_token][ntt.logging.v1alpha2.ListLogsResponse.next_page_token].

ListLogsResponse Message

Response message for method [ListLogs][ntt.logging.v1alpha2.ListLogs]

Name Type Description
logs repeated Log Logs that match the filter included in the request.
next_page_token string If there are more results than have been returned, then this field is set to a non-empty value. To see the additional results, use that value as pageToken in the next call to this method.
execution_errors repeated Status Query execution errors that may have caused the logs data returned to be incomplete.

ListLogsResponse.ErrorDetails Message

ErrorDetails is used when one of the queried regions fails to produce results. It is used in execution_errors field (see subfield ntt.rpc.Status.details).

Name Type Description
region_id string region id which failed to give results.

CreateLogsRequest Message

Request message for method [CreateLogs][ntt.logging.v1alpha2.CreateLogs]

Name Type Description
parent string (parent name of Log) Parent reference of ntt.logging.v1alpha2.Log
logs repeated Log List of logs to create/append. If they have specified name field, it must match provided parent field.

CreateLogsResponse Message

Response message for method [CreateLogs][ntt.logging.v1alpha2.CreateLogs]

Name Type Description
log_names map<uint32s, string (name of Log)> Log names indexed by Create position. All logs, except failed ones will be present. If all logs were written successfully, then map will have keys all from 0 to N-1.
failed_logs repeated CreateLogsResponse.CreateError Logs that failed to be created

CreateLogsResponse.CreateError Message

Describes the result of a failed request to write logs.

Name Type Description
logs repeated Log All logs that failed to be written. This field provides all of the context that would be needed to retry the operation.
status Status The status of the requested write operation.

LogDescriptor Resource

LogDescriptor Resource

Name patterns:

  • logDescriptors/{log_descriptor}
  • projects/{project}/logDescriptors/{log_descriptor}
  • organizations/{organization}/logDescriptors/{log_descriptor}

Parent resources:

This section covers the methods and messages to interact with LogDescriptor resource.

LogDescriptor Methods

Here is the list of LogDescriptor resource methods:

GetLogDescriptor Method

GetLogDescriptor

rpc GetLogDescriptor(GetLogDescriptorRequest) returns (LogDescriptor)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.get

The equivalent REST API is:

GET /v1alpha2/{name=logDescriptors/*} 

GET /v1alpha2/{name=projects/*/logDescriptors/*} 

GET /v1alpha2/{name=organizations/*/logDescriptors/*}

BatchGetLogDescriptors Method

BatchGetLogDescriptors

rpc BatchGetLogDescriptors(BatchGetLogDescriptorsRequest) returns (BatchGetLogDescriptorsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.batchGet

The equivalent REST API is:

GET /v1alpha2/logDescriptors:batchGet

ListLogDescriptors Method

ListLogDescriptors

rpc ListLogDescriptors(ListLogDescriptorsRequest) returns (ListLogDescriptorsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.list

The equivalent REST API is:

GET /v1alpha2/logDescriptors 

GET /v1alpha2/{parent=projects/*}/logDescriptors 

GET /v1alpha2/{parent=organizations/*}/logDescriptors

WatchLogDescriptor Method

WatchLogDescriptor

rpc WatchLogDescriptor(WatchLogDescriptorRequest) returns (WatchLogDescriptorResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.watch

The equivalent REST API is:

POST /v1alpha2/{name=logDescriptors/*}:watch 

POST /v1alpha2/{name=projects/*/logDescriptors/*}:watch 

POST /v1alpha2/{name=organizations/*/logDescriptors/*}:watch

WatchLogDescriptors Method

WatchLogDescriptors

rpc WatchLogDescriptors(WatchLogDescriptorsRequest) returns (WatchLogDescriptorsResponse)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.watch

The equivalent REST API is:

POST /v1alpha2/logDescriptors:watch 

POST /v1alpha2/{parent=projects/*}/logDescriptors:watch 

POST /v1alpha2/{parent=organizations/*}/logDescriptors:watch

CreateLogDescriptor Method

CreateLogDescriptor

rpc CreateLogDescriptor(CreateLogDescriptorRequest) returns (LogDescriptor)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.create

The equivalent REST API is:

POST /v1alpha2/logDescriptors (BODY: log_descriptor)

POST /v1alpha2/{parent=projects/*}/logDescriptors 

POST /v1alpha2/{parent=organizations/*}/logDescriptors

UpdateLogDescriptor Method

UpdateLogDescriptor

rpc UpdateLogDescriptor(UpdateLogDescriptorRequest) returns (LogDescriptor)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.update

The equivalent REST API is:

PUT /v1alpha2/{log_descriptor.name=logDescriptors/*} (BODY: log_descriptor)

PUT /v1alpha2/{log_descriptor.name=projects/*/logDescriptors/*} 

PUT /v1alpha2/{log_descriptor.name=organizations/*/logDescriptors/*}

DeleteLogDescriptor Method

DeleteLogDescriptor

rpc DeleteLogDescriptor(DeleteLogDescriptorRequest) returns (Empty)

with the following messages:

Required Permissions:

  • services/logging.edgelq.com/permissions/logDescriptors.delete

The equivalent REST API is:

DELETE /v1alpha2/{name=logDescriptors/*} 

DELETE /v1alpha2/{name=projects/*/logDescriptors/*} 

DELETE /v1alpha2/{name=organizations/*/logDescriptors/*}

LogDescriptor Messages

Here is the list of LogDescriptor resource messages:

LogDescriptor Message

Name Type Description
name string (name of LogDescriptor) Name of LogDescriptor - contains service name followed by name defined by developer, separated by ‘/’ sign. Example name: “devices.edgelq.com/DeviceFailure”
display_name string Optional. A concise name for the log type that might be displayed in user interfaces. It should be a Title Cased Noun Phrase, without any article or other determiners.
description string Optional. A detailed description of the log type that might be used in documentation.
labels repeated LabelDescriptor A set of labels used to describe instances of this log type.
promoted_label_key_sets repeated LabelKeySet Promoted Label Key Sets allow defining multiple indexing rules for underlying backend enabling query optimizations.
metadata Meta Metadata

GetLogDescriptorRequest Message

A request message of the GetLogDescriptor method.

Name Type Description
name string (name of LogDescriptor) Name of ntt.logging.v1alpha2.LogDescriptor
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetLogDescriptorsRequest Message

A request message of the BatchGetLogDescriptors method.

Name Type Description
names repeated string (name of LogDescriptor) Names of LogDescriptors
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetLogDescriptorsResponse Message

A response message of the BatchGetLogDescriptors method.

Name Type Description
log_descriptors repeated LogDescriptor found LogDescriptors
missing repeated string (name of LogDescriptor) list of not found LogDescriptors

ListLogDescriptorsRequest Message

A request message of the ListLogDescriptors method.

Name Type Description
parent string (parent name of LogDescriptor) Parent name of ntt.logging.v1alpha2.LogDescriptor
page_size int32 Requested page size. Server may return fewer LogDescriptors than requested. If unspecified, server will pick an appropriate default.
page_token string (cursor of LogDescriptor) A token identifying a page of results the server should return. Typically, this is the value of ListLogDescriptorsResponse.next_page_token.
order_by string (orderBy of LogDescriptor) Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination list of field path with order directive, either ‘asc’ or ‘desc’. If direction is not provided, ‘asc’ is assumed. e.g. “state.nested_field asc, state.something.else desc, theme”
filter string (filter of LogDescriptor) Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask
include_paging_info bool Indicates if list response should contain total count and offset (fields current_offset and total_results_count).

ListLogDescriptorsResponse Message

A response message of the ListLogDescriptors method.

Name Type Description
log_descriptors repeated LogDescriptor The list of LogDescriptors
prev_page_token string (cursor of LogDescriptor) A token to retrieve previous page of results. Pass this value in the ListLogDescriptorsRequest.page_token.
next_page_token string (cursor of LogDescriptor) A token to retrieve next page of results. Pass this value in the ListLogDescriptorsRequest.page_token.
current_offset int32 Current offset from the first page or 0 if no page tokens were given, paging info was not requested or there was an error while trying to get it). Page index can be computed from offset and limit provided in a request.
total_results_count int32 Number of total LogDescriptors across all pages or 0, if there are no items, paging info was not requested or there was an error while trying to get it.

WatchLogDescriptorRequest Message

A request message of the WatchLogDescriptor method.

Name Type Description
name string (name of LogDescriptor) Name of ntt.logging.v1alpha2.LogDescriptor
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

WatchLogDescriptorResponse Message

A response message of the WatchLogDescriptor method.

Name Type Description
change LogDescriptorChange

WatchLogDescriptorsRequest Message

A request message of the WatchLogDescriptors method.

Name Type Description
type WatchType Type of a watch. Identifies how server stream data to a client, which fields in a request are allowed and which fields in response are relevant.
parent string (parent name of LogDescriptor) Parent name of ntt.logging.v1alpha2.LogDescriptor
page_size int32 Requested page size. Server may return fewer LogDescriptors than requested. If unspecified, server will pick an appropriate default. Can be populated only for stateful watch type.
page_token string (cursor of LogDescriptor) A token identifying a page of results the server should return. Can be populated only for stateful watch type.
order_by string (orderBy of LogDescriptor) Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination Can be populated only for stateful watch type.
resume_token string A token identifying watch resume point from previous session. Can be populated only for stateless watch type.
starting_time .google.protobuf.Timestamp Point in the time from which we want to start getting updates. This field can be populated only for stateless watch type and if resume token is not known yet. If specified, initial snapshot will NOT be provided. It is assumed client can obtain it using separate means. Watch responses will contain resume tokens which should be used to resume broken connection.
filter string (filter of LogDescriptor) Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view Changes to LogDescriptor that don’t affect any of masked fields won’t be sent back.
view View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask Changes to LogDescriptor that don’t affect any of masked fields won’t be sent back.
max_chunk_size int32 Maximum amount of changes in each response message. Query result response is divided on the server side into chunks with size of a specified amount to limit memory footprint of each message. Responses will hold information whether more elements will continue for the actual change. If unspecified, server will pick an appropriate default.

WatchLogDescriptorsResponse Message

A response message of the WatchLogDescriptors method.

Name Type Description
log_descriptor_changes repeated LogDescriptorChange Changes of LogDescriptors
is_current bool If request specified max_chunk_size (or this limit was enforced if stateless watch has been chosen), then responses with “full changeset” will be divided into chunks. Client should keep receiving messages and, once is_current has value true, combine this recent message with all previous ones where is_current is false. If this is the first is_current in a whole watch stream, then it means that client should have, at this moment, contain snapshot of the current situation (or more accurately, snapshot of situation at the moment of request). All LogDescriptors will be of type Added/Current (depending on watch_type specified in the request). Further responses will be incremental - however messages may still be chunked and is_current logic still applies. is_current is always true for stateful watch if max_chunk_size was left to 0.
page_token_change WatchLogDescriptorsResponse.PageTokenChange When present, PageTokens used for page navigation should be updated. Present only if is_current is true (last chunk).
resume_token string Token that can be used if current connection drops and client needs to reconnect. Populated only for stateless watch type. Present only if is_current is true (last chunk).
snapshot_size int64 Server may occasionally send information how many resources should client have in its state so far (response message without any changes, but with snapshot_size field specified). If client has different value than the one sent by the server, then it should be treated by a client as an error and should reconnect. If value is smaller then 0, then client should ignore this field as unpopulated. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will be never sent together with is_current, is_soft_reset and is_hard_reset flags.
is_soft_reset bool In case of internal issue server may send response message with this flag. It indicates that client should drop all changes from recent responses where is_current is false only! If last message had is_current set to true, client should do nothing and process normally. Resume token received before is still valid. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_hard_reset or snapshot_size.
is_hard_reset bool In case of internal issue server may send response message with this flag. After receiving, client should clear whole state (drop all changes received so far) as server will send new snapshot (LogDescriptors will contains changes of type Current only). Any resume tokens should be discarded as well. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_soft_reset or snapshot_size.

WatchLogDescriptorsResponse.PageTokenChange Message

Name Type Description
prev_page_token string (cursor of LogDescriptor) New token to retrieve previous page of results.
next_page_token string (cursor of LogDescriptor) New token to retrieve next page of results.

CreateLogDescriptorRequest Message

A request message of the CreateLogDescriptor method.

Name Type Description
parent string (parent name of LogDescriptor) Parent name of ntt.logging.v1alpha2.LogDescriptor
log_descriptor LogDescriptor LogDescriptor resource body
response_mask CreateLogDescriptorRequest.ResponseMask Optional masking applied to response object to reduce message response size.

CreateLogDescriptorRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields.

UpdateLogDescriptorRequest Message

A request message of the UpdateLogDescriptor method.

Name Type Description
log_descriptor LogDescriptor LogDescriptor resource body
update_mask .google.protobuf.FieldMask FieldMask applied to request - change will be applied only for fields in the mask
cas UpdateLogDescriptorRequest.CAS Conditional update applied to request if update should be executed only for specific resource state. If this field is populated, then server will fetch existing resource, compare with the one stored in the cas field (after applying field mask) and proceed with update only and only if they match. Otherwise RPC error Aborted will be returned.
allow_missing bool If set to true, and the resource is not found, a new resource will be created. In this situation, ‘field_mask’ is ignored. https://google.aip.dev/134#create-or-update
response_mask UpdateLogDescriptorRequest.ResponseMask reduce message response size.

UpdateLogDescriptorRequest.CAS Message

CAS - Compare and Swap. This object is used if user wants to make update conditional based upon previous resource version.

Name Type Description
conditional_state LogDescriptor Conditional desired state of a resource before update.
field_mask .google.protobuf.FieldMask Field paths from conditional state of resource server should check and compare.

UpdateLogDescriptorRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated. Field body_mask is ignored if set.
updated_fields_only bool Include all fields that were actually updated during processing. Note this may be larger than update mask if some fields were computed additionally. Name is added as well.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields. If skip_entire_response_body is true, this field is ignored.

DeleteLogDescriptorRequest Message

A request message of the DeleteLogDescriptor method.

Name Type Description
name string (name of LogDescriptor) Name of ntt.logging.v1alpha2.LogDescriptor

Logging Service Shared Methods and Messages

Logging Service Shared Messages

Here is the list of Logging service shared messages:

LabelDescriptor Message

Name Type Description
key string The label key.
description string Label description

LabelKeySet Message

LabelKeySet is used for defining PromotedLabelKeySets

Name Type Description
label_keys repeated string

LogDescriptorChange Message

LogDescriptorChange is used by Watch notifications Responses to describe change of single LogDescriptor One of Added, Modified, Removed

Name Type Description
added LogDescriptorChange.Added Added is returned when watched document is added, either created or enters Query view
modified LogDescriptorChange.Modified Modified is returned when watched document is modified
current LogDescriptorChange.Current Current is returned in stateless watch when document enters query view or is modified within.
removed LogDescriptorChange.Removed Removed is returned when LogDescriptor is deleted or leaves Query view

LogDescriptorChange.Added Message

LogDescriptor has been added to query view

Name Type Description
log_descriptor LogDescriptor
view_index int32 Integer describing index of added LogDescriptor in resulting query view.

LogDescriptorChange.Current Message

LogDescriptor has been added or modified in a query view. Version used for stateless watching

Name Type Description
log_descriptor LogDescriptor

LogDescriptorChange.Modified Message

LogDescriptor changed some of it’s fields - contains either full document or masked change

Name Type Description
name string (name of LogDescriptor) Name of modified LogDescriptor
log_descriptor LogDescriptor New version of LogDescriptor or masked difference, depending on mask_changes instrumentation of issued [WatchLogDescriptorRequest] or [WatchLogDescriptorsRequest]
field_mask .google.protobuf.FieldMask Used when mask_changes is set, contains field paths of modified properties.
previous_view_index int32 Previous view index specifies previous position of modified LogDescriptor. When modification doesn’t affect sorted order, value will remain identical to [view_index].
view_index int32 Integer specifying LogDescriptor new index in resulting query view.

LogDescriptorChange.Removed Message

Removed is returned when LogDescriptor is deleted or leaves Query view

Name Type Description
name string (name of LogDescriptor)
view_index int32 Integer specifying removed LogDescriptor index. Not populated in stateless watch type.

TimeInterval Message

A time interval extending just after a start time through an end time. If the start time is the same as the end time, then the interval represents a single point in time.

Name Type Description
end_time .google.protobuf.Timestamp Optional - end of the time interval. If not provided, current time will be assumed.
start_time .google.protobuf.Timestamp Required. The beginning of the time interval. The start time must not be later than the end time.