Declaring your Service

• 1: Service Specification
• 2: Auto-Generated Protobuf Files

1 - Service Specification

At the beginning of this chapter, before continuing, it is worth first mentioning the naming conventions we use: https://cloud.google.com/apis/design/naming_convention

Example api-skeleton for 3rd party app: https://github.com/cloudwan/inventory-manager-example/blob/master/proto/api-skeleton-v1.yaml You can also see API skeletons in the edgelq repository, too.

This document describes api-skeletons in bigger detail than a quick startup.

When you start writing a service, the first (you have a new, empty directory for your service) you need is to do two things:

• Create a subdirectory called “proto” (convention used in all created goten services).

• In the proto directory, create file api-skeleton-$SERVICE_VERSION.yaml file.

  In place of $SERVICE_VERSION, you should put a version of your service, for example, v1alpha for a start.

API skeleton file is used by Goten to bootstrap initial proto files for your service - some of them will be initialized only once, some will always be overwritten by subsequent regeneration.

JSON schema

API-Skeleton schema is based on the protobuf file itself: https://github.com/cloudwan/goten/blob/main/annotations/bootstrap.proto You may check this file to see all possible options.

There is a useful trick you can do with your IDE, so it understands schema and can aid you with prototyping: https://github.com/cloudwan/goten/blob/main/schemas/api-skeleton.schema.json

In your IDE, find JSON Schema mappings, and give a path to this file (for example, to your cloned copy of goten). Match with file pattern api-skeleton. This way, IDE can help with writing it.

Generating protobuf files

Once the API-skeleton file is ready, you can generate protobuf files with:

goten-bootstrap -i "${SERVICEPATH}/proto/api-skeleton-$VERSION.yaml" \
  -o "${SERVICEPATH}/proto"
clang-format-12 -i "${SERVICEPATH}"/proto/$VERSION/**.proto

This utility is provided by the Goten repository - you should set up development env first to have this tool.

Variable SERVICEPATH must point to the directory of your service, and VERSION match the service version. It is highly recommended to use clang formatter on generated files - but there is no particular recommendation from version, 12 comes from current state of the scripts.

Note that you should re-generate every time something changes in api-skeleton.

Header part

The header shows basic information about your service:

name: $SERVICE_NAME
proto:
 package:
   name: $PROTO_PACKAGE_PREFIX
   currentVersion: $SERVICE_VERSION
   goPackage: $GITHUB_LINK_TO_YOUR_SERVICE
   protoImportPathPrefix: $DIRECTORY_NAME_WITH_SERVICE_CODE/proto
 service:
   name: $SERVICE_SHORT_NAME
   defaultHost: $SERVICE_NAME
   oauthScopes: https://apis.edgelq.com

• $SERVICE_NAME

  It must be exactly equal to the service you reserved (see introduction to developer guide).

• $PROTO_PACKAGE_PREFIX

  It will be used as a prefix for a proto package containing the whole of your service.

• $SERVICE_VERSION

  It shows a version of your service. It will also be used as a suffix for a proto package of your service.

• $GITHUB_LINK_TO_YOUR_SERVICE

  It must be an actual Github link.

• $DIRECTORY_NAME_WITH_SERVICE_CODE

  It must be equal to the directory name of your code.

• $SERVICE_SHORT_NAME

  It should be some short service name, not in “domain format”.

The header simply declares what service and what version is being offered. It is advisable to configure your IDE to include $DIRECTORY_NAME_WITH_SERVICE_CODE/proto in your proto paths - it will make traversing through IDE much simpler.

Imported services

Very often you will need to declare services your service imports, this is done usually below the header in the API-skeleton:

imports:
- $IMPORTED_SERVICE_NAME

You must provide the service name in imported if at least one of the below applies:

• One of the resources you declared in your service has a parent resource in the imported service.
• One of the resources you declared in your service has a reference to a resource in the imported service

You do NOT NEED to declare a service you are just “using” via its API. For example, if your client runtimes use proxies.edgelq.com for tunneling, but if you don’t use proxies.edgelq.com on the schema level, then you don’t need to import it.

Goten operates on No-SQL and No-relationship databases (Mongo, Firestore), so it provides its mechanism that provides those things. The benefit of Goten is that it can provide relationships not only between resources within your service but also across services. However, it needs in advance to know which services are going to be used. Runtime libraries/modules provided by Goten/SPEKTRA Edge ensure that databases across services are synchronized (for example, we don’t have a dangling reference that was supposed to be blocking deletion).

When you import a service, you must modify your goten-bootstrap call. For example, if you imported meta.goten.com service in version v1, then you need to run commands like:

goten-bootstrap -i "${SERVICEPATH}/proto/api-skeleton-$VERSION.yaml" \
  -o "${SERVICEPATH}/proto" \
  --import "${GOTENPATH}/meta-service/proto/api-skeleton-v1.yaml"
clang-format-12 -i "${SERVICEPATH}"/proto/$VERSION/**.proto

Note that GOTENPATH must point to the Goten code directory - and this path is the current reflection of the current code.

If you imported let’s say iam.edgelq.com, which imports meta.goten.com, you will need to provide import paths to all relevant API-skeletons:

goten-bootstrap -i "${SERVICEPATH}/proto/api-skeleton-$VERSION.yaml" \
  -o "${SERVICEPATH}/proto" \
  --import "${GOTENPATH}/meta-service/proto/api-skeleton-v1.yaml" \
  --import "${EDGELQROOT}/iam/proto/api-skeleton-v1.yaml"
clang-format-12 -i "${SERVICEPATH}"/proto/$VERSION/**.proto

As of now, goten-bootstrap needs field paths to directly and indirectly imported services.

Resources

Goten is resource-oriented, so you should organize your service around resources:

resources:
- name: $RESOURCE_SINGULAR_NAME # It should be in UpperCamelCase format
  plural: $RESOURCE_PLURAL_NAME # If not provided, it is $RESOURCE_SINGULAR_NAME with 's' added at the end.
  parents:
  - $RESOURCE_PARENT_SERVICE/$RESOURCE_PARENT_NAME # $RESOURCE_PARENT_SERVICE/ can be skipped if $RESOURCE_PARENT_NAME is declared in same service
  scopeAttributes:
  - $SCOPE_ATTRIBUTE_NAME
  idPattern: $ID_PATTERN_REGEX

Certain more advanced elements were omitted from above.

Standard Resource represents an object with the following characteristics:

• It has a name field that makes a unique identifier.
• It has a metadata field that contains meta information (sharding, lifecycle, etc.)
• It has an associated API group with the same name as Resource. That API contains CRUD actions for this resource and custom ones, added to the resource in the API-skeleton file.
• Has a collection in the database. That collection can be accessed via CRUD actions added implicitly by goten, OR more directly from server code via database handle.

Resources can be in parent-child relationships - including multiple parents' support, as you may see in the case of “Message” resource. However, you should also note that the resource “Comment”, despite having only one possible parent “Message”, can too have multiple ancestry paths.

Resource naming

Each resource has a unique identifier and name - it is stored also in a “name” field. Resource naming is a very important topic in Goten therefore it deserves a good explanation. The format of any name is the following:

$PARENT_NAME_BLOCK$SCOPE_ATTRIBUTES_NAME_BLOCK$SELF_IF_BLOCK

There are 3 blocks: $PARENT_NAME_BLOCK, then $SCOPE_ATTRIBUTES_NAME_BLOCK, and finally $SELF_IF_BLOCK.

Let us start with $SELF_IF_BLOCK, which has the following format: $resourcePluralNameCamelCase/$resourceId. It is always present and cannot be skipped. First part, $resourcePluralNameCamelCase is derived from $RESOURCE_PLURAL_NAME variable, but first and later is lower-cased. Variable $resourceId is assigned during creation and can never be updated. It must comply with the regex supplied with the variable $ID_PATTERN_REGEX in the api-skeleton file. Param idPattern can be skipped from resource definition - in that case, the default value will be applied: [a-z][a-z0-9\\-]{0,28}[a-z0-9].

The middle block, $SCOPE_ATTRIBUTES_NAME_BLOCK will be EMPTY if none scopeAttributes were defined for a resource in the api-skeleton file. By syntax, scopeAttributes is an array from 0 to N elements, like:

scopeAttributes:
- AttributeOne
- AttributeTwo

Block $SCOPE_ATTRIBUTES_NAME_BLOCK will be a concatenation of all scope attributes in declared order, for this example, it will be like: attributeOnes/$attributeOneId/attributeTwos/$attributeTwoId/. The last ‘/’ is to ensure it can be concatenated with $SELF_IF_BLOCK. Scope attributes also have singular/plural names and ID pattern regexes.

As of now, Goten provides only one built-in scope attribute that can be attached to a resource: Region. It means, that resource like:

name: SomeName
scopeAttributes:
- Region

will have the following name pattern: regions/$regionId/someNames/$someNameId. This built-in attribute Region is very special and has a significant impact on resources, but in essence: It shows that a resource has specific un-modifiable region it belongs to. All write requests for it will have to be executed by the region the resource belongs to. Region attributes should be considered when modeling for MultiRegion deployments. More details later in this doc.

Finally, we have a block $PARENT_NAME_BLOCK - it is empty if param parents were not present for the given resource in the API skeleton. Unlike scope attributes where all are active, a single resource instance can only have one active parent at the same time. When we specify multiple parents in the API skeleton, we just say that there are many alternate values to $PARENT_NAME_BLOCK. This param is the name of the parent resource. Each value of $PARENT_NAME_BLOCK is then structured in this way: $PARENT_NAME_BLOCK$SCOPE_ATTRIBUTES_NAME_BLOCK$SELF_IF_BLOCK/. Last / ensures that it can be concatenated with $SCOPE_ATTRIBUTES_NAME_BLOCK or $SELF_IF_BLOCK if the former is blank.

Top parent resource must have no parents at all.

It is possible to have an optional parent resource as well if we specify the empty string "" as a parent:

parents:
- SomeOptionalParent
- ""

In the above case, $PARENT_NAME_BLOCK will either be empty or end with someOptionalParents/$someOptionalParentId/.

Note that resource parent is a special kind of reference to different resource types. However, unlike regular references:

• The name of the resource contains actual references to ALL ancestral resources.

• Regular references are somewhere in the resource body

  not in the identifier. Therefore, for example, a GET request automatically shows us not only the resource we want to get but also its whole ancestry path.

• If parent is deleted, all kid resources must be automatically deleted

  asynchronously or in-transaction. Unlike in regular reference, a parent cannot be “unset”.

• Scope attributes from parents are automatically inherited by all child resources. It is not the case for regular references.

Throughout Goten, you may encounter some additional things about resource names:

• Wildcards

  for example name someResource/- indicates ANY resource of SomeResource kind.

• Parent names

  parent name is like name, but without $SELF_IF_BLOCK part. It indicates just the parent collection of resources.

Let’s consider some known resources in iam.edgelq.com service: RoleBinding. It has API-skeleton definition:

name: RoleBinding
parents:
- meta.goten.com/Service # Service is declared in different service
- Project
- Organization
- ""

From above, there are 4 valid name patterns RoleBinding can have:

• services/{service}/roleBindings/{roleBinding}
• projects/{project}/roleBindings/{roleBinding}
• organizations/{organization}/roleBindings/{roleBinding}
• roleBindings/{roleBinding}

Then, PARENT NAME patterns that are valid are (same order):

• services/{service}
• projects/{project}
• organizations/{organization}
• "" - just empty string

With wildcards, we can define (just examples):

• services/{service}/roleBindings/- - This indicates ANY RoleBinding from specific service
• services/-/roleBindings/- - This indicates ANY RoleBinding from ANY service
• services/-/roleBindings/{roleBinding} - This would pick all RoleBindings across all services having the same final ID.

Wildcards can be specified in parent names too.

Resource opt-outs

Developers can opt-out from specific standard features offered by Goten. In most cases, we need to do this for specific CRUD actions that are attached to all resources. For example:

resources:
- name: SomeResourceName
  optOuts:
    basicActions:
    - CreateSomeResourceName

Other cases are more tricky. Standard goten CRUD access, resourceChange, and metadata are used intensively by Goten/SPEKTRA Edge framework, even if you don’t use it yourself. For 3rd party developers, we recommend not disabling anything outside basic actions. Other opt-outs exist for resources that are NOT using a standard database (but a custom one, where developers provide their own driver). They will in this way escape the normal schema system (references to those resources will not work as usual).

Resource opt-ins

Some features are optional - as of now we have just one opt-in, it is a Search addition to the standard CRUD for resource objects. We can enable this in API-skeleton:

resources:
- name: SomeResourceName
  optIns:
    searchable: true

With the above, Goten will add the SearchSomeResourceNames action to the standard CRUD. The search method is very similar to List, but users can also specify a search phrase on top of the filter, field mask, and standard paging fields.

Note that enabling in api-skeleton is not sufficient. The developer will also have to:

• Specify (in protobuf files) a list of fields for search-text indexing
• Configure search store backend during deployment

As of now, we support Algolia search, but we plan to extend this to MongoDB too. In this case, we may be able to use same database for records and searches.

Tenant and authorization separation when prototyping resources

There are certain requirements that service developers must follow when developing on the SPEKTRA Edge platform.

The consideration here is a tenant/authorization separation. SPEKTRA Edge services are designed for multi-tenancy in the heart. Those are organized as two resources in service iam.edgelq.com: Organization and Project. The organization is meant to be a container for child Organizations and Projects. The project is a final tenant. Those resource types are top. They don’t have any parents or scope attributes. Their name patterns are, therefore:

• iam.edgelq.com/Project: projects/{project}
• iam.edgelq.com/Organization: organizations/{organization}

Most of the resources in all services should either belong to the Project as final tenant consumer, less typically Organization. Project is preferable due to the stronger integration:

• Monitoring time series can go to project only
• Limits are capable of limiting resource instances only within projects
• Usage metrics are counted per project.

For the above reasons, it is recommended to let Organization be a container for Projects on the core SPEKTRA Edge platform, and use Project resource as a parent for further resources. Therefore, you should apply the following practice in your service API skeleton:

resource:
# You should declare resource Project in your service!
# It must have specific multiRegion setup.
- name: Project
  multiRegion:
    isPolicyHolder: true

# Under project, you can define resource types that will belong to
# each tenant. Those are to be defined by you.
- name: $CUSTOM_RESOURCE_NAME_1
  parents:
  - Project

- name: $CUSTOM_RESOURCE_NAME_2
  parents:
  - Project

# This resource is still descending from Project, so its fine.
- name: $CUSTOM_RESOURCE_NAME_3
  parents:
  - $CUSTOM_RESOURCE_NAME_1
- - $CUSTOM_RESOURCE_NAME_2

The above setup is integrating with iam.edgelq.com Authorization already: Service IAM recognizes 4 authorization scopes:

• System level (root): /
• Organization level: organizations/
• Project level: projects/
• Service level: services/

This list of scopes matches possible parents of RoleBinding resources. By declaring resources under Project, we are utilizing well-known scope and project admins can manage RoleBindings on their own - and forbid other projects to see/modify their data.

Even if a service in development is meant to be used by a single user (like a private service), it is still recommended to use Project resource

• we will have just one instance in existence.

When you deploy such a service, you will need to configure synchronization between 2 collections: iam.edgelq.com/Project AND your.service.com/Project. You should copy those projects into your collection that are interested in your service.

The reason is that projects follow multi-service design: Their administrators should freely choose which services are used by their projects and which are not. If we have a copy of the Project resource from the iam.edgelq.com service, we can:

• Ensure that all projects in your service are those enabling the given service.
• If the project leaves your service, all child resources will be garbage-collected.
• Project defined by your service can have additional fields not present in iam.edgelq.com/Project.

Synchronization between collections across services will be explained in the fixture controller document.

It is of course recognized that not all resource types are suitable to be put under Project tenant - some resource types are meant to be commonly shared across many tenants, probably in read-only mode, with write reserved for service administrators. If you have resources like this, the best option may be to declare them under meta.goten.com/Service resource:

imports:
- meta.goten.com # Needed in order to use Service as a parent

resources:
- name: $SOME_SERVICE_RESOURCE
  parents:
  - meta.goten.com/Service

Specifically, we should NOT have something like:

resources:
- name: $SOME_GLOBAL_SERVICE_RESOURCE

The reason is that, by declaring global service resource on the root level, IAM permissions required for any CRUD will be on the root level. As a system owner, you will have access to this resource type, BUT users for your service will not have, and you won’t be able to grant them any permissions - because RoleBindings on the root / level cannot be created by anyone but SPEKTRA Edge platform administrators. However, 3rd party service admins will be able to create RoleBindings under the Service resource:

services/$your_service_name/roleBindings/$rbId. From this position, permissions can be granted to users by service admins to access those service-level resources.

You may optionally declare service resources like for a Project:

resources:
- name: Service
  multiRegion:
    isPolicyHolder: true

- name: $SOME_SERVICE_RESOURCE
  parents:
  - Service

However, it will cause the generation of Service resources in your service and you will need to copy your service record from meta.goten.com to your service. But this has some benefits:

• It is in some sense clearer

  While meta.goten.com contains all services in the system, your service will contain only services that are using “your service”. In this case, it will be 1 element collection.

• Other services than yours can become tenants of your service as well! You will need to copy a subset of services from meta.goten.com to your service

  A subset using your service. You can then add extra fields not available in the meta.goten.com service.

If you plan to expose your service to other services, you should declare your Service resource and set up the controller to synchronize meta.goten.com/Service with your.service.com/Service. You should synchronize a subset of services only.

If you wonder why not then handle Project resources in the same way:

imports:
- iam.edgelq.com

resources:
- name: $SOME_RESOURCE
  parents:
  - iam.edgelq.com/Project

In the above pattern, the benefit is, that you don’t have a Projects collection in your service - iam already has. However, it is not suitable if you plan to have a multi-tenant service, where the tenant is a Project. As was mentioned. The project is meant to be able to enable/disable services it uses at a whim. By using a synchronized owned collection of Projects, we can ensure that child resources of projects in your service can be properly cleaned up.

However, if you are certain that your service is meant to be used by private project(s) who are always going to use your service and cannot disable it. Then in fact it is a valid choice to use iam.edgelq.com/Project directly.

API

Before explaining API, let me explain the relationships between services, resources, APIs, and methods:

• Service package contains a set of resources and APIs, each resource and API belongs to a single Service Package.
• API contains a set of actions and each action belongs to a single API only. Each action can also be optionally associated with a single resource (primary resource for action).
• APIs can be either “developer-defined” or “provided with resources”. The primary difference between them is that developer-defined API is explicitly described in the API skeleton, while the other kind is implicit and provided by Goten itself, implicitly, per each resource. For example, for the resource “Permission”, there will be a corresponding “PermissionService”.

Developer-defined APIs are typically declared below resources:

apis:
- name: $API_NAME # Should be UpperCamelCase format.

We did not put an equal mark between “Service package” and “API” to achieve smaller code packages and better granularity. Each resource and API has its code package. Custom actions are grouped according to a service developer, who should think what seems to make more sense or is more convenient. APIs can be considered as a “namespace” for actions.

Action

The action represents a single gRPC method. It can be attached to API or resource:

resources:
- name: $SOME_RESOURCE_NAME
  actions:
  - name: $ACTION_NAME
  # ... CONTINUED HERE ...

apis:
- name: $SOME_API_NAME
  actions:
  - name: $ACTION_NAME
  # ... CONTINUED HERE ...

Below a resource or an API, some common properties of an Action are:

actions:
- name: $ACTION_NAME
  verb: $ACTION_VERB # You can skip, this, and $ACTION_VERB will be equal to $ACTION_NAME and lowerCamelCased.
  opResourceInfo:
    name: $RESOURCE_ACTION_OPERATES_ON # Skip-able, if action is defined within resource already
    isCollection: $TRUE_IF_ACTION_OPERATES_ON_COLLECTION
    isPlural: $TRUE_IF_ACTION_OPERATES_ON_MULTIPLE_RESOURCES
    skipResourceInRequest: $TRUE_IF_REQUEST_DOES_NOT_CONTAIN_RESOURCE_NAME_OR_PARENT_NAME
    requestPaths: $PATHS_TO_RESOURCE_IN_REQUEST   # You can skip if defaults are used
    responsePaths: $PATHS_TO_RESOURCE_IN_RESPONSE # You can skip if not needed
  requestName: $REQUEST_NAME   # You can skip for default, which is ${ACTION_NAME}Request
  responseName: $RESPONSE_NAME # You can skip for default, which is ${ACTION_NAME}Response
  skipRequestMsgGen: $TRUE_IF_YOU_WANT_TO_SKIP_REQUEST_GEN_IN_PROTO_FILE
  skipResponseMsgGen: $TRUE_IF_YOU_WANT_TO_SKIP_RESPONSE_GEN_IN_PROTO_FILE
  streamingRequest: $TRUE_IF_CLIENT_IS_STREAMING
  streamingResponse: $TRUE_IF_SERVER_IS_STREAMING
  withStoreHandle:
    transaction: $LEVEL_FOR_TX
    readOnly: $TRUE_IF_NO_WRITES_EXPECTED

Boolean fields can be skipped if you plan to have “false”, unless you like explicit declarations.

Action - transaction

Fields you must set are name and withStoreHandle. The transaction part decides what happens in the transaction middleware when action is being processed by the backend. 3 types have to be specified for transactions:

• NONE

  we declare that no transaction is needed and all database requests should be handled without a transaction. Suitable for read-only requests.

• SNAPSHOT

  we declare that we will be making writes/deletions to the database after the reads. When transaction is concluded, the database must guarantee that all reads WOULD be repeatable (meaning, no one modified and resource/collection we read!). Note that this also included the “collection” part.

For Goten transactions and API skeleton, the word “SNAPSHOT” is a bit misleading, because what Goten offers here is SERIALIZABLE, since we also protect against write skews (which ARE NOT protected by SNAPSHOT). See https://en.wikipedia.org/wiki/Snapshot_isolation for more details, or https://www.cockroachlabs.com/blog/what-write-skew-looks-like/.

transaction level can also be set to MANUAL - in this case, Goten will not generate code starting a read-only session or snapshot transaction (generated transaction middleware in server code will not be present for given action). This is useful, if we deal with some special action for which we want to have for example many separate transactions executed, and we want to give the developer full control over when and how a transaction is started.

Action - request and response objects

The group of Acton API-skeleton options that should be considered together are requestName, responseName, skipRequestMsgGen and skipResponseMsgGen. Those are all optional, but it’s important to make informed decisions about them. While default request/response names are fine in many cases, occasionally you may want to use some specific, existing object for request or response. Consider actions like CreateRoleBinding in the iam.edgelq.com service. The request name is CreateRoleBindingRequest, but the response is just RoleBinding. For an action DeleteRoleBinding, the request is DeleteRoleBindingRequest, but the response name is google.protobuf.Empty (since we don’t need anything from the response). In those cases, the object is already defined and we don’t need to generate it. We would need to write something like:

- name: CreateRoleBinding
  responseName: RoleBinding
  skipResponseMsgGen: true
- name: DeleteRoleBinding
  responseName: google.protobuf.Empty
  skipResponseMsgGen: true

Action - unary, client streaming, server streaming, or bidi-streaming

The next important decision is to decide about Action in the API skeleton is what kind of action we are defining:

• Is it unary type, meaning single request and single response?

  If so, params streamingRequest and streamingResponse must be equal to false. In this case, you don’t need to write it, since false is a default.

• Sometimes what is needed is server streaming gRPC calls.

  Example case: WatchRoleBindings in iam.edgelq.com. The client first sends a single request, then the server keeps responding with responses (many). In this case, streamingRequest must remain false, but you must set streamingResponse to true.

• It is very rare, as of the moment of this writing theoretical, but action can be exclusively client streaming

  The client opens the stream and keeps sending requests. Example use case: Continuous logging submission. In that case, param streamingRequest must be set to true.

• Occasionally we need full bidirectional streaming

  In this case, both streamingResponse and streamingRequest must be set to true.

Action - operated resource

Almost every action interacts with some resource. For example, the action CreateRoleBinding in iam.edgelq.com operates on the RoleBinding resource. We need to define how action behaves on a resource using the opResourceInfo annotation. The most basic property is name there - we can skip this for Action defined within the resource, but we need to specify if action is defined for a custom API.

There are many modes in how action operates on a resource. For example, CreateRoleBinding operates on a single RoleBinding resource, but ListRoleBindings operates on a collection (or sub-collection). In Goten, we define 4 modes:

• Action operating on the single resource in isolation from the collection.

  Examples: Any Update or Delete operation. When you send for example UpdateRoleBinding, only a single instance of RoleBinding is affected, and the rest of the collection is isolated.

• Action operating on a single resource, but affecting collection (or sub-collection).

  An example of such an action is CreateRoleBinding. It creates only a single instance, but it DOES affect collection. Create operations mean you are inserting something into the collection, and Goten needs to check if the name is unique. The act of creating means that you are reserving some name within the collection namespace, affecting anyone else there.

• Actions operating on multiple resources in isolation from the collection.

  A good example here is BatchGetRoleBindings. You specify many specific instances, but still specific, with isolation to non-specified items.

• Actions operating on multiple resources affecting collection.

  classic is ListRoleBindings. You get many instances from collection (or sub-collection).

You need to pick which mode is active by using fields isCollection and isPlural within opResourceInfo.

A very important part of the action is the requestPaths param. It contains information on how to retrieve information about the resource(s) from the request object (in case of streaming, client message). A lot of code-generated parts/framework modules rely on this kind of information. For example, the Authorizer will extract resource name(s)/collection to determine if the caller has a right to execute an action for a given context. Another example the auditing component will need to know what resource(s) are affected by the action, so it can correctly define the activity logs associated.

Object requestPaths has defaults depending on params isCollection and isPlural.

If isCollection is true, then by default Goten assumes that the request object contains a “parent” field pointing to the affected sub-collection.

For example, ListRoleBindings is like:

syntax = "proto3";

message ListRoleBindings {
  // This annotation enforces that value of this string conforms to parent
  // name patterns of RoleBinding, see resource naming chapter in this
  // document.
  string parent = 1 [(goten.annotations.type).parent_name.resource = "RoleBinding"];
  
  // other fields ...
}

Note that Create requests normally have parent field!

For collection requests then, the default value of requestPaths is:

opResourceInfo:
  isCollection: true
  requestPaths:
    resourceParent:
    - parent

However, if the resource has no parents whatsoever, then the resourceParent slice is empty.

If isCollection is false, then Goten looks into isPlural to determine the default. If the action is of plural type, then the following default applies:

opResourceInfo:
  isCollection: true
  requestPaths:
    resourceName:
    - names

Note that this matches any BatchGet request:

syntax = "proto3";

message BatchGetRoleBindings {
  // This annotation enforces that each value of this slice conforms to
  // name patterns of RoleBinding, see resource naming chapter in this
  // document.
  repeated string names = 1 [(goten.annotations.type).name.resource = "RoleBinding"];
  
  // other fields ...
}

For the non-collection and non-plural actions, the default is:

opResourceInfo:
  isCollection: true
  requestPaths:
    resourceName:
    - name

And the request object is like:

syntax = "proto3";

message GetRoleBindings {
  // This annotation enforces that value of this string conforms to name
  // patterns of RoleBinding, see resource naming chapter in this document.
  string name = 1 [(goten.annotations.type).name.resource = "RoleBinding"];
  
  // other fields ...
}

Note that plural/singular use both resourceName annotations - Goten can figure out whether it deals with repeated or a single string.

Because requestPaths are so important for Action (Authorization, Auditing, Usage tracking…), this is de facto mandatory to specify. Even if default is used, during code generation Goten will fail complaining that requestPaths in API-skeleton don’t match those in actual request objects. For initial prototyping though, it is fine to fail first, then define all fields in the request object, finally correct api-skeleton and re-generate everything again.

Note that in requestPaths you can specify multiple possible field paths. The first populated will be picked. This handles cases where you may have oneof protobuf keywords in use. You may also specify resource body paths if entire objects are there.

If action is associated with the resource, but the request does not have explicit field paths showing which, it will be necessary to indicate that with option skipResourceInRequest:

opResourceInfo:
  skipResourceInRequest: true # If so, then requestPaths will be considered empty

This however renders certain aspects like Authorization more tricky. By default only system admins may execute this.

Param responsePaths is optional - it may be used by Audit/Usage metrics if contains some field paths. However, it should not be entirely overlooked. For example, if there is a special authorization to read some specific and sensitive fields in resources returned by the response, indicating field paths containing such resources will help authorization middleware clear those values from the response (before returning to the user)!

Implicit APIs and actions

For each resource, Goten declares an implicit API with the same name as the resource. Those implicit APIs will have CRUD actions:

• Create<ResourceName>
• Update<ResourceName>
• Delete<ResourceName>
• Get<ResourceName>
• BatchGet<ResourcePluralName>
• List<ResourcePluralName>
• Watch<ResourceName>
• Watch<ResourcePluralNames>

Names of those actions should be generally self-explanatory, the only exception may be watch. Note there are two versions of it - for single resource and collection. The singular version is in a way similar to Get<ResourceName>, The plural is similar to List<ResourcePluralName>. The significant difference is that while Get/List is unary, their Watch equivalents are server-streaming. After the first client request and first server response, the client should simply maintain connection and receive updates of resource/collection (as diff messages) in real time.

Multi-region Design

Goten comes with a framework for setting up multi-region environments. Even a single-region setup is considered just a special case of multi-regional (NumRegions is simply 1). Considering this, it is recommended to prepare an API-skeleton with multi-region in mind, but multi-region features can still be skipped. If desired, you can specify the following:

name: $SERVICE_NAME
proto:
  ## Stuff here...

disableMultiRegion: true

If you do this, then you don’t need to specify any multiRegion spec in any Action or Resource. You will still need to specify in which region your service will be running, but you can program without it. Your service will not have multi-region routing middleware either.

When it comes to the MultiRegion setup, the most important elements are concentrated around resources because resources are the actual state of the service. Code is running in all regions where it runs, actions that don’t operate on any resources can be easily executed without issues on any region. But there are important decisions to make about resources.

One important rule is that:

Each resource within a Service MUST belong to one particular region only. In many cases it is more simple: Edge devices, sites, data centers, etc. have normally some location, which we can easily pinpoint. Other some “policies” are more tricky, because we assume they should apply to all regions. In the case of those non-regional resources, it is necessary to specify the primary region responsible for them. Ultimately, we need to ensure database consistency across regions, and transactions cannot provide guarantees they do if a single resource could be written to by two regions.

Goten ensures that:

• All resources belong to a single region
• Read-only copies are asynchronously copied to all relevant regions (described later)!

The above ensures that resource writes are not breaking, but reads are executed on the nearest possible region.

In api-skeleton, we need to decide which resources are regional. We do this by setting proper scopeAttribute:

resources:
# This resource will be considered non-regional
- name: SomeResourceName1

# This resource will be considered regional
- name: SomeResourceName2
  scopeAttributes:
  - Region

# This resource will be considered regional, because parent is!
- name: SomeResourceName3
  parents:
  - SomeResourceName2

Note that the Region scope attribute, which is inherited automatically by all kid resources, adds the regions/{region}/ block to all resource names! Therefore, whenever you see a resource name with such a block, it means this is a regional resource, and the name itself reveals to which region the resource belongs.

The next thing to learn about designing with multi-region in mind is about an object called MultiRegionPolicy. This is a common protobuf object type defined in the Goten framework. A resource that has this object in its fields is called a multi-region policy-holder. Those types of resources need special annotation in the api-skeleton file. You should have seen this already in fact

• when we described tenant separation in this document. This annotation is very common for projects:

resources:
- name: Project
  multiRegion:
    isPolicyHolder: true

In SPEKTRA Edge, we specify three well-known resource types that are multi-region policy-holders:

• meta.goten.com/Service
• iam.edgelq.com/Organization
• iam.edgelq.com/Project

As you should notice. We recommend in fact to declare Project resource for your service already and if not, import iam.edgelq.com explicitly and make other resources child of iam.edgelq.com/Project. This way, if you already annotated which resources are regional, you may have completed prototyping your service for multi-region. At least in the API skeleton file, and if you do not have any tricky actions with some complex multi-region implications!

MultiRegionPolicy object specifies:

• Primary region ID
• All enabled region IDs
• Cross-region database synchronization criteria (by default, all resources under policy-holder are synchronized across all enabled regions).

The definition of the MultiRegionPolicy object in Goten is there: https://github.com/cloudwan/goten/blob/main/types/multi_region_policy.proto.

MultiRegionPolicy object defines multi-region settings for CHILD resources of policy-holder. It does not affect policy-holder itself! It should be easy to understand why though - note that resource Project is the top one, it does not have any parent resources. Its name pattern is simply projects/{project}.

If we have two create requests like below, we will have issues:

createProjectRequests:
- {"project": {"name": "projects/projectId", "multiRegionPolicy": {"defaultControlRegion": "us-west2", "enabledRegions": ["eastus2", "us-west2"]}}}
- {"project": {"name": "projects/projectId", "multiRegionPolicy": {"defaultControlRegion": "eastus2", "enabledRegions": ["eastus2", "us-west2"]}}}

Region us-west2 will accept the creation of projectId, and eastus2 will have the same project in its region. Because transactions can not guarantee uniqueness here, we are facing a conflict during asynchronous multi-region synchronization!

Therefore, policy-holder resources define multi-region policy for their child resources only, never themselves. Project resource itself is considered global for a service - it is automatically synchronized across all regions enabled in a service and its instances are “owned” by the primary region of a service. Note that a Service itself is a policyholder. When you create your service, you need to pick its primary region and deploy it to all regions where you want it to be running. This is the reason why meta.goten.com/Service resource is a policy-holder too - its enabledRegions field is automatically updated whenever you create a new deployment, and defaultControlRegion is set to the primary region of your service.

Let’s wrap this up with some examples: Let’s define service custom.edgelq.com, which MultiRegionPolicy will be:

{"defaultControlRegion": "us-west2", "enabledRegions": ["eastus2", "japaneast", "us-west2"]}

The API skeleton part is:

name: custom.edgelq.com

imports:
- meta.goten.com

resources:
# This collection is synchronized with iam.edgelq.com/Project
- name: Project
  multiRegion:
    isPolicyHolder: true

- name: EdgeDevice
  parents:
  - Project
  scopeAttributes:
  - Region

- name: Interface
  parents:
  - EdgeDevice

- name: AccessPolicy
  plural: AccessPolicies
  parents:
  - Project

# This resource type is managed by service admins, so child of Service
- name: DeviceType
  parents:
  - meta.goten.com/Service

Let’s declare 2 Project resources with this:

- name: projects/p1
  multiRegionPolicy:
    defaultControlRegion: us-west2
    enabledRegions: [japaneast, us-west2]
- name: projects/p2
  multiRegionPolicy:
    defaultControlRegion: eastus2
    enabledRegions: [eastus2, japaneast]

Let’s define the multi-region syncing/ownership situation of those projects. First of all - the resource Project is a global resource, therefore its situation is defined by MultiRegionSpec of the custom.edgelq.com Service record!

Therefore:

• Project projects/p1 will belong to us-west2, and its read-only copies will be distributed to regions “eastus2” and “japaneast”.
• Project projects/p2 will belong to us-west2, and its read-only copies will be distributed to regions “eastus2” and “japaneast”.

Note that the multiRegionPolicy object of the Project is not any factor here MultiRegionPolicy is applied to descending resources only, never policy-holders. Every resource not descending from policy-holder is subject to MultiRegionPolicy defined for a Service itself.

Now, let’s define some EdgeDevice instances:

- name: projects/p1/regions/japaneast/edgeDevices/dId
- name: projects/p1/regions/us-west2/edgeDevices/dId
- name: projects/p2/regions/japaneast/edgeDevices/dId
- name: projects/p2/regions/eastus2/edgeDevices/dId

What will happen is that:

• Resource projects/p1/regions/japaneast/edgeDevices/dId will belong to Region japaneast, and its read-only copy will go to us-west2.
• Resource projects/p1/regions/us-west2/edgeDevices/dId will belong to Region us-west2, and its read-only copy will go to japaneast.
• Resource projects/p2/regions/japaneast/edgeDevices/dId will belong to Region japaneast, and its read-only copy will go to eastus2.
• Resource projects/p2/regions/eastus2/edgeDevices/dId will belong to Region eastus2, and its read-only copy will go to japaneast.

Service will disallow creation of projects/p1/regions/eastus2/edgeDevices/- or projects/p2/regions/us-west2/edgeDevices/-. Note that read-only copies are distributed to all regions indicated by MultiRegionPolicy of Project ancestor, but ownership is indicated by name.

If we define some Interfaces:

- name: projects/p1/regions/japaneast/edgeDevices/dId/interfaces/ix
- name: projects/p1/regions/us-west2/edgeDevices/dId/interfaces/ix
- name: projects/p2/regions/japaneast/edgeDevices/dId/interfaces/ix
- name: projects/p2/regions/eastus2/edgeDevices/dId/interfaces/ix

What will happen is that:

• Resource projects/p1/regions/japaneast/edgeDevices/dId/interfaces/ix will belong to Region japaneast, and its read-only copy will go to us-west2.
• Resource projects/p1/regions/us-west2/edgeDevices/dId/interfaces/ix will belong to Region us-west2 and its read-only copy will go to japaneast.
• Resource projects/p2/regions/japaneast/edgeDevices/dId/interfaces/ix will belong to Region japaneast, and its read-only copy will go to eastus2.
• Resource projects/p2/regions/eastus2/edgeDevices/dId/interfaces/ix will belong to Region eastus2 and its read-only copy will go to japaneast.

Note that interfaces basically inherit region ownership from EdgeDevice resource, and syncing regions are provided still by MultiRegionPolicy of Projects.

For AccessPolicy resources:

- name: projects/p1/accessPolicies/ap
- name: projects/p2/accessPolicies/ap

• Resource projects/p1/accessPolicies/ap will belong to Region us-west2, and its read-only copy will get to japaneast.
• Resource projects/p2/accessPolicies/ap will belong to Region eastus2, and its read-only copy will get to japaneast

Note that ownership of AccessPolicies is decided by the defaultControlRegion field in the MultiRegionPolicy object of the relevant parent resource. Read-only copies are distributed to the remaining enabled regions for a project.

Finally, for a DeviceType resource like:

- name: services/custom.edgelq.com/deviceTyped/d1

All DeviceType instances will belong to the region us-west2, and their read copies be distributed to japaneast and eastus2, because this is what MultiRegionPolicy of services/custom.edgelq.com tells us.

Note that regions projects can use are limited to those defined for a service. Resources under the project are limited to regions the project specifies. This way tenants within the service only bear resources within their chosen regions.

Whenever the server gets any request, routing middleware will try to use the request to deduce actual regions that should execute the request. Routing middleware is code-generated based on API skeleton annotations. Goten has at the disposal some set of known templates, that can handle 99% of the cases. Normally autopilot can handle many cases, but if there is some tricky part, it’s recommended to continue reading this documentation part, especially the API skeleton MultiRegion annotations for Actions.

MultiRegion customizations for Resource

MultiRegionPolicy contains also an additional field, criteriaForDisabledSync. See MultiRegionPolicy documentation for that. However, it means that read-only copies can be prevented from being shared with other regions, even if the default is to always sync to all enabled regions. If it is important, from an application point of view, to enforce syncing across enabled regions, it can be done via API skeleton for a resource:

resources:
- name: SomeName
  multiRegion:
    syncType: ALWAYS

Value syncType can also be NEVER, in which case no read-only copies be made. This is useful for example when we know particular resources are not needed to be copied we can reduce some workload.

Another API skeleton customization we can make for resources is to disable code-generated multi-region routing for specific CRUD functions. We can do this with:

resources:
- name: SomeName
  multiRegion:
    skipCodeGenBasedRoutingBasicActions:
    - CreateSomeName

With annotation like this, it is possible also to write by hand code for multi-region routing.

MultiRegion customizations for Action (and defaults explained)

Handling actions, unlike resources, is much more difficult. For a resource, we can just take a name and deduce what region owns it, and where we can expect a read-only copy. Syncing is more straightforward. But request routing/stream proxying is a more difficult topic. Naturally, CRUD has some defaults generated. Custom actions - code generated is on the best effort.

There are generally 3 models of request/stream handling in middleware for region-routing:

• If the server receives a request that can be executed locally (write to the owned resource, or read from locally available resources), then middleware just passes the stream/request to the next one.

• If the server receives a request that has to be executed somewhere else, then it opens a connection to another region, sends a request, or opens a streaming proxy (for streaming requests). In this case, middleware does not pass anything to the next local middleware, it just passes data elsewhere. Ideally, we should avoid this, as an extra proxy just adds unnecessary latency.

• There is also a possibility, that a given request should be executed by more than 1 region. For example, imagine ListDevices request from all projects (not specific ones). Since Projects can be located in different regions, no single region is guaranteed to have devices from all projects. Routing middleware will then broadcast requests to all regions, but before doing so, it will modify the filter field to indicate, that they are interested in devices from a particular region only. Once middleware routing gets responses from ALL regions, it will merge all responses into one. If orderBy was specified, then it will need to sort them again and apply paging again. The response can be returned. Note that this particular model is more extreme and should be avoided with proper queries. However, this approach has some specific use cases and is supported by multi-region middleware routing.

Two things that need to be analyzed are:

• What is the resource name on which the action operates?

  Or in case of plural action, resource names? If collection, what is the collection (or sub-collection) value? This value needs to be extracted from the request.

• Once we get resource or collection name(s), should this be executed on the region owning it, or on the region just having its read-only copies?

Goten can deduce those values based on the following API skeleton properties of an Action: withStoreHandle AND opResourceInfo.requestPaths. The rule is simple: If the transaction indicates there will be database writes (SNAPSHOT or MANUAL), then Goten will assume action can be executed on the owning resource region. Similarly, requestPaths annotation is used to determine what are the field paths leading to the resource parent name/name/names.

Note: As of now, Goten does not support write actions working on multiple resources, it has to be single.

If we want to ensure Goten will generate routing middleware code that will force action to be executed on the owning region, we can provide the following annotation:

actions:
- name: SomeName
  multiRegionRouting:
    executeOnOwningRegion: true

It is first recommended to provide paths via opResourceInfo.requestPaths annotation in action, as this is common for many things, including Authorization, etc. However, if we want to use separate resource name/parent field paths, specifically for multi-region routing, we can:

Use this annotation for single resource actions:

actions:
- name: SomeName
  multiRegionRouting:
    resourceFieldPaths:
    - some.path_to.resource_name
    - alternative.path

If you have collection type actions (isPlural is true), then use the scopeFieldPaths annotation instead of resourceFieldPaths.

If you just have an explicit field path in a request object indicating a specific region ID that should execute the request, like:

syntax = "proto3";

message SomeRequest {
  // Region ID where request must be routed.
  string executing_region_id = 1;
}

In this case, you should use the following annotation:

actions:
- name: SomeName
  multiRegionRouting:
    regionIdFieldPaths:
    - executing_region_id

If code-gen multi-region routing is not possible in your case, you may need to explicitly disable it:

actions:
- name: SomeName
  multiRegionRouting:
    skipCodeGenBasedRouting: true

If you disable code-gen-based routing, you can write manually your handler later on, in the Golang.

On top of that, there is some special caveat regarding streams and multi-region routing - It is required that the first client message received from a stream will be able to determine routing behavior. If the stream needs routing, the middleware will open the stream to the proxy region and just forward the first message.

gRPC transcoding customizations

We have a gRPC transcoding feature that allows gRPC services to support REST API clients. You can read more in this document: https://cloud.google.com/endpoints/docs/grpc/transcoding

URL paths for each action will be provided in protobuf files via the google.http.api annotation. We will come back to this topic again in the document about prototyping service in protobuf files. But again, api-skeleton is used to create the first set of protobuf files and many of those files must stay as code-generated - including those defining gRPC transcoding. Therefore, all developer customizations for gRPC transcoding can be done in api-skeleton only. After we re-generate protobuf files from api-skeleton, we just need to verify correctness by looking at these files. In this part we will explain defaults/potential customizations but it is worth noting, that customizations are rarely needed, normally Goten can derive proper defaults.

Let’s first look at the transcoding table for all actions, all types, CRUD, optional Search, and custom ones:

Of course, the HTTP method and pattern must be unique across services. For this reason, as a standard, HTTP pattern contains: service version, resource’s name/parent (when relevant), HTTP method, and finally verb.

Simple examples:

ListRoleBindings in iam.edgelq.com service (version v1), for project p1 will have a REST API path:

/v1/projects/p1/roleBindings - Note that $parent is a valid RoleBinding parent name and, therefore contains “projects/” prefix too.

ListProjects, since they don’t have a parent, would have this path: v1/projects

SearchAlertingPolicies from monitoring (v4 version) would have this path: v4/projects/p1/regions/-/alertingPolicies:search. It assumes the parent is projects/p1/regions/-, therefore policies from the specific project but all regions.

Note that :$verb is often used to distinguish proper action - it uses verb param from Action annotation.

If you don’t have any specific issues/needs, you can finish the gRPC transcoding part now, otherwise, you can check some special customizations that can be made:

REST API paths - complete overrides.

If we need it, we can use the nuclear option and just completely define a path for action on our own. To achieve this, we need to use the http_path_overrides option for Action. Example:

actions:
- name: SomeCustomMethod
  grpcTranscoding:
    httpPathOverrides:
    - /very/custom/path
    - /other/custom/path

Goten bootstrap will produce the following annotation in proto files:

option (google.api.http) = {
 post : "/very/custom/path"
 additional_bindings : {
  post : "/other/custom/path"
 }
};

HTTP prefix

Suppose we have a mixin service “health” that:

• Has its own versioning (v1, v2, v3…).
• Can be attached to other services, but generally API is “separated”.
• We want to add it to any other service

When a user sends a request GET some.edgelq.com/v1alpha/topics, then we are calling the ListTopics method. Suppose that something is wrong with the connection, and we want to debug it. We can do that using the health endpoint. The user should then send the following request: GET some.edgelq.com/v1:healthCheck. We assume that health service serves a method with the verb healthCheck. However, this is not a very nice way of doing this, because “v1” and “v1alpha” are “on the top” and they may look like different versions of the same service. To separate mixin from proper service we can put additional prefixes in the path. For example, this looks better: GET some.edgelq.com/health/v1:healthCheck. This is how we can do it in the API skeleton for health service:

name: health.edgelq.com
proto:
 package:
   name: ntt.health
   currentVersion: v1
   goPackage: github.com/example/health
   protoImportPathPrefix: health/proto
 service:
   name: Health
   defaultHost: health.edgelq.com
   oauthScopes: https://apis.edgelq.com
   httpNamespacePrefix: health

apis:
- name: Health
  actions:
  - name: HealthCheck
    verb: healthCheck
    withStoreHandle:
      transaction: NONE

See field proto.package.service.httpNamespacePrefix. It will decorate all HTTP patterns for ALL methods in this mixin service.

Custom reference path capture

Almost every method associated with some resource contains name=$name or parent=$parent in its HTTP pattern. This is called here “captured reference path”. Those two variants are the most common (which one exactly depends on the isCollection param), but they are not non-negotiable. As an example, we can look at the update request, which has a different path: $resource.name. Generally, the field path in the HTTP pattern, if any, MUST reflect the real field path in a request object. Note that this is determined by the requestPaths annotation for an action. Custom single-resource, no-collection actions default to “name”, and collection ones to “parent”. If you change the field path name, names, or parent to something else, the HTTP capture path will also change.

Example (we assume API Version is v1):

message SomeActionRequest {
 string custom_name = 1 [ (goten.annotations.type).name.resource : "SomeResource"];
}

Api-skeleton file:

apis:
- name: SomeApi
  actions:
  - name: SomeAction
    opResourceInfo:
      name: SomeResource
      requestPaths:
        resourceName: [ "custom_name" ]

Annotation for REST API:

option (google.api.http) = {
 post : "/v1/{custom_name=someResources/*}:someAction"
};

If we have multiple alternatives, we can provide multiple items as resource name:

apis:
- name: SomeApi
  actions:
  - name: SomeAction
    opResourceInfo:
      name: SomeResource
      requestPaths:
        resourceName: [ "custom_name", "other_name" ]

In this case, goten-bootstrap will produce following REST API path:

option (google.api.http) = {
 post : "/v1:someAction"
};

The reason is that we can’t have “OR” in those patterns. To specify the exact reference client needs to simply populate the request body. The same story applies to any BatchGet request that has a “names” field (array). URL has no place for arrays like that, so the name pattern is simply $version/$collection:batchGet.

HTTP method

By default, every custom action uses the POST method. It can be changed simply with:

actions:
- name: SomeCustomMethod
  grpcTranscoding:
    httpMethod: PUT

How to remove :$verb from HTTP path:

actions:
- name: SomeCustomMethod
  grpcTranscoding:
    isBasic: true

However, this option should only be used reasonably for standard CRUD methods. It is provided here more for the completeness of this guide. Verb is something that is best in ensuring path uniqueness for custom methods.

How to customize the HTTP body field:

By default, the body field is equal to the whole request. It is a bit different though for create/update requests, where the body is mapped to resource fields only. If we want the user to be able to specify a selected field only, we can use the following the API skeleton option:

actions:
- name: SomeAction
  grpcTranscoding:
    httpBodyField: some_request_field

This is based on the assumption that SomeAction contains a field called some_request_field. Note that this will prevent users from setting other fields though.

2 - Auto-Generated Protobuf Files

Protobuf files describe:

• Resources - models, database indices, name patterns, views, etc.
• Request/Response object definitions (bodies)
• API groups, each with a list of methods
• Service package metadata information

Note that you can read about API just all by looking at protobuf files.

Example proto files for 3rd party app: https://github.com/cloudwan/inventory-manager-example/tree/master/proto/v1 You can also see files in the edgelq repository too.

Resource protobuf files

For each resource in the API specification, Goten will create 2 protobuf files:

• <resource_name>.proto

  This file will contain the proto definition of a single resource.

• <resource_name>_change.proto

  This file will contain the proto definition of the DIFF object of a resource.

Protobuf with Change object is used for Watch requests, real-time subscriptions.

Be aware that:

• goten-bootstrap will always overwrite <resource_name>_change.proto

  You should never write to it.

• File <resource_name>.proto will be generated for the first time only.

  If you change anything in the API skeleton later that would affect the proto file, you will need to either update the file manually in the way the bootstrap utility would, or rename the file and let a new one be generated. You will need to copy all manually written modifications back to the newly generated file. Typically it means resource fields and additional import files.

When a resource file is generated for the first time, it will have name and metadata fields, plus special annotations applicable for resources only. You will need to replace TODO sections in the resource.

The first notable annotation is google.api.resource, like:

option (google.api.resource) = {
  type : "inventory-manager.examples.edgelq.com/Site"
  pattern : "projects/{project}/regions/{region}/sites/{site}"
};

You should note that this annotation will always show you a list of all possible name patterns. Whenever you change something later in the API specification (parents or scopeAttributes), you will need to modify this annotation manually.

Second, a more important annotation is the one provided by Goten, for example:

option (goten.annotations.resource) = {
  id_pattern : "[a-zA-Z0-9_.-]{1,128}"              // This is default value, this is set initially from api-skeleton idPattern param!
  collection : "sites"                              // Always plural and lowerCamelJson
  plural : "sites"                                  // Equal to collection
  parents : "Project"                               // If there are many parents, we will have many "parents:"
  on_parent_deleted_behavior : ASYNC_CASCADE_DELETE // Highly recommended, typical in SPEKTRA Edge
  scope_attributes : "goten.annotations/Region"     // Set for regional resources
  async_deletion : false                            // If set to true, resource will not disappear immediately after deletion.
};

This one shows basic properties like a list of parents, scope attributes, or what happens when a parent is deleted. Parent deletion will always need to be set for each resource. From SPEKTRA Edge’s perspective, we recommend however cascade deletion (and better to do this asynchronously). You may do this in-transaction deletion if you are certain there will be no more than 10 kid resources at once. Especially project kids should use asynchronous cascade deletion. We strive to make project deletion rather a smooth process (although warning: SOFT delete option is not implemented yet).

Parameter async_deletion should have an additional note: When a resource is deleted, by default its record is removed from the database. However, if async_deletion is true, then it will stay till all backreferences are cleaned up (no resource points at us). In some cases it may take considerable time: for example large project deletion.

We recommend setting async_deletion to true for top resources, like Project.

References to other resources

Setting a reference to other resources is pretty straightforward, it follows this pattern:

message SomeResource {
  option (google.api.resource) = { ... };

  option (goten.annotations.resource) = { ... };
  
  string reference_to_resource_from_current_service = 3 [
    (goten.annotations.type).reference = {
      resource: "OtherResource"
      target_delete_behavior : BLOCK
    }
  ];

  string reference_to_resource_from_different_service = 4 [
    (goten.annotations.type).reference = {
      resource: "different.edgelq.com/DifferentResource"
      target_delete_behavior : BLOCK
    }
  ];
}

Note you always need to specify target deletion behavior. If you just want to hold the resource name, but it is not supposed to be a true reference, then you should use (goten.annotations.type).name.resource annotation.

References to resources from different services or different regions will implicitly switch to ASYNC versions of UNSET/CASCADE_DELETE!

Views

Reading methods (Get, BatchGet, List, Watch, Search - if enabled) normally have a field_mask field in their request bodies. Field mask selects which fields should be returned in the response, or the case of the watch, incremental real-time updates. Apart from field mask field, there is another one: view. View indicates the default field mask that should be applied. If both view and field_mask are specified in a request, then their masks are just merged.

There are the following view types available: NAME, BASIC, DETAIL, and FULL. The first one is a two-element field mask, with fields name and display_name (if it is defined in a resource!). The last one should be self-explanatory. Two other ones by default are undefined and if they are used, they will work as FULL ones. Developers can define any 4 of them, even NAME and FULL - those will be just overwritten. This can be done using annotation goten.annotations.resource.

message SomeResource {
  option (goten.annotations.resource) = {
    ...
    views : [
      {
        view : BASIC
        fields : [
          {path : "name"},
          {path : "some_field"},
          {path : "other_field"}
        ]
      },
      {
        view : DETAIL
        fields : [
          {path : "name"},
          {path : "some_field"},
          {path : "other_field"},
          {path : "outer.nested"}
        ]
      }
    ]
  };
}

Note that you need to specify fields using snake_case. You can specify nested fields too.

Database indices

List/Watch requests work on a “best effort” basis in principle. However, sometimes indices are needed for performance, or, like in the case of Firestore, to make certain queries even possible.

Database indices are declared in protobuf definitions in each resource. During startup, db-controller runtime uses libraries provided by Goten to ensure indices in protobuf match those in the database. Note that you should not create indices on your own unless for experimentation.

Let’s define some examples, for simplicity we show just name patterns and indices annotations, fields can be imagined:

message Device {
  option (google.api.resource) = {
    type : "example.edgelq.com/Device"
    pattern : "projects/{project}/devices/{device}"
  };

  option (goten.annotations.indices) = {
    composite : {
      sorting_groups : [
        {
          name : "byDisplayName",
          order_by : "display_name",
          scopes : [ "projects/{project}/devices/-" ]
        },
        {
          name : "bySerialNumber"
          order_by : "info.serial_number"
          scopes : [
            "projects/-/devices/-",
            "projects/{project}/devices/-"
          ]
        }
      ]
      filters : [
        {
          field_path : "info.model"
          required : true
          restricted_sorting_groups : [ "bySerialNumber" ]
        },
        {
          field_path : "info.maintainer_group"
          reference_patterns : [ "projects/{project}/maintanenceGroups/{maintanenceGroup}" ]
        }
      ]
    }
    single : [ {field_path : "machine_type"} ]
  };
}

There are two indices types: single-field and composite. Single should be pretty straightforward, you specify just the field path (can be nested with dots), and the index should be usable by this field. Composite indices are generated based on sorting groups combined with filters.

Composite indices are optimized for sorting - but as of now, only one sorting field is supported. However, if the sorting field is different from the name, then “name” is additionally added, to ensure sorting is stable. In the above example, composite indices can be divided into two groups - those with sorting by display_name, or info.serial_number.

Note that the sorting field path also is usable for filtering, therefore, if you just need a specific composite index for multiple fields for filtering, you can just pick some field that may be optionally used for sorting too. Apart from that, each sorting group has built-in filter support for name fields, for specified patterns only (scopes).

Attached filters can either be required (and if the filter is not specified in a query, it will not be indexed), or optional (each non-required filter doubles the amount of generated indices.)

Based on the above example, generated composite indices will be:

• filter (name.projectId) orderBy (display_name ASC, name.deviceId ASC)
• filter (name.projectId) orderBy (display_name DESC, name.deviceId DESC)
• filter (name.projectId, info.maintainer_group) orderBy (display_name ASC, name.deviceId ASC)
• filter (name.projectId, info.maintainer_group) orderBy (display_name DESC, name.deviceId DESC)
• filter (info.model) orderBy (info.serial_number ASC, name.projectId ASC, name.deviceId ASC)
• filter (info.model) orderBy (info.serial_number DESC, name.projectId DESC, name.deviceId DESC)
• filter (name.projectId, info.model) orderBy (info.serial_number ASC, name.deviceId ASC)
• filter (name.projectId, info.model) orderBy (info.serial_number DESC, name.deviceId DESC)
• filter (info.model, info.maintainer_group) orderBy (info.serial_number ASC, name.projectId ASC, name.deviceId ASC)
• filter (info.model, info.maintainer_group) orderBy (info.serial_number DESC, name.projectId DESC, name.deviceId DESC)
• filter (name.projectId, info.model, info.maintainer_group) orderBy (info.serial_number ASC, name.deviceId ASC)
• filter (name.projectId, info.model, info.maintainer_group) orderBy (info.serial_number DESC, name.deviceId DESC)

When we sort by display_name, to utilize the composite index, we should also filter by the projectId part of the name field. Additional sorting by name.deviceId part is added implicitly to any order. If we add info.maintainer_group to the filter, we will switch to a different composite index.

If we just filter by display_name (we can use > or < operators too!), and add filter by projectId part of the name, then one of those first composite indices will be used too.

When defining indices - be aware of multiplications. Each sorting group has two multipliers - the next multiply is the number of possible name patterns we add (scopes). Finally, for each non-required field, we multiply the number of indices by 2. Here we generated 12 composite indices and 1 single-field one. The amount of indices is important from the perspective of the database used, in Firestore we can have 200 indices typically per database, and in Mongo 64 per collection.

Cache indices

To improve performance & reduce database usage, Goten & SPEKTRA Edge utilize Redis as a database cache.

Service developers should carefully analyze which queries are mostly used, what is the update rate, etc. With goten cache, we support:

• Get/BatchGet queries

  caching is done by resource name. Invalidation happens for updated/deleted resources for specific instances.

• List/Search queries

  we cache by all query params (filter, parent name, order by, page, phrase in case of search, field mask). If a resource is updated/deleted/created, then we invalidate whole cached query groups by filter only. We will explain more with examples.

We don’t support cache for Watch requests.

To enable cache support for service it is required to:

• Provide cache annotation for each relevant resource in their proto files.
• In server code, during initialization, construct store objects with cache, it’s a very short amount of code.

Let’s define some indices, for simplicity, we show just name patterns and annotations specific to the cache:

message Comment {
  option (google.api.resource) = {
    type : "forum.edgelq.com/Comment"
    pattern : "messages/{message}/comments/{comment}"
    pattern : "topics/{topic}/messages/{message}/comments/{comment}"
  };

  option (goten.annotations.cache) = {
    queries : [
      {eq_field_paths : ["name"]},
      {eq_field_paths : ["name", "user"]}
    ]
    query_reference_patterns : [{
      field_path : "name",
      patterns : [
        "messages/-/comments/-",
        "topics/{topic}/messages/-/comments/-"
      ]
    }]
  };
};

By default, Goten generates this proto annotation for every resource when the resource is initiated for the first time, but a very minimal one, with the index for the name field only.

We will support caching for:

• Get/BatchGet requests

  it is enabled by default and the goten.annotations.cache annotation provides a way to disable it only. Users do not need to do anything here.

• Following List/Search queries which filter/parent SATISFY following filter conditions:

  • Group 1: name = "messages/-/comments/-”
  • Group 2: name = “topics/{topicId}/messages/-/comments/-”
  • Group 3: name = “messages/-/comments/-” AND user = “users/{userId}”
  • Group 4: name = “topics/{topicId}/messages/-/comments/-” AND user = “users/{userId}”

Since caching by exact name is very simple, we will be discussing only list/search queries.

We have 4 groups of indices. This is because:

• We have 2 query sets.

  one for name and, the other for name with user. The name field has 2 name patterns.

• Multiply 2 by 2, you have 4.

As a reminder, the presence of the “parent” field in List/Search requests already implies that the final filter will contain the “name” field.

Let’s put some example queries and how invalidation works then. Queries that will be cache-able:

• LIST { parent = 'topics/t1/messages/m1' filter = '' }

  It will belong to group 2.

• LIST { parent = 'topics/t1/messages/-' filter = '' }

  It will belong to group 2.

• LIST { parent = 'messages/-' filter = '' }

  It will belong to group 1.

• LIST { parent = 'messages/m1' filter = '' }

  It will belong to group 1.

• LIST { parent = 'topics/t1/messages/m1' filter = 'user=”users/u1”' }

  It will belong to groups 2 and 4.

• LIST { parent = 'topics/t1/messages/-' filter = 'user=”users/-”' }

  It will belong to group 2.

This query will not be cached: LIST { parent = 'topics/-/messages/-' filter = '' }

Note that exact queries may belong to more than one group. Also note that groups 3 and 4, which require a user, must be given full user reference without wildcards. If we wanted to enable caching also wildcards, then we would need to provide the following annotation:

 option (goten.annotations.cache) = {
   queries : [
     {eq_field_paths : [ "name" ]},
     {eq_field_paths : [ "name", "user" ]}
   ]
   query_reference_patterns : [ {
     field_path : "name",
     patterns : [
       "messages/-/comments/-",
       "topics/{topic}/messages/-/comments/-"
     ]
   }, {
     field_path : "user",
     patterns : [ "users/-" ]
   } ]
 };

The param that allows us to decide to which degree we allow for wildcards is query_reference_patterns. This param is actually “present” for every name/reference field within the resource body that is present in the queries param. The thing is, if the developer does not provide it, goten will assume some default. That default is to allow ALL name patterns - but allow the last segment of the name field to be a wildcard. In other words, the following annotations are equivalent:

 option (goten.annotations.cache) = {
   queries : [
     {eq_field_paths : [ "name" ]},
     {eq_field_paths : [ "name", "user" ]}
   ]
 };


 option (goten.annotations.cache) = {
   queries : [
     {eq_field_paths : [ "name" ]},
     {eq_field_paths : [ "name", "user" ]}
   ]
   query_reference_patterns : [ {
     field_path : "name",
     patterns : [
       "messages/{message}/comments/-",
       "topics/{topic}/messages/{message}/comments/-"
     ]
   }, {
     field_path : "user",
     patterns : [ "users/{user}" ]
   } ]
 };

Going back to our original 4 groups, let’s explain how invalidation works. Suppose that the following resource is created: Comment { name: “topics/t1/messages/m1/comments/c1”, user = “users/u1” }.

Goten will need to delete the following cached query sets:

• CACHED QUERY SET { name: “topics/t1/messages/-/comments/-” }

  filter group 2

• CACHED QUERY SET { name: “topics/t1/messages/m1/comments/-” }

  filter group 2

• CACHED QUERY SET { name: “topics/t1/messages/m1/comments/c1” }

  filter group 2

• CACHED QUERY SET { name: “topics/t1/messages/m1/comments/c1” user: “users/u1” }

  filter group 4

• CACHED QUERY SET { name: “topics/t1/messages/m1/comments/-” user: “users/u1” }

  filter group 4

• CACHED QUERY SET { name: “topics/t1/messages/-/comments/-” user: “users/u1” }

  filter group 4

You can notice that actually, 2 cached query sets may belong to the same filter group - it’s just with a wildcard and with a message specified. All cached query sets are generated from created comments. If the topic/message/user was different, then we would also have different query sets.

We can say, that we have: 2 query field groups, multiplied by 2 patterns for the name field, multiplied by 1 pattern for the user field, multiplied by 3 variants with wildcards in the name pattern. It gives 12 cached query sets for 4 filter groups.

List/Search query is also classified into query sets. For example, a request SEARCH { phrase = “Error” parent: “topics/t1/messages/m1” filter: “user = users/u2 AND metadata.tags CONTAINS xxx” } would be put in the following cached query sets: CACHED QUERY SET { name: “topics/t1/messages/m1/comments/-” user: “users/u2” }

Note that, unlike for resource instances, we are getting the biggest possible cached query set for actual queries. Thanks to that, if there is some update of comment for a specific user and message, then cached queries for the same message and OTHER users will not be invalidated. It’s worth considering this when designing proto-annotation. If a collection gets a lot of updates in general we are getting a lot of invalidations. In that case, it’s worth putting in more possible query field sets, so we are less affected by the high write rate. The more fields are specified, the less likely the update will cause invalidation.

The last remaining thing to mention regarding cache is what kind of filter conditions are supported. At this moment we cache by two conditions: Equality (=) and IN. In other words, request SEARCH { phrase = “Error” parent: “topics/t1/messages/m1” filter: “user IN [users/u2, users/u3] AND metadata.tags CONTAINS xxx” } would be put in the following cached query sets:

CACHED QUERY SET { name: “topics/t1/messages/m1/comments/-” user: “users/u2” } CACHED QUERY SET { name: “topics/t1/messages/m1/comments/-” user: “users/u3” }

Note that IN queries have a bigger chance of invalidation, because the update of comments from 2 users would cause invalidation. But it’s still better than all users.

Search Indices

If the search feature was enabled in the API specification for a given resource, to make it work it is necessary to add annotation for a resource.

We need to tell:

• Which fields should be fully searchable
• Which fields should be sortable
• Which fields should be filter-able only

Each of those field groups we can define via search specification in the resource. For example, let’s define search spec for an imaginary resource called “Message” (should be easy to understand):

message Message {
 option (google.api.resource) = {
   type : "forum.edgelq.com/Message"
   pattern : "messages/{message}"
   pattern : "topics/{topic}/messages/{message}"
 };

 option (goten.annotations.search) = {
  fully_searchable : [
   "name",                 // Name is also a string
   "user",                 // Some reference field (still string)
   "content",              // string
   "metadata.labels",      // map<string, string>
   "metadata.annotations", // map<string, string>
   "metadata.tags"         // []string
  ]
  filterable_only : [
   "views_count",          // integer
   "metadata.create_time"  // timestamp
  ]
  sortable : [
   "views",                // integer
   "metadata.create_time"  // timestamp
  ]
 };
}

Fully searchable fields will be text-indexed AND filterable. They do not only support string fields (name, content, user), they can also support more complex structures that contain strings internally (metadata tags, annotations, labels.) But generally, they should focus on strings. Filterable fields on the other hand can contain non-string elements like numbers, timestamps, booleans, etc. They will not be text-indexed, but can still be used in filters. As a general rule, developers should put string fields (and objects with strings) in a fully searchable category, otherwise is “filterable only”. Sortable fields are of course self-explanatory, they enable sorting for specific fields in both directions. However, during actual queries, only one field can be sorted at once.

Search backend in use may be different from service to service. However, it is the responsibility of the developer to ensure that their chosen backend will support ALL declared search annotations for all relevant resources.

API Group Protobuf Files

SPEKTRA Edge-based Service is a specific version represented by a single protobuf package. It contains multiple API groups, each containing a set of gRPC methods. By default, Goten creates one API group per resource, and its name is equal to that of a resource. By default, it contains CRUD actions, but the developer can add custom ones too in the API-skeleton file.

Files created by goten-bootstrap for each API group are the following:

• <api_name>_service.proto

  This file contains the definition of an API object with its actions from api-skeleton (with CRUD if applicable).

• <api_name>_custom.proto

  This file will contain definitions of requests/responses for custom actions. Each object contains a TODO section because again, this is something that goten cannot fully provide. Those custom files are created only when there are custom actions in the first place.

Files <api_name>_service.proto are generated each time goten-bootstrap is invoked. But <api_name>_custom.proto is generated for the first time only. If you for example add a custom action after the file exists, the request/response pair will not be generated. Instead, you will either need to rename (temporarily) existing files or add full objects manually. It is not a big issue, however, because code-gen just provides empty messages with an optionally single field inside, and a TODO section to populate the rest of the request/response body.

All API groups within the same service will of course share the same endpoint, they will just have different paths and generated code will be packaged per API.

Files ending with _service.proto should be inspected for beginners, or debugging/verification, as those contain action annotations that influence how the request is executed. Based on this example (snippet from inventory manager):

rpc ListReaderAgents(ListReaderAgentsRequest) returns (ListReaderAgentsResponse) {
  option (google.api.http) = {
    get : "/v1/{parent=projects/*/regions/*}/readerAgents"
  };
  option (goten.annotations.method) = {
    resource : "ReaderAgent"
    is_collection : true
    is_plural : true
    verb : "list"
    request_paths : {resource_parent : [ "parent" ]}
    response_paths : {resource_body : [ "reader_agents" ]}
  };
  option (goten.annotations.tx) = {
    read_only : true
    transaction : NONE
  };
  option (goten.annotations.multi_region_routing) = {
    skip_code_gen_based_routing : false
    execute_on_owning_region : false
  };
}

This declaration defines:

• What is the request, what is the response

• gRPC Transcoding via google.api.http annotation

  you can see HTTP method, URL path, capture reference. In this example, we could send HTTP GET /v1/projects/p1/regions/us-west2/readerAgents to get a list of agents in project p1, region us-west2. It would set the value of the “parent” field in ListReaderAgentsRequest to projects/p1/regions/us-west2

• Annotation goten.annotations.method provides basic information (usually self-explanatory). Important fields are those for request_paths and response_paths

  Usage, Auditing, Authorization, and MultiRegion routing depend on these fields, and they need to exist in request/response objects.

• Annotation (goten.annotation.tx) defines what transaction middleware does

  How the database handle is opened. NONE uses the current connection handle. SNAPSHOT will need a separate session.

• Annotation goten.annotations.multi_region_routing tells how the request is routed and if code-gen is used for it at all.

  In this case, since this is a reading request (List), we do not require a request to be executed on the region owning agents, it can be executed in the region where read-only copies are also available.

Note that all of this is copied/derived from the API specification.

Service Package Definition

Finally, among generated protobuf files there is one last time wrapping up information about the service package (with one version): <service_name>.proto. It looks like:

// Goten Service InventoryManager
option (goten.annotations.service_pkg) = {
  // Human friendly short name
  name : "ServiceName"
  
  // We will have meta.goten.com/Service resource with name services/service-name.edgelq.com
  domain : "service-name.edgelq.com"

  // Current version
  version : "v1"
  
  // All imported services
  imported_services : {
    domain : "imported.edgelq.com"
    version : "v1"
    proto_pkg : "ntt.imported.v1"
  }
};

There can be only one file within a proto package like this.

Goten Protobuf Types and other Annotations

When modeling service in Goten with protobuf files, it is just required to use normal proto in version 3 syntax. There are worth mentioning additional elements to consider:

Set of custom types (you should have seen many of them in standard CRUD):

message ExampleSet {
  // This string must conform to naming pattern of specified resource.
  string name_type = 1 [(goten.annotations.type).name.resource = "ResourceName"];
  
  // This string must conform to the naming pattern of specified resource. Also,
  // references in Goten are validated against actual resources (if specified within
  // resource).
  string reference_type = 2 [(goten.annotations.type).reference = {
    resource : "ResourceName"
    target_delete_behavior : ASYNC_CASCADE_DELETE
  }];
  
  // This string must conform to parent naming pattern of specified resource.
  string parent_name_type = 3 [(goten.annotations.type).parent_name.resource = "ResourceName"];
  
  // This string contains token used for pagination (list/search/watch queries). Its contents
  // are validated into specific value required by ResourceName.
  string cursor_type = 4 [(goten.annotations.type).pager_cursor.resource = "ResourceName"];
  
  // This should contain value like "field_name ASC". Field name must exist within specified ResourceName.
  string order_by_type = 5 [(goten.annotations.type).order_by.resource = "ResourceName"];
  
  // This should contain string with conditions using AND condition: We support  equality conditions (like ==, >),
  // IN, CONTAINS, CONTAINS-ANY, NOT IN, IS NULL... some specific queries may be unsupported by underlying
  // database though. Field paths used must exist within ResourceName.
  string filter_type = 6 [(goten.annotations.type).filter.resource = "ResourceName"];
  
  // This is the only non-string custom type. This annotation forces all values within
  // this mask to be valid within ResourceName.
  google.protobuf.FieldMask field_mask_type = 7 [(goten.annotations.type).field_mask.resource = "ResourceName"];
}

When modeling resources/requests/responses, it is important to keep in mind any input validation, to avoid bugs or more malicious intent. You should use annotations from here: https://github.com/cloudwan/goten/blob/main/annotations/validate.proto

An example is here: https://github.com/cloudwan/goten/blob/main/compiler/validate/example.proto

As of now, we don’t apply default string maximum values (we may in the future), so it is worth considering upfront.