Service Specification

At the beginning of this chapter, before continuing, it is worth first mentioning the naming conventions we use: https://cloud.google.com/apis/design/naming_convention

Example api-skeleton for 3rd party app: https://github.com/cloudwan/inventory-manager-example/blob/master/proto/api-skeleton-v1.yaml You can also see API skeletons in the edgelq repository, too.

This document describes api-skeletons in bigger detail than a quick startup.

When you start writing a service, the first (you have a new, empty directory for your service) you need is to do two things:

• Create a subdirectory called “proto” (convention used in all created goten services).

• In the proto directory, create file api-skeleton-$SERVICE_VERSION.yaml file.

  In place of $SERVICE_VERSION, you should put a version of your service, for example, v1alpha for a start.

API skeleton file is used by Goten to bootstrap initial proto files for your service - some of them will be initialized only once, some will always be overwritten by subsequent regeneration.

JSON schema

API-Skeleton schema is based on the protobuf file itself: https://github.com/cloudwan/goten/blob/main/annotations/bootstrap.proto You may check this file to see all possible options.

There is a useful trick you can do with your IDE, so it understands schema and can aid you with prototyping: https://github.com/cloudwan/goten/blob/main/schemas/api-skeleton.schema.json

In your IDE, find JSON Schema mappings, and give a path to this file (for example, to your cloned copy of goten). Match with file pattern api-skeleton. This way, IDE can help with writing it.

Generating protobuf files

Once the API-skeleton file is ready, you can generate protobuf files with:

goten-bootstrap -i "${SERVICEPATH}/proto/api-skeleton-$VERSION.yaml" \
  -o "${SERVICEPATH}/proto"
clang-format-12 -i "${SERVICEPATH}"/proto/$VERSION/**.proto

This utility is provided by the Goten repository - you should set up development env first to have this tool.

Variable SERVICEPATH must point to the directory of your service, and VERSION match the service version. It is highly recommended to use clang formatter on generated files - but there is no particular recommendation from version, 12 comes from current state of the scripts.

Note that you should re-generate every time something changes in api-skeleton.

Header part

The header shows basic information about your service:

name: $SERVICE_NAME
proto:
 package:
   name: $PROTO_PACKAGE_PREFIX
   currentVersion: $SERVICE_VERSION
   goPackage: $GITHUB_LINK_TO_YOUR_SERVICE
   protoImportPathPrefix: $DIRECTORY_NAME_WITH_SERVICE_CODE/proto
 service:
   name: $SERVICE_SHORT_NAME
   defaultHost: $SERVICE_NAME
   oauthScopes: https://apis.edgelq.com

• $SERVICE_NAME

  It must be exactly equal to the service you reserved (see introduction to developer guide).

• $PROTO_PACKAGE_PREFIX

  It will be used as a prefix for a proto package containing the whole of your service.

• $SERVICE_VERSION

  It shows a version of your service. It will also be used as a suffix for a proto package of your service.

• $GITHUB_LINK_TO_YOUR_SERVICE

  It must be an actual Github link.

• $DIRECTORY_NAME_WITH_SERVICE_CODE

  It must be equal to the directory name of your code.

• $SERVICE_SHORT_NAME

  It should be some short service name, not in “domain format”.

The header simply declares what service and what version is being offered. It is advisable to configure your IDE to include $DIRECTORY_NAME_WITH_SERVICE_CODE/proto in your proto paths - it will make traversing through IDE much simpler.

Imported services

Very often you will need to declare services your service imports, this is done usually below the header in the API-skeleton:

imports:
- $IMPORTED_SERVICE_NAME

You must provide the service name in imported if at least one of the below applies:

• One of the resources you declared in your service has a parent resource in the imported service.
• One of the resources you declared in your service has a reference to a resource in the imported service

You do NOT NEED to declare a service you are just “using” via its API. For example, if your client runtimes use proxies.edgelq.com for tunneling, but if you don’t use proxies.edgelq.com on the schema level, then you don’t need to import it.

Goten operates on No-SQL and No-relationship databases (Mongo, Firestore), so it provides its mechanism that provides those things. The benefit of Goten is that it can provide relationships not only between resources within your service but also across services. However, it needs in advance to know which services are going to be used. Runtime libraries/modules provided by Goten/SPEKTRA Edge ensure that databases across services are synchronized (for example, we don’t have a dangling reference that was supposed to be blocking deletion).

When you import a service, you must modify your goten-bootstrap call. For example, if you imported meta.goten.com service in version v1, then you need to run commands like:

goten-bootstrap -i "${SERVICEPATH}/proto/api-skeleton-$VERSION.yaml" \
  -o "${SERVICEPATH}/proto" \
  --import "${GOTENPATH}/meta-service/proto/api-skeleton-v1.yaml"
clang-format-12 -i "${SERVICEPATH}"/proto/$VERSION/**.proto

Note that GOTENPATH must point to the Goten code directory - and this path is the current reflection of the current code.

If you imported let’s say iam.edgelq.com, which imports meta.goten.com, you will need to provide import paths to all relevant API-skeletons:

goten-bootstrap -i "${SERVICEPATH}/proto/api-skeleton-$VERSION.yaml" \
  -o "${SERVICEPATH}/proto" \
  --import "${GOTENPATH}/meta-service/proto/api-skeleton-v1.yaml" \
  --import "${EDGELQROOT}/iam/proto/api-skeleton-v1.yaml"
clang-format-12 -i "${SERVICEPATH}"/proto/$VERSION/**.proto

As of now, goten-bootstrap needs field paths to directly and indirectly imported services.

Resources

Goten is resource-oriented, so you should organize your service around resources:

resources:
- name: $RESOURCE_SINGULAR_NAME # It should be in UpperCamelCase format
  plural: $RESOURCE_PLURAL_NAME # If not provided, it is $RESOURCE_SINGULAR_NAME with 's' added at the end.
  parents:
  - $RESOURCE_PARENT_SERVICE/$RESOURCE_PARENT_NAME # $RESOURCE_PARENT_SERVICE/ can be skipped if $RESOURCE_PARENT_NAME is declared in same service
  scopeAttributes:
  - $SCOPE_ATTRIBUTE_NAME
  idPattern: $ID_PATTERN_REGEX

Certain more advanced elements were omitted from above.

Standard Resource represents an object with the following characteristics:

• It has a name field that makes a unique identifier.
• It has a metadata field that contains meta information (sharding, lifecycle, etc.)
• It has an associated API group with the same name as Resource. That API contains CRUD actions for this resource and custom ones, added to the resource in the API-skeleton file.
• Has a collection in the database. That collection can be accessed via CRUD actions added implicitly by goten, OR more directly from server code via database handle.

Resources can be in parent-child relationships - including multiple parents' support, as you may see in the case of “Message” resource. However, you should also note that the resource “Comment”, despite having only one possible parent “Message”, can too have multiple ancestry paths.

Resource naming

Each resource has a unique identifier and name - it is stored also in a “name” field. Resource naming is a very important topic in Goten therefore it deserves a good explanation. The format of any name is the following:

$PARENT_NAME_BLOCK$SCOPE_ATTRIBUTES_NAME_BLOCK$SELF_IF_BLOCK

There are 3 blocks: $PARENT_NAME_BLOCK, then $SCOPE_ATTRIBUTES_NAME_BLOCK, and finally $SELF_IF_BLOCK.

Let us start with $SELF_IF_BLOCK, which has the following format: $resourcePluralNameCamelCase/$resourceId. It is always present and cannot be skipped. First part, $resourcePluralNameCamelCase is derived from $RESOURCE_PLURAL_NAME variable, but first and later is lower-cased. Variable $resourceId is assigned during creation and can never be updated. It must comply with the regex supplied with the variable $ID_PATTERN_REGEX in the api-skeleton file. Param idPattern can be skipped from resource definition - in that case, the default value will be applied: [a-z][a-z0-9\\-]{0,28}[a-z0-9].

The middle block, $SCOPE_ATTRIBUTES_NAME_BLOCK will be EMPTY if none scopeAttributes were defined for a resource in the api-skeleton file. By syntax, scopeAttributes is an array from 0 to N elements, like:

scopeAttributes:
- AttributeOne
- AttributeTwo

Block $SCOPE_ATTRIBUTES_NAME_BLOCK will be a concatenation of all scope attributes in declared order, for this example, it will be like: attributeOnes/$attributeOneId/attributeTwos/$attributeTwoId/. The last ‘/’ is to ensure it can be concatenated with $SELF_IF_BLOCK. Scope attributes also have singular/plural names and ID pattern regexes.

As of now, Goten provides only one built-in scope attribute that can be attached to a resource: Region. It means, that resource like:

name: SomeName
scopeAttributes:
- Region

will have the following name pattern: regions/$regionId/someNames/$someNameId. This built-in attribute Region is very special and has a significant impact on resources, but in essence: It shows that a resource has specific un-modifiable region it belongs to. All write requests for it will have to be executed by the region the resource belongs to. Region attributes should be considered when modeling for MultiRegion deployments. More details later in this doc.

Finally, we have a block $PARENT_NAME_BLOCK - it is empty if param parents were not present for the given resource in the API skeleton. Unlike scope attributes where all are active, a single resource instance can only have one active parent at the same time. When we specify multiple parents in the API skeleton, we just say that there are many alternate values to $PARENT_NAME_BLOCK. This param is the name of the parent resource. Each value of $PARENT_NAME_BLOCK is then structured in this way: $PARENT_NAME_BLOCK$SCOPE_ATTRIBUTES_NAME_BLOCK$SELF_IF_BLOCK/. Last / ensures that it can be concatenated with $SCOPE_ATTRIBUTES_NAME_BLOCK or $SELF_IF_BLOCK if the former is blank.

Top parent resource must have no parents at all.

It is possible to have an optional parent resource as well if we specify the empty string "" as a parent:

parents:
- SomeOptionalParent
- ""

In the above case, $PARENT_NAME_BLOCK will either be empty or end with someOptionalParents/$someOptionalParentId/.

Note that resource parent is a special kind of reference to different resource types. However, unlike regular references:

• The name of the resource contains actual references to ALL ancestral resources.

• Regular references are somewhere in the resource body

  not in the identifier. Therefore, for example, a GET request automatically shows us not only the resource we want to get but also its whole ancestry path.

• If parent is deleted, all kid resources must be automatically deleted

  asynchronously or in-transaction. Unlike in regular reference, a parent cannot be “unset”.

• Scope attributes from parents are automatically inherited by all child resources. It is not the case for regular references.

Throughout Goten, you may encounter some additional things about resource names:

• Wildcards

  for example name someResource/- indicates ANY resource of SomeResource kind.

• Parent names

  parent name is like name, but without $SELF_IF_BLOCK part. It indicates just the parent collection of resources.

Let’s consider some known resources in iam.edgelq.com service: RoleBinding. It has API-skeleton definition:

name: RoleBinding
parents:
- meta.goten.com/Service # Service is declared in different service
- Project
- Organization
- ""

From above, there are 4 valid name patterns RoleBinding can have:

• services/{service}/roleBindings/{roleBinding}
• projects/{project}/roleBindings/{roleBinding}
• organizations/{organization}/roleBindings/{roleBinding}
• roleBindings/{roleBinding}

Then, PARENT NAME patterns that are valid are (same order):

• services/{service}
• projects/{project}
• organizations/{organization}
• "" - just empty string

With wildcards, we can define (just examples):

• services/{service}/roleBindings/- - This indicates ANY RoleBinding from specific service
• services/-/roleBindings/- - This indicates ANY RoleBinding from ANY service
• services/-/roleBindings/{roleBinding} - This would pick all RoleBindings across all services having the same final ID.

Wildcards can be specified in parent names too.

Resource opt-outs

Developers can opt-out from specific standard features offered by Goten. In most cases, we need to do this for specific CRUD actions that are attached to all resources. For example:

resources:
- name: SomeResourceName
  optOuts:
    basicActions:
    - CreateSomeResourceName

Other cases are more tricky. Standard goten CRUD access, resourceChange, and metadata are used intensively by Goten/SPEKTRA Edge framework, even if you don’t use it yourself. For 3rd party developers, we recommend not disabling anything outside basic actions. Other opt-outs exist for resources that are NOT using a standard database (but a custom one, where developers provide their own driver). They will in this way escape the normal schema system (references to those resources will not work as usual).

Resource opt-ins

Some features are optional - as of now we have just one opt-in, it is a Search addition to the standard CRUD for resource objects. We can enable this in API-skeleton:

resources:
- name: SomeResourceName
  optIns:
    searchable: true

With the above, Goten will add the SearchSomeResourceNames action to the standard CRUD. The search method is very similar to List, but users can also specify a search phrase on top of the filter, field mask, and standard paging fields.

Note that enabling in api-skeleton is not sufficient. The developer will also have to:

• Specify (in protobuf files) a list of fields for search-text indexing
• Configure search store backend during deployment

As of now, we support Algolia search, but we plan to extend this to MongoDB too. In this case, we may be able to use same database for records and searches.

Tenant and authorization separation when prototyping resources

There are certain requirements that service developers must follow when developing on the SPEKTRA Edge platform.

The consideration here is a tenant/authorization separation. SPEKTRA Edge services are designed for multi-tenancy in the heart. Those are organized as two resources in service iam.edgelq.com: Organization and Project. The organization is meant to be a container for child Organizations and Projects. The project is a final tenant. Those resource types are top. They don’t have any parents or scope attributes. Their name patterns are, therefore:

• iam.edgelq.com/Project: projects/{project}
• iam.edgelq.com/Organization: organizations/{organization}

Most of the resources in all services should either belong to the Project as final tenant consumer, less typically Organization. Project is preferable due to the stronger integration:

• Monitoring time series can go to project only
• Limits are capable of limiting resource instances only within projects
• Usage metrics are counted per project.

For the above reasons, it is recommended to let Organization be a container for Projects on the core SPEKTRA Edge platform, and use Project resource as a parent for further resources. Therefore, you should apply the following practice in your service API skeleton:

resource:
# You should declare resource Project in your service!
# It must have specific multiRegion setup.
- name: Project
  multiRegion:
    isPolicyHolder: true

# Under project, you can define resource types that will belong to
# each tenant. Those are to be defined by you.
- name: $CUSTOM_RESOURCE_NAME_1
  parents:
  - Project

- name: $CUSTOM_RESOURCE_NAME_2
  parents:
  - Project

# This resource is still descending from Project, so its fine.
- name: $CUSTOM_RESOURCE_NAME_3
  parents:
  - $CUSTOM_RESOURCE_NAME_1
- - $CUSTOM_RESOURCE_NAME_2

The above setup is integrating with iam.edgelq.com Authorization already: Service IAM recognizes 4 authorization scopes:

• System level (root): /
• Organization level: organizations/
• Project level: projects/
• Service level: services/

This list of scopes matches possible parents of RoleBinding resources. By declaring resources under Project, we are utilizing well-known scope and project admins can manage RoleBindings on their own - and forbid other projects to see/modify their data.

Even if a service in development is meant to be used by a single user (like a private service), it is still recommended to use Project resource

• we will have just one instance in existence.

When you deploy such a service, you will need to configure synchronization between 2 collections: iam.edgelq.com/Project AND your.service.com/Project. You should copy those projects into your collection that are interested in your service.

The reason is that projects follow multi-service design: Their administrators should freely choose which services are used by their projects and which are not. If we have a copy of the Project resource from the iam.edgelq.com service, we can:

• Ensure that all projects in your service are those enabling the given service.
• If the project leaves your service, all child resources will be garbage-collected.
• Project defined by your service can have additional fields not present in iam.edgelq.com/Project.

Synchronization between collections across services will be explained in the fixture controller document.

It is of course recognized that not all resource types are suitable to be put under Project tenant - some resource types are meant to be commonly shared across many tenants, probably in read-only mode, with write reserved for service administrators. If you have resources like this, the best option may be to declare them under meta.goten.com/Service resource:

imports:
- meta.goten.com # Needed in order to use Service as a parent

resources:
- name: $SOME_SERVICE_RESOURCE
  parents:
  - meta.goten.com/Service

Specifically, we should NOT have something like:

resources:
- name: $SOME_GLOBAL_SERVICE_RESOURCE

The reason is that, by declaring global service resource on the root level, IAM permissions required for any CRUD will be on the root level. As a system owner, you will have access to this resource type, BUT users for your service will not have, and you won’t be able to grant them any permissions - because RoleBindings on the root / level cannot be created by anyone but SPEKTRA Edge platform administrators. However, 3rd party service admins will be able to create RoleBindings under the Service resource:

services/$your_service_name/roleBindings/$rbId. From this position, permissions can be granted to users by service admins to access those service-level resources.

You may optionally declare service resources like for a Project:

resources:
- name: Service
  multiRegion:
    isPolicyHolder: true

- name: $SOME_SERVICE_RESOURCE
  parents:
  - Service

However, it will cause the generation of Service resources in your service and you will need to copy your service record from meta.goten.com to your service. But this has some benefits:

• It is in some sense clearer

  While meta.goten.com contains all services in the system, your service will contain only services that are using “your service”. In this case, it will be 1 element collection.

• Other services than yours can become tenants of your service as well! You will need to copy a subset of services from meta.goten.com to your service

  A subset using your service. You can then add extra fields not available in the meta.goten.com service.

If you plan to expose your service to other services, you should declare your Service resource and set up the controller to synchronize meta.goten.com/Service with your.service.com/Service. You should synchronize a subset of services only.

If you wonder why not then handle Project resources in the same way:

imports:
- iam.edgelq.com

resources:
- name: $SOME_RESOURCE
  parents:
  - iam.edgelq.com/Project

In the above pattern, the benefit is, that you don’t have a Projects collection in your service - iam already has. However, it is not suitable if you plan to have a multi-tenant service, where the tenant is a Project. As was mentioned. The project is meant to be able to enable/disable services it uses at a whim. By using a synchronized owned collection of Projects, we can ensure that child resources of projects in your service can be properly cleaned up.

However, if you are certain that your service is meant to be used by private project(s) who are always going to use your service and cannot disable it. Then in fact it is a valid choice to use iam.edgelq.com/Project directly.

API

Before explaining API, let me explain the relationships between services, resources, APIs, and methods:

• Service package contains a set of resources and APIs, each resource and API belongs to a single Service Package.
• API contains a set of actions and each action belongs to a single API only. Each action can also be optionally associated with a single resource (primary resource for action).
• APIs can be either “developer-defined” or “provided with resources”. The primary difference between them is that developer-defined API is explicitly described in the API skeleton, while the other kind is implicit and provided by Goten itself, implicitly, per each resource. For example, for the resource “Permission”, there will be a corresponding “PermissionService”.

Developer-defined APIs are typically declared below resources:

apis:
- name: $API_NAME # Should be UpperCamelCase format.

We did not put an equal mark between “Service package” and “API” to achieve smaller code packages and better granularity. Each resource and API has its code package. Custom actions are grouped according to a service developer, who should think what seems to make more sense or is more convenient. APIs can be considered as a “namespace” for actions.

Action

The action represents a single gRPC method. It can be attached to API or resource:

resources:
- name: $SOME_RESOURCE_NAME
  actions:
  - name: $ACTION_NAME
  # ... CONTINUED HERE ...

apis:
- name: $SOME_API_NAME
  actions:
  - name: $ACTION_NAME
  # ... CONTINUED HERE ...

Below a resource or an API, some common properties of an Action are:

actions:
- name: $ACTION_NAME
  verb: $ACTION_VERB # You can skip, this, and $ACTION_VERB will be equal to $ACTION_NAME and lowerCamelCased.
  opResourceInfo:
    name: $RESOURCE_ACTION_OPERATES_ON # Skip-able, if action is defined within resource already
    isCollection: $TRUE_IF_ACTION_OPERATES_ON_COLLECTION
    isPlural: $TRUE_IF_ACTION_OPERATES_ON_MULTIPLE_RESOURCES
    skipResourceInRequest: $TRUE_IF_REQUEST_DOES_NOT_CONTAIN_RESOURCE_NAME_OR_PARENT_NAME
    requestPaths: $PATHS_TO_RESOURCE_IN_REQUEST   # You can skip if defaults are used
    responsePaths: $PATHS_TO_RESOURCE_IN_RESPONSE # You can skip if not needed
  requestName: $REQUEST_NAME   # You can skip for default, which is ${ACTION_NAME}Request
  responseName: $RESPONSE_NAME # You can skip for default, which is ${ACTION_NAME}Response
  skipRequestMsgGen: $TRUE_IF_YOU_WANT_TO_SKIP_REQUEST_GEN_IN_PROTO_FILE
  skipResponseMsgGen: $TRUE_IF_YOU_WANT_TO_SKIP_RESPONSE_GEN_IN_PROTO_FILE
  streamingRequest: $TRUE_IF_CLIENT_IS_STREAMING
  streamingResponse: $TRUE_IF_SERVER_IS_STREAMING
  withStoreHandle:
    transaction: $LEVEL_FOR_TX
    readOnly: $TRUE_IF_NO_WRITES_EXPECTED

Boolean fields can be skipped if you plan to have “false”, unless you like explicit declarations.

Action - transaction

Fields you must set are name and withStoreHandle. The transaction part decides what happens in the transaction middleware when action is being processed by the backend. 3 types have to be specified for transactions:

• NONE

  we declare that no transaction is needed and all database requests should be handled without a transaction. Suitable for read-only requests.

• SNAPSHOT

  we declare that we will be making writes/deletions to the database after the reads. When transaction is concluded, the database must guarantee that all reads WOULD be repeatable (meaning, no one modified and resource/collection we read!). Note that this also included the “collection” part.

For Goten transactions and API skeleton, the word “SNAPSHOT” is a bit misleading, because what Goten offers here is SERIALIZABLE, since we also protect against write skews (which ARE NOT protected by SNAPSHOT). See https://en.wikipedia.org/wiki/Snapshot_isolation for more details, or https://www.cockroachlabs.com/blog/what-write-skew-looks-like/.

transaction level can also be set to MANUAL - in this case, Goten will not generate code starting a read-only session or snapshot transaction (generated transaction middleware in server code will not be present for given action). This is useful, if we deal with some special action for which we want to have for example many separate transactions executed, and we want to give the developer full control over when and how a transaction is started.

Action - request and response objects

The group of Acton API-skeleton options that should be considered together are requestName, responseName, skipRequestMsgGen and skipResponseMsgGen. Those are all optional, but it’s important to make informed decisions about them. While default request/response names are fine in many cases, occasionally you may want to use some specific, existing object for request or response. Consider actions like CreateRoleBinding in the iam.edgelq.com service. The request name is CreateRoleBindingRequest, but the response is just RoleBinding. For an action DeleteRoleBinding, the request is DeleteRoleBindingRequest, but the response name is google.protobuf.Empty (since we don’t need anything from the response). In those cases, the object is already defined and we don’t need to generate it. We would need to write something like:

- name: CreateRoleBinding
  responseName: RoleBinding
  skipResponseMsgGen: true
- name: DeleteRoleBinding
  responseName: google.protobuf.Empty
  skipResponseMsgGen: true

Action - unary, client streaming, server streaming, or bidi-streaming

The next important decision is to decide about Action in the API skeleton is what kind of action we are defining:

• Is it unary type, meaning single request and single response?

  If so, params streamingRequest and streamingResponse must be equal to false. In this case, you don’t need to write it, since false is a default.

• Sometimes what is needed is server streaming gRPC calls.

  Example case: WatchRoleBindings in iam.edgelq.com. The client first sends a single request, then the server keeps responding with responses (many). In this case, streamingRequest must remain false, but you must set streamingResponse to true.

• It is very rare, as of the moment of this writing theoretical, but action can be exclusively client streaming

  The client opens the stream and keeps sending requests. Example use case: Continuous logging submission. In that case, param streamingRequest must be set to true.

• Occasionally we need full bidirectional streaming

  In this case, both streamingResponse and streamingRequest must be set to true.

Action - operated resource

Almost every action interacts with some resource. For example, the action CreateRoleBinding in iam.edgelq.com operates on the RoleBinding resource. We need to define how action behaves on a resource using the opResourceInfo annotation. The most basic property is name there - we can skip this for Action defined within the resource, but we need to specify if action is defined for a custom API.

There are many modes in how action operates on a resource. For example, CreateRoleBinding operates on a single RoleBinding resource, but ListRoleBindings operates on a collection (or sub-collection). In Goten, we define 4 modes:

• Action operating on the single resource in isolation from the collection.

  Examples: Any Update or Delete operation. When you send for example UpdateRoleBinding, only a single instance of RoleBinding is affected, and the rest of the collection is isolated.

• Action operating on a single resource, but affecting collection (or sub-collection).

  An example of such an action is CreateRoleBinding. It creates only a single instance, but it DOES affect collection. Create operations mean you are inserting something into the collection, and Goten needs to check if the name is unique. The act of creating means that you are reserving some name within the collection namespace, affecting anyone else there.

• Actions operating on multiple resources in isolation from the collection.

  A good example here is BatchGetRoleBindings. You specify many specific instances, but still specific, with isolation to non-specified items.

• Actions operating on multiple resources affecting collection.

  classic is ListRoleBindings. You get many instances from collection (or sub-collection).

You need to pick which mode is active by using fields isCollection and isPlural within opResourceInfo.

A very important part of the action is the requestPaths param. It contains information on how to retrieve information about the resource(s) from the request object (in case of streaming, client message). A lot of code-generated parts/framework modules rely on this kind of information. For example, the Authorizer will extract resource name(s)/collection to determine if the caller has a right to execute an action for a given context. Another example the auditing component will need to know what resource(s) are affected by the action, so it can correctly define the activity logs associated.

Object requestPaths has defaults depending on params isCollection and isPlural.

If isCollection is true, then by default Goten assumes that the request object contains a “parent” field pointing to the affected sub-collection.

For example, ListRoleBindings is like:

syntax = "proto3";

message ListRoleBindings {
  // This annotation enforces that value of this string conforms to parent
  // name patterns of RoleBinding, see resource naming chapter in this
  // document.
  string parent = 1 [(goten.annotations.type).parent_name.resource = "RoleBinding"];
  
  // other fields ...
}

Note that Create requests normally have parent field!

For collection requests then, the default value of requestPaths is:

opResourceInfo:
  isCollection: true
  requestPaths:
    resourceParent:
    - parent

However, if the resource has no parents whatsoever, then the resourceParent slice is empty.

If isCollection is false, then Goten looks into isPlural to determine the default. If the action is of plural type, then the following default applies:

opResourceInfo:
  isCollection: true
  requestPaths:
    resourceName:
    - names

Note that this matches any BatchGet request:

syntax = "proto3";

message BatchGetRoleBindings {
  // This annotation enforces that each value of this slice conforms to
  // name patterns of RoleBinding, see resource naming chapter in this
  // document.
  repeated string names = 1 [(goten.annotations.type).name.resource = "RoleBinding"];
  
  // other fields ...
}

For the non-collection and non-plural actions, the default is:

opResourceInfo:
  isCollection: true
  requestPaths:
    resourceName:
    - name

And the request object is like:

syntax = "proto3";

message GetRoleBindings {
  // This annotation enforces that value of this string conforms to name
  // patterns of RoleBinding, see resource naming chapter in this document.
  string name = 1 [(goten.annotations.type).name.resource = "RoleBinding"];
  
  // other fields ...
}

Note that plural/singular use both resourceName annotations - Goten can figure out whether it deals with repeated or a single string.

Because requestPaths are so important for Action (Authorization, Auditing, Usage tracking…), this is de facto mandatory to specify. Even if default is used, during code generation Goten will fail complaining that requestPaths in API-skeleton don’t match those in actual request objects. For initial prototyping though, it is fine to fail first, then define all fields in the request object, finally correct api-skeleton and re-generate everything again.

Note that in requestPaths you can specify multiple possible field paths. The first populated will be picked. This handles cases where you may have oneof protobuf keywords in use. You may also specify resource body paths if entire objects are there.

If action is associated with the resource, but the request does not have explicit field paths showing which, it will be necessary to indicate that with option skipResourceInRequest:

opResourceInfo:
  skipResourceInRequest: true # If so, then requestPaths will be considered empty

This however renders certain aspects like Authorization more tricky. By default only system admins may execute this.

Param responsePaths is optional - it may be used by Audit/Usage metrics if contains some field paths. However, it should not be entirely overlooked. For example, if there is a special authorization to read some specific and sensitive fields in resources returned by the response, indicating field paths containing such resources will help authorization middleware clear those values from the response (before returning to the user)!

Implicit APIs and actions

For each resource, Goten declares an implicit API with the same name as the resource. Those implicit APIs will have CRUD actions:

• Create<ResourceName>
• Update<ResourceName>
• Delete<ResourceName>
• Get<ResourceName>
• BatchGet<ResourcePluralName>
• List<ResourcePluralName>
• Watch<ResourceName>
• Watch<ResourcePluralNames>

Names of those actions should be generally self-explanatory, the only exception may be watch. Note there are two versions of it - for single resource and collection. The singular version is in a way similar to Get<ResourceName>, The plural is similar to List<ResourcePluralName>. The significant difference is that while Get/List is unary, their Watch equivalents are server-streaming. After the first client request and first server response, the client should simply maintain connection and receive updates of resource/collection (as diff messages) in real time.

Multi-region Design

Goten comes with a framework for setting up multi-region environments. Even a single-region setup is considered just a special case of multi-regional (NumRegions is simply 1). Considering this, it is recommended to prepare an API-skeleton with multi-region in mind, but multi-region features can still be skipped. If desired, you can specify the following:

name: $SERVICE_NAME
proto:
  ## Stuff here...

disableMultiRegion: true

If you do this, then you don’t need to specify any multiRegion spec in any Action or Resource. You will still need to specify in which region your service will be running, but you can program without it. Your service will not have multi-region routing middleware either.

When it comes to the MultiRegion setup, the most important elements are concentrated around resources because resources are the actual state of the service. Code is running in all regions where it runs, actions that don’t operate on any resources can be easily executed without issues on any region. But there are important decisions to make about resources.

One important rule is that:

Each resource within a Service MUST belong to one particular region only. In many cases it is more simple: Edge devices, sites, data centers, etc. have normally some location, which we can easily pinpoint. Other some “policies” are more tricky, because we assume they should apply to all regions. In the case of those non-regional resources, it is necessary to specify the primary region responsible for them. Ultimately, we need to ensure database consistency across regions, and transactions cannot provide guarantees they do if a single resource could be written to by two regions.

Goten ensures that:

• All resources belong to a single region
• Read-only copies are asynchronously copied to all relevant regions (described later)!

The above ensures that resource writes are not breaking, but reads are executed on the nearest possible region.

In api-skeleton, we need to decide which resources are regional. We do this by setting proper scopeAttribute:

resources:
# This resource will be considered non-regional
- name: SomeResourceName1

# This resource will be considered regional
- name: SomeResourceName2
  scopeAttributes:
  - Region

# This resource will be considered regional, because parent is!
- name: SomeResourceName3
  parents:
  - SomeResourceName2

Note that the Region scope attribute, which is inherited automatically by all kid resources, adds the regions/{region}/ block to all resource names! Therefore, whenever you see a resource name with such a block, it means this is a regional resource, and the name itself reveals to which region the resource belongs.

The next thing to learn about designing with multi-region in mind is about an object called MultiRegionPolicy. This is a common protobuf object type defined in the Goten framework. A resource that has this object in its fields is called a multi-region policy-holder. Those types of resources need special annotation in the api-skeleton file. You should have seen this already in fact

• when we described tenant separation in this document. This annotation is very common for projects:

resources:
- name: Project
  multiRegion:
    isPolicyHolder: true

In SPEKTRA Edge, we specify three well-known resource types that are multi-region policy-holders:

• meta.goten.com/Service
• iam.edgelq.com/Organization
• iam.edgelq.com/Project

As you should notice. We recommend in fact to declare Project resource for your service already and if not, import iam.edgelq.com explicitly and make other resources child of iam.edgelq.com/Project. This way, if you already annotated which resources are regional, you may have completed prototyping your service for multi-region. At least in the API skeleton file, and if you do not have any tricky actions with some complex multi-region implications!

MultiRegionPolicy object specifies:

• Primary region ID
• All enabled region IDs
• Cross-region database synchronization criteria (by default, all resources under policy-holder are synchronized across all enabled regions).

The definition of the MultiRegionPolicy object in Goten is there: https://github.com/cloudwan/goten/blob/main/types/multi_region_policy.proto.

MultiRegionPolicy object defines multi-region settings for CHILD resources of policy-holder. It does not affect policy-holder itself! It should be easy to understand why though - note that resource Project is the top one, it does not have any parent resources. Its name pattern is simply projects/{project}.

If we have two create requests like below, we will have issues:

createProjectRequests:
- {"project": {"name": "projects/projectId", "multiRegionPolicy": {"defaultControlRegion": "us-west2", "enabledRegions": ["eastus2", "us-west2"]}}}
- {"project": {"name": "projects/projectId", "multiRegionPolicy": {"defaultControlRegion": "eastus2", "enabledRegions": ["eastus2", "us-west2"]}}}

Region us-west2 will accept the creation of projectId, and eastus2 will have the same project in its region. Because transactions can not guarantee uniqueness here, we are facing a conflict during asynchronous multi-region synchronization!

Therefore, policy-holder resources define multi-region policy for their child resources only, never themselves. Project resource itself is considered global for a service - it is automatically synchronized across all regions enabled in a service and its instances are “owned” by the primary region of a service. Note that a Service itself is a policyholder. When you create your service, you need to pick its primary region and deploy it to all regions where you want it to be running. This is the reason why meta.goten.com/Service resource is a policy-holder too - its enabledRegions field is automatically updated whenever you create a new deployment, and defaultControlRegion is set to the primary region of your service.

Let’s wrap this up with some examples: Let’s define service custom.edgelq.com, which MultiRegionPolicy will be:

{"defaultControlRegion": "us-west2", "enabledRegions": ["eastus2", "japaneast", "us-west2"]}

The API skeleton part is:

name: custom.edgelq.com

imports:
- meta.goten.com

resources:
# This collection is synchronized with iam.edgelq.com/Project
- name: Project
  multiRegion:
    isPolicyHolder: true

- name: EdgeDevice
  parents:
  - Project
  scopeAttributes:
  - Region

- name: Interface
  parents:
  - EdgeDevice

- name: AccessPolicy
  plural: AccessPolicies
  parents:
  - Project

# This resource type is managed by service admins, so child of Service
- name: DeviceType
  parents:
  - meta.goten.com/Service

Let’s declare 2 Project resources with this:

- name: projects/p1
  multiRegionPolicy:
    defaultControlRegion: us-west2
    enabledRegions: [japaneast, us-west2]
- name: projects/p2
  multiRegionPolicy:
    defaultControlRegion: eastus2
    enabledRegions: [eastus2, japaneast]

Let’s define the multi-region syncing/ownership situation of those projects. First of all - the resource Project is a global resource, therefore its situation is defined by MultiRegionSpec of the custom.edgelq.com Service record!

Therefore:

• Project projects/p1 will belong to us-west2, and its read-only copies will be distributed to regions “eastus2” and “japaneast”.
• Project projects/p2 will belong to us-west2, and its read-only copies will be distributed to regions “eastus2” and “japaneast”.

Note that the multiRegionPolicy object of the Project is not any factor here MultiRegionPolicy is applied to descending resources only, never policy-holders. Every resource not descending from policy-holder is subject to MultiRegionPolicy defined for a Service itself.

Now, let’s define some EdgeDevice instances:

- name: projects/p1/regions/japaneast/edgeDevices/dId
- name: projects/p1/regions/us-west2/edgeDevices/dId
- name: projects/p2/regions/japaneast/edgeDevices/dId
- name: projects/p2/regions/eastus2/edgeDevices/dId

What will happen is that:

• Resource projects/p1/regions/japaneast/edgeDevices/dId will belong to Region japaneast, and its read-only copy will go to us-west2.
• Resource projects/p1/regions/us-west2/edgeDevices/dId will belong to Region us-west2, and its read-only copy will go to japaneast.
• Resource projects/p2/regions/japaneast/edgeDevices/dId will belong to Region japaneast, and its read-only copy will go to eastus2.
• Resource projects/p2/regions/eastus2/edgeDevices/dId will belong to Region eastus2, and its read-only copy will go to japaneast.

Service will disallow creation of projects/p1/regions/eastus2/edgeDevices/- or projects/p2/regions/us-west2/edgeDevices/-. Note that read-only copies are distributed to all regions indicated by MultiRegionPolicy of Project ancestor, but ownership is indicated by name.

If we define some Interfaces:

- name: projects/p1/regions/japaneast/edgeDevices/dId/interfaces/ix
- name: projects/p1/regions/us-west2/edgeDevices/dId/interfaces/ix
- name: projects/p2/regions/japaneast/edgeDevices/dId/interfaces/ix
- name: projects/p2/regions/eastus2/edgeDevices/dId/interfaces/ix

What will happen is that:

• Resource projects/p1/regions/japaneast/edgeDevices/dId/interfaces/ix will belong to Region japaneast, and its read-only copy will go to us-west2.
• Resource projects/p1/regions/us-west2/edgeDevices/dId/interfaces/ix will belong to Region us-west2 and its read-only copy will go to japaneast.
• Resource projects/p2/regions/japaneast/edgeDevices/dId/interfaces/ix will belong to Region japaneast, and its read-only copy will go to eastus2.
• Resource projects/p2/regions/eastus2/edgeDevices/dId/interfaces/ix will belong to Region eastus2 and its read-only copy will go to japaneast.

Note that interfaces basically inherit region ownership from EdgeDevice resource, and syncing regions are provided still by MultiRegionPolicy of Projects.

For AccessPolicy resources:

- name: projects/p1/accessPolicies/ap
- name: projects/p2/accessPolicies/ap

• Resource projects/p1/accessPolicies/ap will belong to Region us-west2, and its read-only copy will get to japaneast.
• Resource projects/p2/accessPolicies/ap will belong to Region eastus2, and its read-only copy will get to japaneast

Note that ownership of AccessPolicies is decided by the defaultControlRegion field in the MultiRegionPolicy object of the relevant parent resource. Read-only copies are distributed to the remaining enabled regions for a project.

Finally, for a DeviceType resource like:

- name: services/custom.edgelq.com/deviceTyped/d1

All DeviceType instances will belong to the region us-west2, and their read copies be distributed to japaneast and eastus2, because this is what MultiRegionPolicy of services/custom.edgelq.com tells us.

Note that regions projects can use are limited to those defined for a service. Resources under the project are limited to regions the project specifies. This way tenants within the service only bear resources within their chosen regions.

Whenever the server gets any request, routing middleware will try to use the request to deduce actual regions that should execute the request. Routing middleware is code-generated based on API skeleton annotations. Goten has at the disposal some set of known templates, that can handle 99% of the cases. Normally autopilot can handle many cases, but if there is some tricky part, it’s recommended to continue reading this documentation part, especially the API skeleton MultiRegion annotations for Actions.

MultiRegion customizations for Resource

MultiRegionPolicy contains also an additional field, criteriaForDisabledSync. See MultiRegionPolicy documentation for that. However, it means that read-only copies can be prevented from being shared with other regions, even if the default is to always sync to all enabled regions. If it is important, from an application point of view, to enforce syncing across enabled regions, it can be done via API skeleton for a resource:

resources:
- name: SomeName
  multiRegion:
    syncType: ALWAYS

Value syncType can also be NEVER, in which case no read-only copies be made. This is useful for example when we know particular resources are not needed to be copied we can reduce some workload.

Another API skeleton customization we can make for resources is to disable code-generated multi-region routing for specific CRUD functions. We can do this with:

resources:
- name: SomeName
  multiRegion:
    skipCodeGenBasedRoutingBasicActions:
    - CreateSomeName

With annotation like this, it is possible also to write by hand code for multi-region routing.

MultiRegion customizations for Action (and defaults explained)

Handling actions, unlike resources, is much more difficult. For a resource, we can just take a name and deduce what region owns it, and where we can expect a read-only copy. Syncing is more straightforward. But request routing/stream proxying is a more difficult topic. Naturally, CRUD has some defaults generated. Custom actions - code generated is on the best effort.

There are generally 3 models of request/stream handling in middleware for region-routing:

• If the server receives a request that can be executed locally (write to the owned resource, or read from locally available resources), then middleware just passes the stream/request to the next one.

• If the server receives a request that has to be executed somewhere else, then it opens a connection to another region, sends a request, or opens a streaming proxy (for streaming requests). In this case, middleware does not pass anything to the next local middleware, it just passes data elsewhere. Ideally, we should avoid this, as an extra proxy just adds unnecessary latency.

• There is also a possibility, that a given request should be executed by more than 1 region. For example, imagine ListDevices request from all projects (not specific ones). Since Projects can be located in different regions, no single region is guaranteed to have devices from all projects. Routing middleware will then broadcast requests to all regions, but before doing so, it will modify the filter field to indicate, that they are interested in devices from a particular region only. Once middleware routing gets responses from ALL regions, it will merge all responses into one. If orderBy was specified, then it will need to sort them again and apply paging again. The response can be returned. Note that this particular model is more extreme and should be avoided with proper queries. However, this approach has some specific use cases and is supported by multi-region middleware routing.

Two things that need to be analyzed are:

• What is the resource name on which the action operates?

  Or in case of plural action, resource names? If collection, what is the collection (or sub-collection) value? This value needs to be extracted from the request.

• Once we get resource or collection name(s), should this be executed on the region owning it, or on the region just having its read-only copies?

Goten can deduce those values based on the following API skeleton properties of an Action: withStoreHandle AND opResourceInfo.requestPaths. The rule is simple: If the transaction indicates there will be database writes (SNAPSHOT or MANUAL), then Goten will assume action can be executed on the owning resource region. Similarly, requestPaths annotation is used to determine what are the field paths leading to the resource parent name/name/names.

Note: As of now, Goten does not support write actions working on multiple resources, it has to be single.

If we want to ensure Goten will generate routing middleware code that will force action to be executed on the owning region, we can provide the following annotation:

actions:
- name: SomeName
  multiRegionRouting:
    executeOnOwningRegion: true

It is first recommended to provide paths via opResourceInfo.requestPaths annotation in action, as this is common for many things, including Authorization, etc. However, if we want to use separate resource name/parent field paths, specifically for multi-region routing, we can:

Use this annotation for single resource actions:

actions:
- name: SomeName
  multiRegionRouting:
    resourceFieldPaths:
    - some.path_to.resource_name
    - alternative.path

If you have collection type actions (isPlural is true), then use the scopeFieldPaths annotation instead of resourceFieldPaths.

If you just have an explicit field path in a request object indicating a specific region ID that should execute the request, like:

syntax = "proto3";

message SomeRequest {
  // Region ID where request must be routed.
  string executing_region_id = 1;
}

In this case, you should use the following annotation:

actions:
- name: SomeName
  multiRegionRouting:
    regionIdFieldPaths:
    - executing_region_id

If code-gen multi-region routing is not possible in your case, you may need to explicitly disable it:

actions:
- name: SomeName
  multiRegionRouting:
    skipCodeGenBasedRouting: true

If you disable code-gen-based routing, you can write manually your handler later on, in the Golang.

On top of that, there is some special caveat regarding streams and multi-region routing - It is required that the first client message received from a stream will be able to determine routing behavior. If the stream needs routing, the middleware will open the stream to the proxy region and just forward the first message.

gRPC transcoding customizations

We have a gRPC transcoding feature that allows gRPC services to support REST API clients. You can read more in this document: https://cloud.google.com/endpoints/docs/grpc/transcoding

URL paths for each action will be provided in protobuf files via the google.http.api annotation. We will come back to this topic again in the document about prototyping service in protobuf files. But again, api-skeleton is used to create the first set of protobuf files and many of those files must stay as code-generated - including those defining gRPC transcoding. Therefore, all developer customizations for gRPC transcoding can be done in api-skeleton only. After we re-generate protobuf files from api-skeleton, we just need to verify correctness by looking at these files. In this part we will explain defaults/potential customizations but it is worth noting, that customizations are rarely needed, normally Goten can derive proper defaults.

Let’s first look at the transcoding table for all actions, all types, CRUD, optional Search, and custom ones:

Of course, the HTTP method and pattern must be unique across services. For this reason, as a standard, HTTP pattern contains: service version, resource’s name/parent (when relevant), HTTP method, and finally verb.

Simple examples:

ListRoleBindings in iam.edgelq.com service (version v1), for project p1 will have a REST API path:

/v1/projects/p1/roleBindings - Note that $parent is a valid RoleBinding parent name and, therefore contains “projects/” prefix too.

ListProjects, since they don’t have a parent, would have this path: v1/projects

SearchAlertingPolicies from monitoring (v4 version) would have this path: v4/projects/p1/regions/-/alertingPolicies:search. It assumes the parent is projects/p1/regions/-, therefore policies from the specific project but all regions.

Note that :$verb is often used to distinguish proper action - it uses verb param from Action annotation.

If you don’t have any specific issues/needs, you can finish the gRPC transcoding part now, otherwise, you can check some special customizations that can be made:

REST API paths - complete overrides.

If we need it, we can use the nuclear option and just completely define a path for action on our own. To achieve this, we need to use the http_path_overrides option for Action. Example:

actions:
- name: SomeCustomMethod
  grpcTranscoding:
    httpPathOverrides:
    - /very/custom/path
    - /other/custom/path

Goten bootstrap will produce the following annotation in proto files:

option (google.api.http) = {
 post : "/very/custom/path"
 additional_bindings : {
  post : "/other/custom/path"
 }
};

HTTP prefix

Suppose we have a mixin service “health” that:

• Has its own versioning (v1, v2, v3…).
• Can be attached to other services, but generally API is “separated”.
• We want to add it to any other service

When a user sends a request GET some.edgelq.com/v1alpha/topics, then we are calling the ListTopics method. Suppose that something is wrong with the connection, and we want to debug it. We can do that using the health endpoint. The user should then send the following request: GET some.edgelq.com/v1:healthCheck. We assume that health service serves a method with the verb healthCheck. However, this is not a very nice way of doing this, because “v1” and “v1alpha” are “on the top” and they may look like different versions of the same service. To separate mixin from proper service we can put additional prefixes in the path. For example, this looks better: GET some.edgelq.com/health/v1:healthCheck. This is how we can do it in the API skeleton for health service:

name: health.edgelq.com
proto:
 package:
   name: ntt.health
   currentVersion: v1
   goPackage: github.com/example/health
   protoImportPathPrefix: health/proto
 service:
   name: Health
   defaultHost: health.edgelq.com
   oauthScopes: https://apis.edgelq.com
   httpNamespacePrefix: health

apis:
- name: Health
  actions:
  - name: HealthCheck
    verb: healthCheck
    withStoreHandle:
      transaction: NONE

See field proto.package.service.httpNamespacePrefix. It will decorate all HTTP patterns for ALL methods in this mixin service.

Custom reference path capture

Almost every method associated with some resource contains name=$name or parent=$parent in its HTTP pattern. This is called here “captured reference path”. Those two variants are the most common (which one exactly depends on the isCollection param), but they are not non-negotiable. As an example, we can look at the update request, which has a different path: $resource.name. Generally, the field path in the HTTP pattern, if any, MUST reflect the real field path in a request object. Note that this is determined by the requestPaths annotation for an action. Custom single-resource, no-collection actions default to “name”, and collection ones to “parent”. If you change the field path name, names, or parent to something else, the HTTP capture path will also change.

Example (we assume API Version is v1):

message SomeActionRequest {
 string custom_name = 1 [ (goten.annotations.type).name.resource : "SomeResource"];
}

Api-skeleton file:

apis:
- name: SomeApi
  actions:
  - name: SomeAction
    opResourceInfo:
      name: SomeResource
      requestPaths:
        resourceName: [ "custom_name" ]

Annotation for REST API:

option (google.api.http) = {
 post : "/v1/{custom_name=someResources/*}:someAction"
};

If we have multiple alternatives, we can provide multiple items as resource name:

apis:
- name: SomeApi
  actions:
  - name: SomeAction
    opResourceInfo:
      name: SomeResource
      requestPaths:
        resourceName: [ "custom_name", "other_name" ]

In this case, goten-bootstrap will produce following REST API path:

option (google.api.http) = {
 post : "/v1:someAction"
};

The reason is that we can’t have “OR” in those patterns. To specify the exact reference client needs to simply populate the request body. The same story applies to any BatchGet request that has a “names” field (array). URL has no place for arrays like that, so the name pattern is simply $version/$collection:batchGet.

HTTP method

By default, every custom action uses the POST method. It can be changed simply with:

actions:
- name: SomeCustomMethod
  grpcTranscoding:
    httpMethod: PUT

How to remove :$verb from HTTP path:

actions:
- name: SomeCustomMethod
  grpcTranscoding:
    isBasic: true

However, this option should only be used reasonably for standard CRUD methods. It is provided here more for the completeness of this guide. Verb is something that is best in ensuring path uniqueness for custom methods.

How to customize the HTTP body field:

By default, the body field is equal to the whole request. It is a bit different though for create/update requests, where the body is mapped to resource fields only. If we want the user to be able to specify a selected field only, we can use the following the API skeleton option:

actions:
- name: SomeAction
  grpcTranscoding:
    httpBodyField: some_request_field

This is based on the assumption that SomeAction contains a field called some_request_field. Note that this will prevent users from setting other fields though.