Developing your Service

How to develop your service.

Full example of sample service: https://github.com/cloudwan/inventory-manager-example

Service development preparation steps (as described in the introduction):

  • Reserving service name (domain format) using IAM API.
  • Creating a repository for the service.
  • Install Go SDK in the minimal version or better, the highest.
  • Setting up development - cloning edgelq, goten, setting env variables.

In your repository, you should first:

  • Create a proto directory.
  • Create a proto/api-skeleton-$VERSION.yaml file
  • Get familiar with API Skeleton doc and write some minimal skeleton. You can always come back at some point later on.
  • Generate protobuf files using the goten-bootstrap tool, which is described in the api-skeleton doc.

After this, service can be worked on, using this document and examples.

Further initialization

With api-skeleton and protobuf files you may model your service, but at this point, you need to start preparing some other common files. First is the go.mod file, which should start with something like this:

module github.com/your_organization/your_repository # REPLACE ME!

go 1.22

require (
	github.com/cloudwan/edgelq v1.X.X # PUT CURRENT VERSIONS
	github.com/cloudwan/goten v1.X.X  # PUT CURRENT VERSIONS
)

replace (
	cloud.google.com/go/firestore => github.com/cloudwan/goten-firestore v1.9.0
	google.golang.org/protobuf => github.com/cloudwan/goten-protobuf v1.26.1
)

Note that we have two special forks that are required in SPEKTRA Edge-based service.

The next crucial file is regenerate.sh, which we typically put at the top of the code repository. Refer to InventoryManager example application.

It includes steps:

  • Setting up the PROTOINCLUDE variable (also via script in SPEKTRA Edge repo)
  • Calling goten-bootstrap with clang formatter to create protobuf files
  • Generating server/client libraries (set of protoc calls)
  • Generating descriptor for REST API transcoding
  • Generating controller code (if business logic controller is needed)
  • Generating code for config files

For the startup part, you should skip business logic controller generation, as you may not have it (or need it). Config files (in the config directory), you should start by copying from the example here: https://github.com/cloudwan/inventory-manager-example/tree/master/config.

You should copy all *.proto files and config.go. You may need to remove the business logic controller config part if you don’t need it.

Another note about config files is resource sharding: For API server config, you must specify the following sharding:

  • byName (always)
  • byProjectId (if you have any resources where the parent contains Project)
  • byServiceId (if you have any resources where parent contains Service)
  • byOrgId (if you have any resources where parent contains Organization)
  • byIamScope (if you have resources where the parent contains either Project, Service, or Organization - de facto always).

Once this is done, you should execute regenerate.sh, and you will have almost all the code for the server, controllers and CLI utility ready.

Whenever you modify a Golang code, or after the regenerate.sh call, you may need to run:

go mod tidy # Ensures dependencies are all good

This will update the go.mod and go.sum files, you need to ensure all dependencies are in sync.

At this point, you are ready to start implementing your service. In the next parts, we will describe what you can find in generated code, and provide various advice on how to write code for your apps yourself.

Generated code

All Golang-generated files have .pb. in its file name. Developers can, and should in some cases, extend generated code (structs) with handwritten files using non-pb extensions. They will not be deleted.

We will describe briefly generated code packages and mention where manually written files have to be added.

Resource packages

The first directory you can explore generated code is the resources directory, it contains one package per resource per API version. Within a single resource module like this, we can find:

  • <resource_name>.pb.access.go

    Contains access interface for a resource collection (CRUD). It may be implemented by a database handle or API client.

  • <resource_name>.pb.collections.go

    Generated collections, developed from times before generics were introduced into Golang. We have standard maps/lists.

  • <resource_name>.pb.descriptor.go

    This file is crucial for the development of generic components. It contains a definition of a descriptor that is tied to a specific resource. It was inspired by the protobuf library, where each proto message has its descriptor. Here we do the same, but this descriptor more focuses on creating resource-specific objects without knowing the type. Descriptors are also registered globally, see github.com/cloudwan/goten/runtime/resource/registry.go.

  • <resource_name>.pb.fieldmask.go

    Contains generated type-safe field mask for a specific resource. Paths should be built with a builder, see below.

  • <resource_name>.pb.fieldpath.go

    Contains generated type-safe field path for a specific resource. Users don’t necessarily need to know its workings, apart from interfaces. Each path should be built with the builder and IDE should help show what is possible to do with field paths.

  • <resource_name>.pb.fieldpathbuilder.go

    Developers are recommended to use this file and its builder. It allows the construction of field paths, also with value variants.

  • <resource_name>.pb.filter.go

    Contains generated type-safe filter for a specific resource. Developers should rather not attempt to build filters directly from this but rather use a builder.

  • <resource_name>.pb.filterbuilder.go

    Developers are recommended to use filter builder in those files. It allows simple concatenations of conditions using functions like Where().Path.Eq(value).

  • <resource_name>.pb.go

    Contains generated resource model in Golang, with getters and setters.

  • <resource_name>.pb.name.go

    Contains Name and Reference objects generated for a specific resource. Note that those types are struct in Go, but string in protobuf. However, this allows much easier manipulation of names/references compared to standard strings.

  • <resource_name>.pb.namebuilder.go

    Contains use-to-use builder for name/reference/parent name types.

  • <resource_name>.pb.object_ext.go

    Contains additional utility-generated functions for copying/merging, and diffing.

  • <resource_name>.pb.pagination.go

    Contains types used by pagination components. Usually, developers don’t need to worry about them, but the function MakePagerQuery is often helpful to construct an initial pager.

  • <resource_name>.pb.parentname.go

    It is like its name equivalent but contains a name object for the parent. This file exists for resources with possible parents.

  • <resource_name>.pb.query.go

    Contains query objects for CRUD operations. Should be used with the Access interface.

  • <resource_name>.pb.validate.go

    Generated validation functions (based on goten annotations). They are automatically called by the generated server code.

  • <resource_name>.pb.view.go

    Contains function to generate default field mask from view object.

  • <resource_name>_change.pb.change.go

    Contains additional utility functions for the ResourceChange object.

  • <resource_name>_change.pb.go

    Contains model of change object in Golang.

  • <resource_name>_change.pb.validate.go

    Generated validation functions (based on goten annotations) but for Change object.

Generated types often implement common interfaces as defined in the package github.com/cloudwan/goten/runtime/resource. Notable interfaces: Access, Descriptor, Filter, Name, Reference, PagerQuery, Query, Resource, Registry (global registry of descriptors).

Field mask/Field path base interfaces can be found in module github.com/cloudwan/goten/runtime/object.

While by default resource packages are considered complete and can be used out of the box, often some additional methods extending resource structs are implemented in separate files.

Client packages

Higher-level modules from resources can be found in client, this is typically the second directory to explore. It contains one package per API group plus one final glue package for the whole service (in a specific version).

API group package directory contains:

  • <api_name>_service.pb.go

    Contains definitions of request/response objects, but excluding those from _custom.proto files.

  • <api_name>_custom.pb.go

    Contains definitions of request/response objects from _custom.proto files.

  • <api_name>_service.pb.validate.go

    Contains validation utilities of request/response objects, excluding those from _custom.proto files.

  • <api_name>_custom.pb.validate.go

    Contains validation utilities of request/response objects from _custom.proto files.

  • <api_name>_service.pb.client.go

    Contains wrapper around gRPC connection object. The wrapper contains all actions offered by an API group in type type-safe manner.

  • <api_name>_service.pb.descriptors.go

    Contains descriptor per each method and one per whole API group.

Usually, developers will need to use just the client wrapper and request/response objects.

Descriptors in this case are more usable for maintainers building generic modules, modules responsible for things like Auditing and usage tracing use method descriptors. Those are often using annotations derived from the API skeleton, like requestPaths.

Client modules contain one final “all” package - it is under a directory name having a short service name. It contains typically two files:

  • <service_short_name>.pb.client.go

    Combines API wrappers from all API groups together as one bundle.

  • <service_short_name>.pb.descriptor.go

    Contains descriptor for the whole service in a specific version, with all metadata. Used for generic modules.

When developing applications, developers are encouraged to maintain a single gRPC Connection object and use only those wrappers (clients for API groups) that are needed. This should reduce compiled binary sizes.

Client packages can be usually considered complete - developers don’t need to provide anything there.

Store packages

Directory store contains packages building on top of resources, and is used by server binary. There is one package per resource plus one final wrapper for the whole service.

Within the resource store package we can find:

  • <resource_name>.pb.cache.go

    It has generated code specifically for cache. It is based on cache protobuf annotations.

  • <resource_name>.pb.store_access.go

    It is a wrapper that takes the store handle in the constructor. It provides convenient CRUD access to resources. Note that this implements interface Access defined in <resource_name>.pb.access.go files (In the resources directory).

One common package for the whole service has a name equal to the short service name. It contains files:

  • <service_short_name>.pb.cache.go

    It wraps up all cache descriptors from all resources.

  • <service_short_name>.pb.go

    It takes generic store handle in the constructor and wraps to provide an interface with CRUD for all resources within the service.

There are no known cases where some custom implementation had ever to be provided within those packages, it can be considered complete on its own.

Server packages

Goten/SPEKTRA Edge strives to provide as much ready-to-use code as possible, and this includes almost full server code in the server directory. Each API group has a separate package, but there is one additional overarching package gluing all API groups together.

For each API group, we have for the server side:

  • <api_name>_service.pb.grpc.go

    Server handler interfaces (per each API group)

  • <api_name>_service.pb.middleware.routing.go

    MultiRegion middleware layer.

  • <api_name>_service.pb.middleware.authorization.go

    Authorization middleware layer (but see more in IAM integration)

  • <api_name>_service.pb.middleware.tx.go

    Transaction middleware layer, regulating access to the store for the call.

  • <api_name>_service.pb.middleware.outer.go

    Outer middleware layer - with validation, Compare And Swap checks, etc.

  • <api_name>_service.pb.server.core.go

    Core server that handles all CRUD functions already.

Note that for CRUD, everything is provided fully out of the box, but often there are custom actions or some extra steps required for some basic CRUD, in that case, it is recommended to write custom middleware between outer middleware and core.

Directory server contains also a glue package for a whole service in a specific version, with files:

  • <service_short_name>.pb.grpc.go

    Constructs server interface by gluing interfaces from all API groups

  • <api_name>_service.pb.middleware.routing.go

    Glue for multiRegion middleware layer.

  • <api_name>_service.pb.middleware.authorization.go

    Glue for the authorization middleware layer

  • <api_name>_service.pb.middleware.tx.go

    Glue for the transaction middleware layer

  • <api_name>_service.pb.middleware.outer.go

    Glue for the outer middleware layer

  • <api_name>_service.pb.server.core.go

    Glue for a core server that handles all CRUD functions.

This last directory with glue will also need manually written code files, like https://github.com/cloudwan/inventory-manager-example/blob/master/server/v1/inventory_manager/inventory_manager.go.

Note that this example server constructor shows the order of middleware execution. It corresponds to the process described in the prerequisites.

Be aware, that transaction middleware MAY be executed more than once for SNAPSHOT transaction types, in case we get ABORTED error. Transaction is retried a couple (typically 10) times. This also means that all middleware after TX must contain code that can be executed more than once. The database is guaranteed to reverse any write changes, BUT it is important to keep a check on another state (for example, if we send requests to other services and a transaction fails, those won’t be reversed!). If we change the request body, changes will be present in the request object on the second run too!

Apart from this, developers need to provide files only if there is a need for custom middleware (which fairly is needed always to some extent)

cli packages

Packages cli are used to create the simplest CLI utility based on cuttle. It is complete and only main.go file will be needed later on (to be explained later).

audit handlers packages

Packages for audithandlers contain one package per version for the whole service-generated handlers for all audited methods and resources. It is complete and some minor customizations are only needed, see the Audit integration document part. These packages need only inclusion in the main file, during server initialization. It is not necessarily needed to understand internal workings here.

access packages

Packages for the access directory contain modules that are built around client ones. There are two differences here.

First, while client contains basic objects for client-side code, access is delivering more high-level modules, that are not necessarily needed for all clients. Splitting them into separate packages allows clients to pick smaller packages.

Second, while client packages are built in one-per-API-group mode, access packages are built on the one-per-resource-type basis, and are focused on CRUD functionality only.

In access, each resource has its package, and finally, we have one glue package for the whole service.

Files generated for each resource:

  • <resource_name>.pb.api_access.go

    This implements interface Access defined in <resource_name>.pb.access.go files (In the resources directory). In the constructor, it takes the client interface as defined in <resource_name>_service.pb.client.go file (In the client directory), the one containing CRUD methods.

  • <resource_name>.pb.query_watcher.go

    Lower level watcher built around Watch<CollectionName> method. It takes the client interface and channel where it will be supplying events in real-time. It simplifies the handling of Watch calls for collections. It hides some level of complexity associated with stateless watch calls like soft resets or partial changes.

  • <resource_name>.pb.watcher.go

    High-level watcher components built around the Watch<CollectionName> method. It can support multiple queries and hides all complexity associated with stateless watch calls (resets, snapshot checks, partial snapshots, partial changes, etc.).

Files generated for a glue package for the whole service in a specific version:

  • <service_short_name>.pb.api_access.go

    Glues all access interfaces for each resource.

Watcher components require special attention and are best used for real-time database update observation. They are used heavily in our applications to provide system reactions in real-time. They can be used by web browsers to provide dynamic changes to a view, by client applications to react swiftly to some configuration updates, or by controllers to keep data in sync. We will cover this topic more in real-time updates topic.

Fixtures

Inside the fixtures directory, you will find some base files containing definitions of various resources that will have to be bootstrapped for your service. Usually, fixtures are created:

  • for the service itself.
  • per each project that enables a given service (dynamic creation).

Those files are not a “code” in any form, but some of those fixtures are still generated and it may be worth adding them here for completeness. We will come back to them in SPEKTRA Edge migration document.

Main files (runtime entry points)

SPEKTRA Edge-based service backend consists of:

  • Server runtime, which handles all incoming gRPC, webGRPC, and REST API calls.
  • DbController runtime, which executes all asynchronous database tasks (like Garbage Collecting, multi-region syncing, etc).
  • Controller runtime, that executes all asynchronous tasks related to business logic to keep the system working. It also handles various bootstrapping tasks, like for IAM integration.

For each runtime, it is necessary to write one main.go file.

Apart from the backend, it is very advisable to create a CLI tool that will allow developers to quickly play with the backend at least. It should use generated cli packages.

Clients for web browsers and agents are not covered by this document, but examples provide some insights into how to create a client agent application running on the edge.

All main file examples can be found here: https://github.com/cloudwan/inventory-manager-example/tree/master/cmd.

Service developers should create a cmd directory with relevant runtimes.

Server

In the main file for the server, we need:

  • Initialize the EnvRegistry component, responsible for interacting with the wider SPEKTRA Edge platform (includes the discovery of endpoints, real-time changes, etc.).

  • Initialize observability components

    SPEKTRA Edge provides Audit for recording many API calls, Monitoring for usage tracking (it can also be used to monitor error counters). It is also possible to initialize tracing.

  • Run the server in the selected version (as detected by envRegistry).

In function running a server in a specific version:

  • We are initializing the database access handle. Note that it needs to support collections for your resources, but also for mixins.

  • We need to initialize a multi-region policy store

    It will observe all resources that are multi-region policy-holders for your service. If you use policyholders from imported services, you may need to add a filter that will guarantee you are not trying to access resources unavailable for your service.

  • We need to initialize AuthInfoProvider, which is common for Authenticator and then Authorization.

  • We need to initialize the Authenticator module.

  • Finally, we initialize the gRPC server object. It does not contain any registered handlers on its own yet

    Only common interceptors like Authentication.

For the gRPC server instance, we need to create and register handlers, it is required to provide server handlers for your particular server, then for mandatory mixins:

  • schema mixin is mandatory, and it provides all methods related to database/schema consistency across multi-service, multi-region, and multi-version environments.
  • limits mixin is mandatory if you want to use Limits integration for your service.
  • Diagnostics mixin is optional for now, but this may change once EnvRegistry gets proper health checks based on gRPC. It should be included.

Mixins provide their API methods - they are separate “services” with their own API skeletons and protobuf files.

Refer to the example in the instructions on how to provide your main file for the server.

Note: As of now, webGRPC or REST API is handled not by server runtime, but by envoyproxy component. Examples include configuration example file (And Kubernetes deployment declaration).

Controller

In the main file for the controller, we need:

  • Initialize the EnvRegistry component, responsible for interacting with the wider SPEKTRA Edge platform (includes the discovery of endpoints, real-time changes, etc.).
  • Initialize observability components.
  • Run controller in selected version (as detected by envRegistry).

For a selected version of the controller, we need to:

  • Create business logic controller virtual nodes manager

    This step is necessary if you have business logic nodes, otherwise, you can skip. You can refer to the business logic controller document for more information on what it is and how to use it.

  • Limits-mixin controller virtual nodes manager is mandatory if you include the Limits feature in your service. You can skip this module if you don’t need Limits. Otherwise, it is needed to execute common Limits logic.

  • Fixtures controller nodes are necessary to:

    • Bootstrap resources related to the service itself (like IAM permissions).
    • Bootstrap resources related to the projects enabling service-like metric descriptors. Note that this means that the controller needs to dynamically create new resources and watch project resources appearing in the service (tenants).

The fixtures controller is described more in the document about SPEKTRA Edge integration. However, since some fixtures are mandatory, it is a practically mandatory component to include.

Refer to the example of how to provide your main file for the controller.

DbController

Db-Controller is a set of modules executing tasks related to the database:

  • MultiRegion syncing.

  • Search database syncing

    If the search is enabled and uses a separate database.

  • Schema consistency (like asynchronous cascade unsets/deletions when some resources are deleted).

In the main file for the controller, we need:

  • Initialize the EnvRegistry component, responsible for interacting with the wider SPEKTRA Edge platform (includes the discovery of endpoints, real-time changes, etc.).
  • Initialize observability components.
  • Configure database and search DB indices, as described in proto files.
  • Run db-syncing controller for all syncing-related tasks
  • Run db-constraint controller for all schema consistency tasks

All db-controller modules are provided by the Goten framework, so developers need to provide just the main file only.

Refer to the example in the instructions on how to provide your main file for db-controller.

CLI

If you have Cuttle installed, you can use core SPEKTRA Edge services with it. However, it is useful, especially when developing a service, to have a similar tool for own service too. Goten generates a CLI module in the cli directory, developers need only to provide their main file for CLI. Refer to the inventory-manager example.

In that example, we include an example service, and we add some mixins, schema-mixin and limits-mixin. Those objects for CLI can access mixin APIs exposed by your service. They can be skipped to reduce code size if you prefer. They contain calls that are relevant for service developers or maintainers. Mixins contain internal APIs and, if there are no bugs, even service developers don’t have to know their internals (and if there is a bug, they can submit an issue). Mixins try to operate on mixin APIs on their own and should do all the job.

Inclusion of Audit is recommended, the default cuttle provided by SPEKTRA Edge will not be able to decode Audit messages for custom services. However, CLI utility with all types of service registered will be.

Refer to the example in the instructions on how to provide your main file for CLI.

Note: Compiled CLI will only work if there is a cuttle locally installed and initialized. Apart from that, you need to add endpoint for your service separately to the environment, if your cuttle environment points to the core SPEKTRA Edge platform.

For example: Suppose that the domain for SPEKTRA Edge is beta.apis.edgelq.com:

cuttle config  environment get staging-env
Environment:  staging-env
Domain:  beta.apis.edgelq.com
Auth data:
    ...
Endpoint specific configs:
+--------------+----------+--------------+-----------------+------------------+---------------+
| SERVICE NAME | ENDPOINT | TLS DISABLED | TLS SKIP VERIFY | TLS SERVICE NAME | TLS CERT FILE |
+--------------+----------+--------------+-----------------+------------------+---------------+
+--------------+----------+--------------+-----------------+------------------+---------------+

Therefore, the connection to the IAM service will be iam.beta.apis.edgelq.com because this is the default domain. Considering 3rd party services use different domains, you will need to add different endpoint-specific settings like:

cuttle config environment set-endpoint \
  staging-env $SERVICE_SHORT_NAME --endpoint $SERVICE_ENDPOINT

Variable $SERVICE_SHORT_NAME should be snake_cased, it is derived from the short name of the service in api-skeleton. For The inventory manager example is inventory_manager (in api-skeleton, the short name is InventoryManager). See https://github.com/cloudwan/inventory-manager-example/blob/master/proto/api-skeleton-v1.yaml, field proto.service.name.

Variable $SERVICE_ENDPOINT must point to your service, like inventory-manager.examples.custom.domain.com:443. Note that you must include the port number, but not the method (like https://).