This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Preparing your Environment

How to prepare your development environment.

1 - Prerequisites

What you need to know before developing SPEKTRA Edge services.

gRPC and Protocol buffers

All core services on the SPEKTRA Edge platform utilize gRPC/Protocol buffer technologies as a way of communication with each other. You can read about those here: https://grpc.io/docs/what-is-grpc.

We are using exclusively the proto3 version.

However, to put things more simply: gRPC is a high-level protocol where client-server can communicate in the following ways:

  • Unary request-response, for example, GET Object, LIST Objects etc.
  • Server streaming: The client initiates the connection, sends a first message and then the server keeps sending a series of messages one after another. Use case: WATCH changes on Object “X”.
  • Client streaming: The client initiates the connection and then keeps sending messages. Example use case: Logging
  • Bidi-streaming: The client initializes the connection and then keeps exchanging messages with the server until the connection is closed.

In other words, gRPC defines methods, which can be unary or streaming. It is built on top of HTTP2, and requests, including streams, have HTTP headers. Messages (payloads) themselves use binary format, so you can’t just use JSON with Curl/Postman utility. The format of messages themselves is defined by protocol buffers. However, those messages always can be dumped into some human-readable format - Cuttle is an example.

Message definitions (or structures) naturally are defined in a human-friendly way. First, you need to define structure in proto files (proto is a file name extension). Basic primitive field types are self-describing - string, int64, uint32, bool, etc. You can add “repeated” before type to make an array: repeated int32 integers = 1; as an example. To declare a map, use map<key, value> to define key-value collection. You can declare message <name> { <body> } inside messages too, to define a child-structure. There is also an “enum” and “oneof”. You can find plenty of examples on the internet, in our services, and in our example inventory-manager app.

In proto files, on top of messages, you also define a list of APIs and methods. Again, see an inventory-manager example, some of our services, or check the internet. They should be simple to understand and with practice, you will get everything. The only thing that may look strange at the beginning are those “numbers” assigned all fields in all messages. But they only inform what is the binary ID identifier that will be used for serialization. It’s not something to be particularly worried about, however - once code is released into production, numbers need to stay as they are. Changes to them would render API incompatible, as messages passed between clients/servers would break.

When developing in Goten, you will need to model requests/responses and resources using protocol buffers. Once you have proto files, you can use a proto compiler that creates relevant source code files (C++, javascript, Golang, Python, etc.). It will create code for messages (structs, getters, setters, standard util functions, etc.) and clients (the client itself is an interface, which contains the same list of methods as defined in proto files for your service).

Some pseudo-golang example:

connectionHandle := EstablishConnection(
   “service.address.com”,
   credentialsObject,
)

// make a client
client := NewYourAppServiceClient(connectionHandle)

// unary request example
response := client.UnaryMethodName(requestObject)

// streaming example
stream := client.StreamingMethodName()
streamSend(clientMsg)
serverMsg := stream.Recv()

Protoc for Golang (or any other language) will create a constructor for the client (NewYourAppServiceClient), with all the methods.

Goten enables also REST API - incoming request objects are converted into protobuf, then outgoing messages back to JSON. This has however some performance penalty.

Goten

Goten is a service development framework for building resource-oriented APIs on top of grpc/protobuf. Service built with Goten consists of:

  • Resources
  • APIs (multiple API groups)
  • Methods (each method belongs to some API group).

All core SPEKTRA Edge services were built using Goten, and this is also required for any third party application.

Developers first need to define resources - with relationships to each other. Goten implicitly creates an API group per each one. For example, in service iam.edgelq.com we have a resource called “Permission”. Therefore, service iam.edgelq.com has an API called “PermissionService”, which contains a set of methods operating on Permission resources. By default, those <Resource>Service APIs contain basic CRUD methods (example for PermissionService):

  • GetPermission
  • BatchGetPermissions
  • ListPermissions
  • WatchPermission
  • WatchPermissions
  • CreatePermission
  • UpdatePermission
  • DeletePermission

Developers can attach more methods (custom ones) to those implicitly created API groups. On top of that, they can create custom additional API groups with custom methods - each may operate on different resources.

The benefit of splitting a single service into many API groups is the packaging - we can have a single package per resource and per API groups (set of methods). Client modules can pick which parts of the service they are interested in and compiled binaries should be smaller. Smaller packages make also modules smaller. Still, within code, we can access all resources defined within the service using the same connection/database handler - and consequently, we can access multiple resources within the service in a single transaction.

Goten is more like a toolbox rather than a single tool, it first contains a set of tools generating code for a service based on YAML/protobuf files. It contains also the definition of reusable protobuf types and a set of runtime libraries linked during compilation.

Service built with Goten will support communication with the following APIs:

  • gRPC

    Recommended as the most native protocol and having the best performance.

  • web-gRPC

    It is gRPC for web browsers (as they can’t use native GRPC).

  • HTTP REST API

    Used gRPC transcoding, where requests/responses are converted from/to JSON before passing for processing.

Request processing by SPEKTRA Edge backends

SPEKTRA Edge-based service can accept gRPC (regular request-response or streams), webGRPC (gRPC for web browsers) with websockets-grpc (for streaming), HTTP REST API (request-response only, no streams).

When a request is received by the backend, it first checks the protocol. Any non-gRPC messages are converted to gRPC before the backend handler is called. The handler then identifies the method and sets up observability components for auditing and usage metrics such as latency and request/response sizes. If configured, it also initiates tracing spanning. The request/stream passes through common interceptors that are universal for each method. These interceptors are often simple, adding tags, catching exceptions, and configuring the logger. One notable component is the Authenticator, which is a module provided by the SPEKTRA Edge framework and is built into every server runtime during compilation and linking. The Authenticator retrieves the authorization header and attempts to identify the holder, referred to as the Principal. There are two primary types of principals: ServiceAccount (optimized for bots) and human (User). If the holder cannot be classified, the Principal is classified as Anonymous. The authenticator then checks with the cache. If the principal is stored there, it validates the claims and proceeds if there is a match. Otherwise, it sends a request to the iam.edgelq.com service to inquire about the identity of the principal (GetPrincipal). If the IAM service itself is making requests, it will ask for its database. During the GetPrincipal execution, IAM identifies the Principal and validates that the service requesting the principal has the right to access it. The basic rule is that if a user or ServiceAccount has any RoleBindings for a given service, then the service is assumed to be allowed to access that principal. In this case, IAM returns the Principal data with the corresponding User or ServiceAccount. The returned data is cached for faster execution in subsequent requests.

Note: In the multi-regional environment, GetPrincipal has additional tricks though - Authenticator needs to identify from authorization a token what regions of iam.edgelq.com will know the given principal, but this is already provided in the Authenticator code.

After authentication finishes, the request/stream reaches a set of code-generated middlewares (layers) specifically for this method. The first middleware MAY be transformer middleware - if the method called by the user uses an older API version, then the request/stream is upgraded to the higher version. Then it proceeds further. However, if the request/stream already uses the newest API, it goes straight to the next part. This next middleware, and the first one if no versioning was needed, is multi-region routing middleware. It inspects the stream/request and decides if the request can be executed locally, or should be proxied to the same service in a different region. The next middleware is the Authorization type. It extracts the Principal object from the current processing context and checks if the given Principal is allowed to execute this request. Authorization middleware uses an Authorizer component that is linked in during a server runtime build process. The authorizer grabs relevant RoleBindings for the user and validates against the method. When possible - RoleBindings are extracted from the local cache. Otherwise, it will need to send another request to iam.edgelq.com, returned RoleBindings will be cached. The authorizer will decide eventually if the request/stream can be processed further or not. If all is good, the next middleware is database transaction middleware - it grabs either the SNAPSHOT-transaction or NO-transaction handle. In the case of snapshot one, it needs additional IO operation on the database. Next middleware is “outer” middleware - it makes basic request/stream validation, for update operations it will verify previous resources, and execute CAS (Compare And Swap) if specified. For creations, it will verify resources did not exist, for deletions it will verify they existed.

Finally, outer middleware passes the request/stream to the proper processing part. It can be two things:

  • Code-generated server core

    it handles all CRUD operations or returns Unimplemented errors for custom actions.

  • Custom middleware, written by engineer/developer.

    This custom middleware must be written like a middleware. It must have a handle to the code-generated server core. Custom middleware should execute finally all custom methods (requests or streams) and not pass to the server core, because it would return an Unimplemented error.

For CRUD operations, it MAY implement some additional processing that is executed BEFORE OR AFTER server core. But eventually, for CRUD operations, custom middleware must pass handling to code-generated server code. During this proper processing server can get/save resources from the database, or connect with other services for some more complex cases. Note that the database handle, that is provided by the Goten framework, MAY not only connect with the actual database BUT also connect with other services (in the same or different regions) if we are executing the save/deletion of a resource with references to other services and/or regions. It works to ensure that services database schemas remain in sync, even if we have cross-service or cross-region references. For cross-service requests, other services will also use Authorizer to confirm requesting user/service is allowed to reference resources they own!

Once the request/stream is processed by the proper handler (optional custom middleware + server core), the request/stream goes through middleware in the reverse order (unwrapping). Here most of the middlewares don’t do anything. For example, outer middleware will just pass the response/exit stream error further back. More important things happen with transaction middleware when exiting - if a snapshot transaction was used, then all resource updates/deletions are submitted to the database at this moment. When a request comes back to routing middleware, usually nothing should happen, the middleware should propagate further back the response or exit stream code. However, if it was decided that the request should have been executed by MORE THAN ONE REGION, then middleware will wait until the other servers in different regions return the response. Once it happens, the final response is merged from multiple regional ones. If there was a transformer middleware for versioning before routing middleware, it would convert the response to an older API, if needed. Interceptors will then be called in reverse, but as of now, this is a simple pass-through. Eventually, the response or stream exit code will be returned by the server. At this point, observability components will also be concluded:

  • Audit (if used for this call) will be notified about the activity.
  • Monitoring service will get usage metrics.
  • Tracing may optionally get spans.

The protocol will be adjusted back to REST API/webGRPC if needed after it exists on the server.

Goten/SPEKTRA Edge provides all of those interceptors/middlewares/procedures based on YAML/protobuf prototypes written by the user. The required code to be written is custom middleware, at least when it comes to backend services.

2 - Setting up your Development Environment

How to setup your development environment.

Setting up development environment

If you do not have Go SDK, you should download and configure it. To check the version required by SPEKTRA Edge, see this file - top shows the required minimum version. As of the moment of this writing, it is 1.21, but it may change. Ensure Go SDK is installed.

You will need to access the following repositories, ensure you have access to:

With Go SDK installed, check the $GOPATH variable: echo $GOPATH. Ensure the following paths exist:

Export variables, as they are referenced by various scripts:

export GOTENPATH=$GOPATH/src/github.com/cloudwan/goten
export EDGELQROOT=$GOPATH/src/github.com/cloudwan/edgelq

You may export them permanently.

Goten/SPEKTRA Edge comes with its own dependencies and plugins, you should install them:

$GOTENPATH/scripts/install-proto-deps.sh
$GOTENPATH/scripts/install-plugins.sh
$EDGELQROOT/scripts/install-edgelq-plugins.sh

You need some repository for your code, like github.com/some-namespace/some-repo. If you do that, ensure the location of this repository is in $GOPATH/src/some-namespace/some-repo, OR it is sym-linked there.

Reserving service on the SPEKTRA Edge platform

Before you begin here, ensure that you have access to some Organization where you have permission to create service projects - typically it means some administrator. You need cuttle configured - at least you should go through the user guide and reach the IAM chapter.

All service resources on the SPEKTRA Edge platform belong to some IAM project. An IAM Project that is capable of “hosting” services is called a “Service Project”. You may create many services under a single service project, but you need to make the first one.

# If you dont plan to create devices/applications resources under your service project, you should skip them
# using core-edgelq-service-opt-outs. It DOES NOT MEAN that final tenant projects using your service will not use
# those services, or that your service wont be able to use devices/applications. It merely says that your project
# will not use them.
cuttle iam setup-service-project project --name 'projects/$SERVICE_PROJECT_ID' --title 'Service Project' \
  --parent-organization 'organizations/$MY_ORGANIZATION_ID' --multi-region-policy '{"enabledRegions":["$REGION_ID"],"defaultControlRegion":"$REGION_ID"}' \
  --core-edgelq-service-opt-outs 'services/devices.edgelq.com' --core-edgelq-service-opt-outs 'services/applications.edgelq.com'

To clarify: A service project is just a container for services, and is used for some simple cases like usage metrics storage or service accounts. Therefore, eventually, you will have two resources:

  • IAM Service project: projects/$SERVICE_PROJECT_ID - it will contain credentials of ServiceAccounts with access to your service or usage metrics.
  • Meta Service: services/$YOUR_SERVICE

The service project is a type of IAM project, but your service tenants will have their projects. Service on its own is a separate entity from a project it belongs to. Therefore, unless your project will need to have some devices/applications resources directly, it is recommended to opt out from those services using --core-edgelq-service-opt-outs arguments. Your service will still be able to import/use devices/applications, and tenants using your service too.

You need to decide on $SERVICE_PROJECT_ID and $REGION_ID, where your service will run. As of this moment, SPEKTRA Edge platform is single-regional, so you will only have one region at your disposal. But it should change in the future. It will be possible to expand your service project (and therefore services) to more regions later on.

You will need to replace the $MY_ORGANIZATION_ID variable with one you have access to.

Once you have service project created, you will need to reserve a service:

cuttle iam reserve-service-name project --name 'projects/$SERVICE_PROJECT_ID' --service 'services/$YOUR_SERVICE_NAME' \
  --admin-account 'projects/$SERVICE_PROJECT_ID/regions/$REGION_ID/serviceAccounts/svc-admin' \
  --admin-key '{"name":"projects/$SERVICE_PROJECT_ID/regions/$REGION_ID/serviceAccounts/svc-admin/serviceAccountKeys/key", "algorithm":"RSA_2048"}' \
  -o json

Now you will need to determine the value of $YOUR_SERVICE_NAME - our 3rd party services are watchdog.edgelq.com, and ztna.edgelq.com. Those look like domains, but the actual public domain you can decide/reserve later on.

Argument --admin-account determines the ServiceAccount resource that will be allowed to create a given Service, and it will be responsible for its future management. If it does not exist, it will be created. You should be able to see it with:

cuttle iam get service-account 'projects/$SERVICE_PROJECT_ID/regions/$REGION_ID/serviceAccounts/svc-admin' -o json

The argument --admin-key is more important as it will create a ServiceAccountKey resource under the specified admin account. However, if the key already exists, you will receive an AlreadyExists error. This may occur if you were already making reservations for different services. If both ServiceAccount and ServiceAccountKey already exist in a given service project, you should skip using the --admin-key argument altogether and simply use previously obtained credentials. The same ServiceAccount can be used for many services. However, if you wish, you can decide to create another --admin-account by providing a different name than what was used before.

If you provide –admin-key argument, you can do this in two ways:

  --admin-key '{"name":"projects/$SERVICE_PROJECT_ID/regions/$REGION_ID/serviceAccounts/svc-admin/serviceAccountKeys/key", "algorithm":"RSA_2048"}'

OR

  --admin-key '{"name":"projects/$SERVICE_PROJECT_ID/regions/$REGION_ID/serviceAccounts/svc-admin/serviceAccountKeys/key", "publicKeyData":"$DATA"}'

In the case of the first example, the response will contain private key data contents that you will need. In the second case, you can create a private/public pair yourself and supply public data ($DATA param). This version should be used if you prefer to keep a private key never known by SPEKTRA Edge services.

You should pay attention to the response returned from cuttle iam reserve-service-name project. More specifically, field nttAdminCredentials:

{
  "nttAdminCredentials": {
    "type": "<TYPE>",
    "client_email": "<CLIENT_EMAIL>",
    "private_key_id": "<KEY_ID>",
    "private_key": "<PRIVATE_KEY>"
  }
}

You should get this value and save in own ntt-credentials.json file (you can name file however you like though):

{
  "type": "<TYPE>",
  "client_email": "<CLIENT_EMAIL>",
  "private_key_id": "<KEY_ID>",
  "private_key": "<PRIVATE_KEY>"
}

Note that, if you created –admin-key with the public key (not algorithm), then <PRIVATE_KEY> will not be present in response. Instead, when saving the ntt-credentials.json file, you should populate this value yourself with the private key.

Credentials need to be kept and not lost. In case it happens, you can use the DeleteServiceAccountKey method. Note that ServiceAccount (admin) for services is just a regular ServiceAccount in iam.edgelq.com service - and you have full CRUD of its ServiceAccountKey instances.

When reserving a service for the first time, you may also decide what Role will be assigned to a ServiceAccount for your service. By default, ServiceAccount will be an admin in the Service namespace, but it will have a limited role assigned in the projects/$SERVICE_PROJECT_ID scope. By default, it is services/iam.edgelq.com/roles/default-admin-project-role. However, for more advanced users, you can pass a custom role, for example, full ownership:

cuttle iam reserve-service-name project --admin-account-project-role 'services/iam.edgelq.com/roles/scope-admin' <OTHER ARGS>

You may manage service projects & services on SPEKTRA Edge dashboard as well. To have same API via Cuttle, see:

cuttle iam list-my-service-projects projects --help  # To see service projects
cuttle iam list-service-reservations project --help  # To see existing service reservations under specific service project.
cuttle iam delete-service-reservation project --help # To delete service reservation
cuttle iam list-project-services project --help      # To see already created services under specific service project

# This command is more advanced, should be used when expanding service project
# to new regions. Will be more covered in next docs in detail, for now its just
# FYI.
cuttle iam add-regional-admin-account-for-services service-account --help

With service reserved, you should continue with the normal development.