Goten organization

Understanding the Goten directory structure and libraries.

In the SPEKTRA Edge repository, we have directories for each service:

edgelq/
 applications/
 audit/
 devices/
 iam/
 limits/
 logging/
 meta/
 monitoring/
 proxies/
 secrets/
 ztp/

All names of these services end with the .edgelq.com suffix, except meta. The full name of the meta service is meta.goten.com. The reason is that the core of this service is not in the SPEKTRA Edge repository, it is in the Goten repository:

goten/
 meta-service/

This is where meta’s api-skeleton is, Protocol buffers files, almost all the code-generated modules, and server implementation. The reason why we talk about meta service first is quite important, because it also teaches the difference between SPEKTRA Edge and Goten.

Goten is called a framework for SPEKTRA Edge, but this framework has two main tool sets:

  1. Compiler

    It takes the schema of your service, and generates all the boilerplate code.

  2. Runtime

    Runtime libraries, which are referenced by generated code, are used heavily throughout all services based on Goten.

Goten provides its schema language on top of Protocol Buffers: It introduces the concept of Service packages (with versions), API groups, actions, resources. Unlike raw Protocol Buffers, we have a full-blown schema with references that can point across regions and services, and those services can also vary in versions. Resources can reference each other, services can import each other.

Goten balances between code-generation and runtime libraries operating on “resources” or “methods”. It is usually the tradeoff between performance, type safety, code size, maintainability, and the readability.

If you look at the meta-service, you will see that it has four resource types:

  1. Region
  2. Service
  3. Resource
  4. Deployment

This is pretty much exactly what Goten provides. To be a robust framework, to provide on its promises, Multi-region, multi-service, and multi-version, Goten needs a concept of a service that contains information about regions, services, etc.

SPEKTRA Edge provides various services on a higher level, but Goten provides the baseline for them and allows relationships between them. The only reason why the “meta” directory exists also in SPEKTRA Edge, is because the meta service also needs extra SPEKTRA Edge integration like authorization layer. In the SPEKTRA Edge repo, we have additional components added to meta, and finally, we have meta main.go files. If you look at the files meta service has in the SPEKTRA Edge repo (for the v1 version, not v1alpha2), you will understand that the edgelq version wraps up what goten provides. There is also a full “v1alpha2” service, which predates the move of meta into Goten. At that time SPEKTRA Edge overrode much of the functionality Goten provides.

Goten Directory Structure

As a framework, goten provides:

  1. Modules related to service schema and prototyping (API skeleton and proto files).
  2. Compilers that generate code based on schema
  3. Runtime libraries linked during the compilation

For schema & prototyping, we have directories:

  • schemas

    This directory contains generated JSON schema for api-skeleton files. It is generated based on file annotations/bootstrap.proto.

  • annotations

    Protobuf is already a kind of language for building APIs, but Goten is said to provide a higher level one. This directory contains various proto options (extra decorations), that enhance standard protobuf language. There is one exceptional file though: bootstrap.proto, which DOES NOT define any options, instead it describes the api-skeleton schema in protobuf. The file in the schemas directory is just a compilation of this file. The annotations directory contains generated Golang code describing those proto options, which you do not normally need to read.

  • types

    Contains set of reusable protobuf messages that are used in services using Goten, for example, “Meta” object (file types/meta.proto) is used in almost every resource type. The difference between annotations and types is that, while annotations describe options we can attach to files/proto messages/fields/enums etc., types contain just reusable objects/enums. Apart from that, each proto file contains compiled Golang objects in the relevant directory.

  • contrib/protobuf/google

    This directory avoids depending on the full protobuf distribution from Google by vendoring only the parts SPEKTRA Edge uses. Subdirectory api maps to annotations, and type to types. One file does not follow that mapping: distribution.proto belongs more with the type directories than with api. It does not appear to be used, and may be removable.

    Contributors do still download some protocol buffers separately (see the scripts directory). That downloaded library is more lightweight and does not contain the types vendored here.

All the above directories can be considered a form of Goten-protobuf language that you should know from the developer guide.

For compilers (code-generators), we have directories:

  • compiler

    Each subdirectory (well, almost) contains a specific compiler that generates some set of files that Goten as a whole generates. For example compiler/server generates server middleware.

  • cmd

    Goten does not come with any runtime on its own. This directory provides main.go files for all compilers (code-generators) Goten has.

Compilers generate code you should already know from the developer guide as well.

Runtime libraries have just a single directory:

  • runtime

    Contains various modules for clients, servers, controllers… Each will be talked about separately in various topic-oriented documents.

  • Compiled types

    types/meta/, and types/multi_region_policy may be considered part of the runtime, they map to types objects. You may say, that while resource proto schema imports goten/types/meta.proto, generated code will refer to Go package goten/types/meta/.

In the developer guide, we had brief mentions of some base runtime types, but we were treating them as black boxes, while in this document set, we will dive in.

Other directories in Goten:

  • example

    Contains some typical services developed on Goten, but without SPEKTRA Edge. The current purpose of them is only to run some integration tests though.

  • prototests

    Contains just some basic tests over base extended types by Goten, but does not delve as deep as tests in the example directory.

  • meta-service

    It contains full service of meta without SPEKTRA Edge components and main files. It is supposed to be wrapped by Goten users, SPEKTRA Edge in our case.

  • scripts

    Contains one-of scripts for installing development tools, reusable scripts for other scripts, or regeneration script that regenerates files from the current goten directory (regenerate.sh).

  • src

    This directory name is the most confusing here. It does not contain anything for the framework. It contains generated Java code of annotations and types directories in Goten. It is generated for the local pom.xml file. This Java module is just an import dependency for Goten, so Java code can use protobuf types defined by Goten. We have some Java code in the SPEKTRA Edge repository, so for this purpose, in Goten, we have a small Java package.

  • tools

    Just some dummy imports to ensure they are present in go.mod/go.sum files in goten.

  • webui

    A generic UI for Goten services. This is no longer maintained, as the front-end teams now build specialized UIs rather than a generic one.

The remaining files worth noting:

  • pom.xml

    This is for building a Java package containing Goten protobuf types.

  • sdk-config.yaml

    This is used to generate the public goten-sdk repository, since goten itself is private. It automates copying the public files from goten to goten-sdk.

  • tools.go

    Ensures the required dependencies are present in go.mod.

SPEKTRA Edge Directory Structure

SPEKTRA Edge is a home repository for all core SPEKTRA Edge services, and an adaptation of meta.goten.com, meaning that its sub directories should be familiar, and you should navigate their code well enough since they are “typical” Goten-built services.

We have a common directory though, with some example elements (more important):

  • api and rpc

    Those directories contain extra protobuf reusable types. You will most likely interact with api.ServiceAccount (not to confuse with the iam.edgelq.com/ServiceAccount resource)!

  • cli_configv1, cli_configv2

    The second directory is used by the cuttle CLI utility, and will be needed for all cuttles for 3rd parties.

  • clientenv

    Those contains obsolete config for client env, but its grpc dialers and authclients (for user authentication) are still in use. Needs some cleanup.

  • consts

    It has a set of various common constants in SPEKTRA Edge.

  • doc

    It wraps protoc-gen-goten-doc with additional functionality, to display needed permissions for actions.

  • fixtrues_controller

    It is the full fixtures controller module.

  • serverenv

    It contains a common set for backend runtimes provided by SPEKTRA Edge (typically server, but some elements are used by controllers too).

  • widecolumn

    It contains a storage alternative to the Goten store, for some advanced cases, we will have a different document design for this.

Other directories:

  • healthcheck

    It contains a simple image that polls health checks of core SPEKTRA Edge services.

  • mixins

    It contains a set of mixins, they will be discussed via separate topics.

  • protoc-gen-npm-apis

    It is a Typescript compiler for the frontend team, maintained by the backend. You should read more about compilers here

  • npm

    It is where code generated by protoc-gen-npm-apis goes.

  • scripts

    Set of common scripts, developers must learn to use primarily regenerate-all-sh whenever they change any api-skeleton or proto file.

  • src

    Contains Java-generated code for the Monitoring Pipeline, which is being phased out and will be documented separately.

In this section