EnvRegistry as Service Discovery

Understanding the role of EnvRegistry module in Meta service.

Meta service provides API allowing inspection global environment, but we also need a side library, called EnvRegistry:

  • It must allow a Deployment to register itself in a Meta service, so others can see it.
  • It must allow the discovery of other services with their deployments and resources.
  • It must provide a way to obtain real-time updates of what is happening in the environment.

Those three items above are the responsibilities of EnvRegistry module.

In the goten repo, this module is defined in the runtime/env_registry/env_registry.go file.

As of now, it can only be used by server, controller, and db-controller runtimes. It may be beneficial for client runtimes someday probably, but we will opt out from “registration” responsibility because the client is not the part of the backend, it cannot self-register in Meta service.

One of the design decisions regarding EnvRegistry is that it must block till initialization is completed, meaning:

  • User of EnvRegistry instance must complete self-registration in Meta Service.
  • EnvRegistry must obtain the current state of services and deployments.

Note that no backend service works in isolation, as part of the Goten design, it is essential that:

  • any backend runtime knows its surroundings before executing its tasks.
  • all backend runtimes must be able to see other services and deployments, which are relevant for them.
  • all backend runtimes must initialize and run the EnvRegistry component and it must be one of the first things to do in the main.go file.

This means, that the backend service, if it cannot successfully pass initialization, will be blocked from any useful work. If you check all run functions in EnvRegistry, you should see they lead to the runInBackground function. It runs several goroutines, but then it waits for a signal showing all is fine. After this, EnvRegistry can be safely used to find other services, and deployments, and make networking connections.

This also guarantees that Meta service contains relevant records for services, in other words, EnvRegistry registration initializes regions, services, deployments, and resources. Note, however:

  • The region resources can be created/updated by meta.goten.com service only. Since meta is the first service, it is responsible for this resource to be initialized.
  • The service resource is created by the first deployment of a given service. So, if we release custom.edgelq.com for the first time, in the first region, it will send a CreateService request. The next deployment of the same service, in the next region, will just send UpdateService. This update must have a new MultiRegionPolicy, where field-enabled regions contain a new region ID.
  • Each deployment is responsible for its deployment resource in Meta.
  • All deployments for a given service are responsible for Resource instances. If a new service is deployed with the server, controller, and db-controller pods, then they may initially be sending clashing create requests. We are fine with those minor races there, since transactions in Meta service, coupled with CAS requests made by EnvRegistry, ensure eventual consistency.

Visit the runInit function, which is one of the goroutines of EnvRegistry executed by runInBackground. It contains procedures for registration of Meta resources finishes after a successful run.

From this process, another emerging design property of EnvRegistry is that it is aware of its context, it knows what Service and Deployment it is associated with. Therefore, it has getters for self Deployment and Service.

Let’s stay for a while in this run process, as it shows other goroutines that are run forever:

  • One goroutine keeps running runDeploymentsWatch
  • Second goroutine keeps running runServicesWatch
  • The final goroutine is the main one, runMainSync

We don’t need real-time watch updates of regions and resources, we need services and their regional deployments only. Normally watch requires a separate goroutine, and it is the same case here. To synchronize actual event processing across multiple real-time updates, we need a “main synchronization loop”, which unites all Go channels.

In the main sync goroutine, we:

  • Process changes detected by runServicesWatch.
  • Process changes detected by runDeploymentsWatch.
  • Catch initialization signal from the runInit function, which guarantees information about our service is stored in Meta.
  • Attachment of new real-time subscribers. When they attach, they must get a snapshot of past events.
  • Detachment of real-time subscribers.

As of additional note: since EnvRegistry is self-aware, it gets only Services and Deployments that are relevant. Those are:

  • Services and Deployments of its Service (obviously)
  • Services and Deployments that are used/imported by the current Service
  • Services and Deployments that are using the current Service

The last two parts are important, it means that EnvRegistry for top service (like meta.goten.com) is aware of all Services and Deployments. Higher levels will see all those below or above them, but they won’t be able to see “neighbors”. The higher the tree, there will be fewer services above, and more below, but the proportion of neighbors will be higher and higher.

It should not be a problem, though, unless we reach the scale of thousands of Services, core SPEKTRA Edge services will however be more pressured than all upstream ones for various reasons.

In the context of SPEKTRA Edge, we made additional implementation decisions, when it comes to SPEKTRA Edge platform deployments:

  • Each service, except meta.goten.com itself, must connect to the regional meta service in its EnvRegistry.

    For example, iam.edgelq.com in us-west2, must connect to Meta service in us-west2. Service custom.edgelq.com in eastus2 must connect to Meta service in eastus2.

  • Server instance of meta.goten.com must use local-mode EnvRegistry. The reason is, that it can’t connect to itself via API, especially since it must succeed in EnvRegistry initialization before running its API server.

  • DbController instance of meta.goten.com is special, and shows the asymmetric nature of SPEKTRA Edge core services regarding regions. As a whole, core SPEKTRA Edge services point to the same primary region, any other is secondary. Therefore, DbController instance of meta.goten.com must:

    • In the primary region, connect to the API server of meta.goten.com in the primary region (intra-region)
    • In the secondary region, connect to the API server of meta.goten.com in the primary region (the secondary region connects to the primary).

Therefore, when we add a new region, the meta-db-controller in the secondary region registers itself in the primary region meta-service. This way primary region gets the awareness of the next region’s creation. The choice of meta-db-controller for this responsibility has more for it, Meta-db-controller will be responsible for syncing the secondary region meta database from the primary one. This will be discussed in the following section of this guide. For now, we just mentioned conventions where EnvRegistry must source information from.