Schema Mixin

Understanding the schema mixin implementation.

Mixins are special kinds of services, that are supposed to be mixed/blended with proper services. Like any service, they have api-skeleton, protobuf files, resources, and server handlers. What they don’t get, is independent deployment. They don’t exist in the Meta Service registry. Instead, their resources and API groups are mixed with proper resources.

Moreover, for schema mixins, we are not validating references to other resources, they are excluded from this mechanism, and it’s up to the developer to keep them valid.

The Goten repository provides schema mixin, under runtime/schema-mixin. If you look at this mixin service, you will see that it has ResourceShadow resource. By mixing the schema mixin with let’s say Meta service, which formally has four resource types, four API groups, we have the following total Meta service with:

  • Resources: Region, Service, Deployment, Resource, ResourceShadow
  • API Groups: Region, Service, Deployment, Resource, ResourceShadow (CRUD plus custom actions).

If you inspect the Meta service database, you will have five collections (unless there are more mixins).

See api-skeleton: https://github.com/cloudwan/goten/blob/main/runtime/schema-mixin/proto/api-skeleton-v1.yaml.

By requiring that ALL services attach to themselves schema-mixin, we can guarantee, that all services can access each other via schema-mixin. This is one of the key ingredients of Goten’s protocol. Some common service is always needed, because, to enable circular communication between two services, which can’t possibly know each other schemas, they need some kind of common protocol.

Take a look at the resource_shadow.proto file. Just a note: You can ignore target_delete_behavior, they are more for informative purposes. But for mixins, Goten does not provide schema management. ResourceShadow is a very special kind of resource, and it exists for every other resource in a deployment (except other mixins). What I mean, let’s take a look at the list of resources that may exist in the Deployment of Meta service in region us-west2, like:

  • regions/us-west2 (Kind: meta.goten.com/Region)
  • services/meta.goten.com (Kind: meta.goten.com/Service)
  • services/meta.goten.com/resources/Region (Kind: meta.goten.com/Resource)
  • services/meta.goten.com/resources/Deployment (Kind: meta.goten.com/Resource)
  • services/meta.goten.com/resources/Service (Kind: meta.goten.com/Resource)
  • services/meta.goten.com/resources/Resource (Kind: meta.goten.com/Resource)
  • services/meta.goten.com/deployments/us-west2 (Kind: meta.goten.com/Deployment)

If those resources exist in the database for meta.goten.com in us-west2, then collection ResourceShadow will have the following resources:

  • resourceShadows/regions/us-west2
  • resourceShadows/services/meta.goten.com
  • resourceShadows/services/meta.goten.com/resources/Region
  • resourceShadows/services/meta.goten.com/resources/Deployment
  • resourceShadows/services/meta.goten.com/resources/Service
  • resourceShadows/services/meta.goten.com/resources/Resource
  • resourceShadows/services/meta.goten.com/deployments/us-west2

Basically it’s a one-to-one mapping, with the following exceptions:

  • if there are other mixin resources, they don’t get ResourceShadows.
  • synced read-only copies from other regions do not get ResourceShadows. For example, resource regions/us-west2 will exist in region us-west2, and resourceShadows/regions/us-west2 will also exist in us-west2. But, if regions/us-west2 is copied to other regions, like eastus2, then resourceShadows/regions/us-west2 WILL NOT exist in eastus2.

This makes Resource shadows rather “closed” within their Deployment.

ResourceShadow instances are created/updated along a resource they represent, during each transaction. It ensures that they are always in sync with a resource. They contain all references to other resources and contain all back reference source deployments. The reason we have back reference deployments, not an exact list, is that the full list would have been massive, imagine a Project instance and 10000 Devices pointing to it. Instead, if let’s say those devices are spread across four regions, ResourceShadow for Project will have 4 back reference sources, more manageable.

Now, with ResourceShadows, we can provide some abstraction needed to facilitate communication between services. However, note that we don’t use standard CRUD at all (for shadows). They were in the past, but the problem with CRUD is that they don’t contain the “API Version” field.

For example, we have the secrets.edgelq.com service in versions v1alpha2 and v1. In the older version, we have a Secret resource with the name pattern projects/{project}/secrets/{secret}. Now, with v1 upgrade, name pattern changed to projects/{project}/regions/{region}/secrets/{secret}. Note that this means, that the ResourceShadow name changes too!

Suppose there are services S1 and S2. S1 imports secrets in v1alpha2, and S2 imports secrets in v1. Suppose both S1 and S2 want to create resources concerning some Secret instance. In this case, they would try to use schema-mixin API, and they would give conflicting resource shadow names, but this conflict arises from a different version, not because of a bug. S1 would try to establish a reference to shadow for projects/{project}/secrets/{secret}, and S2 would use the version with region.

This problem repeats for the whole CRUD for ResourceShadow, so we don’t use it. Instead, we developed a bunch of custom actions you can see in the api-skeleton of schema-mixin like EstablishReferences, ConfirmBlockades, etc. All those requests contain a version field, and the API Server can use versioning transformers to convert between names between versions.

Now, coming back to custom actions for ResourceShadows, see API-skeleton along, recommended to see protobuf with request objects!

We had a flow on how references are established, when API Servers handle writing requsts, this is where schema mixin API is in use.

EstablishReferences is used by Store modules in API Servers, when they save resources with cross-region/service references. This is called the DURING transaction of Store in API Server. It ensures that referenced resources will not be deleted for the next few minutes. It creates tentative blockades in ResourceShadow instances on the other side. You may check the implementation in the goten repo, file runtime/schema-mixin/server/v1/resource_shadow/resource_shadow_service.go. When the transaction concludes, then Deployment asynchronously will send ConfirmBlockades to remove the tentative blockade from referenced ResourceShadow in the target Service. It will leave with a back reference source though!

For deletion requests, the API Server must call CheckIfResourceIsBlocked before proceeding with resource deletion. It must also block deletion if there are tentative blockades in ResourceShadow.

We also described Meta owner flows with three cases. When Meta Ownee Deployment tries to confirm the meta owner, it must use the ConfirmMetaOwner call to a Meta Owner Deployment instance. If all is fine, then we will get a successful response. If there is a version mismatch, Meta Ownee Deployment will send UpgradeMetaOwnerVersion request to itself (its API Server), so the meta owner reference is finally in the desired state. If ConfirmMetaOwner discovers the Meta Owner does not confirm ownership, then Meta Ownee Deployment should use the RemoveMetaOwnerReference call.

When it is Meta Owner Deployment that needs to initiate actions (cases two and three), it needs to use ListMetaOwnees to get meta ownees. When relevant, it will need to call UpgradeMetaOwnerVersion or RemoveMetaOwnerReference, depending on the context of why we are iterating meta ownees.

When we described asynchronous deletions handling, the most important schema-mixin API action is WatchImportedServiceDeletions. This is a real-time watch subscription with versioning support. For example, if we have Services S1 and S2 importing secrets.edgelq.com in versions v1alpha2 and v1, then if some Secret is deleted (with name pattern containing region in v1 only), separate WatchImportedServiceDeletionsResponse is sent to S1 and S2 Deployments, containing shadow ID of secret in version Service desires.

When it comes to the deletion flow, we also use CheckIfHasMetaOwnee, and CheckIfResourceHasDeletionSubscriber. These methods are used when waiting for back-references to be deleted generally.

Since the schema-mixin Server is mixed with proper service, it means we can also access original resources from the Store interface! In total, Schema-mixin is a powerful utility for Goten as protocol cases.

We still need CRUD in ResourceShadows, because:

  • Update, Delete, and Watch functions are used within Deployment itself (where we know all runtimes use the same version).
  • debugging purposes. Developers can use read requests when some bug needs investigation.