MetaMixin Service APIv1alpha2

Understanding the MetaMixin service APIv1alpha2, as known as ntt.meta.mixin.v1alpha2.

Here is the list of resources supported in MetaMixin service APIv1alpha2:

ResourceShadow Resource

ResourceShadow is a hidden handle hidden behind each actual resource in their service-region location (shadows are not present in synced regions). It is used to enforce database constraints across all services and regions. Resource shadow is used for all original resources defined in their services - it does not support mixin service resources.

This section covers the methods and messages to interact with ResourceShadow resource.

ResourceShadow Methods

Here is the list of ResourceShadow resource methods:

GetResourceShadow Method

GetResourceShadow

rpc GetResourceShadow(GetResourceShadowRequest) returns (ResourceShadow)

with the following messages:

The equivalent REST API is:

GET /meta-mixin/v1alpha2/{name=resourceShadows/*}

BatchGetResourceShadows Method

BatchGetResourceShadows

rpc BatchGetResourceShadows(BatchGetResourceShadowsRequest) returns (BatchGetResourceShadowsResponse)

with the following messages:

The equivalent REST API is:

GET /meta-mixin/v1alpha2/resourceShadows:batchGet

ListResourceShadows Method

ListResourceShadows

rpc ListResourceShadows(ListResourceShadowsRequest) returns (ListResourceShadowsResponse)

with the following messages:

The equivalent REST API is:

GET /meta-mixin/v1alpha2/resourceShadows

WatchResourceShadow Method

WatchResourceShadow

rpc WatchResourceShadow(WatchResourceShadowRequest) returns (WatchResourceShadowResponse)

with the following messages:

The equivalent REST API is:

POST /meta-mixin/v1alpha2/{name=resourceShadows/*}:watch

WatchResourceShadows Method

WatchResourceShadows

rpc WatchResourceShadows(WatchResourceShadowsRequest) returns (WatchResourceShadowsResponse)

with the following messages:

The equivalent REST API is:

POST /meta-mixin/v1alpha2/resourceShadows:watch

CreateResourceShadow Method

CreateResourceShadow

rpc CreateResourceShadow(CreateResourceShadowRequest) returns (ResourceShadow)

with the following messages:

The equivalent REST API is:

POST /meta-mixin/v1alpha2/resourceShadows (BODY: resource_shadow)

UpdateResourceShadow Method

UpdateResourceShadow

rpc UpdateResourceShadow(UpdateResourceShadowRequest) returns (ResourceShadow)

with the following messages:

The equivalent REST API is:

PUT /meta-mixin/v1alpha2/{resource_shadow.name=resourceShadows/*} (BODY: resource_shadow)

DeleteResourceShadow Method

DeleteResourceShadow

rpc DeleteResourceShadow(DeleteResourceShadowRequest) returns (Empty)

with the following messages:

The equivalent REST API is:

DELETE /meta-mixin/v1alpha2/{name=resourceShadows/*}

EstablishReferences Method

EstablishReferences

rpc EstablishReferences(EstablishReferencesRequest) returns (Empty)

with the following messages:

The equivalent REST API is:

POST /meta-mixin/v1alpha2/resourceShadows:establishReferences

ConfirmBlockades Method

ConfirmBlockades

rpc ConfirmBlockades(ConfirmBlockadesRequest) returns (Empty)

with the following messages:

The equivalent REST API is:

POST /meta-mixin/v1alpha2/resourceShadows:confirmBlockades

RemoveMetaOwnerReference Method

RemoveMetaOwnerReference

rpc RemoveMetaOwnerReference(RemoveMetaOwnerReferenceRequest) returns (Empty)

with the following messages:

The equivalent REST API is:

POST /meta-mixin/v1alpha2:removeMetaOwnerReference

ResourceShadow Messages

Here is the list of ResourceShadow resource messages:

ResourceShadow Message

Name Type Description
name string Name of ResourceShadow. ID part must contain fully qualified resource name.
resource_type string Type of resource for this shadow
version string Version of API for this resource. It will be always newest, however, if some service uses older version of imported service, imported service should convert shadows to that lower version. TODO: This is NOT implemented.
references ResourceShadow.ReferencesState Current state of all references and information about back refs.
lifecycle ResourceShadow.Lifecycle Lifecycle indicates current life cycle state of resource.
resource_metadata .goten.types.Meta Metadata copy from shadowed resource

ResourceShadow.ServiceRegionPair Message

ServiceRegionPair is a binding of service with region. They are used as service location, showing where objects of interest are.

Name Type Description
service string
region string

ResourceShadow.ReferencesState Message

ReferencesState describes all current references a resource has, deletion behaviors and sources of back refs. It is crucial component in enforcing database rules regarding reference constraints, garbage collection etc. It has information about all references, sources od back refs, describes meta references, blocking, those with asynchronous deletion triggers and so on - just big block of information.

Name Type Description
all repeated string List of all referenced resources - both in schema and metadata.
back_ref_locations repeated ResourceShadow.ServiceRegionPair List of all locations with references to this resource. Its a kind of back reference - except we only show locations where they can be found. To obtain them in full, it is required to make List request for each service-region pair and filter must use CONTAINS condition for field “all”.
with_target_del_handlers repeated string List of references in schema with asynchronous deletion handler (unset or cascade delete). This list is used by garbage collection functions. Length of this list is the same as target_del_handlers. Each reference in this list corresponds to handler with same position in that other list.
target_del_handlers repeated ResourceShadow.ReferencesState.DeletionHandler Deletion handlers attached to each reference in with_target_del_handlers list.
async_delete_subscribers repeated ResourceShadow.ServiceRegionPair List of all service-region pairs which contain resources that are interested in deletion of this resource, because they have schema references with target deletion behavior (or had). It does not include metadata references, as those are removed using different mechanism.
meta_owners repeated string List of all references present in metadata as owners. In comparison with schema references, referencing service may not be aware of referenced service existence.
meta_ownee_locations repeated ResourceShadow.ServiceRegionPair List of all service-region pairs which contain resources that point to this resource as their meta owner.
blocking repeated string List of all blocking references. Each reference in this list has associated details in blockade_details.
blockade_details repeated ResourceShadow.ReferencesState.BlockadeInfo Detailed information about blocking references in blocking field.
blocking_sources repeated ResourceShadow.ServiceRegionPair List of all service-region pairs which have resource blocking current resource. This list needs to be cleared before deletion can take place.
tentative_blockades repeated ResourceShadow.ReferencesState.TentativeBlockade List of resources blocking current resource that are in the process of creation. By design, blockade is installed on blocked resource before blocking resource is created, so we never end up with dandling blocking reference.

ResourceShadow.Lifecycle Message

Lifecycle describes life stage of current resource and additional information about tasks that must be executed if it is in pending or deleting state.

Name Type Description
state ResourceShadow.Lifecycle.State Current resource state
pending_blockades_to_confirm bool If this flag is true, it means this resource has gained new blocking references (usually as a result from create operation, but not necessarily). Db controller needs to remove tentative blocks from ReferencesState object in target resources. This flag is used when State is equal to PENDING
pending_delete bool Pending delete is used by db controller to mark resource as waiting for deletion - but which cannot be deleted yet, because it is being blocked. It means that Delete<Resource> call has not been executed yet. This flag is used when State is equal to PENDING
check_meta_ownees bool This flag informs that db controller should iterate over all meta ownee resources, select those with “requires_owner_reference” flag (See type ntt.meta.OwnerReference), and verify if this resource has still strong reference to them - if not, it means they were disowned and need to be cleaned. This flag is used when constraint store notices that some resource unset some of its references and perhaps it affects meta ownees.
check_meta_owners bool This flag informs that db controller should iterate over all meta owner references, select those with “requires_owner_reference” flag (See type ntt.meta.OwnerReference), and verify if this resource has strong reference to them - if not, it means they were disowned and need to be cleaned. This flag is set when new meta reference with above flag has been added. Db controller needs to take into account that meta ownee may have been created but transaction for meta owner may be still ongoing, so removal needs proper delay.

ResourceShadow.ReferencesState.DeletionHandler Message

DeletionHandler is attached to references with defined asynchronous behavior on target deletion. Synchronous are handled by server.

Name Type Description
target_location ResourceShadow.ServiceRegionPair Location of the target resource
ref_field_path string Field path of the reference
handler_type ResourceShadow.ReferencesState.DeletionHandler.HandlerType Desired behavior for target deletion.

ResourceShadow.ReferencesState.BlockadeInfo Message

BlockadeInfo is attached to references that block target deletion.

Name Type Description
target_location ResourceShadow.ServiceRegionPair Location of blocked resource
ref_field_path string Field path of blocking reference

ResourceShadow.ReferencesState.TentativeBlockade Message

TentativeBlockade describes temporary tentative blockade that needs to be confirmed. Blockades are installed when blocking reference is set for the first time, but before transaction concludes. It is being removed ASAP by controller after transaction finishes. If it does not finish, blockade will be removed after timeout.

Name Type Description
blocking_resource string Back reference resource.
resource_location ResourceShadow.ServiceRegionPair Back reference location.
attach_time .google.protobuf.Timestamp Time when blockade was installed. It is used to determine whether transaction should be assumed as failed.
tx_id string Tx identifier as chosen by server making write. In conjunction with tx_attempt_counter it may automatically invalidate previous tentative blocks, so we dont rely on timeout.
tx_attempt_counter int32 Information which tx attempt it is.

GetResourceShadowRequest Message

A request message of the GetResourceShadow method.

Name Type Description
name string Name of ntt.meta.mixin.v1alpha2.ResourceShadow
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view .goten.types.View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetResourceShadowsRequest Message

A request message of the BatchGetResourceShadows method.

Name Type Description
names repeated string Names of ResourceShadows
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view .goten.types.View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

BatchGetResourceShadowsResponse Message

A response message of the BatchGetResourceShadows method.

Name Type Description
resource_shadows repeated ResourceShadow found ResourceShadows
missing repeated string list of not found ResourceShadows

ListResourceShadowsRequest Message

A request message of the ListResourceShadows method.

Name Type Description
page_size int32 Requested page size. Server may return fewer ResourceShadows than requested. If unspecified, server will pick an appropriate default.
page_token string A token identifying a page of results the server should return. Typically, this is the value of ListResourceShadowsResponse.next_page_token.
order_by string Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination list of field path with order directive, either ‘asc’ or ‘desc’. If direction is not provided, ‘asc’ is assumed. e.g. “state.nested_field asc, state.something.else desc, theme”
filter string Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view .goten.types.View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask
include_paging_info bool Indicates if list response should contain total count and offset (fields current_offset and total_results_count).

ListResourceShadowsResponse Message

A response message of the ListResourceShadows method.

Name Type Description
resource_shadows repeated ResourceShadow The list of ResourceShadows
prev_page_token string A token to retrieve previous page of results. Pass this value in the ListResourceShadowsRequest.page_token.
next_page_token string A token to retrieve next page of results. Pass this value in the ListResourceShadowsRequest.page_token.
current_offset int32 Current offset from the first page or 0 if no page tokens were given, paging info was not requested or there was an error while trying to get it). Page index can be computed from offset and limit provided in a request.
total_results_count int32 Number of total ResourceShadows across all pages or 0, if there are no items, paging info was not requested or there was an error while trying to get it.

WatchResourceShadowRequest Message

A request message of the WatchResourceShadow method.

Name Type Description
name string Name of ntt.meta.mixin.v1alpha2.ResourceShadow
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view
view .goten.types.View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask

WatchResourceShadowResponse Message

A response message of the WatchResourceShadow method.

Name Type Description
change ResourceShadowChange

WatchResourceShadowsRequest Message

A request message of the WatchResourceShadows method.

Name Type Description
type .goten.types.WatchType Type of a watch. Identifies how server stream data to a client, which fields in a request are allowed and which fields in response are relevant.
page_size int32 Requested page size. Server may return fewer ResourceShadows than requested. If unspecified, server will pick an appropriate default. Can be populated only for stateful watch type.
page_token string A token identifying a page of results the server should return. Can be populated only for stateful watch type.
order_by string Order By - https://cloud.google.com/apis/design/design_patterns#list_pagination Can be populated only for stateful watch type.
resume_token string A token identifying watch resume point from previous session. Can be populated only for stateless watch type.
starting_time .google.protobuf.Timestamp Point in the time from which we want to start getting updates. This field can be populated only for stateless watch type and if resume token is not known yet. If specified, initial snapshot will NOT be provided. It is assumed client can obtain it using separate means. Watch responses will contain resume tokens which should be used to resume broken connection.
filter string Filter - filter results by field criteria. Simplified SQL-like syntax with following operators: <=, >=, =, !=, <, >, LIKE, CONTAINS (aliases CONTAIN, HAS, HAVE), IN, IS [NOT] NULL
field_mask .google.protobuf.FieldMask A list of extra fields to be obtained for each response item on top of fields defined by request field view Changes to ResourceShadow that don’t affect any of masked fields won’t be sent back.
view .goten.types.View View defines list of standard response fields present in response items. Additional fields can be amended by request field field_mask Changes to ResourceShadow that don’t affect any of masked fields won’t be sent back.
max_chunk_size int32 Maximum amount of changes in each response message. Query result response is divided on the server side into chunks with size of a specified amount to limit memory footprint of each message. Responses will hold information whether more elements will continue for the actual change. If unspecified, server will pick an appropriate default.

WatchResourceShadowsResponse Message

A response message of the WatchResourceShadows method.

Name Type Description
resource_shadow_changes repeated ResourceShadowChange Changes of ResourceShadows
is_current bool If request specified max_chunk_size (or this limit was enforced if stateless watch has been chosen), then responses with “full changeset” will be divided into chunks. Client should keep receiving messages and, once is_current has value true, combine this recent message with all previous ones where is_current is false. If this is the first is_current in a whole watch stream, then it means that client should have, at this moment, contain snapshot of the current situation (or more accurately, snapshot of situation at the moment of request). All ResourceShadows will be of type Added/Current (depending on watch_type specified in the request). Further responses will be incremental - however messages may still be chunked and is_current logic still applies. is_current is always true for stateful watch if max_chunk_size was left to 0.
page_token_change WatchResourceShadowsResponse.PageTokenChange When present, PageTokens used for page navigation should be updated. Present only if is_current is true (last chunk).
resume_token string Token that can be used if current connection drops and client needs to reconnect. Populated only for stateless watch type. Present only if is_current is true (last chunk).
snapshot_size int64 Server may occasionally send information how many resources should client have in its state so far (response message without any changes, but with snapshot_size field specified). If client has different value than the one sent by the server, then it should be treated by a client as an error and should reconnect. If value is smaller then 0, then client should ignore this field as unpopulated. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will be never sent together with is_current, is_soft_reset and is_hard_reset flags.
is_soft_reset bool In case of internal issue server may send response message with this flag. It indicates that client should drop all changes from recent responses where is_current is false only! If last message had is_current set to true, client should do nothing and process normally. Resume token received before is still valid. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_hard_reset or snapshot_size.
is_hard_reset bool In case of internal issue server may send response message with this flag. After receiving, client should clear whole state (drop all changes received so far) as server will send new snapshot (ResourceShadows will contains changes of type Current only). Any resume tokens should be discarded as well. This field should be checked only for stateless watch. In stateful those kind of errors are handled by the server side. Will never be sent along with is_current, is_soft_reset or snapshot_size.

WatchResourceShadowsResponse.PageTokenChange Message

Name Type Description
prev_page_token string New token to retrieve previous page of results.
next_page_token string New token to retrieve next page of results.

CreateResourceShadowRequest Message

A request message of the CreateResourceShadow method.

Name Type Description
resource_shadow ResourceShadow ResourceShadow resource body
response_mask CreateResourceShadowRequest.ResponseMask Optional masking applied to response object to reduce message response size.

CreateResourceShadowRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields.

UpdateResourceShadowRequest Message

A request message of the UpdateResourceShadow method.

Name Type Description
resource_shadow ResourceShadow ResourceShadow resource body
update_mask .google.protobuf.FieldMask FieldMask applied to request - change will be applied only for fields in the mask
cas UpdateResourceShadowRequest.CAS Conditional update applied to request if update should be executed only for specific resource state. If this field is populated, then server will fetch existing resource, compare with the one stored in the cas field (after applying field mask) and proceed with update only and only if they match. Otherwise RPC error Aborted will be returned.
allow_missing bool If set to true, and the resource is not found, a new resource will be created. In this situation, ‘field_mask’ is ignored. https://google.aip.dev/134#create-or-update
response_mask UpdateResourceShadowRequest.ResponseMask reduce message response size.

UpdateResourceShadowRequest.CAS Message

CAS - Compare and Swap. This object is used if user wants to make update conditional based upon previous resource version.

Name Type Description
conditional_state ResourceShadow Conditional desired state of a resource before update.
field_mask .google.protobuf.FieldMask Field paths from conditional state of resource server should check and compare.

UpdateResourceShadowRequest.ResponseMask Message

ResponseMask allows client to reduce response message size.

Name Type Description
skip_entire_response_body bool If this flag has value true, then response will contain just empty resource without any fields populated. Field body_mask is ignored if set.
updated_fields_only bool Include all fields that were actually updated during processing. Note this may be larger than update mask if some fields were computed additionally. Name is added as well.
body_mask .google.protobuf.FieldMask If this field is populated, then resource in response will contain only specific fields. If skip_entire_response_body is true, this field is ignored.

DeleteResourceShadowRequest Message

A request message of the DeleteResourceShadow method.

Name Type Description
name string Name of ntt.meta.mixin.v1alpha2.ResourceShadow

EstablishReferencesRequest Message

Request message for method [EstablishReferences][ntt.meta.mixin.v1alpha2.EstablishReferences] This request is being sent:

  • by server, when there is a change in references in some resource (like some new reference has been added… This request can also be used to change nature of some reference, like from switch BLOCKING to ASYNC_CASCADE_DELETE in schema).
  • by db-controller, as a way to confirm connection with meta owner (shadows_with_new_meta_ownees). Tx fields are not populated in this case
Name Type Description
requester_service ServiceRegionPair Requester service indicates service and region location of shadow that is trying to establish reference bindings.
referencing_shadow string Shadow with references to new resources.
referenced_shadows repeated string List of all NEW referenced shadows - if resource started pointing to some new resource that was not before.
blocked_shadows repeated string List of blocking references from referencing shadow.
shadows_with_new_del_subs repeated string List of shadows requested to install asynchronous subscription for potential delete operation on requester, because it has references causing asynchronous deletion in schema.
shadows_with_new_meta_ownees repeated string List of shadows requested to install asynchronous subscription for potential delete operation on requester, because it has meta references, which, once empty, need to trigger asynchronous deletion.
tx_id string Identifier of local transaction on an instance that sent this request. Receiver of EstablishReferencesRequest should check, along with tx_attempt field, if this is not the first request - and if not, if there are things to drop from previous one.
tx_attempt int32 Indicator (from 1)

ConfirmBlockadesRequest Message

Request message for method [ConfirmBlockades][ntt.meta.mixin.v1alpha2.ConfirmBlockades] This method is additional to EstablishReferencesRequest and is sent by db controller to confirm existence of blocking references.

Name Type Description
requester_service ServiceRegionPair Location of referencing shadow
referencing_shadow string Referencing shadow that wishes to confirm blocking references.
blocked_shadows repeated string List of blocking references that need to be confirmed.

RemoveMetaOwnerReferenceRequest Message

Request message for method [RemoveMetaOwnerReference][ntt.meta.mixin.v1alpha2.RemoveMetaOwnerReference] This method is used to notify meta ownees that their parent has been deleted.

Name Type Description
owner_shadow_location ServiceRegionPair Location of owner resource
owner_shadow string Reference of owner shadow.
ownee_shadow string Reference of ownee shadow - which points to owner in its metadata.

ResourceShadow Enumerations

Here is the list of ResourceShadow resource enumerations:

ResourceShadow.ReferencesState.DeletionHandler.HandlerType Enumeration

Describes whether reference should be only unset or perhaps owning resource should be deleted.

Name Description
UNDEFINED
CASCADE_DELETE
UNSET

ResourceShadow.Lifecycle.State Enumeration

State points to the current resource state. Some states are watched by db controller.

Name Description
UNDEFINED Never used
ACTIVE Active is the default state where resource will spend 99.9% of its life time. Resource is fully functioning and db controller does not need to bother with it.
PENDING Pending state describes resource that already is fully visible and functioning in database - but requires some attention from db controller, because some references have changed recently.
DELETING Deleting state indicates that delete operation was already executed on resource. Proper resource is already deleted from database (TODO: we need to add deletion with progress bar, where original resource remains in db despite delete method already executed). Shadow will exist in this state until all deletion handlers are executed.

MetaMixin Service Shared Methods and Messages

MetaMixin Service Shared Messages

Here is the list of MetaMixin service shared messages:

ResourceShadowChange Message

ResourceShadowChange is used by Watch notifications Responses to describe change of single ResourceShadow One of Added, Modified, Removed

Name Type Description
added ResourceShadowChange.Added Added is returned when watched document is added, either created or enters Query view
modified ResourceShadowChange.Modified Modified is returned when watched document is modified
current ResourceShadowChange.Current Current is returned in stateless watch when document enters query view or is modified within.
removed ResourceShadowChange.Removed Removed is returned when ResourceShadow is deleted or leaves Query view

ResourceShadowChange.Added Message

ResourceShadow has been added to query view

Name Type Description
resource_shadow ResourceShadow
view_index int32 Integer describing index of added ResourceShadow in resulting query view.

ResourceShadowChange.Current Message

ResourceShadow has been added or modified in a query view. Version used for stateless watching

Name Type Description
resource_shadow ResourceShadow

ResourceShadowChange.Modified Message

ResourceShadow changed some of it’s fields - contains either full document or masked change

Name Type Description
name string Name of modified ResourceShadow
resource_shadow ResourceShadow New version of ResourceShadow or masked difference, depending on mask_changes instrumentation of issued [WatchResourceShadowRequest] or [WatchResourceShadowsRequest]
field_mask .google.protobuf.FieldMask Used when mask_changes is set, contains field paths of modified properties.
previous_view_index int32 Previous view index specifies previous position of modified ResourceShadow. When modification doesn’t affect sorted order, value will remain identical to [view_index].
view_index int32 Integer specifying ResourceShadow new index in resulting query view.

ResourceShadowChange.Removed Message

Removed is returned when ResourceShadow is deleted or leaves Query view

Name Type Description
name string
view_index int32 Integer specifying removed ResourceShadow index. Not populated in stateless watch type.

ServiceRegionPair Message

Name Type Description
service string
region string