API Server Flow

To enforce general schema consistency, we must first properly handle requests coming from users, especially writing ones.

The following rules are executed when API servers get a write call:

• when a writing request is sent to the server, multi-region routing middleware must inspect the request, and ensure that all resources that will be written to (or deleted), are owned by the current region. It must store the MultiRegionPolicy object in the context associated with the current call.
• write requests can only execute write updates for a resources under single multi-region policy! It means that writing across let’s say two projects will not be allowed. It is allowed to have writing operations to global resources though. If there is an attempt to write to multiple resources across different policy holders in a single transaction, the Store object must reject the write.
• Store object must populate the metadata.syncing field when saving. It should use MultiRegionPolicy from context.
• When the server calls the Save or Delete function on the store interface (for whatever Service resource), the following things happen:
  • If this is a creation/update, and the new resource has schema references that were not there before, then the Store is responsible for connecting to those Services and ensuring that resources exist, the relationship is established, and it is allowed to establish references in general. For references to local resources, it also needs to check if all is fine.
  • If this is deletion, the Store is obliged to check if there are any blocking back-references. It needs to connect with Deployments where references may exist, including self. For local synchronous cascade deletion & unset, it must execute them.
  • When Deployment connects with others, it must respect their API versions used.
• Meta owner references are not checked, because it is assumed they may be created later. Meta-owner references are asynchronously checked by the system after the request is completed.

This is a designed flow for API Servers, but we have a couple more flows regarding schema consistency. First, let’s define some corner cases when it comes to blocking references across regions/services. Scenario:

• Deployment D1 gets a write (Creation) to resource R1. Establishes SNAPSHOT transaction.
• R1 references (blocking) R2 in Deployment D2, therefore, on the Save call, D1 must ensure everything is valid.
• Deployment D1 sends a request to establish a blocking reference to R2 for R1. D2 can see R2 is here.
• D2 blocks resource R2 in its SNAPSHOT transaction. Then sends a signal to D1 that all is good.

Two things can happen:

• D1 may fail to save R1 because of the failure of its local transaction. Resource R2 may be left with some blockade.
• Small chance, but after successful blockade on R2, D2 may get delete R2 request, while R1 still does not exist, because D1 did not finish its transaction yet. If D2 asks D1 for R1, D1 will say nothing exists. R2 will be deleted, but then R1 may appear.

Therefore, when D2 blocks resource R2, it is a special tentative blockade with a timeout of up to 5 minutes, if I recall the amount correctly. This is way more than enough since transactions are configured to timeout after one minute. It means R2 will not be possible to delete for this period. Then protocol continues:

• If D1 fails transaction, D2 is responsible to asynchronously remove tentative blockade from R2.
• If D1 succeeds the transaction, then D1 is responsible for informing in an asynchronous manner that tentative blockade on R1 is confirmed.