Resource Limit Tracking

Understanding how the resource limit is tracked.

We need to provide a guarantee that the usage tracker stays in sync with the actual resource counter. The best way to do that is to count during local transactions. However, resource Limit belongs to the Limits service, not the actual servicee. This is why we have Limits Mixin Service in the SPEKTRA Edge repository, mixins/limits.

It injects one resource type: LocalLimitTracker. Note it is a regional resource, but not a child of a Project. This means that no project admin will be able to see this resource ever, or any parent organization. This resource type is hidden, only service admins will be able to see it. This prevents any chance of final user mismanagement as well. Because this resource type is mixed along with final service resources, we can achieve SNAPSHOT transactions between actual resources and trackers. We can even prevent bugs that could result in the usage tracker having invalid values. When we create/update the LocalLimitTracker resource, we can extract the true counter from the local database, see file mixins/limits/server/v1/local_limit_tracker/local_limit_tracker_service.go.

To check how LocalLimitTracker usage is tracked during transactions, check two files:

  1. mixins/limits/resource_allocator/v1/resource_allocator.go
  2. common/store_plugins/resource_allocator.go

This is how the store plugin tracks creations/deletions, at the end of the transaction, it tries to push extra updates, LocalLimitTracker instances for all resource types where several instances changed. This guarantees complete synchronization with the database. But note this does not create LocalLimitTrackers yet.

This is why Limits Mixin comes with not only an API Server (so LocalLimitTrackers can be accessed) but also a controller, see mixins/limits/controller/v1 directory. Inside Limits processor we have:

  • LocalLimitTrackersManager instance, which Creates/Updates/Deletes instances of LocalLimitTracker for every Limit instance in Limits service.
  • Synchronizes Limit instances in limits service using LocalLimitTrackers from its region. It means that there is no actual point in meddling with Limit fields, the controller will fix them anyway, and they don’t participate in actual usage checking anyway.
  • Maintains also PhantomTimeSeries, so we have special store usage metrics, showing how historically resource counters were changing.

Note that the Limits processor in this controller has built-in multi-region features, primary region for project creates/deletes LocalLimitTrackers, but the final regions are maintaining Limit instances and PhantomTimeSeries.