SPEKTRA Edge Limits Service API

Understanding the Limits service API.

The limits service is a core SPEKTRA Edge service that enables service developers or product owners to enforce rules/quotas on the maximum amount of resources that can exist within different scopes:

  • Service scope:

    Global service limits based on the controller capacity.

  • Organization scope:

    Limits assigned to the reseller.

  • Project scope:

    Limits assigned to the individual customer.

Limits are passed from top to bottom. First, the service administrator sets the limits for the whole service. Once service limits are initialized, they can be used to allocate resources to root organizations and projects. Organizations can then pass limits further down to their child organizations or projects.

Each Service is managing its quotas.

Limits/Usage could also be tied to the billing/licensing system.

Features/Capabilities of Limits service:

  • Ensures that limits in child projects/organizations do not exceed those for their parents (organizations or system limits).
  • Ensures correctness and database transaction guarantees.
  • Enables passing down limits in various cases (like region to region within a project, parent organization to child organization, etc).
  • Uses monitoring service to record historical changes in resource consumption.
  • Ensures that limits can be passed only between parent/child organization/project.
  • Enforces parent organization/service consent before modifying limits in child entities.
  • Is designed to work in a multi-regional environment. When a limit is assigned to a multi-regional project, then limits are split across all regions. The system tracks each regional limit individually.

Full API Specifications (with resources):

Resources

Accepted Limit Plans

An accepted limit plan is a resource and is very close to the plan assignments. However, while a plan assignment binds a plan with an assignee, an accepted limit plan binds a limit plan with an assigner for a given assignee. The accepted limit plan represents resource limits that an assigner granted to an assignee.

When an accepted limit plan is created or the limits are increased, limits assigned to the assigner are reserved for an assignee with an immediate effect. When the accepted plan is deleted, or the limits are decreased, the limits are returned to the assigner once they have been cleaned up.

The system constantly syncs configuration from accepted limit plans into plan assignments, in an asynchronous manner. It also means that the organization/service administrators can practically modify plan assignments.

If an organization/service administrator accepts any plan assignment request, they are making indirect modifications to their accepted limit plan instances, but both ways use the same workflow.

Limits

A limit combines a project, resource type, region, and two configurable values:

  1. allowance
  2. current usage

It makes sure that the number of resources in a given project/ region does not exceed the allowed value. It is important to note that Limits are only for projects, final assignees, which can create resources that count into usage.

Each service in each region maintains its own set of trackers for all present projects.

It is also important to note, that each limit is derived from a limit plan assignment that contains a given resource type. They are being created/modified/deleted according to the configuration stored in limit plan assignments. They cannot be modified by administrators.

Limit Assignees

A limit assignee is an SPEKTRA Edge entity capable of receiving limits from their parent. A limit assignee is typically an organization or a project. Service is a special type of assignee, it can assign resources to itself without taking from parent entity limits. Limits for the service should be decided by service owners or developers. Services and organizations are both the assignor and the assignee, only projects are the final assignees.

Limit Assigners

A limit assigner is an SPEKTRA Edge entity capable of passing down their limits to their child entities. Because only services and organizations may have child entities, sub-organizations and projects) the assigner could be either an organization (organization admin) or service (service admin).

Limit Plans

A limit plan is a set of limits per resource type. It must be associated with some SPEKTRA Edge services. Resources it contains must be from that service.

Assignees may have many limit plans assigned at once, but each must be from a different service.

Each service must define default limit plans, but organizations can define them too, for their child entries.

Limit Plan Assignments

A limit plan assignment is a resource that binds a limit plan with an assignee. It may also include extra limits that increase default values predicted by the plan.

limit plan assignments are not modifiable. Changes should be requested by the assignee or should be changed by assigners (it is assumed that assigners will respect any service agreement).

Limit plan Assignments for projects also contain regional distribution of limits.

Limit Plan Assignment Requests

A limit plan Assignment Request is a formal request made by an assignee that, once accepted by the assigner, creates a new plan assignments or modifies an existing one. This resource is a formal API allowing an assignee to request a change in their limits.

A limit plan assignment request can be used for:

  • Assigning/Upgrading/Downgrading plan
  • Extending/Decreasing individual limits for specific resources
  • Unassigning existing limit plan
  • Certain requests may be automatically accepted by the system.

Limit Pools

A limit pool is similar to limit as it also “limits” the number of resources in existence. However, while Limit keeps track of resources in a specific scope and ensures it does not exceed the allocated value, the limit pool serves as a source to limits (and other limit pools). Limit instances belonging to projects, final assignees. Limit Pools belong to services and organizations, which are passing limits down to projects and sub-organizations.

Like limits, the individual limit pool are computed from limit plan assignments belonging to the service/organization. Whenever assignments change, the limit pools are modified accordingly. They cannot be modified by administrators.

Multi-Region Note

Plans define resource limits per each region. For example, when the Service assigns itself a plan with 100K Devices from the devices.edgelq.com service, it means that it will have LimitPool for each region with a value of 100K for Devices.

When the Service expands to a new region, it will get new LimitPool resources for all its resource types. Those new LimitPools will be brand new, and existing LimitPools will not change.

When an Organization is assigned some plan, all resource limits from a plan will be applied for each region in which the organization is enabled. Therefore, LimitPool for a particular resource type will be created for each organization region.

When an Organization expands to a new region, it will get new LimitPool resources, and existing ones will not be touched.

LimitPools for Service and Organizations are always regional - regardless of whether the resource type is regional itself or not.

There is a difference on a Project level though. For example, let’s take two resource types:

  • services/applications.edgelq.com/resources/Distribution

    This is a non-regional resource.

  • services/applications.edgelq.com/resources/Pod

    This is a regional resource.

Suppose we create a Project with a plan that predicts:

  • 10 Distributions, 100 Pods (per Region)

If the Project is enabled for a single region, let’s say us-west2, it will have limits:

  • 10 Distributions, Region: us-west2
  • 100 Pods, Region: us-west2

Parent Organization will have the following reservations on its limit pools:

  • Distributions in us-west2: 10 reserved quota
  • Pods in us-west2: 100 reserved quota

If the project expands to a new region, let’s say eastus2, then the project will have the following limits:

  • 10 Distributions, Region: us-west2
  • 100 Pods, Region: us-west2
  • 100 Pods, Region: eastus2

Parent Organization will have the following reservations on its limit pools:

  • Distributions in us-west2: 10 reserved quota
  • Distributions in eastus2: 10 reserved quota
  • Pods in us-west2: 100 reserved quota
  • Pods in eastus2: 100 reserved quota

Pods should be clear, but Distributions require clarification as, to why a Limit of 10 requires sourcing two LimitPools from both enabled regions. The main thing is, that “global” resources are putting pressure on all regions where Project is enabled - typically, they are synced across all regions. Distribution resources are working in all project regions. One instance of Distribution, even if it is managed by us-west2 only, can create pods in both regions. This is why they source from all limit pools for all enabled regions.

Limits Management

Assignee and Assigner have a different API for them but note that Organization is generally both an Assignee and Assigner

  • Assigner, because it can have child projects and organizations. Assignee, because it has either parent organization, or if this is root organization, assigners are services themselves.

For an Assignee (Project or Organization admin)

The organization is assigned plans based on allowed_services values, plus it automatically gets plans for core SPEKTRA Edge services: devices.edgelq.com and applications.edgelq.com (implicitly). There is no need to do anything to have core SPEKTRA Edge plans assigned, it should be enough to just create an Organization. Plans for particular services are enabled when the Organization is adding a service to the allowed_services list.

Projects are similar, they get assigned core plans automatically, and extra service plans only when they enable services via the enabled_services field. PlanAssignments are created automatically unless the parent entity has no free limit pools.

To see PlanAssignments that were created for organization, see:

cuttle limits list plan-assignments --organization <orgId> -o json

For project:

cuttle limits list plan-assignments --project <projectId> -o json

If you want to see limit pools assigned (for organization):

cuttle limits list limit-pools --organization <orgId> -o json

For a project, we should check limits:

cuttle limits list limits --project <projectId> -o json

If there is a need to change limits for a specific plan, it is required that either:

  • Parent Organization admin (or Service for root organizations assignees) updates AcceptedPlan resource from which PlanAssignment sources information.
  • Assignee admin creates PlanAssignmentRequest, which then is accepted by parent organization admin (or Service admin for root organizations)

Let’s assume that there is Parent Organization, which defined the following custom plan for devices:

# --plan-level PROJECT for project assignees!
cuttle limits create plan  --organization <parentOrgId> devices-service-500-devices \
  --service 'services/devices.edgelq.com' \
  --plan-level ORGANIZATION \
  --business-tier LARGE \
  --resource-limits '{"resource": "services/devices.edgelq.com/resources/Device", "value": 500}'

For example, parent organization admin could then modify AcceptedPlan, apply plan with 500 devices for each region.

cuttle limits update accepted-plan <AcceptedPlanId> \
  --defaultRegionalPlan organizations/<parentOrgId>/plans/devices-service-500-devices

Project admins can request, with let’s say some extensions:

cuttle limits create plan-assignment-request --project <myProjectId> \
  --request '{"assign": {"plan": "organizations/<parentOrgId>/plans/devices-service-500-devices", "extensions": [{"resource": "services/devices.edgelq.com/resources/Device", "value": 50}]}}'

Organization admins are similar:

cuttle limits create plan-assignment-request --organization <myOrgId> \
  --request '{"assign": {"plan": "organizations/<parentOrgId>/plans/devices-service-500-devices", "extensions": [{"resource": "services/devices.edgelq.com/resources/Device", "value": 50}]}}'

Instead of plan assignment, we can also just ask for some extra extensions:

cuttle limits create plan-assignment-request --project <myProjectId> \
  --request '{"extend": {"assignment": "<MyPlanAssignmentName>", "extensions": [{"resource": "services/devices.edgelq.com/resources/Device", "value": 50}]}}'

For an assigner (Organization or Service admin)

Limits service creates automatically AcceptedPlans for all child projects and organizations, depending on the services they enable, and business tiers.

If you are a service admin, you can see all AcceptedPlan instances created for root organizations and projects using this service:

cuttle limits list accepted-plans --service '<myServiceId>'

Organization admin can see plans they accepted with:

cuttle limits list accepted-plans --organization '<myOrgId>'

To see overall state of limit pools, service admins should execute:

cuttle limits list limit-pools --service <myServiceId> -o json

Organization admins can see their limit pools with:

cuttle limits list limit-pools --organization <myOrgId> -o json

Assigners (Service or Organization admins) can define plan templates on their own. Service admins should use fixtures, as described in the Developer guide. Organization admins can potentially use just default plans defined by a Service for their child entities, however, limit values may not match exactly their needs. Instead, they are recommended to define their plans, for example:

cuttle limits create plan --organization <parentOrgId> \
  devices-service-500-devices \
  --service 'services/devices.edgelq.com' \
  --plan-level ORGANIZATION \
  --business-tier LARGE \
  --resource-limits '{"resource": "services/devices.edgelq.com/resources/Device", "value": 500}'

Of course, it is necessary to provide resource limits for all required resource types. Moreover, the plan should be created for all plan-level and business-tier combinations required.

This is how assigners can create plans on their own. When they create child entities, let’s say a Project:

cuttle iam create project <projectId> --parent-organization \
  'organizations/<parentOrgId>' --business-tier <Tier> \
  --multi-region-policy <MultiRegionPolicy> \
  --enabled-services <EnabledService1> ... \
  --enabled-services <EnabledServiceN>

In the above case (Assigner is an Organization), Limits service will create automatically AcceptedPlans for all core SPEKTRA Edge and enabled services, where --plan-level is PROJECT, and --business-tier matches the one specified in <Tier> argument.

Assigner admins can manipulate AcceptedPlans further in two ways:

  • By accepting PlanAssignmentRequests made by child entity admins (if they initiate an action)
  • By directly modifying AcceptedPlan instances

For example, AcceptedPlan can be directly modified from the plan assigned by default:

cuttle limits update accepted-plan <AcceptedPlanId> \
  --defaultRegionalPlan \
    organizations/<parentOrgId>/plans/devices-service-500-devices

PlanAssignmentRequests are created by assignee admins (child project or organizations). Service admins can see requests for them with:

cuttle limits list-approver plan-assignment-requests \
  --approver 'services/<myServiceId>'

Organization admins (as assigners) can see requests for them with:

cuttle limits list-approver plan-assignment-requests \
  --approver 'organizations/<myOrgId>'

Service admins can accept/decline requests with:

cuttle limits accept plan-assignment-request \
  --name '<PlanAssignmentRequestFullName>' \
  --approver 'services/<myServiceId>'
cuttle limits decline plan-assignment-request \
  --name '<PlanAssignmentRequestFullName>' \
  --approver 'services/<myServiceId>'

Organization admins can accept/decline requests with:

cuttle limits accept plan-assignment-request \
  --name '<PlanAssignmentRequestFullName>' \
  --approver 'organizations/<myOrgId>'
cuttle limits decline plan-assignment-request \
  --name '<PlanAssignmentRequestFullName>' \
  --approver 'organizations/<myOrgId>'

Understanding the limits.edgelq.com service APIv1, in proto package ntt.limits.v1.

Understanding the limits.edgelq.com service APIv1alpha2, in proto package ntt.limits.v1alpha2.