## Problem Authorization for users who belong to many organizations is slow. On the list <br>endpoints (`/api/v2/organizations`, `/users`, `/groups`) a user in hundreds of <br>orgs saw multi-second page loads <br>([DEVEX-608](https://linear.app/codercom/issue/DEVEX-608/performance-degrades-for-users-in-many-organizations-across-multiple) <br>/ coder/coder#21890 / Pylon [#2758](<https://github.com/coder/coder/issues/2758>)). This is partial-evaluation bound: `rbac.Prepare` <br>scales with the number of org-scoped roles the subject carries. ## Root cause The known-org path in `check_org_permissions` indexed an N-entry vote map by the <br>object's org id: ```rego vote := allow_map[input.object.org_owner] ``` `input.object.org_owner` is unknown during partial evaluation. Indexing a map by <br>an unknown key cannot reduce to a single expression, so OPA emits one residual <br>query per org membership, and `newPartialAuthorizer` then calls `PrepareForEval` <br>once per residual, making `Prepare` O(N) in org count. The list endpoints <br>intentionally use partial eval; the fan-out is in partial eval itself. ## Change Test the object's org id for membership in a set that is fully known at <br>partial-evaluation time, so the query collapses to a single <br>`organization_id = ANY(ARRAY[...])` residual instead of N residuals: * The known-org clause only ever votes to allow, tested via <br>`org_owner in org_ids_with_vote(role_org_votes, 1)`. * Org-level denies are folded into the org-member level as a ground set <br>difference (`member_allow - org_deny`), so the unknown org id appears in only <br>one positive membership test and the decision never branches on it. * The per-org vote maps are computed once as memoized zero-arg rules <br>(`role_org_votes`, `role_member_votes`, `scope_org_votes`, <br>`scope_member_votes`) instead of through parametrized functions that OPA <br>re-evaluates at every call site. * `role_allow`/`scope_allow`, the `any_org` path, and full evaluation are <br>unchanged in behavior. Semantics are unchanged (see the equivalence argument below). The only <br>representational change is that a denied known org's intermediate `org` vote is <br>now `0` instead of `-1`, compensated by the set difference and not observable in <br>the final `allow` decision. ## Results Measured with `BenchmarkRBACManyOrgs` (added on `main` in coder/coder#27270). Full tables: <br>[B/op and allocs/op](<https://github.com/coder/coder/pull/27244#issuecomment-4984523720>). * Residual queries: O(N) -> O(1). * `Prepare` / `PrepareAndCompile` memory changes from < />quadratic growth on `main` <br>(176 MiB, 7.08M allocs per op at 100 orgs) to near-linear (6.5 MiB, 258k <br>allocs), a < />96% reduction at 100 orgs, with similar wins in time. * Memoizing the vote maps removed an early single-org regression: at 1 org <br>`Prepare` now allocates < />7% fewer bytes and < />9% fewer objects than `main`. * `Authorize` (full evaluation) memory is marginally higher (+1-8%, largest at <br>1 org) and time-neutral. This is the inherent cost of the set-membership form <br>that keeps partial evaluation from fanning out; full evaluation builds an <br>allow set it would not otherwise need. * `go test ./coderd/rbac/...` passes, including `TestAuthorizeDomain` (full- vs <br>partial-eval equivalence) and the regosql suite. A second, independent bottleneck remains (out of scope here): the vote map is <br>still built in O(N^2) in `check_all_org_permissions` <br>(`roles[_].by_org_id[org_id]` scans all roles per org). Fixing it means <br>pre-merging roles' `by_org_id` into one org->perms map in the OPA input, and is <br>tracked as a follow-up. ## Testing * `OrgDenyBlocksMember` (`TestAuthorizeLevels`): an org-level deny blocks a <br>member-allowed action on an owned in-org object, while a clean org is allowed, <br>including an action-scoped deny. * `ScopeOrgDenyBlocksMember` (`TestAuthorizeScope`): the same fold at the scope <br>level. * The shared harness covers full and partial evaluation and asserts the partial <br>result compiles to SQL with zero support rules. <details><summary>Decision log and equivalence argument</summary> ### Why not deny-via-set-membership The first attempt expressed deny as a second set-membership clause (`:= -1 if org_owner in deny_set`). That makes `org`/`scope_org` multi-valued, and the `not org = -1` checks in `role_allow`/`scope_allow` then cause OPA to emit a `data.partial.__not__` support rule that regosql cannot compile (`TestAuthorizeDomain/UserACLList` failed). It failed even when the deny set was empty, purely because the `-1` clause exists. ### Why not deny-via-enumeration A follow-up enumerated only the (usually empty) deny set. It compiled and passed, but it branches on the unknown org id (one ground residual per denied org), which violates the "do not branch on the unknown" rule in `coderd/rbac/POLICY.md`. ### Final approach: allow-only + ground set difference The known-org clause votes only to allow, and the org-level deny gate is moved into the org-member level as `member_allow - org_deny`, a set difference over fully-known sets. The unknown org id is used only in positive `in` tests, so there is no enumeration, no negated membership, and no branching on the unknown. ### Empty-set residual pruning A naive set-membership left unsatisfiable residuals (`org_owner in set()`) for levels with no matching permissions (e.g. the org level for an org-member role, or scope-org for `ScopeAll`), each still costing a `PrepareForEval`. Guarding each membership with a ground `count(...) > 0` lets OPA drop those branches, flattening the residual count across org sizes. ### Memoized vote maps Profiling the single-org path showed the cost was repeated function evaluation: the parametrized helpers rebuilt the same vote map for the org, member, and scope paths on every check. Hoisting the maps into memoized zero-arg complete rules (which OPA evaluates once per query) removed that overhead and eliminated the single-org `Prepare` regression, while composition keeps the policy readable. ### Equivalence (known-org path, `site != -1`) * A: original `org == 1` <=> `org_owner in org_allow` (unchanged). * B: original `org != -1 and member == 1` <=> `org_owner not in org_deny and org_owner in member_allow` <=> `org_owner in (member_allow - org_deny)` = new `org_member == 1`. The critical case (`org` denies, member allows): old blocks it via `not org = -1`; new blocks it because `org_owner` is removed from `member_allow - org_deny`. Same outcome. Deny-wins aggregation is intact because `check_all_org_permissions` still nets an org to `-1` via `to_vote`, landing it in `org_deny`. </details> --- This PR was generated by Coder Agents on behalf of @jeremyruppel.
5.4 KiB
Rego authorization policy
Code style
It's a good idea to consult the Rego style guide. The "Variables and Data Types" section in particular has some helpful and non-obvious advice in it.
Debugging
Open Policy Agent provides a CLI and a playground that can be used for evaluating, formatting, testing, and linting policies.
CLI
Below are some helpful commands you can use for debugging.
For full evaluation, run:
opa eval --format=pretty 'data.authz.allow' -d policy.rego -i input.json
For partial evaluation, run:
opa eval --partial --format=pretty 'data.authz.allow' -d policy.rego \
--unknowns input.object.owner --unknowns input.object.org_owner \
--unknowns input.object.acl_user_list --unknowns input.object.acl_group_list \
-i input.json
Playground
Use the Open Policy Agent Playground while editing to getting linting, code formatting, and help debugging!
You can use the contents of input.json as a starting point for your own testing input. Paste the contents of policy.rego into the left-hand side of the playground, and the contents of input.json into the "Input" section. Click "Evaluate" and you should see something like the following in the output.
{
"allow": true,
"check_scope_allow_list": true,
"org": 0,
"org_member": 0,
"org_memberships": [],
"permission_allow": true,
"role_allow": true,
"scope_allow": true,
"scope_org": 0,
"scope_org_member": 0,
"scope_site": 1,
"scope_user": 0,
"site": 1,
"user": 0
}
Levels
Permissions are evaluated at four levels: site, user, org, org_member.
For each level, two checks are performed:
- Do the subject's permissions allow them to perform this action?
- Does the subject's scope allow them to perform this action?
Each of these checks gets a "vote", which must one of three values:
- -1 to deny (usually because of a negative permission)
- 0 to abstain (no matching permission)
- 1 to allow
If a level abstains, then the decision gets deferred to the next level. When there is no "next" level to defer to it is equivalent to being denied.
Known-org asymmetry (org and org_member levels)
The org and org_member levels are evaluated differently depending on whether the object's org id is known.
When the org id is unknown (partial evaluation, e.g. filtering a list), the org id must be kept out of comprehensions and must not be branched on (see "Unknown values" below). To satisfy that, the known-org path tests the object's org id for membership in a set of allowed org ids instead of looking up its vote:
- The org level (
check_org_permissions, known-org clause) only ever votes1(allow) or abstains; it never votes-1for a known org. Thenot org = -1/not scope_org = -1gates in the allow rules are therefore no-ops for a known org and only block in theany_orgcase. - Org-level deny is instead folded into the org_member level as a ground set
difference (
member_allow - org_deny), so an org-level deny still blocks a member-level allow.
The any_org path ("can the subject do this in any org?") still uses the full
-1/0/1 vote (the max over the vote map), because there is no specific
object org id to be unknown. So do not assume org == -1 signals an org-level
deny for a known org; reconstruct it from org_ids_with_vote(role_org_votes, -1)
if you need it.
Scope
Additionally, each input has a "scope" that can be thought of as a second set of permissions, where each permission belongs to one of the four levels–exactly the same as role permissions. An action is only allowed if it is allowed by both the subject's permissions and their current scope. This is to allow issuing tokens for a subject that have a subset of the full subjects permissions.
For example, you may have a scope like...
{
"by_org_id": {
"<org_id>": {
"member": [{ "resource_type": "workspace", "action": "*" }]
}
}
}
...to limit the token to only accessing workspaces owned by the user within a specific org. This provides some assurances for an admin user, that the token can only access intended resources, rather than having full access to everything.
The final policy decision is determined by evaluating each of these checks in their proper precedence order from the allow rule.
Unknown values
This policy is specifically constructed to compress to a set of queries if 'input.object.owner' and 'input.object.org_owner' are unknown. There is no specific set of rules that will guarantee that this policy has this property, however, there are some tricks. We have tests that enforce this property, so any changes that pass the tests will be okay.
Some general rules to follow:
-
Do not use unknown values in any comprehensions or iterations.
-
Use the unknown values as minimally as possible.
-
Avoid making code branches based on the value of the unknown field.
Unknown values are like a "set" of possible values (which is why rule 1 usually breaks things).
For example, in the org level rules, we calculate the "vote" for all orgs, rather than just the input.object.org_owner. This way, if the org_owner changes, then we don't need to recompute any votes; we already have it for the changed value. This means we don't need branching, because the end result is just a lookup table.