* go get google.golang.org/api
go get: upgraded cloud.google.com/go v0.60.0 => v0.100.2
go get: upgraded github.com/golang/snappy v0.0.1 => v0.0.3
go get: upgraded github.com/googleapis/gax-go/v2 v2.0.5 => v2.1.1
go get: upgraded go.opencensus.io v0.22.5 => v0.23.0
go get: upgraded golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d => v0.0.0-20211104180415-d3ed0bb246c8
go get: upgraded google.golang.org/api v0.29.0 => v0.65.0
* Optionally fetch transitive groups in the Google OIDC connector
* Refactor the google workspace parts of the OIDC code
* Further refactoring
This undoes the user account impersonation changes, and always requires
an admin account again.
* Test coverage
* Address review comments
* Minor refactor and name changes
* Allow domain filtering, tests now bypass addGoogleWorkspaceClaims
* Update `OIDCConnectorV2` to `OIDCConnectorV3`
* Backwards compatibility for OIDCConnector v2
This also removes the extra boolean flag that was added previously.
* Update e-ref
Enterprise builds will break unless gravitational/teleport.e#385
is included.
* Remove JSON schema validation
Removing JSON schema validation from all resource unmarshalers.
--- what JSON schema gets us
Looking at the JSON schema spec and our usage, here are the supposed benefits:
- type validation - make sure incoming data uses the right types for the right fields
- required fields - make sure that mandatory fields are set
- defaulting - set defaults for fields
- documentation - schema definition for our API objects
Note that it does _not_ do:
- fail on unknown fields in data
- fail on a required field with an empty value
--- what replaces it
Based on the above, it may seem like JSON schema provides value.
But it's not the case, let's break it down one by one:
- type validation - unmarshaling JSON into a typed Go struct does this
- required fields - only checks that the field was provided, doesn't actually check that a value is set (e.g. `"name": ""` will pass the `required` check)
- so it's pretty useless for any real validation
- and we already have a separate place for proper validation - `CheckAndSetDefaults` methods
- defaulting - done in `CheckAndSetDefaults` methods
- `Version` is the only annoying field, had to add it in a bunch of objects
- documentation - protobuf definitions are the source of truth for our API schema
--- the benefits
- performance - schema validation does a few rounds of `json.Marshal/Unmarshal` in addition to actual validation; now we simply skip all that
- maintenance - no need to keep protobuf and JSON schema definitions in sync anymore
- creating new API objects - one error-prone step removed
- (future) fewer dependencies - we can _almost_ remove the Go libraries for schema validation (one transient dependency keeping them around)
* Remove services.SkipValidation
No more JSON schema validation so this option is a noop.
Various improvements related to extending the dynamic access
API, including:
- Support for users with no statically defined roles.
- Unify trait mapping logic (e.g. claims_to_roles) across
the connector types.
- Support for matcher syntax and claims_to_roles mappings when
configuring which roles a user is able to request.
- Allow tsh or the web UI to automatically generate wildcard
access requests when dictated by role configuration.
- Allow RBAC configuration to attach annotations to pending
access requests which can be consumed by plugins.
- Allow plugins to attach annotations to approvals/denials
which appear in the audit log, and may also be looked up
later to determine additional info about a resolution.
- Support prompts, request reasons, and approval/denial
reasons for access requests.
Fixes#3604
This commit adds support for cluster_labels
role parameter limiting access to remote clusters by label.
New tctl update rc provides interface to set labels on remote clusters.
Consider two clusers, `one` - root and `remote` - leaf.
```bash
$ tsh clusters
Cluster Name Status
------------ ------
one online
two online
```
Create the trusted cluster join token with labels:
```bash
$ tctl tokens add --type=trusted_cluster --labels=env=prod
```
Every cluster joined using this token will inherit env:prod labels.
Alternatively, update remote cluster labels by modifying
`rc` command. Letting remote clusters to propagate their labels
creates a problem of rogue clusters updating their labels to bad values.
Instead, administrator of root cluster control the labels
using remote clusters API without fear of override:
```bash
$ tctl get rc
kind: remote_cluster
metadata:
name: two
status:
connection: online
last_heartbeat: "2020-09-14T03:13:59.35518164Z"
version: v3
```
```bash
$ tctl update rc/two --set-labels=env=prod
cluster two has been updated
```
```bash
$ tctl get rc
kind: remote_cluster
metadata:
labels:
env: prod
name: two
status:
connection: online
last_heartbeat: "2020-09-14T03:13:59.35518164Z"
```
Update the role to deny access to prod env:
```yaml
kind: role
metadata:
name: dev
spec:
allow:
logins: [root]
node_labels:
'*': '*'
# Cluster labels control what clusters user can connect to. The wildcard ('*') means
# any cluster. If no role in the role set is using labels and cluster is not labeled,
# the cluster labels check is not applied. Otherwise, cluster labels are always enforced.
# This makes the feature backwards-compatible.
cluster_labels:
'env': 'staging'
deny:
# cluster labels control what clusters user can connect to. The wildcard ('*') means
# any cluster. By default none is set in deny rules to preserve backwards compatibility
cluster_labels:
'env': 'prod'
```
```bash
$ tctl create -f dev.yaml
```
Cluster two is now invisible to user with `dev` role.
```bash
$ tsh clusters
Cluster Name Status
------------ ------
one online
```
Spring cleaning!
A very mechanical cleanup using several linters (unused, deadcode,
structcheck). Build and tests still pass so no behavior should be
affected.
This commit adds support for custom OIDC prompt values.
Read about possible prompt values here:
https://openid.net/specs/openid-connect-core-1_0.html#AuthRequest
Three cases are possible:
* Prompt value is not set, this defaults to
OIDC prompt value to select_account value to preserve backwards
compatibility.
```yaml
kind: oidc
version: v2
metadata:
name: connector
spec:
prompt: 'login consent'
```
* Prompt value is set to empty string, it will be omitted
from the auth request.
```yaml
kind: oidc
version: v2
metadata:
name: connector
spec:
prompt: ''
```
* Prompt value is set to non empty string, it will be included
in the auth request as is.
```yaml
kind: oidc
version: v2
metadata:
name: connector
spec:
prompt: 'login consent'
```
Tested with Auth0 OIDC connector on teleport 4.2 enterprise.
This commit expands the usage of the caching layer
for auth server API:
* Introduces in-memory cache that is used to serve all
Auth server API requests. This is done to achieve scalability
on 10K+ node clusters, where each node fetches certificate authorities,
roles, users and join tokens. It is not possible to scale
DynamoDB backend or other backends on 10K reads per seconds
on a single shard or partition. The solution is to introduce
an in-memory cache of the backend state that is always used
for reads.
* In-memory cache has been expanded to support all resources
required by the auth server.
* Experimental `tctl top` command has been introduced to display
common single node metrics.
Replace SQLite Memory Backend with BTree
SQLite in memory backend was suffering from
high tail latencies under load (up to 8 seconds
in 99.9%-ile on load configurations).
This commit replaces the SQLite memory caching
backend with in-memory BTree backend that
brought down tail latencies to 2 seconds (99.9%-ile)
and brought overall performance improvement.
This commit introduces several key changes to
Teleport backend and API infrastructure
in order to achieve scalability improvements
on 10K+ node deployments.
Events and plain keyspace
--------------------------
New backend interface supports events,
pagination and range queries
and moves away from buckets to
plain keyspace, what better aligns
with DynamoDB and Etcd featuring similar
interfaces.
All backend implementations are
exposing Events API, allowing
multiple subscribers to consume the same
event stream and avoid polling database.
Replacing BoltDB, Dir with SQLite
-------------------------------
BoltDB backend does not support
having two processes access the database at the
same time. This prevented Teleport
using BoltDB backend to be live reloaded.
SQLite supports reads/writes by multiple
processes and makes Dir backend obsolete
as SQLite is more efficient on larger collections,
supports transactions and can detect data
corruption.
Teleport automatically migrates data from
Bolt and Dir backends into SQLite.
GRPC API and protobuf resources
-------------------------------
GRPC API has been introduced for
the auth server. The auth server now serves both GRPC
and JSON-HTTP API on the same TLS socket and uses
the same client certificate authentication.
All future API methods should use GRPC and HTTP-JSON
API is considered obsolete.
In addition to that some resources like
Server and CertificateAuthority are now
generated from protobuf service specifications in
a way that is fully backward compatible with
original JSON spec and schema, so the same resource
can be encoded and decoded from JSON, YAML
and protobuf.
All models should be refactored
into new proto specification over time.
Streaming presence service
--------------------------
In order to cut bandwidth, nodes
are sending full updates only when changes
to labels or spec have occured, otherwise
new light-weight GRPC keep alive updates are sent
over to the presence service, reducing
bandwidth usage on multi-node deployments.
In addition to that nodes are no longer polling
auth server for certificate authority rotation
updates, instead they subscribe to event updates
to detect updates as soon as they happen.
This is a new API, so the errors are inevitable,
that's why polling is still done, but
on a way slower rate.
This commit adds two extensions to template variables
in roles and adds support for regular expressions
and group captures in role mapping of trusted clusters.
1. Roles node_labels can expand variables from traits:
allow:
node_labels:
'{{external.key}}': '{{external.val}}'
deny:
node_labels:
'{{external.key}}': '{{external.val}}'
If traits variable is not found, label key pair in allow or
deny rule will be set to empty key or value, so if 'external.val'
trait is missing, the resulting role will not match
allow or deny rule:
allow:
node_labels:
'': 'val'
deny:
node_labels:
'': 'val'
Same thing will happen for missing value:
allow:
node_labels:
'key': ''
deny:
node_labels:
'key': ''
2. Trusted cluster role mapping can now
support advanced expressions:
a. Glob values will math any string, including
empty one
role_map:
- remote: 'cluster-*'
local: [clusteradmin]
a. Regular expression syntax is supported:
Syntax: https://github.com/google/re2/wiki/Syntax
Brackets can be used as a capture group and referred
to with expand variable:
role_map:
- remote: '^clusteradmin-(.*)$'
local: [unprivileged-$1]
Will map incoming role 'clusteradmin-account-1' to 'guest-account-1'.
3. Same regular expression syntax is supported for SAML and OIDC
mappings:
a. Glob matches of values instead of static matches:
claims_to_roles:
- {claim: "roles", value: "gravitational/*", roles: ["clusteradmin"]}
b. Regexp matches with subgroup expands:
attributes_to_roles:
- {name: "roles", value: "^gravitational/(.*)$", roles: ["cluster-$1"]}
This was fixed running the `misspell` linter in fix mode using
`gometalinter`. The exact command I ran was :
```
gometalinter --vendor --disable-all -E misspell --linter='misspell:misspell -w {path}:^(?P<path>.*?\.go):(?P<line>\d+):(?P<col>\d+):\s*(?P<message>.*)$' ./...
```
Some typo were fixed by hand on top of it.
Brian Joerger