This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
Address TODOs, add deprecation warnings and remove as many U2F code references
as possible.
Existing behavior is kept unaltered: it's still possible to inform Teleport of
old U2F AppIDs and U2F configurations are still silently converted to WebAuthn.
There's no reason to break that, so we don't.
Most server-side references to SecondFactorU2F are removed, but client-side
references remain: this makes it possible to interop newer clients with old
clusters (something else may break, but hopefully not this part).
Closes#10375.
* Introduce Github Actions join support
* Go mod tidy
* run goimports on source files
* Address PR comments
* More PR review comments
* Changes to tests based on PR feedback
* Improve error message in github rule validation
* Add support for SHA
* Add short message describing which fields shouldb be included
Application sessions were previously only logged when launching an application
session via the UI, and not from the `tsh app login` command. This has been
corrected. The AppName and AppURI are now passed in as part of the gRPC
request to the auth server, which is then used to emit the audit event.
* Introduce proto types for ProvisionTokenV3
* Add methods to ProvisionTokenV3 to support ProvisionToken iface
* Start building v3 support into the client
* add support for mashalling and unmarshalling ProvisionTokenV3
* Start unit testing ProvisionTokenV3
* Remove oneof to support yaml marshal/unmarshal
* Client should try V3 methods and fallback to v2
* More tests
* Fix join tests
* Fix integration tests
* Switch integration tests to use v3 spec
* Switch iam tests to use ProvisionTokenV3
* Change ec2 join tests to use V3 tokens
* Fix events tests for V3 token
* support ProvisionTokenV3 within API client events handler
* Explicitly specify JoinMethod
* Tidy up final usage of NewProvisionTokenV2FromSpec in tests
* Improve proto docs on ProvisionTokenV3
* Fix bot join tests
* Clarify error message for invalid join method
* Adjust resource version comment
* Fix comments and return error rather than bool in V2() method
* Catch incompatible conversions case
* Include V2 ProvisionToken in tests and add appropriate DELETE IN notes
* Fix linter warnings/unit test failures
* Use nolint rather than lint:ignore
* Add more DELETE IN notes
* Run goimports on join_ec2_test.go
* Address PR comments from tim.
* Add more deprecation/delete in notices
* Improve godoc comments on checkAndSetDEfaults for provider config
* Simplify implementation by dropping client-ahead compatability
* Add some support for client-ahead but with conversion to v3
* Update code comments to include responsible party
* Rename `Role` to `RoleARN` in EC2 configuration for clarity
* Fix tests for Role -> RoleARN rename
* Move MustCreateProvisionToken out of API and into test packages
* Properly go imports files
U2F devices, like the Yubikey4, fail in a slightly unexpected way when
assertions set UP=false. Handle this edge case and also make sure that failures
when checking for excluded credentials won't break the entire ceremony.
Similar to #15975, but in the registration flow this time.
Repro steps:
1. Register a Yubikey 4
2. Attempt to register a different security key
3. Registration fails with a "user presence required" error
* Leverage inventory control stream for instance version metrics
The version metrics relied on loading all resources in the cache on
each auth server to tally the counts. Unmarshalling items from the
cache can be quite expensive, especially when done every 15s for
all resources in a cluster. While auth servers are typically robust
machines the extra computation for calculating a single metric is
a bit costly and will grow with the cluster size.
Now that the inventory control stream provides a simple and efficient
manner to iterate over all connected instances we can leverage it
instead of the cache to calculate the version counts. This is a slight
breaking change however, as an auth server will now only be reporting
the versions of instances connected directly to itself, and not
versions for all instances within the cluster. This can easily be
worked around by aggregating the metric from all auth servers for
a cluster.
Closes#16429
Prior to this change, each individual service (proxy, app, SSH, db, etc)
would spin up its own uploader service. If you are running multiple
Teleport services in the same process, this means you get multiple
uploaders all looking at the same directory, which can result in
duplicate upload events in the audit log.
Additionally, desktop access has (mistakenly) failed to set up this
service, so desktop sessions would only be uploaded if you happened
to also run some other service in the same process that does spin up
the uploader.
Solve these issues by centralizing the uploader service so that it
runs once per process, and each Teleport service doesn't need to think
about whether or not the service should run.
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
U2F devices, like Yubikey 4, don't seem to respect cancels, locking the program
if a different security key is chosen.
There isn't a clean way to cancel when Cancel() doesn't work, so I've added a
timeout to let the program complete. It's OK for the timeout to be short, it's
not tied to user interaction and by the time we get to it all cancels were
already issued.
Replace role/authPref RequireSessionMFA (bool) with RequireMFAType
(string).
- Add new RequireMFAType constant values with custom boolean marshalling.
- Add RequireMFAType to role and auth preference and deprecate
RequireSessionMFA.
- Add session-mfa override login when hardware_key_touch is enforced.
- Add protobuf enum for RequireMFAType.
- Add support for proto enums in protoc-gen-crd and update Kubernetes
Operator manifests.
Update `duo-labs/webauthn` up to `20220122034320`, which is the latest version
we can get without dipping into dependency hell (`etcd` and `opentelemetry` woes
ensue after [2365c59d9f][1]).
`tstranex` could be dropped for a while now (we moved on to WebAuthn-like
interfaces for mocks). `cfssl` was only imported due to what I assume was an
IDE mishap.
I've elected to keep `fxamacker/cbor`, instead of trying to move to
[webauthncbor][2]. fxamacker is solid, past v0, seems more appropriate for
client-side libs and still backs webauthncbor.
There are no updates for `flynn/hid` and `flynn/u2f`.
Release notes for fxamacker/cbor:
https://github.com/fxamacker/cbor/releases/tag/v2.4.0.
[1]: https://github.com/duo-labs/webauthn/commit/2365c59d9fb666cb5f5b6bbae86910632198a8d4
[2]: https://pkg.go.dev/github.com/duo-labs/webauthn@v0.0.0-20220815211337-00c9fb5711f5/protocol/webauthncbor
* Drop tstranex/u2f dependency
* Drop direct dependency to cloudflare/cfssl
* Update fxamacker/cbor/v2 to v2.4.0
* Update duo-labs/webauthn to 2022-01-22
* Fix: Make sure all credentials are set in the user
* Simplify: Drop now unnecessary AuthenticationSelection copy
While looking up github.com/gokyle/hotp I found some old deprecation warnings
and decided to address them.
* Remove HOTP support
* Update comment on checkOTP
* Remove OTPType
* Remove a few more HOTP references
* Add initial version of installer
* Resolve comments
- Use aws waiters when checking commands
- Use SSMRunRequest rather than passing instances
- General comments
* Resolve comments, (rebase) pass scriptname parameter
This resolves comments regarding running on multiple ec2 instances at
once by adding state to the instances cache to check if the instance
is known about and how far into installation it is
* Revert cache
* Dont cache on non discovery nodes
* Resolve some comments
* Move discovery out to its own serviec
* Add a `discovery_service` section
* Fix messed up conflict merge
* Make starting a standalone discovery agent work
* Resolve comments
* Resolve comments
- use a regular events.Emitter
- resolve a thousand typos :)
* Resolve comments
* resolve comments, fix a bad merge
* Fail when a non ec2 matcher type is configured
* fix lint-go
* Resolve comments
* Resolve comments, add initial test (currently broken)
* Fix log string so only 1 pair of [] are used
* Chunk instances for sending commands
* add 'isInitialized' to watchers
* Add test for chunked discovery, log output
* lints
* explicetly set matcher.Tags to "*":"*" if its unset
Coverts getting values in the `FnCache` from `FnCache.Get` to
`utils.FnCacheGet[T any]`. This allows us to have compile time
guarantees for the type being loaded, which allows us to remove
some of the manual type checking required of callers today.
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
* Propagate tctl --insecure flag
* Move ExitCodeError into common package
* Add TryRun helper func to tctl for testing
* Return error instead of exiting directly
* Add test for tctl insecure with profile
* Remove debug println
* Update want err message
* Remove unnecessary errors.Is implementation for ExitCodeError
* Update error message for cert error in test
* Fix flaky TestHandlerConnectionUpgrade
* Test that TryRun error is assignable to ExitCodeError
* gofmt
* Fix import order
Co-authored-by: STeve Huang <xin.huang@goteleport.com>
* added check that blocks enterprise Github organizations using external SSO from using Github auth connectors on Teleport OSS
* added details to docs and changelog
* move check from init to on login
* cache org sso status; address small pieces of feedback
* use utils.FnCache instead of 3rd party cache
* addressed feedback
* addressed feedback
* revised docs
* addressed more feedback
* remove notes in changelog
* removed unused interface
This PR introduces Auth Server CRUD API for managing the lifecycle of dynamic `kube_cluster` resources.
Currently, `kube_cluster` resources cannot be manipulated without using the API, nor are they being listened to. It will be part of future developments.
# Future work
- The `kubernetes_service` will have to watch these `kube_cluster` resources.
- The discovery service will have to create these resources.
Resource manipulation via `tctl` will be introduced later and is not part of the scope.
Part of #16135, #13376
Related to #12048
* Add `where` predicate and Machine ID support to SSH host certificates
This adds `where` RBAC predicate support to `host_cert`
pseudo-resources, and adds support for requesting host certs to
Machine ID (with or without these RBAC predicates).
More specifically, this:
* Makes 3 new set functions available during `where` predicate
evaluation: `all_equal`, `all_end_with`, and `is_subset`. These
make it possible to examine the `host_cert.principals` field.
* Adds a new optional `host_cert` field to the standard `where`
context. It's only ever set specifically in `GenerateHostCert()`.
* Sets a custom context in `GenerateHostCert()` to pass along host
cert parameters.
* Makes current config available to tbot config templates
* Adds a new `ssh_host_cert` config template to Machine ID
* Refactors `tbot/config` tests to use shared mock bot / auth
implementations.
Host certs can be requested in Machine ID by adding a new config
template:
```yaml
destinations:
- directory: /opt/machine-id
configs:
- ssh_host_cert:
principals: [foo.example.com]
```
Bots will need a role like the following:
```yaml
kind: role
metadata:
name: hostcert-bot
version: v5
spec:
allow:
rules:
- resources:
- host_cert
verbs:
- create
where: 'is_subset(host_cert.principals, "foo.example.com")'
```
See the RFD for more information:
https://github.com/gravitational/teleport/blob/master/rfd/0083-machine-id-host-certs.md
* Fix lint failure
* Improve doc comments on the new predicate functions.
* Address review feedback
* Address review feedback
- Remove `AuthenticatedUserClientFromIdentity`'s unnecessary
AuthServer param
- Add missing doc comments
- Pull out `getTestIdent` from the Kubernetes template test
- Move various test helpers into their own file
- Unexport new predicate functions
* Add Default Allow Rules for new resources
When adding a new resource, we usually also set some default access
rules on the preset Roles (editor, access, auditor).
However, for existing clusters those new rules are not applied when we
start the cluster with the new teleport version.
This then requires manual intervention for setting up access to those
resources.
This PR adds a new mechanism to remove the need for manual intervention.
For each of the preset rules, we'll have default rules.
When the server starts, it tries to create the preset roles but if they
exist, instead of failing we will:
- check if the resource exists either in allow or deny rules
- if so, then we don't do anything else
- add the default rule
This assumes it's ok to re-create the default rules when the admin
removes those resources from the allow/deny rule set.
* add log when adding default allow rules
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
Tracing handlers consume a large amount of memory over time, which
makes using one per route suboptimal (See https://github.com/open-telemetry/opentelemetry-go-contrib/issues/2413).
Wrapping all `http.Server` handlers with a tracing handler provides
the same tracing abilities for a fraction of the memory usage. A new
span formatter is added which allows servers to still contain the
request span in the span name.
Noah Stride