This PR presents a watcher for automatic `kube_cluster` discovery for AWS EKS clusters. Given a user with access to the AWS cloud, the auto-discovery service will scan the cloud and register all clusters available in EKS.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived - TTL is 15m - and are generated through the IAM role associated with the process running the `kubernetes_service`.
All clusters must necessarily allow access to the Teleport service by configuring the IAM role in the `configmap/aws-auth` present in the `kube-system` namespace. Without this access, the clusters will not be served correctly.
# Future work
- Support GCP auto-discovery
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633
In “tsh login” show only alerts with on-login label.
In “tsh status” show only alerts with “high” severity. Which license warning should match.
In all “tctl” commands show only alerts with “high” severity.
Moves from github.com/golang/protobuf protoc-gen-go plugin to google.golang.org/
plugins.
This change was a long-time coming, but is now possible to do since our
dependencies are up-to-date.
* Move away from deprecated protoc-gen-go plugin
* Embed unimplemented server in handler.Handler
* Embed unimplemented server in multiplexer_test.go
* Update generated protos
* access requests for teleterm
* removed unused imports and named returns
* remove comment
* using timestamppb instead of string for access requests
* updated proto with some more comments
* updated protos with comments
* using clusterClient, comments, and moving validation to daemon for access request delete
* separated GetAccessRequests into separate RPCs
* protobuf updates
* moved requestid check before resolving cluster
* fullstops in comments
* used standard access_request_id through rpc messages
* updated protofiles
* updated daemon service types to match grpc
* added kube advanced search support
* updated protos for kubes in access requests
* testing tag build
* fix detached head
* new tag build
* protobuf update
* lint fixes
* allow drone windows Connect build to include webapps.e
* protobuf files
* remove drone changes and updated comment
* proto changes with comment fixes and changed field order
* protobuf updates
Add definitions for Device Trust RPCs.
DeviceTrustService is fairly isolated from other services, so it is generated
using plain `protoc-gen-go` instead of Gogo.
teleport.e#514
* Add Device Trust proto definitions
* Allow proto generation without gogo
* Drop Gogo from lib/multiplexer protos
* Update generated protos
* Tidy modules
This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
This PR implements the Kubernetes Tester for the ConnectionDiagnostic feature.
The goal is to validate if the Kubernetes cluster is available and if the user can access it.
A series of checks are performed against Teleport using a Kubernetes Client populated with the user's certificates for accessing the desired cluster.
Currently, the checks are:
- Kubernetes Cluster is up and running
- User's role specifies at least one `kubernetes_groups` or `kubernetes_users`
- User's roles allow RBAC access to the cluster: `kubernetes_labels` match.
- Values specified under `kubernetes_groups` or `kubernetes_users` exist in the cluster and allow the user to list pods in the desired namespace.
Testing:
Post: `/v1/webapi/sites/<teleport-cluster>/diagnostics/connections`
```json
{
"resource_kind": "kube_cluster",
"resource_name": "<cluster_name>",
"kubernetes_namespace": "default"
}
```
Closes#16824
This fixes a goroutine leak caused by the various connect methods
getting stuck waiting to push their error in `errChan` after the
function exited with a success.
Address TODOs, add deprecation warnings and remove as many U2F code references
as possible.
Existing behavior is kept unaltered: it's still possible to inform Teleport of
old U2F AppIDs and U2F configurations are still silently converted to WebAuthn.
There's no reason to break that, so we don't.
Most server-side references to SecondFactorU2F are removed, but client-side
references remain: this makes it possible to interop newer clients with old
clusters (something else may break, but hopefully not this part).
Closes#10375.
* Introduce Github Actions join support
* Go mod tidy
* run goimports on source files
* Address PR comments
* More PR review comments
* Changes to tests based on PR feedback
* Improve error message in github rule validation
* Add support for SHA
* Add short message describing which fields shouldb be included
Do another large batch of dependency updates.
Most updates are around minor versions, so _theoretically_ safe. The following
v0.x updates draw attention:
* cloud.google.com/go/iam (likely safe, as Google is largely trunk-based)
* github.com/Microsoft/go-winio
Notable exceptions are k8s-related modules, which are harder to update for
various reasons.
* Update Go dependencies
* Fix lib/srv/app/aws/endpoints_test.go
Application sessions were previously only logged when launching an application
session via the UI, and not from the `tsh app login` command. This has been
corrected. The AppName and AppURI are now passed in as part of the gRPC
request to the auth server, which is then used to emit the audit event.
Update various "assorted" dependencies, that either are used throughout
(logging, testing), are mostly algorithmic or otherwise difficult to pinpoint
"ownership".
I've dropped a couple of deprecated/mostly meaningless dependencies. I suspect
`github.com/mitchellh/mapstructure` could be dropped to, but I didn't look
further now.
* Drop dependency on Clever/go-utils
* Drop dependency on github.com/pkg/errors
* Update various Go dependencies
* Use CompareAndSwap instead of CAS (uber/atomic)
* Add a comment after replaced dependencies
* Introduce proto types for ProvisionTokenV3
* Add methods to ProvisionTokenV3 to support ProvisionToken iface
* Start building v3 support into the client
* add support for mashalling and unmarshalling ProvisionTokenV3
* Start unit testing ProvisionTokenV3
* Remove oneof to support yaml marshal/unmarshal
* Client should try V3 methods and fallback to v2
* More tests
* Fix join tests
* Fix integration tests
* Switch integration tests to use v3 spec
* Switch iam tests to use ProvisionTokenV3
* Change ec2 join tests to use V3 tokens
* Fix events tests for V3 token
* support ProvisionTokenV3 within API client events handler
* Explicitly specify JoinMethod
* Tidy up final usage of NewProvisionTokenV2FromSpec in tests
* Improve proto docs on ProvisionTokenV3
* Fix bot join tests
* Clarify error message for invalid join method
* Adjust resource version comment
* Fix comments and return error rather than bool in V2() method
* Catch incompatible conversions case
* Include V2 ProvisionToken in tests and add appropriate DELETE IN notes
* Fix linter warnings/unit test failures
* Use nolint rather than lint:ignore
* Add more DELETE IN notes
* Run goimports on join_ec2_test.go
* Address PR comments from tim.
* Add more deprecation/delete in notices
* Improve godoc comments on checkAndSetDEfaults for provider config
* Simplify implementation by dropping client-ahead compatability
* Add some support for client-ahead but with conversion to v3
* Update code comments to include responsible party
* Rename `Role` to `RoleARN` in EC2 configuration for clarity
* Fix tests for Role -> RoleARN rename
* Move MustCreateProvisionToken out of API and into test packages
* Properly go imports files
By only providing the tunnel address from the `reversetunnel.Resolver`
callers would still need to lookup the proxy listener mode to determine
how to dial the address. This results in sending a request to
`/webapi/find` once by the resolver to get the tunnel address and then
a second request to `/webapi/find` by users of the `Resolver` to determine
the proxy listener mode. Propagating the listener mode along with the
tunnel address by the `Resolver` ensures only one `/webapi/find` call
is needed.
This is especially impactful because the `reversetunnel.TunnelAuthDialer`
which is used by the auth http client would do this everytime the
`http.Client` connection pool was empty. When the `http.Client` needed
to dial the auth server it was incurring the additional roundtrip to the
proxy.
Prior to this change, each individual service (proxy, app, SSH, db, etc)
would spin up its own uploader service. If you are running multiple
Teleport services in the same process, this means you get multiple
uploaders all looking at the same directory, which can result in
duplicate upload events in the audit log.
Additionally, desktop access has (mistakenly) failed to set up this
service, so desktop sessions would only be uploaded if you happened
to also run some other service in the same process that does spin up
the uploader.
Solve these issues by centralizing the uploader service so that it
runs once per process, and each Teleport service doesn't need to think
about whether or not the service should run.
Forrest Marshall