Prior to the NodeWatcher being successfully initialized, or when
it became stale any calls to `GetNodes` would return no data or
stale data. This behavior results in the problems described in
#22672.
The NodeWatcher will now fallback to pulling data from the cache
in the event that it has no data or stale data when callers attempt
to retrieve nodes. This should reduce the time that it takes the
a new Proxy or a Proxy with intermittent connectivity issues to
Auth be able to route to nodes.
The impact of this change should be noticeable in tests that
create a Proxy and several Nodes and then immediately try to
open a session on the Nodes.
Closes#22672
* Add agentless installer
* Resolve comments
* Resolve comments
* Use GetCertAuthorities locally
* Try to get IMDS hostname
* Try get imds hostname first
This seems to be how its implemented for non-agentless nodes
* Use FIPS cipher suites
* use the openssh ca, resolve comments
* write keys to /etc/teleport/agentless by default
* Resolve comment
* lints
* test fixes
* Move GCP cloud clients into a separate interface
* Add azure mysql/postgresql flex server support
* Add teleport db configure create support for azure postgres/mysql flex servers
* discover both single and flexi server with 'mysql' and 'postgres' db types.
* go doc gcp interface
* Add is_flexi_server to database Azure config
When the AWS Account ID is provided, we must check that it is a valid
one.
AWS Account ID is a well documented field: 12-digit string
This check is meant to prevent typos when creating Databases mainly from
the Web UI in the context of Teleport Discover.
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
This PR presents a watcher for automatic `kube_cluster` discovery for GCP GKE clusters. Given an identity with access to the GCP cloud, the auto-discovery service will scan the cloud and register all clusters available in Kubernetes Engine.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived and generated through Google OAuth2 associated with the GCP Service Account configured for the Kubernetes Service.
GCP's Service Account must have the following role def attached:
```yaml
description: 'GKE Auto-Discovery'
includedPermissions:
- container.clusters.impersonate
- container.clusters.get
- container.clusters.list
- container.pods.get
- container.selfSubjectAccessReviews.create
- container.selfSubjectRulesReviews.create
name: projects/{projectID}/roles/GKEKubernetesAutoDisc
stage: GA
title: GKEKubernetesAutoDisc
```
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633, #14991
This PR presents a watcher for automatic `kube_cluster` discovery for AWS EKS clusters. Given a user with access to the AWS cloud, the auto-discovery service will scan the cloud and register all clusters available in EKS.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived - TTL is 15m - and are generated through the IAM role associated with the process running the `kubernetes_service`.
All clusters must necessarily allow access to the Teleport service by configuring the IAM role in the `configmap/aws-auth` present in the `kube-system` namespace. Without this access, the clusters will not be served correctly.
# Future work
- Support GCP auto-discovery
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633
This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
* Add initial version of installer
* Resolve comments
- Use aws waiters when checking commands
- Use SSMRunRequest rather than passing instances
- General comments
* Resolve comments, (rebase) pass scriptname parameter
This resolves comments regarding running on multiple ec2 instances at
once by adding state to the instances cache to check if the instance
is known about and how far into installation it is
* Revert cache
* Dont cache on non discovery nodes
* Resolve some comments
* Move discovery out to its own serviec
* Add a `discovery_service` section
* Fix messed up conflict merge
* Make starting a standalone discovery agent work
* Resolve comments
* Resolve comments
- use a regular events.Emitter
- resolve a thousand typos :)
* Resolve comments
* resolve comments, fix a bad merge
* Fail when a non ec2 matcher type is configured
* fix lint-go
* Resolve comments
* Resolve comments, add initial test (currently broken)
* Fix log string so only 1 pair of [] are used
* Chunk instances for sending commands
* add 'isInitialized' to watchers
* Add test for chunked discovery, log output
* lints
* explicetly set matcher.Tags to "*":"*" if its unset
rosstimothy