* Move GCP cloud clients into a separate interface
* Add azure mysql/postgresql flex server support
* Add teleport db configure create support for azure postgres/mysql flex servers
* discover both single and flexi server with 'mysql' and 'postgres' db types.
* go doc gcp interface
* Add is_flexi_server to database Azure config
* Add a new db engine
* Add tests for new engine
* Update tsh db subcommands
* Refactor error message and suggestions for unsupported tsh commands
* Add dynamodb to test plan
* Add AWS external ID to db config and update protos
When the AWS Account ID is provided, we must check that it is a valid
one.
AWS Account ID is a well documented field: 12-digit string
This check is meant to prevent typos when creating Databases mainly from
the Web UI in the context of Teleport Discover.
This PR presents a watcher for automatic `kube_cluster` discovery for GCP GKE clusters. Given an identity with access to the GCP cloud, the auto-discovery service will scan the cloud and register all clusters available in Kubernetes Engine.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived and generated through Google OAuth2 associated with the GCP Service Account configured for the Kubernetes Service.
GCP's Service Account must have the following role def attached:
```yaml
description: 'GKE Auto-Discovery'
includedPermissions:
- container.clusters.impersonate
- container.clusters.get
- container.clusters.list
- container.pods.get
- container.selfSubjectAccessReviews.create
- container.selfSubjectRulesReviews.create
name: projects/{projectID}/roles/GKEKubernetesAutoDisc
stage: GA
title: GKEKubernetesAutoDisc
```
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633, #14991
This PR presents a watcher for automatic `kube_cluster` discovery for AWS EKS clusters. Given a user with access to the AWS cloud, the auto-discovery service will scan the cloud and register all clusters available in EKS.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived - TTL is 15m - and are generated through the IAM role associated with the process running the `kubernetes_service`.
All clusters must necessarily allow access to the Teleport service by configuring the IAM role in the `configmap/aws-auth` present in the `kube-system` namespace. Without this access, the clusters will not be served correctly.
# Future work
- Support GCP auto-discovery
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633
When testing the "not available" case, this test closed the server
and waited for the AWS SDK to make several attempts before finally
giving up.
Instead of shutting down the server, return an error. This triggers
the same error handling branches, but doesn't require multiple
attempts by the SDK.
Before:
$ go test -run TestEC2IsInstanceMetadataAvailable ./lib/cloud/aws -count=1
ok github.com/gravitational/teleport/lib/cloud/aws 4.272s
After:
$ go test -run TestEC2IsInstanceMetadataAvailable ./lib/cloud/aws -count=1
ok github.com/gravitational/teleport/lib/cloud/aws 0.262s
This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
* Add initial version of installer
* Resolve comments
- Use aws waiters when checking commands
- Use SSMRunRequest rather than passing instances
- General comments
* Resolve comments, (rebase) pass scriptname parameter
This resolves comments regarding running on multiple ec2 instances at
once by adding state to the instances cache to check if the instance
is known about and how far into installation it is
* Revert cache
* Dont cache on non discovery nodes
* Resolve some comments
* Move discovery out to its own serviec
* Add a `discovery_service` section
* Fix messed up conflict merge
* Make starting a standalone discovery agent work
* Resolve comments
* Resolve comments
- use a regular events.Emitter
- resolve a thousand typos :)
* Resolve comments
* resolve comments, fix a bad merge
* Fail when a non ec2 matcher type is configured
* fix lint-go
* Resolve comments
* Resolve comments, add initial test (currently broken)
* Fix log string so only 1 pair of [] are used
* Chunk instances for sending commands
* add 'isInitialized' to watchers
* Add test for chunked discovery, log output
* lints
* explicetly set matcher.Tags to "*":"*" if its unset
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
* Add Azure auto-discovery configuration fields
* Init databases if azure matchers are in config
* Use AzureMatchers in db service
* Use all azure subscriptions/resource groups if omitted in matcher
* Add azure config tests
* Go mod tidy to update dependencies
* Add azure response error conversion
* Check for azure access denied and give a helpful error message
* Add azure subscriptions api
* Add azure mysql/postgresql api and wrappers
* Test generic db server for azure
* Make server properties its own type
* Convert server types manually instead of via json
* Move server list method selection logic out of api client
* Update azure db server tests
* Fixup merge
* Update comments
* Update more comments and remove junk code
* Move all azure api into lib/cloud/azure
* Update state and version checks
* Convert Azure DB Server into database
* Add mutex to subscription client for caching, just in case
* Test database conversion from azure db server
* Refactor common code
* Add azure matchers to watch config
* Remove unused imports
* Use common max pages in memorydb users
* Add azure clients to cloud clients
* Add azure fetchers to watcher
* Skip fetcher NotFound errors
* Add azure watcher tests
* Fix test comment
* Add wildcard region matching
* Remove redundant import
* Update lib/srv/db/cloud/watchers/azure.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Move db server listing logic into separate func
* Refactor some db conversion code
* Use local var to cache sub IDs
* Rename azure subscription client
* Rename azure subscription client and remove sub ID caching
* Move azure cloud clients into separate embedded interface
* Update watcher test for 'NotFound' handling
* Embed azureClients struct
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/subscriptions_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Rename azure subscription client and remove sub ID caching
* Add reference links for azure db ports
* Move indirect dep into group
* Wrap all converted azure response errors
* Remove unreachable panic
* Godoc DBServer
* Remove maxPages arg to azure client funcs
* Gofmt
* Spacing between copyright and package
* import order
* Update lib/srv/db/cloud/watchers/azure.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Fix import order
* Refactor region matching
* Fix usage of azure clients to remove max pages arg
* Query azure subscriptions in fetcher.Get if matching wildcard subscription
* Fix azure matcher construction
* Test azure fetcher discovers new subscriptions
* Remove unused test helper func
* Set azure database resource id metadata
* Log errors when fetching from azure subscription fails
* Use more readable protocol in azure db description
* Get azure mysql engine version from labels
* Update lib/srv/db/cloud/watchers/watcher_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/srv/db/cloud/watchers/watcher_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Add Azure auto-discovery configuration fields
* Init databases if azure matchers are in config
* Use AzureMatchers in db service
* Use all azure subscriptions/resource groups if omitted in matcher
* Add azure config tests
* Go mod tidy to update dependencies
* Add azure response error conversion
* Check for azure access denied and give a helpful error message
* Add azure subscriptions api
* Add azure mysql/postgresql api and wrappers
* Test generic db server for azure
* Make server properties its own type
* Convert server types manually instead of via json
* Move server list method selection logic out of api client
* Update azure db server tests
* Fixup merge
* Update comments
* Update more comments and remove junk code
* Move all azure api into lib/cloud/azure
* Update state and version checks
* Add mutex to subscription client for caching, just in case
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/subscriptions_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Rename azure subscription client and remove sub ID caching
* Add reference links for azure db ports
* Move indirect dep into group
* Wrap all converted azure response errors
* Remove unreachable panic
* Godoc DBServer
* Remove maxPages arg to azure client funcs
* Gofmt
* Spacing between copyright and package
* import order
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Add an AWS EC2 instance fetcher
* Resolve comments
- Remove leftover VPCID stuff
- Document public functions
- Return a notfound when there are no nodes fetched instead of nil nil
* Resolve comments
* Fix fileconf_test.go
Andrew Burke