* Move configuration from lib/service to lib/service/servicecfg
The new servicecfg package will hold only configuration for services.
This will allow other packages (like tctl and tsh) to depend on
servicecfg without pulling in all of lib/service (which has a number
of platform-specific details).
This is the first step towards being able to build tctl for Windows.
* Move PAM and BPF config into servicecfg
This breaks a compile-time dependency on BPF/PAM for tctl.
This PR introduces a mechanism to control access to Pods in Kubernetes clusters based on access criteria defined in user Roles or Resource Access Requests.
New `Role` version - `v6` - introduced a new field `kubernetes_resources` that allows admins to restrict access to a subset of cluster resources based on the configuration provided.
This PR implements the resource control for `pods` and allows
```yaml
kubernetes_resources:
- kind: pod
name: nginx-*
namespace: default
- kind: pod
name: *
namespace: dev
```
It also implements the logic blocks for supporting Resource Access Requests to Kubernetes Pods. The tsh boilerplate is not included in this PR and will be included in future PRs.
An example of usage:
```
tsh request search --kind pod --kube-cluster=minikube
Name Namespace Labels Resource ID
--------------------------------- --------- -------------------------------------- ---------------------------------------------
nginx-deployment-6595874d85-6j2zm default app=nginx,pod-template-hash=6595874d85 /tele.teleport.local/pod/minikube/default/...
nginx-deployment-6595874d85-7xgmb default app=nginx,pod-template-hash=6595874d85 /tele.teleport.local/pod/minikube/default/...
nginx-deployment-6595874d85-c4kz8 default app=nginx,pod-template-hash=6595874d85 /tele.teleport.local/pod/minikube/default/...
test default run=test /tele.teleport.local/pod/minikube/default/...
To request access to these resources, run
> tsh request create --resource /tele.teleport.local/pod/minikube/default/nginx-deployment-6595874d85-6j2zm --resource /tele.teleport.local/pod/minikube/default/nginx-deployment-6595874d85-7xgmb --resource /tele.teleport.local/pod/minikube/default/nginx-deployment-6595874d85-c4kz8 --resource /tele.teleport.local/pod/minikube/default/test \
--reason <request reason>
```
Part of #18434
* Enable nolintlint linter
* Fix nolint comments in the api package
* Fix RDP client comment
* Address review comment
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Allow unused for nolintlint linter
* Remove redundant casting
* Add comment on why allowed unused is enabled
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
This PR presents a watcher for automatic `kube_cluster` discovery for Azure AKS clusters. Given a user with access to the Azure cloud, the auto-discovery service will scan the cloud and register all clusters available in AKS .
Once the discovery service creates a `kube_cluster` in Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster depend on the different AKS clusters configurations:
# Authentication
## Local Accounts
If the AKS cluster auth is based on local accounts created during the provisioning phase of the cluster, the agent will use the [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint.
This endpoint returns a `kubeconfig` fully populated with user credentials that Teleport can use to access the cluster.
## AZ Active Directory
When AZ active directory integration is enabled, Azure allows login with AD users. Azure forces the login to happen with dynamic short-lived user tokens. These tokens are generated by calling `credentials.GetToken` with a fixed Scope: `6dae42f8-4368-4678-94ff-3960e28e3630` and with the cluster's `tenant_id`. The token contains the user details as well as `group_ids` to match with authorization rules.
```go
// getAzureToken generates an authentication token for clusters with AD enabled.
func (a *aKSClient) getAzureToken(ctx context.Context, tentantID string, clientCfg *rest.Config) (time.Time, error) {
const (
azureManagedClusterScope = "6dae42f8-4368-4678-94ff-3960e28e3630"
)
cred, err := a.azIdentity(&azidentity.DefaultAzureCredentialOptions{
TenantID: tentantID,
})
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
cliAccessToken, err := cred.GetToken(ctx, policy.TokenRequestOptions{
// azureManagedClusterScope is a fixed scope that identifies azure AKS managed clusters.
Scopes: []string{azureManagedClusterScope},
},
)
if err != nil {
return time.Time{}, trace.Wrap(ConvertResponseError(err))
}
// reset the old exec provider credentials
clientCfg.ExecProvider = nil
clientCfg.BearerToken = cliAccessToken.Token
return cliAccessToken.ExpiresOn, nil
}
```
# Authorization
## Local Accounts
The [`aks:ListClusterUserCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-user-credentials?tabs=HTTP) endpoint returns credentials with enough permissions for Teleport to enroll the cluster.
## AZ AD
### Azure RBAC
When Azure RBAC mode is enabled, the cluster authorization is based on rules specified in the Azure Identity permissions.
The AZ group associated with the AZ identity the Teleport Process is running has to define the following permissions:
```json
{
"Name": "AKS Teleport Discovery Permissions",
"Description": "Required permissions for Teleport auto-discovery.",
"Actions": [],
"NotActions": [],
"DataActions": [
"Microsoft.ContainerService/managedClusters/pods/read",
"Microsoft.ContainerService/managedClusters/users/impersonate/action",
"Microsoft.ContainerService/managedClusters/groups/impersonate/action",
"Microsoft.ContainerService/managedClusters/serviceaccounts/impersonate/action",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectaccessreviews/write",
"Microsoft.ContainerService/managedClusters/authorization.k8s.io/selfsubjectrulesreviews/write",
],
"NotDataActions": [],
"assignableScopes": [
"/subscriptions/{subscription_id}"
]
}
```
If correctly specified, the Azure authentication service automatically grants access to any cluster within `subscription_id`
without any other definition. On the other hand, if it's incorrectly configured, an error is triggered but Teleport cannot gain access to the cluster.
### Kubernetes RBAC
If AZ RBAC integration is disabled, the authorization to the cluster is processed by Kubernetes RBAC. This is done by matching the Az Identity principals (`group_ids`) with `Role`, `ClusterRole` objects that live in the AKS cluster. This mode requires that the `ClusterRole` and `ClusterRoleBinding` must exist and must be well configured for each cluster to enroll.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: teleport-role
rules:
- apiGroups:
- ""
resources:
- users
- groups
- serviceaccounts
verbs:
- impersonate
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- "authorization.k8s.io"
resources:
- selfsubjectaccessreviews
- selfsubjectrulesreviews
verbs:
- create
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: teleport-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-role
subjects:
- kind: Group
name: {group_name}
apiGroup: rbac.authorization.k8s.io
```
#### `ClusterRole` and `ClusterRoleBinding` configured
If cluster operators or previous Teleport run has configured access to the cluster, no further action is required since Teleport already has access to the cluster.
#### Cluster `aks:ListClusterAdminCredentials` returns valid credentials
If the Teleport process has access to [`aks:ListClusterAdminCredentials`](https://learn.microsoft.com/en-us/rest/api/aks/managed-clusters/list-cluster-admin-credentials?tabs=HTTP) and the endpoint returns valid cluster admin credentials, Teleport will automatically create the `ClusterRole` and `ClusterRoleBinding` objects in the cluster configured to the `group_id` that is listed in the access token. In order to extract the `group_id` from the token, Teleport parses the JWT claims and extracts the first element.
If the object creation was successful, Teleport can access it, otherwise, it will use the `aks:BeginRunCommand` method to try to configure access to itself.
#### Cluster `aks:BeginRunCommand` returns valid credentials
When we reach this mode, Teleport tries to run a `kubectl` command against the cluster to configure the `ClusterRole` and `ClusterRoleBinding`. `aks:BeginRunCommand` allows any user with access to that endpoint to run arbitrary commands in the cluster (commands cannot be validated). Teleport will use it as the last resource to configure the access to itself.
If the command failed, Teleport cannot grant access to the cluster and an error is returned.
# UX
Currently, to discover AKS resources created and to have them dynamically served by the `kubernetes_service`one can define the following configuration.
```yaml
discovery_service:
enabled: true
azure:
- subscriptions: ["*"]
types: ["aks"]
regions: ["*"]
tags:
'*': '*'
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- Support AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276, #16281
This PR presents a watcher for dynamic `kube_cluster` resources. The cluster credentials can be set using a `kubeconfig` whose payload is defined in `KubernertesClusterV3.Spec.Kubeconfig`. These credentials are used to connect to the Kubernetes API server and if invalid, the cluster is not served and returns an error.
Currently, `kube_cluster` resources created via the API with a kubeconfig payload can be dynamically served by the `kubernetes_service` by enabling dynamic resources.
```yaml
kubernetes_service:
enabled: true
resources:
labels:
'*': '*'
```
# Future work
- The discovery service will have to create these resources.
- Support Azure and AWS dynamic authentication
Part of #16135, #13376
Related to #12048, #16276
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Introduce new make targets to check and add license headers to files
("make lint-license" and "make fix-license"). License checking is now a part of
"make lint" as well.
Initial attempts used goheader, but it caused "make lint-go" to become about 9x
slower (if not more), plus it only targets go files. Google's addlicense is fast
enough and targets however many file types we want.
Existing files that were missing licenses got the header added, using the
current year as the license date.
* Introduce lint-license and fix-license make targets
* Ignore generated files
* Add license to go files
* Replace irregular licenses with standard copyright/license
* Add license to proto files
* Install addlicense in build.assets Dockerfile
Teleport will fail to start when when a k8s cluster is unavailable when
using the kubeconfig in a `kubernetes_service` configuration. This means
that a single missing cluster can disrupt _all_ of the configured
clusters, even if the others are online.
This change makes failing the cluster credential enumeration a
per-k8s-cluster warning, rather than a stop-the-world error.
It also expands the testing shims inside the k8s proxy to allow more
sophisticated mocked scenarios, in order to test the above.
See-Also: #7215
Added fields:
- kube users/groups
- pod name/namespace
- container name/image
- node name
Container image and node name need to be fetched from the k8s API, they
are not known from just the client request. This fetch is optional, and
if it fails (like due to permission errors), those fields will be
missing.
Since kubernetes_service can talk to k8s API and proxy_service can't,
all session events are now emitted by kubernetes_service and skipped by
the proxy (used to be the other way around).
This change has several parts: cluster registration, cache updates,
routing and a new tctl flag.
> cluster registration
Cluster registration means adding `KubernetesClusters` to `ServerSpec`
for servers with `KindKubeService`.
`kubernetes_service` instances will parse their kubeconfig or local
`kube_cluster_name` and add them to their `ServerSpec` sent to the auth
server. They are effectively declaring that "I can serve k8s requests
for k8s cluster X".
> cache updates
This is just cache plumbing for `kubernetes_service` presence, so that
other teleport processes can fetch all of kube services. It was missed
in the previous PR implementing CRUD for `kubernetes_service`.
> routing
Now the fun part - routing logic. This logic lives in
`/lib/kube/proxy/forwarder.go` and is shared by both `proxy_service`
(with kubernetes integration enabled) and `kubernetes_service`.
The target k8s cluster name is passed in the client cert, along with k8s
users/groups information.
`kubernetes_service` only serves requests for its direct k8s cluster
(from `Forwarder.creds`) and doesn't route requests to other teleport
instances.
`proxy_service` can serve requests:
- directly to a k8s cluster (the way it works pre-5.0)
- to a leaf teleport cluster (also same as pre-5.0, based on
`RouteToCluster` field in the client cert)
- to a `kubernetes_service` (directly or over a tunnel)
The last two modes require the proxy to generate an ephemeral client TLS
cert to do an outbound mTLS connection.
> tctl flag
A flag `--kube-cluster-name` for `tctl auth sign --format=kubernetes`
which allows generating client certs for non-default k8s cluster name
(as long as it's registered in a cluster).
I used this for testing, but it could be used for automation too.
* Make k8s permissions test optional
There are several legitimate cases where it can fail:
- root proxy running inside k8s but without access to local k8s cluster
- root proxy running with a dummy kubeconfig that we recommended in the
past
Leave a ForwarderConfig flag to enforce this check, it will be useful in
kubernetes_service later that should always have the right permissions.
Prior to https://github.com/gravitational/teleport/pull/3811, if users
wanted to run a root proxy without k8s clusters but leaf proxies with
k8s, they had to put a dummy `kubeconfig_file` on the root proxy.
The permissions self-test added in
https://github.com/gravitational/teleport/pull/3812 didn't take that
into account.
So, users who keep the old workaround and upgrade to 4.4 will see their
proxies fail to start. To recover, they have to realize that
`kubeconfig_file` can be removed.
Previously, we needed:
- create on namespaces
- impersonate on all users/groups/service accounts
- list pods in kube-system namespace (via teleport-ci-test-group)
- exec/portforward on kube-dns pod in kube-system namespace (via teleport-ci-test-group)
Now, we need:
- create on namespaces
- create on pods in namespace teletest
- impersonate on all users/groups
- get/exec/portforward on pod test-pod in namespace teletest (via teleport-ci-test-group)
Unfortunately, `resourceNames` in RBAC doesn't work with `create` verbs,
so we can't scope down impersonation to just the right users/groups.
This simplifies a setup using trusted clusters:
- root teleport cluster as identity source and frontend
- leaf teleport clusters as gateways to individual k8s clusters
In this setup, root cluster doesn't talk to any k8s clusters directly.
It only forwards requests to leaf cluster proxies.
Without this change, users would have to put a dummy kubeconfig file on
root proxies just to enable k8s forwarding. With this change, this is no
longer needed.
Fixes#3087
To help operators diagnose mis-configured kubeconfig or k8s RBAC roles,
make proxy verify that impersonation is allowed. If not, fail at
startup.
Note: this requires the proxy to have permissions to create
SelfSubjectAccessReviews.
Fixes#3649
The existing kubeCreds already has tls.Config.
Add a new wrapTransport method to it, which handles added bearer tokens
or basic auth to the request before it goes out.
Use this wrapTransport to inject tokens for both SPDY connections and
catch-all forwarder requests.
Also simplify a few unrelated parts:
- don't manually call serverName verification in SPDY dialer
- have requestCertificate return a finished tls.Config instead of an
intermediate type
Zac Bergquist