## What
First part of the Kubernetes [Discovery RFD](https://github.com/gravitational/teleport/pull/13376/) to introduce a Kubernetes server per cluster.
This PR introduces a separate Kubernetes server that uses the already introduced `KubernetesClusterV3`.
## Compatibility
In previous versions, Kubernetes Clusters were part of regular `ServerV2` resource and this refactoring deprecates the `ServerV2` usage but keeps them for compatibility with previous version.
Everything is backward compatible, so v10 kubernetes agents and trusted clusters can connect fine.
## Next steps
Once this is merged, a new PR will introduce dynamic registration for Kubernetes Clusters discovered through EKS Discovery.
* Embed auth.Cache in auth.Server
* Hit the backend during Auth initialization
* Bypass the cache when rotating CAs
* Services.UpsertTrustedCluster is different
* Bypass the cache in waitForTunnelConnections
* Fix infinite recursion
* More cache bypassing during init and rotations
* Rename Services to Uncached in auth.Server
* Further cleanups
* Don't start the auth cache immediately
* Go back to Services rather than Uncached
* Comments and a missing method
* Permit parallelization in cache init
Allow both the auth and proxy caches to fetch multiple resources
in parallel. All other services are still limited to a single
resource at a time to reduce network spikes when a cluster is
recovering.
Also adds some spans to cache fetch and collection apply to make
identifying cache init traces easier.
Adds a `trace.Tracer` to the `cache.Cache`so
that all operations can be traced. Also, updates
the `utils.FnCache` to link the `context.Context`
passed to the loadFn to any current spans. This
will allow any fetches made by the `FnCache` to
be associated with the call that lead to loading.
Note: There are a few methods in the `auth.Cache`
interface which do not take a `context.Context`.
For the time being all spans for these calls use
a `context.TODO` and will be updated in the future,
where the changes can be made on a per method basis.
Fixes total count errors for the web UI:
- Adds de-duplicating matches for `FakePaginate` func.
- This change does makes `FakePaginate` less efficient b/c
it will always run filter for every resource per fetch versus just
running it on the initial fetch to get the total count.
- Removes inaccurate de-dupping in the web api layer since we
do it from the back now
- Adds another criteria when de-duplicating `applications` by checking
for its `public_addr` as well as its name.
* Remove more deprecated code
In the case where we were using the PostalCode, StreetAddress,
and Province fields rather than ASN.1 extensions we could have
removed more code, but I opted for a phase approach where we stop
populating the fields in v10 (but still accept them), and then
stop reading them entirely in v11.
* Fix lints and remove more unused code
* Restore StreetAddress / PostalCode
We don't yet have dedicated extensions for these
* FnCache context improvements
Modifies the FnCache to be constructed with a context.Context
which is passed to all loadfn provided to Get. This removes
the burden placed on callers to ensure loadfn are not request
context scoped.
* Remove unused backend wrapper from Cache
* Remove double printShutdownStatus
* Fix readyz race condition
* Test coverage for the readyz.monitor fix
* Close listeners immediately in proxy.shutdown
* Use and handle net.ErrClosed correctly
This adapts utils.IsUseOfClosedNetworkError to check for net.ErrClosed
even inside trace.Aggregate errors, makes it so that we always return
something that would pass errors.Is(err, net.ErrClosed) when returning
from a (net.Listener).Accept(), and handles closed listeners within our
various Serve() loops so that we don't hit spurious backoff waits while
shutting down.
* Close listeners early and emitters late
* Test coverage for the proxy listener changes
* Revert some errors back to trace.ConnectionProblem
* Reduce PR scope to just the proxy, add comments
* Improve error logging.
The RemoteClusterTunnelManager was hanging on the call to
GetReverseTunnels because it did not take a context and thus did
not respect cancellation. During certain shutdown/restart scenarios
the auth server may not respond, which causes Sync to never return
and subsequently prevents Run from ever completing.
By requiring a context to be passed to GetReverseTunnels, we can
remove the hard coded context.TODO that was being sent to the
http client and fully respect cancellation.
* Prevent relative expiry from emitting more events than can be processed
This alters relative expiry in four ways:
1) Relative expiry is no longer exclusively run by the auth cache
2) Relative expiry no longer emits delete events
3) There is now a limit to the number of nodes removed per interval
4) Relative expiry now runs more frequently
We can remove the need to emit any fake delete events during relative
expiry by not running it exclusively in the auth cache. All caches
will now run relative expiry themselves, even the components that
don't watch for nodes - this is effectively a noop for them. To
prevent the individual caches from getting too far out of sync, the
expiry interval is set to a much smaller value and we limit the
number of nodes being deleted per interval.
This adds a test, `TestCacheWatchKindExistsInEvents`, that ensures
all `types.WatchKind` registered for all Cache components have a
corresponding `proto.Event` defined. By doing this we can ensure
that for a release there will be no `resource type X is not
suppported` errors causing the Cache to be unhealthy. Unfortunately
this does not prevent issues across releases which may have different
resources being emitted.
Introduce Database Certificate Authority. New CA is used by Database Access to sign database certificates making them independent from Host CA.
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Always use in-memory caches
This also cleans up now-useless fields and constants related to on-disk
caches.
* Remove the cache tombstone mechanism
As we're never reopening the same cache backend twice, this is no longer
useful.
* Warn if a cache directory exists on disk
We can't remove it automatically because we might be in the middle of an
upgrade with a old version of Teleport still running.
* Fix goroutine and memory leak in watchCertAuthorities
The CA Watcher was blocking both on writing to a channel when the watcher
was closed and on HTTP calls that had no request timeout or context passed
to cause cancellation.
All resourceWatcher implementations that had a bug which may cause them to block
on writing to a channel forever were fixed by selecting on the write and ctx.Done.
Adding context.Context to all Get/Put/Post/Delete methods on the auth HTTPClient to
force callers to propagate context. Prior all calls used context.TODO which
prevents requests from being properly cancelled.
Add context propagation to RotateCertAuthority, RotateExternalCertAuthority,
GetCertAuthority, GetCertAuthorities. This is needed to get the correct ctx
from the CertAtuhorityWatcher all the way down to the HTTPClient that makes
the call.
Closes#10648
* The web UI needed to show the total count of resources available after filter
* ListResources response returns as a struct to make adding extra fields easier
* Add support for windows desktop services proxying different desktops
* Add filter to GetWindowsDesktops, remove GetWindowsDesktop and GetWindowsDesktopByName
* Cache cleanup
* Fix cache deletes for Windows desktops
For deletes, the cache only gets the backend key, not the entire
resource. Do what database access does, which is to extract the
host ID from the path, and stuff it in the description field of
the resource header.
* Godoc cleanup
* Fix lint
* Address review comments
* Send error message if no desktop found
* Revert to x/net/websocket
This got converted to gorilla/websocket as part of moderated sessions.
We'll do a more intentional conversion post-release.
* fix lint
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Define sorters for resource Server, AppServer, and DbServer
* Add sorting to ListResources in caching and presence layer
* ListResources now returns nextKey set to the limit+1th item,
previously it returned a possible next key, where there
may or may not be more results.
This adds an rpc UpsertKubeServiceV2 to replace UpsertKubeService. Currently, kubernetes service does not have a keepalive heartbeat unlike app, db, and windows service. This brings functionality in line with the others. This would allow for future use of the keepalive to track connected agents via prometheus metrics.
* Dynamically resolve reverse tunnel address
The reverse tunnel address is currently a static string that is
retrieved from config and passed around for the duration of a
services lifetime. When the `tunnel_public_address` is changed
on the proxy and the proxy is then restarted, all established
reverse tunnels over the old address will fail indefinintely.
As a means to get around this, #8102 introduced a mechanism
that would cause nodes to restart if their connection to the
auth server was down for a period of time. While this did
allow the nodes to pickup the new address after the nodes
restarted it was meant to be a stop gap until a more robust
solution could be applid.
Instead of using a static address, the reverse tunnel address
is now resolved via a `reversetunnel.Resolver`. Anywhere that
previoulsy relied on the static proxy address now will fetch
the actual reverse tunnel address via the webclient by using
the Resolver. In addition this builds on the refactoring done
in #4290 to further simplify the reversetunnel package. Since
we no longer track multiple proxies, all the left over bits
that did so have been removed to accomodate using a dynamic
reverse tunnel address.
Fix flaky unit tests
Addresses issues causing failures in TestCache_Backoff, TestTeleportProcess_reconnectToAuth
and TestResourceWatcher_Backoff. By utilizing FakeClock.BlockUntil tests ensure that the clock
will not be advanced until retry.After has been called. Move retry duration channels to config in order to allow them to be buffered by tests.
Move cache and resourceWatcher watchers from a 10s retry to a jittered backoff retry up to ~1min. Replace the
reconnectToAuthService interval with a retry to add jitter and backoff there as well for when a node restarts due to
changes introduced in #8102.
Fixes#6889.
* Add RBAC for Windows desktop access
This commit adds RBAC checks for Windows Desktops as described in
RFD 33 and RFD 34:
- add Windows desktop logins & labels to role definition
- introduce new file config for host labels based on a regexp match
- auth server API performs access checking for Windows desktop resources
- add RDP client callback to authorize the user
- support user/role locks
- respect the client idle timeout setting
Note: in cases where an connection is terminated to to RBAC, the web UI
currently displays "websocket connection failed" because the connection
is closed from the server. We'll need to follow up with a nice error
message for the client side to improve the UX here.
Other changes:
* Remove OSS RBAC migration marked for deletion
* Stop creating a default admin role
* add wildcard desktop access to the preset access role
Updates #7761
Tiago Silva