* Fix TestDiagnoseSSHConnection to be less strict on the exact unknown user error message, which may vary between OS's.
* Decouple/refactor SkipLocalAuth, UseKeyPrincipal, and static auth methods.
* Resolve comments.
* * Add tests for new methods
* Update proxy ssh principal test to catch regression
* Bump predicate error in switch case priority
* Move configuration from lib/service to lib/service/servicecfg
The new servicecfg package will hold only configuration for services.
This will allow other packages (like tctl and tsh) to depend on
servicecfg without pulling in all of lib/service (which has a number
of platform-specific details).
This is the first step towards being able to build tctl for Windows.
* Move PAM and BPF config into servicecfg
This breaks a compile-time dependency on BPF/PAM for tctl.
* Track active ssh sessions established via the web
Adds open session tracking in `web.Handler` and introduces a new
`web.Server` to prevent closing the web `http.Server` during
a graceful shutdown if there are still open sessions. From the
`http.Server/Shutdown` docs:
> Shutdown does not attempt to close nor wait for hijacked
> connections such as WebSockets. The caller of Shutdown should
> separately notify such long-lived connections of shutdown and wait
> for them to close, if desired.
Because the web sessions use WebSockets, `Shutdown` will not
wait for them to be idle before terminating. This used to work due
to the fact that web sessions dialed the Proxy SSH Port to establish the
SSH session. Which meant that all accounting of active user connections
by `sshutils/server.go` was accurate. But now that sessions are
established directly via the `proxy.Router` the web sessions were
unaccounted for.
The `web.Server` now mimics the behavior of `sshutils/server.go` for
Shutdown. Only once all the active web sessions have been terminated
will the underlying http.Server be shutdown.
Fixes#20227
* WIP adding OpenSSH CA
* fix creating OpenSSH CA for tests
* fix a test name and tctl usage flag
* fix 'tctl status' not displaying openssh CA sometimes
* add OpenSSH CA to TestInit_bootstrap, remove TLS key pair
* add OpenSSH CA to a few more places
* make help text of 'tctl auth rotate' more helpful and accurate
* fix web cert export test
* tbot doesn't need to know about OpenSSH CA
* address feedback
* fix comment formatting
* Change the application access auth flow to use a CORS POST request
* Remove auth state cookie
* Close the response body in tests
* Change the CSP to allow cross origin requests to the application
* Change the cookies to lax mode
* Update lib/web/app/handler_test.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Fix the proxy URL in tests
* Update webassets ref
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
n the context of Teleport Discover, we must be able to known if there's any DatabaseService available to proxy a given Database resource.
If there's none available, we will offer a script for the user to run and install a DatabaseService which proxies the desired Database resource.
By DatabaseService, we mean the process that Teleport binary manages when the `teleport.yaml` config has the following section:
```yaml
db_service:
enabled: "yes"
```
To accomplish this we are creating a new resource: DatabaseService.
The UI will fetch all DatabaseServices and check if there's any ResourceMatcher that matches the DatabaseLabels.
Previous PRs created the DatabaseService resource and its CRUD methods.
This PR adds an heartbeat for DatabaseServices similar to what we have for Databases.
There's also a new command to list DatabaseServices using `tctl`:
```
$ tctl get db_service --format text
Name Resource Matchers
------------------------------------ --------------------------------------
a6065ee9-d5ee-4555-8d47-94a78625277b (Labels: <all databases>)
d4e13f2b-0a55-4e0a-b363-bacfb1a11294 (Labels: env=[prod],aws-tag=[xyz abc])
```
Test using Teleport Cloud
```
dinis@lenix ~/p/cloudagents> tctl get db_services
kind: db_service
metadata:
expires: "2022-12-21T18:05:10Z"
id: 1671645310983808522
name: 2a28d394-900c-42ea-a120-eed918e4526b
spec:
resources:
- labels:
aws-tag:
- xyz
- abc
env: prod
version: v1
dinis@lenix ~/p/cloudagents> tctl status
Cluster marcoacme.cloud.gravitational.io
Version 12.0.0-dev
```
Part of #19032
Related #19363#19469
* Connection Diagnostics: Postgres Database tester
When adding a new resource using the Web UI we want to allow users to
test connecting to it.
We have two connection testers already:
- for SSH Nodes
- for Kube clusters
This PR adds a third tester: Postgres Database.
Most of the required changes for any Database are already present but
we're focusing on Postgres for now. Other databases will be added as
future PRs.
Testing a Database is similar to the other tests:
- generate certs for the logged in user
- connect to the resource using those certs
When generating the certificate, we inject an ID so that the Database
Service can add Connection Diagnostic traces.
* Add TTL field to integration/helpers.UserCredsRequest
This will let us create expired user certs by providing a negative TTL.
* Reissue gateway cert if middleware detects it expired
* Add integration test for gateway cert renewal
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
This change re-enables the AuditOn system test and fixes the TTY connection between the Teleport parent and child process. It should allow the child to send the error code to the parent, which should fix the test.
The CertificateTTL causes problems after the introduction of the fake clock
into the appaccess tests due to the fake clock not being propagated properly
throughout the Teleport stack. Increasing the TTL should make this test much
less flaky.
The appaccess TestTCP* tests are highly reliant on time. This has been
reduced (but not eliminated) by using a fakeClock and a channel for
signaling monitor triggered connection closures.
* Fix flaky basic auth dialer test
* Block on sending error to waiters in ProxyAuthorizer, to avoid racing
against the waiter.
* Change the test to start the node in a separate goroutine and
manipulate the auth creds while it is still attempting to connect.
This will allow us to significantly speed up the test since we can
verify that the proxy authorizer is rejecting bad credentials but then
change the credentials to be valid afterwards, allowing the node to
succeed in registering and avoid long wait for it to fail to register.
* Use buffered chan
* Move the stopall defer up in case the test fails earlier
* Remove extra zero count check
Application access connection monitoring has been introduced so that, when a
lock is created, application access connections will be interrupted until the
lock has been cleared. This includes web sockets and TCP applications.
* Ensure invalid tunnel agent connections get closed
Connections from reverse tunnel agents were being marked
as invalid by the proxy under certain conditions but would
ultimately never be closed. This could lead to scenarios where
the agent thought things were fine but the proxy considered
that agent unhealthy and unroutable.
Pruning of invalid connections used to occur when a proxy
tried to retrieve a connection for that tunnel. This also
further muddied the point in time at which the proxy could
close a connection as it never explicitly stopped tracking
the connection and closed it at the same time.
To remedy this, connections are explicitly closed by the proxy
and removed from the mapping to stop tracking immediately. In order
to prevent a connection that is servicing an active connection
from being closed the proxy now tracks which connections have
sessions. Closing does not occur when there are any active
sessions to prevent them from being force terminated.
When the proxy receives a heartbeat from an agent it now restores
the connection to a valid state. In the event that too many heart
beats have been missed for an agent, the proxy will now terminate
the connection, again only if it is not serving any sessions.
Fixes#15911
* Update proxy handler and authorizer mocks
* Use a condition variable to properly sync checks for the last error received
* Fix test to check for all 3 nodes registering correctly
* Update naming of LastError to be more descriptive
* Refactor authorizer to not use a condition variable
* Remove 3rd node to speed up test
* Make test more robust
* Remove the reset count func
* Close client conns before new requests to be sure
* Connection -> Request terminology
* Change test to not mutate the environment variable, but instead manipulate the auth proxy credentials
* this way we can be sure that the test will work correctly when the credentials match. If we mutate env, we don't know whether the callers are still holding a dialer using the old env variable
* Remove extra node
* fix lint
* Fix req waiting
* Change wording in debug message
* fix comment
* Update integration/helpers/proxy.go
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Fix defer func
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Add local proxy middleware for db cert checking
* Use tls conversion util instead of inline
* Add middleware to local proxy config
* Add middleware configuration in tsh
* Use route to database check and set defaults func
* Dont trigger normal db login flow if using local proxy tunnel
* Split out adding client creds into helper func for testing
* Add integration test for local proxy tunnel db cert middleware
* Add unit test for local proxy middleware
* Update comment
* Make middleware on new conn block
* godoc
* Make any cert check error trigger cert renewal in local proxy middleware
* Move dbcertchecker into lib/client
* Remove unneeded mutex in local proxy and unused func in lib/utils
* Make local proxy middleware integration test more robust
* Print message before mfa prompt in proxy tunnel
* Add before prompt option to test
* Remove unneeded comment
* Change local proxy messages to be more clear
* Pass local proxy opts by reference
* Pass certs in opts instead of cert/key file path
This is so we can check if the error is recoverable while preparing
local proxy options. A tunneled local proxy can ignore the error because
it does not rely on cert files - it can just renew its certs if
necessary.
* Move db route checking back to tsh
* Fix lint err
* Fix typo and print the hint to same writer as the mfa prompt
* Introduce proto types for ProvisionTokenV3
* Add methods to ProvisionTokenV3 to support ProvisionToken iface
* Start building v3 support into the client
* add support for mashalling and unmarshalling ProvisionTokenV3
* Start unit testing ProvisionTokenV3
* Remove oneof to support yaml marshal/unmarshal
* Client should try V3 methods and fallback to v2
* More tests
* Fix join tests
* Fix integration tests
* Switch integration tests to use v3 spec
* Switch iam tests to use ProvisionTokenV3
* Change ec2 join tests to use V3 tokens
* Fix events tests for V3 token
* support ProvisionTokenV3 within API client events handler
* Explicitly specify JoinMethod
* Tidy up final usage of NewProvisionTokenV2FromSpec in tests
* Improve proto docs on ProvisionTokenV3
* Fix bot join tests
* Clarify error message for invalid join method
* Adjust resource version comment
* Fix comments and return error rather than bool in V2() method
* Catch incompatible conversions case
* Include V2 ProvisionToken in tests and add appropriate DELETE IN notes
* Fix linter warnings/unit test failures
* Use nolint rather than lint:ignore
* Add more DELETE IN notes
* Run goimports on join_ec2_test.go
* Address PR comments from tim.
* Add more deprecation/delete in notices
* Improve godoc comments on checkAndSetDEfaults for provider config
* Simplify implementation by dropping client-ahead compatability
* Add some support for client-ahead but with conversion to v3
* Update code comments to include responsible party
* Rename `Role` to `RoleARN` in EC2 configuration for clarity
* Fix tests for Role -> RoleARN rename
* Move MustCreateProvisionToken out of API and into test packages
* Properly go imports files
* Reduce number of auth dials for tsh commands
One of the major areas of latency for `tsh ssh` is creating multiple
auth clients. Since the auth client is lazy and only actually performs
the dial on first use we can create an auth client once and simply
reuse it. This is done by adding an `auth.ClientI` to `ProxyClient`
which is created via `connectToProxy`. All attempts to connect to
the current auth server via the `ProxyClient` will be given the
cached `auth.ClientI`.
The new method of retrieving the current auth client also allowed
to remove a number of calls to `GetSites` which were used to obtain
the current cluster name. The local profile already contains the name
of the cluster and calls to `GetSites` were unnecessary. All instances
which relied on the site name now retrieve from information that the
`ProxyClient` already has.
In an effort to reduce ambiguity and confusion `CurrentClusterAccessPoint`
and `ClusterAccessPoint` were also removed. AccessPoint denotes that
you are connecting to a cache, but the `ProxyClient` is always going
to be hitting the auth server directly. The two have been replaced
with `CurrentCluster` and `ConnectToCluster`, which they were merely
wrappers for anyhow.
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
Following on from #13658, this patch removes more (but unfortunately not
all) usages of the deprecated, list-based port-allocation scheme.
This patch:
1. Updates the integration test `TeleInstance` fixture to use injected
listeners rather than static ports when creating a new proxy node in
a cluster,
2. Updates tests affected by (1) to pre-allocate and inject listeners,
including handling caching the listener FDs between proxy restarts
3. Removed unnecessary port allocations when creating LoadBalancer
fixtures, and
4. Moved the remaining list-base port allocation functions out of helpers
and back into integrations and made private. These functions should
never be used by more than one test package concurrently or there is a
very high chance of a port collision. Rather than just write that rule
down in the comments, I have contained the deprecated code into the
affected package made the compiler enforce the rule for us.
See-Also: #12421
See-Also: #13658
See-Also: #14408
Making all of our integration tests run in entirely parallel requires
a large engineering effort to enforce test isolation and remove all race
conditions between tests.
A lower-effort alternative may be to split apart the various test suites
into their own Go packages, and test those packages in parallel, even if
the tests inside are still executed serially. Auditing the test suites
for races on system-level resources (e.g. files, ports) is much easier
than chasing down every p[ossible race in the testing system.
This patch acts as a trial run, breaking a fairly well-defined and
self-contained test suite out into its own package. Note that the goal of
this change is not necessarily to shave minutes off the build (although
that would be nice), but to act as an illustration of how other, less
well-formed test suites might be broken apart.
See-Also: #12421
See-Also: #14408
Primary Changes:
- Remove reliance on Private Key PEM:
- Update native and keygen packages to return PrivateKey instead of PEM key
- Add new PrivateKey interface which implements crypto.Signer
- Replace PEM encoded private key usage where possible
- Replace calls to tls.(Load)X509KeyPair with keys.(Load)X509KeyPair in
client packages
Minor Changes:
- Remove unused agent.AddedKey return from LoadKey
- Simplify sshutils and removed unused code paths
- Add ecdsa and ed25519 key support
## What
First part of the Kubernetes [Discovery RFD](https://github.com/gravitational/teleport/pull/13376/) to introduce a Kubernetes server per cluster.
This PR introduces a separate Kubernetes server that uses the already introduced `KubernetesClusterV3`.
## Compatibility
In previous versions, Kubernetes Clusters were part of regular `ServerV2` resource and this refactoring deprecates the `ServerV2` usage but keeps them for compatibility with previous version.
Everything is backward compatible, so v10 kubernetes agents and trusted clusters can connect fine.
## Next steps
Once this is merged, a new PR will introduce dynamic registration for Kubernetes Clusters discovered through EKS Discovery.
When desktop access is enabled, the TeleportReady event will not
be emitted until the WindowsDesktopReadyEvent is emitted, and it
turns out we have *never* emitted a WindowsDesktopReadyEvent.
This is likely due to desktop access being copied from kube access
since the very beginning. The same issue was recently fixed for
kube access in #9418.
Ports used by the unit tests have been allocated by pulling them out of a list, with no guarantee that the port is not actually in use. This central allocation point also means that tests cannot be split into separate packages to be run in parallel, as the ports allocated between the various packages will be allocated multiple times and end up intermittently clashing.
There is also no guarantee, even when the tests are run serially, that the ports will not clash with services already running on the machine.
This patch (largely) replaces the use of this centralised port allocation with pre-created listeners injected into the test via the file descriptor import mechanism use by Teleport to pass open ports to child processes.
There are still some cases where the old port allocation system is still in use. I felt this was already getting beyond the bounds of sensibly reviewable, so I have left those for a further PR after this.
See-Also: #12421
See-Also: #14408
Brian Joerger