* Fix TestDiagnoseSSHConnection to be less strict on the exact unknown user error message, which may vary between OS's.
* Decouple/refactor SkipLocalAuth, UseKeyPrincipal, and static auth methods.
* Resolve comments.
* * Add tests for new methods
* Update proxy ssh principal test to catch regression
* Bump predicate error in switch case priority
To prevent users from being able to directly connect to registered
OpenSSH (Agentless) nodes, the OpenSSH CA was introduced. When
connecting to Agentless nodes, the Proxy will forward the connection
so the session can be recorded. The forwarding SSH server is configured
to authenticate to the target node with a certificate signed by OpenSSH
CA, which users do not have access to.
We were properly handling graceful websocket closure, but in some
cases (when terminating a session with ctrl+d, for example), the
failure manifest as a generic connection closed error instead of
a websocket close error.
Fixes#21334
* Track active ssh sessions established via the web
Adds open session tracking in `web.Handler` and introduces a new
`web.Server` to prevent closing the web `http.Server` during
a graceful shutdown if there are still open sessions. From the
`http.Server/Shutdown` docs:
> Shutdown does not attempt to close nor wait for hijacked
> connections such as WebSockets. The caller of Shutdown should
> separately notify such long-lived connections of shutdown and wait
> for them to close, if desired.
Because the web sessions use WebSockets, `Shutdown` will not
wait for them to be idle before terminating. This used to work due
to the fact that web sessions dialed the Proxy SSH Port to establish the
SSH session. Which meant that all accounting of active user connections
by `sshutils/server.go` was accurate. But now that sessions are
established directly via the `proxy.Router` the web sessions were
unaccounted for.
The `web.Server` now mimics the behavior of `sshutils/server.go` for
Shutdown. Only once all the active web sessions have been terminated
will the underlying http.Server be shutdown.
Fixes#20227
Web sessions were dialing with the sever hostname in cases where
the server UUID was known and should have been used which resulted
in #19415. All sessions launched via the "Connect" button from the
UI are guaranteed to use the server UUID. Manual connections via
the UI attempt find a matching host and use the UUID but may fallback
to using the hostname/ip instead if multiple matches are found.
When resolving servers by hostname or address `ListResources` is
now used directly to populate the `SearchKeywords` field to limit
the number of servers returned to only those fuzzily matching the
server. Prior to this **all** servers in the cluster were fetched
and filtered out by the web api.
The error displayed when ambiguous hosts are found has also been
improved from:
```shell
disconnected
err-node-is-ambiguous
```
to the following, which more closely mirrors the error returned by
`tsh`:
```shell
disconnected
error: ambiguous host could match multiple nodes
Hint: try addressing the node by unique id (ex: user@node-id)
```
Closes#19415
* Prevent race from causing remote clients from being closed
`web.SessionContext` was susceptible to races retrieving `auth.ClientI`
for remote sites. The mutex on the remote client cache was not held for
the duration of the `GetUserClient` which meant two simultaneous requests
could find no client existed in the cache, create one and then add it to
the cache. When the second client is added to the cache it would close the
first client created, which results in the `use of closed network connection`
error displaying on the web UI as described in #17173.
To remedy this the portion of `GetUserClient` that retrieves the remote client
is now protected by a `singleflight.Group`. This ensures that races on creating
a client no longer happen.
`web.SessionContext` was slightly refactored to make it easier to test, to
remove some unnecessary operations, and to provide `context.Context` to `GetUserClient`.
Additionally, all `web.SessionContext` variables which were named `ctx` in `lib/web`
have been renamed to avoid confusion as that often refers to a `context.Context`.
Fixes#17173
* Fix web ssh session with proxy recording mode
The agent was not being propagated when establishing an ssh session
via the web which resulted in the error described in #18850. Providing
the agent was straightforward, however, due to the changes from #18656
when using the forward server, as is required with proxy recording mode,
the ssh connection is now being performed directly over a `net.Pipe`.
Due to the synchronous nature of `net.Pipe` this causes a deadlock when
performing the ssh handshake. To mitigate the deadlocks, `DualPipeNetConn`
was changed to leverage `syscall.Socketpair` instead of `net.Pipe`.
`TestTerminal` now has two cases, one for node recording mode and another
for proxy recording mode. In order for the proxy recording mode test to pass
the fake clock used in the test needed to be properly propagated to the
`ssh.CertChecker` and `forward.ServerConfig`.
Fixes#18850
Reduces latency creating ssh sessions via the web ui by:
1) No longer uses `TeleportClient.SSH` to establish a session
2) Reuses the user auth client for the web session to perform MFA ceremony
3) Ensures that connection attempts follow the flow outlined in RFD 93
The web api server now leverages the `proxy.Router` and `srv.SessionController`
directly, instead of doing so indirectly via `TeleportClient.SSH`. Using
the `TeleportClient` required an ssh connection to be established from the web
api server to the proxy ssh server, which are in the same process. This added
overhead can be avoided now that the routing logic and session control logic
exists in a reusable component. To create an interactive session on the node
once the connection is established, `client.NodeClient` is used. A new constructor
was added to facilitate creating one and remove duplicated creation code and a
`RunInteractiveShell` receiver method was added to allow callers outside of
`lib/client` to spawn a session.
`TerminalHandler.issueSessionMFACerts` used to check if per-session mfa was enabled
and perform the mfa ceremony via the `client.ProxyClient` which was constructed
with the `TeleportClient` established from connecting to the proxy ssh server.
This would dial the Auth server under the hood directly and call `IsMFARequired`
and do the ceremony if required. Each web session established via the web ui
already established an auth client with the credentials of the logged in user.
Again overhead is removed by leveraging the existing auth client and performing
the mfa ceremony manually.
Finally `TerminalHandler.makeClient` always attempted to perform the mfa ceremony
prior to returning the `TeleportClient`. As outlined in [RFD 93](https://github.com/gravitational/teleport/blob/master/rfd/0093-offline-access.md),
this causes additional latency and requires Auth connectivity to connect to nodes.
The connection flow is now modified to attempt connection to the nodes first, and
fall back to the mfa ceremony and reconnecting only if the node denies access.
Partially addresses #15167
* Forward traces from the web UI
Adds a `/webapi/traces` endpoint to the proxy web handler
to allow receiving traces from the UI so that they can
be forwarded to the configured exporter in the proxy_service.
To accomodate traces coming in via websockets the tracing
handler is updated to pull tracing context first from the
standard headers and fallback to retrieving it from a query
parameter as web sockets from the UI cannot alter headers.
Additionally update `web.Terminal` to propagate the tracing
context properly and instruments some of the functions
to ensure spans from the UI are properly correlated to spans
on the backend.
This test takes between 10-20 seconds to run, because it waits for
a websocket ping message and pings run on a 10s timer by default.
It turns out, we allow specifying the keepalive interval in the
terminal request, but due to a bug we would always overwrite that
value with what's in the cluster networking config.
Fix the bug and speed up the test by overriding the keepalive interval
to something shorter for this test.
Before:
➜ go test -run TestTerminalPing -race ./lib/web -count=1
ok github.com/gravitational/teleport/lib/web 13.122s
After:
➜ go test -run TestTerminalPing -race ./lib/web -count=1
ok github.com/gravitational/teleport/lib/web 3.394s
* Prevent ssh.Session SendRequest from wrapping payload twice
Payloads were inadvertently being wrapped twice by sessions making
the ssh server unable to parse the request payload. This is largely
only an issue for window-change requests, as they are the only request
being sent directly via a session that contains a payload. All window-change
requests now use ssh.Session WindowChange instead or sending the request
manually.
Primary Changes:
- Remove reliance on Private Key PEM:
- Update native and keygen packages to return PrivateKey instead of PEM key
- Add new PrivateKey interface which implements crypto.Signer
- Replace PEM encoded private key usage where possible
- Replace calls to tls.(Load)X509KeyPair with keys.(Load)X509KeyPair in
client packages
Minor Changes:
- Remove unused agent.AddedKey return from LoadKey
- Simplify sshutils and removed unused code paths
- Add ecdsa and ed25519 key support
Adds a wrapper around `ssh.Session` which injects tracing context
in a similar manner to the `ssh.Client` wrapper. All usages of
`ssh.Session` have now been replaced and have the appropriate
`context.Context` passed along
Part of #12241
* Add tracing instrumentation for ssh clients/servers
Add tracing context to the existing ProxyHelloSignature to provide
span information across ssh connections. To add span context per
ssh session on top of new connections, the same tracing context is
passed in the first global request of the session.
In order to ensure that tracing context is pulled from and inserted
into the proper context.Context, some interfaces and methods were
changed to take one as the first argument.
The web api cannot leverage the NodeWatcher because it needs to
have each request for nodes go through RBAC checks per user. Rolling
back to creating an auth client per user and retrieving nodes
directly from the auth.ClientI.
* Prevent proxy from loading entire node set into memory more than once
When establishing a new session to a node, the proxy would load the
entire node set into memory in an attempt to find the matching host. For
smaller clusters this may not be that problematic. But on larger clusters,
loading >40k nodes into memory from the cache can be quite expensive.
This problem is compounded by the fact that it happened**per** session,
which could potentially cause the proxy to consume all available memory
and be OOM killed.
A new `NodeWatcher` is introduced which will maintain an in memory list
of all nodes per process. The watcher leverages the existing resource
watcher system and stores all nodes as types.Server, to eliminate the
cost incurred by unmarshalling the nodes from the cache. The `NodeWatcher`
provides a way to retrieve a filtered list of nodes in order to reduce the number
of copies made to only the matches.
Alias the "u2f" second factor mode to "webauthn", effectively sunsetting U2F in
favor of WebAuthn.
The change effectively disables "U2F mode" server-side, making Teleport use
WebAuthn instead. This is in line with our compatibility promise, as Teleport
8.x clients are already WebAuthn-capable (and thus have no problems talking to
the cluster).
I have cleaned up a good chunk of U2F references in lib/web and lib/client, plus
a few other places. Changes on lib/auth are just the necessary to get the tests
back to good standing. There is more work to be done, but this seems enough for
a single PR.
#10375
* Remove "Disabled" field from types.Webauthn
* Update generated protos
* Treat second_factor "u2f" as "webauthn"
* Remove references to Webauthn.Disabled
* Remove U2F from lib/web/
* Remove U2F from lib/client/
* Remove U2F from lib/auth/ (partially)
* Fix issues after rebase on master
* Fix typo
Removes the call for wait for ssh.Session end to check for errors
in web terminal which fixes a regression bug where typing "exit"
in web terminal does not return session end event.
PR #8081 removed the need to check for errors as it correctly
returns exit errors whereas before it returned nil.
* Remove JSON schema validation
Removing JSON schema validation from all resource unmarshalers.
--- what JSON schema gets us
Looking at the JSON schema spec and our usage, here are the supposed benefits:
- type validation - make sure incoming data uses the right types for the right fields
- required fields - make sure that mandatory fields are set
- defaulting - set defaults for fields
- documentation - schema definition for our API objects
Note that it does _not_ do:
- fail on unknown fields in data
- fail on a required field with an empty value
--- what replaces it
Based on the above, it may seem like JSON schema provides value.
But it's not the case, let's break it down one by one:
- type validation - unmarshaling JSON into a typed Go struct does this
- required fields - only checks that the field was provided, doesn't actually check that a value is set (e.g. `"name": ""` will pass the `required` check)
- so it's pretty useless for any real validation
- and we already have a separate place for proper validation - `CheckAndSetDefaults` methods
- defaulting - done in `CheckAndSetDefaults` methods
- `Version` is the only annoying field, had to add it in a bunch of objects
- documentation - protobuf definitions are the source of truth for our API schema
--- the benefits
- performance - schema validation does a few rounds of `json.Marshal/Unmarshal` in addition to actual validation; now we simply skip all that
- maintenance - no need to keep protobuf and JSON schema definitions in sync anymore
- creating new API objects - one error-prone step removed
- (future) fewer dependencies - we can _almost_ remove the Go libraries for schema validation (one transient dependency keeping them around)
* Remove services.SkipValidation
No more JSON schema validation so this option is a noop.
Prior to this change, `tsh` will only ever forward the internal key
agent managed by `tsh` to a remote machine.
This change allows a user to specify that `tsh` should forward either
the `tsh`-internal keystore, or the system key agent at `$SSH_AUTH_SOCK`.
This change also brings the `-A` command-line option into line with
OpenSSH.
For more info refer to RFD-0022.
See-Also: #1571
Check whether MFA is required for the current session and send a
challenge over the websocket.
client.IssueUserCertsWithMFA had to be modified to inject proxy's
cached user certs and websocket-based U2F prompt.
* Use fake clock consistently in units tests.
* Split web session management into two interfaces and implement them separately for clear separation
* Split session management into New/Validate to make it aparent where the sessions are created and where existing sessions are managed. Remove ttlmap in favor of a simple map and handle expirations
explicitly.
Add web session management to gRPC server for the cache.
* Reintroduce web sessions APIs under a getter interface.
* Add SubKind to WatchKind for gRPC and add conversions from/to protobuf. Fix web sessions unit tests.
* lib/web: create/insert session context in ValidateSession if the session has not yet been added to session cache.
lib/cache: add event filter for web session in auth cache.
lib/auth: propagate web session subkind in gRPC event.
* Add implicit migrations for legacy web session key path for queries.
* Integrate web token in lib/web
* Add a bearer token when upserting a web session
* Fix tests. Use fake clock wherever possible.
* Converge session cache handling in lib/web
* Clean up and add doc comments where necessary
* Use correct form of sessions/tokens controller for ServerWithRoles. Use fake time in web tests
* Converge the web sessions/tokens handling in lib/auth to match the old behavior w.r.t access checking (e.g. implicit handling of the local user identity).
* Use cached reads and waiters only when necessary. Query sessions/tokens using best-effort - first looking in the cache and falling back to a proxy client
* Properly propagate events about deletes for values with subkind.
* Update to retrofit changes after recent teleport API refactorings
* Update comment on removing legacy code to move the deadline to 7.x
* Do not close the resources on the session when it expires - this beats the purpose of this PR.
Also avoid a race between closing the cached clients and an existing reference to the session by letting the session linger for longer before removing it.
* Move web session/token request structs to the api client proto package
* Only set HTTP fs on the web handler if the UI is enabled
* Properly tear down web session test by releasing resources at the end. Fix the web UI assets configuration by removing DisableUI and instead use the presence of assets (HTTP file system) as an indicator that the web UI has been enabled.
* Decrease the expired session cache clean up threshold to 2m. Only log the expiration error message for errors other than not found
* Add test for terminal disconnect when using two proxies in HA mode
* lib/web: update logging to go through a package-level logger.
Updates https://github.com/gravitational/teleport/issues/4110.
* Unify uses of package-level logger. Update e
* Fix linter warning and tests
* Address review comments
This helps with ELB and similar L5 load balancers that don't respect
TCP-level keep-alives. ELB for example kills connections after 60s of no
application traffic.
With this PR, you can leave a `kubectl exec` session open indefinitely
without any activity.
Brian Joerger