* Fix TestDiagnoseSSHConnection to be less strict on the exact unknown user error message, which may vary between OS's.
* Decouple/refactor SkipLocalAuth, UseKeyPrincipal, and static auth methods.
* Resolve comments.
* * Add tests for new methods
* Update proxy ssh principal test to catch regression
* Bump predicate error in switch case priority
Sets `http.Server.IdleTimeout` and `http.Client.IdleConnTimeout`
on clients and servers which didn't have them set. A default of
360s was chosen to be on par with the default of an NLB without
being identical.
This was added as another safety measure to prevent leaking any
idle connections indefinitely as seen in #22757.
`apidefaults.DefaultDialTimeout` was also renamed to
`apidefaults.DefaultIOTimeout` to better reflect its usage.
Previously doing an `ls -R` would list all resources per profile in
parallel. This had little benefit when one profile needed to list
multiple clusters as those were still done serially. To mitigate this
things are broken up into two separate steps:
1) Connect to all clusters the user has credentials for in parallel
2) List resources for each cluster in parallel
This ensures that by the time a client is established for all available
clusters that resource listing can happen in parallel regardless of
profile. All operations are still restricted to an upper bound, however
those limits were also bumped to help performance.
* Set SkipLocalAuth on identity file use; Simplify logic for SkipLocalAuth.
* Fix TrustedCertsFromCACerts to work with known hosts without proxy host set in host name; Fix HostKeyCallback for keys.
* Add regression test where known hosts are not written with proxy host.
* Fix client key default host key callback.
* Return empty list, not not found error, when there are no trusted certs.
* Restore option to filter trusted host keys in host key callback.
* Update lib/client/identityfile/identity_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
* Expose device verification methods in lib/devicetrust/authz
* Set device fields in auth.Context.GetAccessState
* Enable role-based device verification for SSH
* Enable role-based device verification for DBs
* Add TODO for better SSH error messages
* Do not override device error on lib/kube/proxy
* Run gci
Reduces latency creating ssh sessions via the web ui by:
1) No longer uses `TeleportClient.SSH` to establish a session
2) Reuses the user auth client for the web session to perform MFA ceremony
3) Ensures that connection attempts follow the flow outlined in RFD 93
The web api server now leverages the `proxy.Router` and `srv.SessionController`
directly, instead of doing so indirectly via `TeleportClient.SSH`. Using
the `TeleportClient` required an ssh connection to be established from the web
api server to the proxy ssh server, which are in the same process. This added
overhead can be avoided now that the routing logic and session control logic
exists in a reusable component. To create an interactive session on the node
once the connection is established, `client.NodeClient` is used. A new constructor
was added to facilitate creating one and remove duplicated creation code and a
`RunInteractiveShell` receiver method was added to allow callers outside of
`lib/client` to spawn a session.
`TerminalHandler.issueSessionMFACerts` used to check if per-session mfa was enabled
and perform the mfa ceremony via the `client.ProxyClient` which was constructed
with the `TeleportClient` established from connecting to the proxy ssh server.
This would dial the Auth server under the hood directly and call `IsMFARequired`
and do the ceremony if required. Each web session established via the web ui
already established an auth client with the credentials of the logged in user.
Again overhead is removed by leveraging the existing auth client and performing
the mfa ceremony manually.
Finally `TerminalHandler.makeClient` always attempted to perform the mfa ceremony
prior to returning the `TeleportClient`. As outlined in [RFD 93](https://github.com/gravitational/teleport/blob/master/rfd/0093-offline-access.md),
this causes additional latency and requires Auth connectivity to connect to nodes.
The connection flow is now modified to attempt connection to the nodes first, and
fall back to the mfa ceremony and reconnecting only if the node denies access.
Partially addresses #15167
* Allow connections to nodes when Auth is offline
`tsh ssh user@foo` currently fails if Auth is unreachable. Prior to
connecting to the target, `tsh` sends a `proto.AuthService/IsMFARequired`
request to determine if per-session MFA is required. If necessary, the MFA
ceremony then occurs before `tsh` attempts to connect to the node.
However, nodes ultimately makes the decision on whether the user has
access. To both increase availability of nodes and reduce initial
connection latency, `tsh` can first attempt to connect to a node and
fallback to checking if per-session MFA is required and then performing
the ceremony if necessary followed by attempting another connection to the
node. The major tradeoff is now when per-session MFA is required, `tsh`
will connect to the target **twice**. The added latency is likely to be
minimized by the amount of time it takes to perform the MFA ceremony.
## Connection flows prior to this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: no
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
## Connection flows after this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
* switch underlying protocol used for 'tsh scp' to SFTP
* address TODO
* appease linter
* add method to make it easier for other callers to transfer files
* add tests
* print transfer progress with progress bar by default
Also allow a SIGINT to gracefully stop the SFTP connection. This is
necessary because the progress bar will ignore signals and prevent the
process from exiting.
* address SFTP fork issues
* make tests less flakey
* fix specifying dir for dst not copying files to correct paths
* make tests less flakey (again)
* don't check file access times, often differs when run in CI
* few small fixes from review, simplify Create method now that HTTP FS isn't needed
* create dst files and dirs with src mode
* improved error messages when doing file operations
* expand home dirs in remote paths
* addressed more feedback
* add license to get_home_dir.go
* address minor feedback of tests, add home dir expansion test
* update sftp fork to point to latest commit on master branch
* addressed feedback
* don't cache home dir lookups, only one remote path can ever be used
* Reduce number of auth dials for tsh commands
One of the major areas of latency for `tsh ssh` is creating multiple
auth clients. Since the auth client is lazy and only actually performs
the dial on first use we can create an auth client once and simply
reuse it. This is done by adding an `auth.ClientI` to `ProxyClient`
which is created via `connectToProxy`. All attempts to connect to
the current auth server via the `ProxyClient` will be given the
cached `auth.ClientI`.
The new method of retrieving the current auth client also allowed
to remove a number of calls to `GetSites` which were used to obtain
the current cluster name. The local profile already contains the name
of the cluster and calls to `GetSites` were unnecessary. All instances
which relied on the site name now retrieve from information that the
`ProxyClient` already has.
In an effort to reduce ambiguity and confusion `CurrentClusterAccessPoint`
and `ClusterAccessPoint` were also removed. AccessPoint denotes that
you are connecting to a cache, but the `ProxyClient` is always going
to be hitting the auth server directly. The two have been replaced
with `CurrentCluster` and `ConnectToCluster`, which they were merely
wrappers for anyhow.
* Allow tsh to retrieve cluster details in one request
Prior to connecting to a node via `tsh ssh` we need two bits of
information about the cluster:
1) The session recording mode
2) Whether FIPS is enabled
In order to retrieve the information `tsh` first would send the
global ssh request `RecordingProxyReqType` to determine the
session recording mode. Later on `tsh` would Ping the auth
server to determine if the cluster was running in FIPS mode.
In an effort to reduce the number of round trips to retrieve
this data, a new global ssh request `ClusterDetailsReqType` is
introduced that returns both the session recording mode and
whether FIPS is enabled. This allows `tsh` to leverage the new
request to get all the information it needs, and is extensible
in case more information is needed` in one request which helps
reduce latency for `tsh ssh`.
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Primary Changes:
- Remove reliance on Private Key PEM:
- Update native and keygen packages to return PrivateKey instead of PEM key
- Add new PrivateKey interface which implements crypto.Signer
- Replace PEM encoded private key usage where possible
- Replace calls to tls.(Load)X509KeyPair with keys.(Load)X509KeyPair in
client packages
Minor Changes:
- Remove unused agent.AddedKey return from LoadKey
- Simplify sshutils and removed unused code paths
- Add ecdsa and ed25519 key support
* Add spans to TeleportClient/ProxyClient/NodeSession
* Improve dialer tracing
* Alter default http client span name to include path
* propagate tracing options in NewClientConn
Adds a wrapper around `ssh.Session` which injects tracing context
in a similar manner to the `ssh.Client` wrapper. All usages of
`ssh.Session` have now been replaced and have the appropriate
`context.Context` passed along
Part of #12241
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
* `tsh ssh`: graceful error handling for forwarding.
Both regular and dynamic port forwarding will now:
- continue to accept connections even if an error occurred
for whatever reason (e.g., invalid proxy protocol.)
- log richer information
- shut down the listen loop on parent context closure
* configure golangci-lint misspell to check for anglicized spellings
* Americanize spellings
* fix aws constant value with british spelling 🇬🇧
* update api types with americanized spellings
* use american spellings .cloudbuild/scripts
Reinstates some logic that was removed on #12475 and changes `optsOverride` to a
function, so there is less ambiguity in dealing with booleans / default values.
* Pass proxy address to PromptMFAChallenge calls
* Add coverage for TeleportClient.PromptMFAChallenge
Create spans for all public facing TeleportClient,
ProxyClient, and NodeClient methods. This makes
correlating spans easier to reason about when
looking at `tsh` traces. As a result of creating
spans, some additional context propagation is
required as well to ensure that spans are linked
properly.
This also removes the unused `quiet` argument from
`ConnectToCluster`. It's usage was not consistent
by existing callers, and it was ignored, so in order
to avoid confusion in future calls, it was removed.
#12241
This change adds the --all/-R flag to tsh ls, tsh apps ls, tsh db ls, and tsh kube ls, which lets tsh list resources from across all clusters and logged in proxies.
* Add client side circuit breaker to auth clients
In order to apply back pressure we can utilize a circuit breaker that
monitors error responses from auth server. When tripped it will prevent
all outbound requests to auth for a period of time. This can also help
prevent a potential thundering heard when auth is in an unhealthy state.
By default the circuit breaker will only be tripped if 90% of the
requests made in the monitoring interval fail.
* Span forwarding
Modifies the auth grpc server to implement the OTLP Collector
RegisterTraceServiceServer API (https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/collector/trace/v1/trace_service.proto).
This allows the auth server to receive spans from other services
like `tsh`, `tctl`, and `tbot`. Any spans received by the auth
server will be forwarded to the exporter configured via the
`tracing_service` if it is enabled. All received spans will be
dropped in the event that the`tracing_service` disabled. By
forwarding spans to the auth server, `tsh` doesn't need to
be provided with any of the telemetry backend information
to have its spans exported.
Adds a new `--trace` flag to `tsh` to enable collecting and
forwarding spans to the auth server. When set, the tracing
provider is initialized with a sampling rate of 1.0 to force
all spans to be recorded. Teleport respects the sampling rate
from remote spans, which means that when `--trace` is set, all
spans from `tsh` and any downstream Teleport services will be
recorded and exported regardless of the sampling rate that each
Teleport service is configured with.
* Add tracing instrumentation for ssh clients/servers
Add tracing context to the existing ProxyHelloSignature to provide
span information across ssh connections. To add span context per
ssh session on top of new connections, the same tracing context is
passed in the first global request of the session.
In order to ensure that tracing context is pulled from and inserted
into the proper context.Context, some interfaces and methods were
changed to take one as the first argument.
Add the --mfa-mode flag and wire touch ID into tsh authn/registration.
In the default MFA mode (auto), touch ID should be eager if there is a
credential present in the system. Users may change this behavior using other MFA
modes.
See https://github.com/gravitational/teleport/blob/master/rfd/0054-passwordless-macos.md.
#9160
* Add AuthenticatorAttachment/MFAMode to tsh
* Add support for touch ID in wancli.Login
* Warn users about passwordless in U2F mode
* Add support for touch ID in `tsh mfa add`
* feat: delete app web sessions during logout
* Apply suggestions from code review
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* refactor(auth): add VerbList action to delete user app sessions
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
Fixes a potential stdin hijacking bug by making relogin attempts default to a
single MFA method (the strongest available).
The problematic scenario is as follows:
1. User has both OTP and security keys registered
2. "Relogin" is triggered via a tsh command (say,
`tsh logout; tsh ssh --proxy=example.com llama@myserver`)
3. User is prompted to pick either OTP or security key ("Tap any security key or
enter a code from a OTP device")
4. An stdin read is fired in the background to read the OTP code (via
prompt.Stdin)
5. User picks the security method, thus the stdin read is "abandoned"
In most cases this is fine, as the program ends right after. The issue is when a
relogin is triggered by a long living tsh invocation (again, `tsh ssh ...`): in
this case the stdin hijack causes input to be swallowed.
Forcing a single MFA option avoids the potential stdin hijack, fixing the
problem for all relogin invocations. `tsh login` behavior remains the same.
Note that we have to default to cluster's most secure method _without_ checking
the user devices. The user is not logged in yet, thus the backend cannot reveal
any information about that user.
Fixes#11709.
* Add UseStrongestAuth flag to PromptMFAChallenge
* Add TeleportClient.UseStrongestAuth and set it true for relogin
* Proper testing
* Address review comments
* The web UI needed to show the total count of resources available after filter
* ListResources response returns as a struct to make adding extra fields easier
Brian Joerger