* Use serverUID for scp target
Revert unneeded hostname change
Remove comment
Get name
* Change name to serverId
* Fix lint errors
* Add comment
* Fix comment and signature format
* Fix TestDiagnoseSSHConnection to be less strict on the exact unknown user error message, which may vary between OS's.
* Decouple/refactor SkipLocalAuth, UseKeyPrincipal, and static auth methods.
* Resolve comments.
* * Add tests for new methods
* Update proxy ssh principal test to catch regression
* Bump predicate error in switch case priority
* Improve error messages for tsh login connectivity and ssh port connection errors
* spell fix
* lint fix
* lint fix
* suggestion applied
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Moved message to helper function for ssh proxy connection error
* format function
* Include error message on non-specified port login
* Give error message on attempted proxy hosts
* style coding
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* change to string construction
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* bytes buffer update
* lint fix
* Update message
* Update lib/client/api.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Update lib/client/api.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
* Update tool/tsh/tsh.go
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
While loading a key from storage, Teleport client transverses the
storage directory to read the secrets and reads each file one by one.
This has huge impacts on `tsh kube credentials` because when calling
`kubectl` in parallel, the disk operations become overloaded and the
operation time increases exponentially.
This PR removes the transverse when reading credentials for Kubernetes
clusters and adds a quick read operation that reads the following files:
- `$TSH_HOME/current_profile`
- `$TSH_HOME/$profile.yaml`
- `$TSH_HOME/keys/$PROXY/$USER-kube/$TELEPORT_CLUSTER/$KUBE_CLUSTER-x509.pem`
- `$TSH_HOME/keys/$PROXY/$USER`
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old direct | 9.90s user 3.14s system 64% cpu 20.284 total | 11.08s user 3.60s system 496% cpu 2.955 total | 12.46s user 4.36s system 820% cpu 2.050 total | 12.81s user 4.76s system 811% cpu 2.164 total | 13.02s user 5.07s system 861% cpu 2.100 total|
| old teleport | 18.45s user 7.80s system 51% cpu 51.085 total | 21.58s user 10.03s system 434% cpu 7.270 total | 22.33s user 10.52s system 178% cpu 18.402 total | 22.04s user 10.83s system 96% cpu 33.977 total | 22.57s user 11.45s system 55% cpu 1:01.34 total|
| new direct | 9.51s user 2.90s system 146% cpu 8.460 total | 13.75s user 4.95s system 718% cpu 2.603 total | 12.21s user 4.43s system 832% cpu 1.999 total | 11.72s user 4.50s system 839% cpu 1.933 total | 12.06s user 4.62s system 830% cpu 2.009 total |
| new teleport | 16.10s user 5.97s system 42% cpu 52.104 total | 21.62s user 9.01s system 381% cpu 8.023 total | 23.22s user 10.11s system 540% cpu 6.162 total | 21.64s user 9.81s system 603% cpu 5.208 total | 20.70s user 9.13s system 602% cpu 4.948 total |
---
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old no tsh kube credential | 8.61s user 2.99s system 31% cpu 36.336 total | 9.57s user 3.39s system 283% cpu 4.565 total | 10.22s user 4.03s system 553% cpu 2.574 total | 9.32s user 3.82s system 559% cpu 2.351 total | 11.49s user 4.85s system 624% cpu 2.617 total
| new no tsh kube credential | 8.27s user 2.94s system 24% cpu 45.041 total | 10.85s user 4.31s system 195% cpu 7.767 total | 11.32s user 4.64s system 469% cpu 3.401 total | 10.83s user 4.51s system 553% cpu 2.773 total | 10.90s user 4.84s system 537% cpu 2.930 total |
---
| instances | old total | old avg | **new total** | **total avg** |
| --- | --- | --- | --- | --- |
| 1 | 0.145 | 0.145 | 0.074 | 0.074 |
| 10 | 0.430 | 0.043 | 0.146 | 0.0146 |
| 20 | 0.740 | 0.037 | 0.284 | 0.0142 |
| 30 | 1.108 | 0.037 | 0.417 | 0.0139 |
| 40 | 1.483 | 0.037 | 0.580 | 0.0145 |
| 60 | 13.604 | 0.228 | 0.825 | 0.0137 |
| 80 | 30.236 | 0.378 | 1.143 | 0.0142 |
| 106 | 46.681 | 0.440 | 1.413 | 0.0133 |
Fixes#19313
Co-authored-by: Brian Joerger <bjoerger@goteleport.com>
* Our current SFTP implementation uses a single channel to transfer data. Connections with a high latency cause very slow transfer. github.com/pkg/sftp allows transferring data using concurrent streams, which seems to solve the problem. In order to utilize the feature, the remote file must implement the WriteTo() or ReadFrom() and Stat() methods. I've moved the progress bar to relay the data from a local file, as we can still call Read() or Write() on it to update the progress.
This PR doesn't address the high CPU usage by github.com/pkg/sftp on low latency links.
Closes#20579
* Cleanup
* Fix fileTransferProgress
* Simplify interfaces
* Move helper to utils package
Add comments
* Added comments
* Change canceler implementation to return correct values from Read/Write
* Fix infinite recursive call in cancelWriter.
* Save private key policy in profile.
* Fix tsh status/logout with yubikey unplugged; fix cluster alert on tsh status with hardware_key_touch.
* Set private key policy from ping response.
* Skip cluster alerts on tsh status.
* Set SkipLocalAuth on identity file use; Simplify logic for SkipLocalAuth.
* Fix TrustedCertsFromCACerts to work with known hosts without proxy host set in host name; Fix HostKeyCallback for keys.
* Add regression test where known hosts are not written with proxy host.
* Fix client key default host key callback.
* Return empty list, not not found error, when there are no trusted certs.
* Restore option to filter trusted host keys in host key callback.
* Update lib/client/identityfile/identity_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
Since #16964, `tsh login` defaults to passwordless if it finds suitable platform
credentials - it is safer and more convenient than other authentication methods.
Unfortunately, it is a bit too eager in doing that.
This change adds the following prerequisites for the switch, in addition to
existing conditions:
1. `--auth` must not be specified
2. `--mfa-mode` must be either `auto` or `platform`
(1) avoids defaulting when `--auth=local` is present, as well as for other
custom/future values. (If `--auth=passwordless` is passed, we'll do passwordless
anyway.)
(2) includes both an explicit platform attachment test (which is a bit more
conservative in face of changes), as well as checking for `--mfa-mode=otp`.
#20429 and #20322
Before this commit, the `tsh` HTTP requests that had the extra headers
were those that did not use `roundtrip`.
This commit leverages `http.RoundTripper.RoundTrip` to ensure that all
requests have the the extra headers.
AttemptDeviceLogin, which is the main entry point for device authentication, now
checks the Ping response and skips the attempt entirely if device trust is
disabled.
The main objective is to avoid a needless roundtrip if the feature is disabled,
as one should only pay for what is in use.
There's actual little consequence in attempting the roundtrip, apart from the
added latency on logins, so I've gone with a negative flag ("Disabled" instead
of "Enabled"). The negative is less harmful if, for some reason, it's wrongly
absent (say, because of some future Ping code branch).
gravitational/teleport.e#514
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
Wire device authentication into `tsh`, so it attempts to acquire device
certificates after user login. This affects direct logins (`tsh login`),
indirect logins (RetryWithRelogin) and Connect.
If authentication fails (non-Enterprise cluster, device not enrolled, etc) `tsh`
proceeds as usual, but the final user certificate won't contain device
extensions.
gravitational/teleport.e#514
* Allow connections to nodes when Auth is offline
`tsh ssh user@foo` currently fails if Auth is unreachable. Prior to
connecting to the target, `tsh` sends a `proto.AuthService/IsMFARequired`
request to determine if per-session MFA is required. If necessary, the MFA
ceremony then occurs before `tsh` attempts to connect to the node.
However, nodes ultimately makes the decision on whether the user has
access. To both increase availability of nodes and reduce initial
connection latency, `tsh` can first attempt to connect to a node and
fallback to checking if per-session MFA is required and then performing
the ceremony if necessary followed by attempting another connection to the
node. The major tradeoff is now when per-session MFA is required, `tsh`
will connect to the target **twice**. The added latency is likely to be
minimized by the amount of time it takes to perform the MFA ceremony.
## Connection flows prior to this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: no
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
## Connection flows after this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
* Add local proxy middleware for db cert checking
* Use tls conversion util instead of inline
* Add middleware to local proxy config
* Add middleware configuration in tsh
* Use route to database check and set defaults func
* Dont trigger normal db login flow if using local proxy tunnel
* Split out adding client creds into helper func for testing
* Add integration test for local proxy tunnel db cert middleware
* Add unit test for local proxy middleware
* Update comment
* Make middleware on new conn block
* godoc
* Make any cert check error trigger cert renewal in local proxy middleware
* Move dbcertchecker into lib/client
* Remove unneeded mutex in local proxy and unused func in lib/utils
* Make local proxy middleware integration test more robust
* Print message before mfa prompt in proxy tunnel
* Add before prompt option to test
* Remove unneeded comment
* Change local proxy messages to be more clear
* Pass local proxy opts by reference
* Pass certs in opts instead of cert/key file path
This is so we can check if the error is recoverable while preparing
local proxy options. A tunneled local proxy can ignore the error because
it does not rely on cert files - it can just renew its certs if
necessary.
* Move db route checking back to tsh
* Fix lint err
* Fix typo and print the hint to same writer as the mfa prompt
Label based tsh was sending UUID:0 as the node name to the
IsMFARequired check. The server was checking this name against each
nodes name and uuid. However the port included in the name prevented
any matches from being found. Which resulted in the ssh session being
rejected with an access denied message by the node because mfa was
required but never performed.
This removes the :0 from the node name to the IsMFARequired check and
adds a test case to TestSSHOnMultipleNodes that uses role based
per session mfa.
* switch underlying protocol used for 'tsh scp' to SFTP
* address TODO
* appease linter
* add method to make it easier for other callers to transfer files
* add tests
* print transfer progress with progress bar by default
Also allow a SIGINT to gracefully stop the SFTP connection. This is
necessary because the progress bar will ignore signals and prevent the
process from exiting.
* address SFTP fork issues
* make tests less flakey
* fix specifying dir for dst not copying files to correct paths
* make tests less flakey (again)
* don't check file access times, often differs when run in CI
* few small fixes from review, simplify Create method now that HTTP FS isn't needed
* create dst files and dirs with src mode
* improved error messages when doing file operations
* expand home dirs in remote paths
* addressed more feedback
* add license to get_home_dir.go
* address minor feedback of tests, add home dir expansion test
* update sftp fork to point to latest commit on master branch
* addressed feedback
* don't cache home dir lookups, only one remote path can ever be used
* Reduce number of auth dials for tsh commands
One of the major areas of latency for `tsh ssh` is creating multiple
auth clients. Since the auth client is lazy and only actually performs
the dial on first use we can create an auth client once and simply
reuse it. This is done by adding an `auth.ClientI` to `ProxyClient`
which is created via `connectToProxy`. All attempts to connect to
the current auth server via the `ProxyClient` will be given the
cached `auth.ClientI`.
The new method of retrieving the current auth client also allowed
to remove a number of calls to `GetSites` which were used to obtain
the current cluster name. The local profile already contains the name
of the cluster and calls to `GetSites` were unnecessary. All instances
which relied on the site name now retrieve from information that the
`ProxyClient` already has.
In an effort to reduce ambiguity and confusion `CurrentClusterAccessPoint`
and `ClusterAccessPoint` were also removed. AccessPoint denotes that
you are connecting to a cache, but the `ProxyClient` is always going
to be hitting the auth server directly. The two have been replaced
with `CurrentCluster` and `ConnectToCluster`, which they were merely
wrappers for anyhow.
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
Label based tsh, `tsh ssh root@foo=bar uptime`, may potentially match
a number of nodes that exceeds the users `max_connections` limit.
However, tsh would blindly connect to all matching nodes and execute
the provided command simultaneously. This results in a number of
commands to fail because the user has too many concurrent sessions
open.
Now that #15641 has introduced running multiple commands inside
an `errgroup.Group` we can adjust the limit such that tsh will
never exceed the users `max_connections` limit. In order to
prevent label based tsh from consuming all of a users connections,
the `errgroup.Group` limit is set to half of `max_connections`.
Closes#4626
While looking up github.com/gokyle/hotp I found some old deprecation warnings
and decided to address them.
* Remove HOTP support
* Update comment on checkOTP
* Remove OTPType
* Remove a few more HOTP references
Switches label based ssh to always use the node UUID instead of its
address. This prevents issues that occurred when dialing by address
if the public_addr isn't set correctly.
Closes#3214
This PR implements the SSH Tester for ConnectionDiagnostic feature.
This feature is also known as Test Connection, part of Teleport
Discover.
The goal here is to provide immediante feedback about a newly added
resource. Can the user connect to it?
We are targetting SSH Nodes as a first ResourceKind.
To test the access to an SSH Node we require the ResourceName and a
login username (ssh principal).
Then a series of checks will occur in two places:
- SSH client in the Web server
- SSH server in the SSH agent
The ssh client creates a new Connection Diagnostic with some initial
state.
Then it tries to build up the necessary SSH config
This already gives us a couple of things to check for:
- does the node exist and does the current user (inherited from
websession) can access it?
- is the node accepting TCP connections (in the specific port)?
- is the node accepting SSH protocol on top of the TCP connection?
Then, the ConnectionDiagnosticID is inject into the certificate and the
SSH Server receives it and will also Append traces into it:
- is the requested principal allowed for the current user?
- does the requested principal exist in the target node?
This is not an exhaustive list of checks.
For a complete list of which checks are verified please see the
TestDiagnoseSSHConnection test.
After all those checks, it returns if the Connection was successful and
what all of the traces generated along the way.
Demo:
Michael
