* Connect: Remove leftovers from resource cache removal
* GetAll* RPCs were used in the resource cache and are no longer used.
* ListApps was pretty much never actually used as we didn't ever list apps
in Connect.
* ClusterContext.state.searchValue ceased to be used after we moved to
fetching resources from the backend.
* Remove remaining leftovers from tshd client mock
* Remove unused imports
While loading a key from storage, Teleport client transverses the
storage directory to read the secrets and reads each file one by one.
This has huge impacts on `tsh kube credentials` because when calling
`kubectl` in parallel, the disk operations become overloaded and the
operation time increases exponentially.
This PR removes the transverse when reading credentials for Kubernetes
clusters and adds a quick read operation that reads the following files:
- `$TSH_HOME/current_profile`
- `$TSH_HOME/$profile.yaml`
- `$TSH_HOME/keys/$PROXY/$USER-kube/$TELEPORT_CLUSTER/$KUBE_CLUSTER-x509.pem`
- `$TSH_HOME/keys/$PROXY/$USER`
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old direct | 9.90s user 3.14s system 64% cpu 20.284 total | 11.08s user 3.60s system 496% cpu 2.955 total | 12.46s user 4.36s system 820% cpu 2.050 total | 12.81s user 4.76s system 811% cpu 2.164 total | 13.02s user 5.07s system 861% cpu 2.100 total|
| old teleport | 18.45s user 7.80s system 51% cpu 51.085 total | 21.58s user 10.03s system 434% cpu 7.270 total | 22.33s user 10.52s system 178% cpu 18.402 total | 22.04s user 10.83s system 96% cpu 33.977 total | 22.57s user 11.45s system 55% cpu 1:01.34 total|
| new direct | 9.51s user 2.90s system 146% cpu 8.460 total | 13.75s user 4.95s system 718% cpu 2.603 total | 12.21s user 4.43s system 832% cpu 1.999 total | 11.72s user 4.50s system 839% cpu 1.933 total | 12.06s user 4.62s system 830% cpu 2.009 total |
| new teleport | 16.10s user 5.97s system 42% cpu 52.104 total | 21.62s user 9.01s system 381% cpu 8.023 total | 23.22s user 10.11s system 540% cpu 6.162 total | 21.64s user 9.81s system 603% cpu 5.208 total | 20.70s user 9.13s system 602% cpu 4.948 total |
---
| - | P=1 | P=10 | P=40 | P=80 | P=120 |
| --- | --- | --- | --- | --- | --- |
| old no tsh kube credential | 8.61s user 2.99s system 31% cpu 36.336 total | 9.57s user 3.39s system 283% cpu 4.565 total | 10.22s user 4.03s system 553% cpu 2.574 total | 9.32s user 3.82s system 559% cpu 2.351 total | 11.49s user 4.85s system 624% cpu 2.617 total
| new no tsh kube credential | 8.27s user 2.94s system 24% cpu 45.041 total | 10.85s user 4.31s system 195% cpu 7.767 total | 11.32s user 4.64s system 469% cpu 3.401 total | 10.83s user 4.51s system 553% cpu 2.773 total | 10.90s user 4.84s system 537% cpu 2.930 total |
---
| instances | old total | old avg | **new total** | **total avg** |
| --- | --- | --- | --- | --- |
| 1 | 0.145 | 0.145 | 0.074 | 0.074 |
| 10 | 0.430 | 0.043 | 0.146 | 0.0146 |
| 20 | 0.740 | 0.037 | 0.284 | 0.0142 |
| 30 | 1.108 | 0.037 | 0.417 | 0.0139 |
| 40 | 1.483 | 0.037 | 0.580 | 0.0145 |
| 60 | 13.604 | 0.228 | 0.825 | 0.0137 |
| 80 | 30.236 | 0.378 | 1.143 | 0.0142 |
| 106 | 46.681 | 0.440 | 1.413 | 0.0133 |
Fixes#19313
Co-authored-by: Brian Joerger <bjoerger@goteleport.com>
* services.usagereporter to lib/usagereporter/teleport
* teleterm/daemon/usagereporter to usagereporter/daemon
* usagereporter/daemon type aliases
* Move UsageAnonymizable newtypes in a separate file
* DiscardUsageReporter should be passed by value
* Type renames in usagereporter/teleport
* Move audit event to usage event conversion to lib/usagereporter
* package import renames
* Don't break the e build for now
* move web usage report events to lib/usagereporter
* forgot license blurb
* lib/usagereporter/discover.go to types_discover.go
This commit adds a user message ("Detected security key tap") that is
printed at the earliest opportunity after `tsh` succesfully detects a
security key tap.
This solves a problem users have when there is significant latency in
their connection to their teleport proxy, where there may be zero
visible feedback to the user for upwards of 8 seconds after they tap
their security key.
This often causes users to think that the tap was not detected or
failed, and they attempt to tap again.
This can cause the user to accidentally paste a "yubisneeze" into their
terminal, which is not great from a security perspective.
As an API choice here I added a `TouchAcknowledger` returned by they
`PromptTouch` of the `LoginPrompt` interface.
It may seem like a bit of a weird API, but I did it this way so that all
callers of `PromptTouch` are forced to explicitly acknowledge the touch
or explicitly ignore that return value, they cannot simply forget to ack.
This applies to WebAuthn and Passwordless logins from `tsh`, no changes
are made to Connect.
The Electron app no longer asks tshd to restart gateways. This was changed
in gravitational/webapps#1383.
This PR merely removes the tshd implementation of that RPC.
* Add integration/teleterm_test with tests for get* and list* clusters
Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>
* Move `testAddingRootCluster to integration/teleterm_test
* Add `ClusterWithDetails` type instead of enriching existing `Cluster` with optional properties
* Remove unnecessary condition
* Add license
* Bunch of renames
* Do more in mustLogin
* Remove watcher
---------
Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>
* Adjust go_package of lib/prehog Go protobufs
This makes them follow the pattern set out by api/proto and proto.
* Adjust go_package of lib/teleterm Go protobufs
* Use single buf.gen.yaml to generate JS protos
This also entailed changing the location of lib/teleterm protos and changing
the value of their package specifier to match the conventions in other parts
of the codebase. This is a breaking change but that is fine for Connect
as the protos are used locally only and each build ships with matching
protobufs.
* Make web/packages/teleterm use protobufs from gen-proto-js
We used to copy protobufs over to web/packages/teleterm/src/services/tshd
since webapps used to be in a separate repo.
This is no longer the case, so we can just make teleterm use protobufs
from gen-proto-js.
* Move prehog & teleterm protos into proto/teleport/lib
* Generate JS protos to gen/proto/js
* Move lib/teleterm Go protobufs to gen/proto/go
* Move lib/prehog Go protobufs to gen/proto/go
* Rename lib/teleterm proto package
* Re-enable linter rules for teleterm & prehog
* Update prehogv1 path in usagereporter_test.go
* Use except instead of ignore_only to allow Google API-style responses
* Add UNARY_RPC to api/proto & proto
* Ignore gen/ when running addlicense
* buf-js.gen.yaml: Remove comment about lack of go_package for JS
* Move prehog protos to proto/prehog/v1alpha
* Adjust prehog's go_package to match proto package
* Our current SFTP implementation uses a single channel to transfer data. Connections with a high latency cause very slow transfer. github.com/pkg/sftp allows transferring data using concurrent streams, which seems to solve the problem. In order to utilize the feature, the remote file must implement the WriteTo() or ReadFrom() and Stat() methods. I've moved the progress bar to relay the data from a local file, as we can still call Read() or Write() on it to update the progress.
This PR doesn't address the high CPU usage by github.com/pkg/sftp on low latency links.
Closes#20579
* Cleanup
* Fix fileTransferProgress
* Simplify interfaces
* Move helper to utils package
Add comments
* Added comments
* Change canceler implementation to return correct values from Read/Write
* Fix infinite recursive call in cancelWriter.
* Save private key policy in profile.
* Fix tsh status/logout with yubikey unplugged; fix cluster alert on tsh status with hardware_key_touch.
* Set private key policy from ping response.
* Skip cluster alerts on tsh status.
* Remove CLANG_FORMAT from Makefiles
It was used to format protos but we use Buf for that since v10.
* Move installing grpc_node_plugin into Dockerfile
This commit basically takes grpc_node_plugin compilation from
Dockerfile-teleterm and moves it to Dockerfile.
* Replace Dockerfile-teleterm with Dockerfile
After moving grpc_node_plugin compilation to Dockerfile, the only remaining
thing that Dockerfile-teleterm does is installing rpm so that we can make
an RPM package for Connect during tag builds.
Installing this package can be simply moved to Dockerfile.
* Remove grpc-teleterm Make target in favor of grpc
* Add updated protobufs
It looks like they're a result of someone changing protos in lib/prehog
without running `make grpc-teleterm` separately. Which is why we're getting
rid of grpc-teleterm as a separate Make target in the first place. ;)
* Add integration test for daemon.Service.AddCluster
* Call SaveProfile on clusterClient rather than cfg
This way we don't have to explicitly set ClientStore as
client.NewClient(cfg) does that for us.
Back in the day, the ping response didn't include the actual name of
the cluster. So instead we had to make a separate request for /web/config.js.
However, the ClusterName field was added to /webapi/ping in v10 (#12848).
Since we already ping the server, let's use the name from the ping response
instead.
- Add a generalized client store made up of a key, profile, and trusted certs store. Each sub store can support different backends (~/.tsh, identity_file, in-memory).
- Replace custom identity file handling with in-memory client store.
- Fix issues with trusted certs handling.
Wire device authentication into `tsh`, so it attempts to acquire device
certificates after user login. This affects direct logins (`tsh login`),
indirect logins (RetryWithRelogin) and Connect.
If authentication fails (non-Enterprise cluster, device not enrolled, etc) `tsh`
proceeds as usual, but the final user certificate won't contain device
extensions.
gravitational/teleport.e#514
* Add TTL field to integration/helpers.UserCredsRequest
This will let us create expired user certs by providing a negative TTL.
* Reissue gateway cert if middleware detects it expired
* Add integration test for gateway cert renewal
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
* Log gateway.Close errors during test cleanups
Unless gateway.Close is called on a gateway that was already closed, it
shouldn't return an error.
However, when working on handling expired certs in Connect I ran into a
buggy test where that error from gateway.Close provided a crucial clue
in fixing the bug. But because initially it was simply ignored, it took
me a while to figure out what was going on.
That's why this commit adds logging around those errors.
2 out of those 3 places are helper functions which get used in a variety
of tests, hence why they call t.Cleanup. The other place does call
gateway.Close eventually but we still use t.Cleanup in case the
execution doesn't get to that point.
* Automatically add useful fields to gateway loggers
It's useful to see what resource the gateway is targeting and what is
the URI of the gateway.
Previously the field with URI was hardcoded in cluster_gateways.go or
added only when cfg.Log was nil, meaning that we weren't able to benefit
from it in places such as gateway_test.go.
This commit makes it so that the `resource` and `gateway` fields are
added to any logger that is passed through gateway.Config.
* Remove copylocks warning from Gateway.NewWithLocalPort
gateway.Gateway holds a mutex in one of its fields. NewWithLocalPort
accepted gateway by value so vet was issuing a warning about copying a
lock.
NewWithLocalPort doesn't actually use the copied lock. But it makes
sense to get rid of the warning anyway.
* Make ReissueDBCerts accept tlsca.RouteToDatabase as arg
ReissueDBCerts used to accept a full-blown types.Database object just to
read a couple of fields from it. In the context of Connect, such object
is obtainable only by making a request to the cluster.
However, in the upcoming PR we want to be able to reissue the cert
without having to perform an unnecessary request to the cluster.
gateway.Gateway already holds all data we need to reissue the cert, so
let's make ReissueDBCerts accept tlsca.RouteToDatabase instead of
types.Database to avoid making that extra request.
* Add Gateway.ReloadCert
In the upcoming PR, after we reissue the db cert, we need to be able to
update the cert used by the running alpn.LocalProxy. This commit exposes
exactly that functionality.
Also, this commit adds RWMutex to Gateway to avoid a situation where
multiple goroutines attempt to reload the cert. This shouldn't happen
under normal circumstances but better safe than sorry.
RWMutex is also used for any field on Gateway that has a setter.
* Add basic implementation of LocalProxyMiddleware
* Add OnExpiredCert callback to gateway.Config
This callback will let the layer above gateway.Gateway handle a
situation in which the cert used by the gateway has expired but there's
a client that tries to make a connection through the gateway.
gateway.Gateway doesn't have the ability to reissue the cert by itself,
hence why we need to accept a callback from above.
* Refactor error handling for GetConfigForClient
If GetConfigForClient returns an error, the error is not visible in any
logs or by the client making the request.
Instead of failing, we return a config that doesn't let any client
through. It has ClientAuth set to RequireAndVerifyClientCert, but the
config lacks ClientCAs to verify the client cert against.
This also means that when a client with an invalid cert dials the server,
it's going to fail on net.Conn.Read and not tls.Dialer.DialContext. This
will help us add uniform tests in the next commit.
This use of GetConfigForClient is more similar to what we do in other
parts of the codebase, for example in lib/srv/db/proxyserver.go.
* Add tests for teleterm.Serve with TCP address
Previously we'd only test teleterm.Serve with a unix socket, meaning our
whole TCP setup would not be tested.
Testing the TCP server means that we need to set up proper TLS configs
for clients in tests.
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
To enable feature detection in the Connect application, we need to
ping the auth server to understand which features are enabled.
Previously, we could get away with any cluster information stored in the
cluster profile but a proxy dial is necessary now to get an auth ping response.
* Connect: Accommodate for making gRPC client creds from tshd key pair
For tshd-initiated communication, the tshd process will need to create a
client that will connect to a gRPC server operated by the renderer
process of the Electron app.
On Windows, we use gRPC over TCP with mTLS. Each process creates its own
keypair and saves the public key to a predetermined location.
The previous code assumes that tshd is only going to need server
credentials. This commit makes it possible to create client credentials
from the same key pair.
* Refactor server options
* Expand the comment for createServerCredentials
* Remove unnecessary filepath.Join
* generateAndSaveCert: Use os.CreateTemp
Moves from github.com/golang/protobuf protoc-gen-go plugin to google.golang.org/
plugins.
This change was a long-time coming, but is now possible to do since our
dependencies are up-to-date.
* Move away from deprecated protoc-gen-go plugin
* Embed unimplemented server in handler.Handler
* Embed unimplemented server in multiplexer_test.go
* Update generated protos
* access requests for teleterm
* removed unused imports and named returns
* remove comment
* using timestamppb instead of string for access requests
* updated proto with some more comments
* updated protos with comments
* using clusterClient, comments, and moving validation to daemon for access request delete
* separated GetAccessRequests into separate RPCs
* protobuf updates
* moved requestid check before resolving cluster
* fullstops in comments
* used standard access_request_id through rpc messages
* updated protofiles
* updated daemon service types to match grpc
* added kube advanced search support
* updated protos for kubes in access requests
* testing tag build
* fix detached head
* new tag build
* protobuf update
* lint fixes
* allow drone windows Connect build to include webapps.e
* protobuf files
* remove drone changes and updated comment
* proto changes with comment fixes and changed field order
* protobuf updates
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Rafał Cieślak