* Move configuration from lib/service to lib/service/servicecfg
The new servicecfg package will hold only configuration for services.
This will allow other packages (like tctl and tsh) to depend on
servicecfg without pulling in all of lib/service (which has a number
of platform-specific details).
This is the first step towards being able to build tctl for Windows.
* Move PAM and BPF config into servicecfg
This breaks a compile-time dependency on BPF/PAM for tctl.
* Move the authorizer into its own package.
The Authorizer and associated functions and constants have been moved into
their own package `authz`. This will allow us to more easily create separate
services for gRPC functionality so that we're not pushing as much into the
`lib/auth` package.
Though this PR is large, the only actual code change here is making
`ContextForLocalUser` exported and changing the `permissions_test.go` so that
it doesn't rely on the `lib/auth` package.
* Use context functions.
* Test and linting fixes.
* Add in a temporary alias file to ensure e still compiles.
* Use functions instead of var aliases for performance reasons.
* Adding in godocs, cleaning up tests.
* Fix tests.
* Add a few more aliases.
* sshserver: Correctly handle PuTTY winadj channel requests
Fixes#19365
* Actually send a failure
* dispatch changes
* Reimplement wantReply handling
* Also handle windadj in forwarding server
* lint: Fix typo
* lint: Invert return order to put err last
* Use nil return payload instead
* Use trace.ReplyAlreadyHandled instead
* Handle reply internally and set req.WantReply to false
* Use t.Cleanup()
* behaviour is not a misspelling but whatever makes you happy
* Suggestions from code review
Introduces `limiter.Listener` to provide a consistent and reusable
mechanism for limiting incoming connections per client. The new
listener is used by `sshutils/server.go` instead of manually applying
limits in `HandleConnection`.
This is particularly important now that the Proxy SSH port multiplexes
both SSH and gRPC. Each listener is now wrapped in a `limiter.Listener`
that uses the same `limiter.ConnectionsListener` to ensure that the
connection limits for the Proxy are enforced for all traffic on the
port.
Converts usage of `newFixture` to `newFixtureWithoutDiskBasedLogging`
to prevent directory not empty errors caused by `t.TempDir` still
containing upload parts.
Fixes#19826
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
* Ensure ssh connection rejection errors are returned
#18302 altered the connection flow which changed the communication
via ssh. Prior to the change an ssh channel was opened first, then
global requests were sent. After the change a global request is sent
prior to opening a channel. This caused users to no longer receive a
human readable rejection message a la: `ssh: rejected: administratively
prohibited (too many concurrent ssh connections for user`, instead only
an `EOF` was returned.
To ensure that the correct message is provided to users two changes are
made, although only the client side change is strictly required.
1) the tracing ssh client now returns any rejected errors cached from
opening the tracing channel
2) the ssh server now processes global requests during it's wait period
waiting for a channel to reject and send the replies to any requests
with said rejection
`TestLockInForce` was updated to include attempting to send a global
request so both scenarios are covered as order should not matter.
Session control logic existed within `HandleNewConn` of `srv/regular.Server`.
This prevented any of it from being used by other components that
also needed to enforce session control.
All the logic from within `HandleNewConn` was refactored to a new
`srv.SessionController` object which the `regular.Server` now uses
to perform session control. There were a few additional changes
needed to accomadate that session control now exists outside
the server and to make tests easier to write. Namely, altering
`srv.ComputLockTargets` to not take a `Server` as a parameter and
leveraging a clock within `services.SemaphoreLock`.
This is step 2 in addressing #15167. Before the web apiserver can
leverage the newly introduced proxy.Router and bypass making ssh
connections to the proxy ssh server it needs to be able to perfrom
session control.
* create package to contain proxy peering code
* Refactor proxy routing logic into a reusable object
Routing logic existed within an unexported handler of ssh subsystem
requests, which prevented it from being reused by other components
within the proxy, like the webapi server. This causes significant
latency issues for web sessions because the web apiserver is required
to dial the proxy ssh server to determine how to route to the host.
Since the web apiserver and the proxy ssh server exist in the same
process this is an entirely unnecesarry step that could be avoided
if the routing and ability to established connections were shared
throughout the proxy.
A new `proxy.Router` object is introduced which contains all the
logic that used to exist in `regular.proxySubsys` for determining
how to connect to servers and clusters. All routing within the
`regular.proxySubsys` now leverages the `proxy.Router` to dial
the target.
This is step 1 in addressing #15167. Now that the `proxy.Router`
exists `web.APIServer` will be able to make use of it to avoid
dialing the same process to establish connections.
* Allow tsh to retrieve cluster details in one request
Prior to connecting to a node via `tsh ssh` we need two bits of
information about the cluster:
1) The session recording mode
2) Whether FIPS is enabled
In order to retrieve the information `tsh` first would send the
global ssh request `RecordingProxyReqType` to determine the
session recording mode. Later on `tsh` would Ping the auth
server to determine if the cluster was running in FIPS mode.
In an effort to reduce the number of round trips to retrieve
this data, a new global ssh request `ClusterDetailsReqType` is
introduced that returns both the session recording mode and
whether FIPS is enabled. This allows `tsh` to leverage the new
request to get all the information it needs, and is extensible
in case more information is needed` in one request which helps
reduce latency for `tsh ssh`.
By only providing the tunnel address from the `reversetunnel.Resolver`
callers would still need to lookup the proxy listener mode to determine
how to dial the address. This results in sending a request to
`/webapi/find` once by the resolver to get the tunnel address and then
a second request to `/webapi/find` by users of the `Resolver` to determine
the proxy listener mode. Propagating the listener mode along with the
tunnel address by the `Resolver` ensures only one `/webapi/find` call
is needed.
This is especially impactful because the `reversetunnel.TunnelAuthDialer`
which is used by the auth http client would do this everytime the
`http.Client` connection pool was empty. When the `http.Client` needed
to dial the auth server it was incurring the additional roundtrip to the
proxy.
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Adds a wrapper around `ssh.Session` which injects tracing context
in a similar manner to the `ssh.Client` wrapper. All usages of
`ssh.Session` have now been replaced and have the appropriate
`context.Context` passed along
Part of #12241
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
* configure golangci-lint misspell to check for anglicized spellings
* Americanize spellings
* fix aws constant value with british spelling 🇬🇧
* update api types with americanized spellings
* use american spellings .cloudbuild/scripts
* Make TestLimiter test less flaky
* Update lib/srv/regular/sshserver_test.go
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* Update lib/srv/regular/sshserver_test.go
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* Remove unneeded helper to wait for server to start and add a comment
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* Deflake TestProxyRoundRobin / TestClientDisconnect
Configure the test environment in a way that a disk-based session
uploader will not be used. This prevents the tempdir cleanup from
failing while the uploader is still writing to the directory.
Fixes#13938
* Apply fix to TestProxyRoundRobin
This test fails similarly to TestAgentForward, because the
file-based session uploader runs in the background and is
sometimes still writing to disk when the test goes to clean
up temp directories.
Since this test has nothing to do with the disk-based uploader
or the audit log, we use a discard emitter and set the session
recording mode to synchronous so there are no files on disk
which need to be uploaded.
* Don't GetAuthServers in AuthProxyDialerService
* Don't GetAuthServers in localSite
* Fix lib/web tests
* Connect to a single auth in transport.start
* Review comments
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Attempt to deflake TestAgentForward
This was another case of the t.TempDir() being cleaned up while the
audit logger is still writing to the directory, which happens when
tests don't properly clean up after themselves. Ensure that any
services we spin up closed via cleanup actions.
* Prevent the use of disk-based logging in TestAgentForward
The disk-based logger runs a background process to complete uploads,
which occaisionally fails to finish before the test cleanup tries
to remove the temporary directory.
There are two ways to prevent the use of a disk based logger:
1. Set IsTestStub on the SSH *ServerContext
2. Use a sync session recording mode
Option 2 was selected, because the ServerContext is created by the SSH
server instead of the test, so plumbing that value through would be
a larger change, and I generally dislike test specific modes that can
be mistakenly enabled in non-test situations.
Additionally, update the lib/srv/regular test fixture to allow for
configuring the audit log to use. This allows us to set up a dicarding
logger, since these tests are about agent forwarding behavior and not
audit logging.
* Add client side circuit breaker to auth clients
In order to apply back pressure we can utilize a circuit breaker that
monitors error responses from auth server. When tripped it will prevent
all outbound requests to auth for a period of time. This can also help
prevent a potential thundering heard when auth is in an unhealthy state.
By default the circuit breaker will only be tripped if 90% of the
requests made in the monitoring interval fail.
This adds proxy peering support. A configurable setting that allows for agents
to connect to a subset of proxies and be reachable through any proxy in the
cluster. This is achieved by creating grpc connections between each proxy
server. Client connections can then be passed between proxies to the desired
agent.
* Improve CertAuthorityWatcher
CertAuthorityWatcher and its usage are refactored to allow for
all the following:
- eliminate retransmission of the same CAs
- reduce memory usage by having one local watcher per proxy
- adds the ability to filter only the CAs that are desired
- reduce the time required to send the first CAs
watchCertAuthorities now compares all CAs it receives from the
watcher with the previous CA of the same type and only sends to
the remote site if they are not identical. This is to reduce
unnecessary network traffic which can be problematic for a
root cluster with a larger number of leafs.
The CertAuthorityWatcher is refactored to leverage a fanout
to emit events to any number of watchers, each subscription
can be for a subset of the configured CA types. The proxy
now has only one CertAuthorityWatcher that is passed around
similarly to the LockWatcher. This reduces the memory usage
for proxies, which prior to this has one local CAWatcher per
remote site.
updateCertAuthorities no longer waits on the utils.Retry it
is provided with before starting to watch CAs. By doing this
the proxy no longer has to wait ~8 minutes before it even
starts to watch CAs.
When proxying connections, PuTTY automatically requests the 'simple@putty.projects.tartarus.org' channel
type to indicate that this is a "simple" connection which will not be requesting other channel formats.
A request to Teleport for this channel currently results in the connection being terminated. A better
idea is just to ignore this request.
* Update golangci-lint
To accomodate the recent Go 1.18 upgrade
* Fix new lint warnings as a result of linter upgrade
* Set golangci-lint to Go 1.18 mode
golangci-lint will automatically skip linters that don't have support
for Go 1.18.
See: https://github.com/golangci/golangci-lint/issues/2649
* Prevent proxy from loading entire node set into memory more than once
When establishing a new session to a node, the proxy would load the
entire node set into memory in an attempt to find the matching host. For
smaller clusters this may not be that problematic. But on larger clusters,
loading >40k nodes into memory from the cache can be quite expensive.
This problem is compounded by the fact that it happened**per** session,
which could potentially cause the proxy to consume all available memory
and be OOM killed.
A new `NodeWatcher` is introduced which will maintain an in memory list
of all nodes per process. The watcher leverages the existing resource
watcher system and stores all nodes as types.Server, to eliminate the
cost incurred by unmarshalling the nodes from the cache. The `NodeWatcher`
provides a way to retrieve a filtered list of nodes in order to reduce the number
of copies made to only the matches.
* Replace Upload completer grace period logic with session tracker checking to accurately determine whether an upload has been abandoned
* Update session tracker expiration to be 1 hour, and dynamically extend it while the session is active.
Zac Bergquist