To prevent users from being able to directly connect to registered
OpenSSH (Agentless) nodes, the OpenSSH CA was introduced. When
connecting to Agentless nodes, the Proxy will forward the connection
so the session can be recorded. The forwarding SSH server is configured
to authenticate to the target node with a certificate signed by OpenSSH
CA, which users do not have access to.
* Move configuration from lib/service to lib/service/servicecfg
The new servicecfg package will hold only configuration for services.
This will allow other packages (like tctl and tsh) to depend on
servicecfg without pulling in all of lib/service (which has a number
of platform-specific details).
This is the first step towards being able to build tctl for Windows.
* Move PAM and BPF config into servicecfg
This breaks a compile-time dependency on BPF/PAM for tctl.
* Move the authorizer into its own package.
The Authorizer and associated functions and constants have been moved into
their own package `authz`. This will allow us to more easily create separate
services for gRPC functionality so that we're not pushing as much into the
`lib/auth` package.
Though this PR is large, the only actual code change here is making
`ContextForLocalUser` exported and changing the `permissions_test.go` so that
it doesn't rely on the `lib/auth` package.
* Use context functions.
* Test and linting fixes.
* Add in a temporary alias file to ensure e still compiles.
* Use functions instead of var aliases for performance reasons.
* Adding in godocs, cleaning up tests.
* Fix tests.
* Add a few more aliases.
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* sshserver: Correctly handle PuTTY winadj channel requests
Fixes#19365
* Actually send a failure
* dispatch changes
* Reimplement wantReply handling
* Also handle windadj in forwarding server
* lint: Fix typo
* lint: Invert return order to put err last
* Use nil return payload instead
* Use trace.ReplyAlreadyHandled instead
* Handle reply internally and set req.WantReply to false
* Use t.Cleanup()
* behaviour is not a misspelling but whatever makes you happy
* Suggestions from code review
Migrates metrics counting proxied connections and connection attempts
from `regular/proxy.go` to `proxy.Router` so that connections via
the web UI, tsh ssh via the Proxy ssh and grpc servers are all
accounted for.
Introduces `limiter.Listener` to provide a consistent and reusable
mechanism for limiting incoming connections per client. The new
listener is used by `sshutils/server.go` instead of manually applying
limits in `HandleConnection`.
This is particularly important now that the Proxy SSH port multiplexes
both SSH and gRPC. Each listener is now wrapped in a `limiter.Listener`
that uses the same `limiter.ConnectionsListener` to ensure that the
connection limits for the Proxy are enforced for all traffic on the
port.
Ensures that active connections during a graceful shutdown don't
inadvertently prolong the heartbeat.
The added ReadyEvents ensure that TeleportReady is broadcast if
either the tracing_service or metrics_service is enabled. Not emitting
them prevents graceful shutdown from completing because the uploader
service requires the TeleportReady event before it can complete initialization.
Converts usage of `newFixture` to `newFixtureWithoutDiskBasedLogging`
to prevent directory not empty errors caused by `t.TempDir` still
containing upload parts.
Fixes#19826
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
* Fix web ssh session with proxy recording mode
The agent was not being propagated when establishing an ssh session
via the web which resulted in the error described in #18850. Providing
the agent was straightforward, however, due to the changes from #18656
when using the forward server, as is required with proxy recording mode,
the ssh connection is now being performed directly over a `net.Pipe`.
Due to the synchronous nature of `net.Pipe` this causes a deadlock when
performing the ssh handshake. To mitigate the deadlocks, `DualPipeNetConn`
was changed to leverage `syscall.Socketpair` instead of `net.Pipe`.
`TestTerminal` now has two cases, one for node recording mode and another
for proxy recording mode. In order for the proxy recording mode test to pass
the fake clock used in the test needed to be properly propagated to the
`ssh.CertChecker` and `forward.ServerConfig`.
Fixes#18850
* Ensure ssh connection rejection errors are returned
#18302 altered the connection flow which changed the communication
via ssh. Prior to the change an ssh channel was opened first, then
global requests were sent. After the change a global request is sent
prior to opening a channel. This caused users to no longer receive a
human readable rejection message a la: `ssh: rejected: administratively
prohibited (too many concurrent ssh connections for user`, instead only
an `EOF` was returned.
To ensure that the correct message is provided to users two changes are
made, although only the client side change is strictly required.
1) the tracing ssh client now returns any rejected errors cached from
opening the tracing channel
2) the ssh server now processes global requests during it's wait period
waiting for a channel to reject and send the replies to any requests
with said rejection
`TestLockInForce` was updated to include attempting to send a global
request so both scenarios are covered as order should not matter.
Session control logic existed within `HandleNewConn` of `srv/regular.Server`.
This prevented any of it from being used by other components that
also needed to enforce session control.
All the logic from within `HandleNewConn` was refactored to a new
`srv.SessionController` object which the `regular.Server` now uses
to perform session control. There were a few additional changes
needed to accomadate that session control now exists outside
the server and to make tests easier to write. Namely, altering
`srv.ComputLockTargets` to not take a `Server` as a parameter and
leveraging a clock within `services.SemaphoreLock`.
This is step 2 in addressing #15167. Before the web apiserver can
leverage the newly introduced proxy.Router and bypass making ssh
connections to the proxy ssh server it needs to be able to perfrom
session control.
* create package to contain proxy peering code
* Refactor proxy routing logic into a reusable object
Routing logic existed within an unexported handler of ssh subsystem
requests, which prevented it from being reused by other components
within the proxy, like the webapi server. This causes significant
latency issues for web sessions because the web apiserver is required
to dial the proxy ssh server to determine how to route to the host.
Since the web apiserver and the proxy ssh server exist in the same
process this is an entirely unnecesarry step that could be avoided
if the routing and ability to established connections were shared
throughout the proxy.
A new `proxy.Router` object is introduced which contains all the
logic that used to exist in `regular.proxySubsys` for determining
how to connect to servers and clusters. All routing within the
`regular.proxySubsys` now leverages the `proxy.Router` to dial
the target.
This is step 1 in addressing #15167. Now that the `proxy.Router`
exists `web.APIServer` will be able to make use of it to avoid
dialing the same process to establish connections.
* Allow connections to nodes when Auth is offline
`tsh ssh user@foo` currently fails if Auth is unreachable. Prior to
connecting to the target, `tsh` sends a `proto.AuthService/IsMFARequired`
request to determine if per-session MFA is required. If necessary, the MFA
ceremony then occurs before `tsh` attempts to connect to the node.
However, nodes ultimately makes the decision on whether the user has
access. To both increase availability of nodes and reduce initial
connection latency, `tsh` can first attempt to connect to a node and
fallback to checking if per-session MFA is required and then performing
the ceremony if necessary followed by attempting another connection to the
node. The major tradeoff is now when per-session MFA is required, `tsh`
will connect to the target **twice**. The added latency is likely to be
minimized by the amount of time it takes to perform the MFA ceremony.
## Connection flows prior to this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: no
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
## Connection flows after this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
* switch underlying protocol used for 'tsh scp' to SFTP
* address TODO
* appease linter
* add method to make it easier for other callers to transfer files
* add tests
* print transfer progress with progress bar by default
Also allow a SIGINT to gracefully stop the SFTP connection. This is
necessary because the progress bar will ignore signals and prevent the
process from exiting.
* address SFTP fork issues
* make tests less flakey
* fix specifying dir for dst not copying files to correct paths
* make tests less flakey (again)
* don't check file access times, often differs when run in CI
* few small fixes from review, simplify Create method now that HTTP FS isn't needed
* create dst files and dirs with src mode
* improved error messages when doing file operations
* expand home dirs in remote paths
* addressed more feedback
* add license to get_home_dir.go
* address minor feedback of tests, add home dir expansion test
* update sftp fork to point to latest commit on master branch
* addressed feedback
* don't cache home dir lookups, only one remote path can ever be used
* Allow tsh to retrieve cluster details in one request
Prior to connecting to a node via `tsh ssh` we need two bits of
information about the cluster:
1) The session recording mode
2) Whether FIPS is enabled
In order to retrieve the information `tsh` first would send the
global ssh request `RecordingProxyReqType` to determine the
session recording mode. Later on `tsh` would Ping the auth
server to determine if the cluster was running in FIPS mode.
In an effort to reduce the number of round trips to retrieve
this data, a new global ssh request `ClusterDetailsReqType` is
introduced that returns both the session recording mode and
whether FIPS is enabled. This allows `tsh` to leverage the new
request to get all the information it needs, and is extensible
in case more information is needed` in one request which helps
reduce latency for `tsh ssh`.
By only providing the tunnel address from the `reversetunnel.Resolver`
callers would still need to lookup the proxy listener mode to determine
how to dial the address. This results in sending a request to
`/webapi/find` once by the resolver to get the tunnel address and then
a second request to `/webapi/find` by users of the `Resolver` to determine
the proxy listener mode. Propagating the listener mode along with the
tunnel address by the `Resolver` ensures only one `/webapi/find` call
is needed.
This is especially impactful because the `reversetunnel.TunnelAuthDialer`
which is used by the auth http client would do this everytime the
`http.Client` connection pool was empty. When the `http.Client` needed
to dial the auth server it was incurring the additional roundtrip to the
proxy.
* Add initial version of installer
* Resolve comments
- Use aws waiters when checking commands
- Use SSMRunRequest rather than passing instances
- General comments
* Resolve comments, (rebase) pass scriptname parameter
This resolves comments regarding running on multiple ec2 instances at
once by adding state to the instances cache to check if the instance
is known about and how far into installation it is
* Revert cache
* Dont cache on non discovery nodes
* Resolve some comments
* Move discovery out to its own serviec
* Add a `discovery_service` section
* Fix messed up conflict merge
* Make starting a standalone discovery agent work
* Resolve comments
* Resolve comments
- use a regular events.Emitter
- resolve a thousand typos :)
* Resolve comments
* resolve comments, fix a bad merge
* Fail when a non ec2 matcher type is configured
* fix lint-go
* Resolve comments
* Resolve comments, add initial test (currently broken)
* Fix log string so only 1 pair of [] are used
* Chunk instances for sending commands
* add 'isInitialized' to watchers
* Add test for chunked discovery, log output
* lints
* explicetly set matcher.Tags to "*":"*" if its unset
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
Invert the relationship of `lib/srv` and `lib/bpf`,
`lib/restrictedsession` such that `lib/bpf` is only imported in
`lib/srv/regular`. Since not everything is built with the `bpf`
tag it's important to reduce the surface area of `lib/bpf` such
that it isn't inadvertantly imported. For instance it was entirely
possible to import a package in `tsh` that transitively
depends on `lib/bpf` - which breaks the build since `tsh` is not
compiled with the `bpf` tag.
By refactoring the `srv.Server` interface not to use the `bpf.BPF`
and `restrictedsession.Manager` interfaces directly anything that
imports `lib/srv` now won't require that `-tags=bpf` is set in
order to compile.
* Add an AWS EC2 instance fetcher
* Resolve comments
- Remove leftover VPCID stuff
- Document public functions
- Return a notfound when there are no nodes fetched instead of nil nil
* Resolve comments
* Fix fileconf_test.go
Adds a wrapper around `ssh.Session` which injects tracing context
in a similar manner to the `ssh.Client` wrapper. All usages of
`ssh.Session` have now been replaced and have the appropriate
`context.Context` passed along
Part of #12241
## What
First part of the Kubernetes [Discovery RFD](https://github.com/gravitational/teleport/pull/13376/) to introduce a Kubernetes server per cluster.
This PR introduces a separate Kubernetes server that uses the already introduced `KubernetesClusterV3`.
## Compatibility
In previous versions, Kubernetes Clusters were part of regular `ServerV2` resource and this refactoring deprecates the `ServerV2` usage but keeps them for compatibility with previous version.
Everything is backward compatible, so v10 kubernetes agents and trusted clusters can connect fine.
## Next steps
Once this is merged, a new PR will introduce dynamic registration for Kubernetes Clusters discovered through EKS Discovery.
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
Andrew LeFevre