* clean up pidfile and reloads
* reduce log spam during graceful shutdowns
* hold an exclusive advisory lock on the pidfile
* do systemd reloads with the pidfile rather than $MAINPID
* prioritize signals over internal restarts
* Add pre- and post-detect hooks to the multiplexer
This gets rid of the dedicated "already written" mechanism in
multiplexer.Conn.
* Add a way for net.Conn to override ServerVersion
* Resumption handshake, client and server
* Enable SSH resumption in teleport and tsh
* Disable resumption in a subtest of TestIntegrations/Disconnection
* Add back the removed tests
* minor refactor of runClientResumable
* Disable resumption for a different subtest
* Add an option to disable SSH connection resumption in Connect (#36712)
Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>
Co-authored-by: Grzegorz Zdunek <gzdunek@users.noreply.github.com>
---------
Co-authored-by: Rafał Cieślak <rafal.cieslak@goteleport.com>
Co-authored-by: Grzegorz Zdunek <gzdunek@users.noreply.github.com>
`x/crypto/ssh` is adding a `fipsonly` mode (similar to `crypto/tls/fipsonly`) in
golang/go#64769. Once this has landed and been released, we'll swap to using it.
In the meantime, update our hardcoded algorithm set to match the algorithms listed
in https://go-review.googlesource.com/c/crypto/+/550515.
The only difference is that `aes192-ctr` is kept in `FIPSCiphers`, as it seems to be
approved by NIST SP 800-131A rev 2 (page 17).
Additionally, we now define a list of approved public key authentication algorithms for
SSH server connections. This isn't configurable for normal use case (yet), only handled
for FIPS builds.
Ensure that `proxy_ssh_sessions_total` is incremented in both the
root and the leaf cluster if a session to a node in the leaf cluster
is established through the root cluster. This now also prevents
leaf clusters from terminating during a graceful shutdown if there
is an active connection that was created from the root.
Fixes#32605.
The ssh session was not being closed for web sessions which resulted
in zombie sessions being left around until the ssh service was
restarted. TestTerminal was updated to assert that the session
tracker eventually transitions to the terminated state when the
client terminates the web socket.
Fixes#32120
* Make SSH node listener to use multiplexer to leverage parsing of signed PROXY headers
Before for backward compatibility reason we used regular listener and later
we wrapped connection into handler that could parse signed PROXY header or
proxyHelloSignature for getting correct user IP. But
after removal of proxyHelloSignature we don't need the wrapper anymore.
* Improve error message
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
---------
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Allow node to handle old and new way of client IP propagation on same listener
With addition of signed PROXY headers, node was listening on multiplexer, but because
of that it couldn't processing incoming connection from older proxies
when ProxyHelloSignature was used, because
both ends were waiting for the other side to send data first.
Here we integrate ability to handle PROXY headers into connection itself,
so we can start ssh server without waiting for multiplexer to detect connection
* Remove unneeded code
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Improve comment's wording
Co-authored-by: Michael Wilson <mike@mdwn.dev>
* Fix imports
* Unexport function
* Add godoc
* Remove unneeded comment
* Move ProxyHelloSignature to constants
* Check that proxyline is verified
* Use Warn() instead of Warnf()
* Move ProxyHelloSignature to api/constants
* Add timeout for getting host CA during proxyline verification.
* Rearrange conditions
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Clarify comment.
---------
Co-authored-by: Michael Wilson <mike@mdwn.dev>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
Introduces `limiter.Listener` to provide a consistent and reusable
mechanism for limiting incoming connections per client. The new
listener is used by `sshutils/server.go` instead of manually applying
limits in `HandleConnection`.
This is particularly important now that the Proxy SSH port multiplexes
both SSH and gRPC. Each listener is now wrapped in a `limiter.Listener`
that uses the same `limiter.ConnectionsListener` to ensure that the
connection limits for the Proxy are enforced for all traffic on the
port.
* Ensure ssh connection rejection errors are returned
#18302 altered the connection flow which changed the communication
via ssh. Prior to the change an ssh channel was opened first, then
global requests were sent. After the change a global request is sent
prior to opening a channel. This caused users to no longer receive a
human readable rejection message a la: `ssh: rejected: administratively
prohibited (too many concurrent ssh connections for user`, instead only
an `EOF` was returned.
To ensure that the correct message is provided to users two changes are
made, although only the client side change is strictly required.
1) the tracing ssh client now returns any rejected errors cached from
opening the tracing channel
2) the ssh server now processes global requests during it's wait period
waiting for a channel to reject and send the replies to any requests
with said rejection
`TestLockInForce` was updated to include attempting to send a global
request so both scenarios are covered as order should not matter.
* Allow tsh to retrieve cluster details in one request
Prior to connecting to a node via `tsh ssh` we need two bits of
information about the cluster:
1) The session recording mode
2) Whether FIPS is enabled
In order to retrieve the information `tsh` first would send the
global ssh request `RecordingProxyReqType` to determine the
session recording mode. Later on `tsh` would Ping the auth
server to determine if the cluster was running in FIPS mode.
In an effort to reduce the number of round trips to retrieve
this data, a new global ssh request `ClusterDetailsReqType` is
introduced that returns both the session recording mode and
whether FIPS is enabled. This allows `tsh` to leverage the new
request to get all the information it needs, and is extensible
in case more information is needed` in one request which helps
reduce latency for `tsh ssh`.
* Add additional debug logging for instantiation of ssh server conn
* Rough out introducing mux to ssh listener
* Catch errors from multiplexer serve
* Adapt multiplexer to support ProxyHelloSignature
* Adjust logging for ssh conn creation
* Adjust tests to use semantically correct assert/require helpers
* Address PR comments
* Check Client addr is properly propagated in mux test
* SSH request tracing
Add tracing support for ssh global requests and channels. Wrappers
for `ssh.Client`, `ssh.Channel`, and `ssh.NewChannel` provide a
mechanism for tracing context to be propagated via a `context.Context`.
In order to maintain backwards compatibility the ssh.Client wrapper
tries to open a TracingChannel when constructed. Any servers that
don't support tracing will reject the unknown channel. The client
will only provide tracing context to servers which do NOT reject
the TracingChannel request.
In order to include pass tracing context along all ssh payloads
are wrapped in an Envelope that includes the original payload
AND any tracing context. Servers now try to unmarshal all payloads
into said Envelope when processing messages. If there is an Envelope
provided, a new span will be created and the original payload will
be pass along to handlers.
Part of #12241
* configure golangci-lint misspell to check for anglicized spellings
* Americanize spellings
* fix aws constant value with british spelling 🇬🇧
* update api types with americanized spellings
* use american spellings .cloudbuild/scripts
This change adds IP-based validation for SSH certificates.
There's new option in role definition:
kind: role
metadata:
name: dev
spec:
options:
pin_source_ip: true
When that is set to true client IP must be the same when generating certificates and using them. It uses source_address critical option that should be supported by both teleport and sshd and only applies to certificates we send to user (like in tsh login), we don't pin IP in certificates issued for web UI as they can't leak.
This change also omits machine ID (it uses different code path) - it will be added in separate PR.
Most of the lines changed are from regenerating types.proto, change itself is not that big
Relates #11719
This adds proxy peering support. A configurable setting that allows for agents
to connect to a subset of proxies and be reachable through any proxy in the
cluster. This is achieved by creating grpc connections between each proxy
server. Client connections can then be passed between proxies to the desired
agent.
* Add tracing instrumentation for ssh clients/servers
Add tracing context to the existing ProxyHelloSignature to provide
span information across ssh connections. To add span context per
ssh session on top of new connections, the same tracing context is
passed in the first global request of the session.
In order to ensure that tracing context is pulled from and inserted
into the proper context.Context, some interfaces and methods were
changed to take one as the first argument.
* Remove unused backend wrapper from Cache
* Remove double printShutdownStatus
* Fix readyz race condition
* Test coverage for the readyz.monitor fix
* Close listeners immediately in proxy.shutdown
* Use and handle net.ErrClosed correctly
This adapts utils.IsUseOfClosedNetworkError to check for net.ErrClosed
even inside trace.Aggregate errors, makes it so that we always return
something that would pass errors.Is(err, net.ErrClosed) when returning
from a (net.Listener).Accept(), and handles closed listeners within our
various Serve() loops so that we don't hit spurious backoff waits while
shutting down.
* Close listeners early and emitters late
* Test coverage for the proxy listener changes
* Revert some errors back to trace.ConnectionProblem
* Reduce PR scope to just the proxy, add comments
* Improve error logging.
* Added support for connecting API client through tunnel proxy and web proxy addresses (with identity file).
* Added concurrent dialing logic to dial several possible dialing combinations and seamlessly return the first client to connect.
* Add logger attributes to be able to propagate logger from tests for identifying tests
* Add test case for Server's DeepCopy.
* Update test to using the testing package directly. Update dependency after upstream PR.
Adds support for Concurrent Session Control and a new
semaphore API. Roles now support two new configuration
options, `max_ssh_connections` and `max_ssh_sessions`
which correspond to the total number of authenticated
ssh connections per cluster, and the number of ssh sessions
within a connection respectively. Attempting to exceed
these limits generate variants of the `session.rejected`
audit event and cause the connection/session to be
rejected.
Changes agent channel setup behavior to be consistent
openssh by having servers lazily request agent channels
when they are needed, rather than immediately starting a
single connection-wide channel as soon as forwarding is
requested. Fixes an issue introduced in #3613 which
caused openssh clients to hang on exit due to persistent
agent channel.
This code is not caught by linters because it's exported and they assume
there's some external users.
Since teleport is relatively self-contained, we can tell for sure
whether something is called or not.
Changes the lifetime of agent forwarding to be scoped
to the underlying ssh connection, instead of the
specific ssh channel which initially passed the agent
forwarding request.
Update utils.CertChecker to only check key and certificate algorithms
when in FIPS mode. Otherwise accept keys and certificates generated with
any algorithm.
Don't heartbeat address for nodes connected to clusters over a reverse
tunnel. Print warning to users if listen_addr or public_addr are set as
these are not used.
Updated services.ReverseTunnel to support type (proxy or node). For
proxy types, which represent trusted cluster connections, when a
services.ReverseTunnel is created, it's created on the remote side with
name /reverseTunnels/example.com. For node types, services.ReverseTunnel
is created on the main side as /reverseTunnels/{nodeUUID}.clusterName.
Updated services.TunnelConn to support type (proxy or node). For proxy
types, which represent trusted cluster connections, tunnel connections
are created on the main side under
/tunnelConnections/remote.example.com/{proxyUUID}-remote.example.com.
For nodes, tunnel connections are created on the main side under
/tunnelConnections/example.com/{proxyUUID}-example.com. This allows
searching for tunnel connections by cluster then allows easily creating
a set of proxies that are missing matching services.TunnelConn.
The reverse tunnel server has been updated to handle heartbeats from
proxies as well as nodes. Proxy heartbeat behavior has not changed.
Heartbeats from nodes now add remote connections to the matching local
site. In addition, the reverse tunnel server now proxies connection to
the Auth Server for requests that are already authenticated (a second
authentication to the Auth Server is required).
For registration, nodes try and connect to the Auth Server to fetch host
credentials. Upon failure, nodes now try and fallback to fetching host
credentials from the web proxy.
To establish a connection to an Auth Server, nodes first try and connect
directly, and if the connection fails, fallback to obtaining a
connection to the Auth Server through the reverse tunnel. If a
connection is established directly, node startup behavior has not
changed. If a node establishes a connection through the reverse tunnel,
it creates an AgentPool that attempts to dial back to the cluster and
establish a reverse tunnel.
When nodes heartbeat, they also heartbeat if they are connected directly
to the cluster or through a reverse tunnel. For nodes that are connected
through a reverse tunnel, the proxy subsystem now directs the reverse
tunnel server to establish a connection through the reverse tunnel
instead of directly.
When sending discovery requests, the domain field has been replaced with
tunnelID. The tunnelID field is either the cluster name (same as before)
for proxies, or {nodeUUID}.example.com for nodes.
Edoardo Spadolini