This serves two purposes:
1. on upload, the receiver (node) will exit with error and generate an
SCP error audit event.
2. on download, the receiver (tsh) will print a meaningful message from
the sender (node)
Fixes#2861
This code is not caught by linters because it's exported and they assume
there's some external users.
Since teleport is relatively self-contained, we can tell for sure
whether something is called or not.
Changes the lifetime of agent forwarding to be scoped
to the underlying ssh connection, instead of the
specific ssh channel which initially passed the agent
forwarding request.
Fixed findings:
```
lib/sshutils/server_test.go:163:2: SA4006: this value of `clt` is never used (staticcheck)
clt, err := ssh.Dial("tcp", srv.Addr(), &cc)
^
lib/sshutils/server_test.go:91:3: SA5001: should check returned error before deferring ch.Close() (staticcheck)
defer ch.Close()
^
lib/shell/shell_test.go:33:2: SA4006: this value of `shell` is never used (staticcheck)
shell, err = GetLoginShell("non-existent-user")
^
lib/cgroup/cgroup_test.go:111:2: SA9003: empty branch (staticcheck)
if err != nil {
^
lib/cgroup/cgroup_test.go:119:2: SA5001: should check returned error before deferring service.Close() (staticcheck)
defer service.Close()
^
lib/client/keystore_test.go:138:2: SA4006: this value of `keyCopy` is never used (staticcheck)
keyCopy, err = s.store.GetKey("host.a", "bob")
^
lib/client/api.go:1604:3: SA4004: the surrounding loop is unconditionally terminated (staticcheck)
return makeProxyClient(sshClient, m), nil
^
lib/backend/test/suite.go:156:2: SA4006: this value of `err` is never used (staticcheck)
result, err = s.B.GetRange(ctx, prefix("/prefix/c/c1"), backend.RangeEnd(prefix("/prefix/c/cz")), backend.NoLimit)
^
lib/utils/timeout_test.go:84:2: SA1019: t.Dial is deprecated: Use DialContext instead, which allows the transport to cancel dials as soon as they are no longer needed. If both are set, DialContext takes priority. (staticcheck)
t.Dial = func(network string, addr string) (net.Conn, error) {
^
lib/utils/websocketwriter.go:83:3: SA4006: this value of `err` is never used (staticcheck)
utf8, err = w.encoder.String(string(data))
^
lib/utils/loadbalancer_test.go:134:2: SA4006: this value of `out` is never used (staticcheck)
out, err = Roundtrip(frontend.String())
^
lib/utils/loadbalancer_test.go:209:2: SA4006: this value of `out` is never used (staticcheck)
out, err = RoundtripWithConn(conn)
^
lib/srv/forward/sshserver.go:582:3: SA4004: the surrounding loop is unconditionally terminated (staticcheck)
return
^
lib/service/service.go:347:4: SA4006: this value of `err` is never used (staticcheck)
i, err = auth.GenerateIdentity(process.localAuth, id, principals, dnsNames)
^
lib/service/signals.go:60:3: SA1016: syscall.SIGKILL cannot be trapped (did you mean syscall.SIGTERM?) (staticcheck)
syscall.SIGKILL, // fast shutdown
^
lib/config/configuration_test.go:184:2: SA4006: this value of `conf` is never used (staticcheck)
conf, err = ReadFromFile(s.configFileBadContent)
^
lib/config/configuration.go:129:2: SA5001: should check returned error before deferring reader.Close() (staticcheck)
defer reader.Close()
^
lib/kube/kubeconfig/kubeconfig_test.go:227:2: SA4006: this value of `err` is never used (staticcheck)
tlsCert, err := ca.GenerateCertificate(tlsca.CertificateRequest{
^
lib/srv/sess.go:720:3: SA4006: this value of `err` is never used (staticcheck)
result, err := s.term.Wait()
^
lib/multiplexer/multiplexer_test.go:169:11: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
_, err = fmt.Fprintf(conn, proxyLine.String())
^
lib/multiplexer/multiplexer_test.go:221:11: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
_, err = fmt.Fprintf(conn, proxyLine.String())
^
```
Spring cleaning!
A very mechanical cleanup using several linters (unused, deadcode,
structcheck). Build and tests still pass so no behavior should be
affected.
Update utils.CertChecker to only check key and certificate algorithms
when in FIPS mode. Otherwise accept keys and certificates generated with
any algorithm.
Don't heartbeat address for nodes connected to clusters over a reverse
tunnel. Print warning to users if listen_addr or public_addr are set as
these are not used.
Updated services.ReverseTunnel to support type (proxy or node). For
proxy types, which represent trusted cluster connections, when a
services.ReverseTunnel is created, it's created on the remote side with
name /reverseTunnels/example.com. For node types, services.ReverseTunnel
is created on the main side as /reverseTunnels/{nodeUUID}.clusterName.
Updated services.TunnelConn to support type (proxy or node). For proxy
types, which represent trusted cluster connections, tunnel connections
are created on the main side under
/tunnelConnections/remote.example.com/{proxyUUID}-remote.example.com.
For nodes, tunnel connections are created on the main side under
/tunnelConnections/example.com/{proxyUUID}-example.com. This allows
searching for tunnel connections by cluster then allows easily creating
a set of proxies that are missing matching services.TunnelConn.
The reverse tunnel server has been updated to handle heartbeats from
proxies as well as nodes. Proxy heartbeat behavior has not changed.
Heartbeats from nodes now add remote connections to the matching local
site. In addition, the reverse tunnel server now proxies connection to
the Auth Server for requests that are already authenticated (a second
authentication to the Auth Server is required).
For registration, nodes try and connect to the Auth Server to fetch host
credentials. Upon failure, nodes now try and fallback to fetching host
credentials from the web proxy.
To establish a connection to an Auth Server, nodes first try and connect
directly, and if the connection fails, fallback to obtaining a
connection to the Auth Server through the reverse tunnel. If a
connection is established directly, node startup behavior has not
changed. If a node establishes a connection through the reverse tunnel,
it creates an AgentPool that attempts to dial back to the cluster and
establish a reverse tunnel.
When nodes heartbeat, they also heartbeat if they are connected directly
to the cluster or through a reverse tunnel. For nodes that are connected
through a reverse tunnel, the proxy subsystem now directs the reverse
tunnel server to establish a connection through the reverse tunnel
instead of directly.
When sending discovery requests, the domain field has been replaced with
tunnelID. The tunnelID field is either the cluster name (same as before)
for proxies, or {nodeUUID}.example.com for nodes.
This commit expands the usage of the caching layer
for auth server API:
* Introduces in-memory cache that is used to serve all
Auth server API requests. This is done to achieve scalability
on 10K+ node clusters, where each node fetches certificate authorities,
roles, users and join tokens. It is not possible to scale
DynamoDB backend or other backends on 10K reads per seconds
on a single shard or partition. The solution is to introduce
an in-memory cache of the backend state that is always used
for reads.
* In-memory cache has been expanded to support all resources
required by the auth server.
* Experimental `tctl top` command has been introduced to display
common single node metrics.
Replace SQLite Memory Backend with BTree
SQLite in memory backend was suffering from
high tail latencies under load (up to 8 seconds
in 99.9%-ile on load configurations).
This commit replaces the SQLite memory caching
backend with in-memory BTree backend that
brought down tail latencies to 2 seconds (99.9%-ile)
and brought overall performance improvement.
Added utils.CertChecker that wraps a ssh.CertChecker. The new
certificate checker first checks if the certificate is a valid
certificate for Teleport. At the moment that is 2048-bit RSA then calls
the underlying certificate checker to perform the requested validation.
Created *utils.TrackingConn that wraps the server side net.Conn and is
used to track how much data is transmitted and received over the
net.Conn. At the close of a connection (close of a *srv.ServerContext)
the total data transmitted and received is emitted to the Audit Log.
On the client side, make sure directory mode is requested when sending
multiple files. On the server side, when in directory mode, validate
that the target is a directory.
control how often the server sends keep-alive messages to clients and
after how many missed keep-alive replies the server tears down the
connection to the client.
This commit reduces the amount of typing some users have
to do when using identity file:
* Teleport user is set from the certificate
* Auth preference is set to local
In addition it fixes several UX problems:
* Commands `tctl auth sign` and `tsh login -o` now include
trusted CA keys in the identity file.
* Command `tsh ssh -i` is now noninteractive and only
uses identity file for authentication, in case
if identity file can not authenticate, no other interactive
or non-interactive authentication methods will be used.
Before this commit, tsh tried to fallback to interactive
login mode or read the keys from agent socket.
Fixes#1698.
* Added sync.Pool to take care of many gzip.Writer
allocating a lot of large objects on the heap.
* Reshuffled signal handling, SIGQUIT is now
graceful shutdown, just like in Nginx.
* Signal USR1 prints hepful diagnostic info to stderr.
* Removed gops endpoint and flags.
* Fixed logs in some places.
* Debug flag now adds extra pprof handlers to diagnostic
endpoint.
This commit introduces signal handling.
Parent teleport process is now capable of forking
the child process and passing listeners file descriptors
to the child.
Parent process then can gracefully shutdown
by tracking the amount of current connections and
closing listeners once the amount goes to 0.
Here are the signals handled:
* USR2 signal will cause the parent to fork
a child process and pass listener file descriptors to it.
Child process will close unused file descriptors
and will bind to the used ones.
At this moment two processes - the parent
and the forked child process will be serving requests.
After looking at the traffic and the log files,
administrator can either shut down the parent process
or the child process if the child process is not functioning
as expected.
* TERM, INT signals will trigger graceful process shutdown.
Auth, node and proxy processes will wait until the amount
of active connections goes down to 0 and will exit after that.
* KILL, QUIT signals will cause immediate non-graceful
shutdown.
* HUP signal combines USR2 and TERM signals in a convenient
way: parent process will fork a child process and
self-initate graceful shutdown. This is a more convenient
than USR2/TERM sequence, but less agile and robust
as if the connection to the parent process drops, but
the new process exits with error, administrators
can lock themselves out of the environment.
Additionally, boltdb backend has to be phased out,
as it does not support read/writes by two concurrent
processes. This had required refactoring of the dir
backend to use file locking to allow inter-process
collaboration on read/write operations.
* Do not log EOF errors, avoid polluting logs
* Trim space from tokens when reading from file
* Do not use dir based caching
The caching problem deserves a separate explanation.
Directory backend is not concurrent friendly - it has a
fundamental design flaw - multiple gorotuines writing to the
same file corrupt cache data.
This requires either redesign of the backend or switching
to boltdb backend for caching.
Boltdb backend uses transactions and is safe for concurrent
access. This PR changes local cache to use boltdb instead
of the dir backend that is now used only in tests.
This commit introduced mutual TLS authentication
for auth server API server.
Auth server multiplexes HTTP over SSH - existing
protocol and HTTP over TLS - new protocol
on the same listening socket.
Nodes and users authenticate with 2.5.0 Teleport
using TLS mutual TLS except backwards-compatibility
cases.
This commit aadds multiplexer library of SSH/TLS on the same
listener socket. The multiplexer detects the protocol by the first
3 bytes of the incoming connection and forwards wrapped
connection either to the SSH ot TLS listeners.
The library also supports PROXY line protocol
and wraps connection information with connection details
from the proxy line received by the server
This was fixed running the `misspell` linter in fix mode using
`gometalinter`. The exact command I ran was :
```
gometalinter --vendor --disable-all -E misspell --linter='misspell:misspell -w {path}:^(?P<path>.*?\.go):(?P<line>\d+):(?P<col>\d+):\s*(?P<message>.*)$' ./...
```
Some typo were fixed by hand on top of it.
- Renamed "dir" format to "openssh"
- Replaced self-made key fingerprinting function with a standard one
- Changed fingerprinting from legacy md5 to sha256
First part of addressing #1033 is ability to load credentials from the
credentials file(s).
This commit adds -i flag processing, i.e. a certificate can be fed via a
cert.file and used to login.
In this commit:
1. Minor addition to Makefile to pull new .go files from
tool/teleport/common
2. os.Glob() returns an empty list (instead of an error) if the
file/pattern is not found, so added check for that.
3. sendFile was prematurely sending 'C' command before trying to open a
file. This used to lead to creation of empty files for invlaid
sources.
Also, removed some confusing comments.
Andrew Lytvynov