There seem to be some test stubs in `lib/auth` that can generate TLS
key/cert but without supporting k8s users/groups.
Rewrite cert generation in `kube_integration_test.go` to use the same
logic as `auth.Server.ProcessKubeCSR`.
Original finding list:
```
tool/tctl/common/node_command.go:163:3: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(string(out))
^
tool/tctl/common/status_command.go:110:2: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(view())
^
tool/tctl/common/status_command.go:126:3: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(view())
^
tool/tctl/common/token_command.go:201:3: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(tokensView())
^
tool/tctl/common/token_command.go:207:3: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(string(data))
^
tool/tctl/common/user_command.go:248:2: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(string(out))
^
tool/tctl/common/user_command.go:294:3: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
fmt.Printf(string(out))
^
integration/helpers.go:200:2: SA4006: this value of `err` is never used (staticcheck)
cryptoPubKey, err := sshutils.CryptoPublicKey(cfg.Pub)
^
integration/helpers.go:399:3: SA4006: this value of `roles` is never used (staticcheck)
roles = append(roles, role)
^
integration/helpers.go:597:4: SA4006: this value of `roles` is never used (staticcheck)
roles = append(roles, role)
^
integration/helpers.go:599:4: SA4006: this value of `roles` is never used (staticcheck)
roles = user.Roles
^
integration/integration_test.go:1625:2: SA4006: this value of `err` is never used (staticcheck)
adminsRole, err := services.NewRole(mainAdmins, services.RoleSpecV3{
^
integration/integration_test.go:2185:2: SA4006: this value of `output` is never used (staticcheck)
output, err = runCommand(main, []string{"echo", "hello world"}, cfg, 1)
^
integration/integration_test.go:2340:2: SA4006: this value of `output` is never used (staticcheck)
output, err = runCommand(main, []string{"echo", "hello world"}, cfgProxy, 1)
^
integration/kube_integration_test.go:154:2: SA4006: this value of `err` is never used (staticcheck)
role, err := services.NewRole("kubemaster", services.RoleSpecV3{
^
integration/kube_integration_test.go:321:2: SA4006: this value of `err` is never used (staticcheck)
role, err := services.NewRole("kubemaster", services.RoleSpecV3{
^
integration/kube_integration_test.go:366:2: SA4006: this value of `err` is never used (staticcheck)
role, err := services.NewRole("kubemaster", services.RoleSpecV3{
^
integration/kube_integration_test.go:386:2: SA4006: this value of `err` is never used (staticcheck)
pods, err := s.CoreV1().Pods(kubeSystemNamespace).List(metav1.ListOptions{
^
integration/kube_integration_test.go:465:2: SA4006: this value of `err` is never used (staticcheck)
mainRole, err := services.NewRole("main-kube", services.RoleSpecV3{
^
integration/kube_integration_test.go:579:2: SA4006: this value of `err` is never used (staticcheck)
pods, err := proxyClient.CoreV1().Pods(kubeSystemNamespace).List(metav1.ListOptions{
^
integration/kube_integration_test.go:727:2: SA4006: this value of `err` is never used (staticcheck)
mainRole, err := services.NewRole("main-kube", services.RoleSpecV3{
^
integration/kube_integration_test.go:840:2: SA4006: this value of `err` is never used (staticcheck)
pods, err := proxyClient.CoreV1().Pods(kubeSystemNamespace).List(metav1.ListOptions{
^
integration/kube_integration_test.go:1008:2: SA4006: this value of `err` is never used (staticcheck)
role, err := services.NewRole("kubemaster", services.RoleSpecV3{
^
tool/teleport/common/teleport_test.go:83:2: SA4006: this value of `cmd` is never used (staticcheck)
cmd, conf := Run(Options{
^
tool/teleport/common/teleport_test.go:91:2: SA4006: this value of `cmd` is never used (staticcheck)
cmd, conf = Run(Options{
^
tool/tsh/tsh.go:170:2: SA4006: this value of `cmdLine` is never used (staticcheck)
cmdLine := []string{}
^
integration/helpers.go:399:11: SA4010: this result of append is never used, except maybe in other appends (staticcheck)
roles = append(roles, role)
^
integration/helpers.go:597:12: SA4010: this result of append is never used, except maybe in other appends (staticcheck)
roles = append(roles, role)
^
integration/integration_test.go:1092:7: SA4000: identical expressions on the left and right side of the '&&' operator (staticcheck)
for len(b.Tunnel.GetSites()) < 2 && len(b.Tunnel.GetSites()) < 2 {
^
integration/integration_test.go:1426:6: SA4000: identical expressions on the left and right side of the '&&' operator (staticcheck)
for len(main.Tunnel.GetSites()) < 2 && len(main.Tunnel.GetSites()) < 2 {
^
integration/integration_test.go:1691:6: SA4000: identical expressions on the left and right side of the '&&' operator (staticcheck)
for len(main.Tunnel.GetSites()) < 2 && len(main.Tunnel.GetSites()) < 2 {
^
integration/integration_test.go:1895:6: SA4000: identical expressions on the left and right side of the '&&' operator (staticcheck)
for len(main.Tunnel.GetSites()) < 2 && len(main.Tunnel.GetSites()) < 2 {
^
integration/kube_integration_test.go:548:6: SA4000: identical expressions on the left and right side of the '&&' operator (staticcheck)
for len(main.Tunnel.GetSites()) < 2 && len(main.Tunnel.GetSites()) < 2 {
^
integration/kube_integration_test.go:814:6: SA4000: identical expressions on the left and right side of the '&&' operator (staticcheck)
for len(main.Tunnel.GetSites()) < 2 && len(main.Tunnel.GetSites()) < 2 {
^
```
Changes the lifetime of agent forwarding to be scoped
to the underlying ssh connection, instead of the
specific ssh channel which initially passed the agent
forwarding request.
All changes should be noop, except for
`integration/integration_test.go`.
The integration test was ignoring `recordingMode` test case parameter
and always used `RecordAtNode`. When switching to `recordingMode`, test
cases with `RecordAtProxy` fail with a confusing error about missing
user agent. Filed https://github.com/gravitational/teleport/issues/3606
to track that separately and unblock enabling `structcheck` linter.
TeleInstance manages an auth server and a set of proxies/nodes.
TeleInstance.Stop only stops the auth server. A bunch of tests used it
assuming it also cleans up any running nodes.
This has caused a lot of log spam from failing heartbeats and generally
wasted CPU cycles.
Rename it to Stop to StopAuth to make it's purpose more obvious. Add
TeleInstance.StopAll that cleans up everything, suitable for deferring
in tests.
Some unusual shells like [fish](https://fishshell.com/) don't support
`$(cmd)` nested command syntax.
Print window size as two separate commands separated by newline instead.
Also, scan more of the output, in case the prompt is very long.
This type uses an inner bytes.Buffer to store terminal output. Terminal
is used by concurrent goroutines as io.ReadWriter, so access to the
buffer needs to be synchronized.
* Add monorepo
* Add reset/passwd capability for local users (#3287)
* Add UserTokens to allow password resets
* Pass context down through ChangePasswordWithToken
* Rename UserToken to ResetPasswordToken
* Add auto formatting for proto files
* Add common Marshaller interfaces to reset password token
* Allow enterprise "tctl" reuse OSS user methods (#3344)
* Pass localAuthEnabled flag to UI (#3412)
* Added LocalAuthEnabled prop to WebConfigAuthSetting struct in webconfig.go
* Added LocalAuthEnabled state as part of webCfg in apiserver.go
* update e-refs
* Fix a regression bug after merge
* Update tctl CLI output msgs (#3442)
* Use local user client when resolving user roles
* Update webapps ref
* Add and retrieve fields from Cluster struct (#3476)
* Set Teleport versions for node, auth, proxy init heartbeat
* Add and retrieve fields NodeCount, PublicURL, AuthVersion from Clusters
* Remove debug logging to avoid log pollution when getting public_addr of proxy
* Create helper func GuessProxyHost to get the public_addr of a proxy host
* Refactor newResetPasswordToken to use GuessProxyHost and remove publicUrl func
* Remove webapps submodule
* Add webassets submodule
* Replace webapps sub-module reference with webassets
* Update webassets path in Makefile
* Update webassets
1b11b26 Simplify and clean up Makefile (#62) https://github.com/gravitational/webapps/commit/1b11b26
* Retrieve cluster details for user context (#3515)
* Let GuessProxyHost also return proxy's version
* Unit test GuessProxyHostAndVersion & GetClusterDetails
* Update webassets
4dfef4e Fix build pipeline (#66) https://github.com/gravitational/webapps/commit/4dfef4e
* Update e-ref
* Update webassets
0647568 Fix OSS redirects https://github.com/gravitational/webapps/commit/0647568
* update e-ref
* Update webassets
e0f4189 Address security audit warnings Updates "minimist" package which is used by 7y old "optimist". https://github.com/gravitational/webapps/commit/e0f4189
* Add new attr to Session struct (#3574)
* Add fields ServerHostname and ServerAddr
* Set these fields on newSession
* Ensure webassets submodule during build
* Update e-ref
* Ensure webassets before running unit-tests
* Update E-ref
Co-authored-by: Lisa Kim <lisa@gravitational.com>
Co-authored-by: Pierre Beaucamp <pierre@gravitational.com>
Co-authored-by: Jenkins <jenkins@gravitational.io>
Top-level `make lint` rule that scans everything and a CI-specific rule
for Jenkins.
Currently only enable "unused", since it's reliable. The list will
expand.
Also clean up stragglers that somehow slipped through in #3552.
Updates #3551
This commit fixes#3369, refs #3374
It adds support for kuberenetes_users section in roles,
allowing Teleport proxy to impersonate user identities.
It also extends variable interpolation syntax by adding
suffix and prefix to variables and function `email.local`:
Example:
```yaml
kind: role
version: v3
metadata:
name: admin
spec:
allow:
# extract email local part from the email claim
logins: ['{{email.local(external.email)}}']
# impersonate a kubernetes user with IAM prefix
kubernetes_users: ['IAM#{{external.email}}']
# the deny section uses the identical format as the 'allow' section.
# the deny rules always override allow rules.
deny: {}
```
Some notes on email.local behavior:
* This is the only function supported in the template variables for now
* In case if the email.local will encounter invalid email address,
it will interpolate to empty value, will be removed from resulting
output.
Changes in impersonation behavior:
* By default, if no kubernetes_users is set, which is a majority of cases,
user will impersonate themselves, which is the backwards-compatible behavior.
* As long as at least one `kubernetes_users` is set, the forwarder will start
limiting the list of users allowed by the client to impersonate.
* If the users' role set does not include actual user name, it will be rejected,
otherwise there will be no way to exclude the user from the list).
* If the `kuberentes_users` role set includes only one user
(quite frequently that's the real intent), teleport will default to it,
otherwise it will refuse to select.
This will enable the use case when `kubernetes_users` has just one field to
link the user identity with the IAM role, for example `IAM#{{external.email}}`
* Previous versions of the forwarding proxy were denying all external
impersonation headers, this commit allows 'Impesrsonate-User' and
'Impersonate-Group' header values that are allowed by role set.
* Previous versions of the forwarding proxy ignored 'Deny' section of the roles
when applied to impersonation, this commit fixes that - roles with deny
kubernetes_users and kubernetes_groups section will not allow
impersonation of those users and groups.
Added support for reexec during port forwarding. This allows Teleport
nodes to run PAM code before port forwarding requests. This makes any
memory leaks in PAM code less dangerous as well as bringing port
forwarding logic in-line with execution requests (exec or shell).
This commit fixes#3252
Security patches 4.2 introduced a regression - leaf clusters ignore role mapping
and attempt to use role names coming from identity of the root cluster
whenever GetNodes method was used.
This commit reverts back the logic, however it ensures that the original
fix is preserved - traits and groups are updated on the user object.
Integration test has been extended to avoid the regression in the future.
Added package cgroup to orchestrate cgroups. Only support for cgroup2
was added to utilize because cgroup2 cgroups have unique IDs that can be
used correlated with BPF events.
Added bpf package that contains three BPF programs: execsnoop,
opensnoop, and tcpconnect. The bpf package starts and stops these
programs as well correlating their output with Teleport sessions
and emitting them to the audit log.
Added support for Teleport to re-exec itself before launching a shell.
This allows Teleport to start a child process, capture it's PID, place
the PID in a cgroup, and then continue to process. Once the process is
continued it can be tracked by it's cgroup ID.
Reduced the total number of connections to a host so Teleport does not
quickly exhaust all file descriptors. Exhausting all file descriptors
happens very quickly when disk events are emitted to the audit log which
are emitted at a very high rate.
Added tarballs for exec sessions. Updated session.start and session.end
events with additional metadata. Updated the format of session tarballs
to include enhanced events.
Added file configuration for enhanced session recording. Added code to
startup enhanced session recording and pass package to SSH nodes.
This commit introduces a new iteration of the proxy discovery
algorithm implemented as a separate component from the main
reversetunnel agent/agentpool system. This isolation is
intended to improve our ability to test and reason about the
algorithm in isolation from IO and other implementation
details.
When emitting session related events, emit them with the ID of the host
not of the server. This is because the forwarding server has a randomly
generated ID that will note validate with TLS identity that connects to
the Auth Server.
If an attacker can force a username change at an IdP, upon second login,
the services.User object of the original user can be updated with new
roles and traits. If these new roles and traits differ, the original
user can have their privileges raised (or lowered).
To mitigate this, encode roles and traits within the certificate and use
these when fetching roles to make RBAC decisions. If roles and traits are
not encoded within an certificate (for example for old style SSH
certificates then fallback to using the services.User object and log a
warning.
This commit implements #2543
In SSH terms ProxyJump is a shortcut for SSH client
connecting the proxy/jumphost and requesting .port forwarding to the
target node.
This commit adds support for direct-tcpip request support
in teleport proxy service that is an alias to the existing proxy
subsystem and reuses most of the code.
This commit also adds support to "route to cluster" metadata
encoded in SSH certificate making it possible to have client
SSH certificates to include the metadata that will cause the proxy
to route the client requests to a specific cluster.
`tsh ssh -J proxy:port ` is supported in a limited way:
Only one jump host is supported (-J supports chaining
that teleport does not utilise) and tsh will return with error
in case of two jumphosts: -J a,b will not work.
In case if `tsh ssh -J user@proxy` is used, it overrides
the SSH proxy coming from the tsh profile and port-forwarding
is used instead of the existing teleport proxy subsystem
This commit implements #2872.
Similarly to file://, the scheme `stdout://` could be used complimentary
to the existing external scheme to logs audit logs:
to stdout:
```yaml
audit_events_uri: ['dynamodb://events', 'stdout://',]
```
Just like `file://` scheme it is only possible to use 'stdout://'
scheme when external events and session uploader are defined,
so all audit upload and search features of teleport could work.
This commit fixes issue #2766.
The prior logic in Kubernetes module used
SNI to route requests to the target kubernetes cluster.
This approach created problems with long cluster names
exceeding 61 character DNS label limit and
required setting up DNS wildcard records.
This commit changes the routing to use the metadata
encoded in client's x509 certificate to route the
request to the target cluster.
SNI approach will be supported for several versions
to preserve backwards compatibility.
- Updates the identity files exported by `tctl auth sign` to include the
user's TLS certificate, as well as the set of available TLS root CA
certs.
- Adds a new GRPC-based auth server method, `GenerateUserCerts`, which
exports both SSH and TLS based certs.
This commit refactor discovery protocol
to make it less dependent on the database and
scale better on large numbers of tunnels.
Reverse tunnel is now always sending
back the list of all proxies registered in the
cluster in the form of discovery requests.
Before this commit, reverse tunnel server was comparing
existing TunnelConnection with the Proxies
and sending back the list of proxies that were not
discovered.
This required nodes to register tunnel connections
in the database and servers poll the connections.
On 10K clusters this is not scalable. Instead,
the change assumes that there is not a lot of
proxies so it's OK to send the information about
them back to all connected agents.
Agent pools can make up their own mind about what to
do with the information - they can ignore
the request as long as they observe all agents
connected to the requested proxies.
At the same time, to avoid using too much traffic,
reverse tunnel server only sends the discovery requests
after the first agent heartbeat and in case if
proxy list changes. To make it possible reverse tunnel
sets up a watch on the proxies.
Added "local_auth" to file configuration and "LocalAuth" to
services.ClusterConfig to control cluster-wide local authentication.
Check local auth settings when generating signup tokens, creating local
users, and login.
In case of IOT (whenever teleport nodes are
connecting to the proxy), there is no need
to create ReverseTunnel objects in the backend,
as there is always one reverse tunnel per node.
This commit removes the logic that created
reverse tunnel object in the backed in IOT cases
and refactors some other parts of the code.
Instantiate agent pool (and agent) with a reference to the reverse
tunnel server.
Pass list of principals to agents when initiating a transport dial
request.
The above two changes allow the agent to look up principals in local
site when attempting to connect to a node within a trusted cluster.
Updated services.ReverseTunnel to support type (proxy or node). For
proxy types, which represent trusted cluster connections, when a
services.ReverseTunnel is created, it's created on the remote side with
name /reverseTunnels/example.com. For node types, services.ReverseTunnel
is created on the main side as /reverseTunnels/{nodeUUID}.clusterName.
Updated services.TunnelConn to support type (proxy or node). For proxy
types, which represent trusted cluster connections, tunnel connections
are created on the main side under
/tunnelConnections/remote.example.com/{proxyUUID}-remote.example.com.
For nodes, tunnel connections are created on the main side under
/tunnelConnections/example.com/{proxyUUID}-example.com. This allows
searching for tunnel connections by cluster then allows easily creating
a set of proxies that are missing matching services.TunnelConn.
The reverse tunnel server has been updated to handle heartbeats from
proxies as well as nodes. Proxy heartbeat behavior has not changed.
Heartbeats from nodes now add remote connections to the matching local
site. In addition, the reverse tunnel server now proxies connection to
the Auth Server for requests that are already authenticated (a second
authentication to the Auth Server is required).
For registration, nodes try and connect to the Auth Server to fetch host
credentials. Upon failure, nodes now try and fallback to fetching host
credentials from the web proxy.
To establish a connection to an Auth Server, nodes first try and connect
directly, and if the connection fails, fallback to obtaining a
connection to the Auth Server through the reverse tunnel. If a
connection is established directly, node startup behavior has not
changed. If a node establishes a connection through the reverse tunnel,
it creates an AgentPool that attempts to dial back to the cluster and
establish a reverse tunnel.
When nodes heartbeat, they also heartbeat if they are connected directly
to the cluster or through a reverse tunnel. For nodes that are connected
through a reverse tunnel, the proxy subsystem now directs the reverse
tunnel server to establish a connection through the reverse tunnel
instead of directly.
When sending discovery requests, the domain field has been replaced with
tunnelID. The tunnelID field is either the cluster name (same as before)
for proxies, or {nodeUUID}.example.com for nodes.
This commit expands the usage of the caching layer
for auth server API:
* Introduces in-memory cache that is used to serve all
Auth server API requests. This is done to achieve scalability
on 10K+ node clusters, where each node fetches certificate authorities,
roles, users and join tokens. It is not possible to scale
DynamoDB backend or other backends on 10K reads per seconds
on a single shard or partition. The solution is to introduce
an in-memory cache of the backend state that is always used
for reads.
* In-memory cache has been expanded to support all resources
required by the auth server.
* Experimental `tctl top` command has been introduced to display
common single node metrics.
Replace SQLite Memory Backend with BTree
SQLite in memory backend was suffering from
high tail latencies under load (up to 8 seconds
in 99.9%-ile on load configurations).
This commit replaces the SQLite memory caching
backend with in-memory BTree backend that
brought down tail latencies to 2 seconds (99.9%-ile)
and brought overall performance improvement.
This commit implements disconnects of kubernetes
clients with expired certificates and terminates
idle sessions, according to 'disconnect_expired_cert'
and 'client_idle_timeout' existing settings making
the behavior consistent with SSH sessions.
Added utils.CertChecker that wraps a ssh.CertChecker. The new
certificate checker first checks if the certificate is a valid
certificate for Teleport. At the moment that is 2048-bit RSA then calls
the underlying certificate checker to perform the requested validation.
This commit switches Teleport proxy to use impersonation
API instead of the CSR API.
This allows Teleport to work on EKS clusters, GKE and all
other CNCF compabitble clusters.
This commit updates helm chart RBAC as well.
It introduces extra configuration flag to proxy_service
configuration parameter:
```yaml
proxy_service:
# kubeconfig_file is used for scenarios
# when Teleport Proxy is deployed outside
# of the kubernetes cluster
kubeconfig_file: /path/to/kube/config
```
It deprecates similar flag in auth_service:
```yaml
auth_service:
# DEPRECATED. THIS FLAG IS IGNORED
kubeconfig_file: /path/to/kube/config
```
Created *utils.TrackingConn that wraps the server side net.Conn and is
used to track how much data is transmitted and received over the
net.Conn. At the close of a connection (close of a *srv.ServerContext)
the total data transmitted and received is emitted to the Audit Log.
* Fetch groups for GSuite SSO. (#2456)
Fixes#2455
This commit adds support for fetching
groups for GSuite SSO logins via
OIDC connector interface.
If OIDC connector has a special scope:
`https://www.googleapis.com/auth/admin.directory.group.readonly`
teleport will fetch user's group membership and populate
groups claim.
* Pass kubernetes groups to the remote cluster. (#2484)
This commit allows remote cluster to reference
the kubernetes groups coming from the roles
of the main cluster in the trusted clusters
configuration.
For example, main cluster can have a user
with a role 'main' and kubernetes groups:
kube_groups: ['system:masters']
and SSH logins:
logins: ['root']
Remote cluster can choose to map
this 'main' cluster to it's own:
'remote-admin' cluster in the trusted cluster config:
role_map:
- remote: 'main'
local: 'remote-admin'
The role 'remote-admin' of the remote cluster
can now be templated to use the main cluster role main
logins and kubernetes_groups using variables:
logins: ['{{internal.logins}}']
kubernetes_groups: ['{{internal.kubernetes_groups}}']
This is possible because teleport now encodes
both values in X509 certificate metadata
and remote cluster passes these values as 'internal' traits
to the template engine.
This PR fixes an issue with tsh login.
Here is a flaw in logic described using the following
scenario:
Assume there are two clusters, 'main' and 'east'.
1. User logs into the first cluster 'main'
2. Selects the cluster 'east' in the profile
3. Next day, logs in again
4. Client pulls the trusted CA from the cluster 'main'
as a part of SSH login procedure and adds to the keystore
5. Client connects to cluster 'east' because it is
set as a current cluster in the profile
6. Client attempts to connect to the auth server of the cluster
'east' and fails because it does not trust the certificate
of the 'east' yet, only 'main.
This PR fixes the issue by making sure the client
always connects to the cluster 'main' in the step 5 instead.
This commit introduces several key changes to
Teleport backend and API infrastructure
in order to achieve scalability improvements
on 10K+ node deployments.
Events and plain keyspace
--------------------------
New backend interface supports events,
pagination and range queries
and moves away from buckets to
plain keyspace, what better aligns
with DynamoDB and Etcd featuring similar
interfaces.
All backend implementations are
exposing Events API, allowing
multiple subscribers to consume the same
event stream and avoid polling database.
Replacing BoltDB, Dir with SQLite
-------------------------------
BoltDB backend does not support
having two processes access the database at the
same time. This prevented Teleport
using BoltDB backend to be live reloaded.
SQLite supports reads/writes by multiple
processes and makes Dir backend obsolete
as SQLite is more efficient on larger collections,
supports transactions and can detect data
corruption.
Teleport automatically migrates data from
Bolt and Dir backends into SQLite.
GRPC API and protobuf resources
-------------------------------
GRPC API has been introduced for
the auth server. The auth server now serves both GRPC
and JSON-HTTP API on the same TLS socket and uses
the same client certificate authentication.
All future API methods should use GRPC and HTTP-JSON
API is considered obsolete.
In addition to that some resources like
Server and CertificateAuthority are now
generated from protobuf service specifications in
a way that is fully backward compatible with
original JSON spec and schema, so the same resource
can be encoded and decoded from JSON, YAML
and protobuf.
All models should be refactored
into new proto specification over time.
Streaming presence service
--------------------------
In order to cut bandwidth, nodes
are sending full updates only when changes
to labels or spec have occured, otherwise
new light-weight GRPC keep alive updates are sent
over to the presence service, reducing
bandwidth usage on multi-node deployments.
In addition to that nodes are no longer polling
auth server for certificate authority rotation
updates, instead they subscribe to event updates
to detect updates as soon as they happen.
This is a new API, so the errors are inevitable,
that's why polling is still done, but
on a way slower rate.
Andrew Lytvynov