The HTTP client created by `auth.NewClient` has an idle timeout of
30s and each instance periodically gets cert authorities in
`TeleportProcess.syncRotationStateCycle`. If there has been no
activity when the ticker expires then the request will result in a
brand new connection to Auth to be established. This can cause a
spikes in open file descriptors on the Proxy when there are a large
number of instances connected via a tunnel.
To prevent periodic dialing when retrieving cert authorities the
endpoint has been migrated to the new Trust gRPC service. For now
only the get endpoint has been migrated but the rest of the CRUD
operations for cert authorities should be migrated from the HTTP
api to the gRPC service as well.
* Move the authorizer into its own package.
The Authorizer and associated functions and constants have been moved into
their own package `authz`. This will allow us to more easily create separate
services for gRPC functionality so that we're not pushing as much into the
`lib/auth` package.
Though this PR is large, the only actual code change here is making
`ContextForLocalUser` exported and changing the `permissions_test.go` so that
it doesn't rely on the `lib/auth` package.
* Use context functions.
* Test and linting fixes.
* Add in a temporary alias file to ensure e still compiles.
* Use functions instead of var aliases for performance reasons.
* Adding in godocs, cleaning up tests.
* Fix tests.
* Add a few more aliases.
* Expose device verification methods in lib/devicetrust/authz
* Set device fields in auth.Context.GetAccessState
* Enable role-based device verification for SSH
* Enable role-based device verification for DBs
* Add TODO for better SSH error messages
* Do not override device error on lib/kube/proxy
* Run gci
Refactor `services.AccessMFAParams` and the accompanying `MFAParams` methods so
they may be extended to carry additional data (eg, information required to
perform device trust checks).
This PR refactors existing code without adding any new functionality, so it may
be backported to older branches to alleviate eventual conflicts. Follow up
changes will take advantage of it to add support for device trust.
Related to gravitational/teleport.e#514.
* Add a new db engine
* Add tests for new engine
* Update tsh db subcommands
* Refactor error message and suggestions for unsupported tsh commands
* Add dynamodb to test plan
* Add AWS external ID to db config and update protos
When the AWS Account ID is provided, we must check that it is a valid
one.
AWS Account ID is a well documented field: 12-digit string
This check is meant to prevent typos when creating Databases mainly from
the Web UI in the context of Teleport Discover.
* Connection Diagnostics: Postgres Database tester
When adding a new resource using the Web UI we want to allow users to
test connecting to it.
We have two connection testers already:
- for SSH Nodes
- for Kube clusters
This PR adds a third tester: Postgres Database.
Most of the required changes for any Database are already present but
we're focusing on Postgres for now. Other databases will be added as
future PRs.
Testing a Database is similar to the other tests:
- generate certs for the logged in user
- connect to the resource using those certs
When generating the certificate, we inject an ID so that the Database
Service can add Connection Diagnostic traces.
* Add helper func to check for AWS unrecognized engine error
* Retry RDS queries with individual filters if engine name is unrecognized
* godoc
* pass engine names instead of filters, to make retry logic simpler
* Update mocks to check/apply RDS engine filters
* Update watcher tests
* Add watcher test for unrecognized RDS engines
* Update lib/srv/db/cloud/mocks.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
This PR presents a watcher for automatic `kube_cluster` discovery for GCP GKE clusters. Given an identity with access to the GCP cloud, the auto-discovery service will scan the cloud and register all clusters available in Kubernetes Engine.
Once the discovery service creates a `kube_cluster` on the Auth Server, the Kubernetes Service will start serving it. The credentials used to access the cluster are short-lived and generated through Google OAuth2 associated with the GCP Service Account configured for the Kubernetes Service.
GCP's Service Account must have the following role def attached:
```yaml
description: 'GKE Auto-Discovery'
includedPermissions:
- container.clusters.impersonate
- container.clusters.get
- container.clusters.list
- container.pods.get
- container.selfSubjectAccessReviews.create
- container.selfSubjectRulesReviews.create
name: projects/{projectID}/roles/GKEKubernetesAutoDisc
stage: GA
title: GKEKubernetesAutoDisc
```
Part of #16135, #13376
Related to #12048, #16276, #16281, #16633, #14991
Update various "assorted" dependencies, that either are used throughout
(logging, testing), are mostly algorithmic or otherwise difficult to pinpoint
"ownership".
I've dropped a couple of deprecated/mostly meaningless dependencies. I suspect
`github.com/mitchellh/mapstructure` could be dropped to, but I didn't look
further now.
* Drop dependency on Clever/go-utils
* Drop dependency on github.com/pkg/errors
* Update various Go dependencies
* Use CompareAndSwap instead of CAS (uber/atomic)
* Add a comment after replaced dependencies
* Add Yubikey PrivateKey implementation for use by Teleport clients.
- Add yubikey login logic, reusing previously stored private keys.
- Fix identity file decoding with PIV keys, which sign ecdsa certificates.
- Add libpcsclite-dev pre-req for building on linux.
- Remove unnecessary keys.Signer interface and move its functionality to keys.PrivateKey.
- Move retry and jitter utils to new api/utils/retryutils package.
Replace role/authPref RequireSessionMFA (bool) with RequireMFAType
(string).
- Add new RequireMFAType constant values with custom boolean marshalling.
- Add RequireMFAType to role and auth preference and deprecate
RequireSessionMFA.
- Add session-mfa override login when hardware_key_touch is enforced.
- Add protobuf enum for RequireMFAType.
- Add support for proto enums in protoc-gen-crd and update Kubernetes
Operator manifests.
Making all of our integration tests run in entirely parallel requires
a large engineering effort to enforce test isolation and remove all race
conditions between tests.
A lower-effort alternative may be to split apart the various test suites
into their own Go packages, and test those packages in parallel, even if
the tests inside are still executed serially. Auditing the test suites
for races on system-level resources (e.g. files, ports) is much easier
than chasing down every p[ossible race in the testing system.
This patch acts as a trial run, breaking a fairly well-defined and
self-contained test suite out into its own package. Note that the goal of
this change is not necessarily to shave minutes off the build (although
that would be nice), but to act as an illustration of how other, less
well-formed test suites might be broken apart.
See-Also: #12421
See-Also: #14408
* Add Azure auto-discovery configuration fields
* Init databases if azure matchers are in config
* Use AzureMatchers in db service
* Use all azure subscriptions/resource groups if omitted in matcher
* Add azure config tests
* Go mod tidy to update dependencies
* Add azure response error conversion
* Check for azure access denied and give a helpful error message
* Add azure subscriptions api
* Add azure mysql/postgresql api and wrappers
* Test generic db server for azure
* Make server properties its own type
* Convert server types manually instead of via json
* Move server list method selection logic out of api client
* Update azure db server tests
* Fixup merge
* Update comments
* Update more comments and remove junk code
* Move all azure api into lib/cloud/azure
* Update state and version checks
* Convert Azure DB Server into database
* Add mutex to subscription client for caching, just in case
* Test database conversion from azure db server
* Refactor common code
* Add azure matchers to watch config
* Remove unused imports
* Use common max pages in memorydb users
* Add azure clients to cloud clients
* Add azure fetchers to watcher
* Skip fetcher NotFound errors
* Add azure watcher tests
* Fix test comment
* Add wildcard region matching
* Remove redundant import
* Update lib/srv/db/cloud/watchers/azure.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Move db server listing logic into separate func
* Refactor some db conversion code
* Use local var to cache sub IDs
* Rename azure subscription client
* Rename azure subscription client and remove sub ID caching
* Move azure cloud clients into separate embedded interface
* Update watcher test for 'NotFound' handling
* Embed azureClients struct
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/subscriptions_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Rename azure subscription client and remove sub ID caching
* Add reference links for azure db ports
* Move indirect dep into group
* Wrap all converted azure response errors
* Remove unreachable panic
* Godoc DBServer
* Remove maxPages arg to azure client funcs
* Gofmt
* Spacing between copyright and package
* import order
* Update lib/srv/db/cloud/watchers/azure.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Fix import order
* Refactor region matching
* Fix usage of azure clients to remove max pages arg
* Query azure subscriptions in fetcher.Get if matching wildcard subscription
* Fix azure matcher construction
* Test azure fetcher discovers new subscriptions
* Remove unused test helper func
* Set azure database resource id metadata
* Log errors when fetching from azure subscription fails
* Use more readable protocol in azure db description
* Get azure mysql engine version from labels
* Update lib/srv/db/cloud/watchers/watcher_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/srv/db/cloud/watchers/watcher_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Co-authored-by: Marek Smoliński <marek@goteleport.com>
Primary Changes:
- Remove reliance on Private Key PEM:
- Update native and keygen packages to return PrivateKey instead of PEM key
- Add new PrivateKey interface which implements crypto.Signer
- Replace PEM encoded private key usage where possible
- Replace calls to tls.(Load)X509KeyPair with keys.(Load)X509KeyPair in
client packages
Minor Changes:
- Remove unused agent.AddedKey return from LoadKey
- Simplify sshutils and removed unused code paths
- Add ecdsa and ed25519 key support
* Add Azure auto-discovery configuration fields
* Init databases if azure matchers are in config
* Use AzureMatchers in db service
* Use all azure subscriptions/resource groups if omitted in matcher
* Add azure config tests
* Go mod tidy to update dependencies
* Add azure response error conversion
* Check for azure access denied and give a helpful error message
* Add azure subscriptions api
* Add azure mysql/postgresql api and wrappers
* Test generic db server for azure
* Make server properties its own type
* Convert server types manually instead of via json
* Move server list method selection logic out of api client
* Update azure db server tests
* Fixup merge
* Update comments
* Update more comments and remove junk code
* Move all azure api into lib/cloud/azure
* Update state and version checks
* Add mutex to subscription client for caching, just in case
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/subscriptions_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Update lib/cloud/azure/db_server_test.go
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Rename azure subscription client and remove sub ID caching
* Add reference links for azure db ports
* Move indirect dep into group
* Wrap all converted azure response errors
* Remove unreachable panic
* Godoc DBServer
* Remove maxPages arg to azure client funcs
* Gofmt
* Spacing between copyright and package
* import order
Co-authored-by: Marek Smoliński <marek@goteleport.com>
* Add an AWS EC2 instance fetcher
* Resolve comments
- Remove leftover VPCID stuff
- Document public functions
- Return a notfound when there are no nodes fetched instead of nil nil
* Resolve comments
* Fix fileconf_test.go
Teleport now will try to extract MySQL server version from initial handshake package instead of sending `8.0.0-Teleport` every time. This string can be overridden by new configuration option `mysql.server_version`. On DB service start Teleport will also try to fetch the current version from MySQL/MariaDB instance. After that the server version will be updated on every successful connection to keep it up to date.
Co-authored-by: STeve (Xin) Huang <xin.huang@goteleport.com>
Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com>
Introduce Database Certificate Authority. New CA is used by Database Access to sign database certificates making them independent from Host CA.
Co-authored-by: Marek Smoliński <marek@goteleport.com>
rosstimothy