* Add conditional update methods for `AccessList` and `AccessListMembers`
This PR adds two new methods for conditinally update an AccessList and
AccessListMember resources to avoid overriding changed resources.
This is a preparation for a ineligible status reconciler.
Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
* Update lib/services/simple/access_list.go
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* handle review comments and drop copy methods
* simplify code
---------
Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
The existing resource parser was converted to use typical under
the hood. Aside from the type safety improvements the making the
switch allows a single predicate expression to be parsed once and
evaluated per resource. The previous parser worked backwards, it
created a parser per resource with the same expression which lead
to elevated memory consumption and poor performance.
An additional bug was fixed that prevented predicate filtering from
working properly with the unified resource API due to applying
predicates prior to filtering by kind. This resulted in queriers
for server specific things (resource.spec.hostname) to fail even
when only the server resource was specified in the resource kinds.
Benchamrk results for the changes in this commit compared to master
show a 75% reduction in the test time and a 96% reduction in allocations
during the tests.
```bash
$ benchstat e77e5e080b1848da830a9030d43c38be55a5b8d5.txt 8eff752a93dfe96225a71431af829015e83400fa.txt
goos: darwin
goarch: arm64
pkg: github.com/gravitational/teleport/lib/auth
│ e77e5e080b1848da830a9030d43c38be55a5b8d5.txt │ 8eff752a93dfe96225a71431af829015e83400fa.txt │
│ sec/op │ sec/op vs base │
ListUnifiedResourcesFilter/labels-12 73.88m ± 1% 64.74m ± 6% -12.37% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_path-12 294.06m ± 20% 74.83m ± 5% -74.55% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_index-12 307.63m ± 4% 76.32m ± 4% -75.19% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_lower-12 130.7m ± 5% 130.5m ± 4% ~ (p=0.971 n=10)
ListUnifiedResourcesFilter/search_upper-12 147.4m ± 5% 138.7m ± 2% -5.87% (p=0.003 n=10)
ListUnifiedResources/simple_labels-12 1.748 ± 2% 1.744 ± 2% ~ (p=0.631 n=10)
geomean 246.6m 150.6m -38.93%
│ e77e5e080b1848da830a9030d43c38be55a5b8d5.txt │ 8eff752a93dfe96225a71431af829015e83400fa.txt │
│ B/op │ B/op vs base │
ListUnifiedResourcesFilter/labels-12 735.2Ki ± 0% 724.6Ki ± 1% -1.44% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_path-12 238.873Mi ± 0% 7.585Mi ± 0% -96.82% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_index-12 238.869Mi ± 0% 7.587Mi ± 0% -96.82% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_lower-12 16.77Mi ± 0% 16.77Mi ± 0% ~ (p=0.436 n=10)
ListUnifiedResourcesFilter/search_upper-12 16.78Mi ± 0% 16.76Mi ± 0% ~ (p=0.052 n=10)
ListUnifiedResources/simple_labels-12 1.272Gi ± 0% 1.273Gi ± 0% ~ (p=0.739 n=10)
geomean 49.67Mi 15.69Mi -68.41%
│ e77e5e080b1848da830a9030d43c38be55a5b8d5.txt │ 8eff752a93dfe96225a71431af829015e83400fa.txt │
│ allocs/op │ allocs/op vs base │
ListUnifiedResourcesFilter/labels-12 11.45k ± 0% 11.41k ± 0% -0.37% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_path-12 4962.0k ± 0% 161.5k ± 0% -96.75% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_index-12 4812.0k ± 0% 161.5k ± 0% -96.64% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_lower-12 161.6k ± 0% 161.6k ± 0% ~ (p=0.419 n=10)
ListUnifiedResourcesFilter/search_upper-12 161.7k ± 0% 161.6k ± 0% ~ (p=0.116 n=10)
ListUnifiedResources/simple_labels-12 23.70M ± 0% 23.70M ± 0% +0.01% (p=0.019 n=10)
geomean 743.7k 238.5k -67.93%
```
This PR adds the `patch_type` to `KubernetesWaitingContainer` objects.
This patch type is used to decode the contents correctly depending on
different patch strategies allowed by Kubernetes clients.
Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
* add kubernetes waiting container backend service and API
* remove other resource type
* add unit test
* fix after rebasing
* address feedback
* remove KubeWaitingContainerRequest
* check for nil in validation func
* remove UnwrapResource153
* fix test
* address feedback
* address linter issues
* fix cache watch kind test
* Remove logging from circuit breaker
The logging is never output, and if it was would be very verbose.
Instead of dropping the level down to trace the logger is removed
entirely.
* Add constants from trace library
The logging functionality in trace is being removed in an upcoming
release. To prepare the logging component keys are copied from trace
into teleport.
* Convert trace.ComponentFields to teleport.ComponentFields
* Replace trace.Component with teleport.ComponentKey
* Replace trace logging constants
* AWSOIDC: Move enrollment of EKS clusters to the gRPC service.
* Fix after rebase.
* Add EKS auto discovery for dynamic integration based fetchers.
* Remove commented code.
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Remove unneeded code.
Co-authored-by: Marco André Dinis <marco.dinis@goteleport.com>
* Add missing godoc.
* Initiate map with length.
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Fix typo in the interface name.
* Introduce iam join method for EKS enrollment.
* Correctly process DiscoveredEKSCluster in kube watcher.
* Remove remaining CreateToken related code from Discovery service.
* Change var initialization.
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Remove dev version detection for EKS enrollment.
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
* Revert "Introduce iam join method for EKS enrollment."
This reverts commit 0eb3f4f1226239a5d4e768ebe9129535be318ccb.
* Fix usage of a missing function.
* Refactor cluster features to be a function.
* Change discovery to use gRPC call for EKS enrollment.
* Remove unused mock.
* Correct a comment.
* Remove token creation permission from Discovery service.
* Fix after rebase.
* Improve sorting clusters by region and integration.
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
* Return and error if cluster features are missing.
* Fix tests.
---------
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Marco André Dinis <marco.dinis@goteleport.com>
Co-authored-by: Tiago Silva <tiago.silva@goteleport.com>
The access list interface now supports counting members without unmarshaling.
This method has been added to the generic backend service as well. This will
allow for quicker counts when necessary. This method has been added to the
cache.
Additionally, a status field as per RFD-153 has been added to access lists
that will house the new optional member count field.
* Remove marshal options from GetSessionRecordingConfig
* Add SessionRecordingConfig support to cluster configuration RPC service
Extends the clusterconfigv1.Service to implement RPCs for interacting
with SessionRecordingConfig. The various GetSessionRecordingConfig
interfaces were also all updated to remove the marshal options since
they were never used and served no purpose.
* Remove marshal options from GetClusterNetworkingConfig
* Add ClusterNetworkingConfig support to cluster configuration RPC service
Extends the clusterconfigv1.Service to implement RPCs for interacting
with ClusterNetworkingConfig. The various GetClusterNetworkingConfig
interfaces were also all updated to remove the marshal options since
they were never used and served no purpose.
When relative node expiry was originally implemented it only ran
on Auth and propagated delete events downstream. However, that
was quickly changed in #12002 because the event system could not
keep up in clusters with high churn and would result in buffer
overflow errors in downstream caches. The event system has been
overhauled to use FanoutV2, which no longer suffers from the
burst of events causing problems. The change to not propagate
delete events also breaks any node watchers on the local
cache from ever expiring the server since they never receive
a delete event.
This reverts some of the changes from #12002 such that relative
expiry now only runs on the auth cache and emits delete events.
Each relative expiry interval is however still limited to only
remove up to a fixed number of nodes.
Closes#37527
ListAppSessions was never added to the `auth.Cache` interface or
implemented by `cache.Cache` which means all session listing was
using the backend as the data source. This could cause backend
throttling if the number of sessions was large enough.
* split database CA into server/client CAs
* update migration logging
* include auth component in migrations
* debug log when skipping db client ca migration
* update tctl auth sign for cockroachdb
* update tctl auth sign for oracle
* set ExtKeyUsage for DB certs
* clientAuth for DB Client CA
* serverAuth for DB Server CA
* inject filter for pre-v14.3.1 clients
* inject a filter instead of dropping events in grpcserver
* add DatabaseClientCA type
* debug log when skipping filter injection
* update CA filter injection version check func name and doc
* fix WatchKind.Contains for ca filter
* add CA filter to caCollector
* require CA types in CA watcher config
* apply ca filter in EventsService parser
* Add access lists, members, and reviews to cache.
Access lists, members, and reviews have been added to the cache. To do this,
I have created a simple backend that doesn't perform the validation that is
performed by the main access list backend and only implements a subset of
actions that are needed by the cache.
To support this, I've added in new gRPC methods:
ListAllAccessListMembers and ListAllAccessListReviews. These do not lock
any access lists and additionally do not perform any implicit or dynamic
members, but just simply return the full list of all members and reviews
so that these endpoints can be used to populate the cache.
The general thought here that, since the order of cache events cannot be
guaranteed, we'll rely on the auth server to perform the proper backend
validation and use a simplified backend that simply sees if the items
are present in the cache. If so, we'll return them.
I've had to update a few protos to support this. While I was here, I
fixed a bug in the access list gRPC client where the call tolist of reviews
was not supplying the access list name. Additionally, the user login
state generator will now use the cached access list values.
* Move simple backend to lib/services/simple
* Update to reflect new generic function signatures.
Prefer the stdlib version of the experimental package.
Includes a depguard update to prevent future regressions,
as well as an e-ref update to appease depguard.
Generated with:
$ gofmt -w -r '"golang.org/x/exp/slices" -> "slices"' . && \
make fix-imports
* Discovery: Use Integration credentials for fetchers
* fix typos, improve docs, add clients.closer
* be explicit about using ambient or integration credentials
* remove integration assignment when using ambient credentials
* be explicit about using ambient credentials in db service
* fix cache key for region only sessions
* re-trigger license/cla check
* Add user login state original traits and indicator that original values set.
Original traits has been added to the user login state for use by a watcher in
the enterprise repo that will react to user access changes. This field will be
used to rebuild the last known good state when user access changes.
Additionally, an internal annotation that indicates the original role/trait
fields have been set in the user login state has been introduced so that the
watcher knows whether the user login state can be used by this watcher.
* GCI.
* Tweak tests.
* Further tweak tests.
* Formatting.
* Fix missing argument.
* Standardize delete comment.
Listing users is different from most existing list operations
because multiple resource types are stored under the users prefix.
Care was taken to ensure that these extra resources(passwords,
mfa devices, etc) do not cause lising to omit any users.
In addition to adding server and backend handling for create and
update roles, the services.Access interface was updated to return
a role from the existing Create/UpsertRole methods. Bumps the e
ref to incorporate the associated changes needed there to prevent
breaking the build.
The access lists and members have been removed from the cache. There's not
a lot of value having them in there and they create some uncomfortable
questions since members are dependent on access lists and the cache isn't
designed for dependent operations like this. By removing them we avoid a
lot of pain here.
* DiscoveryConfig: init service and add it to `tctl`
This PR starts the DiscoveryConfig service in gRPC server and allows
`tctl` to interact with those records.
It also adds access to the `editor` role.
Users should be able to RW any DiscoveryConfig.
DiscoveryService should be able to watch those resources, so that it can
act upon any changes.
* add revision
* add upsert method
* improve tctl -f command
* Update e ref
* Complete services.Users interface refactoring
Removes temporary functions required to migrate the interface
without breaking e.
Closes#32949.
services.UsersService now takes a context and returns the user
from write operations as shown in the diff below. The bulk of the
changes are from modifying code to account for the additional
parameter and/or return value. Functional changes to better make
use of the new API will come in follow up PRs.
```diff
// UserGetter is responsible for getting users
type UserGetter interface {
// GetUser returns a user by name
- GetUser(user string, withSecrets bool) (types.User, error)
+ GetUser(ctx context.Context, user string, withSecrets bool) (types.User, error)
}
// UsersService is responsible for basic user management
type UsersService interface {
UserGetter
// CreateUser creates user, only if the user entry does not exist
- CreateUser(user types.User) error
+ CreateUser(ctx context.Context, user types.User) (types.User, error)
// UpdateUser updates an existing user.
- UpdateUser(ctx context.Context, user types.User) error
+ UpdateUser(ctx context.Context, user types.User) (types.User, error)
// UpdateAndSwapUser reads an existing user, runs `fn` against it and writes
// the result to storage. Return `false` from `fn` to avoid storage changes.
// Roughly equivalent to [GetUser] followed by [CompareAndSwapUser].
// Returns the storage user.
UpdateAndSwapUser(ctx context.Context, user string, withSecrets bool, fn func(types.User) (changed bool, err error)) (types.User, error)
// UpsertUser updates parameters about user
- UpsertUser(user types.User) error
+ UpsertUser(ctx context.Context, user types.User) (types.User, error)
// CompareAndSwapUser updates an existing user, but fails if the user does
// not match an expected backend value.
CompareAndSwapUser(ctx context.Context, new, existing types.User) error
// DeleteUser deletes a user with all the keys from the backend
DeleteUser(ctx context.Context, user string) error
// GetUsers returns a list of users registered with the local auth server
- GetUsers(withSecrets bool) ([]types.User, error)
+ GetUsers(ctx context.Context, withSecrets bool) ([]types.User, error)
// DeleteAllUsers deletes all users
- DeleteAllUsers() error
+ DeleteAllUsers(ctx context.Context) error
}
```
Depends on gravitational/teleport.e#2346
Implements step 3 of #32949
* Start migrating services.Identity to use context and return users
Adds new variants of existing methods that are going to be updated
to support propagating context and return users from create, update
and upsert. This is an unfortunate step required because e utilizes
the interface for various functionality. In order to prevent breaking
builds, the temporary methods were added so that e can be converted
to them first, then oss can be updated to the new version of the
interface. Once that is done e will be converted and then the temp
methods will be removed.
* fix typos in comment
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* fix: don't set metadata on existing item in CAS
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* fix: gci
* fix: set resource id on update
---------
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
Access Lists now have audit review recurrence presets. These allow users to
specify review frequencies of 1, 3, 6, or 12 months, and specify the 1st,
15th, or last days of the target month. Presets have been used for their
simplicity over other various recurrence definition mechanisms, as these
presets are much clearer than many of the other options.
* Fix Access List Members cache and eventing.
Two things were happening that were shadowing the Access List members cache
and eventing.
1. In the cache collections, the wrong reader was being assigned to the
lookup map. The correct reader was being used elsewhere, however, so the
caching tests appear to have still been working.
2. The watcher in lib/services/local/events.go apparently collapses prefixes
if they overlap. Prefix `access_list_members` is encompassed by
`access_list`, so the access list members prefix was eliminated from the
watcher. As a result, access list member events were being processed by
the access list parser, which resulted in non-critical warnings.
Local testing and dogfooding has yielded that this has had no apparent impact,
at least in situations without cache propagation. However, I've got a feeling
that this could affect situations with multiple auth servers.
While I'm here, I've eliminated the pointer-to-pointer logic in the access
list unmarshaling, which was excised elsewhere and should be excised here as
well.
* Use ExactKey, fix accessListMemberParser as well.
* Set revision on resources retrieved from the backend
Adds a new `MarshalOption` that ensures the resource revision is
set when unmarshalling a backend item. The new `WithRevision` option
was also applied everywhere that the legacy `WithResourceID` was
being used.
* Prevent storing resource revision in the backend item value
The revision follows the same semantics as the resource id for
marshalling. This prevents both items from showing up in the value
of the backend item, which can prevent compare and swap operations
from completing succesfully. Each backend is responsible for
persisting the revision in some manner.
The existing PreserveResourceID was reused to prevent having to
make multiple copies of a resource when clearing the id and revision.
The marshal option will be updated in a follow up when the resource
id is removed.
* Ignore revision in resource comparisons
* Remove use of require assertions inside Eventually calls
require.Eventually runs the predicate function in a background
goroutine. It is invalid to use require to make assertions
inside the eventually, because require will fail the test if the
assertion fails, and tests can only be failed from the test's
main goroutine.
* Use EventuallyWithT
* Add access list members to the cache.
Access list members have been added to the cache. To support this, the access
list member gRPC client methods have been implemented. A few extra gRPC
methods were added to the access list service. Additionally, a few changes
were made to the cache tests since the backend for the access list now
performs multiple backend calls.
* Further tuning of backend.
* Revert removal of access list members assignment.
* Update lib/cache/cache_test.go
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
* Remove debugging statements, tune legacy header conversion.
* Remove AccessListMembers reference, as it's no longer needed.
---------
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
Brian Joerger