usagereporter: Add Identity Security streaming/aggregating events (#63753)

* proto/prehog: Sync from cloud repo for access graph usage

Sync the `prehog.v1` proto package sfrom the cloud
repo to bring in the changes for access graph usage events.

* proto/accessgraph: Extend EventsStreamV2 response for usage events

Extend the EventsStreamV2 response to allow usage events to be sent back
to teleport.

The usage events that can be sent back are currently all defined in the
`prehog.v1alpha` protobuf package. As this does not have a `go_package`
statement, we need to add a `M` flag to the buf config for generating
the Go code. There is no canonical package for the generated code as the
protos live and are generated in two repos (cloud and teleport).

* proto: Regenerate proto/grpc code for access graph usage events

Regenerate the generated proto code to include the changes for access
graph usage events. This includes the following packages:
- `prehog.v1` (Go and Typescript)
- `accessgraph.v1alpha` (Go)

The contents of this commit were auto-generated with:

    make grpc/host

* usagereporter: Add anonymizer for identity security events

Add types compatible with `Anonymizer` for the two identity security
events:
* `prehogv1a.IdentitySecurityGraphSizeEvent`
* `prehogv1a.IdentitySecurityAuditLogsIngestedEvent`

These events are directly transformed as they contain no anonymizable
data.

* usage: Aggregate access graph queries per user

Extend `aggregating.Reporter` to aggregate the number of access graph
queries per user so we can generate monthly user active reports of
access graph usage for self-hosted Teleport customers.

* aggregating: Rename some identitySecurity -> sessionSummary

Rename some variables and functions in the aggregating reporter to more
tightly scope the naming, using "sessionSummary" instead of
"identitySecurity". Another report is to be added named
"identitySecurity" as an aggregation of more general data about the
product.

The existing report defined in the proto -
`IdentitySecuritySummariesGeneratedReport` has been retained as that is
a larger change to make across repositories and with "Summaries" in the
name, is scoped enough to differentiate it from the upcoming report.

* aggregating: Aggregate identity security events into new report

Aggregate the `IdentitySecurityGraphSizeEvent` and
`IdentitySecurityAuditLogsIngestedEvent` events into the
`IdentitySecurityReport` aggregated report.

Aggregation for the graph size events is simply taking the last size of
each provider, as these are essentially guages reporting the size at a
point in time. Aggregation for the audit log events are an accumulation
by provider giving a total count over the report window.
This commit is contained in:
Cam Hutchison
2026-04-10 03:46:59 +00:00
committed by GitHub
parent 759203dc6a
commit 8a32b4d971
13 changed files with 1662 additions and 426 deletions
+6
View File
@@ -32,6 +32,9 @@ plugins:
# needed by teleport/lib/teleterm/v1/usage_events.proto because we use
# managed mode for the go package name there
- Mprehog/v1alpha/connect.proto=github.com/gravitational/teleport/gen/proto/go/prehog/v1alpha;prehogv1alpha
# needed by proto/accessraph/v1alpha/access_graph_service.go because it imports it for
# usage events
- Mprehog/v1alpha/teleport.proto=github.com/gravitational/teleport/gen/proto/go/prehog/v1alpha;prehogv1alpha
strategy: all
- local:
- go
@@ -43,4 +46,7 @@ plugins:
# needed by teleport/lib/teleterm/v1/usage_events.proto because we use
# managed mode for the go package name there
- Mprehog/v1alpha/connect.proto=github.com/gravitational/teleport/gen/proto/go/prehog/v1alpha;prehogv1alpha
# needed by proto/accessraph/v1alpha/access_graph_service.go because it imports it for
# usage events
- Mprehog/v1alpha/teleport.proto=github.com/gravitational/teleport/gen/proto/go/prehog/v1alpha;prehogv1alpha
strategy: all
File diff suppressed because it is too large Load Diff
+303 -34
View File
@@ -417,8 +417,10 @@ type UserActivityRecord struct {
SamlIdpSessions uint64 `protobuf:"varint,24,opt,name=saml_idp_sessions,json=samlIdpSessions,proto3" json:"saml_idp_sessions,omitempty"`
// counter of session summaries accessed by this user per session type and resource name.
SessionSummariesAccessed []*SessionSummariesAccessedRecord `protobuf:"bytes,25,rep,name=session_summaries_accessed,json=sessionSummariesAccessed,proto3" json:"session_summaries_accessed,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// counter of access graph queries by this user.
AccessGraphQueries uint64 `protobuf:"varint,26,opt,name=access_graph_queries,json=accessGraphQueries,proto3" json:"access_graph_queries,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *UserActivityRecord) Reset() {
@@ -627,6 +629,13 @@ func (x *UserActivityRecord) GetSessionSummariesAccessed() []*SessionSummariesAc
return nil
}
func (x *UserActivityRecord) GetAccessGraphQueries() uint64 {
if x != nil {
return x.AccessGraphQueries
}
return 0
}
type ResourcePresenceReport struct {
state protoimpl.MessageState `protogen:"open.v1"`
// randomly generated UUID for this specific report, 16 bytes (in string order)
@@ -1247,6 +1256,232 @@ func (x *SessionSummariesGeneratedRecord) GetSummariesGenerated() uint64 {
return 0
}
// IdentitySecurityReport is a report of Identity Security non-user usage. It currently
// includes the size of the access graph by provider and count of audit logs ingested
// by provider.
type IdentitySecurityReport struct {
state protoimpl.MessageState `protogen:"open.v1"`
// randomly generated UUID for this specific report, 16 bytes (in string order)
//
// PostHog property: tp.report_uuid (in 8-4-4-4-12 string form)
ReportUuid []byte `protobuf:"bytes,1,opt,name=report_uuid,json=reportUuid,proto3" json:"report_uuid,omitempty"`
// cluster name, anonymized, 32 bytes (HMAC-SHA-256)
//
// PostHog property: tp.cluster_name (in base64)
ClusterName []byte `protobuf:"bytes,2,opt,name=cluster_name,json=clusterName,proto3" json:"cluster_name,omitempty"`
// hostid of the auth that collected this report, anonymized, 32 bytes (HMAC-SHA-256)
//
// PostHog property: tp.reporter_hostid (in base64)
ReporterHostid []byte `protobuf:"bytes,3,opt,name=reporter_hostid,json=reporterHostid,proto3" json:"reporter_hostid,omitempty"`
// beginning of the time window for this data; ending is not specified but is
// intended to be at most 60 minutes
//
// PostHog timestamp (not a property, the ingest time is tp.report_time instead)
StartTime *timestamppb.Timestamp `protobuf:"bytes,4,opt,name=start_time,json=startTime,proto3" json:"start_time,omitempty"`
// graph_size contains the size of each provider's graph: count of identities and
// count of resources.
GraphSizeRecords []*IdentitySecurityGraphSize `protobuf:"bytes,5,rep,name=graph_size_records,json=graphSizeRecords,proto3" json:"graph_size_records,omitempty"`
// audit_logs contains the counts of audit logs ingested from each provider.
AuditLogRecords []*IdentitySecurityAuditLogsIngested `protobuf:"bytes,6,rep,name=audit_log_records,json=auditLogRecords,proto3" json:"audit_log_records,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *IdentitySecurityReport) Reset() {
*x = IdentitySecurityReport{}
mi := &file_prehog_v1_teleport_proto_msgTypes[10]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *IdentitySecurityReport) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*IdentitySecurityReport) ProtoMessage() {}
func (x *IdentitySecurityReport) ProtoReflect() protoreflect.Message {
mi := &file_prehog_v1_teleport_proto_msgTypes[10]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use IdentitySecurityReport.ProtoReflect.Descriptor instead.
func (*IdentitySecurityReport) Descriptor() ([]byte, []int) {
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{10}
}
func (x *IdentitySecurityReport) GetReportUuid() []byte {
if x != nil {
return x.ReportUuid
}
return nil
}
func (x *IdentitySecurityReport) GetClusterName() []byte {
if x != nil {
return x.ClusterName
}
return nil
}
func (x *IdentitySecurityReport) GetReporterHostid() []byte {
if x != nil {
return x.ReporterHostid
}
return nil
}
func (x *IdentitySecurityReport) GetStartTime() *timestamppb.Timestamp {
if x != nil {
return x.StartTime
}
return nil
}
func (x *IdentitySecurityReport) GetGraphSizeRecords() []*IdentitySecurityGraphSize {
if x != nil {
return x.GraphSizeRecords
}
return nil
}
func (x *IdentitySecurityReport) GetAuditLogRecords() []*IdentitySecurityAuditLogsIngested {
if x != nil {
return x.AuditLogRecords
}
return nil
}
// IdentitySecurityGraphSize is submitted by Access Graph for each provider
// for which it has a graph.
type IdentitySecurityGraphSize struct {
state protoimpl.MessageState `protogen:"open.v1"`
// provider is the system containing the identities and resources being counted.
// It is one of teleport, aws, azure, entra, gitlab, github, netiq or okta.
Provider string `protobuf:"bytes,1,opt,name=provider,proto3" json:"provider,omitempty"`
// total_identities is the number of identities in the graph.
TotalIdentities uint64 `protobuf:"varint,2,opt,name=total_identities,json=totalIdentities,proto3" json:"total_identities,omitempty"`
// total_resources is the number of resources in the graph.
TotalResources uint64 `protobuf:"varint,3,opt,name=total_resources,json=totalResources,proto3" json:"total_resources,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *IdentitySecurityGraphSize) Reset() {
*x = IdentitySecurityGraphSize{}
mi := &file_prehog_v1_teleport_proto_msgTypes[11]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *IdentitySecurityGraphSize) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*IdentitySecurityGraphSize) ProtoMessage() {}
func (x *IdentitySecurityGraphSize) ProtoReflect() protoreflect.Message {
mi := &file_prehog_v1_teleport_proto_msgTypes[11]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use IdentitySecurityGraphSize.ProtoReflect.Descriptor instead.
func (*IdentitySecurityGraphSize) Descriptor() ([]byte, []int) {
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{11}
}
func (x *IdentitySecurityGraphSize) GetProvider() string {
if x != nil {
return x.Provider
}
return ""
}
func (x *IdentitySecurityGraphSize) GetTotalIdentities() uint64 {
if x != nil {
return x.TotalIdentities
}
return 0
}
func (x *IdentitySecurityGraphSize) GetTotalResources() uint64 {
if x != nil {
return x.TotalResources
}
return 0
}
// IdentitySecurityAuditLogsIngested tracks the count of log entries ingested by
// identity activity center.
type IdentitySecurityAuditLogsIngested struct {
state protoimpl.MessageState `protogen:"open.v1"`
// provider is the system emitting audit logs. It is one of
// teleport, cloudtrail, kubernetes, github or okta.
Provider string `protobuf:"bytes,1,opt,name=provider,proto3" json:"provider,omitempty"`
// logs_ingested is a count of log entries ingested into Identity Security.
LogsIngested uint64 `protobuf:"varint,2,opt,name=logs_ingested,json=logsIngested,proto3" json:"logs_ingested,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *IdentitySecurityAuditLogsIngested) Reset() {
*x = IdentitySecurityAuditLogsIngested{}
mi := &file_prehog_v1_teleport_proto_msgTypes[12]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *IdentitySecurityAuditLogsIngested) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*IdentitySecurityAuditLogsIngested) ProtoMessage() {}
func (x *IdentitySecurityAuditLogsIngested) ProtoReflect() protoreflect.Message {
mi := &file_prehog_v1_teleport_proto_msgTypes[12]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use IdentitySecurityAuditLogsIngested.ProtoReflect.Descriptor instead.
func (*IdentitySecurityAuditLogsIngested) Descriptor() ([]byte, []int) {
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{12}
}
func (x *IdentitySecurityAuditLogsIngested) GetProvider() string {
if x != nil {
return x.Provider
}
return ""
}
func (x *IdentitySecurityAuditLogsIngested) GetLogsIngested() uint64 {
if x != nil {
return x.LogsIngested
}
return 0
}
type SubmitUsageReportsRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// at most 10 reports of all kinds in a single RPC, each shouldn't exceed 128KiB or so
@@ -1259,13 +1494,15 @@ type SubmitUsageReportsRequest struct {
BotInstanceActivity []*BotInstanceActivityReport `protobuf:"bytes,3,rep,name=bot_instance_activity,json=botInstanceActivity,proto3" json:"bot_instance_activity,omitempty"`
// encoded as a separate tp.identity_security.summaries_generated PostHog event
IdentitySecuritySummariesReport []*IdentitySecuritySummariesGeneratedReport `protobuf:"bytes,4,rep,name=identity_security_summaries_report,json=identitySecuritySummariesReport,proto3" json:"identity_security_summaries_report,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
// encoded as a separate tp.identity_security.usage PostHog event
IdentitySecurityReport []*IdentitySecurityReport `protobuf:"bytes,5,rep,name=identity_security_report,json=identitySecurityReport,proto3" json:"identity_security_report,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *SubmitUsageReportsRequest) Reset() {
*x = SubmitUsageReportsRequest{}
mi := &file_prehog_v1_teleport_proto_msgTypes[10]
mi := &file_prehog_v1_teleport_proto_msgTypes[13]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1277,7 +1514,7 @@ func (x *SubmitUsageReportsRequest) String() string {
func (*SubmitUsageReportsRequest) ProtoMessage() {}
func (x *SubmitUsageReportsRequest) ProtoReflect() protoreflect.Message {
mi := &file_prehog_v1_teleport_proto_msgTypes[10]
mi := &file_prehog_v1_teleport_proto_msgTypes[13]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1290,7 +1527,7 @@ func (x *SubmitUsageReportsRequest) ProtoReflect() protoreflect.Message {
// Deprecated: Use SubmitUsageReportsRequest.ProtoReflect.Descriptor instead.
func (*SubmitUsageReportsRequest) Descriptor() ([]byte, []int) {
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{10}
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{13}
}
func (x *SubmitUsageReportsRequest) GetUserActivity() []*UserActivityReport {
@@ -1321,6 +1558,13 @@ func (x *SubmitUsageReportsRequest) GetIdentitySecuritySummariesReport() []*Iden
return nil
}
func (x *SubmitUsageReportsRequest) GetIdentitySecurityReport() []*IdentitySecurityReport {
if x != nil {
return x.IdentitySecurityReport
}
return nil
}
type SubmitUsageReportsResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
// randomly generated UUID for this specific batch, 16 bytes (in string order)
@@ -1333,7 +1577,7 @@ type SubmitUsageReportsResponse struct {
func (x *SubmitUsageReportsResponse) Reset() {
*x = SubmitUsageReportsResponse{}
mi := &file_prehog_v1_teleport_proto_msgTypes[11]
mi := &file_prehog_v1_teleport_proto_msgTypes[14]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -1345,7 +1589,7 @@ func (x *SubmitUsageReportsResponse) String() string {
func (*SubmitUsageReportsResponse) ProtoMessage() {}
func (x *SubmitUsageReportsResponse) ProtoReflect() protoreflect.Message {
mi := &file_prehog_v1_teleport_proto_msgTypes[11]
mi := &file_prehog_v1_teleport_proto_msgTypes[14]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -1358,7 +1602,7 @@ func (x *SubmitUsageReportsResponse) ProtoReflect() protoreflect.Message {
// Deprecated: Use SubmitUsageReportsResponse.ProtoReflect.Descriptor instead.
func (*SubmitUsageReportsResponse) Descriptor() ([]byte, []int) {
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{11}
return file_prehog_v1_teleport_proto_rawDescGZIP(), []int{14}
}
func (x *SubmitUsageReportsResponse) GetBatchUuid() []byte {
@@ -1380,7 +1624,7 @@ const file_prehog_v1_teleport_proto_rawDesc = "" +
"\x0freporter_hostid\x18\x03 \x01(\fR\x0ereporterHostid\x129\n" +
"\n" +
"start_time\x18\x04 \x01(\v2\x1a.google.protobuf.TimestampR\tstartTime\x127\n" +
"\arecords\x18\x05 \x03(\v2\x1d.prehog.v1.UserActivityRecordR\arecords\"\x99\t\n" +
"\arecords\x18\x05 \x03(\v2\x1d.prehog.v1.UserActivityRecordR\arecords\"\xcb\t\n" +
"\x12UserActivityRecord\x12\x1b\n" +
"\tuser_name\x18\x01 \x01(\fR\buserName\x120\n" +
"\tuser_kind\x18\x0e \x01(\x0e2\x13.prehog.v1.UserKindR\buserKind\x12\x16\n" +
@@ -1410,7 +1654,8 @@ const file_prehog_v1_teleport_proto_rawDesc = "" +
"\x15access_lists_reviewed\x18\x16 \x01(\x04R\x13accessListsReviewed\x12.\n" +
"\x13access_lists_grants\x18\x17 \x01(\x04R\x11accessListsGrants\x12*\n" +
"\x11saml_idp_sessions\x18\x18 \x01(\x04R\x0fsamlIdpSessions\x12g\n" +
"\x1asession_summaries_accessed\x18\x19 \x03(\v2).prehog.v1.SessionSummariesAccessedRecordR\x18sessionSummariesAccessed\"\x9b\x02\n" +
"\x1asession_summaries_accessed\x18\x19 \x03(\v2).prehog.v1.SessionSummariesAccessedRecordR\x18sessionSummariesAccessed\x120\n" +
"\x14access_graph_queries\x18\x1a \x01(\x04R\x12accessGraphQueries\"\x9b\x02\n" +
"\x16ResourcePresenceReport\x12\x1f\n" +
"\vreport_uuid\x18\x01 \x01(\fR\n" +
"reportUuid\x12!\n" +
@@ -1456,12 +1701,29 @@ const file_prehog_v1_teleport_proto_rawDesc = "" +
"\rresource_name\x18\x02 \x01(\tR\fresourceName\x12,\n" +
"\x12total_input_tokens\x18\x04 \x01(\x04R\x10totalInputTokens\x12.\n" +
"\x13total_output_tokens\x18\x05 \x01(\x04R\x11totalOutputTokens\x12/\n" +
"\x13summaries_generated\x18\x06 \x01(\x04R\x12summariesGenerated\"\x8c\x03\n" +
"\x13summaries_generated\x18\x06 \x01(\x04R\x12summariesGenerated\"\xee\x02\n" +
"\x16IdentitySecurityReport\x12\x1f\n" +
"\vreport_uuid\x18\x01 \x01(\fR\n" +
"reportUuid\x12!\n" +
"\fcluster_name\x18\x02 \x01(\fR\vclusterName\x12'\n" +
"\x0freporter_hostid\x18\x03 \x01(\fR\x0ereporterHostid\x129\n" +
"\n" +
"start_time\x18\x04 \x01(\v2\x1a.google.protobuf.TimestampR\tstartTime\x12R\n" +
"\x12graph_size_records\x18\x05 \x03(\v2$.prehog.v1.IdentitySecurityGraphSizeR\x10graphSizeRecords\x12X\n" +
"\x11audit_log_records\x18\x06 \x03(\v2,.prehog.v1.IdentitySecurityAuditLogsIngestedR\x0fauditLogRecords\"\x8b\x01\n" +
"\x19IdentitySecurityGraphSize\x12\x1a\n" +
"\bprovider\x18\x01 \x01(\tR\bprovider\x12)\n" +
"\x10total_identities\x18\x02 \x01(\x04R\x0ftotalIdentities\x12'\n" +
"\x0ftotal_resources\x18\x03 \x01(\x04R\x0etotalResources\"d\n" +
"!IdentitySecurityAuditLogsIngested\x12\x1a\n" +
"\bprovider\x18\x01 \x01(\tR\bprovider\x12#\n" +
"\rlogs_ingested\x18\x02 \x01(\x04R\flogsIngested\"\xe9\x03\n" +
"\x19SubmitUsageReportsRequest\x12B\n" +
"\ruser_activity\x18\x01 \x03(\v2\x1d.prehog.v1.UserActivityReportR\fuserActivity\x12N\n" +
"\x11resource_presence\x18\x02 \x03(\v2!.prehog.v1.ResourcePresenceReportR\x10resourcePresence\x12X\n" +
"\x15bot_instance_activity\x18\x03 \x03(\v2$.prehog.v1.BotInstanceActivityReportR\x13botInstanceActivity\x12\x80\x01\n" +
"\"identity_security_summaries_report\x18\x04 \x03(\v23.prehog.v1.IdentitySecuritySummariesGeneratedReportR\x1fidentitySecuritySummariesReport\";\n" +
"\"identity_security_summaries_report\x18\x04 \x03(\v23.prehog.v1.IdentitySecuritySummariesGeneratedReportR\x1fidentitySecuritySummariesReport\x12[\n" +
"\x18identity_security_report\x18\x05 \x03(\v2!.prehog.v1.IdentitySecurityReportR\x16identitySecurityReport\";\n" +
"\x1aSubmitUsageReportsResponse\x12\x1d\n" +
"\n" +
"batch_uuid\x18\x01 \x01(\fR\tbatchUuid*c\n" +
@@ -1505,7 +1767,7 @@ func file_prehog_v1_teleport_proto_rawDescGZIP() []byte {
}
var file_prehog_v1_teleport_proto_enumTypes = make([]protoimpl.EnumInfo, 3)
var file_prehog_v1_teleport_proto_msgTypes = make([]protoimpl.MessageInfo, 12)
var file_prehog_v1_teleport_proto_msgTypes = make([]protoimpl.MessageInfo, 15)
var file_prehog_v1_teleport_proto_goTypes = []any{
(UserKind)(0), // 0: prehog.v1.UserKind
(UserOrigin)(0), // 1: prehog.v1.UserOrigin
@@ -1520,35 +1782,42 @@ var file_prehog_v1_teleport_proto_goTypes = []any{
(*SPIFFEIDRecord)(nil), // 10: prehog.v1.SPIFFEIDRecord
(*IdentitySecuritySummariesGeneratedReport)(nil), // 11: prehog.v1.IdentitySecuritySummariesGeneratedReport
(*SessionSummariesGeneratedRecord)(nil), // 12: prehog.v1.SessionSummariesGeneratedRecord
(*SubmitUsageReportsRequest)(nil), // 13: prehog.v1.SubmitUsageReportsRequest
(*SubmitUsageReportsResponse)(nil), // 14: prehog.v1.SubmitUsageReportsResponse
(*timestamppb.Timestamp)(nil), // 15: google.protobuf.Timestamp
(*IdentitySecurityReport)(nil), // 13: prehog.v1.IdentitySecurityReport
(*IdentitySecurityGraphSize)(nil), // 14: prehog.v1.IdentitySecurityGraphSize
(*IdentitySecurityAuditLogsIngested)(nil), // 15: prehog.v1.IdentitySecurityAuditLogsIngested
(*SubmitUsageReportsRequest)(nil), // 16: prehog.v1.SubmitUsageReportsRequest
(*SubmitUsageReportsResponse)(nil), // 17: prehog.v1.SubmitUsageReportsResponse
(*timestamppb.Timestamp)(nil), // 18: google.protobuf.Timestamp
}
var file_prehog_v1_teleport_proto_depIdxs = []int32{
15, // 0: prehog.v1.UserActivityReport.start_time:type_name -> google.protobuf.Timestamp
18, // 0: prehog.v1.UserActivityReport.start_time:type_name -> google.protobuf.Timestamp
4, // 1: prehog.v1.UserActivityReport.records:type_name -> prehog.v1.UserActivityRecord
0, // 2: prehog.v1.UserActivityRecord.user_kind:type_name -> prehog.v1.UserKind
10, // 3: prehog.v1.UserActivityRecord.spiffe_ids_issued:type_name -> prehog.v1.SPIFFEIDRecord
1, // 4: prehog.v1.UserActivityRecord.user_origin:type_name -> prehog.v1.UserOrigin
7, // 5: prehog.v1.UserActivityRecord.session_summaries_accessed:type_name -> prehog.v1.SessionSummariesAccessedRecord
15, // 6: prehog.v1.ResourcePresenceReport.start_time:type_name -> google.protobuf.Timestamp
18, // 6: prehog.v1.ResourcePresenceReport.start_time:type_name -> google.protobuf.Timestamp
6, // 7: prehog.v1.ResourcePresenceReport.resource_kind_reports:type_name -> prehog.v1.ResourceKindPresenceReport
2, // 8: prehog.v1.ResourceKindPresenceReport.resource_kind:type_name -> prehog.v1.ResourceKind
15, // 9: prehog.v1.BotInstanceActivityReport.start_time:type_name -> google.protobuf.Timestamp
18, // 9: prehog.v1.BotInstanceActivityReport.start_time:type_name -> google.protobuf.Timestamp
9, // 10: prehog.v1.BotInstanceActivityReport.records:type_name -> prehog.v1.BotInstanceActivityRecord
15, // 11: prehog.v1.IdentitySecuritySummariesGeneratedReport.start_time:type_name -> google.protobuf.Timestamp
18, // 11: prehog.v1.IdentitySecuritySummariesGeneratedReport.start_time:type_name -> google.protobuf.Timestamp
12, // 12: prehog.v1.IdentitySecuritySummariesGeneratedReport.records:type_name -> prehog.v1.SessionSummariesGeneratedRecord
3, // 13: prehog.v1.SubmitUsageReportsRequest.user_activity:type_name -> prehog.v1.UserActivityReport
5, // 14: prehog.v1.SubmitUsageReportsRequest.resource_presence:type_name -> prehog.v1.ResourcePresenceReport
8, // 15: prehog.v1.SubmitUsageReportsRequest.bot_instance_activity:type_name -> prehog.v1.BotInstanceActivityReport
11, // 16: prehog.v1.SubmitUsageReportsRequest.identity_security_summaries_report:type_name -> prehog.v1.IdentitySecuritySummariesGeneratedReport
13, // 17: prehog.v1.TeleportReportingService.SubmitUsageReports:input_type -> prehog.v1.SubmitUsageReportsRequest
14, // 18: prehog.v1.TeleportReportingService.SubmitUsageReports:output_type -> prehog.v1.SubmitUsageReportsResponse
18, // [18:19] is the sub-list for method output_type
17, // [17:18] is the sub-list for method input_type
17, // [17:17] is the sub-list for extension type_name
17, // [17:17] is the sub-list for extension extendee
0, // [0:17] is the sub-list for field type_name
18, // 13: prehog.v1.IdentitySecurityReport.start_time:type_name -> google.protobuf.Timestamp
14, // 14: prehog.v1.IdentitySecurityReport.graph_size_records:type_name -> prehog.v1.IdentitySecurityGraphSize
15, // 15: prehog.v1.IdentitySecurityReport.audit_log_records:type_name -> prehog.v1.IdentitySecurityAuditLogsIngested
3, // 16: prehog.v1.SubmitUsageReportsRequest.user_activity:type_name -> prehog.v1.UserActivityReport
5, // 17: prehog.v1.SubmitUsageReportsRequest.resource_presence:type_name -> prehog.v1.ResourcePresenceReport
8, // 18: prehog.v1.SubmitUsageReportsRequest.bot_instance_activity:type_name -> prehog.v1.BotInstanceActivityReport
11, // 19: prehog.v1.SubmitUsageReportsRequest.identity_security_summaries_report:type_name -> prehog.v1.IdentitySecuritySummariesGeneratedReport
13, // 20: prehog.v1.SubmitUsageReportsRequest.identity_security_report:type_name -> prehog.v1.IdentitySecurityReport
16, // 21: prehog.v1.TeleportReportingService.SubmitUsageReports:input_type -> prehog.v1.SubmitUsageReportsRequest
17, // 22: prehog.v1.TeleportReportingService.SubmitUsageReports:output_type -> prehog.v1.SubmitUsageReportsResponse
22, // [22:23] is the sub-list for method output_type
21, // [21:22] is the sub-list for method input_type
21, // [21:21] is the sub-list for extension type_name
21, // [21:21] is the sub-list for extension extendee
0, // [0:21] is the sub-list for field type_name
}
func init() { file_prehog_v1_teleport_proto_init() }
@@ -1562,7 +1831,7 @@ func file_prehog_v1_teleport_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_prehog_v1_teleport_proto_rawDesc), len(file_prehog_v1_teleport_proto_rawDesc)),
NumEnums: 3,
NumMessages: 12,
NumMessages: 15,
NumExtensions: 0,
NumServices: 1,
},
+337 -2
View File
@@ -249,6 +249,12 @@ export interface UserActivityRecord {
* @generated from protobuf field: repeated prehog.v1.SessionSummariesAccessedRecord session_summaries_accessed = 25;
*/
sessionSummariesAccessed: SessionSummariesAccessedRecord[];
/**
* counter of access graph queries by this user.
*
* @generated from protobuf field: uint64 access_graph_queries = 26;
*/
accessGraphQueries: bigint;
}
/**
* @generated from protobuf message prehog.v1.ResourcePresenceReport
@@ -531,6 +537,109 @@ export interface SessionSummariesGeneratedRecord {
*/
summariesGenerated: bigint;
}
/**
* IdentitySecurityReport is a report of Identity Security non-user usage. It currently
* includes the size of the access graph by provider and count of audit logs ingested
* by provider.
*
* @generated from protobuf message prehog.v1.IdentitySecurityReport
*/
export interface IdentitySecurityReport {
/**
* randomly generated UUID for this specific report, 16 bytes (in string order)
*
* PostHog property: tp.report_uuid (in 8-4-4-4-12 string form)
*
* @generated from protobuf field: bytes report_uuid = 1;
*/
reportUuid: Uint8Array;
/**
* cluster name, anonymized, 32 bytes (HMAC-SHA-256)
*
* PostHog property: tp.cluster_name (in base64)
*
* @generated from protobuf field: bytes cluster_name = 2;
*/
clusterName: Uint8Array;
/**
* hostid of the auth that collected this report, anonymized, 32 bytes (HMAC-SHA-256)
*
* PostHog property: tp.reporter_hostid (in base64)
*
* @generated from protobuf field: bytes reporter_hostid = 3;
*/
reporterHostid: Uint8Array;
/**
* beginning of the time window for this data; ending is not specified but is
* intended to be at most 60 minutes
*
* PostHog timestamp (not a property, the ingest time is tp.report_time instead)
*
* @generated from protobuf field: google.protobuf.Timestamp start_time = 4;
*/
startTime?: Timestamp;
/**
* graph_size contains the size of each provider's graph: count of identities and
* count of resources.
*
* @generated from protobuf field: repeated prehog.v1.IdentitySecurityGraphSize graph_size_records = 5;
*/
graphSizeRecords: IdentitySecurityGraphSize[];
/**
* audit_logs contains the counts of audit logs ingested from each provider.
*
* @generated from protobuf field: repeated prehog.v1.IdentitySecurityAuditLogsIngested audit_log_records = 6;
*/
auditLogRecords: IdentitySecurityAuditLogsIngested[];
}
/**
* IdentitySecurityGraphSize is submitted by Access Graph for each provider
* for which it has a graph.
*
* @generated from protobuf message prehog.v1.IdentitySecurityGraphSize
*/
export interface IdentitySecurityGraphSize {
/**
* provider is the system containing the identities and resources being counted.
* It is one of teleport, aws, azure, entra, gitlab, github, netiq or okta.
*
* @generated from protobuf field: string provider = 1;
*/
provider: string;
/**
* total_identities is the number of identities in the graph.
*
* @generated from protobuf field: uint64 total_identities = 2;
*/
totalIdentities: bigint;
/**
* total_resources is the number of resources in the graph.
*
* @generated from protobuf field: uint64 total_resources = 3;
*/
totalResources: bigint;
}
/**
* IdentitySecurityAuditLogsIngested tracks the count of log entries ingested by
* identity activity center.
*
* @generated from protobuf message prehog.v1.IdentitySecurityAuditLogsIngested
*/
export interface IdentitySecurityAuditLogsIngested {
/**
* provider is the system emitting audit logs. It is one of
* teleport, cloudtrail, kubernetes, github or okta.
*
* @generated from protobuf field: string provider = 1;
*/
provider: string;
/**
* logs_ingested is a count of log entries ingested into Identity Security.
*
* @generated from protobuf field: uint64 logs_ingested = 2;
*/
logsIngested: bigint;
}
/**
* @generated from protobuf message prehog.v1.SubmitUsageReportsRequest
*/
@@ -561,6 +670,12 @@ export interface SubmitUsageReportsRequest {
* @generated from protobuf field: repeated prehog.v1.IdentitySecuritySummariesGeneratedReport identity_security_summaries_report = 4;
*/
identitySecuritySummariesReport: IdentitySecuritySummariesGeneratedReport[];
/**
* encoded as a separate tp.identity_security.usage PostHog event
*
* @generated from protobuf field: repeated prehog.v1.IdentitySecurityReport identity_security_report = 5;
*/
identitySecurityReport: IdentitySecurityReport[];
}
/**
* @generated from protobuf message prehog.v1.SubmitUsageReportsResponse
@@ -832,7 +947,8 @@ class UserActivityRecord$Type extends MessageType<UserActivityRecord> {
{ no: 22, name: "access_lists_reviewed", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ },
{ no: 23, name: "access_lists_grants", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ },
{ no: 24, name: "saml_idp_sessions", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ },
{ no: 25, name: "session_summaries_accessed", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => SessionSummariesAccessedRecord }
{ no: 25, name: "session_summaries_accessed", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => SessionSummariesAccessedRecord },
{ no: 26, name: "access_graph_queries", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ }
]);
}
create(value?: PartialMessage<UserActivityRecord>): UserActivityRecord {
@@ -862,6 +978,7 @@ class UserActivityRecord$Type extends MessageType<UserActivityRecord> {
message.accessListsGrants = 0n;
message.samlIdpSessions = 0n;
message.sessionSummariesAccessed = [];
message.accessGraphQueries = 0n;
if (value !== undefined)
reflectionMergePartial<UserActivityRecord>(this, message, value);
return message;
@@ -946,6 +1063,9 @@ class UserActivityRecord$Type extends MessageType<UserActivityRecord> {
case /* repeated prehog.v1.SessionSummariesAccessedRecord session_summaries_accessed */ 25:
message.sessionSummariesAccessed.push(SessionSummariesAccessedRecord.internalBinaryRead(reader, reader.uint32(), options));
break;
case /* uint64 access_graph_queries */ 26:
message.accessGraphQueries = reader.uint64().toBigInt();
break;
default:
let u = options.readUnknownField;
if (u === "throw")
@@ -1033,6 +1153,9 @@ class UserActivityRecord$Type extends MessageType<UserActivityRecord> {
/* repeated prehog.v1.SessionSummariesAccessedRecord session_summaries_accessed = 25; */
for (let i = 0; i < message.sessionSummariesAccessed.length; i++)
SessionSummariesAccessedRecord.internalBinaryWrite(message.sessionSummariesAccessed[i], writer.tag(25, WireType.LengthDelimited).fork(), options).join();
/* uint64 access_graph_queries = 26; */
if (message.accessGraphQueries !== 0n)
writer.tag(26, WireType.Varint).uint64(message.accessGraphQueries);
let u = options.writeUnknownFields;
if (u !== false)
(u == true ? UnknownFieldHandler.onWrite : u)(this.typeName, message, writer);
@@ -1617,13 +1740,218 @@ class SessionSummariesGeneratedRecord$Type extends MessageType<SessionSummariesG
*/
export const SessionSummariesGeneratedRecord = new SessionSummariesGeneratedRecord$Type();
// @generated message type with reflection information, may provide speed optimized methods
class IdentitySecurityReport$Type extends MessageType<IdentitySecurityReport> {
constructor() {
super("prehog.v1.IdentitySecurityReport", [
{ no: 1, name: "report_uuid", kind: "scalar", T: 12 /*ScalarType.BYTES*/ },
{ no: 2, name: "cluster_name", kind: "scalar", T: 12 /*ScalarType.BYTES*/ },
{ no: 3, name: "reporter_hostid", kind: "scalar", T: 12 /*ScalarType.BYTES*/ },
{ no: 4, name: "start_time", kind: "message", T: () => Timestamp },
{ no: 5, name: "graph_size_records", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => IdentitySecurityGraphSize },
{ no: 6, name: "audit_log_records", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => IdentitySecurityAuditLogsIngested }
]);
}
create(value?: PartialMessage<IdentitySecurityReport>): IdentitySecurityReport {
const message = globalThis.Object.create((this.messagePrototype!));
message.reportUuid = new Uint8Array(0);
message.clusterName = new Uint8Array(0);
message.reporterHostid = new Uint8Array(0);
message.graphSizeRecords = [];
message.auditLogRecords = [];
if (value !== undefined)
reflectionMergePartial<IdentitySecurityReport>(this, message, value);
return message;
}
internalBinaryRead(reader: IBinaryReader, length: number, options: BinaryReadOptions, target?: IdentitySecurityReport): IdentitySecurityReport {
let message = target ?? this.create(), end = reader.pos + length;
while (reader.pos < end) {
let [fieldNo, wireType] = reader.tag();
switch (fieldNo) {
case /* bytes report_uuid */ 1:
message.reportUuid = reader.bytes();
break;
case /* bytes cluster_name */ 2:
message.clusterName = reader.bytes();
break;
case /* bytes reporter_hostid */ 3:
message.reporterHostid = reader.bytes();
break;
case /* google.protobuf.Timestamp start_time */ 4:
message.startTime = Timestamp.internalBinaryRead(reader, reader.uint32(), options, message.startTime);
break;
case /* repeated prehog.v1.IdentitySecurityGraphSize graph_size_records */ 5:
message.graphSizeRecords.push(IdentitySecurityGraphSize.internalBinaryRead(reader, reader.uint32(), options));
break;
case /* repeated prehog.v1.IdentitySecurityAuditLogsIngested audit_log_records */ 6:
message.auditLogRecords.push(IdentitySecurityAuditLogsIngested.internalBinaryRead(reader, reader.uint32(), options));
break;
default:
let u = options.readUnknownField;
if (u === "throw")
throw new globalThis.Error(`Unknown field ${fieldNo} (wire type ${wireType}) for ${this.typeName}`);
let d = reader.skip(wireType);
if (u !== false)
(u === true ? UnknownFieldHandler.onRead : u)(this.typeName, message, fieldNo, wireType, d);
}
}
return message;
}
internalBinaryWrite(message: IdentitySecurityReport, writer: IBinaryWriter, options: BinaryWriteOptions): IBinaryWriter {
/* bytes report_uuid = 1; */
if (message.reportUuid.length)
writer.tag(1, WireType.LengthDelimited).bytes(message.reportUuid);
/* bytes cluster_name = 2; */
if (message.clusterName.length)
writer.tag(2, WireType.LengthDelimited).bytes(message.clusterName);
/* bytes reporter_hostid = 3; */
if (message.reporterHostid.length)
writer.tag(3, WireType.LengthDelimited).bytes(message.reporterHostid);
/* google.protobuf.Timestamp start_time = 4; */
if (message.startTime)
Timestamp.internalBinaryWrite(message.startTime, writer.tag(4, WireType.LengthDelimited).fork(), options).join();
/* repeated prehog.v1.IdentitySecurityGraphSize graph_size_records = 5; */
for (let i = 0; i < message.graphSizeRecords.length; i++)
IdentitySecurityGraphSize.internalBinaryWrite(message.graphSizeRecords[i], writer.tag(5, WireType.LengthDelimited).fork(), options).join();
/* repeated prehog.v1.IdentitySecurityAuditLogsIngested audit_log_records = 6; */
for (let i = 0; i < message.auditLogRecords.length; i++)
IdentitySecurityAuditLogsIngested.internalBinaryWrite(message.auditLogRecords[i], writer.tag(6, WireType.LengthDelimited).fork(), options).join();
let u = options.writeUnknownFields;
if (u !== false)
(u == true ? UnknownFieldHandler.onWrite : u)(this.typeName, message, writer);
return writer;
}
}
/**
* @generated MessageType for protobuf message prehog.v1.IdentitySecurityReport
*/
export const IdentitySecurityReport = new IdentitySecurityReport$Type();
// @generated message type with reflection information, may provide speed optimized methods
class IdentitySecurityGraphSize$Type extends MessageType<IdentitySecurityGraphSize> {
constructor() {
super("prehog.v1.IdentitySecurityGraphSize", [
{ no: 1, name: "provider", kind: "scalar", T: 9 /*ScalarType.STRING*/ },
{ no: 2, name: "total_identities", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ },
{ no: 3, name: "total_resources", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ }
]);
}
create(value?: PartialMessage<IdentitySecurityGraphSize>): IdentitySecurityGraphSize {
const message = globalThis.Object.create((this.messagePrototype!));
message.provider = "";
message.totalIdentities = 0n;
message.totalResources = 0n;
if (value !== undefined)
reflectionMergePartial<IdentitySecurityGraphSize>(this, message, value);
return message;
}
internalBinaryRead(reader: IBinaryReader, length: number, options: BinaryReadOptions, target?: IdentitySecurityGraphSize): IdentitySecurityGraphSize {
let message = target ?? this.create(), end = reader.pos + length;
while (reader.pos < end) {
let [fieldNo, wireType] = reader.tag();
switch (fieldNo) {
case /* string provider */ 1:
message.provider = reader.string();
break;
case /* uint64 total_identities */ 2:
message.totalIdentities = reader.uint64().toBigInt();
break;
case /* uint64 total_resources */ 3:
message.totalResources = reader.uint64().toBigInt();
break;
default:
let u = options.readUnknownField;
if (u === "throw")
throw new globalThis.Error(`Unknown field ${fieldNo} (wire type ${wireType}) for ${this.typeName}`);
let d = reader.skip(wireType);
if (u !== false)
(u === true ? UnknownFieldHandler.onRead : u)(this.typeName, message, fieldNo, wireType, d);
}
}
return message;
}
internalBinaryWrite(message: IdentitySecurityGraphSize, writer: IBinaryWriter, options: BinaryWriteOptions): IBinaryWriter {
/* string provider = 1; */
if (message.provider !== "")
writer.tag(1, WireType.LengthDelimited).string(message.provider);
/* uint64 total_identities = 2; */
if (message.totalIdentities !== 0n)
writer.tag(2, WireType.Varint).uint64(message.totalIdentities);
/* uint64 total_resources = 3; */
if (message.totalResources !== 0n)
writer.tag(3, WireType.Varint).uint64(message.totalResources);
let u = options.writeUnknownFields;
if (u !== false)
(u == true ? UnknownFieldHandler.onWrite : u)(this.typeName, message, writer);
return writer;
}
}
/**
* @generated MessageType for protobuf message prehog.v1.IdentitySecurityGraphSize
*/
export const IdentitySecurityGraphSize = new IdentitySecurityGraphSize$Type();
// @generated message type with reflection information, may provide speed optimized methods
class IdentitySecurityAuditLogsIngested$Type extends MessageType<IdentitySecurityAuditLogsIngested> {
constructor() {
super("prehog.v1.IdentitySecurityAuditLogsIngested", [
{ no: 1, name: "provider", kind: "scalar", T: 9 /*ScalarType.STRING*/ },
{ no: 2, name: "logs_ingested", kind: "scalar", T: 4 /*ScalarType.UINT64*/, L: 0 /*LongType.BIGINT*/ }
]);
}
create(value?: PartialMessage<IdentitySecurityAuditLogsIngested>): IdentitySecurityAuditLogsIngested {
const message = globalThis.Object.create((this.messagePrototype!));
message.provider = "";
message.logsIngested = 0n;
if (value !== undefined)
reflectionMergePartial<IdentitySecurityAuditLogsIngested>(this, message, value);
return message;
}
internalBinaryRead(reader: IBinaryReader, length: number, options: BinaryReadOptions, target?: IdentitySecurityAuditLogsIngested): IdentitySecurityAuditLogsIngested {
let message = target ?? this.create(), end = reader.pos + length;
while (reader.pos < end) {
let [fieldNo, wireType] = reader.tag();
switch (fieldNo) {
case /* string provider */ 1:
message.provider = reader.string();
break;
case /* uint64 logs_ingested */ 2:
message.logsIngested = reader.uint64().toBigInt();
break;
default:
let u = options.readUnknownField;
if (u === "throw")
throw new globalThis.Error(`Unknown field ${fieldNo} (wire type ${wireType}) for ${this.typeName}`);
let d = reader.skip(wireType);
if (u !== false)
(u === true ? UnknownFieldHandler.onRead : u)(this.typeName, message, fieldNo, wireType, d);
}
}
return message;
}
internalBinaryWrite(message: IdentitySecurityAuditLogsIngested, writer: IBinaryWriter, options: BinaryWriteOptions): IBinaryWriter {
/* string provider = 1; */
if (message.provider !== "")
writer.tag(1, WireType.LengthDelimited).string(message.provider);
/* uint64 logs_ingested = 2; */
if (message.logsIngested !== 0n)
writer.tag(2, WireType.Varint).uint64(message.logsIngested);
let u = options.writeUnknownFields;
if (u !== false)
(u == true ? UnknownFieldHandler.onWrite : u)(this.typeName, message, writer);
return writer;
}
}
/**
* @generated MessageType for protobuf message prehog.v1.IdentitySecurityAuditLogsIngested
*/
export const IdentitySecurityAuditLogsIngested = new IdentitySecurityAuditLogsIngested$Type();
// @generated message type with reflection information, may provide speed optimized methods
class SubmitUsageReportsRequest$Type extends MessageType<SubmitUsageReportsRequest> {
constructor() {
super("prehog.v1.SubmitUsageReportsRequest", [
{ no: 1, name: "user_activity", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => UserActivityReport },
{ no: 2, name: "resource_presence", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => ResourcePresenceReport },
{ no: 3, name: "bot_instance_activity", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => BotInstanceActivityReport },
{ no: 4, name: "identity_security_summaries_report", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => IdentitySecuritySummariesGeneratedReport }
{ no: 4, name: "identity_security_summaries_report", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => IdentitySecuritySummariesGeneratedReport },
{ no: 5, name: "identity_security_report", kind: "message", repeat: 1 /*RepeatType.PACKED*/, T: () => IdentitySecurityReport }
]);
}
create(value?: PartialMessage<SubmitUsageReportsRequest>): SubmitUsageReportsRequest {
@@ -1632,6 +1960,7 @@ class SubmitUsageReportsRequest$Type extends MessageType<SubmitUsageReportsReque
message.resourcePresence = [];
message.botInstanceActivity = [];
message.identitySecuritySummariesReport = [];
message.identitySecurityReport = [];
if (value !== undefined)
reflectionMergePartial<SubmitUsageReportsRequest>(this, message, value);
return message;
@@ -1653,6 +1982,9 @@ class SubmitUsageReportsRequest$Type extends MessageType<SubmitUsageReportsReque
case /* repeated prehog.v1.IdentitySecuritySummariesGeneratedReport identity_security_summaries_report */ 4:
message.identitySecuritySummariesReport.push(IdentitySecuritySummariesGeneratedReport.internalBinaryRead(reader, reader.uint32(), options));
break;
case /* repeated prehog.v1.IdentitySecurityReport identity_security_report */ 5:
message.identitySecurityReport.push(IdentitySecurityReport.internalBinaryRead(reader, reader.uint32(), options));
break;
default:
let u = options.readUnknownField;
if (u === "throw")
@@ -1677,6 +2009,9 @@ class SubmitUsageReportsRequest$Type extends MessageType<SubmitUsageReportsReque
/* repeated prehog.v1.IdentitySecuritySummariesGeneratedReport identity_security_summaries_report = 4; */
for (let i = 0; i < message.identitySecuritySummariesReport.length; i++)
IdentitySecuritySummariesGeneratedReport.internalBinaryWrite(message.identitySecuritySummariesReport[i], writer.tag(4, WireType.LengthDelimited).fork(), options).join();
/* repeated prehog.v1.IdentitySecurityReport identity_security_report = 5; */
for (let i = 0; i < message.identitySecurityReport.length; i++)
IdentitySecurityReport.internalBinaryWrite(message.identitySecurityReport[i], writer.tag(5, WireType.LengthDelimited).fork(), options).join();
let u = options.writeUnknownFields;
if (u !== false)
(u == true ? UnknownFieldHandler.onWrite : u)(this.typeName, message, writer);
@@ -22,6 +22,8 @@ import (
"context"
"encoding/binary"
"log/slog"
"maps"
"slices"
"sync"
"time"
@@ -42,7 +44,8 @@ const (
userActivityReportGranularity = 15 * time.Minute
resourceReportGranularity = time.Hour
botInstanceActivityReportGranularity = 15 * time.Minute
identitySecurityReportGranularity = 15 * time.Minute
sessionSummaryReportGranularity = 15 * time.Minute
identitySecurityReportGranularity = 24 * time.Hour
rollbackGrace = time.Minute
reportTTL = 60 * 24 * time.Hour
@@ -125,8 +128,8 @@ func NewReporter(ctx context.Context, cfg ReporterConfig) (*Reporter, error) {
return r, nil
}
// identitySecuritySummariesGeneratedKey uniquely identifies a session summary by session type and resource name
type identitySecuritySummariesGeneratedKey struct {
// sessionSummariesGeneratedKey uniquely identifies a session summary by session type and resource name
type sessionSummariesGeneratedKey struct {
sessionType string
resourceName string
}
@@ -198,8 +201,11 @@ func (r *Reporter) AnonymizeAndSubmit(events ...usagereporter.Anonymizable) {
*usagereporter.AccessRequestReviewEvent,
*usagereporter.AccessListReviewCreateEvent,
*usagereporter.AccessListGrantsToUserEvent,
*usagereporter.TagExecuteQueryEvent,
*usagereporter.SessionSummaryCreateEvent,
*usagereporter.SessionSummaryAccessEvent:
*usagereporter.SessionSummaryAccessEvent,
*usagereporter.IdentitySecurityGraphSizeEvent,
*usagereporter.IdentitySecurityAuditLogsIngestedEvent:
filtered = append(filtered, event)
}
}
@@ -324,13 +330,13 @@ func (r *Reporter) run(ctx context.Context) {
// sessionSummariesGenerated tracks AI-generated session summaries with token usage
// map[key]*SessionSummariesGeneratedRecord
sessionSummariesGenerated := make(map[identitySecuritySummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord)
identitySecurityStartTime := r.clock.Now().UTC().Truncate(userActivityReportGranularity)
identitySecurityWindowStart := identitySecurityStartTime.Add(-rollbackGrace)
identitySecurityWindowEnd := identitySecurityStartTime.Add(userActivityReportGranularity)
sessionSummariesGenerated := make(map[sessionSummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord)
sessionSummariesStartTime := r.clock.Now().UTC().Truncate(userActivityReportGranularity)
sessionSummariesWindowStart := sessionSummariesStartTime.Add(-rollbackGrace)
sessionSummariesWindowEnd := sessionSummariesStartTime.Add(userActivityReportGranularity)
incrementSessionSummariesGenerated := func(sessionType string, resourceName string, inputTokens uint64, outputTokens uint64) {
key := identitySecuritySummariesGeneratedKey{
key := sessionSummariesGeneratedKey{
sessionType: sessionType,
resourceName: resourceName,
}
@@ -347,6 +353,31 @@ func (r *Reporter) run(ctx context.Context) {
record.SummariesGenerated++
}
// identitySecurityGraphSize tracks the size of access graphs by provider.
identitySecurityGraphSize := make(map[string]*prehogv1.IdentitySecurityGraphSize)
// identitySecurityAuditLogsIngested tracks the count of audit logs ingested by provider.
identitySecurityAuditLogsIngested := make(map[string]*prehogv1.IdentitySecurityAuditLogsIngested)
identitySecurityStartTime := r.clock.Now().UTC().Truncate(identitySecurityReportGranularity)
identitySecurityWindowStart := identitySecurityStartTime.Add(-rollbackGrace)
identitySecurityWindowEnd := identitySecurityStartTime.Add(identitySecurityReportGranularity)
recordIdentitySecurityGraphSize := func(provider string, identities, resources uint64) {
// Store the latest received graph size for the provider, overwriting any previous ones.
identitySecurityGraphSize[provider] = &prehogv1.IdentitySecurityGraphSize{
Provider: provider,
TotalIdentities: identities,
TotalResources: resources,
}
}
incrementIdentitySecurityAuditLogsIngested := func(provider string, count uint64) {
record := identitySecurityAuditLogsIngested[provider]
if record == nil {
record = &prehogv1.IdentitySecurityAuditLogsIngested{Provider: provider}
identitySecurityAuditLogsIngested[provider] = record
}
record.LogsIngested += count
}
botInstanceActivityStartTime := r.clock.Now().UTC().Truncate(botInstanceActivityReportGranularity)
botInstanceActivityWindowStart := botInstanceActivityStartTime.Add(-rollbackGrace)
botInstanceActivityWindowEnd := botInstanceActivityStartTime.Add(botInstanceActivityReportGranularity)
@@ -449,23 +480,44 @@ Ingest:
resourcePresences = make(map[prehogv1.ResourceKind]map[string]struct{}, len(resourcePresences))
}
if now := r.clock.Now().UTC(); now.Before(identitySecurityWindowStart) || !now.Before(identitySecurityWindowEnd) {
if now := r.clock.Now().UTC(); now.Before(sessionSummariesWindowStart) || !now.Before(sessionSummariesWindowEnd) {
if len(sessionSummariesGenerated) > 0 {
wg.Add(1)
go func(
ctx context.Context,
startTime time.Time,
summaries map[identitySecuritySummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord,
summaries map[sessionSummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord,
) {
defer wg.Done()
r.persistIdentitySecuritySummariesGenerated(ctx, startTime, summaries)
}(ctx, identitySecurityStartTime, sessionSummariesGenerated)
}(ctx, sessionSummariesStartTime, sessionSummariesGenerated)
}
sessionSummariesStartTime = now.Truncate(sessionSummaryReportGranularity)
sessionSummariesWindowStart = sessionSummariesStartTime.Add(-rollbackGrace)
sessionSummariesWindowEnd = sessionSummariesStartTime.Add(sessionSummaryReportGranularity)
sessionSummariesGenerated = make(map[sessionSummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord, len(sessionSummariesGenerated))
}
if now := r.clock.Now().UTC(); now.Before(identitySecurityWindowStart) || !now.Before(identitySecurityWindowEnd) {
if len(identitySecurityGraphSize) > 0 || len(identitySecurityAuditLogsIngested) > 0 {
wg.Add(1)
go func(
ctx context.Context,
startTime time.Time,
graphSizes map[string]*prehogv1.IdentitySecurityGraphSize,
logsIngested map[string]*prehogv1.IdentitySecurityAuditLogsIngested,
) {
defer wg.Done()
r.persistIdentitySecurity(ctx, startTime, graphSizes, logsIngested)
}(ctx, identitySecurityStartTime, identitySecurityGraphSize, identitySecurityAuditLogsIngested)
}
identitySecurityStartTime = now.Truncate(identitySecurityReportGranularity)
identitySecurityWindowStart = identitySecurityStartTime.Add(-rollbackGrace)
identitySecurityWindowEnd = identitySecurityStartTime.Add(identitySecurityReportGranularity)
sessionSummariesGenerated = make(map[identitySecuritySummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord, len(sessionSummariesGenerated))
identitySecurityGraphSize = make(map[string]*prehogv1.IdentitySecurityGraphSize, len(identitySecurityGraphSize))
identitySecurityAuditLogsIngested = make(map[string]*prehogv1.IdentitySecurityAuditLogsIngested, len(identitySecurityAuditLogsIngested))
}
switch te := ae.(type) {
@@ -542,6 +594,12 @@ Ingest:
if te.Success {
incrementSessionSummariesGenerated(te.SessionType, te.ResourceName, te.TotalInputTokens, te.TotalOutputTokens)
}
case *usagereporter.TagExecuteQueryEvent:
userRecord(te.UserName, prehogv1alpha.UserKind_USER_KIND_HUMAN).AccessGraphQueries++
case *usagereporter.IdentitySecurityGraphSizeEvent:
recordIdentitySecurityGraphSize(te.Provider, te.TotalIdentities, te.TotalResources)
case *usagereporter.IdentitySecurityAuditLogsIngestedEvent:
incrementIdentitySecurityAuditLogsIngested(te.Provider, te.LogsIngested)
}
if ae != nil && r.ingested != nil {
@@ -562,7 +620,11 @@ Ingest:
}
if len(sessionSummariesGenerated) > 0 {
r.persistIdentitySecuritySummariesGenerated(ctx, identitySecurityStartTime, sessionSummariesGenerated)
r.persistIdentitySecuritySummariesGenerated(ctx, sessionSummariesStartTime, sessionSummariesGenerated)
}
if len(identitySecurityGraphSize) > 0 || len(identitySecurityAuditLogsIngested) > 0 {
r.persistIdentitySecurity(ctx, identitySecurityStartTime, identitySecurityGraphSize, identitySecurityAuditLogsIngested)
}
wg.Wait()
@@ -716,7 +778,7 @@ func (r *Reporter) persistResourcePresence(ctx context.Context, startTime time.T
func (r *Reporter) persistIdentitySecuritySummariesGenerated(
ctx context.Context,
startTime time.Time,
sessionSummariesGenerated map[identitySecuritySummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord,
sessionSummariesGenerated map[sessionSummariesGeneratedKey]*prehogv1.SessionSummariesGeneratedRecord,
) {
records := make([]*prehogv1.SessionSummariesGeneratedRecord, 0, len(sessionSummariesGenerated))
for _, record := range sessionSummariesGenerated {
@@ -756,3 +818,43 @@ func (r *Reporter) persistIdentitySecuritySummariesGenerated(
)
}
}
func (r *Reporter) persistIdentitySecurity(
ctx context.Context,
startTime time.Time,
graphSizes map[string]*prehogv1.IdentitySecurityGraphSize,
logsIngested map[string]*prehogv1.IdentitySecurityAuditLogsIngested,
) {
graphSizeRecords := slices.Collect(maps.Values(graphSizes))
logsIngestedRecords := slices.Collect(maps.Values(logsIngested))
anonymizedClusterName := r.anonymizer.AnonymizeNonEmpty(r.clusterName)
anonymizedHostID := r.anonymizer.AnonymizeNonEmpty(r.hostID)
reports, err := prepareIdentitySecurityReports(anonymizedClusterName, anonymizedHostID, startTime, graphSizeRecords, logsIngestedRecords)
if err != nil {
r.logger.ErrorContext(ctx, "Failed to prepare identity security report, dropping data.",
"start_time", startTime,
"lost_records", len(graphSizeRecords)+len(logsIngestedRecords),
"error", err,
)
return
}
for _, report := range reports {
if err := r.svc.upsertIdentitySecurityReport(ctx, report, reportTTL); err != nil {
r.logger.ErrorContext(ctx, "Failed to persist identity security report, dropping data.",
"start_time", startTime,
"lost_records", len(report.GraphSizeRecords)+len(report.AuditLogRecords),
"error", err,
)
continue
}
reportUUID, _ := uuid.FromBytes(report.ReportUuid)
r.logger.DebugContext(ctx, "Persisted identity security report.",
"report_uuid", reportUUID,
"start_time", startTime,
"records", len(report.GraphSizeRecords)+len(report.AuditLogRecords),
)
}
}
@@ -471,7 +471,7 @@ func TestReporterSessionSummariesAccessed(t *testing.T) {
recvIngested()
// Wait for the aggregation window to complete
time.Sleep(identitySecurityReportGranularity)
time.Sleep(sessionSummaryReportGranularity)
synctest.Wait()
require.Equal(t, types.OpPut, recvBackendEvent().Type)
@@ -629,3 +629,115 @@ func TestReporterIdentitySecuritySummariesGenerated(t *testing.T) {
}
})
}
func TestReporterIdentitySecurity(t *testing.T) {
synctest.Test(t, func(t *testing.T) {
ctx := t.Context()
clk := clockwork.NewRealClock()
bk, err := memory.New(memory.Config{
Clock: clk,
})
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, bk.Close()) })
// Set up a watcher to not have to poll the backend for newly added items
w, err := bk.NewWatcher(ctx, backend.Watch{})
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, w.Close()) })
recvBackendEvent := func() backend.Event {
return <-w.Events()
}
require.Equal(t, types.OpInit, recvBackendEvent().Type)
clusterName, err := services.NewClusterNameWithRandomID(types.ClusterNameSpecV2{
ClusterName: "clustername",
})
require.NoError(t, err)
anonymizer, err := utils.NewHMACAnonymizer(utils.AnonymizationKeyString("0123456789abcdef"))
require.NoError(t, err)
r, err := NewReporter(ctx, ReporterConfig{
Backend: bk,
Clock: clk,
ClusterName: clusterName,
HostID: "host-id",
Anonymizer: anonymizer,
})
require.NoError(t, err)
svc := reportService{bk}
r.ingested = make(chan usagereporter.Anonymizable, 10)
recvIngested := func() {
<-r.ingested
}
r.AnonymizeAndSubmit(&usagereporter.IdentitySecurityGraphSizeEvent{
Provider: "teleport",
TotalIdentities: 1,
TotalResources: 2,
})
recvIngested()
// Later size should overwrite earlier size for the same provider.
r.AnonymizeAndSubmit(&usagereporter.IdentitySecurityGraphSizeEvent{
Provider: "teleport",
TotalIdentities: 3,
TotalResources: 4,
})
recvIngested()
r.AnonymizeAndSubmit(&usagereporter.IdentitySecurityAuditLogsIngestedEvent{
Provider: "teleport",
LogsIngested: 5,
})
recvIngested()
// Audit log counts are accumulated.
r.AnonymizeAndSubmit(&usagereporter.IdentitySecurityAuditLogsIngestedEvent{
Provider: "teleport",
LogsIngested: 7,
})
recvIngested()
time.Sleep(identitySecurityReportGranularity)
for putCount := 0; putCount < 2; {
if recvBackendEvent().Type == types.OpPut {
putCount++
}
}
reports, err := svc.listIdentitySecurityReports(ctx, 10)
require.NoError(t, err)
require.Len(t, reports, 2)
expectedCluster := anonymizer.AnonymizeNonEmpty(clusterName.GetClusterName())
expectedHostID := anonymizer.AnonymizeNonEmpty("host-id")
var graphReport *prehogv1.IdentitySecurityReport
var auditReport *prehogv1.IdentitySecurityReport
for _, report := range reports {
require.Equal(t, expectedCluster, report.ClusterName)
require.Equal(t, expectedHostID, report.ReporterHostid)
if len(report.GraphSizeRecords) > 0 {
graphReport = report
}
if len(report.AuditLogRecords) > 0 {
auditReport = report
}
}
require.NotNil(t, graphReport)
require.Len(t, graphReport.GraphSizeRecords, 1)
require.Equal(t, "teleport", graphReport.GraphSizeRecords[0].Provider)
require.Equal(t, uint64(3), graphReport.GraphSizeRecords[0].TotalIdentities)
require.Equal(t, uint64(4), graphReport.GraphSizeRecords[0].TotalResources)
require.NotNil(t, auditReport)
require.Len(t, auditReport.AuditLogRecords, 1)
require.Equal(t, "teleport", auditReport.AuditLogRecords[0].Provider)
require.Equal(t, uint64(12), auditReport.AuditLogRecords[0].LogsIngested)
})
}
@@ -42,6 +42,7 @@ const (
botInstanceActivityReportsPrefix = "botInstanceActivityReports"
userActivityReportsPrefix = "userActivityReports"
identitySecuritySummariesGeneratedReportsPrefix = "identitySecuritySummariesGeneratedReports"
identitySecurityReportsPrefix = "identitySecurityReports"
// usageReportingLock is a lock that should be held when submitting usage
// reports to the upstream service. Whilst the underlying key refers
// specifically to "userActivityReports", this is inaccurate.
@@ -241,6 +242,62 @@ func prepareIdentitySecuritySummariesGeneratedReports(
return reports, nil
}
// identitySecurityReportKey returns the backend key for an identity security report with
// a given UUID and start time, such that reports with an earlier start time
// will appear earlier in lexicographic ordering.
func identitySecurityReportKey(reportUUID uuid.UUID, startTime time.Time) backend.Key {
return backend.NewKey(identitySecurityReportsPrefix, startTime.Format(time.RFC3339), reportUUID.String())
}
func prepareIdentitySecurityReports(
clusterName, reporterHostID []byte, startTime time.Time,
graphSizeRecords []*prehogv1.IdentitySecurityGraphSize,
logsIngestedRecords []*prehogv1.IdentitySecurityAuditLogsIngested,
) ([]*prehogv1.IdentitySecurityReport, error) {
var reports []*prehogv1.IdentitySecurityReport
newReport := func() *prehogv1.IdentitySecurityReport {
reportUUID := uuid.New()
return &prehogv1.IdentitySecurityReport{
ReportUuid: reportUUID[:],
ClusterName: clusterName,
ReporterHostid: reporterHostID,
StartTime: timestamppb.New(startTime),
}
}
for len(graphSizeRecords) > 0 {
report := newReport()
report.GraphSizeRecords = graphSizeRecords
for proto.Size(report) > maxItemSize {
if len(report.GraphSizeRecords) <= 1 {
return nil, trace.LimitExceeded("failed to marshal identity security report within size limit (this is a bug)")
}
report.GraphSizeRecords = report.GraphSizeRecords[:len(report.GraphSizeRecords)/2]
}
graphSizeRecords = graphSizeRecords[len(report.GraphSizeRecords):]
reports = append(reports, report)
}
for len(logsIngestedRecords) > 0 {
report := newReport()
report.AuditLogRecords = logsIngestedRecords
for proto.Size(report) > maxItemSize {
if len(report.AuditLogRecords) <= 1 {
return nil, trace.LimitExceeded("failed to marshal identity security report within size limit (this is a bug)")
}
report.AuditLogRecords = report.AuditLogRecords[:len(report.AuditLogRecords)/2]
}
logsIngestedRecords = logsIngestedRecords[len(report.AuditLogRecords):]
reports = append(reports, report)
}
return reports, nil
}
// reportService is a [backend.Backend] wrapper that handles usage reports.
type reportService struct {
b backend.Backend
@@ -551,3 +608,73 @@ func (r reportService) listIdentitySecuritySummariesGeneratedReports(
return reports, nil
}
func (r reportService) upsertIdentitySecurityReport(
ctx context.Context, report *prehogv1.IdentitySecurityReport, ttl time.Duration,
) error {
marshaledReport, err := proto.Marshal(report)
if err != nil {
return trace.Wrap(err)
}
reportUUID, err := uuid.FromBytes(report.GetReportUuid())
if err != nil {
return trace.Wrap(err)
}
startTime := report.GetStartTime().AsTime()
if startTime.IsZero() {
return trace.BadParameter("missing start_time")
}
if _, err := r.b.Put(ctx, backend.Item{
Key: identitySecurityReportKey(reportUUID, startTime),
Value: marshaledReport,
Expires: startTime.Add(ttl),
}); err != nil {
return trace.Wrap(err)
}
return nil
}
func (r reportService) deleteIdentitySecurityReport(
ctx context.Context, report *prehogv1.IdentitySecurityReport,
) error {
reportUUID, err := uuid.FromBytes(report.GetReportUuid())
if err != nil {
return trace.Wrap(err)
}
startTime := report.GetStartTime().AsTime()
if startTime.IsZero() {
return trace.BadParameter("missing start_time")
}
if err := r.b.Delete(ctx, identitySecurityReportKey(reportUUID, startTime)); err != nil {
return trace.Wrap(err)
}
return nil
}
func (r reportService) listIdentitySecurityReports(
ctx context.Context, count int,
) ([]*prehogv1.IdentitySecurityReport, error) {
rangeStart := backend.ExactKey(identitySecurityReportsPrefix)
result, err := r.b.GetRange(ctx, rangeStart, backend.RangeEnd(rangeStart), count)
if err != nil {
return nil, trace.Wrap(err)
}
reports := make([]*prehogv1.IdentitySecurityReport, 0, len(result.Items))
for _, item := range result.Items {
report := &prehogv1.IdentitySecurityReport{}
if err := proto.Unmarshal(item.Value, report); err != nil {
return nil, trace.Wrap(err)
}
reports = append(reports, report)
}
return reports, nil
}
@@ -150,6 +150,14 @@ func newIdentitySecuritySummariesGeneratedReport(startTime time.Time) *prehogv1.
}
}
func newIdentitySecurityReport(startTime time.Time) *prehogv1.IdentitySecurityReport {
u := uuid.New()
return &prehogv1.IdentitySecurityReport{
ReportUuid: u[:],
StartTime: timestamppb.New(startTime),
}
}
func TestResourcePresenceReporting(t *testing.T) {
ctx := context.Background()
clk := clockwork.NewFakeClock()
@@ -124,13 +124,19 @@ func RunSubmitter(ctx context.Context, cfg SubmitterConfig) {
func submitOnce(ctx context.Context, c SubmitterConfig) {
svc := reportService{c.Backend}
userActivityReports, err := svc.listUserActivityReports(ctx, submitBatchSize)
if err != nil {
c.Logger.ErrorContext(ctx, "Failed to load usage reports for submission.", "error", err)
return
freeBatchSize := submitBatchSize
var err error
var userActivityReports []*prehogv1.UserActivityReport
if freeBatchSize > 0 {
userActivityReports, err = svc.listUserActivityReports(ctx, freeBatchSize)
if err != nil {
c.Logger.ErrorContext(ctx, "Failed to load usage reports for submission.", "error", err)
return
}
freeBatchSize -= len(userActivityReports)
}
freeBatchSize := submitBatchSize - len(userActivityReports)
var resourcePresenceReports []*prehogv1.ResourcePresenceReport
if freeBatchSize > 0 {
resourcePresenceReports, err = svc.listResourcePresenceReports(ctx, freeBatchSize)
@@ -138,9 +144,9 @@ func submitOnce(ctx context.Context, c SubmitterConfig) {
c.Logger.ErrorContext(ctx, "Failed to load resource counts reports for submission.", "error", err)
return
}
freeBatchSize -= len(resourcePresenceReports)
}
freeBatchSize = submitBatchSize - len(userActivityReports) - len(resourcePresenceReports)
var botInstanceActivityReports []*prehogv1.BotInstanceActivityReport
if freeBatchSize > 0 {
botInstanceActivityReports, err = svc.listBotInstanceActivityReports(ctx, freeBatchSize)
@@ -148,9 +154,9 @@ func submitOnce(ctx context.Context, c SubmitterConfig) {
c.Logger.ErrorContext(ctx, "Failed to load bot instance activity reports for submission.", "error", err)
return
}
freeBatchSize -= len(botInstanceActivityReports)
}
freeBatchSize = submitBatchSize - len(userActivityReports) - len(resourcePresenceReports) - len(botInstanceActivityReports)
var identitySecuritySummariesReports []*prehogv1.IdentitySecuritySummariesGeneratedReport
if freeBatchSize > 0 {
identitySecuritySummariesReports, err = svc.listIdentitySecuritySummariesGeneratedReports(ctx, freeBatchSize)
@@ -158,9 +164,20 @@ func submitOnce(ctx context.Context, c SubmitterConfig) {
c.Logger.ErrorContext(ctx, "Failed to load identity security summaries reports for submission.", "error", err)
return
}
freeBatchSize -= len(identitySecuritySummariesReports)
}
totalReportCount := len(userActivityReports) + len(resourcePresenceReports) + len(botInstanceActivityReports) + len(identitySecuritySummariesReports)
var identitySecurityReports []*prehogv1.IdentitySecurityReport
if freeBatchSize > 0 {
identitySecurityReports, err = svc.listIdentitySecurityReports(ctx, freeBatchSize)
if err != nil {
c.Logger.ErrorContext(ctx, "Failed to load identity security reports for submission.", "error", err)
return
}
freeBatchSize -= len(identitySecurityReports)
}
totalReportCount := submitBatchSize - freeBatchSize
if totalReportCount < 1 {
err := ClearAlert(ctx, c.Status)
@@ -207,6 +224,15 @@ func submitOnce(ctx context.Context, c SubmitterConfig) {
}
}
if len(identitySecurityReports) > 0 {
if t := identitySecurityReports[0].GetStartTime().AsTime(); t.Before(oldest) {
oldest = t
}
if t := identitySecurityReports[len(identitySecurityReports)-1].GetStartTime().AsTime(); t.After(newest) {
newest = t
}
}
debugPayload := fmt.Sprintf("%v %q", time.Now().Round(0), c.HostID)
if err := svc.createUsageReportingLock(ctx, submitLockDuration, []byte(debugPayload)); err != nil {
if trace.IsAlreadyExists(err) {
@@ -225,6 +251,7 @@ func submitOnce(ctx context.Context, c SubmitterConfig) {
ResourcePresence: resourcePresenceReports,
BotInstanceActivity: botInstanceActivityReports,
IdentitySecuritySummariesReport: identitySecuritySummariesReports,
IdentitySecurityReport: identitySecurityReports,
})
if err != nil {
c.Logger.ErrorContext(ctx, "Failed to send usage reports.",
@@ -284,6 +311,11 @@ func submitOnce(ctx context.Context, c SubmitterConfig) {
lastErr = err
}
}
for _, report := range identitySecurityReports {
if err := svc.deleteIdentitySecurityReport(ctx, report); err != nil {
lastErr = err
}
}
if lastErr != nil {
c.Logger.WarnContext(ctx, "Failed to delete some usage reports after successful send.", "last_error", lastErr)
@@ -56,14 +56,16 @@ func TestSubmitOnce(t *testing.T) {
var submittedPresence []*prehogv1.ResourcePresenceReport
var submittedBotInstanceActivity []*prehogv1.BotInstanceActivityReport
var submittedIdentitySecuritySummaries []*prehogv1.IdentitySecuritySummariesGeneratedReport
var submittedIdentitySecurityReports []*prehogv1.IdentitySecurityReport
submitOk := func(ctx context.Context, req *prehogv1.SubmitUsageReportsRequest) (uuid.UUID, error) {
if l := len(req.UserActivity) + len(req.ResourcePresence) + len(req.BotInstanceActivity) + len(req.IdentitySecuritySummariesReport); l > submitBatchSize {
if l := len(req.UserActivity) + len(req.ResourcePresence) + len(req.BotInstanceActivity) + len(req.IdentitySecuritySummariesReport) + len(req.IdentitySecurityReport); l > submitBatchSize {
return uuid.Nil, trace.LimitExceeded("got %v reports, expected at most %v", l, submitBatchSize)
}
submitted = append(submitted, req.UserActivity...)
submittedPresence = append(submittedPresence, req.ResourcePresence...)
submittedBotInstanceActivity = append(submittedBotInstanceActivity, req.BotInstanceActivity...)
submittedIdentitySecuritySummaries = append(submittedIdentitySecuritySummaries, req.IdentitySecuritySummariesReport...)
submittedIdentitySecurityReports = append(submittedIdentitySecurityReports, req.IdentitySecurityReport...)
return uuid.New(), nil
}
submitErr := func(ctx context.Context, req *prehogv1.SubmitUsageReportsRequest) (uuid.UUID, error) {
@@ -84,16 +86,24 @@ func TestSubmitOnce(t *testing.T) {
resCountReport := newResourcePresenceReport(time.Now().UTC())
require.NoError(t, svc.upsertResourcePresenceReport(ctx, resCountReport, reportTTL))
identityReport := newIdentitySecurityReport(time.Now().UTC())
require.NoError(t, svc.upsertIdentitySecurityReport(ctx, identityReport, reportTTL))
// successful submit, no alerts, no leftover reports
submitOnce(ctx, scfg)
require.Len(t, submitted, 1)
require.True(t, proto.Equal(reportFresh, submitted[0]))
require.Len(t, submittedIdentitySecurityReports, 1)
require.True(t, proto.Equal(identityReport, submittedIdentitySecurityReports[0]))
reports, err := svc.listUserActivityReports(ctx, 10)
require.NoError(t, err)
require.Empty(t, reports)
rReports, err := svc.listResourcePresenceReports(ctx, 10)
require.NoError(t, err)
require.Empty(t, rReports)
identityReports, err := svc.listIdentitySecurityReports(ctx, 10)
require.NoError(t, err)
require.Empty(t, identityReports)
submitted = nil
@@ -215,9 +225,10 @@ func testSubmitOnceIdentitySecuritySummaries(t *testing.T) {
var submittedResourcePresence []*prehogv1.ResourcePresenceReport
var submittedBotInstanceActivity []*prehogv1.BotInstanceActivityReport
var submittedIdentitySecuritySummaries []*prehogv1.IdentitySecuritySummariesGeneratedReport
var submittedIdentitySecurityReports []*prehogv1.IdentitySecurityReport
submitOk := func(ctx context.Context, req *prehogv1.SubmitUsageReportsRequest) (uuid.UUID, error) {
totalReports := len(req.UserActivity) + len(req.ResourcePresence) + len(req.BotInstanceActivity) + len(req.IdentitySecuritySummariesReport)
totalReports := len(req.UserActivity) + len(req.ResourcePresence) + len(req.BotInstanceActivity) + len(req.IdentitySecuritySummariesReport) + len(req.IdentitySecurityReport)
if totalReports > submitBatchSize {
return uuid.Nil, trace.LimitExceeded("got %v reports, expected at most %v", totalReports, submitBatchSize)
}
@@ -225,6 +236,7 @@ func testSubmitOnceIdentitySecuritySummaries(t *testing.T) {
submittedResourcePresence = append(submittedResourcePresence, req.ResourcePresence...)
submittedBotInstanceActivity = append(submittedBotInstanceActivity, req.BotInstanceActivity...)
submittedIdentitySecuritySummaries = append(submittedIdentitySecuritySummaries, req.IdentitySecuritySummariesReport...)
submittedIdentitySecurityReports = append(submittedIdentitySecurityReports, req.IdentitySecurityReport...)
return uuid.New(), nil
}
@@ -242,6 +254,7 @@ func testSubmitOnceIdentitySecuritySummaries(t *testing.T) {
submitOnce(ctx, scfg)
require.Len(t, submittedIdentitySecuritySummaries, 1)
require.True(t, proto.Equal(identityReport1, submittedIdentitySecuritySummaries[0]))
require.Empty(t, submittedIdentitySecurityReports)
// Verify report was deleted after submission
reports, err := svc.listIdentitySecuritySummariesGeneratedReports(ctx, 10)
@@ -253,6 +266,7 @@ func testSubmitOnceIdentitySecuritySummaries(t *testing.T) {
submittedResourcePresence = nil
submittedBotInstanceActivity = nil
submittedIdentitySecuritySummaries = nil
submittedIdentitySecurityReports = nil
// Test 2: Submit mixed report types respecting batch size
// Add 5 user activity reports
@@ -299,6 +313,7 @@ func testSubmitOnceIdentitySecuritySummaries(t *testing.T) {
submittedResourcePresence = nil
submittedBotInstanceActivity = nil
submittedIdentitySecuritySummaries = nil
submittedIdentitySecurityReports = nil
// Test 3: Priority ordering - user activity fills batch first, then resource, then bot, then identity
// Add enough reports to test multiple batches
+31
View File
@@ -2219,3 +2219,34 @@ func (e *DiscoveryConfigEvent) Anonymize(a utils.Anonymizer) prehogv1a.SubmitEve
},
}
}
// IdentitySecurityGraphSizeEvent is emitted when the size of a providers access graph is updated.
type IdentitySecurityGraphSizeEvent prehogv1a.IdentitySecurityGraphSizeEvent
// Anonymize anonymizes the event.
func (e *IdentitySecurityGraphSizeEvent) Anonymize(a utils.Anonymizer) prehogv1a.SubmitEventRequest {
return prehogv1a.SubmitEventRequest{
Event: &prehogv1a.SubmitEventRequest_IdentitySecurityGraphSizeEvent{
IdentitySecurityGraphSizeEvent: &prehogv1a.IdentitySecurityGraphSizeEvent{
Provider: e.Provider,
TotalIdentities: e.TotalIdentities,
TotalResources: e.TotalResources,
},
},
}
}
// IdentitySecurityAuditLogsIngestedEvent is emitted when logs are ingested into indentity activity center
type IdentitySecurityAuditLogsIngestedEvent prehogv1a.IdentitySecurityAuditLogsIngestedEvent
// Anonymize anonymizes the event.
func (e *IdentitySecurityAuditLogsIngestedEvent) Anonymize(a utils.Anonymizer) prehogv1a.SubmitEventRequest {
return prehogv1a.SubmitEventRequest{
Event: &prehogv1a.SubmitEventRequest_IdentitySecurityAuditLogsIngestedEvent{
IdentitySecurityAuditLogsIngestedEvent: &prehogv1a.IdentitySecurityAuditLogsIngestedEvent{
Provider: e.Provider,
LogsIngested: e.LogsIngested,
},
},
}
}
@@ -33,6 +33,7 @@ import "accessgraph/v1alpha/resources.proto";
import "google/protobuf/empty.proto";
import "google/protobuf/struct.proto";
import "google/protobuf/timestamp.proto";
import "prehog/v1alpha/teleport.proto";
import "teleport/auditlog/v1/auditlog.proto";
option go_package = "github.com/gravitational/teleport/gen/proto/go/accessgraph/v1alpha;accessgraphv1alpha";
@@ -309,6 +310,9 @@ message EventsStreamV2Response {
oneof action {
// event is a audit event that should be logged by Teleport.
AuditEvent event = 1;
// usage_event is a usage event to be submitted to the auth server, as an analog of
// AuthService.SubmitUsageEvent.
UsageEvent usage_event = 2;
}
}
@@ -320,6 +324,16 @@ message AuditEvent {
}
}
// UsageEvent is a usage event that access graph can submit to teleport
message UsageEvent {
oneof event {
// graph_size provides the size of a provider's graph
prehog.v1alpha.IdentitySecurityGraphSizeEvent graph_size = 1;
// audit_logs_ingested provides a count of new audit logs ingested by a provider.
prehog.v1alpha.IdentitySecurityAuditLogsIngestedEvent audit_logs_ingested = 2;
}
}
// AuditLogStreamRequest is sent from the client to the server over the
// bi-directional AuditLogStream. It encapsulates distinct client actions for
// configuring the export stream, sending batches of audit log events, and
+62
View File
@@ -146,6 +146,8 @@ message UserActivityRecord {
uint64 saml_idp_sessions = 24;
// counter of session summaries accessed by this user per session type and resource name.
repeated SessionSummariesAccessedRecord session_summaries_accessed = 25;
// counter of access graph queries by this user.
uint64 access_graph_queries = 26;
}
// UserOrigin is the origin of a user account.
@@ -349,6 +351,64 @@ message SessionSummariesGeneratedRecord {
uint64 summaries_generated = 6;
}
// IdentitySecurityReport is a report of Identity Security non-user usage. It currently
// includes the size of the access graph by provider and count of audit logs ingested
// by provider.
message IdentitySecurityReport {
// randomly generated UUID for this specific report, 16 bytes (in string order)
//
// PostHog property: tp.report_uuid (in 8-4-4-4-12 string form)
bytes report_uuid = 1;
// cluster name, anonymized, 32 bytes (HMAC-SHA-256)
//
// PostHog property: tp.cluster_name (in base64)
bytes cluster_name = 2;
// hostid of the auth that collected this report, anonymized, 32 bytes (HMAC-SHA-256)
//
// PostHog property: tp.reporter_hostid (in base64)
bytes reporter_hostid = 3;
// beginning of the time window for this data; ending is not specified but is
// intended to be at most 60 minutes
//
// PostHog timestamp (not a property, the ingest time is tp.report_time instead)
google.protobuf.Timestamp start_time = 4;
// graph_size contains the size of each provider's graph: count of identities and
// count of resources.
repeated IdentitySecurityGraphSize graph_size_records = 5;
// audit_logs contains the counts of audit logs ingested from each provider.
repeated IdentitySecurityAuditLogsIngested audit_log_records = 6;
}
// IdentitySecurityGraphSize is submitted by Access Graph for each provider
// for which it has a graph.
message IdentitySecurityGraphSize {
// provider is the system containing the identities and resources being counted.
// It is one of teleport, aws, azure, entra, gitlab, github, netiq or okta.
string provider = 1;
// total_identities is the number of identities in the graph.
uint64 total_identities = 2;
// total_resources is the number of resources in the graph.
uint64 total_resources = 3;
}
// IdentitySecurityAuditLogsIngested tracks the count of log entries ingested by
// identity activity center.
message IdentitySecurityAuditLogsIngested {
// provider is the system emitting audit logs. It is one of
// teleport, cloudtrail, kubernetes, github or okta.
string provider = 1;
// logs_ingested is a count of log entries ingested into Identity Security.
uint64 logs_ingested = 2;
}
message SubmitUsageReportsRequest {
// at most 10 reports of all kinds in a single RPC, each shouldn't exceed 128KiB or so
//
@@ -360,6 +420,8 @@ message SubmitUsageReportsRequest {
repeated BotInstanceActivityReport bot_instance_activity = 3;
// encoded as a separate tp.identity_security.summaries_generated PostHog event
repeated IdentitySecuritySummariesGeneratedReport identity_security_summaries_report = 4;
// encoded as a separate tp.identity_security.usage PostHog event
repeated IdentitySecurityReport identity_security_report = 5;
}
message SubmitUsageReportsResponse {