opensource/coder
3
0
Fork 0
mirror of https://github.com/coder/coder.git synced 2026-09-24 15:04:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix: fix false positive disconnected agent metric reporting (#24225)

Browse Source 
We noticed during higher active workspace counts that the agent
connection metric, generated via a query to the database, would report a
relatively high amount of agents as disconnected. Somewhere between 5
and 20%. However, other metrics such as # of websocket connections would
suggest that all agent connections are healthy.

Looking at the `Agents` function in prometheus metrics, plus the query
execution time (not accounting for actual database RT time) revealed
that this reporting of agents as disconnected was almost certainly false
positives due to clock drift in the way we're generating the metric
values. At 10k metrics, with a p50 of 2ms and p99 of 5ms, the entire
`agents` function could take upwards of 50s to execute. Because we were
doing a query/database RT to query th apps for each agent individually,
and grabbing a `time.Now` value on each iteration of that loop, it's
likely the portion of agents that were reported as disconnected were
those that had last heartbeat the furthest in the past.

The fix here is to set a consistent `now` before fetching agent data to
avoid clock drift inflating the inactive timeout comparison, and replace
the per-agent app query N+1 with a single batched lookup to prevent loop
execution time from pushing agents over the disconnected threshold.

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Callum Styan
2026-04-14 22:23:06 -07:00
committed by GitHub
co-authored by Claude Sonnet 4.6
parent 0080bcbf33
commit 730edba87a
6 changed files with 59 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
coderd/database/modelmethods.go
+3 -4
View File
@@ -15,7 +15,6 @@ import (
"golang.org/x/oauth2"
"golang.org/x/xerrors"
"github.com/coder/coder/v2/coderd/database/dbtime"
"github.com/coder/coder/v2/coderd/rbac"
"github.com/coder/coder/v2/coderd/rbac/policy"
)
@@ -624,7 +623,7 @@ type WorkspaceAgentConnectionStatus struct {
DisconnectedAt *time.Time `json:"disconnected_at"`
}
func (a WorkspaceAgent) Status(inactiveTimeout time.Duration) WorkspaceAgentConnectionStatus {
func (a WorkspaceAgent) Status(now time.Time, inactiveTimeout time.Duration) WorkspaceAgentConnectionStatus {
connectionTimeout := time.Duration(a.ConnectionTimeoutSeconds) * time.Second
status := WorkspaceAgentConnectionStatus{
@@ -643,7 +642,7 @@ func (a WorkspaceAgent) Status(inactiveTimeout time.Duration) WorkspaceAgentConn
switch {
case !a.FirstConnectedAt.Valid:
switch {
case connectionTimeout > 0 && dbtime.Now().Sub(a.CreatedAt) > connectionTimeout:
case connectionTimeout > 0 && now.Sub(a.CreatedAt) > connectionTimeout:
// If the agent took too long to connect the first time,
// mark it as timed out.
status.Status = WorkspaceAgentStatusTimeout
@@ -658,7 +657,7 @@ func (a WorkspaceAgent) Status(inactiveTimeout time.Duration) WorkspaceAgentConn
// If we've disconnected after our last connection, we know the
// agent is no longer connected.
status.Status = WorkspaceAgentStatusDisconnected
case dbtime.Now().Sub(a.LastConnectedAt.Time) > inactiveTimeout:
case now.Sub(a.LastConnectedAt.Time) > inactiveTimeout:
// The connection died without updating the last connected.
status.Status = WorkspaceAgentStatusDisconnected
// Client code needs an accurate disconnected at if the agent has been inactive.