Add client side circuit breaker to auth clients (#10282)

* Add client side circuit breaker to auth clients

In order to apply back pressure we can utilize a circuit breaker that
monitors error responses from auth server. When tripped it will prevent
all outbound requests to auth for a period of time. This can also help
prevent a potential thundering heard when auth is in an unhealthy state.
By default the circuit breaker will only be tripped if 90% of the
requests made in the monitoring interval fail.

api/defaults/defaults.go

@@ -51,6 +51,27 @@ const (
 
 	// SessionTrackerTTL defines the default base ttl of a session tracker.
 	SessionTrackerTTL = time.Hour
+
+	// BreakerInterval is the period in time the circuit breaker will
+	// tally metrics for
+	BreakerInterval = time.Minute
+
+	// TrippedPeriod is the default period of time the circuit breaker will
+	// remain in breaker.StateTripped before transitioning to breaker.StateRecovering. No
+	// outbound requests are allowed for the duration of this period.
+	TrippedPeriod = 60 * time.Second
+
+	// RecoveryLimit is the default number of consecutive successful requests needed to transition
+	// from breaker.StateRecovering to breaker.StateStandby
+	RecoveryLimit = 3
+
+	// BreakerRatio is the default ratio of failed requests to successful requests that will
+	// result in the circuit breaker transitioning to breaker.StateTripped
+	BreakerRatio = 0.9
+
+	// BreakerRatioMinExecutions is the minimum number of requests before the ratio tripper
+	// will consider examining the request pass rate
+	BreakerRatioMinExecutions = 10
 )
 
 var (