fix: reduce cost of prebuild failure (#17697)

Relates to https://github.com/coder/coder/issues/17432

### Part 1:

Notes:
- `GetPresetsAtFailureLimit` SQL query is added, which is similar to
`GetPresetsBackoff`, they use same CTEs: `filtered_builds`,
`time_sorted_builds`, but they are still different.

- Query is executed on every loop iteration. We can consider marking
specific preset as permanently failed as an optimization to avoid
executing query on every loop iteration. But I decided don't do it for
now.

- By default `FailureHardLimit` is set to 3.

- `FailureHardLimit` is configurable. Setting it to zero - means that
hard limit is disabled.

### Part 2

Notes:
- `PrebuildFailureLimitReached` notification is added.
- Notification is sent to template admins.
- Notification is sent only the first time, when hard limit is reached.
But it will `log.Warn` on every loop iteration.
- I introduced this enum:
```sql
CREATE TYPE prebuild_status AS ENUM (
  'normal',           -- Prebuilds are working as expected; this is the default, healthy state.
  'hard_limited',     -- Prebuilds have failed repeatedly and hit the configured hard failure limit; won't be retried anymore.
  'validation_failed' -- Prebuilds failed due to a non-retryable validation error (e.g. template misconfiguration); won't be retried.
);
```
`validation_failed` not used in this PR, but I think it will be used in
next one, so I wanted to save us an extra migration.

- Notification looks like this:
<img width="472" alt="image"
src="https://github.com/user-attachments/assets/e10efea0-1790-4e7f-a65c-f94c40fced27"
/>

### Latest notification views:
<img width="463" alt="image"
src="https://github.com/user-attachments/assets/11310c58-68d1-4075-a497-f76d854633fe"
/>
<img width="725" alt="image"
src="https://github.com/user-attachments/assets/6bbfe21a-91ac-47c3-a9d1-21807bb0c53a"
/>

codersdk/deployment.go

@@ -807,6 +807,12 @@ type PrebuildsConfig struct {
 	// ReconciliationBackoffLookback determines the time window to look back when calculating
 	// the number of failed prebuilds, which influences the backoff strategy.
 	ReconciliationBackoffLookback serpent.Duration `json:"reconciliation_backoff_lookback" typescript:",notnull"`
+
+	// FailureHardLimit defines the maximum number of consecutive failed prebuild attempts allowed
+	// before a preset is considered to be in a hard limit state. When a preset hits this limit,
+	// no new prebuilds will be created until the limit is reset.
+	// FailureHardLimit is disabled when set to zero.
+	FailureHardLimit serpent.Int64 `json:"failure_hard_limit" typescript:"failure_hard_limit"`
 }
 
 const (
@@ -3086,6 +3092,17 @@ Write out the current server config as YAML to stdout.`,
 			Annotations: serpent.Annotations{}.Mark(annotationFormatDuration, "true"),
 			Hidden:      true,
 		},
+		{
+			Name:        "Failure Hard Limit",
+			Description: "Maximum number of consecutive failed prebuilds before a preset hits the hard limit; disabled when set to zero.",
+			Flag:        "workspace-prebuilds-failure-hard-limit",
+			Env:         "CODER_WORKSPACE_PREBUILDS_FAILURE_HARD_LIMIT",
+			Value:       &c.Prebuilds.FailureHardLimit,
+			Default:     "3",
+			Group:       &deploymentGroupPrebuilds,
+			YAML:        "failure_hard_limit",
+			Hidden:      true,
+		},
 	}
 
 	return opts