fix: preserve Vale severity in CI annotations and add three-severity demo (#26587)

Closes DOCS-426. Follow-up to
[#26586](https://github.com/coder/coder/pull/26586) (DOCS-425, strip),
which merged first.

## Problem

The Vale problem matcher at `.github/vale-problem-matcher.json`
hard-codes `"severity": "warning"`. Every Vale finding renders as a
GitHub `warning` annotation, regardless of Vale's actual severity. Nick
observed this on PR [#25501](https://github.com/coder/coder/pull/25501):
error-level findings from `Coder.BrandNames` appear as warnings.

This collapsed the doctrine's three-severity ladder (`error` / `warning`
/ `suggestion`) into a single advisory channel for the reader of a PR
diff. This PR restores the ladder visually so contributors and reviewers
see each rule's intended severity.

## Root cause

GitHub Actions problem matchers expect either a regex capture group for
severity or a hard-coded severity. Vale's `--output=line` format
produces `path:line:col:rule:message` with severity stripped, so the
matcher had no severity to capture and fell back on the hard-coded
value.

## Fix

### Commit 1: severity rendering

Switch the Vale prose lint step to `vale --output=JSON` and pipe through
`jq` to emit GitHub workflow commands directly. Drop the problem matcher
file.

| Vale severity | GitHub workflow command |
|---|---|
| `suggestion` | `::notice::` |
| `warning` | `::warning::` |
| `error` | `::error::` |

Message bodies are URL-encoded for `%`, `\r`, and `\n` per the GitHub
Actions workflow command spec. The Vale step stays advisory
(`continue-on-error: true`, `vale --no-exit`); rendering becomes correct
but the step never fails the job.

### Commit 2: three-severity demo

Three throwaway `Coder.Demo*` rules at `level: suggestion`, `level:
warning`, and `level: error`, plus a
`docs/.style/_vale-annotation-demo.md` file that triggers each rule
exactly once. Together with the rendering fix above, this PR's CI
surfaces three GitHub annotations in three distinct severities (notice,
warning, error). Use the Files Changed view to inspect rendering.

The demo files live permanently in `docs/.style/`, which is excluded
from coder.com. They re-trigger annotations only on PRs that touch the
demo file itself, so they don't pollute CI on day-to-day PRs.

## Sample output

<img width="1443" height="1293" alt="image"
src="https://github.com/user-attachments/assets/fb337315-7b55-40b3-9983-828b2d5399fc"
/>

<img width="1443" height="1293" alt="image"
src="https://github.com/user-attachments/assets/b02d575d-5905-4c6d-b145-ad5df6e04f11"
/>

## Out of scope

Blocking merge on `error`-level findings is the natural next step but is
sequenced as the **final** step of the prose-style rollout. It was
prototyped in this PR (commit 3, since backed out) and verified
end-to-end against the demo doc. The work moved to
[DOCS-433](https://linear.app/codercom/issue/DOCS-433/block-merge-on-vale-error-level-findings-final-step-of-prose-style)
so the corpus of enabled rules is broad enough by the time the gate
lands that it catches real violations rather than novelty failures from
a single rule.

## Expected CI state on this PR

`lint-docs` passes. The three demo annotations render at lines 17 / 19 /
21 of `docs/.style/_vale-annotation-demo.md` as `::notice::`,
`::warning::`, and `::error::` respectively. The `::error::` annotation
does not fail the job because the Vale step is still advisory under this
PR.

Local verification of the rendering pipeline:

```
$ printf '%s\n' 'docs/.style/_vale-annotation-demo.md' \
    | xargs -d '\n' vale --no-exit --output=JSON \
    | jq -r '...'
::notice  file=docs/.style/_vale-annotation-demo.md,line=17,col=3,title=Coder.DemoSuggestion::[Demo] Suggestion-level Vale annotation.
::warning file=docs/.style/_vale-annotation-demo.md,line=19,col=3,title=Coder.DemoWarning::[Demo] Warning-level Vale annotation.
::error   file=docs/.style/_vale-annotation-demo.md,line=21,col=3,title=Coder.DemoError::[Demo] Error-level Vale annotation.
```

<details>
<summary>Decision log</summary>

- **Workflow commands vs custom Vale template + updated matcher**: chose
workflow commands because the transform is a 10-line jq pipeline with no
extra files to maintain, and it bypasses GitHub Actions problem-matcher
limitations entirely. The custom-template option would have kept the
matcher infrastructure but required an additional Go template file under
`.github/`.
- **Throwaway demo rules vs reusing existing rules**: chose throwaway
because we wanted each severity to fire deterministically from a single
unambiguous marker. Reusing existing rules would couple the demo to
corpus content and obscure the signal.
- **Demo persists vs drops before merge**: persists. The merge-gate
constraint that originally forced the demo to drop is gone (deferred to
DOCS-433). The four demo files live in `docs/.style/`, excluded from
coder.com, and only annotate PRs that touch them. They double as a
permanent canary so a future regression in severity rendering surfaces
immediately on whichever PR introduces it, and as the verification
artifact DOCS-433 uses when re-installing the merge gate.
- **`docs/.style/_vale-annotation-demo.md` filename**: underscore prefix
follows Coder convention for files that exist outside the normal docs
taxonomy. Not surfaced on coder.com/docs because `docs/.style/` is
excluded from the manifest, deploy workflow, and docs preview.
- **Merge-block deferred to DOCS-433**: the rendering fix and the merge
gate are independent changes. Shipping the rendering first lets
contributors see the three-severity ladder while the rule catalogue is
still small and the false-positive policy hasn't been stress-tested yet.
The gate lands as the final step of the rollout, after the catalogue is
broad enough that the gate covers real prose-style policy rather than
one rule's enforcement.

</details>

---
*Filed via [Coder Agents](https://coder.com/docs/ai-coder/agents) on
Nick's behalf.*
This commit is contained in:
Nick Vigilante
2026-06-23 12:55:15 -04:00
committed by GitHub
parent 854d280834
commit 6acf32701e
6 changed files with 77 additions and 21 deletions
-18
View File
@@ -1,18 +0,0 @@
{
"problemMatcher": [
{
"owner": "vale",
"severity": "warning",
"pattern": [
{
"regexp": "^(.+):(\\d+):(\\d+):([^:]+):(.+)$",
"file": 1,
"line": 2,
"column": 3,
"code": 4,
"message": 5
}
]
}
]
}
+19 -3
View File
@@ -234,9 +234,25 @@ jobs:
echo "No changed Markdown files under docs/ on disk; skipping Vale."
exit 0
fi
echo "::add-matcher::.github/vale-problem-matcher.json"
printf '%s\n' "$files" | xargs -d '\n' mise exec "aqua:errata-ai/vale" -- vale --no-exit --output=line
echo "::remove-matcher owner=vale::"
# Vale's --output=line strips per-finding severity, so the
# previous problem-matcher approach collapsed every finding to
# a single hard-coded severity. Use --output=JSON instead and
# emit GitHub workflow commands directly so error/warning/
# suggestion render with their actual Vale severities. URL-
# encode message bodies for `%`, `\r`, and `\n` per the
# GitHub Actions workflow command spec. See DOCS-426.
printf '%s\n' "$files" \
| xargs -d '\n' mise exec "aqua:errata-ai/vale" -- vale --no-exit --output=JSON \
| jq -r '
to_entries[]
| .key as $file
| .value[]
| (if .Severity == "suggestion" then "notice"
elif .Severity == "warning" then "warning"
else "error" end) as $level
| (.Message | gsub("%"; "%25") | gsub("\r"; "%0D") | gsub("\n"; "%0A")) as $msg
| "::\($level) file=\($file),line=\(.Line),col=\(.Span[0]),title=\(.Check)::\($msg)"
'
- name: Save Vale styles
# Only the default branch is trusted to write the cache, so PR