mirror of
https://github.com/coder/coder.git
synced 2026-10-16 14:48:56 +08:00
Nick Vigilante
f1ed84ffab
## Summary The docs sidebar is now written as YAML files under `docs/manifest/`, one route and its direct children per file, and a new tool, `scripts/docsmanifestgen`, compiles them into `docs/manifest.json`. Moving two top-level sections is now a 2-line change in `docs/manifest/index.yml` instead of a diff of hundreds of lines in a 2,600-line JSON file. The compiled `docs/manifest.json` is byte-identical to `main`, so coder.com, docs-preview, deploy-docs, and offlinedocs need no changes. Refs [DOCS-1131](https://linear.app/codercom/issue/DOCS-1131/author-the-docs-sidebar-in-per-section-yaml-compiled-to) ## Changes - **Sources:** `docs/manifest/index.yml` holds `versions` and the section files in sidebar order. Each section file (`about.yml`, `admin.yml`, and so on) holds one top-level route. - **One level per file:** a file holds one route and its direct children. A child with children of its own, or with `children_from`, lives in its own file in a directory named after the parent file, and the parent lists it as `- include: admin/users.yml`. For example, `get-started.yml` includes `get-started/customize-your-template.yml`, which lists that page's children. The 8 section files split into 53 source files. - **`scripts/docsmanifestgen`:** `build` compiles the sources into `docs/manifest.json`, `fmt` formats them, and `check` validates them. - Keys must be known (a typo like `titel:` fails with a line number) and appear in a fixed order in both the YAML and the JSON. - Every `path` must exist, and every file under `docs/manifest/` must be referenced from `index.yml`, an `include`, or a `children_from`. - A grandchild in the same file fails with a message to move the child into its own file, and an `include` entry can't set other keys or name a file that is already used. - A comment on the same line as a `title` or `description` is rejected, because an unquoted ` #` would silently cut off the value. - **Generated fragments:** `clidocgen` and `apidocgen` no longer rewrite the manifest. They write the Command Line and REST API children to `docs/manifest/generated/*.json`, which the sources pull in with `children_from`. The tool accepts fragments as YAML or JSON. - The hand-written REST API descriptions and states moved into the YAML, attached to generated routes by title. A YAML entry that matches no generated route, or disagrees with one, fails the build instead of being dropped silently. - **Shared code:** loading, validation, and fragment merging live in `scripts/docgenenv/sources.go`, so the generators and the tool share one loader. `docgenenv.Route` gains YAML tags and the source-only `ChildrenFrom` and `Include` fields (`json:"-"`). `LoadManifest` is removed because nothing reads the compiled JSON anymore. - **Makefile:** - `docs/manifest.json` is built from the sources and fragments, which replaces the `_gen/manifest-staging.json` handoff. - Each fragment has its own rule, and the fragments are added to the `.PRECIOUS`, `GEN_FILES`, and `gen/mark-fresh` lists. - `DOCS_MANIFEST_SOURCES` finds the source files recursively. - New targets: `fmt/docs-manifest` (part of `fmt`), `lint/docs-manifest` (part of `lint`), and `gen/docs-manifest`, the quick path after a sidebar-only edit that skips regenerating the CLI and API fragments. - **`.gitattributes`:** marks `docs/manifest.json` and `docs/manifest/generated/*` as generated so GitHub collapses them in diffs. - **Guidance:** the content guidelines, the style guide, `DOCS_STYLE_GUIDE.md`, the `write-docs` and `doc-check` skills, `docs/README.md`, and `docs/tutorials/example-guide.md` now point to the YAML sources instead of `docs/manifest.json`. ## Testing - Split today's `docs/manifest.json` into the sources, rebuilt it, and confirmed the output is byte-identical to `main`. After splitting the sources to one level per file, the rebuilt `docs/manifest.json` and a fresh `clidocgen` CLI fragment were still byte-identical, and `LoadManifestSources` still finds the REST API and Command Line routes through their includes. After rebasing onto `main`, both commits rebuild `main`'s `docs/manifest.json` byte for byte. - Hand-edited `docs/manifest.json` and a YAML source, then rebuilt: the YAML change persisted and the JSON edit was overwritten. CI's `make -B gen` plus `check_unstaged.sh` fails on a committed hand edit to the JSON. - Deleted both fragments and ran `make docs/manifest.json` in a fresh worktree, which regenerated them through swag and `clidocgen`. `docs/manifest.json`, the fragments, and every generated CLI and API page had no diff. - Swapped User Guides and Administration in `index.yml`: 2 changed lines in the source, 166 generated lines in `docs/manifest.json`. Swapping back restored the file exactly. - `go test` for `scripts/docgenenv`, `scripts/docsmanifestgen`, `scripts/clidocgen`, and `scripts/apidocgen/postprocess`. The new tests cover merging, includes, every validation error, idempotent formatting that keeps comments, and a check that the key order matches the struct field order. - `golangci-lint` on the changed packages, and the full `make pre-commit` hook (gen, fmt, lint including `lint/docs-manifest`, and the slim build). ## Notes - Open PRs that edit `docs/manifest.json` will need to redo their sidebar changes in the YAML sources after this merges. - #30161 (merged) stopped escaping `&`, `<`, and `>` in `docs/manifest.json`. This PR carries that over: all manifest JSON escaping is in `docgenenv.MarshalJSON`, which sets `enc.SetEscapeHTML(false)`, and `Manifest.Marshal` from #30161 is gone because the generators no longer write the manifest. - #30129 (merged) flattened `admin/external-auth/index.md` and `ai-coder/agents/tools/index.md` to single files, and the YAML sources use the new paths. - The CLI index and REST API page front matter still update only on a full `make gen` when their sidebar metadata changes, as it did before. That gap and the overall cost of `make gen` are tracked in [DOCS-1141](https://linear.app/codercom/issue/DOCS-1141/reduce-the-compute-cost-of-make-gen). <details> <summary>Decision log</summary> ### Goal Make the sidebar easy to reorder and hard to break, with strict key order and YAML sources, while keeping `docs/manifest.json` as the file downstream systems read. ### Jsonnet vs. a custom tool Jsonnet was evaluated first and rejected, based on tests against go-jsonnet v0.22.0: - Jsonnet always sorts object keys alphabetically, and there's no option to keep them in order (`preserve_order` isn't a parameter of `manifestJsonEx` or `objectFieldsEx`). Passing today's manifest through Jsonnet and then biome changed 2,114 of 2,478 lines. - Unknown keys pass through silently, so a `titel:` typo ends up in the output. - The sources would be `.jsonnet` files, not YAML. Reading YAML takes `std.parseYaml(importstr ...)`, and the keys still come out sorted. - Restoring key order would need a Go step after Jsonnet anyway, and that step is most of this tool. Decoding YAML into Go structs with yaml.v3 (already a dependency) gives key order from the struct, rejects unknown keys, keeps comments, and reproduced the current manifest byte for byte. ### Design choices - **Ordered by `index.yml`:** reordering sections touches only the index. - **One route and its direct children per file:** every file stays short and flat, so a subsection can be found, reviewed, and moved as one file. Files nest by parent file name rather than by docs path, so the directory tree matches the sidebar tree. Include paths are relative to `docs/manifest/`, like `children_from`, so every reference to a file is spelled the same way. A child with `children_from` also gets its own file, so the rule has no exceptions. - **Generated subtrees in their own fragments:** the generators can't safely rewrite hand-edited YAML, so they own separate files. Hand-written metadata for generated routes lives in the YAML, attached by title, so descriptions never live in a generated file. - **Key order enforced in both sources and output:** `fmt` rewrites the sources, `check` fails on unformatted sources, and the output order comes from the `docgenenv.Route` field order. - **`&` left unescaped:** the output matches `main` after #30161, which stopped escaping `&` downstream. - **`gen/docs-manifest` separate from `fmt`:** keeping manifest assembly out of `fmt` means `make -j gen fmt` never has two jobs writing `docs/manifest.json` at once. - **YAML pitfalls:** unquoted `no` and `123` stay strings because the fields are typed as strings, an unquoted `: ` is a parse error, and the silent ` #` truncation is rejected by the comment check. ### Out of scope - Replacing `&` with "and" in titles (a style question). - Reducing the cost of `make gen` (DOCS-1141). </details> --- _This PR was generated by Coder Agents on behalf of @nickvigilante._