mirror of
https://github.com/moeru-ai/airi.git
synced 2026-10-30 19:34:10 +08:00
feat(core-agent,stage-ui): now speech formula, math, chem correctly (#3054)
This commit is contained in:
Neko
@@ -118,6 +118,7 @@ words:
|
||||
- Equirectangular
|
||||
- esaxx
|
||||
- esbuild
|
||||
- evals
|
||||
- eventa
|
||||
- Factorio
|
||||
- fakemic
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
# Speech text preprocessor with a MathJSON layer
|
||||
|
||||
Status: accepted
|
||||
|
||||
## Context
|
||||
|
||||
`toSpeechText` in `@proj-airi/core-agent` turns a markdown reply into the text that a TTS provider reads. It reads
|
||||
TeX formulas, chemistry, units, and prices as words in Chinese, English, Japanese, and Korean.
|
||||
|
||||
The formula reader has one hand-written pipeline:
|
||||
|
||||
```text
|
||||
TeX ──parseTex──▶ TexNode (syntax) ──reader (guesses meaning and builds words) ──▶ speech
|
||||
```
|
||||
|
||||
The reader mixes two jobs. It guesses what the notation means, for example that `\frac{d}{dx}` is a derivative or
|
||||
that `0^{+}` is a side of a limit. It also builds the words of each language. A new language or a new reading touches
|
||||
the same functions as a new rule of meaning. The chemistry reader parses a small part of mhchem by itself.
|
||||
|
||||
The formula speech eval (`evals/tts-speech-formula`) showed what goes wrong when a voice gets the text: a digit, a
|
||||
letter, a unit, or a structure that the voice reads in another language or that the listener cannot parse. The fixes
|
||||
of 2026-10-19 raised the rule engine from 89.5% to 98.5% without TTS, and from 70% to 94% with the official TTS. Each
|
||||
fix added another guess to the reader.
|
||||
|
||||
Popular libraries already parse TeX and mhchem. KaTeX parses TeX for the renderer of the stage apps, about 17.8
|
||||
million downloads each week. mhchemParser is the mhchem parser of KaTeX and MathJax. Compute Engine parses TeX into
|
||||
MathJSON, but it is a computer algebra system. It changes what the writer wrote: `50\%` becomes `0.5`, `3\times10^8`
|
||||
becomes one number, `AB` becomes a product, and `\overline{AB}` becomes a complex conjugate.
|
||||
|
||||
## Decision
|
||||
|
||||
### The goal is a reading for a person who listens
|
||||
|
||||
AIRI reads a reply aloud. A reading must keep the meaning of the formula and sound like a teacher who explains it. It
|
||||
is not a dictation: it does not name details that only matter to a person who writes the formula down, such as the end
|
||||
of a subscript or the case of a lone capital letter. A reading that changes the meaning, as silent brackets after a
|
||||
minus sign do, is a defect. A reading that adds words only for a dictation is a defect too.
|
||||
|
||||
### Three layers for formulas
|
||||
|
||||
```text
|
||||
TeX ──KaTeX parse + normalize──▶ syntax tree ──lower──▶ MathJSON (meaning) ──render + locale──▶ speech
|
||||
```
|
||||
|
||||
- **Parse.** `katex.__parse` with `throwOnError: false` is the syntax parser. KaTeX is the parser of the formulas that
|
||||
the user sees, so speech and display agree: an override gives rehype-katex the catalog KaTeX. The catalog pins the
|
||||
exact version, because `__parse` is not a public API. KaTeX returns a command that it does not know as text, so `\foo{x}` still reads. Unbalanced braces are balanced
|
||||
first, and input that KaTeX still cannot parse falls back to the source text without backslashes.
|
||||
- **Normalize.** The KaTeX tree is a layout tree. Normalization joins what KaTeX splits for display, for example the
|
||||
digits of `3.14` and of `10^{8}`, unwraps the groups around scripts and arguments, reads negations from the MathML
|
||||
branch, and pairs delimiters. The result is the small syntax tree of `tex.ts`.
|
||||
- **Lower.** One module turns the syntax tree into MathJSON. It holds every rule of meaning, for example `d/dx` as
|
||||
`D`, a one-sided limit, an interval after `\in`, a product with a parenthesized sum, and a function signature.
|
||||
Lowering keeps what the writer wrote. It never simplifies or evaluates.
|
||||
- **Render.** One module reads MathJSON. It chooses words only by the head of each expression and the locale. It has
|
||||
no rule of meaning.
|
||||
|
||||
### MathJSON, extended for speech
|
||||
|
||||
The tree uses the MathJSON format: a number, a string symbol, or an array whose first item is the head. Standard heads
|
||||
keep their MathJSON meaning, so the eval can compare them with Compute Engine. Speech needs more than standard
|
||||
MathJSON keeps, so the tree also has these heads:
|
||||
|
||||
| Head | Meaning | Why standard MathJSON is not enough |
|
||||
| --- | --- | --- |
|
||||
| `Percent` | `50\%` | Standard parsing turns it into `0.5`. |
|
||||
| `Name` | `AB`, `ABC` of geometry | Standard parsing reads juxtaposition as a product. |
|
||||
| `Segment`, `Arc` | `\overline{AB}`, `\overset{\frown}{AB}` | Standard parsing has a conjugate or no meaning. |
|
||||
| `Unit` | `m`, `km/h`, `m/s^2` | The unit keeps its symbol, denominator, and power. Standard MathJSON has no unit reading. |
|
||||
| `Celsius` | `25^\circ\mathrm{C}` | Standard parsing turns degrees into radians. |
|
||||
| `Chemistry` | the content of `\ce{}` | MathJSON has no chemistry. |
|
||||
| `Delimiter`, `Group` | parentheses that the writer wrote | Lowering decides whether they are read: `Group` is read as a quantity, `Delimiter` is silent. |
|
||||
| `Mapping`, `Clauses` | `f: X \to Y, x \mapsto y` | A signature and comma-separated statements read as one sentence each. |
|
||||
| `Apply` | `f(x)`, `H_k(M, N)` | Standard parsing calls only known functions. Other names and their parentheses become a product. |
|
||||
| `Sequence` | `r, s \in \Sigma` | The items of a list read with a pause. Without it, `r, s` reads as the product `r s`. |
|
||||
| `Arrow` | `H_k(M) \to H_k(M, N)` | `To` reads "approaches". An arrow between two structures is a map and reads "to". |
|
||||
| `Superscript` | `\mathbb{N}^{+}`, `f^{(n)}` | A sign or a label in parentheses is a mark, not a power. |
|
||||
| `Bracket`, `Placeholder` | `[M]`, `p(\cdot)` | Square brackets around one item and an empty argument slot are read. |
|
||||
| `Expectation`, `Probability`, `Variance`, `Covariance` | `\mathbb{E}_{x \sim p}[f(x)]` | Standard parsing reads `\mathbb{E}` as a set, and its brackets as a factor. The subscript is the condition. |
|
||||
| `Distributed` | `x \sim p_t` | Standard `\sim` is similarity. On the right of `\sim`, a calligraphic name or a density letter is a distribution. |
|
||||
| `BigUnion`, `BigIntersection` | `\bigcup_{i=1}^{n} A_i` | Standard parsing has no bounds for them, so the upper bound reads as a power. |
|
||||
| `Compose`, `Assign`, `GreaterApprox`, `LessApprox`, `NotDivides`, `Proves` | `f \circ g`, `x \coloneqq y`, `n \gtrsim 10` | Lowering keeps the meaning that the renderer needs. `Ceil` and `Floor` are standard heads. |
|
||||
| `Inverse`, `Ordinal` | `f^{-1}(x)`, `\cos^{-1}`, `n^{\text{th}}` | Standard parsing reads them as powers, and the listener hears 1/f. |
|
||||
| `Cases`, `Case`, `Matrix`, `Determinant`, `Row` | `\begin{cases}`, `\begin{pmatrix}` | Each case reads with its condition, and each row with its cells. Without them, the cells run together. |
|
||||
| `Raw` | `\ce{H2O}` before its extension lowers it | The parser keeps the source of an extension command. |
|
||||
|
||||
TeX writes a call and a product the same way, as in `f(x)` and `n(n+1)`. Lowering reads parentheses after a name as a
|
||||
call when they hold a comma, hold no operator, or follow a name with scripts, an accent, or a usual function letter.
|
||||
Parentheses after a function command, as `\ln(x+1)`, or after upright text, as `\operatorname{tr}(A)`, are a call too.
|
||||
Other parentheses after a letter multiply. A call reads "f of x" in English. In Chinese, Japanese, and Korean a simple
|
||||
call keeps the reading of a product, and a call with more than one argument or with an operator says its parentheses.
|
||||
|
||||
`1e-3` written without spaces is scientific notation, and the parser rewrites it as `1\times 10^{-3}`. `2e - 1` with
|
||||
spaces keeps Euler's number. Digits grouped as `1,000,000` or `1{,}000` join, but one plain group, as `[1,100]`, stays a
|
||||
list.
|
||||
|
||||
`3\times10^{8}` stays `Multiply` and `Power`. Standard parsing folds it into one number, and the speech tree keeps
|
||||
it as the writer wrote it.
|
||||
|
||||
A symbol keeps its letter case. The renderer says the case only when one formula uses both cases of a letter.
|
||||
|
||||
### Chemistry with mhchemParser
|
||||
|
||||
`\ce{}` and `\pu{}` go through `mhchemParser.toTex`. KaTeX parses its output, and lowering turns it into `Chemistry`
|
||||
nodes: species, coefficients, charges, states, hydrate dots, and arrows. The hand-written mhchem grammar is removed.
|
||||
|
||||
### Streamed segments
|
||||
|
||||
A speech producer used to hold the whole reply until `end()`, so an answer started to speak only after the model
|
||||
ended. Markdown cannot be parsed one token at a time: a formula, a code block, a link, or a table that is cut in two
|
||||
reads as its source. The pipeline now has a segmenter before the transformation:
|
||||
|
||||
```text
|
||||
model text ──MarkdownSegmenter──▶ segment ──toSpeechText──▶ spoken text ──TTS chunker──▶ TTS requests
|
||||
```
|
||||
|
||||
- `MarkdownSegmenter` ends a segment after a sentence end or a line, only where nothing is open: fenced code,
|
||||
`$$...$$`, `\[...\]`, inline code, `$...$`, `\(...\)`, link brackets, emphasis, or a table. A `$` before a price does
|
||||
not open a formula.
|
||||
- A line ends a segment only when the next line has started. An indented line can continue a list item, and parsed
|
||||
alone it would be a code block. A line of `=` makes a heading.
|
||||
- `toSpeechText` reads each segment. `before` is the spoken text of the segment before it, so a display formula that
|
||||
starts a segment reads in the language of the text before it.
|
||||
- A flush follows each segment, because the chunker reads one character past a sentence end.
|
||||
- Constructs that refer to later text, as reference link definitions, read as text. Model replies rarely use them.
|
||||
|
||||
### A preprocessor for all spoken text
|
||||
|
||||
`toSpeechText` becomes a host for preprocessor plugins. A plugin owns one kind of content, for example formulas,
|
||||
money, or numbers in prose. The words come from locale packs, so a contributor adds the reading of a language without
|
||||
changing a plugin.
|
||||
|
||||
```text
|
||||
markdown ──\( \[ to $──▶ prepare hooks ──remark──▶ mdast ──formula and prose hooks──▶ spoken text
|
||||
│
|
||||
├─ money (prepare: escape prices, prose: read prices)
|
||||
├─ math (formula: KaTeX → MathJSON → render,
|
||||
│ prose: bare TeX such as \ce{H2O})
|
||||
└─ future: rewrites, special pronunciations, prose numbers
|
||||
```
|
||||
|
||||
A plugin has three optional hooks:
|
||||
|
||||
| Hook | Input | Rule |
|
||||
| --- | --- | --- |
|
||||
| `prepare` | the markdown before parsing | Each plugin gets the output of the plugin before it. |
|
||||
| `formula` | one TeX formula and its language | The first plugin that returns words wins. The source is kept when none does. |
|
||||
| `prose` | a run of text or inline code, its language, and `readFormula` | Each plugin gets the output of the plugin before it. |
|
||||
|
||||
- A plugin is trusted in-process code, as voice input plugins are, see
|
||||
[voice plugins watch and finalize](2026-10-16-voice-plugins-watch-and-finalize.md).
|
||||
- Installation order decides which plugin sees a node first. There is no dependency graph. The default order is
|
||||
money, then math, so a price reads before the math plugin looks for bare TeX.
|
||||
- `options.preprocessors` replaces the default list. A caller adds a plugin by putting it in front of
|
||||
`defaultSpeechPreprocessors()` or after it.
|
||||
- The rules of formulas are a rule system without the speech pipeline, so other projects can use them. They are in
|
||||
three packages:
|
||||
- `@proj-airi/mathjson`: the MathJSON types and helpers.
|
||||
- `@proj-airi/speech-transformer-math`: parse, lower, render, units, and the locale packs.
|
||||
- `@proj-airi/speech-transformer-chem`: mhchem chemistry, as an extension of the math reader.
|
||||
- `MathSpeech` is a class. Each instance owns its locale packs and extensions, so two callers, or two tests, can read
|
||||
differently in one process. core-agent creates one instance, `mathSpeech`, with chemistry.
|
||||
- An extension (`MathSpeechExtension`) names raw commands, for example `\ce`. KaTeX does not parse them. The parser keeps
|
||||
their source in a `Raw` node, the extension lowers it to its own heads, and its render handlers read those heads.
|
||||
- Chemistry and units are not preprocessor plugins. They appear only inside formulas, and the formula hook already
|
||||
selects the first reader.
|
||||
- A locale pack (`MathLocale`) holds the words of one language for formulas and units. A chemistry locale
|
||||
(`ChemLocale`) holds the words of chemistry for the same language. Money is not a formula rule, so its words are in
|
||||
the money plugin of core-agent. `mathSpeech.register`
|
||||
adds a pack or replaces one, and returns a function that restores the earlier pack. A language without a pack uses
|
||||
English. The math plugin uses the pack of its sentence.
|
||||
- The tests of each language are next to its locale pack, for example `locales/ko.test.ts`.
|
||||
- Unit packs name each SI prefix and unit once. `speakUnit` composes them, so a pack does not list each pair.
|
||||
|
||||
## Scope
|
||||
|
||||
- `packages/core-agent/src/voice/output/`: the preprocessor host, and the math and money plugins.
|
||||
- `packages/mathjson`, `packages/speech-transformer-math`, and `packages/speech-transformer-chem`: the rules of
|
||||
formulas, units, and chemistry, and their locale packs for `zh`, `en`, `ja`, and `ko`.
|
||||
- `evals/tts-speech-formula`: a structural comparison of the lowered MathJSON with Compute Engine, a speed benchmark,
|
||||
and an optional MathBridge dataset.
|
||||
|
||||
Non-goals:
|
||||
|
||||
- Compute Engine in `@proj-airi/core-agent`. It stays a dependency of the eval only.
|
||||
- Evaluation, simplification, or type inference of formulas.
|
||||
- Thai and other languages without a locale pack.
|
||||
- Changes to the TTS providers or to the speech stream.
|
||||
|
||||
## Module dependencies
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
SpeechText[toSpeechText] --> Host[preprocessor host]
|
||||
Host --> Math[math plugin]
|
||||
Host --> Money[money plugin]
|
||||
Math --> Instance[MathSpeech instance]
|
||||
Instance --> KaTeX[katex __parse]
|
||||
Instance --> Lower[lower to MathJSON]
|
||||
Instance --> Chem[chemistry extension]
|
||||
Chem --> Mhchem[mhchemParser]
|
||||
Chem --> KaTeX
|
||||
Money --> MoneyLocales[money words]
|
||||
Money --> Numbers[spellNumber]
|
||||
Lower --> Render[render]
|
||||
Render --> Locales[locale packs]
|
||||
Eval[formula speech eval] --> Lower
|
||||
Eval --> CE[Compute Engine]
|
||||
```
|
||||
|
||||
## Affected files
|
||||
|
||||
```text
|
||||
packages/core-agent/src/voice/output/speech-transformation/
|
||||
index.ts `toSpeechText`, the host of the preprocessor plugins
|
||||
segmenter.ts `MarkdownSegmenter`, the safe ends of streamed markdown
|
||||
language.ts primary language subtags, and the language of the words around a formula
|
||||
markdown.ts code spans, and `\(...\)` and `\[...\]` delimiters
|
||||
preprocessor/types.ts plugin contracts
|
||||
preprocessor/math/ the mathSpeech instance, the math plugin, and bare TeX
|
||||
preprocessor/money/ the money plugin, prices, and money words
|
||||
packages/mathjson/src/index.ts types and helpers of the speech MathJSON
|
||||
packages/speech-transformer-math/src/
|
||||
speech.ts MathSpeech: locale packs, extensions, and the call stack
|
||||
extension.ts the extension contract
|
||||
parse.ts KaTeX parse, raw commands, unknown commands, fallback
|
||||
lower.ts KaTeX tree to MathJSON
|
||||
render.ts MathJSON to words
|
||||
locales/{zh,en,ja,ko}.ts locale packs, with their tests next to them
|
||||
packages/speech-transformer-chem/src/
|
||||
chem.ts mhchemParser to Chemistry nodes, and their reading
|
||||
index.ts the chemistry extension
|
||||
locales/{zh,en,ja,ko}.ts chemistry words and compound names, with their tests next to them
|
||||
evals/tts-speech-formula/
|
||||
eval/test-objects/rule/src/ structural comparison with Compute Engine
|
||||
eval/cases/math-bridge/ optional MathBridge cases
|
||||
```
|
||||
|
||||
## Sequence
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Reply as markdown reply
|
||||
participant Host as preprocessor host
|
||||
participant Math as math plugin
|
||||
participant KaTeX
|
||||
participant Lower
|
||||
participant Render
|
||||
Reply->>Host: toSpeechText(markdown, language)
|
||||
Host->>Host: prepare hooks
|
||||
Host->>Math: formula hook with the TeX and the sentence language
|
||||
Math->>KaTeX: __parse(tex)
|
||||
alt KaTeX cannot parse the source
|
||||
KaTeX-->>Math: ParseError
|
||||
Math->>Math: the source text without backslashes
|
||||
else
|
||||
KaTeX-->>Math: KaTeX tree, with unknown commands as text
|
||||
end
|
||||
Math->>Lower: lower(tree)
|
||||
Lower-->>Math: MathJSON
|
||||
Math->>Render: render(MathJSON, locale pack)
|
||||
Render-->>Host: words
|
||||
Host-->>Reply: spoken text
|
||||
```
|
||||
|
||||
## Measurements
|
||||
|
||||
Measured on 2026-10-20 with `vitest bench` on 14 formulas of the eval:
|
||||
|
||||
| Layer | Time for all formulas |
|
||||
| --- | --- |
|
||||
| Parse (KaTeX and normalize) | 0.25 ms |
|
||||
| Lower | 0.06 ms |
|
||||
| Render | 0.10 ms |
|
||||
| Whole pipeline | 0.52 ms |
|
||||
| The replaced hand-written parser and reader | 0.22 ms |
|
||||
|
||||
The pipeline is about 2.4 times slower than the replaced reader, about 37 µs for each formula. KaTeX parsing is the
|
||||
largest layer. A reply has few formulas, so the cost is small next to TTS.
|
||||
|
||||
Lowering was first the slowest layer, because each symbol scanned all alias groups. An index of the spellings made it
|
||||
eight times faster.
|
||||
|
||||
## Test plan
|
||||
|
||||
- Unit tests for each layer: parsing and fallback, lowering of each head, rendering of each head in each locale, and
|
||||
the host order of plugins.
|
||||
- The 188 existing speech tests stay unchanged as a regression net. A change of their expected text needs a reason in
|
||||
the commit.
|
||||
- The eval compares lowered MathJSON with Compute Engine on the standard heads, and lists the differences in a
|
||||
snapshot.
|
||||
- A benchmark measures the old reader and the new pipeline on the formulas of the eval, and reports each layer.
|
||||
- The full eval runs without TTS for every test object. Runs with TTS need account credit.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Endpoints and keys only. The matrix, the model names, and the voices are in
|
||||
# eval/tasks/read-aloud/config.ts, and the rewrite model is in eval/test-objects/llm/vieval.config.ts.
|
||||
|
||||
# The listener writes down what it heard as LaTeX. Any OpenAI-compatible endpoint works.
|
||||
LISTENER_BASE_URL='https://api.openai.com/v1/'
|
||||
LISTENER_API_KEY=''
|
||||
|
||||
# CLIProxyAPI as provider
|
||||
# LISTENER_BASE_URL='http://localhost:8317/v1'
|
||||
# LISTENER_API_KEY='local-key'
|
||||
|
||||
# The rewrite model of the `llm` test object.
|
||||
REWRITE_BASE_URL='https://api.openai.com/v1/'
|
||||
REWRITE_API_KEY=''
|
||||
|
||||
# CLIProxyAPI as provider
|
||||
# REWRITE_BASE_URL='http://localhost:8317/v1'
|
||||
# REWRITE_API_KEY='local-key'
|
||||
|
||||
# Speech recognition, needed when the `tts` axis has a provider.
|
||||
# A whisper.cpp server started with `--inference-path /v1/audio/transcriptions` also works.
|
||||
ASR_BASE_URL='https://api.openai.com/v1/'
|
||||
ASR_API_KEY=''
|
||||
|
||||
# Gemini as an ASR model of the `asr` axis. It uses the Gemini `generateContent` API, not an OpenAI-compatible endpoint.
|
||||
# Set the API root: Google itself, or a proxy route that forwards the Gemini API.
|
||||
ASR_GEMINI_BASE_URL='https://generativelanguage.googleapis.com/v1beta/'
|
||||
ASR_GEMINI_API_KEY=''
|
||||
|
||||
# One pair per TTS provider in the `tts` axis: TTS_<PROVIDER>_BASE_URL and TTS_<PROVIDER>_API_KEY.
|
||||
TTS_OFFICIAL_BASE_URL='https://api.airi.build/api/v1/'
|
||||
TTS_OFFICIAL_API_KEY=''
|
||||
|
||||
TTS_AZURE_BASE_URL='http://127.0.0.1:5933/v1/'
|
||||
TTS_AZURE_API_KEY=''
|
||||
|
||||
# Optional, for development: case ids and languages to run. Empty runs all 67 cases in all four languages.
|
||||
EVAL_CASES=''
|
||||
EVAL_LANGUAGES=''
|
||||
# Optional: ASR models to run instead of the default `asr` axis, for example `gemini-3.8-flash`.
|
||||
EVAL_ASR=''
|
||||
@@ -0,0 +1,200 @@
|
||||
# @proj-airi/evals-tts-speech-formula
|
||||
|
||||
A [vieval](https://vieval.dev/) comparison that measures how well a listener understands the formulas that AIRI reads aloud. It covers LaTeX math, chemistry, units, and dollar amounts in Chinese, English, Japanese, and Korean.
|
||||
|
||||
For each case, a test object turns a reply into spoken text. The text optionally goes through TTS and ASR. Then a listener model writes down what it heard as LaTeX, and a deterministic grader compares that answer with the reference. The listener model never grades.
|
||||
|
||||
## What the suite measures
|
||||
|
||||
AIRI reads a reply aloud to a person. The person must understand what the formula says, as from a teacher. The person
|
||||
does not write it down. So the goal of a reading is that it keeps the meaning and sounds natural:
|
||||
|
||||
- A reading that changes the meaning is wrong, for example `v - (x - \epsilon)` read as "v minus x minus epsilon".
|
||||
- A reading that spells markup is wrong, for example "open parenthesis x close parenthesis" for `f(x)`.
|
||||
- A reading does not need to name every detail that a dictation needs. A lone `\Sigma` can be "sigma", and a compound
|
||||
subscript does not need an end word.
|
||||
|
||||
The listener and the grader measure whether the meaning survives. They are a proxy: a listener model writes LaTeX, so
|
||||
`formula.correct` also rewards words that only help a dictation. Read `formula.correct` as the floor of meaning, and
|
||||
`formula.natural` as the reading that a person wants to hear. A Chinese, Japanese, or Korean equation reads its
|
||||
substances without coefficients, as a textbook does, so `formula.correct` is 0 for it although the listener heard it.
|
||||
|
||||
A judge model also reads the spoken text. It checks that the text says the meaning of the formula instead of spelling its markup. A listener model can write down "open parenthesis x close parenthesis" as easily as "of x", so correctness alone rewards spelled readings.
|
||||
|
||||
```text
|
||||
markdown -> test object -> spoken text -> [TTS -> ASR] -> listener model (3 votes) -> LaTeX -> grader -> formula.correct: 1 or 0
|
||||
\-> judge model (3 votes) -> natural, mixed, or spelled -> formula.natural: 1, 0.5, or 0
|
||||
```
|
||||
|
||||
The task records two named scores, and vieval derives the headline score from them:
|
||||
|
||||
| Score | Grader | Value |
|
||||
| --- | --- | --- |
|
||||
| `formula.correct` | listener votes and the grader | 1 when a strict majority writes the formula back correctly |
|
||||
| `formula.natural` | judge votes | 1 for natural, 0.5 for mixed, 0 for spelled, the median |
|
||||
| `formula.pass` (primary) | derived | 1 when the reading is correct and at least mixed |
|
||||
|
||||
Compare `formula.correct` for correctness and `formula.pass` for correct readings that a person wants to hear. Reports of vieval 0.9 and older cannot be read, so run them again.
|
||||
|
||||
## How to use it
|
||||
|
||||
1. Copy `.env.example` to `.env` and set the endpoints and keys. The file holds nothing else.
|
||||
2. Set the matrix in `eval/tasks/read-aloud/config.ts`: the `tts`, `asr`, and `listener` axes, the TTS models and their voices, and the languages.
|
||||
3. Compare all test objects:
|
||||
|
||||
```shell
|
||||
pnpm -F @proj-airi/evals-tts-speech-formula eval:compare
|
||||
```
|
||||
|
||||
The script builds `@proj-airi/core-agent` first, so the `rule` test object reads the code in your working tree. It writes the comparison to `.vieval/comparison.json`.
|
||||
|
||||
To run one test object and keep its full report, run its workspace config:
|
||||
|
||||
```shell
|
||||
pnpm -F @proj-airi/core-agent build
|
||||
cd evals/tts-speech-formula
|
||||
pnpm exec vieval run --config ./eval/test-objects/rule/vieval.config.ts --report-out .vieval/reports/rule
|
||||
```
|
||||
|
||||
Each case writes `spoken.txt` and `result.json` as artifacts. A case with a TTS provider also writes `speech.mp3`. To compare two retained reports case by case, use `vieval report compare <left> <right>`. Cases match by `benchmark.case.id`, which is `<case>|<language>|<tts>` and leaves the test object out.
|
||||
|
||||
To add a TTS provider, add a `speechProviderFrom` and a `speechModelFrom` entry to `modelPlugins` in `eval/tasks/read-aloud/config.ts`, add the model id to the `tts` axis, and set the provider env in `.env`. Every provider takes the same OpenAI-compatible `/audio/speech` request. Thus, the official AIRI provider, unspeech, and other services differ only in their endpoint, model, and voices. Set `sendsLanguage: true` on a model whose endpoint takes the language of the text in `extra_body.language`, as the official AIRI provider does. The clip then gets the BCP 47 tag of the case language, for example `en-US`, and the tag is part of its cache key.
|
||||
|
||||
## Structure
|
||||
|
||||
```text
|
||||
vieval.config.ts the comparison
|
||||
eval/
|
||||
cases/formula-speech/ the dataset: 70 cases and the four languages
|
||||
evaluators/
|
||||
speech/ TTS and ASR. `speech-models.ts` is a vieval plugin, see below
|
||||
listener/ the listener model: Velin prompt, votes, and cached answers
|
||||
judge/ the judge model: Velin prompt, votes, and cached verdicts of the reading style
|
||||
grader/ the deterministic grader and its unit tests
|
||||
tasks/read-aloud/ the shared task that runs one test object through the evaluators, and the shared config
|
||||
shared/ environment variables, cache keys, and Velin rendering
|
||||
test-objects/<object>/
|
||||
vieval.config.ts the workspace of the test object
|
||||
src/read-aloud.ts the adapter: reply in, spoken text out
|
||||
tasks/read-aloud/formula-speech/<language>.eval.ts
|
||||
index.ts the package entry that the test objects import
|
||||
```
|
||||
|
||||
| Vieval layer | In this suite |
|
||||
| --- | --- |
|
||||
| Comparison | `vieval.config.ts`, on the `formula-speech` dataset. |
|
||||
| Workspace | One test object under `eval/test-objects/`: `raw`, `rule`, `llm`, or `sre`. |
|
||||
| Project | One per test object, named `formula-speech-<object>`. |
|
||||
| Run matrix | `tts`: `none`, then TTS model ids. The `llm` object adds `rewrite`, its rewrite model. |
|
||||
| Eval matrix | `listener`, the listener model, `judge`, the judge model, and `asr`, the ASR model when `tts` has a provider. They are part of the measurement, not of the test object. |
|
||||
| Task | `read-aloud` in one language. During development, `EVAL_LANGUAGES=<language>,<language>` limits a run to some languages. |
|
||||
| Case | One of the 70 cases. During development, `EVAL_CASES=<id>,<id>` limits a run to some cases. |
|
||||
|
||||
The test objects are:
|
||||
|
||||
- `raw`: the markdown as the model wrote it. This is AIRI without formula speech.
|
||||
- `rule`: `toSpeechText` of `@proj-airi/core-agent`, the rule engine that AIRI ships.
|
||||
- `llm`: a model rewrites the reply for speech. It is a reference for the upper bound. It is too slow for the live path. Its rewrite prompt is in its own `src/`.
|
||||
- `sre`: speech-rule-engine ClearSpeak. It has no Chinese or Japanese locale, so those languages get the English reading.
|
||||
|
||||
The prompts are [Velin](https://github.com/moeru-ai/velin) markdown. Keep each LaTeX example of a prompt in a code span: Velin escapes backslashes and underscores in plain text.
|
||||
|
||||
To add a test object, add a directory under `eval/test-objects/` with a `ReadAloud` adapter, a config that calls `defineFormulaSpeechConfig`, and one task file per language. The comparison finds it automatically.
|
||||
|
||||
`tts: none` sends the spoken text straight to the listener. It measures only what AIRI hands to TTS, independent of the provider. This is the floor that AIRI controls. A real provider shows which parts of that text the provider then reads wrong.
|
||||
|
||||
## ASR
|
||||
|
||||
The `asr` axis selects the speech recognizer of every TTS clip. Two are registered:
|
||||
|
||||
- `apple-speech`: Apple Speech on macOS, through `@xsai-apple-speech/transcription-native`. It runs locally and needs no key. It uses one recognizer per language: `zh-CN`, `en-US`, `ja-JP`, or `ko-KR`. Install these locales in the macOS speech settings. It is the default, because `whisper-1` fell into repetition loops on short clips.
|
||||
- `whisper-1`: any OpenAI-compatible `/audio/transcriptions` endpoint, set with `ASR_BASE_URL` and `ASR_API_KEY`.
|
||||
- `gemini-3.8-flash`: Gemini through its `generateContent` API, set with `ASR_GEMINI_BASE_URL` and `ASR_GEMINI_API_KEY`. The base URL is the Gemini API root, for example `https://generativelanguage.googleapis.com/v1beta/`. Gemini has no `/audio/transcriptions` endpoint, so `evaluators/speech/gemini-transcription.ts` is a transcription executor that sends the clip as inline audio.
|
||||
|
||||
Put several in the `asr` axis to compare them on the same clips. `EVAL_ASR=apple-speech,gemini-3.8-flash` selects them for one run.
|
||||
|
||||
## Speech models plugin
|
||||
|
||||
`eval/evaluators/speech/speech-models.ts` registers TTS and ASR models the way `vieval/plugins/chat-models` registers chat models. It uses only the public vieval API, so it can move into vieval as `vieval/plugins/speech-models`.
|
||||
|
||||
| Chat models | Speech models |
|
||||
| --- | --- |
|
||||
| `ChatProviders`, `chatProviderFrom` | `SpeechProviders`, `speechProviderFrom` |
|
||||
| `ChatModels`, `chatModelFrom` | `SpeechModels`, `speechModelFrom`, `transcriptionModelFrom` |
|
||||
| `modelFromRun`, `modelFromEval` | `speechModelFromRun`, `speechModelFromEval`, `transcriptionModelFromRun`, `transcriptionModelFromEval` |
|
||||
| `openaiFromRunContext` | `openaiSpeechFromRunContext`, `openaiTranscriptionFromRunContext`, `voiceFor` |
|
||||
|
||||
The plugins register every declared model, and the matrix only selects them by id, as with chat models. The models go into the shared `config.models` registry. Provider endpoints and keys are optional at load time, so an unused provider needs no env. A selected model without its base URL fails when the task resolves it, and the error names its provider. `parameters.capability` is `speech` or `transcription`, and a lookup rejects a model with the wrong capability. A TTS model has a default `voice` and a `voices` map by language. A transcription provider can also be a `TranscriptionExecutor`: an object that returns xsai request options with its own `fetch` for each call, for example a native recognizer. Like the chat plugin, it does not send requests. The suite sends them with `@xsai/generate-speech` and `@xsai/generate-transcription`.
|
||||
|
||||
## Grading
|
||||
|
||||
| Mode | A case passes when |
|
||||
| --- | --- |
|
||||
| `symbolic` | The answer is equal after normalization, or symbolically equivalent with [Compute Engine](https://cortexjs.io/compute-engine/). `6/π²` does not pass for `π²/6`. Limits, sums, and integrals must match operator by operator. |
|
||||
| `norm` | The answer is equal after normalization. |
|
||||
| `chem` | The answer has the same species, coefficients, and arrow class. The order of terms is free. |
|
||||
| `currency` | The answer has the same dollar amounts and the same inline math. |
|
||||
|
||||
Speech cannot carry letter case, so the grader folds letter case. When the reference uses both cases of one letter, for example `a \in A`, the grader keeps case.
|
||||
|
||||
Each case gets three listener answers. The case passes with a strict majority.
|
||||
|
||||
Each answer also has a hearing status: `ok`, `partial`, or `unclear`. The listener sets it through structured output. An `unclear` vote scores 0 with the grading rule `unclear`, so a report tells an unintelligible clip apart from a wrong reading. The `tts_speech_formula.hearing` metric is the status of most votes.
|
||||
|
||||
Each case also gets three judge verdicts on the spoken text: `natural`, `mixed`, or `spelled`. The judge never sees the answer of the listener and never judges correctness. The case takes the median verdict as the `tts_speech_formula.reading` metric, and records its score as the vieval `judge` score. The judge reads the spoken text, not the ASR transcript, so ASR errors do not count against the test object.
|
||||
|
||||
### Judge calibration
|
||||
|
||||
A person labels a fixed set of readings, and the share of labels that the judge matches measures the judge. The set is
|
||||
`eval/evaluators/judge/calibration/calibration.json`: 15 readings for each language, spread over the test objects and
|
||||
the verdicts of the judge.
|
||||
|
||||
1. After a change of the judge prompt, run the eval on the cases of the set, then refresh the verdicts with
|
||||
`pnpm -F @proj-airi/evals-tts-speech-formula run judge:collect`. It keeps the set. `-- --pick` picks a new one.
|
||||
2. Label with `pnpm -F @proj-airi/evals-tts-speech-formula run judge:label`. It opens a local page on port 5299. Keys
|
||||
1, 2, and 3 choose natural, mixed, and spelled. The verdict of the judge shows after the label.
|
||||
|
||||
On 2026-10-20 the judge matched 31 of 43 labels (72%). After the prompt took the labels into account, it matched 34 (79%).
|
||||
|
||||
The `tts_speech_formula.leftover_notation` metric counts the symbols and LaTeX command names that a test object left in its spoken text, for example `^`, `/`, `$`, or `mapsto`. It needs no TTS. The test `eval/test-objects/rule/src/read-aloud.test.ts` lists these leftovers of the rule engine for every case, without any model call.
|
||||
|
||||
The test `eval/test-objects/rule/src/structure.test.ts` compares the MathJSON of the rule engine (`MathSpeech.toMathJson` of `@proj-airi/speech-transformer-math`) with Compute Engine on the standard heads, for example `Divide`, `Limit`, `D`, and `Interval`. Its snapshot lists each formula where the two trees differ. It also runs without any model call.
|
||||
|
||||
Answers are cached by model, prompt, transcript, and vote number. Thus, an unchanged transcript gets the same answer in every run, and run-to-run changes come only from the test object.
|
||||
|
||||
## History
|
||||
|
||||
The suite was first written in Python for Goal task T-157, with a sympy grader. The cached answers of the two full T-157 runs were graded again with this grader. It agrees with the sympy grader on 99.88% (edge-tts) and 99.94% (official StepFun) of the 3216 votes of each run. The remaining differences are these:
|
||||
|
||||
- `\sum (1/2)^k` against `\sum 1/2^k`: Compute Engine does not prove that the two summands are equal, so this grader fails the answer.
|
||||
- An answer with a trailing `?`: sympy cannot parse it, but Compute Engine ignores the character.
|
||||
|
||||
The prompts were also rewritten as structured Velin markdown. The scores are therefore not comparable with the T-157 baseline. Use a new baseline from this suite.
|
||||
|
||||
## Usage
|
||||
|
||||
The suite records what each run spends. It records only calls that reach a provider: a cached answer makes no call, so a rerun records almost nothing.
|
||||
|
||||
| Call | Record | Where |
|
||||
| --- | --- | --- |
|
||||
| Listener, judge, and `llm` rewrite | Token usage and latency, as vieval `inference.*` events | The inference panel of each case in the viewer, and `events.jsonl` |
|
||||
| TTS | `tts_speech_formula.tts.characters_sent` | Case metrics |
|
||||
| ASR | `tts_speech_formula.asr.requests` | Case metrics |
|
||||
|
||||
TTS and ASR use metrics because the vieval inference events accept only chat calls. The suite records no prices.
|
||||
|
||||
## When to use it
|
||||
|
||||
- Use it before and after you change `math-speech` or `toSpeechText` in `@proj-airi/core-agent`.
|
||||
- Use it to compare TTS providers on the same spoken text.
|
||||
|
||||
## When not to use it
|
||||
|
||||
- Do not use it as a unit test. It calls paid models. The unit tests of the rule engine are in `packages/core-agent`.
|
||||
- Do not use its score as a measure of human understanding. The listener is a model that reads an ASR transcript, not a person who hears audio.
|
||||
|
||||
## Known issues
|
||||
|
||||
- In vieval 0.4.0, two `describeTask` calls in one eval file get the same task id, and the second task overwrites the cases of the first. Thus, each language has its own task file.
|
||||
- `vieval compare` writes the report of each test object to a temporary directory. To keep the cases and artifacts of one test object, run its workspace config with `--report-out`.
|
||||
- Vieval loads config files with Node type stripping, which does not resolve a relative import without its extension. Thus, the test object configs import the shared config through the separate package export `@proj-airi/evals-tts-speech-formula/config`.
|
||||
@@ -0,0 +1,697 @@
|
||||
import type { Language } from './languages'
|
||||
|
||||
/**
|
||||
* How a case is graded. Each mode compares the reference with what a listener wrote down.
|
||||
*
|
||||
* - `symbolic`: equal after normalization, or equivalent by symbolic math. `6/π²` is not `π²/6`.
|
||||
* - `norm`: equal after normalization only. Use it when two different forms must not pass.
|
||||
* - `chem`: the same species, coefficients, and arrow class.
|
||||
* - `currency`: the same dollar amounts, and the same inline math.
|
||||
*/
|
||||
export type GradingMode = 'symbolic' | 'norm' | 'chem' | 'currency'
|
||||
|
||||
/** One expected item of a currency sentence: an amount in US dollars, or an inline formula. */
|
||||
export type CurrencyItem = readonly ['USD', number] | readonly ['MATH', string]
|
||||
|
||||
export interface FormulaSpeechCase {
|
||||
/** Stable id. Reports use it to match the same case across runs. */
|
||||
id: string
|
||||
/** The group of notation that the case covers, for example `limit` or `chem`. */
|
||||
stratum: string
|
||||
/** Where the case comes from: a Goal task, a textbook, or a dataset. */
|
||||
source: string
|
||||
mode: GradingMode
|
||||
/** Reference LaTeX, or the expected items of a currency sentence. */
|
||||
reference: string | readonly CurrencyItem[]
|
||||
/** The markdown that the methods get: one text for all languages, or one sentence per language. */
|
||||
input: string | Readonly<Record<Language, string>>
|
||||
/**
|
||||
* Overrides the letter case policy of the grader.
|
||||
*
|
||||
* @default derived from the reference: `sensitive` when it uses both cases of one letter
|
||||
*/
|
||||
letterCase?: 'fold' | 'sensitive'
|
||||
note: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The stratified regression set of formula speech, 70 cases in 12 strata.
|
||||
* It was first built for Goal task T-157 and is read in Chinese, English, Japanese, and Korean.
|
||||
*/
|
||||
export const formulaSpeechCases: readonly FormulaSpeechCase[] = [
|
||||
{
|
||||
id: 'lim-basel',
|
||||
stratum: 'limit',
|
||||
source: 'goal/T-150/T-151/T-153',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\lim_{n \to \infty}\sum_{i=1}^{n}\frac{1}{i^2}=\frac{\pi^2}{6}`,
|
||||
input: String.raw`$\lim_{n \to \infty}\sum_{i=1}^{n}\frac{1}{i^2}=\frac{\pi^2}{6}$`,
|
||||
note: '目标公式；6/π² 倒置陷阱',
|
||||
},
|
||||
{
|
||||
id: 'lim-sinx',
|
||||
stratum: 'limit',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\lim_{x \to 0}\frac{\sin x}{x}=1`,
|
||||
input: String.raw`$\lim_{x \to 0}\frac{\sin x}{x}=1$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'lim-e',
|
||||
stratum: 'limit',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\lim_{n \to \infty}\left(1+\frac{1}{n}\right)^{n}=e`,
|
||||
input: String.raw`$\lim_{n \to \infty}\left(1+\frac{1}{n}\right)^{n}=e$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'lim-recip',
|
||||
stratum: 'limit',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\lim_{x \to \infty}\frac{1}{x}=0`,
|
||||
input: String.raw`$\lim_{x \to \infty}\frac{1}{x}=0$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'lim-onesided',
|
||||
stratum: 'limit',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\lim_{x \to 0^{+}}\ln x=-\infty`,
|
||||
input: String.raw`$\lim_{x \to 0^{+}}\ln x=-\infty$`,
|
||||
note: '单侧极限 0^+',
|
||||
},
|
||||
{
|
||||
id: 'lim-fx-inf',
|
||||
stratum: 'limit',
|
||||
source: 'T-153',
|
||||
mode: 'norm',
|
||||
reference: String.raw`f(x) \to \infty`,
|
||||
input: String.raw`$f(x) \to \infty$`,
|
||||
note: '趋于 → 关系',
|
||||
},
|
||||
{
|
||||
id: 'sum-gauss',
|
||||
stratum: 'sum',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sum_{k=1}^{n} k=\frac{n(n+1)}{2}`,
|
||||
input: String.raw`$\sum_{k=1}^{n} k=\frac{n(n+1)}{2}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'sum-geo',
|
||||
stratum: 'sum',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sum_{k=0}^{\infty} \frac{1}{2^{k}}=2`,
|
||||
input: String.raw`$\sum_{k=0}^{\infty} \frac{1}{2^{k}}=2$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'sum-sq',
|
||||
stratum: 'sum',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sum_{i=1}^{n} i^{2}`,
|
||||
input: String.raw`$\sum_{i=1}^{n} i^{2}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'sum-mb-ux',
|
||||
stratum: 'sum',
|
||||
source: 'MathBridge(style)',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sum_{i=1}^{n} x_{i}^{2}`,
|
||||
input: String.raw`$\sum_{i=1}^{n} x_{i}^{2}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'sum-prod',
|
||||
stratum: 'sum',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\prod_{k=1}^{n} k=n!`,
|
||||
input: String.raw`$\prod_{k=1}^{n} k=n!$`,
|
||||
note: '连乘 ∏',
|
||||
},
|
||||
{
|
||||
id: 'frac-half',
|
||||
stratum: 'fraction',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\frac{1}{2}+\frac{1}{3}=\frac{5}{6}`,
|
||||
input: String.raw`$\frac{1}{2}+\frac{1}{3}=\frac{5}{6}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'frac-pi2-6',
|
||||
stratum: 'fraction',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\frac{\pi^2}{6}`,
|
||||
input: String.raw`$\frac{\pi^2}{6}$`,
|
||||
note: '分子分母次序陷阱（韩/日/中“分之”语序）',
|
||||
},
|
||||
{
|
||||
id: 'frac-6-pi2',
|
||||
stratum: 'fraction',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\frac{6}{\pi^2}`,
|
||||
input: String.raw`$\frac{6}{\pi^2}$`,
|
||||
note: '与上一条互为倒数，判分必须区分',
|
||||
},
|
||||
{
|
||||
id: 'frac-nested',
|
||||
stratum: 'fraction',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\frac{1}{1+\frac{1}{x}}`,
|
||||
input: String.raw`$\frac{1}{1+\frac{1}{x}}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'frac-mb-7-4',
|
||||
stratum: 'fraction',
|
||||
source: 'MathBridge#200057(简化)',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`q \geq \frac{7}{4}`,
|
||||
input: String.raw`$q \geq \frac{7}{4}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'frac-mb-sigma',
|
||||
stratum: 'fraction',
|
||||
source: 'MathBridge#3000060(简化)',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sigma_{0}=\frac{f-1}{2}`,
|
||||
input: String.raw`$\sigma_{0}=\frac{f-1}{2}$`,
|
||||
note: '(f-1)/2 vs f-1/2 分组',
|
||||
},
|
||||
{
|
||||
id: 'root-quadratic',
|
||||
stratum: 'root',
|
||||
source: 'T-153',
|
||||
mode: 'norm',
|
||||
reference: String.raw`x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}`,
|
||||
input: String.raw`$x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}$`,
|
||||
note: '±',
|
||||
},
|
||||
{
|
||||
id: 'root-sqrt2',
|
||||
stratum: 'root',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\sqrt{2}\approx 1.414`,
|
||||
input: String.raw`$\sqrt{2}\approx 1.414$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'root-cube',
|
||||
stratum: 'root',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sqrt[3]{8}=2`,
|
||||
input: String.raw`$\sqrt[3]{8}=2$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'root-sum',
|
||||
stratum: 'root',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sqrt{a^2+b^2}`,
|
||||
input: String.raw`$\sqrt{a^2+b^2}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'root-mb-s',
|
||||
stratum: 'root',
|
||||
source: 'MathBridge#1200092',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\sqrt{s}=14`,
|
||||
input: String.raw`$\sqrt{s}=14$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-perp',
|
||||
stratum: 'relation',
|
||||
source: 'T-150/T-153',
|
||||
mode: 'norm',
|
||||
reference: String.raw`AB \perp CD`,
|
||||
input: String.raw`$AB \perp CD$`,
|
||||
note: '垂直',
|
||||
},
|
||||
{
|
||||
id: 'rel-parallel',
|
||||
stratum: 'relation',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'norm',
|
||||
reference: String.raw`l_1 \parallel l_2`,
|
||||
input: String.raw`$l_1 \parallel l_2$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-leq',
|
||||
stratum: 'relation',
|
||||
source: 'T-153',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`x \leq 3`,
|
||||
input: String.raw`$x \leq 3$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-geq',
|
||||
stratum: 'relation',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`|x| \geq 0`,
|
||||
input: String.raw`$|x| \geq 0$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-neq',
|
||||
stratum: 'relation',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`a \neq b`,
|
||||
input: String.raw`$a \neq b$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-approx',
|
||||
stratum: 'relation',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\pi \approx 3.14`,
|
||||
input: String.raw`$\pi \approx 3.14$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-sim',
|
||||
stratum: 'relation',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\triangle ABC \sim \triangle DEF`,
|
||||
input: String.raw`$\triangle ABC \sim \triangle DEF$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-mb-chain',
|
||||
stratum: 'relation',
|
||||
source: 'MathBridge#600008',
|
||||
mode: 'norm',
|
||||
reference: String.raw`x_1 \leq \cdots \leq x_n`,
|
||||
input: String.raw`$x_1 \leq \cdots \leq x_n$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'rel-angle',
|
||||
stratum: 'relation',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\angle ABC=90^\circ`,
|
||||
input: String.raw`$\angle ABC=90^\circ$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'set-in',
|
||||
stratum: 'set',
|
||||
source: 'T-153',
|
||||
mode: 'norm',
|
||||
reference: String.raw`x \in A`,
|
||||
input: String.raw`$x \in A$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'set-notin',
|
||||
stratum: 'set',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\sqrt{2} \notin \mathbb{Q}`,
|
||||
input: String.raw`$\sqrt{2} \notin \mathbb{Q}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'set-subseteq',
|
||||
stratum: 'set',
|
||||
source: 'T-153',
|
||||
mode: 'norm',
|
||||
reference: String.raw`A \subseteq B`,
|
||||
input: String.raw`$A \subseteq B$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'set-cup',
|
||||
stratum: 'set',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`A \cup B=\{1,2,3\}`,
|
||||
input: String.raw`$A \cup B=\{1,2,3\}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'set-cap-empty',
|
||||
stratum: 'set',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`A \cap B=\varnothing`,
|
||||
input: String.raw`$A \cap B=\varnothing$`,
|
||||
note: String.raw`空集写法 \emptyset/\varnothing 等价`,
|
||||
},
|
||||
{
|
||||
id: 'set-interval',
|
||||
stratum: 'set',
|
||||
source: 'MathBridge#3000013',
|
||||
mode: 'norm',
|
||||
reference: String.raw`x \in (0, b]`,
|
||||
input: String.raw`$x \in (0, b]$`,
|
||||
note: '半开区间',
|
||||
},
|
||||
{
|
||||
id: 'set-builder',
|
||||
stratum: 'set',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\{x \mid x > 0\}`,
|
||||
input: String.raw`$\{x \mid x > 0\}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'chem-water',
|
||||
stratum: 'chem',
|
||||
source: 'T-150/T-153',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{2H2 + O2 -> 2H2O}`,
|
||||
input: String.raw`$\ce{2H2 + O2 -> 2H2O}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'chem-co2',
|
||||
stratum: 'chem',
|
||||
source: 'textbook',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{C + O2 -> CO2}`,
|
||||
input: String.raw`$\ce{C + O2 -> CO2}$`,
|
||||
note: 'CO 与 CO2 必须区分',
|
||||
},
|
||||
{
|
||||
id: 'chem-co',
|
||||
stratum: 'chem',
|
||||
source: 'textbook',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{2C + O2 -> 2CO}`,
|
||||
input: String.raw`$\ce{2C + O2 -> 2CO}$`,
|
||||
note: '与上一条对照',
|
||||
},
|
||||
{
|
||||
id: 'chem-h2o2',
|
||||
stratum: 'chem',
|
||||
source: 'textbook',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{2H2O2 -> 2H2O + O2}`,
|
||||
input: String.raw`$\ce{2H2O2 -> 2H2O + O2}$`,
|
||||
note: 'H2O2 vs H2O',
|
||||
},
|
||||
{
|
||||
id: 'chem-lime',
|
||||
stratum: 'chem',
|
||||
source: 'textbook',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{CaCO3 -> CaO + CO2}`,
|
||||
input: String.raw`$\ce{CaCO3 -> CaO + CO2}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'chem-ca-oh',
|
||||
stratum: 'chem',
|
||||
source: 'T-156',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{Ca(OH)2 + CO2 -> CaCO3 + H2O}`,
|
||||
input: String.raw`$\ce{Ca(OH)2 + CO2 -> CaCO3 + H2O}$`,
|
||||
note: '括号原子团',
|
||||
},
|
||||
{
|
||||
id: 'chem-haber',
|
||||
stratum: 'chem',
|
||||
source: 'textbook',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{N2 + 3H2 <=> 2NH3}`,
|
||||
input: String.raw`$\ce{N2 + 3H2 <=> 2NH3}$`,
|
||||
note: '可逆箭头',
|
||||
},
|
||||
{
|
||||
id: 'chem-hydrate',
|
||||
stratum: 'chem',
|
||||
source: 'T-150',
|
||||
mode: 'chem',
|
||||
reference: String.raw`\ce{CuSO4*5H2O}`,
|
||||
input: String.raw`$\ce{CuSO4*5H2O}$`,
|
||||
note: '结晶水',
|
||||
},
|
||||
{
|
||||
id: 'unit-g',
|
||||
stratum: 'unit',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`9.8\,\mathrm{m/s^2}`,
|
||||
input: String.raw`$9.8\,\mathrm{m/s^2}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'unit-c',
|
||||
stratum: 'unit',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`3\times10^{8}\,\mathrm{m/s}`,
|
||||
input: String.raw`$3\times10^{8}\,\mathrm{m/s}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'unit-temp',
|
||||
stratum: 'unit',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`25^\circ\mathrm{C}`,
|
||||
input: String.raw`$25^\circ\mathrm{C}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'unit-kg',
|
||||
stratum: 'unit',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`5\,\mathrm{kg}`,
|
||||
input: String.raw`$5\,\mathrm{kg}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'unit-pct',
|
||||
stratum: 'unit',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'norm',
|
||||
reference: String.raw`50\%`,
|
||||
input: String.raw`$50\%$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'cur-two',
|
||||
stratum: 'currency',
|
||||
source: 'textbook',
|
||||
mode: 'currency',
|
||||
reference: [['USD', 3], ['USD', 5]],
|
||||
input: {
|
||||
zh: '苹果 $3，香蕉 $5。',
|
||||
en: 'Apples cost $3 and bananas cost $5.',
|
||||
ja: 'りんごは $3、バナナは $5 です。',
|
||||
ko: '사과는 $3, 바나나는 $5입니다.',
|
||||
},
|
||||
note: '两个 $ 会被误配成一对数学定界符',
|
||||
},
|
||||
{
|
||||
id: 'cur-decimal',
|
||||
stratum: 'currency',
|
||||
source: 'textbook',
|
||||
mode: 'currency',
|
||||
reference: [['USD', 19.99]],
|
||||
input: {
|
||||
zh: '这本书售价 $19.99。',
|
||||
en: 'The book costs $19.99.',
|
||||
ja: 'この本は $19.99 です。',
|
||||
ko: '이 책은 $19.99입니다.',
|
||||
},
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'cur-range',
|
||||
stratum: 'currency',
|
||||
source: 'textbook',
|
||||
mode: 'currency',
|
||||
reference: [['USD', 10], ['USD', 20]],
|
||||
input: {
|
||||
zh: '预算在 $10 到 $20 之间。',
|
||||
en: 'The budget is between $10 and $20.',
|
||||
ja: '予算は $10 から $20 の間です。',
|
||||
ko: '예산은 $10에서 $20 사이입니다.',
|
||||
},
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'cur-thousand',
|
||||
stratum: 'currency',
|
||||
source: 'textbook',
|
||||
mode: 'currency',
|
||||
reference: [['USD', 1200]],
|
||||
input: {
|
||||
zh: '月租 $1,200。',
|
||||
en: 'The rent is $1,200 a month.',
|
||||
ja: '家賃は月 $1,200 です。',
|
||||
ko: '월세는 $1,200입니다.',
|
||||
},
|
||||
note: '千分位逗号',
|
||||
},
|
||||
{
|
||||
id: 'cur-mixed',
|
||||
stratum: 'currency',
|
||||
source: 'textbook',
|
||||
mode: 'currency',
|
||||
reference: [['USD', 5], ['MATH', 'x^2']],
|
||||
input: {
|
||||
zh: '门票 $5，面积是 $x^2$。',
|
||||
en: 'A ticket is $5, and the area is $x^2$.',
|
||||
ja: 'チケットは $5、面積は $x^2$ です。',
|
||||
ko: '입장권은 $5이고 넓이는 $x^2$입니다.',
|
||||
},
|
||||
note: '货币与数学定界符混排',
|
||||
},
|
||||
{
|
||||
id: 'case-fold-tri',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\triangle ABC`,
|
||||
input: String.raw`$\triangle ABC$`,
|
||||
note: '全大写，语音无法区分大小写 → 折叠',
|
||||
},
|
||||
{
|
||||
id: 'case-fold-ema',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: 'E=mc^2',
|
||||
input: '$E=mc^2$',
|
||||
note: 'E/m/c 各不同字母 → 折叠',
|
||||
},
|
||||
{
|
||||
id: 'case-fold-fma',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: 'F=ma',
|
||||
input: '$F=ma$',
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'case-sens-aA',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`a \in A`,
|
||||
input: String.raw`$a \in A$`,
|
||||
note: '同一字母大小写同时出现 → 必须区分（需读出“大写”）',
|
||||
},
|
||||
{
|
||||
id: 'case-sens-xX',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`f: X \to Y, x \mapsto y`,
|
||||
input: String.raw`$f: X \to Y, x \mapsto y$`,
|
||||
note: 'x/X、y/Y 同时出现 → 必须区分',
|
||||
},
|
||||
{
|
||||
id: 'case-sens-nN',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'symbolic',
|
||||
reference: 'N=n+1',
|
||||
input: '$N=n+1$',
|
||||
note: 'N 与 n 同时出现 → 必须区分',
|
||||
},
|
||||
{
|
||||
id: 'case-sens-omega',
|
||||
stratum: 'case',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\Omega \neq \omega`,
|
||||
input: String.raw`$\Omega \neq \omega$`,
|
||||
note: '希腊字母大小写同时出现 → 必须区分',
|
||||
},
|
||||
{
|
||||
id: 'misc-int',
|
||||
stratum: 'misc',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\int_0^1 x^2\,dx=\frac{1}{3}`,
|
||||
input: String.raw`$\int_0^1 x^2\,dx=\frac{1}{3}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'misc-deriv',
|
||||
stratum: 'misc',
|
||||
source: 'textbook',
|
||||
mode: 'norm',
|
||||
reference: String.raw`\frac{d}{dx}x^{n}=nx^{n-1}`,
|
||||
input: String.raw`$\frac{d}{dx}x^{n}=nx^{n-1}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'misc-neg-pow',
|
||||
stratum: 'misc',
|
||||
source: 'airi math-speech test',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`x^{-1}=\frac{1}{x}`,
|
||||
input: String.raw`$x^{-1}=\frac{1}{x}$`,
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'misc-binom',
|
||||
stratum: 'misc',
|
||||
source: 'T-156',
|
||||
mode: 'symbolic',
|
||||
reference: '(a+b)^2=a^2+2ab+b^2',
|
||||
input: '$(a+b)^2=a^2+2ab+b^2$',
|
||||
note: '',
|
||||
},
|
||||
{
|
||||
id: 'prob-fm-straight',
|
||||
stratum: 'probability',
|
||||
source: 'AIRI reply, 2026-10-20 (straight-path Flow Matching)',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`x_t=(1-t)\epsilon+tx`,
|
||||
input: String.raw`$$x_t=(1-t)\epsilon+tx$$`,
|
||||
note: 'The brackets of (1-t) bind it to epsilon. Read silent, the formula became 1 - t epsilon.',
|
||||
},
|
||||
{
|
||||
id: 'prob-fm-target',
|
||||
stratum: 'probability',
|
||||
source: 'AIRI reply, 2026-10-20 (straight-path Flow Matching)',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\mathcal{L}=\mathbb{E}\left[\left\|v_\theta(x_t,t)-(x-\epsilon)\right\|^2\right]`,
|
||||
input: String.raw`$$\mathcal{L}=\mathbb{E}\left[\left\|v_\theta(x_t,t)-(x-\epsilon)\right\|^2\right]$$`,
|
||||
note: 'The brackets after the minus decide the sign of epsilon. Read silent, the formula became v - x - epsilon.',
|
||||
},
|
||||
{
|
||||
id: 'prob-fm-loss',
|
||||
stratum: 'probability',
|
||||
source: 'AIRI reply, 2026-10-20 (Flow Matching, arXiv 2210.02747 eq. 5)',
|
||||
mode: 'symbolic',
|
||||
reference: String.raw`\mathcal{L}_{FM}(\theta) = \mathbb{E}_{t \sim \mathcal{U}[0,1],\, x \sim p_t}\left[\|v_\theta(x,t) - u_t(x)\|^2\right]`,
|
||||
input: String.raw`$$\mathcal{L}_{FM}(\theta) = \mathbb{E}_{t \sim \mathcal{U}[0,1],\, x \sim p_t}\left[\|v_\theta(x,t) - u_t(x)\|^2\right]$$`,
|
||||
note: '期望、范数与服从：方括号和范数线曾被读成"方括号""括号"，\\sim 曾被读成"相似于"',
|
||||
},
|
||||
]
|
||||
@@ -0,0 +1,2 @@
|
||||
export * from './cases'
|
||||
export * from './languages'
|
||||
@@ -0,0 +1,24 @@
|
||||
/** The languages that the suite reads every case in. */
|
||||
export const languages = [
|
||||
'zh',
|
||||
'en',
|
||||
'ja',
|
||||
'ko',
|
||||
] as const
|
||||
|
||||
export type Language = typeof languages[number]
|
||||
|
||||
/** BCP 47 code that AIRI uses for the voice of each language. */
|
||||
export const voiceLanguage: Readonly<Record<Language, string>> = {
|
||||
zh: 'zh-CN',
|
||||
en: 'en-US',
|
||||
ja: 'ja-JP',
|
||||
ko: 'ko-KR',
|
||||
}
|
||||
|
||||
export const languageName: Readonly<Record<Language, string>> = {
|
||||
zh: 'Mandarin Chinese',
|
||||
en: 'English',
|
||||
ja: 'Japanese',
|
||||
ko: 'Korean',
|
||||
}
|
||||
@@ -0,0 +1,207 @@
|
||||
import process from 'node:process'
|
||||
|
||||
import { createHash } from 'node:crypto'
|
||||
import { createReadStream, createWriteStream } from 'node:fs'
|
||||
import { access, mkdir, rename, writeFile } from 'node:fs/promises'
|
||||
import { dirname } from 'node:path'
|
||||
import { pipeline } from 'node:stream/promises'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import katex from 'katex'
|
||||
|
||||
import { DuckDBDataProtocol } from '@duckdb/duckdb-wasm'
|
||||
import { connect } from '@proj-airi/duckdb-wasm'
|
||||
import { getBundles } from '@proj-airi/duckdb-wasm/bundles/default-node'
|
||||
|
||||
import 'katex/contrib/mhchem'
|
||||
|
||||
/**
|
||||
* Samples MathBridge (https://huggingface.co/datasets/Kyudan/MathBridge, MIT license) into `sample.json`.
|
||||
*
|
||||
* The dataset is one parquet file of 23 million rows. The script downloads the file at a pinned revision into the
|
||||
* ignored `.cache/` folder, checks its SHA-256, and queries it with DuckDB. The same revision and seed always give the
|
||||
* same sample. Only a new sample needs the file: the eval reads the committed `sample.json`.
|
||||
*
|
||||
* - The `uniform` stratum takes {@link uniformCases} gradable formulas in a seeded random order over all rows.
|
||||
* - Each topic stratum takes {@link topicCases} formulas with one kind of notation, for example an expectation or a
|
||||
* norm. Uniform rows rarely hold them, so the rules for them would have no cases without these strata.
|
||||
* - A formula longer than {@link shortLength} characters gets a `-long` suffix, so a report can tell them apart.
|
||||
*
|
||||
* Run: pnpm -F @proj-airi/evals-tts-speech-formula run cases:math-bridge
|
||||
*
|
||||
* Call stack:
|
||||
*
|
||||
* main
|
||||
* -> download, once per revision, then verify
|
||||
* -> connect (@proj-airi/duckdb-wasm), registerFileURL with NODE_FS
|
||||
* -> uniform query and topic query, see {@link gradableRows}
|
||||
* -> sample.json
|
||||
*/
|
||||
|
||||
const dataset = 'Kyudan/MathBridge'
|
||||
/** The commit of the dataset repository. Change it, and {@link sha256}, to sample a newer release. */
|
||||
const revision = '353fb2255f6959ad2940865346c38689cfce902a'
|
||||
/** The SHA-256 of `train.parquet` at {@link revision}, as the `x-linked-etag` header of the LFS file reports it. */
|
||||
const sha256 = '49f5c87a7086de1130842fc80aab7bb11218bf020bf3b4e937682714a1c68b57'
|
||||
const fileUrl = `https://huggingface.co/datasets/${dataset}/resolve/${revision}/train.parquet`
|
||||
const cacheFile = fileURLToPath(new URL(`../../../.cache/mathbridge/train-${revision}.parquet`, import.meta.url))
|
||||
|
||||
/** The seed of the sample order. Change it to draw another sample. */
|
||||
const seed = 20261020
|
||||
/** Cases of the uniform stratum, so a run costs a known number of listener calls. */
|
||||
const uniformCases = 120
|
||||
/** Cases of each topic stratum. */
|
||||
const topicCases = 12
|
||||
/**
|
||||
* Candidates per case that the query returns. Some candidates use macros of their paper and are dropped, see
|
||||
* {@link usesStandardTex}, so the query returns more than it keeps.
|
||||
*/
|
||||
const candidateFactor = 4
|
||||
/** Longer formulas get a `-long` suffix. They are kept up to {@link maxLength}, because real replies have them. */
|
||||
const shortLength = 80
|
||||
const maxLength = 200
|
||||
|
||||
/**
|
||||
* Notation that the uniform rows rarely hold, by stratum name. A formula goes to the first topic that matches.
|
||||
* The patterns are RE2, the regular expressions of DuckDB. MathBridge writes a space after each token, so they allow
|
||||
* spaces inside braces.
|
||||
*/
|
||||
const topics: readonly (readonly [string, string])[] = [
|
||||
['expectation', String.raw`\\mathbb\s*\{\s*E\s*\}|\\mathbb\s+E\b|\\operatorname\s*\{\s*E\s*\}`],
|
||||
['variance', String.raw`\\(operatorname|mathrm)\s*\{\s*(Var|Cov)\s*\}|\\Pr\b|\\mathbb\s*\{\s*P\s*\}`],
|
||||
['norm', String.raw`\\\||\\[lr]?Vert\b`],
|
||||
['distribution', String.raw`\\sim\b`],
|
||||
['calligraphic-call', String.raw`\\mathcal\s*\{\s*[A-Z]\s*\}(\s*_\s*\{[^{}]*\})?(\s*\\left)?\s*\(`],
|
||||
]
|
||||
|
||||
/** One row of `sample.json`. ./index reads the same shape. */
|
||||
interface MathBridgeRow {
|
||||
/** `uniform`, or a topic of {@link topics}. A `-long` suffix marks a formula longer than {@link shortLength}. */
|
||||
stratum: string
|
||||
offset: number
|
||||
contextBefore: string
|
||||
equation: string
|
||||
contextAfter: string
|
||||
spokenEnglish: string
|
||||
}
|
||||
|
||||
/** Downloads the pinned file once. A partial download goes to a temporary name, so a broken run leaves no bad cache. */
|
||||
async function download(): Promise<void> {
|
||||
const exists = await access(cacheFile).then(() => true, () => false)
|
||||
if (!exists) {
|
||||
process.stdout.write(`downloading ${fileUrl}\n`)
|
||||
await mkdir(dirname(cacheFile), { recursive: true })
|
||||
const response = await fetch(fileUrl)
|
||||
if (!response.ok || !response.body)
|
||||
throw new Error(`MathBridge download: ${response.status}`)
|
||||
const partial = `${cacheFile}.partial`
|
||||
await pipeline(response.body, createWriteStream(partial))
|
||||
await rename(partial, cacheFile)
|
||||
}
|
||||
|
||||
const hash = createHash('sha256')
|
||||
await pipeline(createReadStream(cacheFile), hash)
|
||||
const actual = hash.digest('hex')
|
||||
if (actual !== sha256)
|
||||
throw new Error(`MathBridge file ${cacheFile} has SHA-256 ${actual}, expected ${sha256}. Delete it and run again.`)
|
||||
}
|
||||
|
||||
/**
|
||||
* The SQL of the rows that a listener can grade: notation without prose, a size that fits one spoken sentence, and a
|
||||
* sentence before it. `priority` is the DuckDB hash of the row index and the seed. Sorting by it gives a random order
|
||||
* that does not depend on the order of the file. The catalog pins the DuckDB version, so the same seed repeats it.
|
||||
* About 88% of the first 16 million rows hold notation, but under 1% of the rest do: they are single letters, for
|
||||
* example `$ A $`. Gradable rows therefore come from the first 16 million rows, and that is the data, not the hash.
|
||||
*/
|
||||
function gradableRows(): string {
|
||||
return `
|
||||
WITH source AS (
|
||||
SELECT
|
||||
file_row_number AS row_offset,
|
||||
trim(regexp_replace(equation, '^\\s*\\$+|\\$+\\s*$', '', 'g')) AS tex,
|
||||
equation, context_before, context_after, "spoken_English" AS spoken_english
|
||||
FROM read_parquet('mathbridge.parquet', file_row_number = true)
|
||||
)
|
||||
SELECT *, hash(row_offset, ${seed}) AS priority
|
||||
FROM source
|
||||
WHERE length(tex) BETWEEN 6 AND ${maxLength}
|
||||
AND regexp_matches(tex, '[\\\\^_=<>]')
|
||||
AND NOT regexp_matches(tex, '\\\\(text|mbox|label|ref|cite)|\\\\mathrm\\{[a-z]{4,}')
|
||||
AND length(context_before) > 0`
|
||||
}
|
||||
|
||||
/**
|
||||
* True when KaTeX renders the formula, so it uses only standard commands. A formula with a macro that its paper
|
||||
* defines, for example `\\bxi` or `\\NN`, cannot be read by anyone without the paper, and a reply never writes it.
|
||||
*/
|
||||
function usesStandardTex(tex: string): boolean {
|
||||
try {
|
||||
katex.renderToString(tex, { throwOnError: true, strict: 'ignore' })
|
||||
return true
|
||||
}
|
||||
catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/** Escapes a value for a single-quoted SQL string. The patterns are constants, so this only doubles quotes. */
|
||||
function sqlString(value: string): string {
|
||||
return `'${value.replaceAll('\'', '\'\'')}'`
|
||||
}
|
||||
|
||||
async function main() {
|
||||
await download()
|
||||
|
||||
const client = await connect({ bundles: getBundles() })
|
||||
try {
|
||||
// NODE_FS reads the file from disk in pieces, so DuckDB never holds the whole gigabyte in WebAssembly memory.
|
||||
await client.db.registerFileURL('mathbridge.parquet', cacheFile, DuckDBDataProtocol.NODE_FS, false)
|
||||
|
||||
const topicCase = topics.map(([name, pattern]) => `WHEN regexp_matches(tex, ${sqlString(pattern)}) THEN ${sqlString(name)}`).join(' ')
|
||||
const toRow = (row: Record<string, unknown>, stratum: string): MathBridgeRow => ({
|
||||
stratum: String(row.tex).length > shortLength ? `${stratum}-long` : stratum,
|
||||
offset: Number(row.row_offset),
|
||||
contextBefore: String(row.context_before ?? ''),
|
||||
equation: String(row.equation ?? ''),
|
||||
contextAfter: String(row.context_after ?? ''),
|
||||
spokenEnglish: String(row.spoken_english ?? ''),
|
||||
})
|
||||
|
||||
const uniform = await client.query(`${gradableRows()} ORDER BY priority LIMIT ${uniformCases * candidateFactor}`)
|
||||
const byTopic = await client.query(`
|
||||
SELECT * FROM (SELECT *, CASE ${topicCase} END AS topic FROM (${gradableRows()}))
|
||||
WHERE topic IS NOT NULL
|
||||
QUALIFY row_number() OVER (PARTITION BY topic ORDER BY priority) <= ${topicCases * candidateFactor}
|
||||
ORDER BY topic, priority`)
|
||||
|
||||
// Candidates come in the seeded order. Each stratum takes the first standard formulas up to its quota. A formula
|
||||
// in both strata stays in the uniform one, so the uniform stratum keeps its size.
|
||||
const kept = new Set<number>()
|
||||
const sample: MathBridgeRow[] = []
|
||||
const taken = new Map<string, number>()
|
||||
let dropped = 0
|
||||
for (const [rows, stratumOf, quota] of [[uniform, () => 'uniform', uniformCases], [byTopic, (row: Record<string, unknown>) => String(row.topic), topicCases]] as const) {
|
||||
for (const row of rows as Record<string, unknown>[]) {
|
||||
const stratum = stratumOf(row)
|
||||
if ((taken.get(stratum) ?? 0) >= quota || kept.has(Number(row.row_offset)))
|
||||
continue
|
||||
if (!usesStandardTex(String(row.tex))) {
|
||||
dropped++
|
||||
continue
|
||||
}
|
||||
kept.add(Number(row.row_offset))
|
||||
taken.set(stratum, (taken.get(stratum) ?? 0) + 1)
|
||||
sample.push(toRow(row, stratum))
|
||||
}
|
||||
}
|
||||
|
||||
await writeFile(new URL('./sample.json', import.meta.url), `${JSON.stringify({ source: `https://huggingface.co/datasets/${dataset}`, license: 'MIT', revision, seed, rows: sample }, null, 2)}\n`)
|
||||
const counts = Object.entries(Object.groupBy(sample, row => row.stratum)).map(([name, rows]) => `${name} ${rows?.length ?? 0}`)
|
||||
process.stdout.write(`kept ${sample.length} rows: ${counts.join(', ')}. Dropped ${dropped} with macros of their paper.\n`)
|
||||
}
|
||||
finally {
|
||||
await client.close()
|
||||
}
|
||||
}
|
||||
|
||||
void main()
|
||||
@@ -0,0 +1,30 @@
|
||||
import { toSpeechText } from '@proj-airi/core-agent'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { mathBridgeCases } from '.'
|
||||
import { leftoverNotation } from '../../evaluators/notation'
|
||||
|
||||
describe('mathBridgeCases', () => {
|
||||
it('loads the committed sample with unique ids', () => {
|
||||
expect(mathBridgeCases.length).toBeGreaterThan(0)
|
||||
expect(new Set(mathBridgeCases.map(item => item.id)).size).toBe(mathBridgeCases.length)
|
||||
})
|
||||
|
||||
// Runs the rule engine on arXiv notation without any model call. The snapshot lists what it still leaves as
|
||||
// notation. `\left\ {` and `\deltat` are artifacts of the MathBridge tokenizer, not notation that a reply writes.
|
||||
it('lists the notation that the rule engine leaves in arXiv formulas', () => {
|
||||
const leftovers: Record<string, string> = {}
|
||||
for (const item of mathBridgeCases) {
|
||||
const markdown = item.input as string
|
||||
const found = leftoverNotation(toSpeechText(markdown, { language: 'en-US' }), markdown)
|
||||
if (found.length > 0)
|
||||
leftovers[item.id] = `${item.reference as string} => ${found.join(' ')}`
|
||||
}
|
||||
|
||||
expect(leftovers).toMatchInlineSnapshot(`
|
||||
{
|
||||
"mb-4602321": "\\mathbb { P } \\Gamma\\backslash \\mathbb { CH } ^9 =M => backslash",
|
||||
}
|
||||
`)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,44 @@
|
||||
import type { FormulaSpeechCase } from '../formula-speech/cases'
|
||||
|
||||
import { readFileSync } from 'node:fs'
|
||||
|
||||
/** One row of `sample.json`, written by ./fetch. */
|
||||
interface MathBridgeRow {
|
||||
/** `uniform`, or a topic such as `expectation`. A `-long` suffix marks a formula longer than 80 characters. */
|
||||
stratum: string
|
||||
offset: number
|
||||
contextBefore: string
|
||||
equation: string
|
||||
contextAfter: string
|
||||
spokenEnglish: string
|
||||
}
|
||||
|
||||
const sample = JSON.parse(readFileSync(new URL('./sample.json', import.meta.url), 'utf8')) as { rows: MathBridgeRow[] }
|
||||
|
||||
/**
|
||||
* Repairs what the MathBridge tokenizer split. It writes `\{` as `\ {` and `\%` as `\ %`. In TeX `\ ` is a space and
|
||||
* `%` starts a comment, so `68.3\ %` loses its percent sign and `\ { 1, 2 \ }` loses its set braces.
|
||||
*/
|
||||
function untokenize(tex: string): string {
|
||||
return tex.replace(/\\ ([{}%])/g, '\\$1')
|
||||
}
|
||||
|
||||
/**
|
||||
* Formulas of MathBridge (MIT license), a sample of formulas from arXiv papers of 2023 and open textbooks.
|
||||
*
|
||||
* The cases add realistic notation to the textbook set. Each input is the formula alone, because the listener writes
|
||||
* down one formula and the grader compares formulas. The sentence around it stays in `note`. The dataset is English,
|
||||
* so a run selects it with `EVAL_DATASET=math-bridge EVAL_LANGUAGES=en`.
|
||||
*/
|
||||
export const mathBridgeCases: readonly FormulaSpeechCase[] = sample.rows.map((row) => {
|
||||
const tex = untokenize(row.equation.replace(/^\s*\$+|\$+\s*$/g, '').trim())
|
||||
return {
|
||||
id: `mb-${row.offset}`,
|
||||
stratum: `math-bridge-${row.stratum}`,
|
||||
source: `MathBridge#${row.offset}`,
|
||||
mode: 'symbolic',
|
||||
reference: tex,
|
||||
input: `$${tex}$`,
|
||||
note: `${row.contextBefore} [formula] ${row.contextAfter}`.slice(0, 200),
|
||||
}
|
||||
})
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,136 @@
|
||||
import { stripWrappers } from './normalize'
|
||||
|
||||
type Arrow = '<=>' | '->'
|
||||
|
||||
/** The reversible class is tried first, because `<=>` also contains `=` and `->`-like text. */
|
||||
const arrows: readonly (readonly [RegExp, Arrow])[] = [
|
||||
[/<=>|<->|\\rightleftharpoons|\\leftrightarrow|⇌|↔|\\rightleftarrows/, '<=>'],
|
||||
// `=` is the arrow of Chinese textbooks.
|
||||
[/->|→|\\rightarrow|\\longrightarrow|\\to(?![a-z])|=(?!=)|\\xrightarrow\{[^}]*\}|-\[[^\]]*\]>/, '->'],
|
||||
]
|
||||
|
||||
const uppercase = /\p{Lu}/u
|
||||
const lowercase = /\p{Ll}/u
|
||||
const digit = /\d/
|
||||
|
||||
function readCount(text: string, start: number): [count: number, end: number] {
|
||||
let end = start
|
||||
while (end < text.length && digit.test(text[end]!))
|
||||
end++
|
||||
return [end > start ? Number(text.slice(start, end)) : 1, end]
|
||||
}
|
||||
|
||||
/** Counts the elements of a formula from `start` to the closing bracket of the current group. */
|
||||
function readGroup(text: string, start: number): [counts: Map<string, number>, end: number] {
|
||||
const counts = new Map<string, number>()
|
||||
let index = start
|
||||
while (index < text.length) {
|
||||
const char = text[index]!
|
||||
if (char === '(' || char === '[') {
|
||||
const [inner, afterGroup] = readGroup(text, index + 1)
|
||||
const [multiplier, afterCount] = readCount(text, afterGroup)
|
||||
for (const [element, count] of inner)
|
||||
counts.set(element, (counts.get(element) ?? 0) + count * multiplier)
|
||||
index = afterCount
|
||||
}
|
||||
else if (char === ')' || char === ']') {
|
||||
return [counts, index + 1]
|
||||
}
|
||||
else if (uppercase.test(char)) {
|
||||
const element = char + (index + 1 < text.length && lowercase.test(text[index + 1]!) ? text[index + 1]! : '')
|
||||
const [count, afterCount] = readCount(text, index + element.length)
|
||||
counts.set(element, (counts.get(element) ?? 0) + count)
|
||||
index = afterCount
|
||||
}
|
||||
else {
|
||||
index++
|
||||
}
|
||||
}
|
||||
return [counts, index]
|
||||
}
|
||||
|
||||
/**
|
||||
* Keys one species by its element counts, so `Ca(OH)2` and `CaO2H2` are the same species.
|
||||
* A hydrate such as `CuSO4*5H2O` adds the water of crystallization.
|
||||
*/
|
||||
function speciesKey(formula: string): string {
|
||||
const total = new Map<string, number>()
|
||||
for (const part of formula.split(/\\cdot|\\bullet|[*·.•⋅]/)) {
|
||||
const trimmed = part.trim()
|
||||
const [multiplier, start] = readCount(trimmed, 0)
|
||||
const [counts] = readGroup(trimmed.slice(start), 0)
|
||||
for (const [element, count] of counts)
|
||||
total.set(element, (total.get(element) ?? 0) + count * multiplier)
|
||||
}
|
||||
return [...total].map(([element, count]) => `${element}${count}`).sort().join(' ')
|
||||
}
|
||||
|
||||
/** Parses an equation into its arrow class and one multiset of `(coefficient, species)` per side. */
|
||||
function parseChem(source: string): { arrow: Arrow | undefined, sides: Map<string, number>[] } {
|
||||
let text = stripWrappers(source)
|
||||
text = /\\ce\{(.*)\}/s.exec(text)?.[1] ?? text
|
||||
// State marks, gas and precipitate marks, and text notes do not change the species.
|
||||
text = text.replace(/\\text\{[^}]*\}|\^\{?[\\a-z]*\}?(?![0-9+-])|\s+v(?=\s|$)|↑|↓|\(g\)|\(s\)|\(aq\)|\(l\)/g, ' ')
|
||||
text = text.replace(/_\{?(\d+)\}?/g, '$1')
|
||||
|
||||
let arrow: Arrow | undefined
|
||||
let parts = [text]
|
||||
for (const [pattern, name] of arrows) {
|
||||
const match = pattern.exec(text)
|
||||
if (match) {
|
||||
arrow = name
|
||||
parts = [text.slice(0, match.index), text.slice(match.index + match[0].length)]
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
const sides = parts.map((side) => {
|
||||
const terms = new Map<string, number>()
|
||||
for (const term of side.split(/\s\+\s|\s*\+\s*(?=[0-9A-Z(])/)) {
|
||||
const trimmed = term.trim()
|
||||
if (!trimmed)
|
||||
continue
|
||||
const match = /^(\d+)\s*(?=[A-Z([])(.+)$/.exec(trimmed)
|
||||
const coefficient = match ? Number(match[1]) : 1
|
||||
const formula = match ? match[2]! : trimmed
|
||||
if (!/[A-Z]/.test(formula))
|
||||
continue
|
||||
const key = `${coefficient}|${speciesKey(formula)}`
|
||||
terms.set(key, (terms.get(key) ?? 0) + 1)
|
||||
}
|
||||
return terms
|
||||
})
|
||||
return { arrow, sides }
|
||||
}
|
||||
|
||||
function sameMultiset(left: Map<string, number>, right: Map<string, number>): boolean {
|
||||
return left.size === right.size && [...left].every(([key, count]) => right.get(key) === count)
|
||||
}
|
||||
|
||||
/** The species of a side without their coefficients. */
|
||||
function speciesOf(side: Map<string, number>): Set<string> {
|
||||
return new Set([...side.keys()].map(key => key.split('|')[1]!))
|
||||
}
|
||||
|
||||
/**
|
||||
* Tells whether two chemical equations have the same species, coefficients, and arrow class.
|
||||
* The order of terms is free. A wrong species (CO for CO2), coefficient, or arrow class fails.
|
||||
*
|
||||
* A hypothesis without any coefficient passes with the same species and arrow. Chinese, Japanese, and Korean readings
|
||||
* name the substances of an equation without coefficients, as a textbook does, so the listener cannot hear them. A
|
||||
* hypothesis that writes coefficients must write the right ones.
|
||||
*/
|
||||
export function sameChemistry(reference: string, hypothesis: string): boolean {
|
||||
const expected = parseChem(reference)
|
||||
const actual = parseChem(hypothesis)
|
||||
if (expected.arrow !== actual.arrow || expected.sides.length !== actual.sides.length)
|
||||
return false
|
||||
if (expected.sides.every((side, index) => sameMultiset(side, actual.sides[index]!)))
|
||||
return true
|
||||
const withoutCoefficients = actual.sides.every(side => [...side.keys()].every(key => key.startsWith('1|')))
|
||||
return withoutCoefficients && expected.sides.every((side, index) => {
|
||||
const left = speciesOf(side)
|
||||
const right = speciesOf(actual.sides[index]!)
|
||||
return left.size === right.size && [...left].every(species => right.has(species))
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import type { CurrencyItem } from '../../cases/formula-speech/cases'
|
||||
|
||||
import { canonicalize } from './normalize'
|
||||
|
||||
/** A dollar amount with a sign before it (`$5`, `US$5`, `USD 5`), or a word after it (`5 dollars`, `5美元`). */
|
||||
const amount = /(?:\$|USD|US\$)\s*(\d[\d,]*(?:\.\d+)?)|(\d[\d,]*(?:\.\d+)?)\s*(?:dollars?|USD|美元|美金|ドル|달러)/g
|
||||
|
||||
/**
|
||||
* Lists the US dollar amounts in a sentence, sorted.
|
||||
*
|
||||
* @example
|
||||
* dollarAmounts('Apples cost \\$3, and rent is 1,200 dollars')
|
||||
* // => [3, 1200]
|
||||
*/
|
||||
export function dollarAmounts(text: string): number[] {
|
||||
const normalized = text.replaceAll('\\$', '$').replaceAll('＄', '$')
|
||||
return [...normalized.matchAll(amount)]
|
||||
.map(match => Number((match[1] ?? match[2]!).replaceAll(',', '')))
|
||||
.sort((left, right) => left - right)
|
||||
}
|
||||
|
||||
/** Tells whether a sentence keeps every dollar amount, and no other amount, and every inline formula of the reference. */
|
||||
export function sameCurrencySentence(reference: readonly CurrencyItem[], hypothesis: string): boolean {
|
||||
const expected = reference.flatMap(item => item[0] === 'USD' ? [item[1]] : []).sort((left, right) => left - right)
|
||||
const actual = dollarAmounts(hypothesis)
|
||||
if (actual.length !== expected.length || actual.some((value, index) => value !== expected[index]))
|
||||
return false
|
||||
|
||||
const sentence = canonicalize(hypothesis)
|
||||
return reference.every(item => item[0] !== 'MATH' || sentence.includes(canonicalize(item[1])))
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
import type { FormulaSpeechCase } from '../../cases/formula-speech/cases'
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { grade, majorityScore } from '.'
|
||||
import { formulaSpeechCases } from '../../cases/formula-speech/cases'
|
||||
|
||||
const cases = new Map(formulaSpeechCases.map(item => [item.id, item]))
|
||||
|
||||
describe('grade', () => {
|
||||
// Ported from the T-157 scorer self test. It covers the three judge defects that the manual calibration found,
|
||||
// letter case, strict chemistry, currency, and units.
|
||||
it.each([
|
||||
['lim-sinx', String.raw`\lim_{x \to 0}\frac{x}{\sin x}=1`, 0, 'review #9: symbolic evaluated both limits to 1; operators must match'],
|
||||
['lim-basel', String.raw`\lim_{n\to\infty}\sum_{i=1}^{n}\frac{1}{i}=\frac{\pi^2}{6}`, 0, 'summand 1/i is not 1/i^2'],
|
||||
['lim-onesided', String.raw`\lim_{x \to 0^+} \ln x = -\infty`, 1, 'review #6: 0^{+} vs 0^+ (natural `}}` was read as an escaped brace)'],
|
||||
['set-notin', String.raw`\sqrt{2}\notin\mathbb{Q}`, 1, 'review #96/#104: \\notin\\mathbb{Q} without a space'],
|
||||
['set-cup', String.raw`A \cup B = \{1,2,3\}`, 1, 'escaped set braces still compare'],
|
||||
['chem-hydrate', String.raw`\ce{CuSO4 \cdot 5H2O}`, 1, 'review B#7: \\cdot hydrate dot'],
|
||||
['lim-onesided', String.raw`\lim_{x \to +0} \ln x = -\infty`, 1, 'Japanese x→+0 is 0^+'],
|
||||
['lim-basel', String.raw`\lim_{n \to \infty}\sum_{i=1}^{n}\frac{1}{i^2}=\frac{6}{\pi^2}`, 0, 'T-153 ko 6/π² inversion must fail'],
|
||||
['lim-basel', String.raw`\lim_{n\to\infty}\sum_{i=1}^{n}\frac{1}{i^{2}}=\frac{\pi^{2}}{6}`, 1, 'same formula, other braces'],
|
||||
['frac-pi2-6', String.raw`\frac{6}{\pi^2}`, 0, 'inverse fraction'],
|
||||
['frac-6-pi2', String.raw`6/\pi^2`, 1, 'slash form is equivalent (symbolic)'],
|
||||
['sum-gauss', String.raw`\sum_{k=1}^{n} k = \frac{n^2+n}{2}`, 1, 'expanded numerator (symbolic)'],
|
||||
['frac-mb-sigma', String.raw`\sigma_0 = f - \frac{1}{2}`, 0, '(f-1)/2 vs f-1/2 grouping'],
|
||||
['lim-sinx', String.raw`\lim_{x\to 0} \frac{\sin(x)}{x} = 1`, 1, 'parenthesised argument'],
|
||||
['case-fold-tri', String.raw`\triangle abc`, 1, 'case folded: speech cannot carry case'],
|
||||
['rel-perp', String.raw`ab \perp cd`, 1, 'T-153 relations: case folded'],
|
||||
['case-fold-ema', String.raw`e = m c^2`, 1, 'case folded'],
|
||||
['prob-fm-loss', String.raw`l_{fm}(\theta)=\mathbb{E}_{t\sim U([0,1]),\,x\sim p_t}\left[\left\|v_\theta(x,t)-u_t(x)\right\|^2\right]`, 1, 'L has one case, and U([0,1]) is U[0,1]'],
|
||||
['prob-fm-loss', String.raw`l_{fm}(\theta)=\mathbb{E}_{t\sim u[0,1],\,x\sim p_t}\left[\left\|v_\theta(x,t)-u_t(x)\right\|^2\right]`, 0, 'U and u are both in the reference'],
|
||||
['case-sens-aA', String.raw`a \in a`, 0, 'a and A both in reference: listener must hear "capital"'],
|
||||
['case-sens-aA', String.raw`a \in A`, 1, 'distinction kept'],
|
||||
['case-sens-nN', String.raw`n = n + 1`, 0, 'N vs n lost'],
|
||||
['case-sens-omega', String.raw`\omega \neq \omega`, 0, 'Ω vs ω lost'],
|
||||
['case-sens-omega', String.raw`\Omega \ne \omega`, 1, 'kept; \\ne alias'],
|
||||
['chem-water', String.raw`\ce{H2 + O2 -> H2O}`, 1, 'no coefficients: the substances of a textbook reading'],
|
||||
['chem-water', String.raw`\ce{H2 + O2 -> H2O2}`, 0, 'no coefficients, but another product'],
|
||||
['chem-water', String.raw`\ce{2H2 + O2 -> H2O}`, 0, 'one coefficient dropped'],
|
||||
['chem-water', String.raw`\ce{O2 + 2H2 -> 2H2O}`, 1, 'reactant order is free'],
|
||||
['chem-water', String.raw`2H_2 + O_2 \rightarrow 2H_2O`, 1, 'plain LaTeX arrow form'],
|
||||
['chem-water', String.raw`\ce{2H2 + O2 = 2H2O}`, 1, '= is the Chinese textbook arrow'],
|
||||
['chem-water', String.raw`\ce{2H2 + O2 <=> 2H2O}`, 0, 'wrong arrow class'],
|
||||
['chem-water', String.raw`\ce{2H2 + O2 -> 2H2O2}`, 0, 'H2O2 is not H2O'],
|
||||
['chem-co2', String.raw`\ce{C + O2 -> CO}`, 0, 'CO is not CO2'],
|
||||
['chem-ca-oh', String.raw`\ce{CaOH2 + CO2 -> CaCO3 + H2O}`, 0, 'lost (OH) group changes the formula'],
|
||||
['chem-ca-oh', String.raw`\ce{Ca(OH)2 + CO2 -> CaCO3 v + H2O}`, 1, 'precipitate mark ignored'],
|
||||
['chem-hydrate', String.raw`\ce{CuSO4.5H2O}`, 1, 'hydrate dot form'],
|
||||
['chem-hydrate', String.raw`\ce{CuSO4}`, 0, 'water of crystallisation lost'],
|
||||
['chem-haber', String.raw`\ce{N2 + 3H2 -> 2NH3}`, 0, 'reversible arrow lost'],
|
||||
['cur-two', String.raw`苹果 \$3，香蕉 \$5`, 1, 'two USD amounts'],
|
||||
['cur-two', String.raw`苹果 3，香蕉 5`, 0, 'dollar sign lost'],
|
||||
['cur-thousand', String.raw`The rent is \$1,200 a month`, 1, 'thousands separator'],
|
||||
['cur-thousand', String.raw`The rent is \$1 200`, 0, 'read as 1 and 200'],
|
||||
['cur-mixed', String.raw`A ticket is \$5, and the area is $x^2$`, 1, 'money + math'],
|
||||
['cur-mixed', String.raw`A ticket is 5, and the area is x`, 0, 'both lost'],
|
||||
['unit-temp', String.raw`25°C`, 1, 'degree sign'],
|
||||
['unit-g', String.raw`9.8\ \text{m/s}^2`, 1, 'text unit'],
|
||||
['unit-g', String.raw`9.8\ \text{m/s}`, 0, 'square lost'],
|
||||
['set-cap-empty', String.raw`A \cap B = \emptyset`, 1, 'empty set alias'],
|
||||
['set-interval', String.raw`x \in [0, b]`, 0, 'closed interval is not half-open'],
|
||||
] as const)('grades %s as %s: %s', (id, hypothesis, expected, _reason) => {
|
||||
expect(grade(cases.get(id)!, hypothesis).score).toBe(expected)
|
||||
})
|
||||
|
||||
it('accepts the spaced braces of MathBridge references', () => {
|
||||
// MathBridge writes a space after each token, as in `_ { u }`. The answer `_u` is the same formula.
|
||||
const item = (reference: string): FormulaSpeechCase => ({ id: 'mb', stratum: 'math-bridge', source: 'MathBridge', mode: 'symbolic', reference, input: `$${reference}$`, note: '' })
|
||||
expect(grade(item(String.raw`\lambda_ { \mathrm { u } } \rightarrow\infty`), String.raw`\lambda_u \to \infty`).score).toBe(1)
|
||||
expect(grade(item(String.raw`b\in\mathbb { N } ^ { + }`), String.raw`b\in\mathbb{N}^{+}`).score).toBe(1)
|
||||
expect(grade(item(String.raw`R_ { 0 , \mathrm { b } } =H_ { \mathrm { b } }`), String.raw`R_{0,b}=H_b`).score).toBe(1)
|
||||
})
|
||||
|
||||
it('scores an empty answer 0 and does not throw on malformed LaTeX', () => {
|
||||
expect(grade(cases.get('lim-basel')!, '')).toEqual({ score: 0, rule: 'empty' })
|
||||
expect(grade(cases.get('lim-basel')!, String.raw`\frac{{{`).score).toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('majorityScore', () => {
|
||||
it('passes only with a strict majority', () => {
|
||||
expect(majorityScore([1, 0, 1])).toBe(1)
|
||||
expect(majorityScore([1, 0])).toBe(0)
|
||||
expect(majorityScore([0, 0, 1])).toBe(0)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,63 @@
|
||||
import type { FormulaSpeechCase } from '../../cases/formula-speech/cases'
|
||||
import type { KeptCase } from './normalize'
|
||||
|
||||
import { sameChemistry } from './chem'
|
||||
import { sameCurrencySentence } from './currency'
|
||||
import { canonicalize, caseSensitiveLetters } from './normalize'
|
||||
import { symbolicallyEquivalent } from './symbolic'
|
||||
|
||||
/**
|
||||
* The grade of one written-down answer.
|
||||
*
|
||||
* `rule` names the check that decided it: `norm`, `symbolic`, `chem`, or `currency`.
|
||||
* `empty` means the listener wrote nothing, and `error` means the grader failed on malformed input.
|
||||
* `unclear` means the listener could not hear anything recoverable. The read-aloud task sets it, not {@link grade}.
|
||||
*/
|
||||
export interface Grade {
|
||||
score: 0 | 1
|
||||
rule: 'norm' | 'symbolic' | 'chem' | 'currency' | 'empty' | 'error' | 'unclear'
|
||||
}
|
||||
|
||||
/**
|
||||
* Grades what a listener wrote down against the reference of a case. The result is deterministic, 1 or 0.
|
||||
* The listener model never grades. Malformed LaTeX scores 0 and never fails a run.
|
||||
*/
|
||||
export function grade(item: FormulaSpeechCase, hypothesis: string | undefined): Grade {
|
||||
try {
|
||||
return gradeUnsafe(item, hypothesis)
|
||||
}
|
||||
catch {
|
||||
return { score: 0, rule: 'error' }
|
||||
}
|
||||
}
|
||||
|
||||
function gradeUnsafe(item: FormulaSpeechCase, hypothesis: string | undefined): Grade {
|
||||
if (!hypothesis?.trim())
|
||||
return { score: 0, rule: 'empty' }
|
||||
|
||||
// Only currency cases have a list reference.
|
||||
if (typeof item.reference !== 'string')
|
||||
return { score: sameCurrencySentence(item.reference, hypothesis) ? 1 : 0, rule: 'currency' }
|
||||
|
||||
const reference = item.reference
|
||||
if (item.mode === 'chem')
|
||||
return { score: sameChemistry(reference, hypothesis) ? 1 : 0, rule: 'chem' }
|
||||
|
||||
const keep: KeptCase = item.letterCase === 'sensitive' ? 'all' : item.letterCase === 'fold' ? new Set() : caseSensitiveLetters(reference)
|
||||
if (canonicalize(reference, keep) === canonicalize(hypothesis, keep))
|
||||
return { score: 1, rule: 'norm' }
|
||||
if (item.mode === 'symbolic' && symbolicallyEquivalent(reference, hypothesis, keep))
|
||||
return { score: 1, rule: 'symbolic' }
|
||||
return { score: 0, rule: item.mode }
|
||||
}
|
||||
|
||||
/**
|
||||
* Combines the grades of several listener votes. The item passes only with a strict majority of passing votes.
|
||||
*
|
||||
* @example
|
||||
* majorityScore([1, 0, 1])
|
||||
* // => 1
|
||||
*/
|
||||
export function majorityScore(scores: readonly (0 | 1)[]): 0 | 1 {
|
||||
return 2 * scores.reduce<number>((sum, score) => sum + score, 0) > scores.length ? 1 : 0
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
const greekNames = new Set(['alpha', 'beta', 'gamma', 'delta', 'epsilon', 'zeta', 'eta', 'theta', 'iota', 'kappa', 'lambda', 'mu', 'nu', 'xi', 'omicron', 'pi', 'rho', 'sigma', 'tau', 'upsilon', 'phi', 'chi', 'psi', 'omega'])
|
||||
|
||||
/**
|
||||
* Rewrites that make two spellings of one notation compare equal. Order matters: a later rule sees the output of the earlier ones.
|
||||
* Set braces become sentinels first, so that a natural closing `}}` is never read as an escaped brace.
|
||||
*/
|
||||
const aliases: readonly (readonly [RegExp, string])[] = [
|
||||
[/\\dfrac|\\tfrac/g, String.raw`\frac`],
|
||||
[/\\leqslant|\\leqq|\\le(?![a-z])/g, String.raw`\leq`],
|
||||
[/\\geqslant|\\geqq|\\ge(?![a-z])/g, String.raw`\geq`],
|
||||
[/\\ne(?![a-z])|\\not=/g, String.raw`\neq`],
|
||||
[/\\rightarrow|\\longrightarrow|\\to(?![a-z])/g, String.raw`\to`],
|
||||
[/\\emptyset|\\varnothing|\\O(?![a-zA-Z])/g, String.raw`\emptyset`],
|
||||
[/\\dots|\\ldots|\\cdots|\.\.\.|…/g, String.raw`\cdots`],
|
||||
[/\\times/g, String.raw`\cdot`],
|
||||
[/\\ast|\*/g, String.raw`\cdot`],
|
||||
[/\\colon/g, ':'],
|
||||
[/\\vert|\\mid|\\lvert|\\rvert/g, '|'],
|
||||
[/\\varepsilon/g, String.raw`\epsilon`],
|
||||
[/\\varphi/g, String.raw`\phi`],
|
||||
[/\\degree|°|\\textdegree/g, String.raw`^\circ`],
|
||||
[/\\lbrace|\\\{/g, '\x01'],
|
||||
[/\\rbrace|\\\}/g, '\x02'],
|
||||
[/\\mapsto|\\longmapsto/g, String.raw`\mapsto`],
|
||||
[/\\(?:mathbf|mathbb|mathcal|mathrm|text|textrm|operatorname|mathit|boldsymbol)\s*\{([^{}]*)\}/g, ' $1 '],
|
||||
[/\\(?:mathbf|mathbb|mathcal|mathrm|text|textrm|operatorname|mathit|boldsymbol)(?![a-z])/g, ''],
|
||||
[/\\Delta(?=\s*ABC)|\\bigtriangleup/g, String.raw`\triangle`],
|
||||
[/π/g, String.raw`\pi`],
|
||||
[/∞/g, String.raw`\infty`],
|
||||
[/≤/g, String.raw`\leq`],
|
||||
[/≥/g, String.raw`\geq`],
|
||||
[/≠/g, String.raw`\neq`],
|
||||
[/≈/g, String.raw`\approx`],
|
||||
[/∈/g, String.raw`\in`],
|
||||
[/∉/g, String.raw`\notin`],
|
||||
[/⊆/g, String.raw`\subseteq`],
|
||||
[/⊥/g, String.raw`\perp`],
|
||||
[/∥/g, String.raw`\parallel`],
|
||||
[/√/g, String.raw`\sqrt`],
|
||||
[/[×·⋅]/g, String.raw`\cdot`],
|
||||
[/[−–]/g, '-'],
|
||||
[/∠/g, String.raw`\angle`],
|
||||
[/△/g, String.raw`\triangle`],
|
||||
// Japanese textbooks write a one-sided limit as `x→+0`.
|
||||
[/\\to\s*\+\s*0(?![0-9.])/g, String.raw`\to 0^+`],
|
||||
[/\\to\s*-\s*0(?![0-9.])/g, String.raw`\to 0^-`],
|
||||
[/\\%/g, '%'],
|
||||
[/\\lim\\limits/g, String.raw`\lim`],
|
||||
[/\\sum\\limits/g, String.raw`\sum`],
|
||||
]
|
||||
|
||||
/** Commands and characters that only change spacing or size. */
|
||||
const spacing = /\\left|\\right|\\bigg|\\Bigg|\\big|\\Big|\\displaystyle|\\,|\\;|\\:|\\!|\\quad|\\qquad|\\ |~|\\limits|\$/g
|
||||
|
||||
/**
|
||||
* Removes the wrappers around an answer: math delimiters, code fences, equation environments, and final punctuation.
|
||||
* A listener who heard raw LaTeX read aloud writes "dollar backslash triangle"; that spoken markup is not content.
|
||||
*
|
||||
* @example
|
||||
* stripWrappers('$$\\backslash triangle ABC$$.')
|
||||
* // => '\\triangle ABC'
|
||||
*/
|
||||
export function stripWrappers(source: string): string {
|
||||
let text = source.trim()
|
||||
text = text.replace(/\\backslash\s*(?=\\)/g, '')
|
||||
text = text.replace(/\\backslash\s*([A-Z]+)/gi, (_, word: string) => `\\${word}`)
|
||||
text = text.trim().replace(/^\\\$\s*|\s*\\\$$/g, '')
|
||||
text = text.replace(/^```(?:latex|tex)?|```$/g, '').trim()
|
||||
text = text.replace(/^\$\$?|\$\$?$/g, '').trim()
|
||||
text = text.replace(/^\\\[|\\\]$|^\\\(|\\\)$/g, '').trim()
|
||||
text = text.replace(/\\begin\{(?:equation|align)\*?\}|\\end\{(?:equation|align)\*?\}/g, '')
|
||||
return text.replace(/[.。,，]+$/, '').trim()
|
||||
}
|
||||
|
||||
/**
|
||||
* The letters whose case a comparison keeps: `all`, or a set of small Latin letters and small Greek names. A letter
|
||||
* outside the set is folded.
|
||||
*/
|
||||
export type KeptCase = 'all' | ReadonlySet<string>
|
||||
|
||||
/**
|
||||
* Lowers the case of every letter outside a command name, except the letters in `keep`. A capital Greek command becomes
|
||||
* the small one.
|
||||
*
|
||||
* @example
|
||||
* foldLetterCase('\\Omega = E \\cdot X')
|
||||
* // => '\\omega = e \\cdot x'
|
||||
*/
|
||||
export function foldLetterCase(source: string, keep: ReadonlySet<string> = new Set()): string {
|
||||
const text = source.replace(/\\([A-Z][a-z]+)/g, (match, name: string) => greekNames.has(name.toLowerCase()) && !keep.has(name.toLowerCase()) ? `\\${name.toLowerCase()}` : match)
|
||||
let result = ''
|
||||
for (let index = 0; index < text.length;) {
|
||||
if (text[index] === '\\') {
|
||||
const command = /^\\(?:[A-Z]+|.)?/i.exec(text.slice(index))![0]
|
||||
result += command
|
||||
index += command.length
|
||||
continue
|
||||
}
|
||||
const char = text[index]!
|
||||
result += keep.has(char.toLowerCase()) ? char : char.toLowerCase()
|
||||
index++
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* The letters whose case must survive speech for this reference.
|
||||
*
|
||||
* Speech carries letter case only as a word, as "capital A", and a reading says it only for a letter that the formula
|
||||
* uses in both cases, for example `a \\in A` or `\\Omega \\neq \\omega`. Those letters keep their case in the
|
||||
* comparison. The other letters are folded, so `\\mathcal{L}_{FM}` read as "L sub F M" matches `l_{fm}`.
|
||||
*/
|
||||
export function caseSensitiveLetters(reference: string): Set<string> {
|
||||
// A calligraphic or a blackboard letter is still a letter, as `\\mathcal{U}` beside `u_t`. Upright words are names.
|
||||
const body = reference.replace(/\\(?:mathbb|mathcal)\{([^}]*)\}/g, '$1').replace(/\\(?:mathrm|text|operatorname)\{[^}]*\}/g, '')
|
||||
const commands = [...body.matchAll(/\\([A-Z]+)/gi)].map(match => match[1]!)
|
||||
const sensitive = new Set<string>()
|
||||
for (const name of commands) {
|
||||
const small = name.toLowerCase()
|
||||
if (greekNames.has(small) && commands.includes(small) && commands.includes(small[0]!.toUpperCase() + small.slice(1)))
|
||||
sensitive.add(small)
|
||||
}
|
||||
|
||||
const plain = body.replace(/\\[A-Z]+/gi, ' ')
|
||||
const letters = new Set([
|
||||
// The first letter of each run, so `\\sin` and `ABC` do not count as many single letters.
|
||||
...[...plain.matchAll(/(?<![\\A-Z])([A-Z])/gi)].map(match => match[1]!),
|
||||
// Every capital inside a run of capitals, for example each letter of `ABC`.
|
||||
...[...body.matchAll(/[A-Z](?=[A-Z])|(?<=[A-Z])[A-Z]/g)].map(match => match[0]),
|
||||
])
|
||||
for (const letter of letters) {
|
||||
if (letters.has(letter.toLowerCase()) && letters.has(letter.toUpperCase()))
|
||||
sensitive.add(letter.toLowerCase())
|
||||
}
|
||||
return sensitive
|
||||
}
|
||||
|
||||
/**
|
||||
* Brings an answer to one canonical LaTeX string: aliases, no spacing, single-token braces removed, thousands separators removed.
|
||||
* One space stays after each command, so `\triangle ABC` does not become the command `\triangleABC`.
|
||||
*
|
||||
* @example
|
||||
* canonicalize('$\\left( x^{2} \\right) \\le \\dfrac 1 2$')
|
||||
* // => '(x^2)\\leq \\frac{1}{2}'
|
||||
*/
|
||||
export function canonicalize(source: string, keep: KeptCase = new Set()): string {
|
||||
let text = stripWrappers(source)
|
||||
for (const [pattern, replacement] of aliases)
|
||||
text = text.replace(pattern, replacement)
|
||||
text = text.replace(spacing, ' ')
|
||||
text = text.replace(/\\infty/g, String.raw`\infty `)
|
||||
text = text.replaceAll('\x01', '\\{').replaceAll('\x02', '\\}')
|
||||
// Three passes unwrap nested single tokens, for example `x^{{2}}`. MathBridge writes its references with a space
|
||||
// after each token, as in `\lambda_ { u }`, so a space can come between the script mark and its brace.
|
||||
for (let pass = 0; pass < 3; pass++)
|
||||
text = text.replace(/([\^_])\s*\{\s*([A-Z0-9+-]|\\[A-Z]+)\s*\}/gi, '$1$2')
|
||||
text = text.replace(/\\frac\s*([0-9A-Z])\s*([0-9A-Z])/gi, String.raw`\frac{$1}{$2}`)
|
||||
text = text.replace(/\\sqrt\s*([0-9A-Z])(?![A-Z])/gi, String.raw`\sqrt{$1}`)
|
||||
text = text.replace(/\^\\circ\s*/g, String.raw`^\circ `)
|
||||
text = text.replace(/(\d),(\d{3})(?!\d)/g, '$1$2')
|
||||
// A distribution on an interval is written `U[0,1]` or `U([0,1])`: parentheses around one bracketed interval add nothing.
|
||||
text = text.replace(/\(\s*(\[[^()[\]]*[\])])\s*\)/g, '$1')
|
||||
text = text.replace(/(\\[A-Z]+)\s*/gi, '$1\x00')
|
||||
text = text.replace(/\s+/g, '').replaceAll('\x00', ' ')
|
||||
return keep === 'all' ? text : foldLetterCase(text, keep)
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import type { Expression } from '@cortex-js/compute-engine'
|
||||
|
||||
import type { KeptCase } from './normalize'
|
||||
|
||||
import { ComputeEngine, isFunction } from '@cortex-js/compute-engine'
|
||||
|
||||
import { foldLetterCase, stripWrappers } from './normalize'
|
||||
|
||||
const engine = new ComputeEngine()
|
||||
|
||||
const relations = new Set(['Equal', 'NotEqual', 'Less', 'LessEqual', 'Greater', 'GreaterEqual'])
|
||||
|
||||
/**
|
||||
* Operators that the engine evaluates to a value. `lim sin x / x` and `lim x / sin x` both evaluate to 1,
|
||||
* so these operators must match one by one before values are compared.
|
||||
*/
|
||||
const evaluatedOperators = new Set(['Limit', 'Sum', 'Product', 'Integrate', 'D', 'Derivative'])
|
||||
|
||||
/** Prepares LaTeX for the parser: no sizing commands, one spelling of each relation, and products written out. */
|
||||
function prepare(source: string, keep: KeptCase): string {
|
||||
let text = stripWrappers(source)
|
||||
text = text.replace(/\\left|\\right|\\,|\\;|\\!|\\displaystyle|\\limits/g, '')
|
||||
text = text.replace(/\\dfrac|\\tfrac/g, String.raw`\frac`)
|
||||
text = text.replace(/\\le(?![a-z])|\\leqslant/g, String.raw`\leq`)
|
||||
text = text.replace(/\\ge(?![a-z])|\\geqslant/g, String.raw`\geq`)
|
||||
text = text.replaceAll('π', String.raw`\pi`).replaceAll('∞', String.raw`\infty`).replaceAll('−', '-')
|
||||
// `n(n+1)` is a product here. Only f, g, and h name functions in the cases.
|
||||
text = text.replace(/(?<![\\A-Z])([a-ei-z])\s*\(/gi, String.raw`$1 \cdot (`)
|
||||
return keep === 'all' ? text : foldLetterCase(text, keep)
|
||||
}
|
||||
|
||||
/** Equal by structure, by value for constants, or as an identity in the free variables, for example `n(n+1)` and `n^2+n`. */
|
||||
function same(left: Expression, right: Expression): boolean {
|
||||
return left.isSame(right) || left.isEqual(right) === true || left.isIdenticallyEqual(right) === true
|
||||
}
|
||||
|
||||
function operandsOf(expression: Expression): readonly Expression[] {
|
||||
return isFunction(expression) ? expression.ops : []
|
||||
}
|
||||
|
||||
function evaluatedParts(expression: Expression): Expression[] {
|
||||
const own = evaluatedOperators.has(expression.operator) ? [expression] : []
|
||||
return [...own, ...operandsOf(expression).flatMap(evaluatedParts)]
|
||||
}
|
||||
|
||||
/** Each limit, sum, product, integral, and derivative must match its counterpart, argument by argument. */
|
||||
function evaluatedPartsMatch(left: Expression, right: Expression): boolean {
|
||||
const byJson = (expression: Expression) => JSON.stringify(expression.json)
|
||||
const leftParts = evaluatedParts(left).sort((a, b) => byJson(a).localeCompare(byJson(b)))
|
||||
const rightParts = evaluatedParts(right).sort((a, b) => byJson(a).localeCompare(byJson(b)))
|
||||
if (leftParts.length !== rightParts.length)
|
||||
return false
|
||||
|
||||
return leftParts.every((part, index) => {
|
||||
const counterpart = rightParts[index]!
|
||||
if (part.operator !== counterpart.operator)
|
||||
return false
|
||||
if (part.isSame(counterpart))
|
||||
return true
|
||||
const ops = operandsOf(part)
|
||||
const counterpartOps = operandsOf(counterpart)
|
||||
return ops.length === counterpartOps.length && ops.every((op, opIndex) => same(op, counterpartOps[opIndex]!))
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Tells whether a hypothesis is symbolically equivalent to the reference.
|
||||
*
|
||||
* Relations must have the same operator and equivalent sides. An equation may also swap its sides.
|
||||
* Unparsable input is not equivalent; it never throws.
|
||||
*/
|
||||
export function symbolicallyEquivalent(reference: string, hypothesis: string, keep: KeptCase): boolean {
|
||||
try {
|
||||
const expected = engine.parse(prepare(reference, keep))
|
||||
const actual = engine.parse(prepare(hypothesis, keep))
|
||||
if (!expected.isValid || !actual.isValid)
|
||||
return false
|
||||
if (expected.isSame(actual))
|
||||
return true
|
||||
if (!evaluatedPartsMatch(expected, actual))
|
||||
return false
|
||||
|
||||
if (relations.has(expected.operator) || relations.has(actual.operator)) {
|
||||
if (expected.operator !== actual.operator || !isFunction(expected) || !isFunction(actual))
|
||||
return false
|
||||
if (same(expected.op1, actual.op1) && same(expected.op2, actual.op2))
|
||||
return true
|
||||
return expected.operator === 'Equal' && same(expected.op1, actual.op2) && same(expected.op2, actual.op1)
|
||||
}
|
||||
return same(expected, actual)
|
||||
}
|
||||
catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,845 @@
|
||||
{
|
||||
"seed": 20261020,
|
||||
"items": [
|
||||
{
|
||||
"id": "rule|mb-8715240|en",
|
||||
"object": "rule",
|
||||
"caseId": "mb-8715240",
|
||||
"language": "en",
|
||||
"reference": "\\mathcal { I }",
|
||||
"spoken": "I",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; “I” naturally reads the letter in the formula."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|mb-3510988|en",
|
||||
"object": "llm",
|
||||
"caseId": "mb-3510988",
|
||||
"language": "en",
|
||||
"reference": "\\|e_ { \\vect u_3 } \\|_ { \\infty }",
|
||||
"spoken": "The norm, with subscript infinity, of e, with subscript vector u, subscript three.",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the subscripts are read in words."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|mb-7855753|en",
|
||||
"object": "sre",
|
||||
"caseId": "mb-7855753",
|
||||
"language": "en",
|
||||
"reference": "\\mathbb E [ N_\\alpha^T ( v ) ]",
|
||||
"spoken": "double struck E of N sub alpha to the T power of v",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the reading uses standard words for the expectation, subscript, and power."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": "Check whether \"N sub alpha to the T power of v\" keeps the grouping of N_alpha^T(v)."
|
||||
},
|
||||
{
|
||||
"id": "rule|mb-519492|en",
|
||||
"object": "rule",
|
||||
"caseId": "mb-519492",
|
||||
"language": "en",
|
||||
"reference": "k=1 , \\dots , d",
|
||||
"spoken": "k equals one, dot dot dot, d",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; “and so on, up to” is a natural way to read the ellipsis."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Read the ellipsis as \"and so on\" or \"up to\", not \"dot dot dot\"."
|
||||
},
|
||||
{
|
||||
"id": "sre|frac-6-pi2|en",
|
||||
"object": "sre",
|
||||
"caseId": "frac-6-pi2",
|
||||
"language": "en",
|
||||
"reference": "\\frac{6}{\\pi^2}",
|
||||
"spoken": "the fraction with numerator 6 and denominator pi squared",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the numerator-and-denominator phrasing naturally preserves the grouping of pi squared."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "A simple fraction reads shorter: \"six over pi squared\"."
|
||||
},
|
||||
{
|
||||
"id": "rule|mb-3064229|en",
|
||||
"object": "rule",
|
||||
"caseId": "mb-3064229",
|
||||
"language": "en",
|
||||
"reference": "k\\in [ K ]",
|
||||
"spoken": "lowercase k is an element of open bracket capital K close bracket",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "“Lowercase” and “capital” spell out letter case, and “open bracket” and “close bracket” name brackets around the set."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|misc-binom|en",
|
||||
"object": "sre",
|
||||
"caseId": "misc-binom",
|
||||
"language": "en",
|
||||
"reference": "(a+b)^2=a^2+2ab+b^2",
|
||||
"spoken": "open paren a plus b close paren squared equals a squared plus 2 a b plus b squared",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading names the parentheses around the compound expression instead of reading its meaning naturally."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Do not read the parentheses. Say \"the square of the sum of a and b\"."
|
||||
},
|
||||
{
|
||||
"id": "llm|mb-1727433|en",
|
||||
"object": "llm",
|
||||
"caseId": "mb-1727433",
|
||||
"language": "en",
|
||||
"reference": "\\Delta_k < 0",
|
||||
"spoken": "Delta sub k is less than zero.",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; “Delta sub k is less than zero” is a natural reading."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": "Good reading."
|
||||
},
|
||||
{
|
||||
"id": "rule|mb-8575803|en",
|
||||
"object": "rule",
|
||||
"caseId": "mb-8575803",
|
||||
"language": "en",
|
||||
"reference": "\\mathcal { B } ( B \\to \\ , K^ { * } \\ , \\nu\\ , \\bar { \\nu } )",
|
||||
"spoken": "B of B approaches, K superscript star, nu, nu bar",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "“Superscript” spells out the markup for the star rather than reading it as “star,” and “approaches” is not a spelled part."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "The reading of the decay and of \"nu, nu bar\" is unclear."
|
||||
},
|
||||
{
|
||||
"id": "llm|mb-6904738|en",
|
||||
"object": "llm",
|
||||
"caseId": "mb-6904738",
|
||||
"language": "en",
|
||||
"reference": "G^ { ( p ) }",
|
||||
"spoken": "G superscript, open parenthesis, p, close parenthesis.",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "“Open parenthesis” and “close parenthesis” spell out the grouping instead of reading the superscript as G to the power of p."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|mb-5565465|en",
|
||||
"object": "sre",
|
||||
"caseId": "mb-5565465",
|
||||
"language": "en",
|
||||
"reference": "\\frac { 7 } { 60 } \\mathbf { k } \\cdot \\mathbf { k } ^\\prime",
|
||||
"spoken": "7 over 60 bold k times bold k prime",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "“Bold” names formatting markup for both k’s."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Only one k appears, so \"bold\" is not needed: \"k times k prime\"."
|
||||
},
|
||||
{
|
||||
"id": "llm|chem-water|ja",
|
||||
"object": "llm",
|
||||
"caseId": "chem-water",
|
||||
"language": "ja",
|
||||
"reference": "\\ce{2H2 + O2 -> 2H2O}",
|
||||
"spoken": "係数二、エイチの下付き数字二、プラス、オーの下付き数字二、右向き矢印、係数二、エイチの下付き数字二、オー",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "No part is spelled; the element symbols are read by letter, and the subscripts and reaction arrow are described in Japanese."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Reads every subscript and the arrow by name. It does not work as a reading of an equation."
|
||||
},
|
||||
{
|
||||
"id": "llm|misc-deriv|ja",
|
||||
"object": "llm",
|
||||
"caseId": "misc-deriv",
|
||||
"language": "ja",
|
||||
"reference": "\\frac{d}{dx}x^{n}=nx^{n-1}",
|
||||
"spoken": "分子がディー、分母がディーエックスの分数に、エックスのエヌ乗。イコール、エヌかけるエックスの、肩にエヌひく一。",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The phrase 「肩に」 spells out the exponent’s visual placement instead of reading it as a power."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Mostly right. Unsure whether 肩に is a usual Japanese reading."
|
||||
},
|
||||
{
|
||||
"id": "sre|root-quadratic|ja",
|
||||
"object": "sre",
|
||||
"caseId": "root-quadratic",
|
||||
"language": "ja",
|
||||
"reference": "x=\\frac{-b\\pm\\sqrt{b^2-4ac}}{2a}",
|
||||
"spoken": "x equals the fraction with numerator negative b plus or minus the square root of b squared minus 4 a c and denominator 2 a",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading is mostly in English rather than Japanese."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Reads in English, not Japanese. A reading in another language does not pass."
|
||||
},
|
||||
{
|
||||
"id": "sre|chem-h2o2|ja",
|
||||
"object": "sre",
|
||||
"caseId": "chem-h2o2",
|
||||
"language": "ja",
|
||||
"reference": "\\ce{2H2O2 -> 2H2O + O2}",
|
||||
"spoken": "2 empty normal H of sub 2 normal O sub 2 right arrow 2 empty normal H of sub 2 normal O plus normal O sub 2",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading spells the chemical symbols with markup words such as “empty” and “normal,” and says “right arrow” in English."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|sum-geo|ja",
|
||||
"object": "rule",
|
||||
"caseId": "sum-geo",
|
||||
"language": "ja",
|
||||
"reference": "\\sum_{k=0}^{\\infty} \\frac{1}{2^{k}}=2",
|
||||
"spoken": "ケーがゼロから無限大までのにのケー乗分のいちの和 イコール に",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the sum, fraction, power, and equality are read in standard Japanese mathematical terms."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|case-sens-aA|ja",
|
||||
"object": "llm",
|
||||
"caseId": "case-sens-aA",
|
||||
"language": "ja",
|
||||
"reference": "a \\in A",
|
||||
"spoken": "小文字のエーは、大文字のエーに属する。",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the reading naturally distinguishes lowercase a from uppercase A."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": "Shorter case words are enough, for example 小 a and 大 A."
|
||||
},
|
||||
{
|
||||
"id": "sre|frac-nested|ja",
|
||||
"object": "sre",
|
||||
"caseId": "frac-nested",
|
||||
"language": "ja",
|
||||
"reference": "\\frac{1}{1+\\frac{1}{x}}",
|
||||
"spoken": "the fraction with numerator 1 and denominator 1 plus 1 over x",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading is mostly in English rather than Japanese."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Reads in English, not Japanese. A reading in another language does not pass."
|
||||
},
|
||||
{
|
||||
"id": "llm|set-builder|ja",
|
||||
"object": "llm",
|
||||
"caseId": "set-builder",
|
||||
"language": "ja",
|
||||
"reference": "\\{x \\mid x > 0\\}",
|
||||
"spoken": "左中かっこ、エックス、縦棒、エックス大なりゼロ、右中かっこ。",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "It spells out the set braces as “left brace” and “right brace,” and reads the set-builder bar as “vertical bar.”"
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|frac-6-pi2|zh",
|
||||
"object": "rule",
|
||||
"caseId": "frac-6-pi2",
|
||||
"language": "zh",
|
||||
"reference": "\\frac{6}{\\pi^2}",
|
||||
"spoken": "派的平方分之6",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; this is a natural reading of the fraction."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|unit-g|zh",
|
||||
"object": "llm",
|
||||
"caseId": "unit-g",
|
||||
"language": "zh",
|
||||
"reference": "9.8\\,\\mathrm{m/s^2}",
|
||||
"spoken": "九点八米每秒平方",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the quantity is read naturally in Mandarin Chinese."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|rel-parallel|zh",
|
||||
"object": "llm",
|
||||
"caseId": "rel-parallel",
|
||||
"language": "zh",
|
||||
"reference": "l_1 \\parallel l_2",
|
||||
"spoken": "小写字母 l，下标一，平行于小写字母 l，下标二。",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The reading spells out “小写字母” even though the formula contains only one case of l."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "The letter has one case in the formula, so the case is not needed. If it is, say 小 l."
|
||||
},
|
||||
{
|
||||
"id": "sre|frac-6-pi2|zh",
|
||||
"object": "sre",
|
||||
"caseId": "frac-6-pi2",
|
||||
"language": "zh",
|
||||
"reference": "\\frac{6}{\\pi^2}",
|
||||
"spoken": "the fraction with numerator 6 and denominator pi squared",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "The reading is in English rather than Mandarin Chinese."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Not Chinese. The fraction also reads shorter."
|
||||
},
|
||||
{
|
||||
"id": "rule|case-sens-aA|zh",
|
||||
"object": "rule",
|
||||
"caseId": "case-sens-aA",
|
||||
"language": "zh",
|
||||
"reference": "a \\in A",
|
||||
"spoken": "小写a 属于 大写A",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Use the short form 小a and 大A. The case words can go unless a variant of the letter appears."
|
||||
},
|
||||
{
|
||||
"id": "llm|set-builder|zh",
|
||||
"object": "llm",
|
||||
"caseId": "set-builder",
|
||||
"language": "zh",
|
||||
"reference": "\\{x \\mid x > 0\\}",
|
||||
"spoken": "左花括号，x，使得 x 大于零，右花括号。",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading names the left and right braces around the set."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "The braces of a set are clear from the meaning and need no reading."
|
||||
},
|
||||
{
|
||||
"id": "sre|misc-binom|zh",
|
||||
"object": "sre",
|
||||
"caseId": "misc-binom",
|
||||
"language": "zh",
|
||||
"reference": "(a+b)^2=a^2+2ab+b^2",
|
||||
"spoken": "open paren a plus b close paren squared equals a squared plus 2 a b plus b squared",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading is mostly in English rather than Mandarin Chinese, and it names the main parentheses as “open paren” and “close paren.”"
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Not Chinese, and the parentheses are read. Better: a 与 b 的和的平方 等于 a 方加 2ab 加 b 方."
|
||||
},
|
||||
{
|
||||
"id": "rule|set-interval|zh",
|
||||
"object": "rule",
|
||||
"caseId": "set-interval",
|
||||
"language": "zh",
|
||||
"reference": "x \\in (0, b]",
|
||||
"spoken": "x 属于 左开右闭区间0到b",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; “左开右闭区间” naturally describes the interval notation."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|root-sqrt2|zh",
|
||||
"object": "llm",
|
||||
"caseId": "root-sqrt2",
|
||||
"language": "zh",
|
||||
"reference": "\\sqrt{2}\\approx 1.414",
|
||||
"spoken": "根号二约等于一点四一四。",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the formula is read naturally in Mandarin Chinese."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|rel-leq|zh",
|
||||
"object": "sre",
|
||||
"caseId": "rel-leq",
|
||||
"language": "zh",
|
||||
"reference": "x \\leq 3",
|
||||
"spoken": "x is less than or equal to 3",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The comparison is read in English rather than Mandarin Chinese."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Reads in English, not Chinese. A reading in another language does not pass."
|
||||
},
|
||||
{
|
||||
"id": "llm|set-builder|en",
|
||||
"object": "llm",
|
||||
"caseId": "set-builder",
|
||||
"language": "en",
|
||||
"reference": "\\{x \\mid x > 0\\}",
|
||||
"spoken": "Open brace, x such that x is greater than zero, close brace.",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "It spells out the opening and closing braces around the set."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "The braces of a set need no reading."
|
||||
},
|
||||
{
|
||||
"id": "rule|mb-1836666|en",
|
||||
"object": "rule",
|
||||
"caseId": "mb-1836666",
|
||||
"language": "en",
|
||||
"reference": "\\lVert\\ \\ # 1\\ \\rVert { C^ { -1 } } \\leq c_0",
|
||||
"spoken": "lVert # 1 rVert C^ -1 leq c_0",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading leaves the LaTeX commands and symbols `lVert`, `rVert`, `^`, `leq`, and `_` spelled out rather than reading their mathematical meaning."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "The reading is not understandable: it leaves command names such as lVert."
|
||||
},
|
||||
{
|
||||
"id": "llm|mb-1167649|en",
|
||||
"object": "llm",
|
||||
"caseId": "mb-1167649",
|
||||
"language": "en",
|
||||
"reference": "\\Sigma_ { B_T }",
|
||||
"spoken": "Capital sigma, subscript, open brace, capital B, subscript capital T, close brace.",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading spells the braces and subscript markup, and adds unnecessary case labels for sigma, B, and T."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|mb-37396|en",
|
||||
"object": "sre",
|
||||
"caseId": "mb-37396",
|
||||
"language": "en",
|
||||
"reference": "\\qKoec_i ( x_1 ) \\neq +\\infty",
|
||||
"spoken": "backslash qKoec sub i times open paren x sub 1 close paren is not equal to positive infinity",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "It says the LaTeX command name “backslash qKoec” and spells out the parentheses around the function argument."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|unit-g|ja",
|
||||
"object": "sre",
|
||||
"caseId": "unit-g",
|
||||
"language": "ja",
|
||||
"reference": "9.8\\,\\mathrm{m/s^2}",
|
||||
"spoken": "9.8 empty meters divided by seconds squared",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "“Empty” and “divided by” spell spacing and fraction notation rather than reading the unit naturally in Japanese."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "\"empty\" is not a reading."
|
||||
},
|
||||
{
|
||||
"id": "llm|case-sens-nN|ja",
|
||||
"object": "llm",
|
||||
"caseId": "case-sens-nN",
|
||||
"language": "ja",
|
||||
"reference": "N=n+1",
|
||||
"spoken": "大文字のエヌ、イコール、小文字のエヌ、プラス、一。",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "The case words for N and n are natural because both cases appear in the formula; no part is spelled."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Shorter case words are enough, unless the formula has several variants of n."
|
||||
},
|
||||
{
|
||||
"id": "sre|chem-co2|ja",
|
||||
"object": "sre",
|
||||
"caseId": "chem-co2",
|
||||
"language": "ja",
|
||||
"reference": "\\ce{C + O2 -> CO2}",
|
||||
"spoken": "Celsius plus normal O sub 2 right arrow Celsius normal O sub 2",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading is mostly English, and “sub” and “right arrow” spell the notation rather than reading it in Japanese."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Reads C as Celsius, and the oxygen molecule is lost."
|
||||
},
|
||||
{
|
||||
"id": "sre|chem-co|ja",
|
||||
"object": "sre",
|
||||
"caseId": "chem-co",
|
||||
"language": "ja",
|
||||
"reference": "\\ce{2C + O2 -> 2CO}",
|
||||
"spoken": "2 empty Celsius plus normal O sub 2 right arrow 2 empty Celsius normal O",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading uses the English phrases “empty Celsius,” “plus,” “normal,” and “right arrow” instead of Japanese readings."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Reads the formula symbols, not the chemical formula."
|
||||
},
|
||||
{
|
||||
"id": "rule|lim-fx-inf|ja",
|
||||
"object": "rule",
|
||||
"caseId": "lim-fx-inf",
|
||||
"language": "ja",
|
||||
"reference": "f(x) \\to \\infty",
|
||||
"spoken": "エフエックスを無限大に近づける",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the reading uses natural Japanese mathematical phrasing."
|
||||
},
|
||||
"label": "natural",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|chem-hydrate|zh",
|
||||
"object": "rule",
|
||||
"caseId": "chem-hydrate",
|
||||
"language": "zh",
|
||||
"reference": "\\ce{CuSO4*5H2O}",
|
||||
"spoken": "CuSO4 点 5 H2O",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "It leaves the chemical formulas “CuSO4” and “H2O” as source notation instead of reading the element symbols letter by letter."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|unit-temp|zh",
|
||||
"object": "sre",
|
||||
"caseId": "unit-temp",
|
||||
"language": "zh",
|
||||
"reference": "25^\\circ\\mathrm{C}",
|
||||
"spoken": "25 raised to the composed with power Celsius",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading uses the English phrase “raised to the ... power” instead of Mandarin Chinese."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": "Should be 25 摄氏度."
|
||||
},
|
||||
{
|
||||
"id": "llm|chem-h2o2|zh",
|
||||
"object": "llm",
|
||||
"caseId": "chem-h2o2",
|
||||
"language": "zh",
|
||||
"reference": "\\ce{2H2O2 -> 2H2O + O2}",
|
||||
"spoken": "系数二，过氧化氢分子，化学式中氢的下标是二，氧的下标是二，反应生成系数二的水分子，化学式中氢的下标是二，再加一个氧气分子，化学式中氧的下标是二。",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the coefficients, compound names, and subscript readings use natural Mandarin terminology."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Too mechanical."
|
||||
},
|
||||
{
|
||||
"id": "sre|lim-recip|zh",
|
||||
"object": "sre",
|
||||
"caseId": "lim-recip",
|
||||
"language": "zh",
|
||||
"reference": "\\lim_{x \\to \\infty}\\frac{1}{x}=0",
|
||||
"spoken": "lim over x right arrow infinity of 1 over x equals 0",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading is mostly English, and “right arrow” spells the arrow symbol."
|
||||
},
|
||||
"label": "mixed",
|
||||
"note": "Unsure whether lim reads this way."
|
||||
},
|
||||
{
|
||||
"id": "sre|case-sens-xX|zh",
|
||||
"object": "sre",
|
||||
"caseId": "case-sens-xX",
|
||||
"language": "zh",
|
||||
"reference": "f: X \\to Y, x \\mapsto y",
|
||||
"spoken": "f colon X right arrow Y comma x right arrow from bar y",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading spells the symbols as “colon,” “right arrow,” “comma,” and “right arrow from bar” instead of reading their meaning in Mandarin."
|
||||
},
|
||||
"label": "spelled",
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|frac-mb-sigma|ko",
|
||||
"object": "llm",
|
||||
"caseId": "frac-mb-sigma",
|
||||
"language": "ko",
|
||||
"reference": "\\sigma_{0}=\\frac{f-1}{2}",
|
||||
"spoken": "시그마 아래첨자 영은 분수 시작, 분자 에프 빼기 일, 분모 이, 분수 끝과 같다.",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The reading spells out fraction markup with “분수 시작” and “분수 끝.”"
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|lim-recip|ko",
|
||||
"object": "llm",
|
||||
"caseId": "lim-recip",
|
||||
"language": "ko",
|
||||
"reference": "\\lim_{x \\to \\infty}\\frac{1}{x}=0",
|
||||
"spoken": "리미트 엑스가 무한대로 갈 때, 분수의 분자는 일, 분모는 엑스, 그 값은 영입니다.",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the limit and fraction are read in natural Korean mathematical language."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|root-sqrt2|ko",
|
||||
"object": "llm",
|
||||
"caseId": "root-sqrt2",
|
||||
"language": "ko",
|
||||
"reference": "\\sqrt{2}\\approx 1.414",
|
||||
"spoken": "제곱근 이는 약 일 점 사 일 사입니다.",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the formula is read naturally in Korean."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|set-builder|ko",
|
||||
"object": "llm",
|
||||
"caseId": "set-builder",
|
||||
"language": "ko",
|
||||
"reference": "\\{x \\mid x > 0\\}",
|
||||
"spoken": "왼쪽 중괄호, 엑스, 세로줄, 엑스가 영보다 크다, 오른쪽 중괄호.",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading spells out the set’s opening and closing braces and names the vertical bar instead of reading the set-builder structure naturally."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "llm|set-cup|ko",
|
||||
"object": "llm",
|
||||
"caseId": "set-cup",
|
||||
"language": "ko",
|
||||
"reference": "A \\cup B=\\{1,2,3\\}",
|
||||
"spoken": "에이는 비와의 합집합이며, 중괄호를 열고 일, 이, 삼을 차례로 적은 뒤 중괄호를 닫은 집합과 같다.",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "It spells the set braces by saying to open and close the curly braces."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|case-sens-aA|ko",
|
||||
"object": "rule",
|
||||
"caseId": "case-sens-aA",
|
||||
"language": "ko",
|
||||
"reference": "a \\in A",
|
||||
"spoken": "소문자 에이는 대문자 에이의 원소",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the reading naturally distinguishes lowercase a from uppercase A."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|lim-e|ko",
|
||||
"object": "rule",
|
||||
"caseId": "lim-e",
|
||||
"language": "ko",
|
||||
"reference": "\\lim_{n \\to \\infty}\\left(1+\\frac{1}{n}\\right)^{n}=e",
|
||||
"spoken": "엔이 무한대로 갈 때 괄호 일 더하기 분자가 일, 분모가 엔인 분수 괄호 닫고의 엔 제곱의 극한은 e",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The reading is natural; it uses the standard fraction wording and names the meaningful parentheses around the compound expression."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|prob-fm-loss|ko",
|
||||
"object": "rule",
|
||||
"caseId": "prob-fm-loss",
|
||||
"language": "ko",
|
||||
"reference": "\\mathcal{L}_{FM}(\\theta) = \\mathbb{E}_{t \\sim \\mathcal{U}[0,1],\\, x \\sim p_t}\\left[\\|v_\\theta(x,t) - u_t(x)\\|^2\\right]",
|
||||
"spoken": "엘 아래첨자 에프 엠 세타는 티가 대문자 유 영부터 일까지의 닫힌구간을 따름, 엑스가 피 아래첨자 티를 따름에 대한 브이 아래첨자 세타 괄호 엑스, 티 괄호 닫고 빼기 소문자 유 아래첨자 티 엑스의 노름의 제곱의 기댓값",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the subscripts, function arguments, interval, and norm are read naturally in Korean."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|set-builder|ja",
|
||||
"object": "rule",
|
||||
"caseId": "set-builder",
|
||||
"language": "ja",
|
||||
"reference": "\\{x \\mid x > 0\\}",
|
||||
"spoken": "エックス 大なり ゼロようなエックスの集合",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; it naturally reads the set of x such that x is greater than zero."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|set-interval|ko",
|
||||
"object": "rule",
|
||||
"caseId": "set-interval",
|
||||
"language": "ko",
|
||||
"reference": "x \\in (0, b]",
|
||||
"spoken": "엑스는 영 초과 비 이하인 구간의 원소",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the interval and membership are read naturally in Korean."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "rule|sum-gauss|ja",
|
||||
"object": "rule",
|
||||
"caseId": "sum-gauss",
|
||||
"language": "ja",
|
||||
"reference": "\\sum_{k=1}^{n} k=\\frac{n(n+1)}{2}",
|
||||
"spoken": "ケーがいちからエヌまでのケーの和 イコール 分子がエヌかっこエヌプラスいちかっことじ、分母がにの分数",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The spoken reading spells the parentheses in the denominator expression by naming them."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|case-sens-xX|ko",
|
||||
"object": "sre",
|
||||
"caseId": "case-sens-xX",
|
||||
"language": "ko",
|
||||
"reference": "f: X \\to Y, x \\mapsto y",
|
||||
"spoken": "f 콜론 X 오른쪽 화살표 Y 콤마 x 세로선이 붙은 오른쪽 화살표 y",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The reading spells the mapsto symbol as “세로선이 붙은 오른쪽 화살표” instead of reading its mapping meaning."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|chem-water|ko",
|
||||
"object": "sre",
|
||||
"caseId": "chem-water",
|
||||
"language": "ko",
|
||||
"reference": "\\ce{2H2 + O2 -> 2H2O}",
|
||||
"spoken": "2 기본 H 의 아래첨자 2 더하기 기본 O 아래첨자 2 오른쪽 화살표 2 기본 H 의 아래첨자 2 기본 O",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "“기본” spells font formatting, and “오른쪽 화살표” names the arrow symbol rather than reading its meaning."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|lim-recip|ko",
|
||||
"object": "sre",
|
||||
"caseId": "lim-recip",
|
||||
"language": "ko",
|
||||
"reference": "\\lim_{x \\to \\infty}\\frac{1}{x}=0",
|
||||
"spoken": "x 오른쪽 화살표 무한대 에 대한 리미트 의 x 분의 1 은 0",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The reading names the arrow as a right arrow instead of reading the limit condition naturally."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|misc-binom|ko",
|
||||
"object": "sre",
|
||||
"caseId": "misc-binom",
|
||||
"language": "ko",
|
||||
"reference": "(a+b)^2=a^2+2ab+b^2",
|
||||
"spoken": "괄호 열고 a 더하기 b 괄호 닫고 제곱 은 a 제곱 더하기 2 a b 더하기 b 제곱",
|
||||
"judge": {
|
||||
"reading": "spelled",
|
||||
"reason": "The reading names the main parentheses around a plus b instead of using a natural phrase for the squared sum."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|sum-sq|ko",
|
||||
"object": "sre",
|
||||
"caseId": "sum-sq",
|
||||
"language": "ko",
|
||||
"reference": "\\sum_{i=1}^{n} i^{2}",
|
||||
"spoken": "시그마 i 는 1 에서 n 까지 i 제곱",
|
||||
"judge": {
|
||||
"reading": "natural",
|
||||
"reason": "No part is spelled; the summation notation is read in a conventional Korean shorthand."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
},
|
||||
{
|
||||
"id": "sre|unit-temp|ko",
|
||||
"object": "sre",
|
||||
"caseId": "unit-temp",
|
||||
"language": "ko",
|
||||
"reference": "25^\\circ\\mathrm{C}",
|
||||
"spoken": "25 의 합성 함수 제곱 섭씨",
|
||||
"judge": {
|
||||
"reading": "mixed",
|
||||
"reason": "The reading spells the superscript as “제곱,” which changes the degree symbol into an exponent."
|
||||
},
|
||||
"label": null,
|
||||
"note": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
import type { CalibrationFile, CalibrationItem } from './types'
|
||||
|
||||
import process from 'node:process'
|
||||
|
||||
import { glob, readFile, rename, stat, writeFile } from 'node:fs/promises'
|
||||
|
||||
import { formulaSpeechCases } from '../../../cases/formula-speech'
|
||||
import { mathBridgeCases } from '../../../cases/math-bridge'
|
||||
|
||||
/**
|
||||
* Collects the readings that a person labels in the calibration tool, see ./README.md.
|
||||
*
|
||||
* It reads the newest `result.json` of each case from the reports.
|
||||
*
|
||||
* - Without `--pick`, it keeps the items of `calibration.json` and refreshes only their judge verdicts, so the set
|
||||
* stays the same while the judge prompt changes. Agreement is then measured on the same readings every time.
|
||||
* - With `--pick`, or when the file is empty, it picks {@link perLanguage} readings for each language. The picks spread
|
||||
* over the test objects and over the verdicts of the judge. Labeled items always stay.
|
||||
*
|
||||
* Run: pnpm -F @proj-airi/evals-tts-speech-formula run judge:collect [-- --pick]
|
||||
*/
|
||||
|
||||
const reportsRoot = new URL('../../../../.vieval/reports/', import.meta.url)
|
||||
const calibrationFile = new URL('./calibration.json', import.meta.url)
|
||||
/** The test objects that write their own words. `raw` is left out: it only passes the source through. */
|
||||
const objects = ['rule', 'llm', 'sre'] as const
|
||||
const verdicts = ['natural', 'mixed', 'spelled'] as const
|
||||
const perLanguage = 15
|
||||
const seed = 20261020
|
||||
/** Cases of the current datasets. Reports keep results of cases that a newer sample dropped, and those are left out. */
|
||||
const currentCases = new Set([...formulaSpeechCases, ...mathBridgeCases].map(item => item.id))
|
||||
|
||||
interface ResultFile {
|
||||
id: string
|
||||
language: string
|
||||
tts: string
|
||||
reference: unknown
|
||||
spoken: string
|
||||
reading?: string
|
||||
verdicts?: { reason: string, reading: string }[]
|
||||
}
|
||||
|
||||
/** FNV-1a over the seed and a key, so the pick order is random but repeats with the same seed. */
|
||||
function order(key: string): number {
|
||||
let hash = 2166136261 ^ seed
|
||||
for (const char of key) {
|
||||
hash ^= char.codePointAt(0)!
|
||||
hash = Math.imul(hash, 16777619)
|
||||
}
|
||||
return hash >>> 0
|
||||
}
|
||||
|
||||
/** The newest judged reading of each object, case, and language, without TTS. */
|
||||
async function newestReadings(): Promise<CalibrationItem[]> {
|
||||
const newest = new Map<string, { time: number, item: CalibrationItem }>()
|
||||
for (const object of objects) {
|
||||
for await (const path of glob(`${object}/*/tts-speech-formula/attempt-*/run-*/artifacts/*/*/result.json`, { cwd: reportsRoot.pathname })) {
|
||||
const file = new URL(path, reportsRoot)
|
||||
const result = JSON.parse(await readFile(file, 'utf8')) as ResultFile
|
||||
if (!currentCases.has(result.id) || result.tts !== 'none' || result.reading === undefined || typeof result.reference !== 'string' || !result.spoken.trim())
|
||||
continue
|
||||
const id = `${object}|${result.id}|${result.language}`
|
||||
const time = (await stat(file)).mtimeMs
|
||||
if ((newest.get(id)?.time ?? 0) >= time)
|
||||
continue
|
||||
newest.set(id, {
|
||||
time,
|
||||
item: {
|
||||
id,
|
||||
object,
|
||||
caseId: result.id,
|
||||
language: result.language,
|
||||
reference: result.reference,
|
||||
spoken: result.spoken,
|
||||
judge: { reading: result.reading, reason: result.verdicts?.[0]?.reason ?? '' },
|
||||
label: null,
|
||||
note: '',
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
return [...newest.values()].map(({ item }) => item)
|
||||
}
|
||||
|
||||
/**
|
||||
* Picks readings of one language in rounds over each object and each verdict, so a verdict that the judge rarely gives
|
||||
* still gets picks. A spoken text that is already picked is skipped, because two objects can say the same words.
|
||||
*/
|
||||
function pick(readings: readonly CalibrationItem[]): CalibrationItem[] {
|
||||
const buckets = new Map<string, CalibrationItem[]>()
|
||||
for (const item of [...readings].sort((left, right) => order(left.id) - order(right.id))) {
|
||||
const key = `${item.object}|${item.judge.reading}`
|
||||
buckets.set(key, [...(buckets.get(key) ?? []), item])
|
||||
}
|
||||
const picked: CalibrationItem[] = []
|
||||
const spoken = new Set<string>()
|
||||
while (picked.length < perLanguage && [...buckets.values()].some(bucket => bucket.length > 0)) {
|
||||
for (const verdict of verdicts) {
|
||||
for (const object of objects) {
|
||||
const item = buckets.get(`${object}|${verdict}`)?.shift()
|
||||
if (!item || spoken.has(item.spoken) || picked.length >= perLanguage)
|
||||
continue
|
||||
spoken.add(item.spoken)
|
||||
picked.push(item)
|
||||
}
|
||||
}
|
||||
}
|
||||
return picked
|
||||
}
|
||||
|
||||
/** Written next to the file and renamed over it, as the labeling server does, so a reader never sees half a file. */
|
||||
async function write(items: CalibrationItem[]) {
|
||||
const partial = new URL('./calibration.json.partial', import.meta.url)
|
||||
await writeFile(partial, `${JSON.stringify({ seed, items } satisfies CalibrationFile, null, 2)}\n`)
|
||||
await rename(partial, calibrationFile)
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const previous = await readFile(calibrationFile, 'utf8').then(text => JSON.parse(text) as CalibrationFile, () => ({ items: [] }))
|
||||
const labeled = new Map(previous.items.filter(item => item.label !== null).map(item => [item.id, item]))
|
||||
|
||||
const readings = await newestReadings()
|
||||
if (previous.items.length > 0 && !process.argv.includes('--pick')) {
|
||||
const newest = new Map(readings.map(item => [item.id, item]))
|
||||
const items = previous.items.map(item => ({ ...item, judge: newest.get(item.id)?.judge ?? item.judge }))
|
||||
await write(items)
|
||||
process.stdout.write(`refreshed the judge verdicts of ${items.length} readings, ${labeled.size} labeled\n`)
|
||||
return
|
||||
}
|
||||
|
||||
const languages = [...new Set(readings.map(item => item.language))].sort()
|
||||
const items = languages.flatMap(language => pick(readings.filter(item => item.language === language)))
|
||||
// A labeled item keeps its label and note. Its judge verdict is the newest one.
|
||||
.map(item => ({ ...item, label: labeled.get(item.id)?.label ?? null, note: labeled.get(item.id)?.note ?? '' }))
|
||||
// Labeled items that the new pick left out stay, so collecting again never loses work.
|
||||
const kept = new Set(items.map(item => item.id))
|
||||
for (const item of labeled.values()) {
|
||||
if (!kept.has(item.id))
|
||||
items.push(item)
|
||||
}
|
||||
|
||||
await write(items)
|
||||
process.stdout.write(`collected ${items.length} readings from ${readings.length}, ${labeled.size} labeled\n`)
|
||||
}
|
||||
|
||||
void main()
|
||||
@@ -0,0 +1,74 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Judge calibration</title>
|
||||
<link rel="stylesheet" href="./style.css" />
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<header class="top">
|
||||
<div class="title">
|
||||
<h1>Judge calibration</h1>
|
||||
<p class="stats" id="stats">Loading…</p>
|
||||
<div class="progress" aria-hidden="true"><div class="progress-bar" id="progress"></div></div>
|
||||
</div>
|
||||
<div class="filters">
|
||||
<label class="field">
|
||||
<span>Language</span>
|
||||
<select id="language"><option value="">All languages</option></select>
|
||||
</label>
|
||||
<label class="field">
|
||||
<span>Show</span>
|
||||
<select id="show">
|
||||
<option value="unlabeled">Unlabeled first</option>
|
||||
<option value="all">All</option>
|
||||
<option value="disagree">Disagreements</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="stage" id="stage" aria-live="polite"></section>
|
||||
|
||||
<nav class="pager">
|
||||
<button class="button" id="previous" type="button"><kbd>←</kbd> Previous</button>
|
||||
<span class="position" id="position"></span>
|
||||
<button class="button" id="next" type="button">Next <kbd>Enter</kbd></button>
|
||||
</nav>
|
||||
|
||||
<p class="hint">
|
||||
<kbd>1</kbd> natural · <kbd>2</kbd> mixed · <kbd>3</kbd> spelled · press the same key again to clear ·
|
||||
the judge verdict shows after you label.
|
||||
</p>
|
||||
</main>
|
||||
|
||||
<div class="toast" id="toast" role="status"></div>
|
||||
|
||||
<template id="card-template">
|
||||
<article class="card">
|
||||
<div class="meta"></div>
|
||||
<div class="formula"></div>
|
||||
<code class="source"></code>
|
||||
<p class="spoken"></p>
|
||||
<div class="choices" role="radiogroup" aria-label="Reading style">
|
||||
<button class="choice" type="button" role="radio" data-label="natural"><kbd>1</kbd><span class="choice-name">Natural</span><span class="choice-help">says the meaning</span></button>
|
||||
<button class="choice" type="button" role="radio" data-label="mixed"><kbd>2</kbd><span class="choice-name">Mixed</span><span class="choice-help">one or two spelled parts</span></button>
|
||||
<button class="choice" type="button" role="radio" data-label="spelled"><kbd>3</kbd><span class="choice-name">Spelled</span><span class="choice-help">reads the markup</span></button>
|
||||
</div>
|
||||
<label class="note">
|
||||
<span>Note</span>
|
||||
<textarea rows="2" placeholder="Which part is spelled, or how it should be read. Markdown works."></textarea>
|
||||
</label>
|
||||
<div class="note-preview markdown" hidden></div>
|
||||
<div class="judge" hidden></div>
|
||||
<div class="tools">
|
||||
<button class="button ghost" type="button" data-action="copy">Copy translation prompt</button>
|
||||
</div>
|
||||
</article>
|
||||
</template>
|
||||
|
||||
<script type="module" src="./main.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,266 @@
|
||||
/// <reference types="vite/client" />
|
||||
|
||||
import type { CalibrationFile, CalibrationItem, ReadingLabel } from './types'
|
||||
|
||||
import katex from 'katex'
|
||||
import rehypeStringify from 'rehype-stringify'
|
||||
import remarkParse from 'remark-parse'
|
||||
import remarkRehype from 'remark-rehype'
|
||||
|
||||
import { unified } from 'unified'
|
||||
|
||||
import 'katex/contrib/mhchem'
|
||||
import 'katex/dist/katex.min.css'
|
||||
|
||||
const labels: readonly ReadingLabel[] = ['natural', 'mixed', 'spelled']
|
||||
const languageNames: Readonly<Record<string, string>> = { zh: 'Mandarin Chinese', en: 'English', ja: 'Japanese', ko: 'Korean' }
|
||||
|
||||
const stage = document.querySelector<HTMLElement>('#stage')!
|
||||
const template = document.querySelector<HTMLTemplateElement>('#card-template')!
|
||||
const stats = document.querySelector<HTMLElement>('#stats')!
|
||||
const progress = document.querySelector<HTMLElement>('#progress')!
|
||||
const position = document.querySelector<HTMLElement>('#position')!
|
||||
const toast = document.querySelector<HTMLElement>('#toast')!
|
||||
const languageSelect = document.querySelector<HTMLSelectElement>('#language')!
|
||||
const showSelect = document.querySelector<HTMLSelectElement>('#show')!
|
||||
const previousButton = document.querySelector<HTMLButtonElement>('#previous')!
|
||||
const nextButton = document.querySelector<HTMLButtonElement>('#next')!
|
||||
|
||||
/**
|
||||
* Renders the markdown of a judge reason or a note. remark-rehype drops raw HTML, so the text cannot inject markup.
|
||||
*/
|
||||
const markdown = unified().use(remarkParse).use(remarkRehype).use(rehypeStringify)
|
||||
function renderMarkdown(text: string): string {
|
||||
return String(markdown.processSync(text))
|
||||
}
|
||||
|
||||
let items: CalibrationItem[] = []
|
||||
/**
|
||||
* The items in the order of the current filters. It is computed when the page loads or a filter changes, never when a
|
||||
* label changes, so labeling an item never moves the card to another item.
|
||||
*/
|
||||
let order: CalibrationItem[] = []
|
||||
let index = 0
|
||||
|
||||
function computeOrder() {
|
||||
const language = languageSelect.value
|
||||
const show = showSelect.value
|
||||
const filtered = items.filter(item => !language || item.language === language)
|
||||
if (show === 'disagree')
|
||||
order = filtered.filter(item => item.label !== null && item.label !== item.judge.reading)
|
||||
else if (show === 'unlabeled')
|
||||
order = [...filtered.filter(item => item.label === null), ...filtered.filter(item => item.label !== null)]
|
||||
else
|
||||
order = filtered
|
||||
index = 0
|
||||
}
|
||||
|
||||
let toastTimer: ReturnType<typeof setTimeout> | undefined
|
||||
function showToast(message: string, error = false) {
|
||||
toast.textContent = message
|
||||
toast.classList.toggle('error', error)
|
||||
toast.classList.add('visible')
|
||||
clearTimeout(toastTimer)
|
||||
toastTimer = setTimeout(() => toast.classList.remove('visible'), error ? 4000 : 1400)
|
||||
}
|
||||
|
||||
function renderStats() {
|
||||
const labeled = items.filter(item => item.label !== null)
|
||||
const agree = labeled.filter(item => item.label === item.judge.reading).length
|
||||
const rate = labeled.length ? `${Math.round(100 * agree / labeled.length)}%` : '–'
|
||||
stats.textContent = `${labeled.length} of ${items.length} labeled · the judge agrees on ${agree} of ${labeled.length} (${rate})`
|
||||
progress.style.width = `${items.length ? 100 * labeled.length / items.length : 0}%`
|
||||
}
|
||||
|
||||
/** Renders the formula with KaTeX. A formula that KaTeX cannot render shows a short notice above its source. */
|
||||
function renderFormula(target: HTMLElement, tex: string) {
|
||||
try {
|
||||
target.innerHTML = katex.renderToString(tex, { displayMode: true, throwOnError: true })
|
||||
}
|
||||
catch {
|
||||
target.innerHTML = '<p class="formula-error">KaTeX cannot render this source.</p>'
|
||||
}
|
||||
}
|
||||
|
||||
/** A prompt for any agent: translate the reading and say how a native speaker would hear it. */
|
||||
function translationPrompt(item: CalibrationItem): string {
|
||||
return [
|
||||
`This is a ${languageNames[item.language] ?? item.language} reading of a math formula for a text-to-speech voice.`,
|
||||
`LaTeX: ${item.reference}`,
|
||||
`Reading: ${item.spoken}`,
|
||||
'Translate the reading word by word into Chinese. Then say whether a native speaker hears it as a teacher reading the meaning, or as someone spelling the markup, such as naming every bracket or saying command names.',
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
/** The scripts that a reading in each language must use. A reading without them is in another language. */
|
||||
const languageScripts: Readonly<Record<string, RegExp>> = {
|
||||
zh: /\p{Script=Han}/u,
|
||||
ja: /[\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Han}]/u,
|
||||
ko: /\p{Script=Hangul}/u,
|
||||
}
|
||||
|
||||
function chip(text: string, strong = false): HTMLSpanElement {
|
||||
const element = document.createElement('span')
|
||||
element.className = strong ? 'chip strong' : 'chip'
|
||||
element.textContent = text
|
||||
return element
|
||||
}
|
||||
|
||||
/**
|
||||
* Updates only what a label changes: the chosen choice, the verdict of the judge, and the stats. The note box stays,
|
||||
* so its focus and cursor stay too.
|
||||
*/
|
||||
function renderLabel(card: HTMLElement, item: CalibrationItem) {
|
||||
for (const choice of card.querySelectorAll<HTMLButtonElement>('.choice'))
|
||||
choice.setAttribute('aria-checked', String(choice.dataset.label === item.label))
|
||||
|
||||
const judge = card.querySelector<HTMLElement>('.judge')!
|
||||
// The verdict shows only after the person labels, so it does not decide the label for them.
|
||||
judge.hidden = item.label === null
|
||||
if (item.label !== null) {
|
||||
const agree = item.label === item.judge.reading
|
||||
judge.className = `judge ${agree ? 'agree' : 'disagree'}`
|
||||
judge.innerHTML = `<div class="judge-head">${agree ? '✓ The judge agrees' : '✗ The judge disagrees'}: ${item.judge.reading}</div>`
|
||||
+ `<div class="markdown">${renderMarkdown(item.judge.reason)}</div>`
|
||||
}
|
||||
renderStats()
|
||||
}
|
||||
|
||||
function renderNotePreview(card: HTMLElement, item: CalibrationItem) {
|
||||
const preview = card.querySelector<HTMLElement>('.note-preview')!
|
||||
preview.hidden = !item.note.trim()
|
||||
preview.innerHTML = renderMarkdown(item.note)
|
||||
}
|
||||
|
||||
/** Renders the card of the current item. `direction` slides it in from the side it comes from. */
|
||||
function renderCard(direction: 'next' | 'previous' | 'none' = 'none') {
|
||||
index = Math.min(Math.max(index, 0), Math.max(order.length - 1, 0))
|
||||
const item = order[index]
|
||||
previousButton.disabled = index <= 0
|
||||
nextButton.disabled = index >= order.length - 1
|
||||
position.textContent = order.length ? `${index + 1} / ${order.length}` : ''
|
||||
renderStats()
|
||||
if (!item) {
|
||||
stage.innerHTML = '<div class="card empty">No readings match the filters.</div>'
|
||||
return
|
||||
}
|
||||
|
||||
const card = template.content.firstElementChild!.cloneNode(true) as HTMLElement
|
||||
if (direction !== 'none')
|
||||
card.classList.add(`enter-${direction}`)
|
||||
card.querySelector('.meta')!.append(
|
||||
chip(languageNames[item.language] ?? item.language, true),
|
||||
chip(item.object),
|
||||
chip(item.caseId),
|
||||
)
|
||||
renderFormula(card.querySelector<HTMLElement>('.formula')!, item.reference)
|
||||
card.querySelector('.source')!.textContent = item.reference
|
||||
const spoken = card.querySelector<HTMLElement>('.spoken')!
|
||||
spoken.textContent = item.spoken
|
||||
spoken.lang = item.language
|
||||
// A test object can answer in another language, for example sre speaks only English. The text is shown as it came,
|
||||
// so the warning says that this is the reading itself and not a display error.
|
||||
const script = languageScripts[item.language]
|
||||
if (script && !script.test(item.spoken)) {
|
||||
const warning = document.createElement('p')
|
||||
warning.className = 'language-warning'
|
||||
warning.textContent = `This reading is not in ${languageNames[item.language] ?? item.language}. The test object wrote it this way. A reading in another language does not pass.`
|
||||
spoken.after(warning)
|
||||
}
|
||||
|
||||
for (const choice of card.querySelectorAll<HTMLButtonElement>('.choice'))
|
||||
choice.addEventListener('click', () => choose(card, item, choice.dataset.label as ReadingLabel))
|
||||
|
||||
const noteBox = card.querySelector<HTMLTextAreaElement>('textarea')!
|
||||
noteBox.value = item.note
|
||||
// The note is kept on the item as the person types, so a label chosen before the note is saved carries the new note.
|
||||
noteBox.addEventListener('input', () => {
|
||||
item.note = noteBox.value
|
||||
renderNotePreview(card, item)
|
||||
})
|
||||
noteBox.addEventListener('change', () => void save(item, 'Note saved'))
|
||||
|
||||
card.querySelector<HTMLButtonElement>('[data-action="copy"]')!.addEventListener('click', () => {
|
||||
navigator.clipboard.writeText(translationPrompt(item))
|
||||
.then(() => showToast('Translation prompt copied'))
|
||||
.catch(() => showToast('The browser blocked the clipboard', true))
|
||||
})
|
||||
|
||||
stage.replaceChildren(card)
|
||||
renderLabel(card, item)
|
||||
renderNotePreview(card, item)
|
||||
}
|
||||
|
||||
/** Saves of the page, one at a time, so a later save never lands before an earlier one. */
|
||||
let saves: Promise<unknown> = Promise.resolve()
|
||||
|
||||
/** Saves the label and the note of an item to calibration.json through the dev server. */
|
||||
function save(item: CalibrationItem, message: string): Promise<void> {
|
||||
const body = JSON.stringify({ id: item.id, label: item.label, note: item.note })
|
||||
const run = saves.then(async () => {
|
||||
const response = await fetch('/api/items', { method: 'POST', body })
|
||||
if (!response.ok)
|
||||
throw new Error(`${response.status} ${await response.text()}`)
|
||||
})
|
||||
saves = run.catch(() => undefined)
|
||||
return run.then(
|
||||
() => showToast(message),
|
||||
(error: unknown) => showToast(`Not saved: ${String(error)}`, true),
|
||||
)
|
||||
}
|
||||
|
||||
/** Chooses a label, or clears it when it is the label the item already has. The card stays on the item. */
|
||||
function choose(card: HTMLElement, item: CalibrationItem, label: ReadingLabel) {
|
||||
item.label = item.label === label ? null : label
|
||||
const choice = card.querySelector<HTMLButtonElement>(`.choice[data-label="${label}"]`)!
|
||||
choice.classList.remove('pulse')
|
||||
// Reading the layout restarts the animation when the same choice is pressed twice in a row.
|
||||
void choice.offsetWidth
|
||||
choice.classList.add('pulse')
|
||||
renderLabel(card, item)
|
||||
void save(item, item.label === null ? 'Label cleared' : `Saved as ${item.label}`)
|
||||
}
|
||||
|
||||
function move(step: number) {
|
||||
const next = index + step
|
||||
if (next < 0 || next >= order.length)
|
||||
return
|
||||
index = next
|
||||
renderCard(step > 0 ? 'next' : 'previous')
|
||||
}
|
||||
|
||||
document.addEventListener('keydown', (event) => {
|
||||
if (event.target instanceof HTMLTextAreaElement || event.target instanceof HTMLSelectElement || event.isComposing || event.metaKey || event.ctrlKey || event.altKey)
|
||||
return
|
||||
const card = stage.querySelector<HTMLElement>('.card')
|
||||
const item = order[index]
|
||||
const chosen = ['1', '2', '3'].indexOf(event.key)
|
||||
if (card && item && chosen >= 0)
|
||||
choose(card, item, labels[chosen]!)
|
||||
else if (event.key === 'ArrowRight' || event.key === 'Enter')
|
||||
move(1)
|
||||
else if (event.key === 'ArrowLeft')
|
||||
move(-1)
|
||||
else
|
||||
return
|
||||
event.preventDefault()
|
||||
})
|
||||
previousButton.addEventListener('click', () => move(-1))
|
||||
nextButton.addEventListener('click', () => move(1))
|
||||
for (const select of [languageSelect, showSelect]) {
|
||||
select.addEventListener('change', () => {
|
||||
computeOrder()
|
||||
renderCard('none')
|
||||
})
|
||||
}
|
||||
|
||||
async function load() {
|
||||
const file = await (await fetch('/api/items')).json() as CalibrationFile
|
||||
items = file.items
|
||||
for (const language of [...new Set(items.map(item => item.language))].sort())
|
||||
languageSelect.append(new Option(languageNames[language] ?? language, language))
|
||||
computeOrder()
|
||||
renderCard()
|
||||
}
|
||||
|
||||
void load()
|
||||
@@ -0,0 +1,548 @@
|
||||
/*
|
||||
* Neutral surfaces with the AIRI default hue (220.44) as the accent, see DESIGN.md.
|
||||
* The label colors name a state only with the text and the check mark next to them.
|
||||
*/
|
||||
:root {
|
||||
color-scheme: light;
|
||||
--hue: 220.44;
|
||||
--bg: oklch(98.5% 0.004 var(--hue));
|
||||
--surface: oklch(100% 0 0);
|
||||
--surface-muted: oklch(96.5% 0.006 var(--hue));
|
||||
--border: oklch(91% 0.008 var(--hue));
|
||||
--border-strong: oklch(84% 0.012 var(--hue));
|
||||
--text: oklch(24% 0.015 var(--hue));
|
||||
--muted: oklch(52% 0.015 var(--hue));
|
||||
--primary: oklch(62% 0.14 var(--hue));
|
||||
--primary-soft: oklch(94% 0.04 var(--hue));
|
||||
--natural: oklch(62% 0.15 150);
|
||||
--natural-soft: oklch(95% 0.05 150);
|
||||
--mixed: oklch(70% 0.15 70);
|
||||
--mixed-soft: oklch(96% 0.05 80);
|
||||
--spelled: oklch(60% 0.18 25);
|
||||
--spelled-soft: oklch(95% 0.04 25);
|
||||
--shadow: 0 1px 2px oklch(20% 0.02 var(--hue) / 6%), 0 8px 24px oklch(20% 0.02 var(--hue) / 6%);
|
||||
--radius: 14px;
|
||||
--ease: cubic-bezier(0.2, 0.8, 0.2, 1);
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
color-scheme: dark;
|
||||
--bg: oklch(17% 0.01 var(--hue));
|
||||
--surface: oklch(21% 0.012 var(--hue));
|
||||
--surface-muted: oklch(25% 0.012 var(--hue));
|
||||
--border: oklch(30% 0.012 var(--hue));
|
||||
--border-strong: oklch(38% 0.015 var(--hue));
|
||||
--text: oklch(94% 0.006 var(--hue));
|
||||
--muted: oklch(70% 0.012 var(--hue));
|
||||
--primary: oklch(72% 0.13 var(--hue));
|
||||
--primary-soft: oklch(30% 0.05 var(--hue));
|
||||
--natural-soft: oklch(30% 0.06 150);
|
||||
--mixed-soft: oklch(32% 0.06 80);
|
||||
--spelled-soft: oklch(30% 0.06 25);
|
||||
--shadow: 0 1px 2px oklch(0% 0 0 / 30%), 0 8px 24px oklch(0% 0 0 / 25%);
|
||||
}
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font: 15px/1.6 ui-sans-serif, system-ui, -apple-system, 'Segoe UI', 'PingFang SC', 'Hiragino Sans', 'Apple SD Gothic Neo', sans-serif;
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
main {
|
||||
max-width: 880px;
|
||||
margin: 0 auto;
|
||||
padding: 28px 16px 72px;
|
||||
}
|
||||
|
||||
kbd {
|
||||
display: inline-grid;
|
||||
place-items: center;
|
||||
min-width: 1.6em;
|
||||
height: 1.6em;
|
||||
padding: 0 0.4em;
|
||||
border: 1px solid var(--border-strong);
|
||||
border-bottom-width: 2px;
|
||||
border-radius: 6px;
|
||||
background: var(--surface);
|
||||
color: var(--muted);
|
||||
font: 600 11px/1 ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
}
|
||||
|
||||
/* Header */
|
||||
|
||||
.top {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 16px;
|
||||
align-items: flex-end;
|
||||
justify-content: space-between;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.title {
|
||||
flex: 1 1 280px;
|
||||
}
|
||||
|
||||
h1 {
|
||||
margin: 0;
|
||||
font-size: 22px;
|
||||
font-weight: 650;
|
||||
letter-spacing: -0.01em;
|
||||
}
|
||||
|
||||
.stats {
|
||||
margin: 2px 0 10px;
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.progress {
|
||||
height: 6px;
|
||||
border-radius: 999px;
|
||||
background: var(--surface-muted);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.progress-bar {
|
||||
width: 0;
|
||||
height: 100%;
|
||||
border-radius: inherit;
|
||||
background: var(--primary);
|
||||
transition: width 400ms var(--ease);
|
||||
}
|
||||
|
||||
.filters {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.field {
|
||||
display: grid;
|
||||
gap: 4px;
|
||||
font-size: 12px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
select {
|
||||
appearance: none;
|
||||
min-width: 150px;
|
||||
padding: 8px 32px 8px 12px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
background: var(--surface) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='12' height='12' viewBox='0 0 24 24' fill='none' stroke='%23888' stroke-width='2.5' stroke-linecap='round' stroke-linejoin='round'%3E%3Cpath d='m6 9 6 6 6-6'/%3E%3C/svg%3E") no-repeat right 12px center;
|
||||
color: var(--text);
|
||||
font: inherit;
|
||||
font-size: 14px;
|
||||
cursor: pointer;
|
||||
transition: border-color 150ms var(--ease), box-shadow 150ms var(--ease);
|
||||
}
|
||||
|
||||
select:hover {
|
||||
border-color: var(--border-strong);
|
||||
}
|
||||
|
||||
/* Buttons */
|
||||
|
||||
.button,
|
||||
.choice {
|
||||
font: inherit;
|
||||
color: inherit;
|
||||
cursor: pointer;
|
||||
user-select: none;
|
||||
-webkit-tap-highlight-color: transparent;
|
||||
transition:
|
||||
background-color 150ms var(--ease),
|
||||
border-color 150ms var(--ease),
|
||||
color 150ms var(--ease),
|
||||
box-shadow 150ms var(--ease),
|
||||
transform 120ms var(--ease);
|
||||
}
|
||||
|
||||
.button {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 8px 14px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
background: var(--surface);
|
||||
font-size: 14px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.button:hover:not(:disabled) {
|
||||
border-color: var(--border-strong);
|
||||
background: var(--surface-muted);
|
||||
}
|
||||
|
||||
.button:active:not(:disabled),
|
||||
.choice:active {
|
||||
transform: scale(0.97);
|
||||
}
|
||||
|
||||
.button:disabled {
|
||||
opacity: 0.4;
|
||||
cursor: default;
|
||||
}
|
||||
|
||||
.button.ghost {
|
||||
border-color: transparent;
|
||||
background: transparent;
|
||||
color: var(--muted);
|
||||
padding-inline: 10px;
|
||||
}
|
||||
|
||||
.button.ghost:hover {
|
||||
background: var(--surface-muted);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
:is(.button, .choice, select, textarea):focus-visible {
|
||||
outline: none;
|
||||
box-shadow: 0 0 0 3px oklch(from var(--primary) l c h / 35%);
|
||||
}
|
||||
|
||||
/* Card */
|
||||
|
||||
.stage {
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.card {
|
||||
padding: 24px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
background: var(--surface);
|
||||
box-shadow: var(--shadow);
|
||||
}
|
||||
|
||||
.card.enter-next {
|
||||
animation: enter-next 260ms var(--ease);
|
||||
}
|
||||
|
||||
.card.enter-previous {
|
||||
animation: enter-previous 260ms var(--ease);
|
||||
}
|
||||
|
||||
@keyframes enter-next {
|
||||
from { opacity: 0; transform: translateX(24px); }
|
||||
}
|
||||
|
||||
@keyframes enter-previous {
|
||||
from { opacity: 0; transform: translateX(-24px); }
|
||||
}
|
||||
|
||||
.meta {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 6px;
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.chip {
|
||||
padding: 2px 10px;
|
||||
border-radius: 999px;
|
||||
background: var(--surface-muted);
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.chip.strong {
|
||||
background: var(--primary-soft);
|
||||
color: var(--primary);
|
||||
}
|
||||
|
||||
.formula {
|
||||
overflow-x: auto;
|
||||
padding: 8px 0;
|
||||
font-size: 1.15em;
|
||||
}
|
||||
|
||||
.formula .katex-display {
|
||||
margin: 0.6em 0;
|
||||
}
|
||||
|
||||
.formula-error {
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.source {
|
||||
display: block;
|
||||
padding: 8px 12px;
|
||||
border-radius: 8px;
|
||||
background: var(--surface-muted);
|
||||
color: var(--muted);
|
||||
font: 12px/1.5 ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-all;
|
||||
}
|
||||
|
||||
.spoken {
|
||||
margin: 18px 0;
|
||||
padding: 18px 20px;
|
||||
border-left: 3px solid var(--primary);
|
||||
border-radius: 4px 12px 12px 4px;
|
||||
background: var(--primary-soft);
|
||||
font-size: 21px;
|
||||
line-height: 1.7;
|
||||
word-break: break-word;
|
||||
}
|
||||
|
||||
.language-warning {
|
||||
margin: -8px 0 16px;
|
||||
padding: 8px 12px;
|
||||
border-radius: 8px;
|
||||
background: var(--spelled-soft);
|
||||
color: var(--text);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
/* Label choices */
|
||||
|
||||
.choices {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.choice {
|
||||
position: relative;
|
||||
display: grid;
|
||||
grid-template-columns: auto 1fr;
|
||||
grid-template-rows: auto auto;
|
||||
column-gap: 10px;
|
||||
align-items: center;
|
||||
padding: 12px 14px;
|
||||
border: 1.5px solid var(--border);
|
||||
border-radius: 12px;
|
||||
background: var(--surface);
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.choice kbd {
|
||||
grid-row: span 2;
|
||||
}
|
||||
|
||||
.choice-name {
|
||||
font-weight: 650;
|
||||
}
|
||||
|
||||
.choice-help {
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.choice:hover {
|
||||
border-color: var(--border-strong);
|
||||
background: var(--surface-muted);
|
||||
}
|
||||
|
||||
.choice[aria-checked='true']::after {
|
||||
content: '✓';
|
||||
position: absolute;
|
||||
top: 8px;
|
||||
right: 12px;
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.choice[data-label='natural'][aria-checked='true'] {
|
||||
border-color: var(--natural);
|
||||
background: var(--natural-soft);
|
||||
}
|
||||
|
||||
.choice[data-label='mixed'][aria-checked='true'] {
|
||||
border-color: var(--mixed);
|
||||
background: var(--mixed-soft);
|
||||
}
|
||||
|
||||
.choice[data-label='spelled'][aria-checked='true'] {
|
||||
border-color: var(--spelled);
|
||||
background: var(--spelled-soft);
|
||||
}
|
||||
|
||||
.choice[data-label='natural'][aria-checked='true']::after { color: var(--natural); }
|
||||
.choice[data-label='mixed'][aria-checked='true']::after { color: var(--mixed); }
|
||||
.choice[data-label='spelled'][aria-checked='true']::after { color: var(--spelled); }
|
||||
|
||||
/* A short pulse confirms the choice where the eye already is. */
|
||||
.choice.pulse {
|
||||
animation: pulse 320ms var(--ease);
|
||||
}
|
||||
|
||||
@keyframes pulse {
|
||||
40% { transform: scale(1.03); }
|
||||
}
|
||||
|
||||
/* Note and judge */
|
||||
|
||||
.note {
|
||||
display: grid;
|
||||
gap: 6px;
|
||||
margin-top: 18px;
|
||||
color: var(--muted);
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
textarea {
|
||||
width: 100%;
|
||||
min-height: 64px;
|
||||
padding: 10px 12px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 10px;
|
||||
background: var(--surface);
|
||||
color: var(--text);
|
||||
font: inherit;
|
||||
font-size: 14px;
|
||||
resize: vertical;
|
||||
transition: border-color 150ms var(--ease), box-shadow 150ms var(--ease);
|
||||
}
|
||||
|
||||
textarea:hover {
|
||||
border-color: var(--border-strong);
|
||||
}
|
||||
|
||||
.judge,
|
||||
.note-preview {
|
||||
margin-top: 14px;
|
||||
padding: 12px 14px;
|
||||
border-radius: 10px;
|
||||
font-size: 14px;
|
||||
animation: reveal 220ms var(--ease);
|
||||
}
|
||||
|
||||
.note-preview {
|
||||
background: var(--surface-muted);
|
||||
}
|
||||
|
||||
.judge {
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.judge.agree {
|
||||
border-color: var(--natural);
|
||||
background: var(--natural-soft);
|
||||
}
|
||||
|
||||
.judge.disagree {
|
||||
border-color: var(--spelled);
|
||||
background: var(--spelled-soft);
|
||||
}
|
||||
|
||||
.judge-head {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
margin-bottom: 4px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
@keyframes reveal {
|
||||
from { opacity: 0; transform: translateY(-4px); }
|
||||
}
|
||||
|
||||
.markdown > :first-child { margin-top: 0; }
|
||||
.markdown > :last-child { margin-bottom: 0; }
|
||||
.markdown p { margin: 0.3em 0; }
|
||||
.markdown code {
|
||||
padding: 0.1em 0.35em;
|
||||
border-radius: 5px;
|
||||
background: oklch(from var(--text) l c h / 8%);
|
||||
font: 0.92em ui-monospace, SFMono-Regular, Menlo, monospace;
|
||||
}
|
||||
.markdown ul,
|
||||
.markdown ol { margin: 0.3em 0; padding-left: 1.4em; }
|
||||
|
||||
.tools {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
margin-top: 10px;
|
||||
}
|
||||
|
||||
/* Pager and hint */
|
||||
|
||||
.pager {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
margin-top: 16px;
|
||||
}
|
||||
|
||||
.position {
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
font-variant-numeric: tabular-nums;
|
||||
}
|
||||
|
||||
.hint {
|
||||
margin-top: 20px;
|
||||
color: var(--muted);
|
||||
font-size: 13px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.empty {
|
||||
padding: 48px 24px;
|
||||
color: var(--muted);
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
/* Toast */
|
||||
|
||||
.toast {
|
||||
position: fixed;
|
||||
top: 20px;
|
||||
right: 20px;
|
||||
padding: 10px 16px;
|
||||
border-radius: 999px;
|
||||
background: var(--text);
|
||||
color: var(--bg);
|
||||
font-size: 13px;
|
||||
box-shadow: var(--shadow);
|
||||
opacity: 0;
|
||||
pointer-events: none;
|
||||
transform: translateY(-8px);
|
||||
transition: opacity 200ms var(--ease), transform 200ms var(--ease);
|
||||
}
|
||||
|
||||
.toast.visible {
|
||||
opacity: 1;
|
||||
transform: translateY(0);
|
||||
}
|
||||
|
||||
.toast.error {
|
||||
background: var(--spelled);
|
||||
color: white;
|
||||
}
|
||||
|
||||
/* Small screens */
|
||||
|
||||
@media (max-width: 640px) {
|
||||
.filters { width: 100%; }
|
||||
.field { flex: 1; }
|
||||
select { min-width: 0; width: 100%; }
|
||||
.card { padding: 18px 16px; }
|
||||
.spoken { font-size: 18px; }
|
||||
.choices { grid-template-columns: 1fr; }
|
||||
.pager .button { padding-inline: 10px; }
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
*,
|
||||
*::before,
|
||||
*::after {
|
||||
animation: none !important;
|
||||
transition: none !important;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
/** A reading style, the same values as the judge answers. */
|
||||
export type ReadingLabel = 'natural' | 'mixed' | 'spelled'
|
||||
|
||||
/** One reading that a person labels, to measure how often the judge agrees with a person. */
|
||||
export interface CalibrationItem {
|
||||
/** `<object>|<case id>|<language>`. */
|
||||
id: string
|
||||
object: string
|
||||
caseId: string
|
||||
language: string
|
||||
/** The LaTeX source of the case. */
|
||||
reference: string
|
||||
/** The spoken text that the test object wrote. */
|
||||
spoken: string
|
||||
/** The median verdict of the judge when the item was collected, and the reason of its first vote. */
|
||||
judge: { reading: string, reason: string }
|
||||
/** The label of a person, or null before labeling. */
|
||||
label: ReadingLabel | null
|
||||
note: string
|
||||
}
|
||||
|
||||
export interface CalibrationFile {
|
||||
seed?: number
|
||||
items: CalibrationItem[]
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
import type { Plugin } from 'vite'
|
||||
|
||||
import type { CalibrationFile, ReadingLabel } from './types'
|
||||
|
||||
import { readFile, rename, writeFile } from 'node:fs/promises'
|
||||
import { env } from 'node:process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { defineConfig } from 'vite'
|
||||
|
||||
/** `CALIBRATION_FILE` points the tool at another file, for example a copy to test the tool without touching labels. */
|
||||
const calibrationFile = env.CALIBRATION_FILE ?? fileURLToPath(new URL('./calibration.json', import.meta.url))
|
||||
const labels = new Set<ReadingLabel>(['natural', 'mixed', 'spelled'])
|
||||
|
||||
/**
|
||||
* Runs the updates of the file one at a time.
|
||||
*
|
||||
* Leaving a note box and clicking a label sends two saves at once. Two read-modify-write updates that overlap wrote
|
||||
* over each other and left two JSON documents in one file, so every update waits for the one before it. The new
|
||||
* file is written next to the old one and renamed over it, so a failed write never leaves half a file.
|
||||
*/
|
||||
let updates: Promise<unknown> = Promise.resolve()
|
||||
function update(change: (file: CalibrationFile) => boolean): Promise<boolean> {
|
||||
const run = updates.then(async () => {
|
||||
const file = JSON.parse(await readFile(calibrationFile, 'utf8')) as CalibrationFile
|
||||
if (!change(file))
|
||||
return false
|
||||
const partial = `${calibrationFile}.partial`
|
||||
await writeFile(partial, `${JSON.stringify(file, null, 2)}\n`)
|
||||
await rename(partial, calibrationFile)
|
||||
return true
|
||||
})
|
||||
updates = run.catch(() => undefined)
|
||||
return run
|
||||
}
|
||||
|
||||
/**
|
||||
* Serves `calibration.json` to the page and writes each label back to it.
|
||||
*
|
||||
* - `GET /api/items` returns the file.
|
||||
* - `POST /api/items` with `{ id, label, note }` sets the label of one item. `label: null` clears it.
|
||||
*
|
||||
* The file is the only state, so a label is saved as soon as the person chooses it.
|
||||
*/
|
||||
function calibrationApi(): Plugin {
|
||||
return {
|
||||
name: 'calibration-api',
|
||||
configureServer(server) {
|
||||
server.middlewares.use('/api/items', async (request, response) => {
|
||||
if (request.method === 'GET') {
|
||||
response.setHeader('content-type', 'application/json')
|
||||
response.end(await readFile(calibrationFile, 'utf8'))
|
||||
return
|
||||
}
|
||||
if (request.method !== 'POST') {
|
||||
response.statusCode = 405
|
||||
response.end()
|
||||
return
|
||||
}
|
||||
|
||||
let body = ''
|
||||
for await (const chunk of request)
|
||||
body += chunk
|
||||
const { id, label, note } = JSON.parse(body) as { id: string, label: ReadingLabel | null, note: string }
|
||||
if (label !== null && !labels.has(label)) {
|
||||
response.statusCode = 400
|
||||
response.end('unknown label')
|
||||
return
|
||||
}
|
||||
const found = await update((file) => {
|
||||
const item = file.items.find(candidate => candidate.id === id)
|
||||
if (!item)
|
||||
return false
|
||||
item.label = label
|
||||
item.note = note
|
||||
return true
|
||||
})
|
||||
response.statusCode = found ? 200 : 404
|
||||
response.end(found ? 'ok' : 'unknown item')
|
||||
})
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/** The labeling tool of the judge calibration set. Run: pnpm -F @proj-airi/evals-tts-speech-formula run judge:label */
|
||||
export default defineConfig({
|
||||
root: fileURLToPath(new URL('.', import.meta.url)),
|
||||
plugins: [calibrationApi()],
|
||||
server: { port: Number(env.CALIBRATION_PORT ?? 5299), open: !env.CALIBRATION_FILE },
|
||||
})
|
||||
@@ -0,0 +1,32 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { judgePrompt, medianReading } from '.'
|
||||
|
||||
describe('medianReading', () => {
|
||||
const votes = (...readings: ('natural' | 'mixed' | 'spelled')[]) => readings.map(reading => ({ reading, reason: '' }))
|
||||
|
||||
it('takes the reading of two votes of three', () => {
|
||||
expect(medianReading(votes('natural', 'spelled', 'natural'))).toBe('natural')
|
||||
expect(medianReading(votes('spelled', 'natural', 'spelled'))).toBe('spelled')
|
||||
})
|
||||
|
||||
it('selects mixed when three votes differ', () => {
|
||||
expect(medianReading(votes('natural', 'mixed', 'spelled'))).toBe('mixed')
|
||||
})
|
||||
})
|
||||
|
||||
describe('judgePrompt', () => {
|
||||
// Velin escapes backslashes in plain text, so the LaTeX example must stay in a code span.
|
||||
it('names the language and keeps the LaTeX example unchanged', async () => {
|
||||
const prompt = await judgePrompt('ja')
|
||||
expect(prompt).toContain('Japanese')
|
||||
expect(prompt).toContain(`\`${String.raw`$\frac{1}{2}$`}\``)
|
||||
expect(prompt).not.toContain('<script')
|
||||
})
|
||||
|
||||
it('lists the standard readings of each language as natural', async () => {
|
||||
const prompt = await judgePrompt('ja')
|
||||
for (const reading of ['大なりイコール', '下付き', '下标', '아래첨자', 'dot dot dot'])
|
||||
expect(prompt).toContain(reading)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,107 @@
|
||||
import type { TaskRunContext } from 'vieval/config'
|
||||
|
||||
import type { Language } from '../../cases/formula-speech/languages'
|
||||
import type { Endpoint } from '../listener'
|
||||
|
||||
import { object, parse, picklist, string } from 'valibot'
|
||||
|
||||
import { languageName } from '../../cases/formula-speech/languages'
|
||||
import { cacheKey } from '../../shared/cache-key'
|
||||
import { generateTextWithTelemetry } from '../../shared/telemetry'
|
||||
import { renderVelinPrompt } from '../../shared/velin'
|
||||
|
||||
/**
|
||||
* How a reading says a formula.
|
||||
*
|
||||
* - `natural`: it says the meaning, as a teacher reads the formula to a class.
|
||||
* - `mixed`: it is mostly natural, with one or two parts that spell markup.
|
||||
* - `spelled`: it spells the markup, for example "open parenthesis x close parenthesis", or leaves the source.
|
||||
*/
|
||||
export type Reading = 'natural' | 'mixed' | 'spelled'
|
||||
|
||||
/**
|
||||
* The judge score of each reading, recorded as `formula.natural`. The headline score `formula.pass` needs a correct
|
||||
* reading and at least the mixed score, so a reading that the listener can write down but that spells markup fails.
|
||||
*/
|
||||
export const readingScores: Readonly<Record<Reading, number>> = { natural: 1, mixed: 0.5, spelled: 0 }
|
||||
|
||||
/** Independent judge answers per case. The case takes the median, so one unstable answer does not decide it. */
|
||||
export const judgeVotes = 3
|
||||
|
||||
/** One vote of the judge. */
|
||||
export interface Verdict {
|
||||
reading: Reading
|
||||
/** One short sentence that names the spelled parts. Reports show it, so a reader can check the judge. */
|
||||
reason: string
|
||||
}
|
||||
|
||||
/** The answer contract in `judge.velin.md`. The JSON Schema below sends the same shape to the provider. */
|
||||
const verdictSchema = object({ reason: string(), reading: picklist(['natural', 'mixed', 'spelled']) })
|
||||
|
||||
const verdictFormat = {
|
||||
type: 'json_schema',
|
||||
json_schema: {
|
||||
name: 'judge_verdict',
|
||||
strict: true,
|
||||
schema: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
// The reason comes first, so the model names the spelled parts before it chooses the reading.
|
||||
required: ['reason', 'reading'],
|
||||
properties: {
|
||||
reason: { type: 'string' },
|
||||
reading: { type: 'string', enum: ['natural', 'mixed', 'spelled'] },
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
/** The system prompt of the judge. It judges only the style of a reading. The listener and the grader judge correctness. */
|
||||
export function judgePrompt(language: Language): Promise<string> {
|
||||
return renderVelinPrompt(new URL('./judge.velin.md', import.meta.url), { language: languageName[language] })
|
||||
}
|
||||
|
||||
/**
|
||||
* The median reading of several votes. With three votes, two votes decide, and a tie of three different readings
|
||||
* selects `mixed`.
|
||||
*
|
||||
* @example
|
||||
* medianReading([{ reading: 'natural', reason: '' }, { reading: 'spelled', reason: '' }, { reading: 'natural', reason: '' }])
|
||||
* // => 'natural'
|
||||
*/
|
||||
export function medianReading(verdicts: readonly Verdict[]): Reading {
|
||||
const sorted = verdicts.map(({ reading }) => reading).sort((left, right) => readingScores[left] - readingScores[right])
|
||||
return sorted[Math.floor((sorted.length - 1) / 2)] ?? 'spelled'
|
||||
}
|
||||
|
||||
/**
|
||||
* Judges whether a reading says the meaning of a formula or spells its markup.
|
||||
*
|
||||
* It judges the spoken text that the test object sends to TTS, not the ASR transcript, because ASR errors are not part
|
||||
* of the reading. An empty reading makes no call and is `spelled`: it says nothing of the formula.
|
||||
*
|
||||
* Each vote is cached by model, prompt, source, reading, and vote number, so an unchanged reading gets the same verdict
|
||||
* in every run and under every TTS model. `context` receives the usage of each call that reaches the model.
|
||||
*/
|
||||
export async function judgeReading(spoken: string, source: string, language: Language, vote: number, endpoint: Endpoint, context: TaskRunContext, signal: AbortSignal): Promise<Verdict> {
|
||||
if (!spoken.trim())
|
||||
return { reading: 'spelled', reason: 'The reading is empty.' }
|
||||
|
||||
const system = await judgePrompt(language)
|
||||
const file = context.cache.namespace('judge').file({ ext: 'json', key: [endpoint.model, cacheKey(system, source, spoken, String(vote))] })
|
||||
if (await file.exists())
|
||||
return file.readJson<Verdict>()
|
||||
|
||||
const { text } = await generateTextWithTelemetry(context, 'judge', {
|
||||
...endpoint,
|
||||
abortSignal: signal,
|
||||
maxCompletionTokens: 400,
|
||||
messages: [{ role: 'system', content: system }, { role: 'user', content: `LaTeX source: ${source}\nSpoken text: ${spoken}` }],
|
||||
responseFormat: verdictFormat,
|
||||
temperature: 0,
|
||||
})
|
||||
// A malformed verdict throws. The case then fails as a judge error instead of scoring a wrong 0.
|
||||
const verdict = parse(verdictSchema, JSON.parse(text ?? ''))
|
||||
await file.writeJson(verdict)
|
||||
return verdict
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
<script setup>
|
||||
const props = defineProps({
|
||||
// The language name of the reading, for example `Mandarin Chinese`.
|
||||
language: { type: String, required: true },
|
||||
})
|
||||
</script>
|
||||
|
||||
## Role
|
||||
|
||||
You judge how a math formula is read aloud in {{ props.language }}. A text-to-speech voice reads the text, and a listener only hears it.
|
||||
|
||||
You get the LaTeX source and the spoken text. The source can also be a sentence with prices and math. Then judge only how the math and the money are read.
|
||||
|
||||
## What to judge
|
||||
|
||||
Judge only the style of the reading. Another check grades whether the reading is correct, so never judge correctness. Never judge length alone.
|
||||
|
||||
A natural reading says the meaning with the words that a teacher uses to read the formula to a class, in {{ props.language }}, as short as the meaning allows.
|
||||
|
||||
- It can add a structure word where the grouping would be lost, for example "the quantity p plus q, squared" or "end sub".
|
||||
- It can use "the fraction with numerator ... and denominator ..." when the numerator or the denominator is compound or has a power. For `6/\pi^2` the long form is natural, because "six over pi squared" can be heard as `(6/\pi)^2`. For `1/2`, the long form is mixed.
|
||||
- It can name brackets that carry meaning, for example the square brackets of a class, or the parentheses of a call with more than one argument.
|
||||
- It says the case of a letter only when the formula has both cases of that letter, and then in the short form, for example 大A and 小a, or capital A. `a` and `A` are the two cases of one letter, so `b \in B` needs its case words, and so do `\psi` and `\Psi`, and `A` and `B` in `g: A \to B, a \mapsto b`. Saying the case of a letter that has one case in the formula, for example capital for a lone `\Sigma`, is markup.
|
||||
|
||||
## Standard readings
|
||||
|
||||
These are the words of textbooks and classrooms. They are natural, never spelled markup, even when they name a symbol or a script. The list shows examples, not every word: another common word with the same meaning, such as Japanese 添字 for a subscript, is natural too.
|
||||
|
||||
- English: "f of x", "x squared", "x sub one", "x to the power of n", "one over two", "the sum from k equals one to n of", "the set of x such that", "and so on" or "up to" for an ellipsis, "x prime", "x bar", "x hat".
|
||||
- Mandarin Chinese: 下标 for a subscript, 的平方 and 次方 for powers, 分之 for a fraction, 分子 and 分母 for a long fraction, p 与 q 的和的平方 for `(p+q)^2`, 依此类推 for an ellipsis, English loanwords for functions such as sine or log.
|
||||
- Japanese: 下付き for a subscript, の二乗 and 乗 for powers, 分の for a fraction, 分子が and 分母が for a long fraction, イコール, ノットイコール, 大なり, 小なり, 大なりイコール, 小なりイコール, ニアリーイコール, かっこ and かっことじ around a compound part, 大文字の and 小文字の for letter case.
|
||||
- Korean: 아래첨자 for a subscript, 제곱 for powers, 분의 for a fraction, 괄호 and 괄호 닫고 around a compound part, 대문자 and 소문자 for letter case.
|
||||
- In Japanese and Korean, かっこ and かっことじ, or 괄호 and 괄호 닫고, around a compound part is how a teacher groups it, also around the main structure: `(p+q)^2` is かっこピープラスキューかっことじの二乗, and `\sqrt{p^2+q^2}` is ルートかっこ…かっことじ. These languages have no other phrase for the group, so these words are natural, not spelled.
|
||||
- Chemistry in every language: the element symbols of a formula read letter by letter, for example "N A H C O three" for `NaHCO3`, and the names of compounds.
|
||||
|
||||
## Spelled markup
|
||||
|
||||
A spelled reading reads the markup instead of the meaning.
|
||||
|
||||
- It names most parentheses, brackets, braces, or commas, for example "open parenthesis x close parenthesis" for `f(x)`. A bracket word around every simple item is markup, even in the languages above.
|
||||
- It names where a script or an argument starts and ends when the grouping is already clear, for example "start exponent", "subscript beginning", or "subscript ending".
|
||||
- It says LaTeX command names that are not words of the language, for example "mathbb", "frac", "varsigma", "overline", "backslash", "left", or "right".
|
||||
- It leaves LaTeX source or symbols for the voice, for example `$\frac{1}{2}$`, `^`, `_`, or `\alpha`.
|
||||
- It names the brackets of the main structure when the language has a natural phrase for it, for example "open paren p plus q close paren squared" for `(p+q)^2` in English, where "the square of the quantity p plus q" exists, or 左花括号 and 右花括号 around a set. This is spelled, not mixed. Japanese and Korean group words are not this case, see the standard readings.
|
||||
- It says "dot dot dot" or 点点点 for an ellipsis. This makes the reading at least mixed.
|
||||
|
||||
## Language
|
||||
|
||||
The reading must be in {{ props.language }}, apart from letter names, loanwords, and the names of functions. A reading that is mostly in another language is `spelled`, even when its words would be natural in that other language: the listener asked for {{ props.language }} and cannot use it.
|
||||
|
||||
## Answer
|
||||
|
||||
First write `reason`: one short English sentence that names each spelled part, or says that no part is spelled. Then write `reading`, and make it agree with the reason.
|
||||
|
||||
- `natural`: the reason names no spelled part.
|
||||
- `mixed`: most of the reading is natural, and the reason names one or two spelled parts.
|
||||
- `spelled`: most of the reading spells markup, it leaves the source, or it is mostly in another language.
|
||||
@@ -0,0 +1,21 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { listenerPrompt } from '.'
|
||||
|
||||
// Velin renders markdown and writes markdown back, which escapes backslashes, underscores, and braces in plain text.
|
||||
// The prompts keep every LaTeX example in a code span, so the listener sees it unchanged.
|
||||
describe('listenerPrompt', () => {
|
||||
it.each(['formula', 'chem', 'sentence'] as const)('keeps every LaTeX example unchanged for the %s form', async (form) => {
|
||||
const prompt = await listenerPrompt('zh', form)
|
||||
for (const example of [String.raw`\frac{B}{A}`, String.raw`\{x \mid ...\}`, String.raw`\sum_{k=1}^{n}`, String.raw`\sigma_0`, String.raw`7\,\mathrm{kg}`, String.raw`^\circ\mathrm{C}`])
|
||||
expect(prompt).toContain(`\`${example}\``)
|
||||
expect(prompt).not.toContain('<div')
|
||||
expect(prompt).toContain('Mandarin Chinese')
|
||||
})
|
||||
|
||||
it('tells the listener only about the form of the audio', async () => {
|
||||
expect(await listenerPrompt('en', 'chem')).toContain(String.raw`\ce{...}`)
|
||||
expect(await listenerPrompt('en', 'sentence')).toContain(String.raw`\$19.99`)
|
||||
expect(await listenerPrompt('en', 'formula')).not.toContain(String.raw`\ce{...}`)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,164 @@
|
||||
import type { TaskRunContext } from 'vieval/config'
|
||||
|
||||
import type { GradingMode } from '../../cases/formula-speech/cases'
|
||||
import type { Language } from '../../cases/formula-speech/languages'
|
||||
|
||||
import { object, parse, picklist, string } from 'valibot'
|
||||
|
||||
import { languageName } from '../../cases/formula-speech/languages'
|
||||
import { cacheKey } from '../../shared/cache-key'
|
||||
import { generateTextWithTelemetry } from '../../shared/telemetry'
|
||||
import { renderVelinPrompt } from '../../shared/velin'
|
||||
|
||||
/** An OpenAI-compatible endpoint and the model to call on it. */
|
||||
export interface Endpoint {
|
||||
baseURL: string
|
||||
apiKey: string
|
||||
model: string
|
||||
}
|
||||
|
||||
/** What the listener is told about the form of the audio. It never sees the reference. */
|
||||
export type AnswerForm = 'formula' | 'chem' | 'sentence'
|
||||
|
||||
/**
|
||||
* The system prompt of the listener. It asks for a literal transcription, never a correction toward a known formula,
|
||||
* so that a misread formula stays wrong and the deterministic grader can see it.
|
||||
*/
|
||||
export function listenerPrompt(language: Language, form: AnswerForm): Promise<string> {
|
||||
return renderVelinPrompt(new URL('./listener.velin.md', import.meta.url), { language: languageName[language], form })
|
||||
}
|
||||
|
||||
/**
|
||||
* Independent listener answers per case. The case passes with a strict majority of passing answers.
|
||||
* Three votes hold an unstable answer of the listener model to one vote of three.
|
||||
*/
|
||||
export const listenerVotes = 3
|
||||
|
||||
export function answerFormOf(mode: GradingMode): AnswerForm {
|
||||
if (mode === 'chem')
|
||||
return 'chem'
|
||||
if (mode === 'currency')
|
||||
return 'sentence'
|
||||
return 'formula'
|
||||
}
|
||||
|
||||
/**
|
||||
* How well the listener could hear the transcript.
|
||||
*
|
||||
* - `ok`: it heard all of it.
|
||||
* - `partial`: some parts were garbled or missing. The answer has the parts that it heard, and the grader grades it.
|
||||
* - `unclear`: nothing was recoverable. The grader scores the vote 0 with the rule `unclear`, so a report can tell
|
||||
* an unintelligible clip apart from a wrong reading.
|
||||
*/
|
||||
export type HearingStatus = 'ok' | 'partial' | 'unclear'
|
||||
|
||||
/** One vote of the listener. */
|
||||
export interface ListenerAnswer {
|
||||
/** The LaTeX that the listener wrote down. Empty when the status is `unclear` or the transcript was empty. */
|
||||
latex: string
|
||||
status: HearingStatus
|
||||
}
|
||||
|
||||
/** The answer contract in `listener.velin.md`. The JSON Schema below sends the same shape to the provider. */
|
||||
const answerSchema = object({ latex: string(), status: picklist(['ok', 'partial', 'unclear']) })
|
||||
|
||||
/** Structured output, so the hearing status is a field and never mixed into the LaTeX. */
|
||||
const answerFormat = {
|
||||
type: 'json_schema',
|
||||
json_schema: {
|
||||
name: 'listener_answer',
|
||||
strict: true,
|
||||
schema: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
required: ['latex', 'status'],
|
||||
properties: {
|
||||
latex: { type: 'string' },
|
||||
status: { type: 'string', enum: ['ok', 'partial', 'unclear'] },
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
/** JSON escapes that are also the first letter of a TeX command: `\f` of `\frac`, `\t` of `\times`, `\n` of `\neq`. */
|
||||
const swallowedCommandLetters: Readonly<Record<string, string>> = { '\f': 'f', '\t': 't', '\n': 'n', '\b': 'b', '\r': 'r' }
|
||||
|
||||
/**
|
||||
* Restores TeX commands that a JSON parser turned into control characters.
|
||||
*
|
||||
* A model can write `"\frac"` in its JSON answer without escaping the backslash. `JSON.parse` then reads `\f` as a form
|
||||
* feed, and the answer becomes `rac{1}{x}`. A LaTeX answer has no control characters, so each one is a lost command.
|
||||
* Cached answers from before this repair get it when they are read.
|
||||
*
|
||||
* @example
|
||||
* // JSON.parse('{"latex": "x^{-1}=\frac{1}{x}"}').latex has a form feed in place of `\f`.
|
||||
* restoreTexCommands('x^{-1}=\u000Crac{1}{x}')
|
||||
* // => 'x^{-1}=\\frac{1}{x}'
|
||||
*/
|
||||
function restoreTexCommands(latex: string): string {
|
||||
return latex.replace(/[\f\t\n\u0008\r]/g, control => `\\${swallowedCommandLetters[control]}`)
|
||||
}
|
||||
|
||||
/** True when a LaTeX answer still holds words: CJK text, or a Latin word of four or more letters outside commands. */
|
||||
function hasWords(answer: string): boolean {
|
||||
const bare = answer.replace(/\\[A-Z]+/gi, ' ')
|
||||
return /[\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Han}\p{Script=Hangul}]/u.test(bare) || /[A-Za-z][a-z]{3,}/.test(bare)
|
||||
}
|
||||
|
||||
/** The model sometimes answers with display math over several lines. All of it is kept, on one line. */
|
||||
function oneLine(answer: string): string {
|
||||
return answer.trim().split('\n').map(line => line.trim()).filter(Boolean).join(' ')
|
||||
}
|
||||
|
||||
async function ask(endpoint: Endpoint, system: string, prompt: string, context: TaskRunContext, signal: AbortSignal): Promise<ListenerAnswer> {
|
||||
const { text } = await generateTextWithTelemetry(context, 'listener', {
|
||||
...endpoint,
|
||||
abortSignal: signal,
|
||||
// WORKAROUND:
|
||||
// A garbled transcript makes the listener model write without end, and the case never finishes.
|
||||
// whisper-1 can fall into a repetition loop on a short clip, for example `L-R-R-R-...` for Korean `l 1은 l 2와 평행`.
|
||||
// Source: the `rel-parallel` case in Korean on the official StepFun TTS, 2026-10-19. T-157 used the same cap.
|
||||
// Remove when the listener model stops on its own for such input. The step timeout in tasks/read-aloud stays as a backstop.
|
||||
maxCompletionTokens: 2000,
|
||||
messages: [{ role: 'system', content: system }, { role: 'user', content: prompt }],
|
||||
responseFormat: answerFormat,
|
||||
temperature: 0,
|
||||
})
|
||||
// A malformed answer throws. The case then fails as a listener error instead of scoring a wrong 0.
|
||||
const answer = parse(answerSchema, JSON.parse(text ?? ''))
|
||||
return { latex: oneLine(restoreTexCommands(answer.latex)), status: answer.status }
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes down, as LaTeX, what the listener heard, with how well it could hear it.
|
||||
* An empty transcript makes no call and is `unclear`.
|
||||
*
|
||||
* Each vote is cached by model, prompt, transcript, and vote number. An unchanged transcript therefore gets the same answer
|
||||
* in every run, and run-to-run noise comes only from the system under test.
|
||||
* A formula answer that still holds words gets one format repair. The repair is kept only when it is clean.
|
||||
* `context` is the case context: it holds the cache, and it receives the usage of each call that reaches the model.
|
||||
*/
|
||||
export async function writeDown(heard: string, language: Language, form: AnswerForm, vote: number, endpoint: Endpoint, context: TaskRunContext, signal: AbortSignal): Promise<ListenerAnswer> {
|
||||
if (!heard.trim())
|
||||
return { latex: '', status: 'unclear' }
|
||||
|
||||
const system = await listenerPrompt(language, form)
|
||||
const file = context.cache.namespace('listener').file({ ext: 'json', key: [endpoint.model, cacheKey(system, heard, String(vote))] })
|
||||
if (await file.exists()) {
|
||||
const cached = await file.readJson<ListenerAnswer>()
|
||||
return { ...cached, latex: restoreTexCommands(cached.latex) }
|
||||
}
|
||||
|
||||
const prompt = `ASR transcript: ${heard}`
|
||||
let answer = await ask(endpoint, system, prompt, context, signal)
|
||||
if (form !== 'sentence' && answer.status !== 'unclear' && hasWords(answer.latex)) {
|
||||
const repaired = await ask(endpoint, system, `${prompt}\nYour previous answer was: ${answer.latex}\nIt still contains words. Write it again as notation only, still exactly as heard.`, context, signal)
|
||||
// A failed repair keeps the first answer. The grader then scores it as written.
|
||||
.catch(() => answer)
|
||||
if (!hasWords(repaired.latex))
|
||||
answer = repaired
|
||||
}
|
||||
|
||||
await file.writeJson(answer)
|
||||
return answer
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
<script setup>
|
||||
const props = defineProps({
|
||||
// The language name of the voice, for example `Mandarin Chinese`.
|
||||
language: { type: String, required: true },
|
||||
// `formula`, `chem`, or `sentence`. The listener knows the form of the audio, but never the reference.
|
||||
form: { type: String, required: true },
|
||||
})
|
||||
</script>
|
||||
|
||||
## Role
|
||||
|
||||
You are a listener. A text-to-speech voice read something in {{ props.language }}, and you get the ASR transcript of that audio. The transcript can contain homophone errors, for example 派, パイ, or 파이 for pi, and "E B" for "A B".
|
||||
|
||||
## What the audio was
|
||||
|
||||
<div v-if="props.form === 'chem'">
|
||||
|
||||
The audio was one chemical equation or formula. Write it as `\ce{...}` with standard chemical capitalisation, for example `KNO3`, `SO3`, and `Mg(OH)2`.
|
||||
|
||||
- Species said by name (hydrogen, 氢气, 水素, 수소, water, 水, 물) become their usual formula, for example `H2` and `H2O`.
|
||||
- Keep the coefficients, the species, and the arrow that were said.
|
||||
- Use `<=>` only if a reversible reaction was said.
|
||||
|
||||
</div>
|
||||
<div v-else-if="props.form === 'sentence'">
|
||||
|
||||
The audio was a sentence that may contain prices and math.
|
||||
|
||||
- Keep the words.
|
||||
- Write every amount of money as `\$N`, for example `\$19.99`. "two dollars" is `\$2`.
|
||||
- Put any math between `$...$`.
|
||||
|
||||
</div>
|
||||
<div v-else>
|
||||
|
||||
The audio was one math formula. Write it down as a single LaTeX formula with NO words of any language left in it.
|
||||
|
||||
</div>
|
||||
|
||||
## Rules
|
||||
|
||||
1. Write exactly what was said.
|
||||
- Never correct, complete, or "fix" it toward a well-known formula.
|
||||
- If a part is missing or garbled, leave it missing or write what was literally said.
|
||||
- A wrong-sounding number or letter stays as heard: "n minus e" is `n-e`, not `n-1`.
|
||||
2. Fractions follow the spoken order.
|
||||
- Chinese "A分之B", Japanese "A分のB", and Korean "A 분의 B" all mean `\frac{B}{A}`. This also holds when A is 1: Korean "1분의 x" is `\frac{x}{1}`.
|
||||
- English "A over B" means `\frac{A}{B}`.
|
||||
- "numerator X, denominator Y" means `\frac{X}{Y}`.
|
||||
3. Japanese 小なり and 大なり (also heard as ショーなり and だいなり) are `<` and `>`. With イコール, they are `\leq` and `\geq`.
|
||||
4. Letter case: write a Latin or Greek letter in upper case only if the speech says so (capital, upper, cap, 大写, 大文字, 대문자). The capitalisation in the ASR text is a guess and does not count. Otherwise, write the letter in lower case. This rule does not apply to chemistry.
|
||||
5. Structure words become notation.
|
||||
- "the set of x such that ..." is `\{x \mid ...\}`.
|
||||
- "sum from k=1 to n of" is `\sum_{k=1}^{n}`.
|
||||
- "the quantity p plus q, squared" is `(p+q)^2`.
|
||||
- A power applies only to the letter or the group right before it, as in written notation. "a b squared", Chinese "ab的平方" and "ab方", Japanese "エービーの二乗", and Korean "에이 비 제곱" are `ab^2`. Only a spoken group, as "the quantity a b, squared" or "a与b的积的平方", is `(ab)^2`.
|
||||
- "all of that over 2" is `\frac{...}{2}`.
|
||||
- "limit as n approaches infinity" is `\lim_{n \to \infty}`.
|
||||
- "f x" or "f of x" is `f(x)`.
|
||||
- "sigma sub 0" is the letter `\sigma_0`. Only "sum" or "the sum from ... to ..." is `\sum`.
|
||||
- "line 1" is `l_1`.
|
||||
- "the (set of) rational / real / integer / natural / complex numbers" is `\mathbb{Q}` / `\mathbb{R}` / `\mathbb{Z}` / `\mathbb{N}` / `\mathbb{C}`.
|
||||
6. Units become SI symbols in `\mathrm{}`, for example `7\,\mathrm{kg}`, `\mathrm{m/s^2}`, and `^\circ\mathrm{C}`.
|
||||
|
||||
## Output
|
||||
|
||||
Answer with a JSON object that has two fields:
|
||||
|
||||
- `latex`: what you wrote down, on one line, with no explanation.
|
||||
- `status`: how well you could hear it.
|
||||
- `ok`: you heard all of it.
|
||||
- `partial`: some parts are garbled or missing. `latex` has the parts that you heard.
|
||||
- `unclear`: the transcript says nothing recoverable. `latex` is empty.
|
||||
@@ -0,0 +1,25 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { leftoverNotation } from '.'
|
||||
|
||||
describe('leftoverNotation', () => {
|
||||
it('finds characters that a voice cannot read as words', () => {
|
||||
expect(leftoverNotation('9.8 m/s^2', String.raw`$9.8\,\mathrm{m/s^2}$`)).toEqual(['/', '^'])
|
||||
expect(leftoverNotation('りんごは $3、バナナは $5 です。', 'りんごは $3、バナナは $5 です。')).toEqual(['$', '$'])
|
||||
})
|
||||
|
||||
it('finds a command name of the input that came through as a word, also next to CJK text', () => {
|
||||
expect(leftoverNotation('f X 趋于 Y x mapsto y', String.raw`$f: X \to Y, x \mapsto y$`)).toEqual(['mapsto'])
|
||||
expect(leftoverNotation('x mapsto是y', String.raw`$x \mapsto y$`)).toEqual(['mapsto'])
|
||||
})
|
||||
|
||||
it('accepts command names that are also words of a reading', () => {
|
||||
expect(leftoverNotation('3 times 10 to the power of 8', String.raw`$3 \times 10^{8}$`)).toEqual([])
|
||||
expect(leftoverNotation('omega is not equal to Omega', String.raw`$\Omega \neq \omega$`)).toEqual([])
|
||||
expect(leftoverNotation('l 1 is parallel to l 2', String.raw`$l_1 \parallel l_2$`)).toEqual([])
|
||||
})
|
||||
|
||||
it('accepts a command name that the input does not use', () => {
|
||||
expect(leftoverNotation('the vector maps to its image', '$v$')).toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,67 @@
|
||||
/**
|
||||
* Characters that a TTS voice cannot read as words. Each one leaves the reading to the TTS provider:
|
||||
* some read `$5` and `m/s` well, and others skip or misread them, mostly in Japanese and Korean.
|
||||
*/
|
||||
const notationCharacters = /[\\^_{}$|~/]/g
|
||||
|
||||
/**
|
||||
* LaTeX command names that are also words of a reading, so finding them in spoken text is not a leftover:
|
||||
* Greek letter names, and English words such as `times` in "3 times 10" or `right` in "right arrow".
|
||||
*/
|
||||
const spokenCommandNames = new Set([
|
||||
'alpha',
|
||||
'beta',
|
||||
'gamma',
|
||||
'delta',
|
||||
'epsilon',
|
||||
'zeta',
|
||||
'theta',
|
||||
'iota',
|
||||
'kappa',
|
||||
'lambda',
|
||||
'omicron',
|
||||
'sigma',
|
||||
'upsilon',
|
||||
'omega',
|
||||
'nabla',
|
||||
'angle',
|
||||
'left',
|
||||
'right',
|
||||
'over',
|
||||
'parallel',
|
||||
'partial',
|
||||
'prime',
|
||||
'star',
|
||||
'dagger',
|
||||
'subset',
|
||||
'supset',
|
||||
'tilde',
|
||||
'times',
|
||||
'triangle',
|
||||
])
|
||||
|
||||
/**
|
||||
* Finds the notation that a test object left in its spoken text, so a TTS voice gets symbols instead of words.
|
||||
*
|
||||
* It finds two kinds: a character that has no reading, such as `^`, `/`, or `$`, and a LaTeX command name of the input
|
||||
* that came through as a word, such as `mapsto`. It runs on text only, so it shows these defects without TTS or ASR.
|
||||
* Command names shorter than four letters are not checked, because many of them are also words, for example `in`.
|
||||
*
|
||||
* @example
|
||||
* leftoverNotation('9.8 m/s^2', '$9.8\\,\\mathrm{m/s^2}$')
|
||||
* // => ['/', '^']
|
||||
* leftoverNotation('f X approaches Y x mapsto y', '$f: X \\to Y, x \\mapsto y$')
|
||||
* // => ['mapsto']
|
||||
*/
|
||||
export function leftoverNotation(spoken: string, markdown: string): string[] {
|
||||
const leftovers = Array.from(spoken.matchAll(notationCharacters), match => match[0])
|
||||
const commandNames = new Set(Array.from(markdown.matchAll(/\\([a-z]+)/gi), match => match[1]!))
|
||||
for (const name of commandNames) {
|
||||
if (name.length < 4 || spokenCommandNames.has(name.toLowerCase()))
|
||||
continue
|
||||
// `\b` is an ASCII word boundary, so a name next to CJK text, for example `mapsto是`, still matches.
|
||||
if (new RegExp(`\\b${name}\\b`, 'i').test(spoken))
|
||||
leftovers.push(name)
|
||||
}
|
||||
return leftovers
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import { generateTranscription } from '@xsai/generate-transcription'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import { createGeminiTranscription } from './gemini-transcription'
|
||||
|
||||
const parameters = { apiKey: 'gemini-key', baseURL: 'https://gemini.example/v1beta' }
|
||||
|
||||
function transcribeClip() {
|
||||
const request = createGeminiTranscription().transcription({ language: 'ja', model: 'gemini-test', parameters })
|
||||
return generateTranscription({ ...request, file: new Blob([new Uint8Array([1, 2, 3])], { type: 'audio/mpeg' }), fileName: 'speech.mp3', language: 'ja' })
|
||||
}
|
||||
|
||||
describe('createGeminiTranscription', () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks()
|
||||
})
|
||||
|
||||
it('sends the clip as inline audio to generateContent and answers with the transcript', async () => {
|
||||
const upstream = vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(JSON.stringify({
|
||||
candidates: [{ content: { parts: [{ text: 'thinking', thought: true }, { text: ' 2分の1 ' }] } }],
|
||||
})))
|
||||
|
||||
const result = await transcribeClip()
|
||||
|
||||
expect(result).toEqual({ text: '2分の1' })
|
||||
const [url, init] = upstream.mock.calls[0]!
|
||||
expect(String(url)).toBe('https://gemini.example/v1beta/models/gemini-test:generateContent')
|
||||
expect(init?.headers).toMatchObject({ 'x-goog-api-key': 'gemini-key' })
|
||||
const body = JSON.parse(String(init?.body))
|
||||
expect(body.contents[0].parts[0].text).toContain('"ja"')
|
||||
expect(body.contents[0].parts[1].inline_data).toEqual({ data: 'AQID', mime_type: 'audio/mpeg' })
|
||||
})
|
||||
|
||||
it('answers an empty transcript when Gemini returns no candidate', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(JSON.stringify({ candidates: [] })))
|
||||
|
||||
expect(await transcribeClip()).toEqual({ text: '' })
|
||||
})
|
||||
|
||||
it('passes a failed upstream status through, so xsai reports it and vieval can retry it', async () => {
|
||||
vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response('rate limited', { status: 429 }))
|
||||
|
||||
await expect(transcribeClip()).rejects.toThrow('429')
|
||||
})
|
||||
|
||||
it('names the missing parameter of the provider', () => {
|
||||
expect(() => createGeminiTranscription().transcription({ model: 'gemini-test', parameters: { baseURL: parameters.baseURL } })).toThrow('has no "apiKey" parameter')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,102 @@
|
||||
import type { TranscriptionExecutor } from './speech-models'
|
||||
|
||||
import { Buffer } from 'node:buffer'
|
||||
|
||||
// NOTICE:
|
||||
// This module has no runtime relative import, because `eval/tasks/read-aloud/config.ts` imports it through the package
|
||||
// export `./gemini-transcription`. Vieval loads configs with Node type stripping, which cannot follow an extensionless
|
||||
// relative import. Type imports are stripped, so they are safe.
|
||||
// Source: the same constraint as `speech-models.ts`, see eval/index.ts.
|
||||
// Remove this notice when vieval loads configs with an extension-resolving loader.
|
||||
|
||||
/** The text part of a Gemini `generateContent` answer. Thought parts carry the reasoning, not the transcript. */
|
||||
interface GenerateContentResponse {
|
||||
candidates?: Array<{ content?: { parts?: Array<{ text?: string, thought?: boolean }> } }>
|
||||
}
|
||||
|
||||
/**
|
||||
* Tells the model to act as a speech recognizer. The listener turns the words into notation later, so this step
|
||||
* must keep the spoken words and must not use what it knows about formulas to fix or complete them.
|
||||
*/
|
||||
function transcriptionPrompt(language: string | undefined): string {
|
||||
const spoken = language ? `The speech is in the language with the BCP 47 code "${language}".` : ''
|
||||
return [
|
||||
`Transcribe this audio verbatim, as a speech recognizer does. ${spoken}`,
|
||||
'Write the words that you hear. Do not convert spoken math into notation or LaTeX.',
|
||||
'Do not correct, complete, or explain anything. Output only the transcript.',
|
||||
'If you hear no intelligible speech, output nothing.',
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
function requiredParameter(parameters: Readonly<Record<string, unknown>>, name: string, model: string): string {
|
||||
const value = parameters[name]
|
||||
if (typeof value !== 'string' || !value)
|
||||
throw new Error(`Gemini transcription model "${model}" has no "${name}" parameter. Set it on its speech provider, for example through its env.`)
|
||||
return value
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a transcription executor for Gemini models, through the Gemini `generateContent` API.
|
||||
*
|
||||
* Use when:
|
||||
* - the `asr` axis selects a Gemini model. Gemini has no `/audio/transcriptions` endpoint, so the executor's `fetch`
|
||||
* turns the xsai transcription request into one `generateContent` call with the clip as inline audio.
|
||||
*
|
||||
* Expects:
|
||||
* - the provider parameters `baseURL`, the Gemini API root such as `https://generativelanguage.googleapis.com/v1beta/`,
|
||||
* and `apiKey`
|
||||
*
|
||||
* Returns:
|
||||
* - an executor whose `fetch` answers `{ text }`, as an OpenAI-compatible transcription endpoint does. A failed upstream
|
||||
* response is returned unchanged, so xsai reports its status and vieval can retry a rate limit.
|
||||
*/
|
||||
export function createGeminiTranscription(): TranscriptionExecutor {
|
||||
return {
|
||||
id: 'gemini',
|
||||
transcription: ({ model, parameters }) => {
|
||||
const apiKey = requiredParameter(parameters, 'apiKey', model)
|
||||
const baseURL = requiredParameter(parameters, 'baseURL', model)
|
||||
const root = baseURL.endsWith('/') ? baseURL : `${baseURL}/`
|
||||
|
||||
return {
|
||||
baseURL: root,
|
||||
model,
|
||||
fetch: async (_input: RequestInfo | URL, init?: RequestInit) => {
|
||||
// xsai sends the clip and its options as multipart form data. See `generateTranscription` of @xsai/generate-transcription.
|
||||
if (!(init?.body instanceof FormData))
|
||||
throw new TypeError('Gemini transcription expects the multipart request of xsai.')
|
||||
const file = init.body.get('file')
|
||||
if (!(file instanceof Blob))
|
||||
throw new TypeError('Gemini transcription request has no audio file.')
|
||||
const language = init.body.get('language')
|
||||
|
||||
const response = await fetch(new URL(`models/${model}:generateContent`, root), {
|
||||
body: JSON.stringify({
|
||||
contents: [{
|
||||
parts: [
|
||||
{ text: transcriptionPrompt(typeof language === 'string' ? language : undefined) },
|
||||
{ inline_data: { data: Buffer.from(await file.arrayBuffer()).toString('base64'), mime_type: file.type || 'audio/mpeg' } },
|
||||
],
|
||||
}],
|
||||
// A transcript of the same clip must not change between runs.
|
||||
generationConfig: { temperature: 0 },
|
||||
}),
|
||||
headers: { 'Content-Type': 'application/json', 'x-goog-api-key': apiKey },
|
||||
method: 'POST',
|
||||
signal: init.signal,
|
||||
})
|
||||
if (!response.ok)
|
||||
return response
|
||||
|
||||
const answer = await response.json() as GenerateContentResponse
|
||||
const parts = answer.candidates?.[0]?.content?.parts
|
||||
// No candidate means the model heard nothing to transcribe, so the transcript is empty and the listener marks the case `unclear`.
|
||||
const text = parts == null
|
||||
? ''
|
||||
: parts.filter(part => !part.thought && typeof part.text === 'string').map(part => part.text).join('').trim()
|
||||
return new Response(JSON.stringify({ text }), { headers: { 'Content-Type': 'application/json' } })
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
import type { TaskCacheRuntime } from 'vieval/core/runner'
|
||||
|
||||
import type { Language } from '../../cases/formula-speech/languages'
|
||||
import type { OpenAISpeechModelRuntimeConfig, TranscriptionRequestOptions } from './speech-models'
|
||||
|
||||
import { Buffer } from 'node:buffer'
|
||||
|
||||
import { generateSpeech } from '@xsai/generate-speech'
|
||||
import { generateTranscription } from '@xsai/generate-transcription'
|
||||
|
||||
import { voiceLanguage } from '../../cases/formula-speech/languages'
|
||||
import { cacheKey } from '../../shared/cache-key'
|
||||
import { voiceFor } from './speech-models'
|
||||
|
||||
export * from './speech-models'
|
||||
|
||||
/** One synthesized clip, kept in the vieval cache so that a rerun reuses it. */
|
||||
export interface SpeechClip {
|
||||
/** Cache file of the mp3. Reports copy it as the case artifact. */
|
||||
path: string
|
||||
bytes: Buffer
|
||||
/** True when the clip came from the cache, so this run sent no TTS request for it. */
|
||||
cached: boolean
|
||||
}
|
||||
|
||||
/** One transcript of a clip. */
|
||||
export interface Transcript {
|
||||
text: string
|
||||
/** True when the transcript came from the cache, so this run sent no ASR request for it. */
|
||||
cached: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Synthesizes the spoken text with the TTS model that the `tts` axis selected.
|
||||
* The clip is cached by endpoint, model, voice, and text. A model that takes the language also gets the BCP 47 tag of the
|
||||
* case language, and the tag is part of its cache key, so a clip that was read without a language is not reused.
|
||||
*/
|
||||
export async function synthesize(speech: OpenAISpeechModelRuntimeConfig, text: string, language: Language, cache: TaskCacheRuntime, signal: AbortSignal): Promise<SpeechClip> {
|
||||
const voice = voiceFor(speech, language)
|
||||
if (!voice)
|
||||
throw new Error(`TTS model ${speech.model} has no voice for ${language}.`)
|
||||
|
||||
const tag = speech.sendsLanguage ? voiceLanguage[language] : undefined
|
||||
const keyParts = tag ? [speech.baseURL, voice, text, tag] : [speech.baseURL, voice, text]
|
||||
const file = cache.namespace('speech').file({ ext: 'mp3', key: [speech.model, cacheKey(...keyParts)] })
|
||||
if (await file.exists())
|
||||
return { path: file.path, bytes: await file.readBuffer(), cached: true }
|
||||
|
||||
const audio = await generateSpeech({
|
||||
abortSignal: signal,
|
||||
apiKey: speech.apiKey,
|
||||
baseURL: speech.baseURL,
|
||||
input: text,
|
||||
model: speech.model,
|
||||
responseFormat: 'mp3',
|
||||
voice,
|
||||
...(tag ? { extraBody: { language: tag } } : {}),
|
||||
})
|
||||
const bytes = Buffer.from(audio)
|
||||
await file.writeBuffer(bytes)
|
||||
return { path: file.path, bytes, cached: false }
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribes a clip with the ASR model that the `asr` axis selected.
|
||||
* `modelId` is the registered model id, because an executor model has no endpoint that tells two models apart.
|
||||
* The transcript is cached by model id, language, and audio content.
|
||||
*/
|
||||
export async function transcribe(transcription: TranscriptionRequestOptions, modelId: string, clip: SpeechClip, language: Language, cache: TaskCacheRuntime, signal: AbortSignal): Promise<Transcript> {
|
||||
const file = cache.namespace('transcription').file({ ext: 'json', key: [modelId, cacheKey(language, clip.bytes.toString('base64'))] })
|
||||
if (await file.exists())
|
||||
return { text: (await file.readJson<{ text: string }>()).text, cached: true }
|
||||
|
||||
const { text } = await generateTranscription({
|
||||
...transcription,
|
||||
abortSignal: signal,
|
||||
file: new Blob([new Uint8Array(clip.bytes)], { type: 'audio/mpeg' }),
|
||||
fileName: 'speech.mp3',
|
||||
language,
|
||||
// A transcript of the same clip must not change between runs.
|
||||
temperature: '0',
|
||||
})
|
||||
const transcript = text.replace(/\s*\n\s*/g, ' ').trim()
|
||||
await file.writeJson({ text: transcript })
|
||||
return { text: transcript, cached: false }
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { openaiSpeechFromRunContext, openaiTranscriptionFromRunContext, speechModelFrom, speechModelFromRun, speechModelMatrix, SpeechModels, speechProviderFrom, SpeechProviders, transcriptionModelFrom, transcriptionModelFromEval, transcriptionRequestFromRunContext, voiceFor } from './speech-models'
|
||||
|
||||
describe('speechProviders', () => {
|
||||
it('resolves requiredEnv and optionalEnv provider parameters from config env', async () => {
|
||||
const plugin = SpeechProviders({
|
||||
providers: [speechProviderFrom({ id: 'tts', inferenceExecutor: 'openai', optionalEnv: { apiKey: 'TTS_API_KEY' }, requiredEnv: { baseURL: 'TTS_BASE_URL' } })],
|
||||
})
|
||||
|
||||
const result = await plugin.configVieval?.({ env: { TTS_BASE_URL: 'http://127.0.0.1:5933/v1/' }, models: [] })
|
||||
|
||||
expect(result?.speechProviders?.[0]?.parameters).toEqual({ baseURL: 'http://127.0.0.1:5933/v1/' })
|
||||
})
|
||||
|
||||
it('throws when a required env value is missing', () => {
|
||||
const plugin = SpeechProviders({
|
||||
providers: [speechProviderFrom({ id: 'tts', inferenceExecutor: 'openai', requiredEnv: { baseURL: 'TTS_BASE_URL' } })],
|
||||
})
|
||||
|
||||
expect(() => plugin.configVieval?.({ env: {}, models: [] })).toThrow('TTS_BASE_URL')
|
||||
})
|
||||
})
|
||||
|
||||
describe('speechModels', () => {
|
||||
const createMatrixContext = async () => {
|
||||
const providers = SpeechProviders({
|
||||
providers: [speechProviderFrom({ id: 'local', inferenceExecutor: 'openai', parameters: { apiKey: 'provider-key', baseURL: 'http://127.0.0.1:5933/v1/' } })],
|
||||
})
|
||||
const models = SpeechModels({
|
||||
models: [
|
||||
speechModelFrom({ id: 'azure', model: 'microsoft/v1', provider: 'local', voice: 'en-US-AriaNeural', voices: { zh: 'zh-CN-XiaoxiaoNeural' } }),
|
||||
transcriptionModelFrom({ model: 'whisper-1', parameters: { apiKey: '' }, provider: 'local' }),
|
||||
],
|
||||
})
|
||||
// A hook may return nothing to keep the config it got.
|
||||
const base = { env: {}, models: [] }
|
||||
const withProviders = await providers.configVieval!(base) ?? base
|
||||
const config = await models.configVieval!(withProviders) ?? withProviders
|
||||
|
||||
return {
|
||||
cache: {} as never,
|
||||
models: config.models ?? [],
|
||||
task: {
|
||||
entry: { description: 'desc', directory: 'd', filePath: '/tmp/a.eval.ts', id: 'entry', name: 'entry' },
|
||||
id: 'task-1',
|
||||
inferenceExecutor: { id: 'default' },
|
||||
matrix: {
|
||||
eval: { asr: 'whisper-1' },
|
||||
meta: { evalRowId: 'asr=whisper-1', runRowId: 'tts=azure' },
|
||||
run: { tts: 'azure' },
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
it('adds speech and transcription models to the shared model registry with their provider parameters', async () => {
|
||||
const context = await createMatrixContext()
|
||||
|
||||
expect(context.models.map(model => model.id)).toEqual(['azure', 'openai:whisper-1'])
|
||||
expect(context.models[0]?.parameters).toMatchObject({ apiKey: 'provider-key', baseURL: 'http://127.0.0.1:5933/v1/', capability: 'speech' })
|
||||
})
|
||||
|
||||
it('resolves models from matrix axes and turns them into runtime config', async () => {
|
||||
const context = await createMatrixContext()
|
||||
|
||||
const speech = openaiSpeechFromRunContext(speechModelFromRun(context, { axis: 'tts' }))
|
||||
expect(speech.model).toBe('microsoft/v1')
|
||||
expect(voiceFor(speech, 'zh')).toBe('zh-CN-XiaoxiaoNeural')
|
||||
expect(voiceFor(speech, 'ja')).toBe('en-US-AriaNeural')
|
||||
|
||||
// A model parameter wins over the provider parameter, so a local ASR server can run without a key.
|
||||
const transcription = openaiTranscriptionFromRunContext(transcriptionModelFromEval(context, { axis: 'asr' }))
|
||||
expect(transcription).toEqual({ apiKey: '', baseURL: 'http://127.0.0.1:5933/v1/', inferenceExecutor: 'openai', model: 'whisper-1' })
|
||||
})
|
||||
|
||||
it('rejects a model with the wrong capability, an unknown model, and a missing axis', async () => {
|
||||
const context = await createMatrixContext()
|
||||
const swapped = { ...context, task: { ...context.task, matrix: { ...context.task.matrix, run: { tts: 'whisper-1' } } } }
|
||||
|
||||
expect(() => speechModelFromRun(swapped, { axis: 'tts' })).toThrow('is not a speech model')
|
||||
expect(() => speechModelFromRun(context, { axis: 'voice' })).toThrow('Missing run matrix axis "voice".')
|
||||
expect(() => transcriptionModelFromEval({ ...context, models: [] }, { axis: 'asr' })).toThrow('Unknown configured speech model "whisper-1"')
|
||||
})
|
||||
|
||||
it('registers a model without env, and fails only when a task resolves it without its base URL', async () => {
|
||||
const providers = SpeechProviders({
|
||||
providers: [speechProviderFrom({ id: 'azure', inferenceExecutor: 'openai', optionalEnv: { baseURL: 'TTS_AZURE_BASE_URL' } })],
|
||||
})
|
||||
const models = SpeechModels({ models: [speechModelFrom({ id: 'azure', model: 'microsoft/v1', provider: 'azure' })] })
|
||||
const base = { env: {}, models: [] }
|
||||
const withProviders = await providers.configVieval!(base) ?? base
|
||||
const config = await models.configVieval!(withProviders) ?? withProviders
|
||||
const azure = config.models?.[0]
|
||||
|
||||
expect(azure?.id).toBe('azure')
|
||||
expect(() => openaiSpeechFromRunContext(azure!)).toThrow('Model "azure" has no "baseURL" parameter. Set it on speech provider "azure"')
|
||||
})
|
||||
|
||||
it('resolves a transcription executor per call, with the language of the call and the resolved provider env', async () => {
|
||||
const calls: Array<{ language?: string, model: string, parameters: Readonly<Record<string, unknown>> }> = []
|
||||
const executorFetch = async () => new Response(JSON.stringify({ text: 'ok' }))
|
||||
const providers = SpeechProviders({
|
||||
providers: [speechProviderFrom({
|
||||
id: 'native',
|
||||
inferenceExecutor: {
|
||||
id: 'native-asr',
|
||||
transcription: (options) => {
|
||||
calls.push(options)
|
||||
return { baseURL: 'native://asr/', fetch: executorFetch, model: options.model }
|
||||
},
|
||||
},
|
||||
optionalEnv: { apiKey: 'NATIVE_API_KEY' },
|
||||
})],
|
||||
})
|
||||
const models = SpeechModels({ models: [transcriptionModelFrom({ id: 'native', model: 'native-model', provider: 'native' })] })
|
||||
const base = { env: { NATIVE_API_KEY: 'native-key' }, models: [] }
|
||||
const withProviders = await providers.configVieval!(base) ?? base
|
||||
const config = await models.configVieval!(withProviders) ?? withProviders
|
||||
const native = config.models![0]!
|
||||
|
||||
expect(native.inferenceExecutorId).toBe('native-asr')
|
||||
const request = transcriptionRequestFromRunContext(native, { language: 'ja' })
|
||||
expect(request).toMatchObject({ baseURL: 'native://asr/', model: 'native-model' })
|
||||
expect(request.fetch).toBe(executorFetch)
|
||||
expect(calls).toEqual([{ language: 'ja', model: 'native-model', parameters: { apiKey: 'native-key', capability: 'transcription' } }])
|
||||
})
|
||||
|
||||
it('sends the language only for a model that takes it', async () => {
|
||||
const providers = SpeechProviders({
|
||||
providers: [speechProviderFrom({ id: 'local', inferenceExecutor: 'openai', parameters: { baseURL: 'http://127.0.0.1:5933/v1/' } })],
|
||||
})
|
||||
const models = SpeechModels({
|
||||
models: [
|
||||
speechModelFrom({ id: 'official', model: 'stepfun/stepaudio-2.5-tts', provider: 'local', sendsLanguage: true, voice: 'jilingshaonv' }),
|
||||
speechModelFrom({ id: 'azure', model: 'microsoft/v1', provider: 'local', voice: 'en-US-AriaNeural' }),
|
||||
],
|
||||
})
|
||||
const base = { env: {}, models: [] }
|
||||
const withProviders = await providers.configVieval!(base) ?? base
|
||||
const config = await models.configVieval!(withProviders) ?? withProviders
|
||||
const [official, azure] = config.models!
|
||||
|
||||
expect(openaiSpeechFromRunContext(official!).sendsLanguage).toBe(true)
|
||||
expect(openaiSpeechFromRunContext(azure!).sendsLanguage).toBe(false)
|
||||
})
|
||||
|
||||
it('throws when a model references an unknown provider', () => {
|
||||
const plugin = SpeechModels({ models: [speechModelFrom({ model: 'tts-1', provider: 'missing' })] })
|
||||
|
||||
expect(() => plugin.configVieval?.({ models: [] })).toThrow('Unknown speech provider "missing"')
|
||||
})
|
||||
|
||||
it('builds a matrix axis from model names', () => {
|
||||
expect(speechModelMatrix('tts', 'azure', 'official', 'azure')).toEqual({ tts: ['azure', 'official'] })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,486 @@
|
||||
import type { generateTranscription } from '@xsai/generate-transcription'
|
||||
import type { ConfigHookPlugin, MatrixDefinition, ModelDefinition, TaskRunContext } from 'vieval/config'
|
||||
|
||||
import process from 'node:process'
|
||||
|
||||
import { resolveModelByName } from 'vieval/config'
|
||||
import { envFrom, requiredEnvFrom } from 'vieval/core/inference-executors'
|
||||
|
||||
// WORKAROUND:
|
||||
// This module is one file, unlike `chat-models`, which splits its runtime config into its own module.
|
||||
// Configs import it, and vieval 0.4.0 loads configs with Node type stripping, which cannot follow an extensionless relative import.
|
||||
// Source: see eval/index.ts.
|
||||
// Remove when vieval loads configs with an extension-resolving loader, or when this module moves into vieval.
|
||||
//
|
||||
// This module mirrors `vieval/plugins/chat-models` for speech. It depends only on the public vieval API,
|
||||
// so it can move into vieval as `vieval/plugins/speech-models`.
|
||||
|
||||
/**
|
||||
* What a speech model does. A `speech` model synthesizes speech from text (TTS).
|
||||
* A `transcription` model turns speech into text (ASR).
|
||||
*/
|
||||
export type SpeechModelCapability = 'speech' | 'transcription'
|
||||
|
||||
/**
|
||||
* One provider definition that speech models reference by id.
|
||||
*
|
||||
* A provider is an OpenAI-compatible endpoint (`POST /audio/speech` and `POST /audio/transcriptions`), which most TTS and
|
||||
* ASR services expose, for example unspeech and a whisper.cpp server. A transcription provider can instead be a
|
||||
* {@link TranscriptionExecutor}, for example a native speech recognizer.
|
||||
*/
|
||||
export interface SpeechProviderDefinition {
|
||||
/**
|
||||
* Stable provider id referenced by `speechModelFrom({ provider })`.
|
||||
*/
|
||||
id: string
|
||||
inferenceExecutor: 'openai' | TranscriptionExecutor
|
||||
/**
|
||||
* Optional provider parameters resolved from env.
|
||||
*
|
||||
* Expects:
|
||||
* - map key is the provider parameter name, for example `apiKey`
|
||||
* - map value is the env key name
|
||||
*/
|
||||
optionalEnv?: Record<string, string>
|
||||
/**
|
||||
* Literal provider parameters.
|
||||
*/
|
||||
parameters?: Record<string, unknown>
|
||||
/**
|
||||
* Required provider parameters resolved from env. A missing key throws when the config loads.
|
||||
*
|
||||
* Expects:
|
||||
* - map key is the provider parameter name, for example `baseURL`
|
||||
* - map value is the env key name
|
||||
*/
|
||||
requiredEnv?: Record<string, string>
|
||||
}
|
||||
|
||||
/**
|
||||
* xsai request options of one transcription call. A runtime without an HTTP endpoint provides its own `fetch`,
|
||||
* as xsai providers do, so callers send every model the same `generateTranscription` request.
|
||||
*/
|
||||
export interface TranscriptionRequestOptions {
|
||||
apiKey?: string
|
||||
baseURL: string | URL
|
||||
/** The `fetch` of an xsai provider, so the same type serves every provider. */
|
||||
fetch?: Parameters<typeof generateTranscription>[0]['fetch']
|
||||
model: string
|
||||
}
|
||||
|
||||
/**
|
||||
* A transcription runtime that is not an OpenAI-compatible endpoint, for example Apple Speech on macOS.
|
||||
*/
|
||||
export interface TranscriptionExecutor {
|
||||
/** Stable id, used as the `inferenceExecutorId` of its models. */
|
||||
id: string
|
||||
/**
|
||||
* Builds the request options of one call. `language` is the language that the caller passes to `generateTranscription`,
|
||||
* so a runtime that needs one recognizer per language, for example per locale, can select it.
|
||||
* `parameters` are the provider and model parameters, with env already resolved. An executor that wraps an HTTP API
|
||||
* that is not OpenAI-compatible reads its endpoint and key from them.
|
||||
*/
|
||||
transcription: (options: { language?: string, model: string, parameters: Readonly<Record<string, unknown>> }) => TranscriptionRequestOptions
|
||||
}
|
||||
|
||||
/**
|
||||
* Speech-model specialization of the canonical `ModelDefinition`.
|
||||
* The capability and the voices are in `parameters`, so the model stays in the shared `config.models` registry.
|
||||
*/
|
||||
export type SpeechModelDefinition = Omit<ModelDefinition, 'inferenceExecutor'> & {
|
||||
inferenceExecutor: 'openai' | TranscriptionExecutor
|
||||
provider: string
|
||||
}
|
||||
|
||||
interface SpeechModelFromBaseOptions {
|
||||
/**
|
||||
* Alias names used by matrix lookup.
|
||||
*/
|
||||
aliases?: string[]
|
||||
/**
|
||||
* Stable model id. Matrix axes select the model by this id, its model name, or an alias.
|
||||
*
|
||||
* @default `openai:<model>`
|
||||
*/
|
||||
id?: string
|
||||
/**
|
||||
* Concrete model name sent to the provider.
|
||||
*/
|
||||
model: string
|
||||
/**
|
||||
* Literal model parameters. They override provider parameters with the same name.
|
||||
*/
|
||||
parameters?: Record<string, unknown>
|
||||
/**
|
||||
* Provider id from `SpeechProviders`.
|
||||
*/
|
||||
provider: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Builder input for `speechModelFrom`.
|
||||
*/
|
||||
export interface SpeechModelFromOptions extends SpeechModelFromBaseOptions {
|
||||
/**
|
||||
* The endpoint takes the language of the text as a BCP 47 tag in `extra_body.language`, as the AIRI speech API does.
|
||||
* Without it, a provider guesses the language, and a voice can read a short text, such as a digit, in its own language.
|
||||
*
|
||||
* @default false
|
||||
*/
|
||||
sendsLanguage?: boolean
|
||||
/**
|
||||
* The voice for any language that `voices` does not name.
|
||||
*/
|
||||
voice?: string
|
||||
/**
|
||||
* The voice of each language, keyed by the language code that eval code passes to `voiceFor`.
|
||||
*/
|
||||
voices?: Record<string, string>
|
||||
}
|
||||
|
||||
/**
|
||||
* Builder input for `transcriptionModelFrom`.
|
||||
*/
|
||||
export interface TranscriptionModelFromOptions extends SpeechModelFromBaseOptions {}
|
||||
|
||||
export interface SpeechModelsPluginOptions {
|
||||
/**
|
||||
* Speech and transcription model definitions to append to `config.models`.
|
||||
*/
|
||||
models: readonly SpeechModelDefinition[]
|
||||
}
|
||||
|
||||
export interface SpeechProvidersPluginOptions {
|
||||
/**
|
||||
* Env source for env-backed provider parameters.
|
||||
*
|
||||
* @default config.env, then process.env
|
||||
*/
|
||||
env?: NodeJS.ProcessEnv
|
||||
providers: readonly SpeechProviderDefinition[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Partial config shape that the speech plugins read and write.
|
||||
*/
|
||||
export interface SpeechPluginConfig {
|
||||
env?: NodeJS.ProcessEnv
|
||||
models?: ModelDefinition[]
|
||||
speechProviders?: SpeechProviderDefinition[]
|
||||
}
|
||||
|
||||
export type SpeechPlugin = ConfigHookPlugin<SpeechPluginConfig>
|
||||
|
||||
export type MatrixModelScope = 'eval' | 'run'
|
||||
|
||||
export interface SpeechModelFromMatrixOptions {
|
||||
/**
|
||||
* Matrix axis whose selected value is a model id, model name, or alias.
|
||||
*/
|
||||
axis: string
|
||||
}
|
||||
|
||||
type MatrixModelContext = Pick<TaskRunContext, 'models' | 'task'>
|
||||
|
||||
/**
|
||||
* Builds one speech-provider definition.
|
||||
*
|
||||
* Use when:
|
||||
* - several speech or transcription models share one endpoint and key
|
||||
*/
|
||||
export function speechProviderFrom(options: SpeechProviderDefinition): SpeechProviderDefinition {
|
||||
return { ...options }
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds one TTS model definition.
|
||||
*
|
||||
* Use when:
|
||||
* - a run or eval matrix axis selects the TTS model, for example to compare providers
|
||||
*/
|
||||
export function speechModelFrom(options: SpeechModelFromOptions): SpeechModelDefinition {
|
||||
return createModelDefinition(options, {
|
||||
capability: 'speech',
|
||||
sendsLanguage: options.sendsLanguage ?? false,
|
||||
voice: options.voice,
|
||||
voices: options.voices ?? {},
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds one ASR model definition.
|
||||
*
|
||||
* Use when:
|
||||
* - a matrix axis selects the speech recognition model, for example as part of the evaluator
|
||||
*/
|
||||
export function transcriptionModelFrom(options: TranscriptionModelFromOptions): SpeechModelDefinition {
|
||||
return createModelDefinition(options, { capability: 'transcription' })
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a matrix axis from configured speech model names.
|
||||
*
|
||||
* Returns:
|
||||
* - matrix axis object compatible with `runMatrix.extend/override` and `evalMatrix.extend/override`
|
||||
*/
|
||||
export function speechModelMatrix(axis: string, ...names: string[]): MatrixDefinition {
|
||||
return { [axis]: Array.from(new Set(names)) }
|
||||
}
|
||||
|
||||
/**
|
||||
* Contributes speech providers to config and resolves their env-backed parameters.
|
||||
* Register it before `SpeechModels`, which reads the providers.
|
||||
*/
|
||||
export function SpeechProviders(options: SpeechProvidersPluginOptions): SpeechPlugin {
|
||||
return {
|
||||
configVieval(config) {
|
||||
const env = config.env ?? options.env ?? process.env
|
||||
return {
|
||||
...config,
|
||||
speechProviders: [
|
||||
...(config.speechProviders ?? []),
|
||||
...options.providers.map(provider => ({ ...provider, parameters: resolveProviderParameters(provider, env) })),
|
||||
],
|
||||
}
|
||||
},
|
||||
name: 'vieval:speech-providers',
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Contributes speech and transcription models to the shared `config.models` registry.
|
||||
* Each model takes the parameters of its provider, and its own parameters win.
|
||||
*/
|
||||
export function SpeechModels(options: SpeechModelsPluginOptions): SpeechPlugin {
|
||||
return {
|
||||
configVieval(config) {
|
||||
const providers = new Map((config.speechProviders ?? []).map(provider => [provider.id, provider]))
|
||||
const models = options.models.map((model) => {
|
||||
const provider = providers.get(model.provider)
|
||||
if (provider == null)
|
||||
throw new Error(`Unknown speech provider "${model.provider}" referenced by model "${model.id}".`)
|
||||
const inferenceExecutor = provider.inferenceExecutor
|
||||
return {
|
||||
...model,
|
||||
inferenceExecutor,
|
||||
inferenceExecutorId: typeof inferenceExecutor === 'string' ? inferenceExecutor : inferenceExecutor.id,
|
||||
parameters: { ...provider.parameters, ...model.parameters },
|
||||
}
|
||||
})
|
||||
|
||||
return {
|
||||
...config,
|
||||
models: [...(config.models ?? []), ...models],
|
||||
}
|
||||
},
|
||||
name: 'vieval:speech-models',
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves a configured speech or transcription model from one scoped matrix axis.
|
||||
*
|
||||
* Expects:
|
||||
* - `options.axis` to exist in `context.task.matrix[scope]`
|
||||
* - the selected model to have the expected capability
|
||||
*/
|
||||
export function speechModelFromMatrix(
|
||||
context: MatrixModelContext,
|
||||
scope: MatrixModelScope,
|
||||
capability: SpeechModelCapability,
|
||||
options: SpeechModelFromMatrixOptions,
|
||||
): ModelDefinition {
|
||||
const selected = context.task.matrix[scope][options.axis]
|
||||
if (selected == null)
|
||||
throw new Error(`Missing ${scope} matrix axis "${options.axis}".`)
|
||||
|
||||
const model = resolveModelByName(context.models, String(selected))
|
||||
if (model == null)
|
||||
throw new Error(`Unknown configured speech model "${String(selected)}" from ${scope} matrix axis "${options.axis}".`)
|
||||
if (model.parameters?.capability !== capability)
|
||||
throw new Error(`Model "${model.id}" from ${scope} matrix axis "${options.axis}" is not a ${capability} model.`)
|
||||
|
||||
return model
|
||||
}
|
||||
|
||||
/** Resolves a TTS model from one run-matrix axis, for example when the TTS provider is part of the system under test. */
|
||||
export function speechModelFromRun(context: MatrixModelContext, options: SpeechModelFromMatrixOptions): ModelDefinition {
|
||||
return speechModelFromMatrix(context, 'run', 'speech', options)
|
||||
}
|
||||
|
||||
/** Resolves a TTS model from one eval-matrix axis, for example when TTS only renders text for an evaluator. */
|
||||
export function speechModelFromEval(context: MatrixModelContext, options: SpeechModelFromMatrixOptions): ModelDefinition {
|
||||
return speechModelFromMatrix(context, 'eval', 'speech', options)
|
||||
}
|
||||
|
||||
/** Resolves an ASR model from one run-matrix axis, for example when speech recognition is under test. */
|
||||
export function transcriptionModelFromRun(context: MatrixModelContext, options: SpeechModelFromMatrixOptions): ModelDefinition {
|
||||
return speechModelFromMatrix(context, 'run', 'transcription', options)
|
||||
}
|
||||
|
||||
/** Resolves an ASR model from one eval-matrix axis, for example when speech recognition is part of the evaluator. */
|
||||
export function transcriptionModelFromEval(context: MatrixModelContext, options: SpeechModelFromMatrixOptions): ModelDefinition {
|
||||
return speechModelFromMatrix(context, 'eval', 'transcription', options)
|
||||
}
|
||||
|
||||
function createModelDefinition(options: SpeechModelFromBaseOptions, parameters: Record<string, unknown>): SpeechModelDefinition {
|
||||
return {
|
||||
aliases: options.aliases ?? [],
|
||||
id: options.id ?? `openai:${options.model}`,
|
||||
inferenceExecutor: 'openai',
|
||||
inferenceExecutorId: 'openai',
|
||||
model: options.model,
|
||||
parameters: { ...options.parameters, ...parameters },
|
||||
provider: options.provider,
|
||||
}
|
||||
}
|
||||
|
||||
function resolveProviderParameters(provider: SpeechProviderDefinition, env: NodeJS.ProcessEnv): Record<string, unknown> {
|
||||
const parameters: Record<string, unknown> = { ...provider.parameters }
|
||||
for (const [name, key] of Object.entries(provider.optionalEnv ?? {})) {
|
||||
const value = envFrom(env, { name: key, type: 'string' })
|
||||
if (value != null)
|
||||
parameters[name] = value
|
||||
}
|
||||
for (const [name, key] of Object.entries(provider.requiredEnv ?? {}))
|
||||
parameters[name] = requiredEnvFrom(env, { name: key, type: 'string' })
|
||||
return parameters
|
||||
}
|
||||
|
||||
/**
|
||||
* Runtime config of one OpenAI-compatible TTS model, ready for a `POST /audio/speech` request.
|
||||
*/
|
||||
export interface OpenAISpeechModelRuntimeConfig {
|
||||
/**
|
||||
* API key. A local server, for example unspeech without auth, needs none.
|
||||
*
|
||||
* @default ''
|
||||
*/
|
||||
apiKey: string
|
||||
baseURL: string
|
||||
inferenceExecutor: 'openai'
|
||||
model: string
|
||||
/** True when the endpoint takes the language of the text in `extra_body.language`. */
|
||||
sendsLanguage: boolean
|
||||
/** The voice for any language that `voices` does not name. */
|
||||
voice?: string
|
||||
/** The voice of each language. */
|
||||
voices: Readonly<Record<string, string>>
|
||||
}
|
||||
|
||||
/**
|
||||
* Runtime config of one OpenAI-compatible ASR model, ready for a `POST /audio/transcriptions` request.
|
||||
*/
|
||||
export interface OpenAITranscriptionModelRuntimeConfig {
|
||||
/**
|
||||
* API key. A local server, for example whisper.cpp, needs none.
|
||||
*
|
||||
* @default ''
|
||||
*/
|
||||
apiKey: string
|
||||
baseURL: string
|
||||
inferenceExecutor: 'openai'
|
||||
model: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the runtime config of a TTS model that a speech-model helper resolved.
|
||||
*
|
||||
* Expects:
|
||||
* - `model` to be a `speech` model with a `baseURL` parameter
|
||||
*/
|
||||
export function openaiSpeechFromRunContext(model: ModelDefinition): OpenAISpeechModelRuntimeConfig {
|
||||
const parameters = parametersOf(model, 'speech')
|
||||
return {
|
||||
apiKey: optionalString(parameters, 'apiKey', model) ?? '',
|
||||
baseURL: requiredString(parameters, 'baseURL', model),
|
||||
inferenceExecutor: 'openai',
|
||||
model: model.model,
|
||||
sendsLanguage: parameters.sendsLanguage === true,
|
||||
voice: optionalString(parameters, 'voice', model),
|
||||
voices: stringRecord(parameters.voices, model),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the runtime config of an ASR model that a speech-model helper resolved.
|
||||
*
|
||||
* Expects:
|
||||
* - `model` to be a `transcription` model with a `baseURL` parameter
|
||||
*/
|
||||
export function openaiTranscriptionFromRunContext(model: ModelDefinition): OpenAITranscriptionModelRuntimeConfig {
|
||||
const parameters = parametersOf(model, 'transcription')
|
||||
return {
|
||||
apiKey: optionalString(parameters, 'apiKey', model) ?? '',
|
||||
baseURL: requiredString(parameters, 'baseURL', model),
|
||||
inferenceExecutor: 'openai',
|
||||
model: model.model,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the xsai request options of one transcription call, for an OpenAI-compatible model or a {@link TranscriptionExecutor}.
|
||||
*
|
||||
* Expects:
|
||||
* - `model` to be a `transcription` model
|
||||
* - `language` to be the language of the call. An executor can select its recognizer from it.
|
||||
*/
|
||||
export function transcriptionRequestFromRunContext(model: ModelDefinition, options: { language?: string } = {}): TranscriptionRequestOptions {
|
||||
if (model.parameters?.capability !== 'transcription')
|
||||
throw new Error(`Model "${model.id}" is not a transcription model.`)
|
||||
if (isTranscriptionExecutor(model.inferenceExecutor))
|
||||
return model.inferenceExecutor.transcription({ language: options.language, model: model.model, parameters: model.parameters ?? {} })
|
||||
return openaiTranscriptionFromRunContext(model)
|
||||
}
|
||||
|
||||
function isTranscriptionExecutor(value: unknown): value is TranscriptionExecutor {
|
||||
return typeof value === 'object' && value != null && 'transcription' in value && typeof value.transcription === 'function'
|
||||
}
|
||||
|
||||
/**
|
||||
* Selects the voice of one language: the language voice, or the default voice.
|
||||
* Returns `undefined` when the model has no voice for the language.
|
||||
*/
|
||||
export function voiceFor(config: Pick<OpenAISpeechModelRuntimeConfig, 'voice' | 'voices'>, language: string): string | undefined {
|
||||
return config.voices[language] ?? config.voice
|
||||
}
|
||||
|
||||
function parametersOf(model: ModelDefinition, capability: string): Record<string, unknown> {
|
||||
const parameters = model.parameters ?? {}
|
||||
if (model.inferenceExecutorId !== 'openai')
|
||||
throw new Error(`Expected an openai model, got "${model.inferenceExecutorId}" for "${model.id}".`)
|
||||
if (parameters.capability !== capability)
|
||||
throw new Error(`Model "${model.id}" is not a ${capability} model.`)
|
||||
return parameters
|
||||
}
|
||||
|
||||
function optionalString(parameters: Record<string, unknown>, name: string, model: ModelDefinition): string | undefined {
|
||||
const value = parameters[name]
|
||||
if (value == null || value === '')
|
||||
return undefined
|
||||
if (typeof value !== 'string')
|
||||
throw new Error(`Parameter "${name}" of model "${model.id}" must be a string.`)
|
||||
return value
|
||||
}
|
||||
|
||||
function requiredString(parameters: Record<string, unknown>, name: string, model: ModelDefinition): string {
|
||||
const value = optionalString(parameters, name, model)
|
||||
if (value != null)
|
||||
return value
|
||||
|
||||
// Providers resolve env lazily, so an unused model needs no env. A selected model reports what is missing here.
|
||||
const provider = 'provider' in model && typeof model.provider === 'string' ? ` Set it on speech provider "${model.provider}", for example through its env.` : ''
|
||||
throw new Error(`Model "${model.id}" has no "${name}" parameter.${provider}`)
|
||||
}
|
||||
|
||||
function isStringRecord(value: unknown): value is Record<string, string> {
|
||||
return typeof value === 'object' && value != null && Object.values(value).every(item => typeof item === 'string')
|
||||
}
|
||||
|
||||
function stringRecord(value: unknown, model: ModelDefinition): Record<string, string> {
|
||||
if (value == null)
|
||||
return {}
|
||||
if (!isStringRecord(value))
|
||||
throw new Error(`Parameter "voices" of model "${model.id}" must map languages to voice names.`)
|
||||
return value
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
// The package entry for test objects.
|
||||
//
|
||||
// WORKAROUND:
|
||||
// `./config`, `./speech-models`, and `./gemini-transcription` are separate package exports, so that a vieval config can import them.
|
||||
// vieval 0.4.0 loads configs with Node type stripping, which does not resolve a relative import without its extension,
|
||||
// and this file re-exports such imports. The repository lint rule forbids the `.ts` extension in relative imports.
|
||||
// Source: `Cannot find module .../src/method-config` when a test object config imported this module.
|
||||
// Remove when vieval loads configs through a resolver that adds extensions, for example jiti or Vite.
|
||||
export * from './cases/formula-speech'
|
||||
export * from './evaluators/grader'
|
||||
export * from './evaluators/listener'
|
||||
export * from './evaluators/speech'
|
||||
export * from './shared/cache-key'
|
||||
export * from './shared/telemetry'
|
||||
export * from './shared/velin'
|
||||
export * from './tasks/read-aloud'
|
||||
@@ -0,0 +1,12 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
|
||||
/**
|
||||
* Builds one cache key segment from every input that changes a cached answer.
|
||||
*
|
||||
* @example
|
||||
* cacheKey('zh', '$x$')
|
||||
* // => 'b5f3…' (32 hex characters)
|
||||
*/
|
||||
export function cacheKey(...parts: string[]): string {
|
||||
return createHash('sha256').update(JSON.stringify(parts)).digest('hex').slice(0, 32)
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { usageRecord } from './telemetry'
|
||||
|
||||
describe('usageRecord', () => {
|
||||
it('uses the Chat Completions names that the vieval viewer reads', () => {
|
||||
expect(usageRecord({ inputTokens: 120, outputTokens: 30, totalTokens: 150 })).toEqual({ completion_tokens: 30, prompt_tokens: 120, total_tokens: 150 })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,44 @@
|
||||
import type { GenerateTextResult } from '@xsai/generate-text'
|
||||
import type { TaskRunContext } from 'vieval/config'
|
||||
|
||||
import { generateText } from '@xsai/generate-text'
|
||||
import { emitChatModelErrorTelemetry, emitChatModelRequestTelemetry, emitChatModelResponseTelemetry } from 'vieval/plugins/chat-models'
|
||||
|
||||
type GenerateTextOptions = Parameters<typeof generateText>[0]
|
||||
|
||||
/**
|
||||
* Converts xsai usage to the Chat Completions field names that the vieval viewer reads.
|
||||
* xsai normalizes the provider usage to camel case, so the provider's own names are gone at this point.
|
||||
*
|
||||
* @example
|
||||
* usageRecord({ inputTokens: 120, outputTokens: 30, totalTokens: 150 })
|
||||
* // => { prompt_tokens: 120, completion_tokens: 30, total_tokens: 150 }
|
||||
*/
|
||||
export function usageRecord(usage: GenerateTextResult['totalUsage']): Record<string, number> {
|
||||
return { completion_tokens: usage.outputTokens, prompt_tokens: usage.inputTokens, total_tokens: usage.totalTokens }
|
||||
}
|
||||
|
||||
/**
|
||||
* Calls a chat model and reports the call to vieval: one `inference.requested` event, then `inference.responded` with the
|
||||
* token usage and the latency, or `inference.failed`. The viewer sums the usage per case and per model.
|
||||
*
|
||||
* Only calls that reach the provider report usage. A cached answer makes no call, so a report shows what the run spent.
|
||||
*/
|
||||
export async function generateTextWithTelemetry(context: TaskRunContext, role: string, options: GenerateTextOptions): Promise<GenerateTextResult> {
|
||||
const provider = { id: role, model: options.model }
|
||||
emitChatModelRequestTelemetry(context, { data: { role }, provider })
|
||||
const startedAt = performance.now()
|
||||
try {
|
||||
const result = await generateText(options)
|
||||
emitChatModelResponseTelemetry(context, {
|
||||
latencyMs: Math.round(performance.now() - startedAt),
|
||||
provider,
|
||||
response: { usage: usageRecord(result.totalUsage) },
|
||||
})
|
||||
return result
|
||||
}
|
||||
catch (error) {
|
||||
emitChatModelErrorTelemetry(context, { error, provider })
|
||||
throw error
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
import { readFile } from 'node:fs/promises'
|
||||
|
||||
import { renderMarkdownString } from '@velin-dev/core-vue/render-node'
|
||||
|
||||
/**
|
||||
* Rendered prompts by file and props. A run asks for the same few prompts thousands of times,
|
||||
* and each render compiles a Vue component, so every distinct prompt renders once per process.
|
||||
*/
|
||||
const rendered = new Map<string, Promise<string>>()
|
||||
|
||||
/**
|
||||
* Renders a Velin markdown prompt.
|
||||
*
|
||||
* Velin escapes backslashes, underscores, and braces in plain text when it writes markdown back.
|
||||
* Keep every LaTeX example of a prompt in a code span.
|
||||
*/
|
||||
export function renderVelinPrompt(file: URL, props: Readonly<Record<string, string>>): Promise<string> {
|
||||
const key = JSON.stringify([file.href, props])
|
||||
let prompt = rendered.get(key)
|
||||
if (!prompt) {
|
||||
prompt = readFile(file, 'utf8')
|
||||
.then(source => renderMarkdownString(source, props))
|
||||
.then(result => result.rendered.trim())
|
||||
rendered.set(key, prompt)
|
||||
}
|
||||
return prompt
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
import type { MatrixDefinition } from 'vieval/config'
|
||||
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { createGeminiTranscription } from '@proj-airi/evals-tts-speech-formula/gemini-transcription'
|
||||
import { speechModelFrom, SpeechModels, speechProviderFrom, SpeechProviders, transcriptionModelFrom } from '@proj-airi/evals-tts-speech-formula/speech-models'
|
||||
import { createAppleSpeechProvider } from '@xsai-apple-speech/transcription-native'
|
||||
import { defineConfig, loadEnv } from 'vieval'
|
||||
import { chatModelFrom, ChatModels, chatProviderFrom, ChatProviders } from 'vieval/plugins/chat-models'
|
||||
|
||||
/** Each language is one task file. During development, `EVAL_LANGUAGES=zh,en` limits a run to some of them. */
|
||||
const languages = [
|
||||
'zh',
|
||||
'en',
|
||||
'ja',
|
||||
'ko',
|
||||
]
|
||||
|
||||
/**
|
||||
* The listener answers three votes per case, so 4 cases send up to 12 requests at once.
|
||||
* `vieval run` runs each of the four test objects in its own process, and this pool is per process,
|
||||
* so the listener and ASR proxies get up to 48 requests at once. With 8, they answered 429 `model_cooldown`.
|
||||
*/
|
||||
const caseConcurrency = 4
|
||||
|
||||
/** The Apple Speech locale of each task language. Every one must be installed in the macOS speech settings. */
|
||||
const appleSpeechLocales: Readonly<Record<string, string>> = { zh: 'zh-CN', en: 'en-US', ja: 'ja-JP', ko: 'ko-KR' }
|
||||
|
||||
/**
|
||||
* Apple Speech on macOS. It is a native recognizer, not an endpoint, so it is a transcription executor.
|
||||
* Creating the provider loads nothing. The native addon loads on the first transcription.
|
||||
*/
|
||||
const appleSpeech = createAppleSpeechProvider()
|
||||
|
||||
/** The package root. Every test object reads `.env` and `.env.test` from here, so one file configures all of them. */
|
||||
const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), '../../..')
|
||||
|
||||
/**
|
||||
* The model plugins. They register every model below, whether the matrix selects it or not, as `ChatModels` does.
|
||||
*
|
||||
* Endpoints and keys come from `.env` and are optional at load time. Thus, a provider that the matrix does not select
|
||||
* needs no env. A selected model without its base URL fails when the task resolves it.
|
||||
*/
|
||||
const modelPlugins = [
|
||||
ChatProviders({
|
||||
providers: [chatProviderFrom({
|
||||
id: 'listener',
|
||||
inferenceExecutor: 'openai',
|
||||
optionalEnv: { baseURL: 'LISTENER_BASE_URL' },
|
||||
requiredEnv: { apiKey: 'LISTENER_API_KEY' },
|
||||
})],
|
||||
}),
|
||||
ChatModels({
|
||||
models: [chatModelFrom({
|
||||
model: 'gpt-6-luna',
|
||||
provider: 'listener',
|
||||
})],
|
||||
}),
|
||||
SpeechProviders({
|
||||
providers: [
|
||||
speechProviderFrom({
|
||||
id: 'official',
|
||||
inferenceExecutor: 'openai',
|
||||
optionalEnv: { apiKey: 'TTS_OFFICIAL_API_KEY', baseURL: 'TTS_OFFICIAL_BASE_URL' },
|
||||
}),
|
||||
speechProviderFrom({
|
||||
id: 'azure',
|
||||
inferenceExecutor: 'openai',
|
||||
optionalEnv: { apiKey: 'TTS_AZURE_API_KEY', baseURL: 'TTS_AZURE_BASE_URL' },
|
||||
}),
|
||||
speechProviderFrom({
|
||||
id: 'asr',
|
||||
inferenceExecutor: 'openai',
|
||||
optionalEnv: { apiKey: 'ASR_API_KEY', baseURL: 'ASR_BASE_URL' },
|
||||
}),
|
||||
speechProviderFrom({
|
||||
id: 'apple-speech',
|
||||
inferenceExecutor: {
|
||||
id: 'apple-speech',
|
||||
// One recognizer per locale. The `speech` transcriber is the model for long-form audio, not dictation.
|
||||
transcription: ({ language }) => appleSpeech.transcription({
|
||||
locale: appleSpeechLocales[language ?? 'en'] ?? 'en-US',
|
||||
transcriber: 'speech',
|
||||
}),
|
||||
},
|
||||
}),
|
||||
speechProviderFrom({
|
||||
id: 'gemini',
|
||||
// Gemini has no `/audio/transcriptions` endpoint, so the executor sends a `generateContent` request with the clip.
|
||||
inferenceExecutor: createGeminiTranscription(),
|
||||
optionalEnv: { apiKey: 'ASR_GEMINI_API_KEY', baseURL: 'ASR_GEMINI_BASE_URL' },
|
||||
}),
|
||||
],
|
||||
}),
|
||||
SpeechModels({
|
||||
models: [
|
||||
// The official AIRI provider and its default voice.
|
||||
speechModelFrom({
|
||||
id: 'official',
|
||||
model: 'stepfun/stepaudio-2.5-tts',
|
||||
provider: 'official',
|
||||
// The AIRI speech API passes the language to StepFun. Without it, this Chinese voice read English digits in Chinese.
|
||||
sendsLanguage: true,
|
||||
voice: 'jilingshaonv',
|
||||
// NOTICE:
|
||||
// `jilingshaonv` still read short English digits in Chinese in 3 of 9 clips with `language: en`.
|
||||
// `lively-girl` read them in English in 18 of 18 clips. No voice read bare Japanese digits reliably, so Japanese keeps the default.
|
||||
// Source: the voice A/B check of 2026-10-19 (official API, Gemini transcripts).
|
||||
// Remove when `DEFAULT_TTS_VOICES` of the official API recommends an English voice for `en-US` and this suite reads it.
|
||||
voices: { en: 'lively-girl' },
|
||||
}),
|
||||
// Azure through unspeech.
|
||||
speechModelFrom({
|
||||
id: 'azure',
|
||||
model: 'microsoft/v1',
|
||||
provider: 'azure',
|
||||
voices: { zh: 'zh-CN-XiaoxiaoNeural', en: 'en-US-AriaNeural', ja: 'ja-JP-NanamiNeural', ko: 'ko-KR-SunHiNeural' },
|
||||
}),
|
||||
transcriptionModelFrom({
|
||||
id: 'whisper-1',
|
||||
model: 'whisper-1',
|
||||
provider: 'asr',
|
||||
}),
|
||||
transcriptionModelFrom({
|
||||
id: 'apple-speech',
|
||||
model: 'apple-speech',
|
||||
provider: 'apple-speech',
|
||||
}),
|
||||
transcriptionModelFrom({
|
||||
id: 'gemini-3.8-flash',
|
||||
model: 'gemini-3.8-flash',
|
||||
provider: 'gemini',
|
||||
}),
|
||||
],
|
||||
}),
|
||||
]
|
||||
|
||||
/** Languages are tasks, not matrix values, so a development filter can drop some without changing what a task measures. */
|
||||
function selectedLanguages(env: NodeJS.ProcessEnv): string[] {
|
||||
const selected = env.EVAL_LANGUAGES?.split(',').map(language => language.trim()).filter(Boolean)
|
||||
return selected?.length ? languages.filter(language => selected.includes(language)) : languages
|
||||
}
|
||||
|
||||
/** A plugin that registers models: the chat or speech plugins of vieval. */
|
||||
type ModelPlugin
|
||||
= | ReturnType<typeof ChatProviders>
|
||||
| ReturnType<typeof ChatModels>
|
||||
| ReturnType<typeof SpeechProviders>
|
||||
| ReturnType<typeof SpeechModels>
|
||||
|
||||
/** What a test object adds to the shared config, for example the model that it calls. */
|
||||
export interface FormulaSpeechConfigOptions {
|
||||
/** Plugins that register the models of the test object. They run after the shared plugins. */
|
||||
plugins?: ModelPlugin[]
|
||||
/** Run-matrix axes of the test object, for example the model that the `llm` object rewrites with. */
|
||||
runMatrix?: MatrixDefinition
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds the workspace config of one test object.
|
||||
*
|
||||
* A test object is one way to turn a reply into spoken text. Its project runs the read-aloud tasks under its own `tasks/`
|
||||
* with the shared matrix, plus the axes of its own options.
|
||||
*/
|
||||
export function defineFormulaSpeechConfig(testObject: string, options: FormulaSpeechConfigOptions = {}) {
|
||||
const env = loadEnv('test', packageRoot, '')
|
||||
const selectedTts = env.EVAL_TTS?.split(',').map(id => id.trim()).filter(Boolean)
|
||||
const selectedAsr = env.EVAL_ASR?.split(',').map(id => id.trim()).filter(Boolean)
|
||||
|
||||
const matrix = {
|
||||
/** `none` grades the spoken text without TTS: the floor that AIRI controls. Every other value is a TTS model. `EVAL_TTS=none` skips TTS. */
|
||||
tts: selectedTts?.length ? selectedTts : ['none', 'official'],
|
||||
/**
|
||||
* Transcribes the clip of every TTS model. Unused when `tts` is only `none`. Add `whisper-1` or `gemini-3.8-flash` to compare.
|
||||
* `EVAL_ASR=gemini-3.8-flash` selects other ASR models for one run.
|
||||
*/
|
||||
asr: selectedAsr?.length ? selectedAsr : ['apple-speech'],
|
||||
/** Writes down what it heard as LaTeX. */
|
||||
listener: ['gpt-6-luna'],
|
||||
/** Judges whether the spoken text says the meaning or spells markup. It uses the endpoint of the listener. */
|
||||
judge: ['gpt-6-luna'],
|
||||
}
|
||||
|
||||
return defineConfig({
|
||||
env,
|
||||
plugins: [
|
||||
...modelPlugins,
|
||||
...(options.plugins ?? []),
|
||||
],
|
||||
scheduler: { concurrency: { case: caseConcurrency } },
|
||||
projects: [
|
||||
{
|
||||
name: `formula-speech-${testObject}`,
|
||||
root: '.',
|
||||
include: selectedLanguages(env).map(language => `tasks/read-aloud/formula-speech/${language}.eval.ts`),
|
||||
runMatrix: {
|
||||
override: {
|
||||
tts: matrix.tts,
|
||||
...options.runMatrix,
|
||||
},
|
||||
},
|
||||
evalMatrix: {
|
||||
override: {
|
||||
// The listener and the ASR model are part of the measurement, not of the test object.
|
||||
listener: matrix.listener,
|
||||
judge: matrix.judge,
|
||||
...(matrix.tts.some(id => id !== 'none') ? { asr: matrix.asr } : {}),
|
||||
},
|
||||
},
|
||||
},
|
||||
],
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,300 @@
|
||||
import type { TaskRunContext } from 'vieval/config'
|
||||
|
||||
import type { Language } from '../../cases/formula-speech'
|
||||
import type { Grade } from '../../evaluators/grader'
|
||||
import type { Reading } from '../../evaluators/judge'
|
||||
import type { HearingStatus } from '../../evaluators/listener'
|
||||
|
||||
import { env } from 'node:process'
|
||||
|
||||
import { errorMessageFrom } from '@moeru/std'
|
||||
import { defineMetric, defineScore, describeTask } from 'vieval'
|
||||
import { createRetryPolicy, isRetriableProviderError, readRetryAfter, runWithRetry } from 'vieval/core/inference-executors'
|
||||
import { modelFromEval, openaiFromRunContext } from 'vieval/plugins/chat-models'
|
||||
|
||||
import { formulaSpeechCases } from '../../cases/formula-speech'
|
||||
import { mathBridgeCases } from '../../cases/math-bridge'
|
||||
import { grade, majorityScore } from '../../evaluators/grader'
|
||||
import { judgeReading, judgeVotes, medianReading, readingScores } from '../../evaluators/judge'
|
||||
import { answerFormOf, listenerVotes, writeDown } from '../../evaluators/listener'
|
||||
import { leftoverNotation } from '../../evaluators/notation'
|
||||
import { openaiSpeechFromRunContext, speechModelFromRun, synthesize, transcribe, transcriptionModelFromEval, transcriptionRequestFromRunContext } from '../../evaluators/speech'
|
||||
|
||||
/**
|
||||
* `EVAL_DATASET` selects the cases: `formula-speech`, the stratified set, or `math-bridge`, a sample of arXiv formulas.
|
||||
* `EVAL_CASES` limits the run to some case ids, for example while a new rule is in progress.
|
||||
*/
|
||||
const dataset = env.EVAL_DATASET === 'math-bridge' ? mathBridgeCases : formulaSpeechCases
|
||||
const selectedIds = env.EVAL_CASES?.split(',').map(id => id.trim()).filter(Boolean)
|
||||
const cases = selectedIds?.length ? dataset.filter(item => selectedIds.includes(item.id)) : dataset
|
||||
|
||||
/** The listener wrote the formula back, so its reading is correct. A strict majority of the listener votes decides it. */
|
||||
const correct = defineScore('formula.correct', {
|
||||
description: { 'en': 'A listener model wrote the formula back from what it heard, and the grader matched it with the reference. A strict majority of the votes decides it.', 'zh-Hans': '听写模型根据听到的内容写回公式，判分器与参考答案比对一致。多数票决定。' },
|
||||
grader: { id: 'grader', kind: 'rule', reducer: 'majority', votes: listenerVotes },
|
||||
label: { 'en': 'Written back correctly', 'zh-Hans': '听写正确' },
|
||||
})
|
||||
/** The judge saw a reading that says the meaning: 1 for natural, 0.5 for mixed, and 0 for spelled markup. */
|
||||
const natural = defineScore('formula.natural', {
|
||||
description: { 'en': 'A judge model read the spoken text: 1 when it says the meaning, 0.5 when one or two parts spell markup, and 0 when it spells markup. It is the median of the votes.', 'zh-Hans': '评委模型阅读朗读文本：读出含义为 1，有一两处逐个念记号为 0.5，逐个念记号为 0。取投票中位数。' },
|
||||
grader: { id: 'judge', kind: 'model', reducer: 'median', votes: judgeVotes },
|
||||
label: { 'en': 'Natural reading', 'zh-Hans': '读法自然' },
|
||||
})
|
||||
/**
|
||||
* The headline score: the reading is correct and a listener wants to hear it. A mixed reading passes, because one word
|
||||
* for a bracket that carries meaning is not a fault. It replaces the mean of the two scores in vieval 0.9.
|
||||
*/
|
||||
const pass = defineScore('formula.pass', {
|
||||
combine: { all: [correct, { atLeast: readingScores.mixed, score: natural }] },
|
||||
description: { 'en': 'The reading is correct, and it is natural or mixed.', 'zh-Hans': '读法正确，并且自然或基本自然。' },
|
||||
label: { 'en': 'Correct and natural', 'zh-Hans': '正确且自然' },
|
||||
primary: true,
|
||||
})
|
||||
|
||||
const strata = defineMetric('tts_speech_formula.stratum', {
|
||||
description: { 'en': 'The group of notation that the case covers.', 'zh-Hans': '用例覆盖的记法类别。' },
|
||||
label: { 'en': 'Stratum', 'zh-Hans': '类别' },
|
||||
})
|
||||
const gradingRule = defineMetric('tts_speech_formula.rule', {
|
||||
description: { 'en': 'The grader check that decided the score.', 'zh-Hans': '决定分数的判分规则。' },
|
||||
label: { 'en': 'Grading rule', 'zh-Hans': '判分规则' },
|
||||
})
|
||||
const leftoverSymbols = defineMetric('tts_speech_formula.leftover_notation', {
|
||||
better: 'lower',
|
||||
description: { 'en': 'Symbols and LaTeX command names that the test object left in the spoken text, so TTS gets notation instead of words. It needs no TTS.', 'zh-Hans': '测试对象留在朗读文本里的符号和 LaTeX 命令名。TTS 拿到的是符号而不是词。不需要 TTS。' },
|
||||
label: { 'en': 'Leftover notation', 'zh-Hans': '残留记号' },
|
||||
unit: 'symbols',
|
||||
})
|
||||
const ttsCharacters = defineMetric('tts_speech_formula.tts.characters_sent', {
|
||||
better: 'lower',
|
||||
description: { 'en': 'Characters that this run sent to the TTS provider. A cached clip sends none. Most TTS providers bill by characters.', 'zh-Hans': '本次运行发给 TTS 服务的字符数。命中缓存时为 0。多数 TTS 服务按字符计费。' },
|
||||
label: { 'en': 'TTS characters sent', 'zh-Hans': 'TTS 发送字符数' },
|
||||
unit: 'characters',
|
||||
})
|
||||
const asrRequests = defineMetric('tts_speech_formula.asr.requests', {
|
||||
better: 'lower',
|
||||
description: { 'en': 'ASR requests that this run sent. A cached transcript sends none.', 'zh-Hans': '本次运行发出的 ASR 请求数。命中缓存时为 0。' },
|
||||
label: { 'en': 'ASR requests', 'zh-Hans': 'ASR 请求数' },
|
||||
unit: 'requests',
|
||||
})
|
||||
const hearing = defineMetric('tts_speech_formula.hearing', {
|
||||
description: { 'en': 'How well the listener could hear the case: ok, partial, or unclear. It is the status of most votes, and the worse one on a tie.', 'zh-Hans': '听写者能否听清：ok、partial 或 unclear。取多数票的状态，平票时取更差的。' },
|
||||
label: { 'en': 'Hearing', 'zh-Hans': '可听清程度' },
|
||||
})
|
||||
const readingMetric = defineMetric('tts_speech_formula.reading', {
|
||||
description: { 'en': 'How the judge saw the reading: natural says the meaning, spelled reads the markup, and mixed is between. It is the median of the judge votes.', 'zh-Hans': '评委对朗读方式的判断：natural 读出含义，spelled 逐个念记号，mixed 介于两者之间。取评委投票的中位数。' },
|
||||
label: { 'en': 'Reading', 'zh-Hans': '朗读方式' },
|
||||
})
|
||||
const votes = defineMetric('tts_speech_formula.votes', {
|
||||
description: { 'en': 'The grade of each listener answer. The case passes with a strict majority.', 'zh-Hans': '每次听写的得分。多数票通过才算通过。' },
|
||||
label: { 'en': 'Listener votes', 'zh-Hans': '听写投票' },
|
||||
})
|
||||
|
||||
/** Worst first, so a tie between two statuses selects the worse one. */
|
||||
const hearingOrder: readonly HearingStatus[] = ['unclear', 'partial', 'ok']
|
||||
|
||||
/**
|
||||
* The hearing status of a case: the status of most votes. On a tie, the worse status wins, so one clear vote
|
||||
* does not hide that the other votes could not hear the clip.
|
||||
*
|
||||
* @example
|
||||
* majorityHearing(['ok', 'unclear', 'unclear'])
|
||||
* // => 'unclear'
|
||||
* majorityHearing(['ok', 'partial', 'unclear'])
|
||||
* // => 'unclear'
|
||||
*/
|
||||
function majorityHearing(statuses: readonly HearingStatus[]): HearingStatus {
|
||||
let selected = hearingOrder[0]!
|
||||
let selectedCount = -1
|
||||
for (const status of hearingOrder) {
|
||||
const count = statuses.filter(item => item === status).length
|
||||
if (count > selectedCount) {
|
||||
selected = status
|
||||
selectedCount = count
|
||||
}
|
||||
}
|
||||
return selected
|
||||
}
|
||||
|
||||
/**
|
||||
* What a read-aloud adapter gets besides the reply: the case context of vieval.
|
||||
* An adapter that calls a model resolves it from `models` and `task.matrix`, caches its answers in `cache`,
|
||||
* and reports its usage to `events`, for example with `generateTextWithTelemetry`.
|
||||
*/
|
||||
export type ReadAloudContext = TaskRunContext & { signal: AbortSignal }
|
||||
|
||||
/**
|
||||
* The adapter of a test object: it turns one reply into the text that a TTS provider reads.
|
||||
* Each test object under `eval/test-objects/` implements one.
|
||||
*/
|
||||
export type ReadAloud = (markdown: string, language: Language, context: ReadAloudContext) => Promise<string>
|
||||
|
||||
// WORKAROUND:
|
||||
// Some calls never answer, and one case then holds the whole run for many minutes.
|
||||
// The official StepFun TTS sometimes holds a request open for more than 5 minutes, and a model can write without end.
|
||||
// Source: T-152 §5.1 (StepFun hangs over 5 minutes), and the Korean `rel-parallel` case on 2026-10-19.
|
||||
// Remove when every provider answers or fails within this time. The timeout fails the try, and `step` sends the call again.
|
||||
const stepTimeoutMs = 120_000
|
||||
|
||||
/** Total tries of one step, including the first. The waits between them add up to about 70 seconds. */
|
||||
const stepMaxAttempts = 4
|
||||
|
||||
// WORKAROUND:
|
||||
// The listener and rewrite proxies answer 429 `model_cooldown` for some seconds when four workspaces call them at once,
|
||||
// and the ASR proxy sometimes drops the TLS connection. The case retry of vieval waited only 0.5, 1, and 2 seconds,
|
||||
// so all retries of a case failed inside one cooldown.
|
||||
// Source: the comparison run attempt-2026-10-19T06-36-24-868Z, where 93 `llm` cases and 33 `rule` cases failed this way.
|
||||
// Remove when vieval limits and cools down calls per provider across workspaces.
|
||||
/**
|
||||
* Wait before retry `attempt` of a step: the provider's `Retry-After` when it sent a longer one, else 10, 20, then
|
||||
* 40 seconds, plus up to a quarter more.
|
||||
* The default backoff of vieval starts at 0.5 seconds, which is far shorter than one cooldown of the proxies.
|
||||
* The random part spreads the cases that one cooldown stopped together, so they do not call again at the same moment.
|
||||
*/
|
||||
function stepRetryDelayMs(attempt: number, error: unknown): number {
|
||||
const backoffMs = 10_000 * 2 ** (attempt - 1)
|
||||
return Math.max(readRetryAfter(error) ?? 0, backoffMs + Math.random() * backoffMs / 4)
|
||||
}
|
||||
|
||||
/**
|
||||
* Retries the calls that a rate limit, a server error, a dropped connection, or a timeout stopped
|
||||
* (`isRetriableProviderError` of vieval). `runWithRetry` stops waiting and retrying once the run is interrupted.
|
||||
*/
|
||||
const stepRetryPolicy = createRetryPolicy({ delayMs: stepRetryDelayMs, maxAttempts: stepMaxAttempts })
|
||||
|
||||
/** The provider error that `step` wrapped, or the error itself when no step wrapped it. */
|
||||
function causeOf(error: unknown): unknown {
|
||||
return error instanceof Error && error.cause != null ? error.cause : error
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one external call with a timeout and names it in the error.
|
||||
* A retriable failure tries the call again after a wait (`stepRetryPolicy`). Other errors and an aborted run fail
|
||||
* the step at once.
|
||||
* Node reports a network failure as `fetch failed` and keeps the reason, for example `ECONNRESET`, in `cause`.
|
||||
* The report then shows the step, the message, and the reason together. The thrown error keeps the provider error
|
||||
* in `cause`, so the case retry below can still classify it.
|
||||
*/
|
||||
async function step<T>(name: string, signal: AbortSignal, run: (signal: AbortSignal) => Promise<T>): Promise<T> {
|
||||
try {
|
||||
return await runWithRetry(() => run(AbortSignal.any([signal, AbortSignal.timeout(stepTimeoutMs)])), stepRetryPolicy, { signal })
|
||||
}
|
||||
catch (error) {
|
||||
const cause = error instanceof Error && error.cause != null ? ` (${errorMessageFrom(error.cause) ?? 'no reason'})` : ''
|
||||
throw new Error(`${name}: ${errorMessageFrom(error) ?? 'unknown error'}${cause}`, { cause: error })
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Registers the read-aloud task of one language for one test object.
|
||||
*
|
||||
* WORKAROUND:
|
||||
* Each language has its own eval file, because two tasks of one file overwrite each other's cases.
|
||||
* In vieval 0.4.0, the task id comes from the eval file, not from the `describeTask` name, so both tasks get one id.
|
||||
* Source: the first comparison run, where the `zh` cases of a file disappeared behind the `en` cases.
|
||||
* Remove when vieval derives the task id from the task name. Then one eval file can register all four languages.
|
||||
*
|
||||
* Call stack:
|
||||
*
|
||||
* eval/test-objects/<object>/tasks/read-aloud/formula-speech/<language>.eval.ts
|
||||
* -> {@link describeFormulaSpeechTask}
|
||||
* -> {@link ReadAloud} of the test object, then synthesize and transcribe (../../evaluators/speech) unless `tts` is `none`
|
||||
* -> writeDown (./listener), one call per vote
|
||||
* -> grade (./score)
|
||||
*/
|
||||
export function describeFormulaSpeechTask(language: Language, readAloud: ReadAloud) {
|
||||
describeTask(language, ({ casesFromInputs }) => {
|
||||
casesFromInputs('case', cases, async (context) => {
|
||||
const item = context.matrix.inputs
|
||||
// `none` sends the spoken text straight to the listener. It measures what AIRI hands to TTS, independent of any provider.
|
||||
const tts = String(context.task.matrix.run.tts)
|
||||
const { cache, signal } = context
|
||||
|
||||
// Reports of different test objects match one sample by this id, so it leaves the test object out.
|
||||
context.recordMetric('benchmark.case.id', `${item.id}|${language}|${tts}`)
|
||||
context.recordMetric(strata, item.stratum)
|
||||
|
||||
const markdown = typeof item.input === 'string' ? item.input : item.input[language]
|
||||
const spoken = await step('read aloud', signal, stepSignal => readAloud(markdown, language, { ...context, signal: stepSignal }))
|
||||
const leftover = leftoverNotation(spoken, markdown)
|
||||
context.recordMetric(leftoverSymbols, leftover.length)
|
||||
await context.recordArtifact('spoken.txt', { body: `${spoken}\n`, contentType: 'text/plain', label: { 'en': 'Spoken text', 'zh-Hans': '朗读文本' }, purpose: 'tts_speech_formula.spoken_text' })
|
||||
|
||||
let heard = spoken
|
||||
if (tts !== 'none' && spoken.trim()) {
|
||||
const speech = openaiSpeechFromRunContext(speechModelFromRun(context, { axis: 'tts' }))
|
||||
const clip = await step(`TTS ${speech.model}`, signal, stepSignal => synthesize(speech, spoken, language, cache, stepSignal))
|
||||
await context.recordArtifact('speech.mp3', { contentType: 'audio/mpeg', label: { 'en': 'Speech', 'zh-Hans': '语音' }, path: clip.path, purpose: 'tts_speech_formula.speech' })
|
||||
const asrModel = transcriptionModelFromEval(context, { axis: 'asr' })
|
||||
const transcription = transcriptionRequestFromRunContext(asrModel, { language })
|
||||
const transcript = await step(`ASR ${asrModel.id}`, signal, stepSignal => transcribe(transcription, asrModel.id, clip, language, cache, stepSignal))
|
||||
heard = transcript.text
|
||||
// Chat calls report token usage through vieval inference events. TTS and ASR calls are not chat calls,
|
||||
// and the vieval event schema accepts only chat, so their billing basis goes into metrics.
|
||||
context.recordMetric(ttsCharacters, clip.cached ? 0 : spoken.length)
|
||||
context.recordMetric(asrRequests, transcript.cached ? 0 : 1)
|
||||
}
|
||||
|
||||
const listener = openaiFromRunContext(modelFromEval(context, { axis: 'listener' }))
|
||||
// An OpenAI-compatible provider without a base URL is OpenAI itself.
|
||||
const endpoint = { apiKey: listener.apiKey, baseURL: listener.baseURL || 'https://api.openai.com/v1/', model: listener.model }
|
||||
let answers: Awaited<ReturnType<typeof writeDown>>[] = []
|
||||
let grades: Grade[] = []
|
||||
let score: 0 | 1 | undefined
|
||||
// An empty transcript of a spoken clip says nothing about the reading: the Gemini ASR returned nothing for some
|
||||
// long Chinese, Japanese, and Korean clips in every run. The case keeps its judge score, and its correctness is unscored.
|
||||
if (tts !== 'none' && spoken.trim() && !heard.trim()) {
|
||||
context.recordUnscored(correct, 'asr_empty', { rationale: 'The ASR returned an empty transcript.' })
|
||||
}
|
||||
else {
|
||||
const form = answerFormOf(item.mode)
|
||||
answers = await step(`listener ${endpoint.model}`, signal, stepSignal => Promise.all(Array.from({ length: listenerVotes }, (_, vote) => writeDown(heard, language, form, vote, endpoint, context, stepSignal))))
|
||||
// An unintelligible vote scores 0 under its own rule, so a report tells an unclear clip apart from a wrong reading.
|
||||
grades = answers.map((answer): Grade => answer.status === 'unclear' ? { score: 0, rule: 'unclear' } : grade(item, answer.latex))
|
||||
score = majorityScore(grades.map(({ score: vote }) => vote))
|
||||
context.recordMetric(hearing, majorityHearing(answers.map(({ status }) => status)))
|
||||
|
||||
// A score of 0 is a result, not a failure. The case fails only when a provider, a model, or the listener cannot be reached.
|
||||
const rule = grades.find(({ score: vote }) => vote === score)?.rule ?? grades[0]!.rule
|
||||
context.recordScore(correct, score, { metadata: { listener: endpoint.model }, rationale: `${grades.map(({ score: vote }) => vote).join(', ')} by ${rule}` })
|
||||
context.recordMetric(votes, grades.map(({ score: vote }) => vote))
|
||||
context.recordMetric(gradingRule, rule)
|
||||
}
|
||||
|
||||
// The listener checks that the reading is correct. The judge checks that it says the meaning instead of spelling
|
||||
// markup, because a listener model can write down "open parenthesis x close parenthesis" as well as "of x".
|
||||
// The judge reads the spoken text, not the transcript, so it judges only the test object.
|
||||
const judge = openaiFromRunContext(modelFromEval(context, { axis: 'judge' }))
|
||||
const judgeEndpoint = { apiKey: judge.apiKey, baseURL: judge.baseURL || 'https://api.openai.com/v1/', model: judge.model }
|
||||
const verdicts = await step(`judge ${judgeEndpoint.model}`, signal, stepSignal => Promise.all(Array.from({ length: judgeVotes }, (_, vote) => judgeReading(spoken, markdown, language, vote, judgeEndpoint, context, stepSignal))))
|
||||
const reading: Reading = medianReading(verdicts)
|
||||
context.recordScore(natural, readingScores[reading], {
|
||||
grader: { id: 'judge', kind: 'model', model: judgeEndpoint.model, reducer: 'median', votes: judgeVotes },
|
||||
rationale: verdicts.find(verdict => verdict.reading === reading)?.reason,
|
||||
})
|
||||
context.recordMetric(readingMetric, reading)
|
||||
await context.recordArtifact('result.json', {
|
||||
body: `${JSON.stringify({ id: item.id, language, tts, markdown, spoken, leftover, heard, reference: item.reference, answers, grades, score, verdicts, reading }, null, 2)}\n`,
|
||||
contentType: 'application/json',
|
||||
label: { 'en': 'Result', 'zh-Hans': '结果' },
|
||||
purpose: 'tts_speech_formula.result',
|
||||
})
|
||||
})
|
||||
}, {
|
||||
// `pass` is derived from the two recorded scores, so the task announces it before the first case.
|
||||
scores: [correct, natural, pass],
|
||||
// WORKAROUND:
|
||||
// Providers sometimes drop a connection, and the case fails with `fetch failed` for a reason outside the test object.
|
||||
// The ASR proxy and the listener proxy dropped some requests during the first full runs.
|
||||
// Source: the `ja` and `ko` cases of the official TTS in the smoke run on 2026-10-19, which passed when run again.
|
||||
// Remove when the providers are stable. A retry replays the case, but every finished step returns from the cache,
|
||||
// so only the failed call runs again. A score of 0 is not a failure, so retries never hide a wrong reading.
|
||||
// `step` already retries the calls that a rate limit or a dropped connection stops, so this retry is the last
|
||||
// resort when a provider stays unavailable for all tries of a step. It waits 60 seconds, longer than one cooldown
|
||||
// of the proxies, or the provider's `Retry-After` when that is longer.
|
||||
// `step` wraps the provider error in `cause`. A bug or a grader error is not a provider error, so it fails at once.
|
||||
autoRetry: 1,
|
||||
autoRetryDelay: (_retryIndex, error) => Math.max(readRetryAfter(causeOf(error)) ?? 0, 60_000),
|
||||
autoRetryIf: error => isRetriableProviderError(causeOf(error)),
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { rewritePrompt } from './read-aloud'
|
||||
|
||||
describe('rewritePrompt', () => {
|
||||
it('names the language of the voice', async () => {
|
||||
expect(await rewritePrompt('ko')).toContain('natural Korean words')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,40 @@
|
||||
import type { Language, ReadAloud } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { cacheKey, generateTextWithTelemetry, languageName, renderVelinPrompt } from '@proj-airi/evals-tts-speech-formula'
|
||||
import { modelFromRun, openaiFromRunContext } from 'vieval/plugins/chat-models'
|
||||
|
||||
export function rewritePrompt(language: Language): Promise<string> {
|
||||
return renderVelinPrompt(new URL('./rewrite.velin.md', import.meta.url), { language: languageName[language] })
|
||||
}
|
||||
|
||||
/**
|
||||
* A model rewrites the reply for speech. It is the upper reference, but too slow for the live path.
|
||||
* The answer is cached by model, prompt, and input, so a rerun makes no new call.
|
||||
*/
|
||||
export const readAloud: ReadAloud = async (markdown, language, context) => {
|
||||
const { cache, signal } = context
|
||||
// The `rewrite` run axis selects the model. `vieval.config.ts` of this test object registers it.
|
||||
const rewrite = openaiFromRunContext(modelFromRun(context, { axis: 'rewrite' }))
|
||||
// An OpenAI-compatible provider without a base URL is OpenAI itself.
|
||||
const endpoint = { apiKey: rewrite.apiKey, baseURL: rewrite.baseURL || 'https://api.openai.com/v1/', model: rewrite.model }
|
||||
const system = await rewritePrompt(language)
|
||||
const file = cache.namespace('rewrite').file({ ext: 'json', key: [endpoint.model, cacheKey(system, markdown)] })
|
||||
if (await file.exists())
|
||||
return (await file.readJson<{ text: string }>()).text
|
||||
|
||||
const { text } = await generateTextWithTelemetry(context, 'rewrite', {
|
||||
...endpoint,
|
||||
abortSignal: signal,
|
||||
// WORKAROUND:
|
||||
// A rewrite model can write without end, and the case never finishes.
|
||||
// xsai sends no output limit unless the caller sets one, and a spoken reply needs far fewer tokens.
|
||||
// Source: the same failure of the listener model, see eval/evaluators/listener.
|
||||
// Remove when the rewrite model always stops on its own.
|
||||
maxCompletionTokens: 2000,
|
||||
messages: [{ role: 'system', content: system }, { role: 'user', content: markdown }],
|
||||
temperature: 0,
|
||||
})
|
||||
const spoken = text?.trim() ?? ''
|
||||
await file.writeJson({ text: spoken })
|
||||
return spoken
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
<script setup>
|
||||
const props = defineProps({
|
||||
// The language name of the voice, for example `Mandarin Chinese`.
|
||||
language: { type: String, required: true },
|
||||
})
|
||||
</script>
|
||||
|
||||
## Role
|
||||
|
||||
You prepare text for a text-to-speech voice that speaks {{ props.language }}.
|
||||
|
||||
## Task
|
||||
|
||||
Rewrite the message so that a listener who only hears it can write down every formula exactly.
|
||||
|
||||
- Read math, chemistry, units, and money as natural {{ props.language }} words.
|
||||
- Read the structure too: which part is the numerator, where a group starts and ends, and the upper and lower limits.
|
||||
- Keep all other words.
|
||||
|
||||
## Output
|
||||
|
||||
- Do not explain or add content.
|
||||
- Do not use LaTeX, markdown, or math symbols.
|
||||
- Output only the text to speak.
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('en', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ja', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ko', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('zh', readAloud)
|
||||
@@ -0,0 +1,30 @@
|
||||
import { defineFormulaSpeechConfig } from '@proj-airi/evals-tts-speech-formula/config'
|
||||
import { chatModelFrom, ChatModels, chatProviderFrom, ChatProviders } from 'vieval/plugins/chat-models'
|
||||
|
||||
/** The rewrite model. It is a run axis of this test object, so a report names the model that wrote the spoken text. */
|
||||
const rewriteModels = ['gpt-6-luna']
|
||||
|
||||
export default defineFormulaSpeechConfig('llm', {
|
||||
plugins: [
|
||||
ChatProviders({
|
||||
providers: [
|
||||
chatProviderFrom({
|
||||
id: 'rewrite',
|
||||
inferenceExecutor: 'openai',
|
||||
optionalEnv: { baseURL: 'REWRITE_BASE_URL' },
|
||||
requiredEnv: { apiKey: 'REWRITE_API_KEY' },
|
||||
}),
|
||||
],
|
||||
}),
|
||||
ChatModels({
|
||||
models: rewriteModels.map((model) => {
|
||||
return chatModelFrom({ id: `rewrite:${model}`, model, provider: 'rewrite' })
|
||||
}),
|
||||
}),
|
||||
],
|
||||
runMatrix: {
|
||||
rewrite: rewriteModels.map((model) => {
|
||||
return `rewrite:${model}`
|
||||
}),
|
||||
},
|
||||
})
|
||||
@@ -0,0 +1,4 @@
|
||||
import type { ReadAloud } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
/** The markdown as the model wrote it. This is AIRI without formula speech, the lower reference. */
|
||||
export const readAloud: ReadAloud = async markdown => markdown
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('en', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ja', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ko', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('zh', readAloud)
|
||||
@@ -0,0 +1,3 @@
|
||||
import { defineFormulaSpeechConfig } from '@proj-airi/evals-tts-speech-formula/config'
|
||||
|
||||
export default defineFormulaSpeechConfig('raw')
|
||||
@@ -0,0 +1,23 @@
|
||||
import { toSpeechText } from '@proj-airi/core-agent'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { formulaSpeechCases, languages, voiceLanguage } from '../../../cases/formula-speech'
|
||||
import { leftoverNotation } from '../../../evaluators/notation'
|
||||
|
||||
describe('rule read-aloud', () => {
|
||||
// Runs the rule engine on every case, without TTS or ASR, so it costs nothing.
|
||||
// The snapshot lists the notation that the rule engine still hands to TTS. Fix the rule engine to shrink it.
|
||||
it('lists the notation that the rule engine leaves in the spoken text', () => {
|
||||
const leftovers: Record<string, string> = {}
|
||||
for (const item of formulaSpeechCases) {
|
||||
for (const language of languages) {
|
||||
const markdown = typeof item.input === 'string' ? item.input : item.input[language]
|
||||
const found = leftoverNotation(toSpeechText(markdown, { language: voiceLanguage[language] }), markdown)
|
||||
if (found.length > 0)
|
||||
leftovers[`${item.id}|${language}`] = found.join(' ')
|
||||
}
|
||||
}
|
||||
|
||||
expect(leftovers).toMatchInlineSnapshot(`{}`)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,7 @@
|
||||
import type { ReadAloud } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { toSpeechText } from '@proj-airi/core-agent'
|
||||
import { voiceLanguage } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
/** `toSpeechText` of `@proj-airi/core-agent`, the rule engine that AIRI ships. The voice language selects the formula language. */
|
||||
export const readAloud: ReadAloud = async (markdown, language) => toSpeechText(markdown, { language: voiceLanguage[language] })
|
||||
@@ -0,0 +1,123 @@
|
||||
import type { MathJson } from '@proj-airi/mathjson'
|
||||
|
||||
import { ComputeEngine } from '@cortex-js/compute-engine'
|
||||
import { MathSpeech } from '@proj-airi/speech-transformer-math'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { formulaSpeechCases } from '../../../cases/formula-speech'
|
||||
|
||||
// The built-in packs. The chemistry cases are skipped below, so the reader has no extension.
|
||||
const reader = new MathSpeech()
|
||||
|
||||
/**
|
||||
* The standard MathJSON heads that both trees use with one meaning. Other heads differ by design, see
|
||||
* docs/ai/adr/2026-10-20-speech-text-preprocessor.md: Compute Engine reads `AB` as a product and folds `50\%`, and the
|
||||
* speech tree keeps names, percents, units, and groups.
|
||||
*/
|
||||
const sharedHeads = new Set([
|
||||
'Equal',
|
||||
'NotEqual',
|
||||
'Less',
|
||||
'Greater',
|
||||
'LessEqual',
|
||||
'GreaterEqual',
|
||||
'Element',
|
||||
'NotElement',
|
||||
'SubsetEqual',
|
||||
'Subset',
|
||||
'Perpendicular',
|
||||
'Parallel',
|
||||
'Similar',
|
||||
'Congruent',
|
||||
'To',
|
||||
'Divide',
|
||||
'Power',
|
||||
'Sqrt',
|
||||
'Root',
|
||||
'Sum',
|
||||
'Product',
|
||||
'Integrate',
|
||||
'Limit',
|
||||
'D',
|
||||
'Interval',
|
||||
'Set',
|
||||
'Abs',
|
||||
'Union',
|
||||
'Intersection',
|
||||
'Add',
|
||||
'Subtract',
|
||||
'Negate',
|
||||
'PlusMinus',
|
||||
'Factorial',
|
||||
'Binomial',
|
||||
])
|
||||
|
||||
/** Compute Engine names of shared meanings that differ from the speech tree. */
|
||||
const computeEngineNames: Readonly<Record<string, string>> = { Tilde: 'Similar', TildeFullEqual: 'Congruent' }
|
||||
/** The signature `f: X \to Y` holds its arrow in `Mapping`. */
|
||||
const speechNames: Readonly<Record<string, string>> = { Mapping: 'To' }
|
||||
/**
|
||||
* Heads that chain. Compute Engine writes `a + b + c` as one `Add`, and the speech tree nests `Add` in pairs, so a
|
||||
* nested head of the same name counts once.
|
||||
*/
|
||||
const chained = new Set(['Add', 'Multiply', 'Equal', 'Less', 'Greater', 'LessEqual', 'GreaterEqual'])
|
||||
/** Compute Engine units are built from `Divide` and `Power`. The speech tree has `Unit`, so unit subtrees are skipped. */
|
||||
const unitHeads = new Set(['__unit__', 'Unit'])
|
||||
|
||||
/** Counts the shared heads of a MathJSON tree, for example `{ Divide: 2, Power: 1 }`. */
|
||||
function headCounts(expression: unknown, rename: Readonly<Record<string, string>> = {}): Map<string, number> {
|
||||
const counts = new Map<string, number>()
|
||||
const visit = (value: unknown, parent?: string) => {
|
||||
if (!Array.isArray(value))
|
||||
return
|
||||
const [head, ...args] = value as [unknown, ...unknown[]]
|
||||
if (typeof head !== 'string' || unitHeads.has(head))
|
||||
return
|
||||
const name = rename[head] ?? head
|
||||
if (sharedHeads.has(name) && (!chained.has(name) || parent !== name))
|
||||
counts.set(name, (counts.get(name) ?? 0) + 1)
|
||||
args.forEach(argument => visit(argument, name))
|
||||
}
|
||||
visit(expression)
|
||||
return counts
|
||||
}
|
||||
|
||||
/** Lists the heads that one tree has more often than the other, for example `Power -1` when the speech tree lacks one. */
|
||||
function difference(speech: Map<string, number>, reference: Map<string, number>): string {
|
||||
const heads = [...new Set([...speech.keys(), ...reference.keys()])].sort()
|
||||
return heads
|
||||
.map(head => [head, (speech.get(head) ?? 0) - (reference.get(head) ?? 0)] as const)
|
||||
.filter(([, delta]) => delta !== 0)
|
||||
.map(([head, delta]) => `${head} ${delta > 0 ? '+' : ''}${delta}`)
|
||||
.join(', ')
|
||||
}
|
||||
|
||||
describe('speech MathJSON against Compute Engine', () => {
|
||||
// Runs on every formula case without a model call. The snapshot lists where the shared heads differ, so a change in
|
||||
// lowering shows up here before an eval run. A `+` is a head that only the speech tree has, and a `-` one that it lacks.
|
||||
it('finds the same standard structures as Compute Engine', () => {
|
||||
const engine = new ComputeEngine()
|
||||
const differences: Record<string, string> = {}
|
||||
for (const item of formulaSpeechCases) {
|
||||
// Currency cases are sentences, and Compute Engine has no chemistry.
|
||||
if (typeof item.input !== 'string' || item.input.includes('\\ce'))
|
||||
continue
|
||||
const tex = item.input.replace(/^\$|\$$/g, '')
|
||||
const speech: MathJson = reader.toMathJson(tex)
|
||||
const reference = engine.parse(tex, { form: 'raw' }).json
|
||||
const found = difference(headCounts(speech, speechNames), headCounts(reference, computeEngineNames))
|
||||
if (found)
|
||||
differences[item.id] = found
|
||||
}
|
||||
|
||||
// Known differences that the speech tree keeps by design:
|
||||
// - root-quadratic: Compute Engine reads `-b \pm \sqrt{...}` as a `Measurement` with an uncertainty.
|
||||
// - unit-c: Compute Engine folds `3\times10^{8}` into one number, so it has no `Power`.
|
||||
expect(differences).toMatchInlineSnapshot(`
|
||||
{
|
||||
"root-quadratic": "PlusMinus +1",
|
||||
"unit-c": "Power +1",
|
||||
}
|
||||
`)
|
||||
})
|
||||
})
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('en', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ja', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ko', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('zh', readAloud)
|
||||
@@ -0,0 +1,3 @@
|
||||
import { defineFormulaSpeechConfig } from '@proj-airi/evals-tts-speech-formula/config'
|
||||
|
||||
export default defineFormulaSpeechConfig('rule')
|
||||
@@ -0,0 +1,55 @@
|
||||
import type { Language, ReadAloud } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { createRequire } from 'node:module'
|
||||
|
||||
import katex from 'katex'
|
||||
|
||||
import 'katex/contrib/mhchem'
|
||||
|
||||
/**
|
||||
* Reads formulas with speech-rule-engine.
|
||||
*
|
||||
* The engine keeps one global locale. Calls are queued, so concurrent cases cannot switch the locale in the middle of another read.
|
||||
*/
|
||||
class SpeechRuleEngineReader {
|
||||
// WORKAROUND:
|
||||
// The ESM entry `speech-rule-engine/js/index.js` loads its CLI, which requires `commander`, and it cannot find the locale files.
|
||||
// The CommonJS bundle that `main` names works, but the package ships types only for the ESM sources.
|
||||
// Source: node_modules/speech-rule-engine/package.json (no `exports` or `types`) and js/common/cli.js.
|
||||
// Remove when speech-rule-engine ships typed exports that load locales.
|
||||
private readonly engine: typeof import('speech-rule-engine/js/common/system.js') = createRequire(import.meta.url)('speech-rule-engine')
|
||||
private queue: Promise<unknown> = Promise.resolve()
|
||||
private locale: string | undefined
|
||||
|
||||
read(markdown: string, language: Language): Promise<string> {
|
||||
const next = this.queue.then(() => this.readNow(markdown, language))
|
||||
this.queue = next.catch(() => undefined)
|
||||
return next
|
||||
}
|
||||
|
||||
private async readNow(markdown: string, language: Language): Promise<string> {
|
||||
// Only English and Korean have locales. Chinese and Japanese get the English reading, as an SRE integration in AIRI would.
|
||||
const locale = language === 'ko' ? 'ko' : 'en'
|
||||
if (this.locale !== locale) {
|
||||
await this.engine.setupEngine({ locale, domain: 'clearspeak', modality: 'speech', style: 'default' })
|
||||
await this.engine.engineReady()
|
||||
this.locale = locale
|
||||
}
|
||||
return markdown
|
||||
.replace(/\$\$([\s\S]+?)\$\$|\$([^$]+)\$/g, (_, block: string | undefined, inline: string | undefined) => ` ${this.engine.toSpeech(mathml(block ?? inline ?? ''))} `)
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
}
|
||||
}
|
||||
|
||||
/** KaTeX MathML without the outer span. The span makes SRE also read the TeX annotation. */
|
||||
function mathml(tex: string): string {
|
||||
return katex.renderToString(tex, { output: 'mathml', throwOnError: false, strict: 'ignore' })
|
||||
.replace(/^<span class="katex">/, '')
|
||||
.replace(/<\/span>$/, '')
|
||||
}
|
||||
|
||||
const speechRuleEngine = new SpeechRuleEngineReader()
|
||||
|
||||
/** speech-rule-engine ClearSpeak on each formula. It has no Chinese or Japanese locale, so those languages get the English reading. */
|
||||
export const readAloud: ReadAloud = async (markdown, language) => speechRuleEngine.read(markdown, language)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('en', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ja', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('ko', readAloud)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
import { describeFormulaSpeechTask } from '@proj-airi/evals-tts-speech-formula'
|
||||
|
||||
import { readAloud } from '../../../src/read-aloud'
|
||||
|
||||
describeFormulaSpeechTask('zh', readAloud)
|
||||
@@ -0,0 +1,3 @@
|
||||
import { defineFormulaSpeechConfig } from '@proj-airi/evals-tts-speech-formula/config'
|
||||
|
||||
export default defineFormulaSpeechConfig('sre')
|
||||
@@ -0,0 +1,63 @@
|
||||
{
|
||||
"name": "@proj-airi/evals-tts-speech-formula",
|
||||
"type": "module",
|
||||
"private": true,
|
||||
"description": "Vieval suite for how well listeners understand formulas that AIRI reads aloud",
|
||||
"author": {
|
||||
"name": "Moeru AI Project AIRI Team",
|
||||
"email": "airi@moeru.ai",
|
||||
"url": "https://github.com/moeru-ai"
|
||||
},
|
||||
"license": "MIT",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/moeru-ai/airi.git",
|
||||
"directory": "evals/tts-speech-formula"
|
||||
},
|
||||
"exports": {
|
||||
".": "./eval/index.ts",
|
||||
"./config": "./eval/tasks/read-aloud/config.ts",
|
||||
"./gemini-transcription": "./eval/evaluators/speech/gemini-transcription.ts",
|
||||
"./speech-models": "./eval/evaluators/speech/speech-models.ts"
|
||||
},
|
||||
"scripts": {
|
||||
"eval:compare": "pnpm -F @proj-airi/core-agent build && vieval run --config ./vieval.config.ts --with-viewer",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"cases:math-bridge": "tsx eval/cases/math-bridge/fetch.ts",
|
||||
"judge:collect": "tsx eval/evaluators/judge/calibration/collect.ts",
|
||||
"judge:label": "vite --config eval/evaluators/judge/calibration/vite.config.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@cortex-js/compute-engine": "catalog:",
|
||||
"@duckdb/duckdb-wasm": "catalog:",
|
||||
"@moeru/std": "catalog:",
|
||||
"@proj-airi/core-agent": "workspace:^",
|
||||
"@proj-airi/duckdb-wasm": "catalog:",
|
||||
"@proj-airi/mathjson": "workspace:^",
|
||||
"@proj-airi/speech-transformer-math": "workspace:^",
|
||||
"@velin-dev/core-vue": "catalog:",
|
||||
"@vieval/ui": "catalog:",
|
||||
"@xsai-apple-speech/transcription-native": "catalog:",
|
||||
"@xsai/generate-speech": "catalog:",
|
||||
"@xsai/generate-text": "catalog:",
|
||||
"@xsai/generate-transcription": "catalog:",
|
||||
"katex": "catalog:",
|
||||
"speech-rule-engine": "catalog:",
|
||||
"valibot": "catalog:",
|
||||
"web-worker": "catalog:"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@xsai-apple-speech/transcription-native-darwin-arm64": "catalog:",
|
||||
"@xsai-apple-speech/transcription-native-darwin-x64": "catalog:"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "catalog:",
|
||||
"rehype-stringify": "catalog:",
|
||||
"remark-parse": "catalog:",
|
||||
"remark-rehype": "catalog:",
|
||||
"unified": "catalog:",
|
||||
"vieval": "catalog:",
|
||||
"vite": "catalog:",
|
||||
"vitest": "catalog:vitest"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ESNext",
|
||||
"lib": [
|
||||
"ESNext",
|
||||
"DOM"
|
||||
],
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"types": [
|
||||
"node"
|
||||
],
|
||||
"strict": true,
|
||||
"noEmit": true,
|
||||
"isolatedModules": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"skipLibCheck": true
|
||||
},
|
||||
"include": [
|
||||
"eval/**/*.ts",
|
||||
"vieval.config.ts",
|
||||
"vitest.config.ts"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
import { defineConfig } from 'vieval'
|
||||
|
||||
/**
|
||||
* Compares the test objects on the formula-speech dataset.
|
||||
*
|
||||
* Each directory under `eval/test-objects/` is one workspace: one way to turn a reply into spoken text.
|
||||
* `vieval run` runs every workspace in its own process, and `benchmark.case.id` aligns their cases, which leaves the test object out.
|
||||
* The listener, ASR, and TTS answers are cached once for all test objects under `.vieval/cache` next to this file.
|
||||
*/
|
||||
export default defineConfig({
|
||||
id: 'tts-speech-formula',
|
||||
cache: { namespace: 'formula-speech-v1' },
|
||||
workspaces: ['eval/test-objects/*'],
|
||||
})
|
||||
@@ -0,0 +1,8 @@
|
||||
import { defineConfig } from 'vitest/config'
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
// Unit tests cover the evaluators and the adapters. `*.eval.ts` files call paid models and run through vieval.
|
||||
include: ['eval/**/*.test.ts'],
|
||||
},
|
||||
})
|
||||
@@ -262,8 +262,31 @@ await response.finish()
|
||||
```
|
||||
|
||||
Producers can synthesize concurrently. Playback follows their reservation order.
|
||||
A producer holds its text until `end()`. Then `toSpeechText` removes markdown that has no sound, for example emphasis delimiters, link URLs, and code blocks.
|
||||
A producer streams its text to TTS by segments. `MarkdownSegmenter` ends a segment after a sentence or a line, only where no formula, code, link, emphasis, or table is open. Then `toSpeechText` reads the segment and removes markdown that has no sound, for example emphasis delimiters, link URLs, and code blocks.
|
||||
So an answer starts to speak after its first sentence, and a formula that later writes complete is still read as one formula.
|
||||
Symbols that markdown keeps as text, for example `2 * 3` and `x ~= 3`, stay in the speech text.
|
||||
TeX formulas, `\ce{}` chemistry, units, and dollar prices are read as words of the sentence language: Chinese, English, Japanese, or Korean.
|
||||
The rules are in [`@proj-airi/speech-transformer-math`](../speech-transformer-math) and [`@proj-airi/speech-transformer-chem`](../speech-transformer-chem), see [the ADR](../../docs/ai/adr/2026-10-20-speech-text-preprocessor.md).
|
||||
`mathSpeech` is the instance that AIRI uses: the built-in locale packs, with chemistry.
|
||||
|
||||
`toMathJson(tex)` returns the MathJSON of a formula without reading it, for tools that inspect formulas.
|
||||
`options.language` is the BCP 47 code of the voice. The words around a formula or a price select its language first.
|
||||
|
||||
`toSpeechText` is a host of preprocessor plugins. The default plugins are `moneySpeechPreprocessor` and `mathSpeechPreprocessor`.
|
||||
A plugin can have three hooks: `prepare` rewrites the markdown, `formula` reads one formula, and `prose` rewrites a run of text.
|
||||
To add a reading, pass `preprocessors: [myPlugin, ...defaultSpeechPreprocessors()]`.
|
||||
|
||||
```ts
|
||||
const spoken = toSpeechText(markdown, {
|
||||
preprocessors: [
|
||||
{ name: 'brand', prose: text => text.replaceAll('AIRI', 'ai ree') },
|
||||
...defaultSpeechPreprocessors(),
|
||||
],
|
||||
})
|
||||
```
|
||||
|
||||
The words of each language are in a locale pack of `@proj-airi/speech-transformer-math`.
|
||||
To add a language or change one, call `mathSpeech.register(pack)`. A language without a pack reads in English.
|
||||
Special tokens and flushes keep their position relative to the text.
|
||||
Cancelling an acknowledgment releases its place without reporting a user interruption.
|
||||
Whole-turn interruption aborts generation, synthesis, queued audio, and playback for that response.
|
||||
|
||||
@@ -44,11 +44,14 @@
|
||||
"dependencies": {
|
||||
"@moeru/eventa": "catalog:",
|
||||
"@moeru/std": "catalog:",
|
||||
"@proj-airi/mathjson": "workspace:^",
|
||||
"@proj-airi/pipelines-audio": "workspace:^",
|
||||
"@proj-airi/plugin-protocol": "workspace:^",
|
||||
"@proj-airi/provider-inference": "workspace:^",
|
||||
"@proj-airi/server-sdk": "workspace:^",
|
||||
"@proj-airi/server-shared": "workspace:^",
|
||||
"@proj-airi/speech-transformer-chem": "workspace:^",
|
||||
"@proj-airi/speech-transformer-math": "workspace:^",
|
||||
"@proj-airi/stream-kit": "workspace:^",
|
||||
"@xsai-ext/providers": "catalog:",
|
||||
"@xsai-ext/responses": "catalog:",
|
||||
@@ -62,6 +65,7 @@
|
||||
"nanoid": "catalog:",
|
||||
"rehype-parse": "catalog:",
|
||||
"rehype-stringify": "catalog:",
|
||||
"remark-math": "catalog:",
|
||||
"remark-parse": "catalog:",
|
||||
"unified": "catalog:",
|
||||
"unist-util-visit": "catalog:",
|
||||
|
||||
@@ -26,6 +26,10 @@ export type {
|
||||
export type { Interruption, VoiceInterruptionEvent } from './interruption'
|
||||
export { SpeechStream, VoiceResponse } from './output/response'
|
||||
export type { SpeechAudio, SpeechClip, SpeechClipEnd, SpeechOutput } from './output/response'
|
||||
export { defaultSpeechPreprocessors, toSpeechText } from './output/speech-transformation'
|
||||
export type { SpeechTextOptions } from './output/speech-transformation'
|
||||
export { mathSpeech, mathSpeechPreprocessor, moneySpeechPreprocessor, speakMath, toMathJson } from './output/speech-transformation/preprocessor'
|
||||
export type { SpeechFormulaContext, SpeechPreprocessor, SpeechProseContext } from './output/speech-transformation/preprocessor'
|
||||
export type {
|
||||
AudioPluginTask,
|
||||
FinalizeContext,
|
||||
|
||||
@@ -155,6 +155,47 @@ describe('voiceController output', () => {
|
||||
await controller.close()
|
||||
})
|
||||
|
||||
it('reads a formula that two writes split as one formula in the voice language', async () => {
|
||||
const requested: string[] = []
|
||||
const playback = new Playback({ nowMs: () => 0, play: () => ({ done: Promise.resolve({ throughMs: 1 }), stop: async () => ({ throughMs: 0 }) }) })
|
||||
const controller = new VoiceController({ speech: () => ({ playback, language: 'zh-CN', synthesize: async (request) => {
|
||||
requested.push(request.text)
|
||||
return new Blob([request.text])
|
||||
} }) })
|
||||
const response = controller.openResponse({ sessionId: 'alice', turnId: 'formula' })
|
||||
const speech = response.openSpeech({ purpose: 'answer' })
|
||||
// A model streams tokens, so the closing `$` and the `\\to` arrive in later writes.
|
||||
speech.write('结果是 $\\lim_{n \\t')
|
||||
speech.write('o \\infty} \\frac{1}{n} = 0$。\n\n$$\n')
|
||||
speech.write('x \\in A\n$$')
|
||||
speech.end()
|
||||
|
||||
expect(await response.finish()).toBe('finished')
|
||||
expect(requested).toEqual(['结果是 当n趋于无穷大时 n分之1的极限 等于 0。', 'x 属于 A'])
|
||||
await controller.close()
|
||||
})
|
||||
|
||||
it('synthesizes the first sentence before the model ends, and holds an open formula', async () => {
|
||||
const requested: string[] = []
|
||||
const playback = new Playback({ nowMs: () => 0, play: () => ({ done: Promise.resolve({ throughMs: 1 }), stop: async () => ({ throughMs: 0 }) }) })
|
||||
const controller = new VoiceController({ speech: () => ({ playback, language: 'en-US', synthesize: async (request) => {
|
||||
requested.push(request.text)
|
||||
return new Blob([request.text])
|
||||
} }) })
|
||||
const response = controller.openResponse({ sessionId: 'alice', turnId: 'stream' })
|
||||
const speech = response.openSpeech({ purpose: 'answer' })
|
||||
speech.write('The answer is ready. The area is $x^')
|
||||
await vi.waitFor(() => expect(requested).toEqual(['The answer is ready.']))
|
||||
|
||||
// The formula is still open, so its source does not reach TTS.
|
||||
speech.write('2$. Done')
|
||||
speech.end()
|
||||
|
||||
expect(await response.finish()).toBe('finished')
|
||||
expect(requested).toEqual(['The answer is ready.', 'The area is x squared.', 'Done'])
|
||||
await controller.close()
|
||||
})
|
||||
|
||||
it('retains speech-onset samples while waiting for playback silence', async () => {
|
||||
const source = createPushStream<import('@proj-airi/pipelines-audio').PcmBlock>()
|
||||
const audio = new AudioInput(pcmSource(source.stream), { historyMs: 4 })
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import type { AudioPlayback, IntentHandle, PcmBlock, PlaybackGroup, PlaybackItem, PlaybackReceipt, SpeechPipelineEvents, SpeechPipelineOptions } from '@proj-airi/pipelines-audio'
|
||||
|
||||
import type { Outcome } from '@proj-airi/stream-kit'
|
||||
|
||||
import type { TurnRef } from '../turn'
|
||||
@@ -9,7 +8,8 @@ import { createPushStream } from '@proj-airi/stream-kit'
|
||||
import { nanoid } from 'nanoid/non-secure'
|
||||
|
||||
import { errorFromCause, errorMessageFromValue } from '../../utils/error'
|
||||
import { toSpeechText } from './speech-text'
|
||||
import { toSpeechText } from './speech-transformation'
|
||||
import { MarkdownSegmenter } from './speech-transformation/segmenter'
|
||||
|
||||
/** Each synthesized part carries optional display text. Audio ownership transfers to playback. */
|
||||
export interface SpeechAudio {
|
||||
@@ -43,6 +43,8 @@ export type SpeechOutput = {
|
||||
readonly onPlaybackStart?: (clip: SpeechClip) => void
|
||||
/** Runs once for each started clip after playback ends, stops, or fails. */
|
||||
readonly onPlaybackEnd?: (clip: SpeechClip, end: SpeechClipEnd) => void
|
||||
/** BCP 47 code of the voice. The words around a formula or a price select its language first. */
|
||||
readonly language?: string
|
||||
} & ({
|
||||
readonly synthesize: SpeechPipelineOptions<Blob>['tts']
|
||||
} | {
|
||||
@@ -51,7 +53,7 @@ export type SpeechOutput = {
|
||||
})
|
||||
|
||||
/**
|
||||
* Marks where a special token or a flush occurs in the buffered text, so that markdown conversion keeps its position.
|
||||
* Marks where a special token or a flush occurs in the streamed text, so that markdown conversion keeps its position.
|
||||
* It is a Unicode private use character. Model text does not contain it, and producer text is stripped of it.
|
||||
*/
|
||||
const controlMarker = '\uE000'
|
||||
@@ -75,9 +77,12 @@ export class SpeechStream {
|
||||
private settled = false
|
||||
|
||||
private timer: ReturnType<typeof setTimeout> | undefined
|
||||
/** Markdown that `end()` converts to speech text. A control marker holds the place of each entry in `controls`. */
|
||||
private markdown = ''
|
||||
/** Holds the text after the last safe end of the streamed markdown. A control marker holds the place of a control. */
|
||||
private readonly segmenter = new MarkdownSegmenter()
|
||||
/** Controls whose markers are still in the segmenter, in order. */
|
||||
private readonly controls: SpeechControl[] = []
|
||||
/** The spoken text of the last segment, for the language of a formula that starts the next one. */
|
||||
private spoken = ''
|
||||
readonly signal = this.abort.signal
|
||||
readonly done = this.completion.promise
|
||||
|
||||
@@ -133,38 +138,39 @@ export class SpeechStream {
|
||||
const clip = { id: item.segmentId, text: item.text }
|
||||
let started = false
|
||||
void Promise.race([previous, this.done])
|
||||
.then(async () => {
|
||||
if (this.signal.aborted || response.signal.aborted)
|
||||
return 'stopped' as const
|
||||
.then(async () => {
|
||||
if (this.signal.aborted || response.signal.aborted)
|
||||
return 'stopped' as const
|
||||
|
||||
return response.playback.enqueue({
|
||||
id: item.id,
|
||||
audio: item.audio,
|
||||
signal: this.signal,
|
||||
onStart: () => {
|
||||
started = true
|
||||
listeners.start.forEach(listener => listener({ item, startedAt: Date.now() }))
|
||||
output.onPlaybackStart?.(clip)
|
||||
} })
|
||||
})
|
||||
.then((result) => {
|
||||
if (started)
|
||||
output.onPlaybackEnd?.(clip, this.clipEnd(result))
|
||||
return response.playback.enqueue({
|
||||
id: item.id,
|
||||
audio: item.audio,
|
||||
signal: this.signal,
|
||||
onStart: () => {
|
||||
started = true
|
||||
listeners.start.forEach(listener => listener({ item, startedAt: Date.now() }))
|
||||
output.onPlaybackStart?.(clip)
|
||||
},
|
||||
})
|
||||
})
|
||||
.then((result) => {
|
||||
if (started)
|
||||
output.onPlaybackEnd?.(clip, this.clipEnd(result))
|
||||
|
||||
if (!pending.delete(item))
|
||||
return
|
||||
if (!pending.delete(item))
|
||||
return
|
||||
|
||||
if (result === 'ended')
|
||||
listeners.end.forEach(listener => listener({ item, endedAt: Date.now() }))
|
||||
else
|
||||
listeners.interrupt.forEach(listener => listener({ item, interruptedAt: Date.now(), reason: result }))
|
||||
if (result === 'ended')
|
||||
listeners.end.forEach(listener => listener({ item, endedAt: Date.now() }))
|
||||
else
|
||||
listeners.interrupt.forEach(listener => listener({ item, interruptedAt: Date.now(), reason: result }))
|
||||
|
||||
// REVIEW: can we have more manage-able error handling
|
||||
// across our pipelines?
|
||||
if (result === 'failed')
|
||||
this.fail(new Error('Speech playback failed'))
|
||||
// REVIEW: can we have more manage-able error handling
|
||||
// across our pipelines?
|
||||
if (result === 'failed')
|
||||
this.fail(new Error('Speech playback failed'))
|
||||
// REVIEW: must we do this here?
|
||||
}, cause => this.fail(errorFromCause(cause, 'Speech playback failed')))
|
||||
}, cause => this.fail(errorFromCause(cause, 'Speech playback failed')))
|
||||
},
|
||||
stopByIntent: (_id, reason) => {
|
||||
for (const item of pending)
|
||||
@@ -249,33 +255,32 @@ export class SpeechStream {
|
||||
return signal.aborted ? { status, reason: errorMessageFromValue(signal.reason) } : { status }
|
||||
}
|
||||
|
||||
// REVIEW:
|
||||
// Every purpose holds its text until `end()`, so markdown is converted with the full context.
|
||||
// read-aloud already has the full text, but answer and notification start to speak only after the model ends.
|
||||
// answer needs a streaming commit at clause boundaries. See the chunker punctuation in pipelines-audio.
|
||||
// Text streams to TTS by segments, so an answer starts to speak after its first sentence. The segmenter ends a segment
|
||||
// only where markdown parses alone: a formula, a code block, or a link that is still open waits for its end.
|
||||
|
||||
/** Special tokens keep their position relative to the text. They are delivered when the producer ends. */
|
||||
/** Special tokens keep their position relative to the text. */
|
||||
special(value: string) {
|
||||
if (this.sealed || this.response.signal.aborted)
|
||||
return
|
||||
|
||||
this.markdown += controlMarker
|
||||
this.controls.push({ type: 'special', value })
|
||||
this.deliver(this.segmenter.push(controlMarker))
|
||||
}
|
||||
|
||||
/** Sends the text so far, unless a formula or another construct is still open. Then the flush follows its end. */
|
||||
flush() {
|
||||
if (this.sealed)
|
||||
return
|
||||
|
||||
this.markdown += controlMarker
|
||||
this.controls.push({ type: 'flush' })
|
||||
this.deliver([...this.segmenter.push(controlMarker), ...this.segmenter.flush()])
|
||||
}
|
||||
|
||||
write(text: string): { status: 'accepted' | 'closed' } {
|
||||
if (this.sealed || this.response.signal.aborted)
|
||||
return { status: 'closed' }
|
||||
|
||||
this.markdown += text.replaceAll(controlMarker, '')
|
||||
this.deliver(this.segmenter.push(text.replaceAll(controlMarker, '')))
|
||||
return { status: 'accepted' }
|
||||
}
|
||||
|
||||
@@ -284,24 +289,46 @@ export class SpeechStream {
|
||||
return
|
||||
|
||||
if (!this.response.signal.aborted)
|
||||
this.deliver()
|
||||
this.deliver(this.segmenter.end())
|
||||
this.sealed = true
|
||||
this.text?.close()
|
||||
this.intent?.end()
|
||||
}
|
||||
|
||||
/** Sends the speech text and the controls in source order. Controls that conversion lost are sent last, in order. */
|
||||
private deliver() {
|
||||
const parts = toSpeechText(this.markdown, { marker: controlMarker }).split(controlMarker)
|
||||
parts.forEach((text, index) => {
|
||||
if (text)
|
||||
this.deliverText(text)
|
||||
/**
|
||||
* Sends each segment as speech text, with its controls in source order. Controls that conversion lost are sent after
|
||||
* the text of their segment, in order.
|
||||
*/
|
||||
private deliver(segments: readonly string[]) {
|
||||
if (this.response.signal.aborted)
|
||||
return
|
||||
|
||||
const control = this.controls[index]
|
||||
if (control && index < parts.length - 1)
|
||||
this.deliverControl(control)
|
||||
})
|
||||
this.controls.slice(parts.length - 1).forEach(control => this.deliverControl(control))
|
||||
for (const segment of segments) {
|
||||
const count = segment.split(controlMarker).length - 1
|
||||
const controls = this.controls.splice(0, count)
|
||||
const parts = toSpeechText(segment, { marker: controlMarker, language: this.output.language, before: this.spoken }).split(controlMarker)
|
||||
// Segments end inside a paragraph or between blocks. The space or the line break between them stays for TTS.
|
||||
const separator = /\n\s*$/.test(segment) ? '\n' : /[ \t]$/.test(segment) ? ' ' : ''
|
||||
parts.forEach((text, index) => {
|
||||
const last = index === parts.length - 1
|
||||
const spoken = last && text ? text + separator : text
|
||||
if (spoken)
|
||||
this.deliverText(spoken)
|
||||
if (text.trim())
|
||||
this.spoken = text
|
||||
|
||||
const control = controls[index]
|
||||
if (control && !last)
|
||||
this.deliverControl(control)
|
||||
})
|
||||
controls.slice(parts.length - 1).forEach(control => this.deliverControl(control))
|
||||
// NOTICE:
|
||||
// The TTS chunker reads one grapheme past a sentence end, to tell `3.14` and `...` from an end. Without more text,
|
||||
// the last sentence of a segment waits for the next segment, which can be a long formula or code block.
|
||||
// Source: chunkTtsInput in packages/pipelines-audio/src/processors/tts-chunker.ts.
|
||||
// Remove when the chunker ends a chunk at the end of its input without lookahead.
|
||||
this.intent?.writeFlush()
|
||||
}
|
||||
}
|
||||
|
||||
private deliverText(text: string) {
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { toSpeechText } from './speech-text'
|
||||
|
||||
describe('toSpeechText', () => {
|
||||
it('removes markup that has no sound', () => {
|
||||
expect(toSpeechText('This is **important** and *soft*.')).toBe('This is important and soft.')
|
||||
expect(toSpeechText('这是**重要**的事情。')).toBe('这是重要的事情。')
|
||||
expect(toSpeechText('Read [the guide](https://example.com/guide) first.')).toBe('Read the guide first.')
|
||||
expect(toSpeechText('Run `pnpm test` now.')).toBe('Run pnpm test now.')
|
||||
expect(toSpeechText('A  sleeps.')).toBe('A cat sleeps.')
|
||||
})
|
||||
|
||||
it('puts a line break between blocks so that each block ends a TTS chunk', () => {
|
||||
expect(toSpeechText('# Plan\n\n- First step\n- Second step\n\n> Done')).toBe('Plan\nFirst step\nSecond step\nDone')
|
||||
})
|
||||
|
||||
it('skips code blocks, HTML, and thematic breaks', () => {
|
||||
expect(toSpeechText('Before.\n\n```ts\nconst a = 1\n```\n\n---\n\n<div>hidden</div>\n\nAfter.')).toBe('Before.\nAfter.')
|
||||
})
|
||||
|
||||
it('keeps symbols that the writer meant as operators', () => {
|
||||
expect(toSpeechText('2 * 3 = 6')).toBe('2 * 3 = 6')
|
||||
expect(toSpeechText('2*3 = 6')).toBe('2*3 = 6')
|
||||
expect(toSpeechText('2**10 = 1024')).toBe('2**10 = 1024')
|
||||
expect(toSpeechText('x ~= 3.14, ~3 hours')).toBe('x ~= 3.14, ~3 hours')
|
||||
})
|
||||
|
||||
it('keeps delimiters that CommonMark reads as emphasis between ASCII operands', () => {
|
||||
expect(toSpeechText('x**2 + y**2 = r**2')).toBe('x**2 + y**2 = r**2')
|
||||
expect(toSpeechText('a*b*c')).toBe('a*b*c')
|
||||
})
|
||||
|
||||
it('keeps each marker, also inside unspoken nodes, in source order', () => {
|
||||
const marker = ''
|
||||
|
||||
expect(toSpeechText(`**${marker}Hi** there.`, { marker })).toBe(`${marker}Hi there.`)
|
||||
expect(toSpeechText(`See [docs](https://e.com/${marker}).`, { marker })).toBe(`See docs${marker}.`)
|
||||
expect(toSpeechText(`One.\n\n\`\`\`\n${marker}\n\`\`\`\n\nTwo.${marker}`, { marker })).toBe(`One.\n${marker}\nTwo.${marker}`)
|
||||
})
|
||||
})
|
||||
@@ -1,113 +0,0 @@
|
||||
import type { Nodes, PhrasingContent, Root, RootContent } from 'mdast'
|
||||
import type { Plugin } from 'unified'
|
||||
|
||||
import remarkParse from 'remark-parse'
|
||||
|
||||
import { unified } from 'unified'
|
||||
|
||||
export interface SpeechTextOptions {
|
||||
/**
|
||||
* One character that the output keeps as often as the source has it, also inside unspoken nodes.
|
||||
* The speech stream puts it where special tokens and flushes occur, then splits the output on it.
|
||||
*/
|
||||
marker?: string
|
||||
}
|
||||
|
||||
/** A delimiter between two ASCII operands is an operator, for example `x**2` or `a*b*c`, not emphasis. */
|
||||
const asciiOperand = /^[A-Z0-9]$/i
|
||||
|
||||
/**
|
||||
* Compiles markdown to the text that a TTS provider reads.
|
||||
*
|
||||
* Markup that has no sound is removed: emphasis delimiters, heading and list markers, link URLs, code
|
||||
* blocks, and HTML. Blocks are joined with a line break, so the TTS chunker pauses between them.
|
||||
* Symbols that the parser keeps as text, for example `2 * 3`, `2**10`, or `x ~= 3`, stay unchanged.
|
||||
*/
|
||||
export const remarkSpeech: Plugin<[SpeechTextOptions?], Root, string> = function (options) {
|
||||
this.compiler = (tree, file) => compile(tree as Root, String(file), options?.marker)
|
||||
}
|
||||
|
||||
export function toSpeechText(markdown: string, options?: SpeechTextOptions): string {
|
||||
return String(unified().use(remarkParse).use(remarkSpeech, options).processSync(markdown))
|
||||
}
|
||||
|
||||
function compile(tree: Root, source: string, marker: string | undefined): string {
|
||||
// TODO: Read literal symbols as words, for example `*` as "times" and `~=` as "approximately equals".
|
||||
// This needs a spoken lexicon for each language. Its place, packages/i18n or a separate table, is not decided.
|
||||
// TODO: Select the spoken language for each span. Replies often switch languages in one sentence,
|
||||
// so the voice language alone cannot select the words.
|
||||
|
||||
function count(text: string): number {
|
||||
return marker ? text.split(marker).length - 1 : 0
|
||||
}
|
||||
|
||||
function range(node: Nodes): string {
|
||||
const start = node.position?.start.offset
|
||||
const end = node.position?.end.offset
|
||||
return start === undefined || end === undefined ? '' : source.slice(start, end)
|
||||
}
|
||||
|
||||
/** Adds the markers that the source of a node has but its spoken text lost, for example in a link URL. */
|
||||
function keepMarkers(node: Nodes, spoken: string): string {
|
||||
const missing = count(range(node)) - count(spoken)
|
||||
return marker && missing > 0 ? spoken + marker.repeat(missing) : spoken
|
||||
}
|
||||
|
||||
function isOperator(node: Nodes): boolean {
|
||||
const start = node.position?.start.offset
|
||||
const end = node.position?.end.offset
|
||||
if (start === undefined || end === undefined)
|
||||
return false
|
||||
|
||||
return asciiOperand.test(source[start - 1] ?? '') && asciiOperand.test(source[end] ?? '')
|
||||
}
|
||||
|
||||
function blocks(nodes: readonly RootContent[]): string {
|
||||
return nodes.map(block).filter(Boolean).join('\n')
|
||||
}
|
||||
|
||||
function block(node: RootContent): string {
|
||||
switch (node.type) {
|
||||
case 'paragraph':
|
||||
case 'heading':
|
||||
return phrasing(node.children)
|
||||
case 'blockquote':
|
||||
case 'list':
|
||||
case 'listItem':
|
||||
return blocks(node.children)
|
||||
// TODO: Tell the listener that the reply has code, instead of a skip without a sound.
|
||||
default:
|
||||
return keepMarkers(node, '')
|
||||
}
|
||||
}
|
||||
|
||||
function phrasing(nodes: readonly PhrasingContent[]): string {
|
||||
return nodes.map(inline).join('')
|
||||
}
|
||||
|
||||
function inline(node: PhrasingContent): string {
|
||||
switch (node.type) {
|
||||
case 'text':
|
||||
case 'inlineCode':
|
||||
return node.value
|
||||
case 'emphasis':
|
||||
case 'strong':
|
||||
case 'delete':
|
||||
// CommonMark reads `x**2 + y**2` as strong text, but the writer meant two powers.
|
||||
// The source text, delimiters included, keeps the operators.
|
||||
return isOperator(node) ? range(node) : phrasing(node.children)
|
||||
case 'link':
|
||||
case 'linkReference':
|
||||
return keepMarkers(node, phrasing(node.children))
|
||||
case 'image':
|
||||
case 'imageReference':
|
||||
return keepMarkers(node, node.alt ?? '')
|
||||
case 'break':
|
||||
return '\n'
|
||||
default:
|
||||
return keepMarkers(node, '')
|
||||
}
|
||||
}
|
||||
|
||||
return blocks(tree.children)
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { toSpeechText } from '.'
|
||||
|
||||
describe('toSpeechText', () => {
|
||||
it('removes markup that has no sound', () => {
|
||||
expect(toSpeechText('This is **important** and *soft*.')).toBe('This is important and soft.')
|
||||
expect(toSpeechText('这是**重要**的事情。')).toBe('这是重要的事情。')
|
||||
expect(toSpeechText('Read [the guide](https://example.com/guide) first.')).toBe('Read the guide first.')
|
||||
expect(toSpeechText('Run `pnpm test` now.')).toBe('Run pnpm test now.')
|
||||
expect(toSpeechText('A  sleeps.')).toBe('A cat sleeps.')
|
||||
})
|
||||
|
||||
it('puts a line break between blocks so that each block ends a TTS chunk', () => {
|
||||
expect(toSpeechText('# Plan\n\n- First step\n- Second step\n\n> Done')).toBe('Plan\nFirst step\nSecond step\nDone')
|
||||
})
|
||||
|
||||
it('skips code blocks, HTML, and thematic breaks', () => {
|
||||
expect(toSpeechText('Before.\n\n```ts\nconst a = 1\n```\n\n---\n\n<div>hidden</div>\n\nAfter.')).toBe('Before.\nAfter.')
|
||||
})
|
||||
|
||||
it('keeps symbols that the writer meant as operators', () => {
|
||||
expect(toSpeechText('2 * 3 = 6')).toBe('2 * 3 = 6')
|
||||
expect(toSpeechText('2*3 = 6')).toBe('2*3 = 6')
|
||||
expect(toSpeechText('2**10 = 1024')).toBe('2**10 = 1024')
|
||||
expect(toSpeechText('x ~= 3.14, ~3 hours')).toBe('x ~= 3.14, ~3 hours')
|
||||
})
|
||||
|
||||
it('keeps delimiters that CommonMark reads as emphasis between ASCII operands', () => {
|
||||
expect(toSpeechText('x**2 + y**2 = r**2')).toBe('x**2 + y**2 = r**2')
|
||||
expect(toSpeechText('a*b*c')).toBe('a*b*c')
|
||||
})
|
||||
|
||||
it('keeps each marker, also inside unspoken nodes, in source order', () => {
|
||||
const marker = ''
|
||||
|
||||
expect(toSpeechText(`**${marker}Hi** there.`, { marker })).toBe(`${marker}Hi there.`)
|
||||
expect(toSpeechText(`See [docs](https://e.com/${marker}).`, { marker })).toBe(`See docs${marker}.`)
|
||||
expect(toSpeechText(`One.\n\n\`\`\`\n${marker}\n\`\`\`\n\nTwo.${marker}`, { marker })).toBe(`One.\n${marker}\nTwo.${marker}`)
|
||||
})
|
||||
|
||||
// The cases of the T-151 baseline recording, in Chinese, English, Japanese, and Korean sentences.
|
||||
// Before this change each formula reached the TTS provider as TeX source, with `$` and backslashes.
|
||||
it.each([
|
||||
['a-inline', 'zh', '巴塞尔问题的结论是 $\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}$，这个结果由欧拉证明。', '巴塞尔问题的结论是 当n趋于无穷大时 i从1到n对i的平方分之1求和的极限 等于 分子是派的平方 分母是6的分数，这个结果由欧拉证明。'],
|
||||
['a-inline', 'en', 'The Basel problem states that $\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}$, which Euler proved.', 'The Basel problem states that the limit as n approaches infinity of the sum from i equals one to n of the fraction with numerator one and denominator i squared equals pi squared over six, which Euler proved.'],
|
||||
['a-inline', 'ja', 'バーゼル問題の結論は $\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}$ で、オイラーが証明しました。', 'バーゼル問題の結論は エヌを無限大に近づけたときのアイがいちからエヌまでのアイの二乗分のいちの和の極限 イコール 分子がパイの二乗、分母がろくの分数 で、オイラーが証明しました。'],
|
||||
['a-inline', 'ko', '바젤 문제의 결론은 $\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}$ 이며, 오일러가 증명했습니다.', '바젤 문제의 결론은 엔이 무한대로 갈 때 아이가 일부터 엔까지 아이 제곱 분의 일의 합의 극한은 분자가 파이 제곱, 분모가 육인 분수 이며, 오일러가 증명했습니다.'],
|
||||
['a-block', 'zh', '巴塞尔问题的结论是：\n\n$$\n\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}\n$$\n\n这个结果由欧拉证明。', '巴塞尔问题的结论是：\n当n趋于无穷大时 i从1到n对i的平方分之1求和的极限 等于 分子是派的平方 分母是6的分数\n这个结果由欧拉证明。'],
|
||||
['a-block', 'en', 'The Basel problem states that:\n\n$$\n\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}\n$$\n\nEuler proved this result.', 'The Basel problem states that:\nthe limit as n approaches infinity of the sum from i equals one to n of the fraction with numerator one and denominator i squared equals pi squared over six\nEuler proved this result.'],
|
||||
['a-block', 'ja', 'バーゼル問題の結論は次のとおりです。\n\n$$\n\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}\n$$\n\nオイラーがこれを証明しました。', 'バーゼル問題の結論は次のとおりです。\nエヌを無限大に近づけたときのアイがいちからエヌまでのアイの二乗分のいちの和の極限 イコール 分子がパイの二乗、分母がろくの分数\nオイラーがこれを証明しました。'],
|
||||
['a-block', 'ko', '바젤 문제의 결론은 다음과 같습니다.\n\n$$\n\\lim_{n \\to \\infty}\\sum_{i=1}^{n}\\frac{1}{i^2}=\\frac{\\pi^2}{6}\n$$\n\n오일러가 이를 증명했습니다.', '바젤 문제의 결론은 다음과 같습니다.\n엔이 무한대로 갈 때 아이가 일부터 엔까지 아이 제곱 분의 일의 합의 극한은 분자가 파이 제곱, 분모가 육인 분수\n오일러가 이를 증명했습니다.'],
|
||||
['b1-water', 'zh', '氢气在氧气中燃烧的化学方程式是 $\\ce{2H2 + O2 -> 2H2O}$。', '氢气在氧气中燃烧的化学方程式是 氢气 和 氧气 反应生成 水。'],
|
||||
['b1-water', 'en', 'When hydrogen burns in oxygen, the equation is $\\ce{2H2 + O2 -> 2H2O}$.', 'When hydrogen burns in oxygen, the equation is two H two plus O two yields two H two O.'],
|
||||
['b1-water', 'ja', '水素が酸素中で燃える反応式は $\\ce{2H2 + O2 -> 2H2O}$ です。', '水素が酸素中で燃える反応式は 水素と酸素が反応して 水ができる です。'],
|
||||
['b1-water', 'ko', '수소가 산소 속에서 타는 반응식은 $\\ce{2H2 + O2 -> 2H2O}$ 입니다.', '수소가 산소 속에서 타는 반응식은 수소와 산소가 반응하여 물이 생성된다 입니다.'],
|
||||
['b2-h2so4', 'zh', '硫酸的化学式是 $H_2SO_4$，它是一种强酸。', '硫酸的化学式是 H2 S O4，它是一种强酸。'],
|
||||
['b2-h2so4', 'en', 'The chemical formula of sulfuric acid is $H_2SO_4$, a strong acid.', 'The chemical formula of sulfuric acid is H two S O four, a strong acid.'],
|
||||
['b2-h2so4', 'ja', '硫酸の化学式は $H_2SO_4$ で、強い酸です。', '硫酸の化学式は エイチにエスオーよん で、強い酸です。'],
|
||||
['b2-h2so4', 'ko', '황산의 화학식은 $H_2SO_4$ 이며, 강산입니다.', '황산의 화학식은 에이치 아래첨자 이 에스 오 아래첨자 사 이며, 강산입니다.'],
|
||||
['b3-fe3', 'zh', '三价铁离子写作 $\\ce{Fe^{3+}}$，它的溶液呈黄色。', '三价铁离子写作 Fe 3价正离子，它的溶液呈黄色。'],
|
||||
['b3-fe3', 'en', 'The iron three ion is written as $\\ce{Fe^{3+}}$ in chemistry.', 'The iron three ion is written as F E three plus in chemistry.'],
|
||||
['b3-fe3', 'ja', '鉄(III)イオンは $\\ce{Fe^{3+}}$ と書き、その水溶液は黄色です。', '鉄(III)イオンは エフ イー さん価の陽イオン と書き、その水溶液は黄色です。'],
|
||||
['b3-fe3', 'ko', '철 3가 이온은 $\\ce{Fe^{3+}}$ 로 쓰며, 그 용액은 노란색입니다.', '철 3가 이온은 에프 E 삼가 양이온 로 쓰며, 그 용액은 노란색입니다.'],
|
||||
['b4-nh3', 'zh', '合成氨是一个可逆反应：$\\ce{N2 + 3H2 <=> 2NH3}$。', '合成氨是一个可逆反应：氮气 和 氢气 可逆反应生成 氨气。'],
|
||||
['b4-nh3', 'en', 'The Haber process is a reversible reaction: $\\ce{N2 + 3H2 <=> 2NH3}$.', 'The Haber process is a reversible reaction: N two plus three H two is in equilibrium with two N H three.'],
|
||||
['b4-nh3', 'ja', 'アンモニアの合成は可逆反応です：$\\ce{N2 + 3H2 <=> 2NH3}$。', 'アンモニアの合成は可逆反応です：窒素と水素が可逆的に反応して アンモニアができる。'],
|
||||
['b4-nh3', 'ko', '암모니아 합성은 가역 반응입니다: $\\ce{N2 + 3H2 <=> 2NH3}$.', '암모니아 합성은 가역 반응입니다: 질소와 수소가 가역적으로 반응하여 암모니아가 생성된다.'],
|
||||
['c1-perp', 'zh', '已知 $AB \\perp CD$，垂足为 O。', '已知 A B 垂直于 C D，垂足为 O。'],
|
||||
['c1-perp', 'en', 'Given that $AB \\perp CD$, the foot of the perpendicular is O.', 'Given that A B is perpendicular to C D, the foot of the perpendicular is O.'],
|
||||
['c1-perp', 'ja', '$AB \\perp CD$ のとき、垂線の足を O とします。', 'エービーとシーディーは垂直 のとき、垂線の足を O とします。'],
|
||||
['c1-perp', 'ko', '$AB \\perp CD$ 일 때, 수선의 발은 O 입니다.', '에이 비는 씨 디에 수직 일 때, 수선의 발은 O 입니다.'],
|
||||
['c2-parallel', 'zh', '如果 $l_1 \\parallel l_2$，那么同位角相等。', '如果 l1 平行于 l2，那么同位角相等。'],
|
||||
['c2-parallel', 'en', 'If $l_1 \\parallel l_2$, then the corresponding angles are equal.', 'If l one is parallel to l two, then the corresponding angles are equal.'],
|
||||
['c2-parallel', 'ja', '$l_1 \\parallel l_2$ ならば、同位角は等しい。', 'エルいちとエルには平行 ならば、同位角は等しい。'],
|
||||
['c2-parallel', 'ko', '$l_1 \\parallel l_2$ 이면 동위각은 같습니다.', '엘 아래첨자 일은 엘 아래첨자 이와 평행 이면 동위각은 같습니다.'],
|
||||
['c3-leq', 'zh', '题目给出的条件是 $a \\leq b$。', '题目给出的条件是 a 小于等于 b。'],
|
||||
['c3-leq', 'en', 'The problem gives the condition $a \\leq b$.', 'The problem gives the condition a is less than or equal to b.'],
|
||||
['c3-leq', 'ja', '問題の条件は $a \\leq b$ です。', '問題の条件は エー 小なりイコール ビー です。'],
|
||||
['c3-leq', 'ko', '문제의 조건은 $a \\leq b$ 입니다.', '문제의 조건은 에이는 비보다 작거나 같음 입니다.'],
|
||||
['c4-in', 'zh', '设元素 $x \\in A$，则 x 满足该条件。', '设元素 x 属于 A，则 x 满足该条件。'],
|
||||
['c4-in', 'en', 'Let the element $x \\in A$, so x satisfies the condition.', 'Let the element x is an element of A, so x satisfies the condition.'],
|
||||
['c4-in', 'ja', '要素 $x \\in A$ とすると、x はこの条件を満たします。', '要素 エックスはエーに属する とすると、x はこの条件を満たします。'],
|
||||
['c4-in', 'ko', '원소 $x \\in A$ 라고 하면, x는 이 조건을 만족합니다.', '원소 엑스는 에이의 원소 라고 하면, x는 이 조건을 만족합니다.'],
|
||||
['c5-subseteq', 'zh', '这两个集合满足 $A \\subseteq B$。', '这两个集合满足 A 包含于 B。'],
|
||||
['c5-subseteq', 'en', 'These two sets satisfy $A \\subseteq B$.', 'These two sets satisfy A is a subset of or equal to B.'],
|
||||
['c5-subseteq', 'ja', 'この二つの集合は $A \\subseteq B$ を満たします。', 'この二つの集合は エーはビーに含まれる を満たします。'],
|
||||
['c5-subseteq', 'ko', '두 집합은 $A \\subseteq B$ 를 만족합니다.', '두 집합은 에이는 비의 부분집합 를 만족합니다.'],
|
||||
['c6-angle', 'zh', '在这个直角三角形中，$\\angle ABC = 90^\\circ$。', '在这个直角三角形中，角A B C 等于 90度。'],
|
||||
['c6-angle', 'en', 'In this right triangle, $\\angle ABC = 90^\\circ$.', 'In this right triangle, angle A B C equals ninety degrees.'],
|
||||
['c6-angle', 'ja', 'この直角三角形では、$\\angle ABC = 90^\\circ$ です。', 'この直角三角形では、角エービーシー イコール きゅうじゅう度 です。'],
|
||||
['c6-angle', 'ko', '이 직각삼각형에서 $\\angle ABC = 90^\\circ$ 입니다.', '이 직각삼각형에서 각 에이 비 씨는 구십도 입니다.'],
|
||||
['c7-similar', 'zh', '由此可得 $\\triangle ABC \\sim \\triangle DEF$。', '由此可得 三角形A B C 相似于 三角形D E F。'],
|
||||
['c7-similar', 'en', 'It follows that $\\triangle ABC \\sim \\triangle DEF$.', 'It follows that triangle A B C is similar to triangle D E F.'],
|
||||
['c7-similar', 'ja', 'したがって、$\\triangle ABC \\sim \\triangle DEF$ です。', 'したがって、三角形エービーシーと三角形ディーイーエフは相似 です。'],
|
||||
['c7-similar', 'ko', '따라서 $\\triangle ABC \\sim \\triangle DEF$ 입니다.', '따라서 삼각형 에이 비 씨와 삼각형 디 E 에프는 닮음 입니다.'],
|
||||
])('reads the baseline formula %s in %s', (_, __, markdown, speech) => {
|
||||
expect(toSpeechText(markdown)).toBe(speech)
|
||||
})
|
||||
|
||||
it('reads a formula in the language of its sentence, and in the voice language without words around it', () => {
|
||||
expect(toSpeechText('$x \\in A$')).toBe('x is an element of A')
|
||||
expect(toSpeechText('$x \\in A$', { language: 'ja-JP' })).toBe('エックスはエーに属する')
|
||||
expect(toSpeechText('$x \\in A$', { language: 'ko-KR' })).toBe('엑스는 에이의 원소')
|
||||
expect(toSpeechText('$x \\in A$', { language: 'zh-CN' })).toBe('x 属于 A')
|
||||
// Thai has no lexicon yet. English is used.
|
||||
expect(toSpeechText('$x \\in A$', { language: 'th-TH' })).toBe('x is an element of A')
|
||||
// Han text without kana is Japanese only when the voice is Japanese.
|
||||
expect(toSpeechText('元素 $x \\in A$。', { language: 'ja-JP' })).toBe('元素 エックスはエーに属する。')
|
||||
expect(toSpeechText('Then 设 $x \\in A$。')).toBe('Then 设 x 属于 A。')
|
||||
expect(toSpeechText('First, $a \\leq b$. 然后 $x \\in A$。')).toBe('First, a is less than or equal to b. 然后 x 属于 A。')
|
||||
})
|
||||
|
||||
it('reads the other TeX delimiters and TeX without delimiters', () => {
|
||||
expect(toSpeechText('Inline \\(a_1 \\times b_1\\) here.')).toBe('Inline a one times b one here.')
|
||||
expect(toSpeechText('结果：\\[\\frac{1}{2}\\]')).toBe('结果：2分之1')
|
||||
expect(toSpeechText('集合 $A=\\{x \\mid x \\in \\mathbb{R}\\}$。')).toBe('集合 A 等于 满足x 属于 实数集的x的集合。')
|
||||
expect(toSpeechText('设 \\alpha 为锐角，且 a \\leq b。')).toBe('设 阿尔法 为锐角，且 a 小于等于 b。')
|
||||
expect(toSpeechText('Water is \\ce{H2O}, and E = mc^2 holds.')).toBe('Water is H two O, and E equals m c squared holds.')
|
||||
expect(toSpeechText('Use `\\frac{1}{2}` here.')).toBe('Use one over two here.')
|
||||
})
|
||||
|
||||
it('reads a script without braces in TeX without delimiters as the whole number', () => {
|
||||
// TeX applies `^` to one character, but a reply that writes `2^10` without `$` means 2 to the 10th power.
|
||||
expect(toSpeechText('2 的 10 次方也写成 2^10。')).toBe('2 的 10 次方也写成 2的10次方。')
|
||||
expect(toSpeechText('也写成 $2^10$。')).toBe('也写成 2的1次方0。')
|
||||
})
|
||||
|
||||
it('reads prices as words, and keeps paths and code that only look like TeX', () => {
|
||||
expect(toSpeechText('It costs $5 and $10 today.')).toBe('It costs five dollars and ten dollars today.')
|
||||
expect(toSpeechText('价格$5，另一个$10。')).toBe('价格5美元，另一个10美元。')
|
||||
expect(toSpeechText('A ticket is $5, and the area is $x^2$.')).toBe('A ticket is five dollars, and the area is x squared.')
|
||||
expect(toSpeechText('チケットは $5、面積は $x^2$ です。')).toBe('チケットは ごドル、面積は エックスの二乗 です。')
|
||||
expect(toSpeechText('Open C:\\Users\\neko now.')).toBe('Open C:\\Users\\neko now.')
|
||||
expect(toSpeechText('Before.\n\n```tex\n\\(x\\)\n```\n\nAfter.')).toBe('Before.\nAfter.')
|
||||
})
|
||||
|
||||
it('keeps a marker inside a formula', () => {
|
||||
const marker = '\uE000'
|
||||
|
||||
expect(toSpeechText(`$a ${marker}\\leq b$ ok.`, { marker })).toBe(`a is less than or equal to b${marker} ok.`)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,223 @@
|
||||
import type { Nodes, PhrasingContent, Root, RootContent } from 'mdast'
|
||||
import type { Plugin } from 'unified'
|
||||
|
||||
import type { SpeechPreprocessor } from './preprocessor'
|
||||
|
||||
import remarkMath from 'remark-math'
|
||||
import remarkParse from 'remark-parse'
|
||||
|
||||
import { unified } from 'unified'
|
||||
|
||||
import { detectSpeechLanguage, toSpeechLanguage } from './language'
|
||||
import { normalizeMathDelimiters } from './markdown'
|
||||
import { mathSpeechPreprocessor, moneySpeechPreprocessor } from './preprocessor'
|
||||
|
||||
export interface SpeechTextOptions {
|
||||
/**
|
||||
* One character that the output keeps as often as the source has it, also inside unspoken nodes.
|
||||
* The speech stream puts it where special tokens and flushes occur, then splits the output on it.
|
||||
*/
|
||||
marker?: string
|
||||
/**
|
||||
* BCP 47 code of the voice or the character, for example `zh-CN`.
|
||||
* The words around a formula or a price select its language first. This code is used when they do not.
|
||||
*/
|
||||
language?: string
|
||||
/**
|
||||
* The spoken text before this markdown, when the markdown is one segment of a streamed reply, see ./segmenter.
|
||||
* A formula without words around it, as a display formula that starts the segment, reads in its language.
|
||||
*/
|
||||
before?: string
|
||||
/**
|
||||
* The preprocessor plugins, in the order that their hooks run.
|
||||
* The default is the money plugin, then the math plugin. A list replaces the default, so it includes them to keep them.
|
||||
*/
|
||||
preprocessors?: readonly SpeechPreprocessor[]
|
||||
}
|
||||
|
||||
/** The default preprocessor plugins: money first, so a price reads before the math plugin looks for bare TeX. */
|
||||
export function defaultSpeechPreprocessors(): SpeechPreprocessor[] {
|
||||
return [moneySpeechPreprocessor(), mathSpeechPreprocessor()]
|
||||
}
|
||||
|
||||
/** A delimiter between two ASCII operands is an operator, for example `x**2` or `a*b*c`, not emphasis. */
|
||||
const asciiOperand = /^[A-Z0-9]$/i
|
||||
/** A sentence ends at these characters. The words of the sentence select the language of its formulas. */
|
||||
const sentenceEnd = /[.!?。！？\n]/
|
||||
|
||||
/**
|
||||
* Compiles markdown to the text that a TTS provider reads.
|
||||
*
|
||||
* Markup that has no sound is removed: emphasis delimiters, heading and list markers, link URLs, code
|
||||
* blocks, and HTML. Blocks are joined with a line break, so the TTS chunker pauses between them.
|
||||
* Symbols that the parser keeps as text, for example `2 * 3`, `2**10`, or `x ~= 3`, stay unchanged.
|
||||
*
|
||||
* TeX formulas in `$...$`, `$$...$$`, `\(...\)`, `\[...\]`, and known commands without delimiters, such as
|
||||
* `\ce{H2O}`, are read as words. The words of the sentence around a formula select its language, and
|
||||
* `options.language` is used when they do not.
|
||||
*/
|
||||
export const remarkSpeech: Plugin<[SpeechTextOptions?], Root, string> = function (options) {
|
||||
this.compiler = (tree, file) => compile(tree as Root, String(file), options)
|
||||
}
|
||||
|
||||
/**
|
||||
* Compiles markdown to the text that a TTS provider reads, with the preprocessor plugins of `options`.
|
||||
*
|
||||
* `\(...\)` and `\[...\]` become `$` delimiters first, because remark-math reads only `$`. Then each `prepare` hook
|
||||
* rewrites the markdown, in plugin order.
|
||||
*/
|
||||
export function toSpeechText(markdown: string, options?: SpeechTextOptions): string {
|
||||
const preprocessors = options?.preprocessors ?? defaultSpeechPreprocessors()
|
||||
const prepared = preprocessors.reduce((text, preprocessor) => preprocessor.prepare?.(text) ?? text, normalizeMathDelimiters(markdown))
|
||||
return String(unified().use(remarkParse).use(remarkMath).use(remarkSpeech, { ...options, preprocessors }).processSync(prepared))
|
||||
}
|
||||
|
||||
function compile(tree: Root, source: string, options: SpeechTextOptions | undefined): string {
|
||||
// TODO: Read literal symbols outside formulas as words, for example `*` as "times" and `~=` as "approximately equals".
|
||||
// A prose preprocessor plugin can read them with the words of the locale packs of `mathSpeech`, see ./preprocessor/math.
|
||||
const marker = options?.marker
|
||||
const fallback = toSpeechLanguage(options?.language)
|
||||
const preprocessors = options?.preprocessors ?? defaultSpeechPreprocessors()
|
||||
|
||||
function count(text: string): number {
|
||||
return marker ? text.split(marker).length - 1 : 0
|
||||
}
|
||||
|
||||
function range(node: Nodes): string {
|
||||
const start = node.position?.start.offset
|
||||
const end = node.position?.end.offset
|
||||
return start === undefined || end === undefined ? '' : source.slice(start, end)
|
||||
}
|
||||
|
||||
/** Adds the markers that the source of a node has but its spoken text lost, for example in a link URL. */
|
||||
function keepMarkers(node: Nodes, spoken: string): string {
|
||||
const missing = count(range(node)) - count(spoken)
|
||||
return marker && missing > 0 ? spoken + marker.repeat(missing) : spoken
|
||||
}
|
||||
|
||||
function isOperator(node: Nodes): boolean {
|
||||
const start = node.position?.start.offset
|
||||
const end = node.position?.end.offset
|
||||
if (start === undefined || end === undefined)
|
||||
return false
|
||||
|
||||
return asciiOperand.test(source[start - 1] ?? '') && asciiOperand.test(source[end] ?? '')
|
||||
}
|
||||
|
||||
/** The text of a node without its formulas. */
|
||||
function plain(node: Nodes): string {
|
||||
if (node.type === 'text')
|
||||
return node.value
|
||||
return 'children' in node ? node.children.map(child => plain(child as Nodes)).join('') : ''
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a formula with the first formula hook that returns words, or keeps the source when none does.
|
||||
* Its markers move to its end, because words have no place for them.
|
||||
*/
|
||||
function speakFormula(source: string, context: string): string {
|
||||
const tex = marker ? source.replaceAll(marker, '') : source
|
||||
const language = detectSpeechLanguage(context, fallback)
|
||||
let spoken = tex
|
||||
for (const preprocessor of preprocessors) {
|
||||
const words = preprocessor.formula?.(tex, { language })
|
||||
if (words !== undefined) {
|
||||
spoken = words
|
||||
break
|
||||
}
|
||||
}
|
||||
return spoken + (marker ?? '').repeat(count(source))
|
||||
}
|
||||
|
||||
/** Runs the prose hooks over a run of text. `context` is the words of its sentence. */
|
||||
function speakProse(text: string, kind: 'text' | 'code', context: string): string {
|
||||
const language = detectSpeechLanguage(context, fallback)
|
||||
return preprocessors.reduce((rest, preprocessor) => preprocessor.prose?.(rest, { language, fallback, kind, readFormula: speakFormula }) ?? rest, text)
|
||||
}
|
||||
|
||||
/**
|
||||
* Inline math follows the Pandoc rule, so `$5 and $10` stays text:
|
||||
* the content does not start or end with a space, and a digit does not follow the closing `$`.
|
||||
*/
|
||||
function isFormula(node: Extract<PhrasingContent, { type: 'inlineMath' }>): boolean {
|
||||
if (range(node).startsWith('$$'))
|
||||
return true
|
||||
const after = sourceAfter(node)
|
||||
return !/^\s|\s$/.test(node.value) && !/^\d/.test(after)
|
||||
}
|
||||
|
||||
function sourceAfter(node: Nodes): string {
|
||||
const end = node.position?.end.offset
|
||||
return end === undefined ? '' : source.slice(end, end + 1)
|
||||
}
|
||||
|
||||
/** The words of the sentence that holds a formula, in the paragraph `nodes`. Other sentences are the fallback. */
|
||||
function sentenceAround(nodes: readonly PhrasingContent[], index: number): string {
|
||||
const before = nodes.slice(0, index).map(plain).join('')
|
||||
const after = nodes.slice(index + 1).map(plain).join('')
|
||||
const sentence = before.split(sentenceEnd).at(-1)! + after.split(sentenceEnd)[0]!
|
||||
if (/\p{L}{2,}/u.test(sentence))
|
||||
return sentence
|
||||
return /\p{L}{2,}/u.test(before + after) ? before + after : options?.before ?? ''
|
||||
}
|
||||
|
||||
function blocks(nodes: readonly RootContent[]): string {
|
||||
return nodes.map((node, index) => block(node, nodes, index)).filter(Boolean).join('\n')
|
||||
}
|
||||
|
||||
function block(node: RootContent, siblings: readonly RootContent[], index: number): string {
|
||||
switch (node.type) {
|
||||
case 'paragraph':
|
||||
case 'heading':
|
||||
return phrasing(node.children)
|
||||
case 'blockquote':
|
||||
case 'list':
|
||||
case 'listItem':
|
||||
return blocks(node.children)
|
||||
case 'math': {
|
||||
// A display formula reads in the language of the text before it, or after it when it opens the reply.
|
||||
// In a streamed segment, the text before it can be in an earlier segment.
|
||||
const previous = siblings[index - 1] ? plain(siblings[index - 1]!) : options?.before ?? ''
|
||||
const context = [previous, siblings[index + 1] ? plain(siblings[index + 1]!) : ''].join('\n')
|
||||
return keepMarkers(node, speakFormula(node.value, context))
|
||||
}
|
||||
// TODO: Tell the listener that the reply has code, instead of a skip without a sound.
|
||||
default:
|
||||
return keepMarkers(node, '')
|
||||
}
|
||||
}
|
||||
|
||||
function phrasing(nodes: readonly PhrasingContent[]): string {
|
||||
return nodes.map((node, index) => inline(node, nodes, index)).join('')
|
||||
}
|
||||
|
||||
function inline(node: PhrasingContent, siblings: readonly PhrasingContent[], index: number): string {
|
||||
switch (node.type) {
|
||||
case 'text':
|
||||
// A voice can read `$5` in another language, so a price is read as words of its sentence.
|
||||
return speakProse(node.value, 'text', sentenceAround(siblings, index) + node.value)
|
||||
case 'inlineCode':
|
||||
return speakProse(node.value, 'code', node.value)
|
||||
case 'inlineMath':
|
||||
return isFormula(node) ? keepMarkers(node, speakFormula(node.value, sentenceAround(siblings, index))) : keepMarkers(node, range(node))
|
||||
case 'emphasis':
|
||||
case 'strong':
|
||||
case 'delete':
|
||||
// CommonMark reads `x**2 + y**2` as strong text, but the writer meant two powers.
|
||||
// The source text, delimiters included, keeps the operators.
|
||||
return isOperator(node) ? range(node) : phrasing(node.children)
|
||||
case 'link':
|
||||
case 'linkReference':
|
||||
return keepMarkers(node, phrasing(node.children))
|
||||
case 'image':
|
||||
case 'imageReference':
|
||||
return keepMarkers(node, node.alt ?? '')
|
||||
case 'break':
|
||||
return '\n'
|
||||
default:
|
||||
return keepMarkers(node, '')
|
||||
}
|
||||
}
|
||||
|
||||
return blocks(tree.children)
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { detectSpeechLanguage, toSpeechLanguage } from './language'
|
||||
|
||||
describe('language selection', () => {
|
||||
it('maps BCP 47 codes to primary language subtags', () => {
|
||||
expect(toSpeechLanguage('zh-CN')).toBe('zh')
|
||||
expect(toSpeechLanguage('cmn-Hans-CN')).toBe('zh')
|
||||
expect(toSpeechLanguage('yue-HK')).toBe('zh')
|
||||
expect(toSpeechLanguage('en_US')).toBe('en')
|
||||
expect(toSpeechLanguage('ko')).toBe('ko')
|
||||
expect(toSpeechLanguage('th-TH')).toBe('th')
|
||||
expect(toSpeechLanguage(undefined)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('reads the script of the words around a formula', () => {
|
||||
expect(detectSpeechLanguage('バーゼル問題の結論は')).toBe('ja')
|
||||
expect(detectSpeechLanguage('바젤 문제의 결론은')).toBe('ko')
|
||||
expect(detectSpeechLanguage('巴塞尔问题的结论是')).toBe('zh')
|
||||
expect(detectSpeechLanguage('巴塞尔问题', 'ja')).toBe('ja')
|
||||
expect(detectSpeechLanguage('The Basel problem', 'zh')).toBe('en')
|
||||
expect(detectSpeechLanguage(' , ', 'ko')).toBe('ko')
|
||||
expect(detectSpeechLanguage('')).toBe('en')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* The primary language subtag of a BCP 47 code: `zh` for `zh-CN`, `ko` for `ko_KR`. Chinese dialect codes are `zh`,
|
||||
* because the readers have one Chinese pack.
|
||||
*/
|
||||
export function toSpeechLanguage(code: string | undefined): string | undefined {
|
||||
if (code === undefined)
|
||||
return undefined
|
||||
const primary = code.toLowerCase().split(/[-_]/)[0] ?? ''
|
||||
if (primary === 'cmn' || primary === 'yue')
|
||||
return 'zh'
|
||||
return primary || undefined
|
||||
}
|
||||
|
||||
const kana = /[\p{Script=Hiragana}\p{Script=Katakana}]/u
|
||||
const hangul = /\p{Script=Hangul}/u
|
||||
const han = /\p{Script=Han}/u
|
||||
const latinWord = /[a-z]{2,}/i
|
||||
|
||||
/**
|
||||
* Selects the language of a formula or a price from the words around it, as a primary language subtag.
|
||||
* Han characters without kana are Chinese, unless the voice is Japanese.
|
||||
*/
|
||||
export function detectSpeechLanguage(context: string, fallback?: string): string {
|
||||
if (kana.test(context))
|
||||
return 'ja'
|
||||
if (hangul.test(context))
|
||||
return 'ko'
|
||||
if (han.test(context))
|
||||
return fallback === 'ja' ? 'ja' : 'zh'
|
||||
if (latinWord.test(context))
|
||||
return 'en'
|
||||
return fallback ?? 'en'
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { normalizeMathDelimiters } from './markdown'
|
||||
|
||||
describe('normalizeMathDelimiters', () => {
|
||||
it('converts the parenthesis and bracket delimiters outside code', () => {
|
||||
expect(normalizeMathDelimiters(String.raw`a \(x\) b \[y\]`)).toBe('a $x$ b $$y$$')
|
||||
expect(normalizeMathDelimiters('`\\(x\\)` and \\(x\\)')).toBe('`\\(x\\)` and $x$')
|
||||
expect(normalizeMathDelimiters('```\n\\[x\\]\n```')).toBe('```\n\\[x\\]\n```')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,18 @@
|
||||
/** Code that markdown keeps as written: fenced blocks and inline code. Math delimiters inside it are not math. */
|
||||
export const codeSpans = /(```[\s\S]*?```|~~~[\s\S]*?~~~|`[^`\n]*`)/
|
||||
|
||||
|
||||
/**
|
||||
* Converts `\(...\)` to `$...$` and `\[...\]` to `$$...$$`, outside code.
|
||||
* CommonMark reads `\(` as an escaped parenthesis, so the backslash is gone after parsing.
|
||||
*/
|
||||
export function normalizeMathDelimiters(markdown: string): string {
|
||||
return markdown
|
||||
.split(codeSpans)
|
||||
.map((part, index) => index % 2 === 1
|
||||
? part
|
||||
: part
|
||||
.replace(/\\\[([\s\S]+?)\\\]/g, (_, tex: string) => `$$${tex}$$`)
|
||||
.replace(/\\\(([\s\S]+?)\\\)/g, (_, tex: string) => `$${tex}$`))
|
||||
.join('')
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import type { SpeechPreprocessor } from '.'
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { mathSpeechPreprocessor, moneySpeechPreprocessor } from '.'
|
||||
import { defaultSpeechPreprocessors, toSpeechText } from '..'
|
||||
|
||||
describe('speech preprocessor host', () => {
|
||||
it('reads formulas and prices with the default plugins', () => {
|
||||
expect(defaultSpeechPreprocessors().map(preprocessor => preprocessor.name)).toEqual(['money', 'math'])
|
||||
expect(toSpeechText('Apples cost $3, and $x^2$ is a square.')).toBe('Apples cost three dollars, and x squared is a square.')
|
||||
})
|
||||
|
||||
it('runs prose hooks in installation order, each on the output of the one before', () => {
|
||||
const calls: string[] = []
|
||||
const tag = (name: string): SpeechPreprocessor => ({
|
||||
name,
|
||||
prose: (text) => {
|
||||
calls.push(name)
|
||||
return `${text}[${name}]`
|
||||
},
|
||||
})
|
||||
|
||||
expect(toSpeechText('hello', { preprocessors: [tag('a'), tag('b')] })).toBe('hello[a][b]')
|
||||
expect(calls).toEqual(['a', 'b'])
|
||||
})
|
||||
|
||||
it('uses the first formula hook that returns words', () => {
|
||||
const quiet: SpeechPreprocessor = { name: 'quiet', formula: () => undefined }
|
||||
const named: SpeechPreprocessor = { name: 'named', formula: (tex, { language }) => tex === 'E=mc^2' ? `${language}: mass energy` : undefined }
|
||||
|
||||
expect(toSpeechText('See $E=mc^2$ here.', { preprocessors: [quiet, named, mathSpeechPreprocessor()] })).toBe('See en: mass energy here.')
|
||||
expect(toSpeechText('See $x+1$ here.', { preprocessors: [quiet, named, mathSpeechPreprocessor()] })).toBe('See x plus one here.')
|
||||
})
|
||||
|
||||
it('keeps the source of a formula that no plugin reads', () => {
|
||||
expect(toSpeechText('Value $x+1$.', { preprocessors: [moneySpeechPreprocessor()] })).toBe('Value x+1.')
|
||||
})
|
||||
|
||||
it('runs prepare hooks before parsing', () => {
|
||||
const shout: SpeechPreprocessor = { name: 'shout', prepare: markdown => markdown.toUpperCase() }
|
||||
expect(toSpeechText('*hi*', { preprocessors: [shout] })).toBe('HI')
|
||||
})
|
||||
|
||||
it('reads bare TeX in prose through the formula hooks', () => {
|
||||
const formula: SpeechPreprocessor = { name: 'formula', formula: () => 'water' }
|
||||
expect(toSpeechText(String.raw`Drink \ce{H2O} daily.`, { preprocessors: [mathSpeechPreprocessor(), formula] })).toBe('Drink H two O daily.')
|
||||
expect(toSpeechText(String.raw`Drink \ce{H2O} daily.`, { preprocessors: [formula, mathSpeechPreprocessor()] })).toBe('Drink water daily.')
|
||||
})
|
||||
|
||||
it('leaves prices in inline code', () => {
|
||||
expect(toSpeechText('Run `echo $5` now.')).toBe('Run echo $5 now.')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,3 @@
|
||||
export { mathSpeech, mathSpeechPreprocessor, speakMath, toMathJson } from './math'
|
||||
export { moneySpeechPreprocessor } from './money'
|
||||
export type { SpeechFormulaContext, SpeechPreprocessor, SpeechProseContext } from './types'
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { findBareLatex } from './bare-latex'
|
||||
|
||||
describe('findBareLatex', () => {
|
||||
function spans(text: string) {
|
||||
return findBareLatex(text).map(({ start, end }) => text.slice(start, end))
|
||||
}
|
||||
|
||||
it('finds known commands and scripts with their operands', () => {
|
||||
expect(spans(String.raw`设 \alpha 为锐角，且 a \leq b。`)).toEqual([String.raw`\alpha`, String.raw`a \leq b`])
|
||||
expect(spans(String.raw`The bare \ce{H2O} is water, and \frac{1}{2} is half.`)).toEqual([String.raw`\ce{H2O}`, String.raw`\frac{1}{2}`])
|
||||
expect(spans('E = mc^2 holds.')).toEqual(['E = mc^2'])
|
||||
expect(spans('file_name and x_{i+1}')).toEqual(['x_{i+1}'])
|
||||
})
|
||||
|
||||
it('ignores paths and unknown commands', () => {
|
||||
expect(spans(String.raw`Open C:\Users\neko now.`)).toEqual([])
|
||||
expect(spans(String.raw`\newcommand is not read.`)).toEqual([])
|
||||
})
|
||||
})
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
import { mathSpeech } from './speech'
|
||||
|
||||
/** Commands that start a formula in plain text. Unknown commands, such as a path `C:\Users`, do not. */
|
||||
const knownCommands = new Set([
|
||||
...Object.keys(mathSpeech.locale('en').words),
|
||||
...Object.keys(mathSpeech.locale('en').relations),
|
||||
...Object.keys(mathSpeech.locale('en').operators),
|
||||
// The commands of extensions, for example `\ce` of chemistry.
|
||||
...mathSpeech.rawCommands.map(command => `\\${command}`),
|
||||
...['frac', 'dfrac', 'tfrac', 'sqrt', 'text', 'mathrm', 'mathbb', 'overline', 'vec', 'hat', 'left', 'right', 'sum', 'prod', 'int', 'binom', 'overset', 'prime', 'circ'].map(name => `\\${name}`),
|
||||
].filter(key => key.startsWith('\\')))
|
||||
|
||||
/** One symbol that can continue a formula: an operator, a digit, an uppercase name, or one lowercase letter. */
|
||||
const mathToken = /^(?:\s+|\d+(?:\.\d+)?|[A-Z]+|[a-z](?![a-z])|[=+\-*/<>()|'!^_[\]∞⊥∥≤≥≠≈∈∉⊆⊂∠△°±×÷→])/
|
||||
const trigger = /\\([a-z]+)|[a-z0-9)]\^[{\w]|_\{/gi
|
||||
|
||||
function readBalanced(text: string, start: number): number {
|
||||
let depth = 0
|
||||
for (let index = start; index < text.length; index++) {
|
||||
if (text[index] === '\\') {
|
||||
index++
|
||||
continue
|
||||
}
|
||||
if (text[index] === '{')
|
||||
depth++
|
||||
if (text[index] === '}' && --depth === 0)
|
||||
return index + 1
|
||||
}
|
||||
return text.length
|
||||
}
|
||||
|
||||
/** Finds TeX that a reply writes without delimiters, for example `设 \alpha 为锐角` or `\ce{H2O}`. */
|
||||
export function findBareLatex(text: string): { start: number, end: number }[] {
|
||||
const spans: { start: number, end: number }[] = []
|
||||
for (const match of text.matchAll(trigger)) {
|
||||
if (match.index < (spans.at(-1)?.end ?? 0))
|
||||
continue
|
||||
if (match[1] !== undefined && !knownCommands.has(`\\${match[1]}`))
|
||||
continue
|
||||
|
||||
// Extend left over the operand before the trigger, for example `a` in `a \leq b`.
|
||||
let start = match.index
|
||||
while (start > 0) {
|
||||
const before = text.slice(0, start)
|
||||
const token = /(?:\s+|\d+(?:\.\d+)?|[A-Z]+|(?<![a-z])[a-z]|[=+\-*/<>()|'!^_])$/.exec(before)
|
||||
if (!token || before.length - token[0].length < (spans.at(-1)?.end ?? 0))
|
||||
break
|
||||
start -= token[0].length
|
||||
}
|
||||
|
||||
let end = match.index
|
||||
while (end < text.length) {
|
||||
const rest = text.slice(end)
|
||||
if (rest[0] === '\\') {
|
||||
end += /^\\(?:[a-z]+|.)/i.exec(rest)![0].length
|
||||
continue
|
||||
}
|
||||
if (rest[0] === '{') {
|
||||
end = readBalanced(text, end)
|
||||
continue
|
||||
}
|
||||
const token = mathToken.exec(rest)
|
||||
if (!token)
|
||||
break
|
||||
end += token[0].length
|
||||
}
|
||||
|
||||
while (start < end && /\s/.test(text[start]!))
|
||||
start++
|
||||
while (end > start && /\s/.test(text[end - 1]!))
|
||||
end--
|
||||
if (end > start)
|
||||
spans.push({ start, end })
|
||||
}
|
||||
return spans
|
||||
}
|
||||
|
||||
/**
|
||||
* Braces a multi-digit script in TeX that a reply writes without delimiters.
|
||||
* TeX applies `^` and `_` to one character, but `2^10` in plain text means 2 to the 10th power.
|
||||
* Formulas inside `$` keep the TeX rule, because the renderer shows them by that rule.
|
||||
*
|
||||
* @example
|
||||
* braceBareScripts('2^10 + x_12')
|
||||
* // => '2^{10} + x_{12}'
|
||||
*/
|
||||
export function braceBareScripts(tex: string): string {
|
||||
return tex.replace(/([\^_])(\d{2,}(?:\.\d+)?|\d+\.\d+)/g, '$1{$2}')
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
import type { SpeechPreprocessor } from '../types'
|
||||
|
||||
import { braceBareScripts, findBareLatex } from './bare-latex'
|
||||
import { speakMath } from './speech'
|
||||
|
||||
export { mathSpeech, speakMath, toMathJson } from './speech'
|
||||
|
||||
/**
|
||||
* Reads TeX formulas, chemistry in `\ce{}`, and units as words.
|
||||
*
|
||||
* Formulas go through `mathSpeech`, see ./speech. In prose, the plugin also finds TeX without delimiters, for example
|
||||
* `\ce{H2O}`, and reads it as a formula.
|
||||
*/
|
||||
export function mathSpeechPreprocessor(): SpeechPreprocessor {
|
||||
return {
|
||||
name: 'math',
|
||||
formula: (tex, { language }) => speakMath(tex, language),
|
||||
prose: (text, { readFormula }) => {
|
||||
const spans = findBareLatex(text)
|
||||
if (spans.length === 0)
|
||||
return text
|
||||
|
||||
const context = spans.reduceRight((rest, { start, end }) => rest.slice(0, start) + rest.slice(end), text)
|
||||
let result = ''
|
||||
let last = 0
|
||||
for (const { start, end } of spans) {
|
||||
result += text.slice(last, start) + readFormula(braceBareScripts(text.slice(start, end)), context)
|
||||
last = end
|
||||
}
|
||||
return result + text.slice(last)
|
||||
},
|
||||
}
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { speakMath } from './speech'
|
||||
|
||||
// The readings of each language are tested next to their locale packs, in @proj-airi/speech-transformer-math and
|
||||
// @proj-airi/speech-transformer-chem. This file tests only how AIRI uses them.
|
||||
describe('speakMath', () => {
|
||||
it('reads formulas and chemistry with the default instance', () => {
|
||||
expect(speakMath(String.raw`x^2`, 'en')).toBe('x squared')
|
||||
expect(speakMath(String.raw`\ce{2H2 + O2 -> 2H2O}`, 'zh')).toBe('氢气 和 氧气 反应生成 水')
|
||||
})
|
||||
})
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
import type { MathJson } from '@proj-airi/mathjson'
|
||||
|
||||
import { chemistry } from '@proj-airi/speech-transformer-chem'
|
||||
import { MathSpeech } from '@proj-airi/speech-transformer-math'
|
||||
|
||||
/**
|
||||
* The formula reader of AIRI: the built-in locale packs of `@proj-airi/speech-transformer-math`, with chemistry.
|
||||
* A contributor adds or changes a language with `mathSpeech.register`.
|
||||
*/
|
||||
export const mathSpeech = new MathSpeech({ extensions: [chemistry()] })
|
||||
|
||||
/** Reads TeX math source as words of one language. `\ce{}` is read as chemistry. See {@link MathSpeech.speak}. */
|
||||
export function speakMath(tex: string, language: string): string {
|
||||
return mathSpeech.speak(tex, language)
|
||||
}
|
||||
|
||||
/** Lowers TeX math source into MathJSON, without reading it. See {@link MathSpeech.toMathJson}. */
|
||||
export function toMathJson(tex: string): MathJson {
|
||||
return mathSpeech.toMathJson(tex)
|
||||
}
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
import type { SpeechPreprocessor } from '../types'
|
||||
|
||||
import { escapePrices, speakPrices } from './prices'
|
||||
|
||||
export { escapePrices, speakPrices } from './prices'
|
||||
|
||||
/**
|
||||
* Reads dollar amounts in prose as words of their sentence, for example `$3` as "three dollars".
|
||||
*
|
||||
* Before parsing, it escapes the `$` of a price with the Pandoc rule, so `$5 and $10` is not a formula.
|
||||
* Inline code keeps its prices as written.
|
||||
*/
|
||||
export function moneySpeechPreprocessor(): SpeechPreprocessor {
|
||||
return {
|
||||
name: 'money',
|
||||
prepare: escapePrices,
|
||||
prose: (text, { kind, language }) => kind === 'text' ? speakPrices(text, language) : text,
|
||||
}
|
||||
}
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
import { spellNumber } from '@proj-airi/speech-transformer-math'
|
||||
|
||||
/** The words of money in one language. */
|
||||
export interface MoneyLocale {
|
||||
/** Reads an amount of US dollars from its digits, for example `1200` or `19.99`. */
|
||||
dollars: (digits: string) => string
|
||||
}
|
||||
|
||||
/**
|
||||
* The money words of each language, by primary language subtag.
|
||||
* Chinese keeps the digits, because a Chinese voice reads them. The other languages spell them, so the voice does not
|
||||
* read them in another language.
|
||||
*/
|
||||
const locales: Readonly<Record<string, MoneyLocale>> = {
|
||||
en: { dollars: digits => `${spellNumber(digits, 'en')} ${digits === '1' ? 'dollar' : 'dollars'}` },
|
||||
ja: { dollars: digits => `${spellNumber(digits, 'ja')}ドル` },
|
||||
ko: { dollars: digits => `${spellNumber(digits, 'ko')} 달러` },
|
||||
zh: { dollars: digits => `${digits}美元` },
|
||||
}
|
||||
|
||||
/** The money words of a language, or English for a language without them. */
|
||||
export function moneyLocale(language: string): MoneyLocale {
|
||||
return locales[language] ?? locales.en!
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
|
||||
import { escapePrices, speakPrices } from './prices'
|
||||
|
||||
describe('prices', () => {
|
||||
it('escapes the dollar sign of a price, and keeps formulas', () => {
|
||||
// remark-math paired `$5, and the area is $` as one formula before.
|
||||
expect(escapePrices('A ticket is $5, and the area is $x^2$.')).toBe('A ticket is \\$5, and the area is $x^2$.')
|
||||
expect(escapePrices('It costs $5 and $10 today.')).toBe('It costs \\$5 and \\$10 today.')
|
||||
expect(escapePrices('$2 + 3$ and $5$ and $3\\times10^{8}$')).toBe('$2 + 3$ and $5$ and $3\\times10^{8}$')
|
||||
expect(escapePrices('Use `$5` here.')).toBe('Use `$5` here.')
|
||||
})
|
||||
|
||||
it.each([
|
||||
['Apples cost $3 and $1.', 'en', 'Apples cost three dollars and one dollar.'],
|
||||
['家賃は月 $1,200 です。', 'ja', '家賃は月 せんにひゃくドル です。'],
|
||||
['사과는 $3, 바나나는 $5입니다.', 'ko', '사과는 삼 달러, 바나나는 오 달러입니다.'],
|
||||
['这本书售价 $19.99。', 'zh', '这本书售价 19.99美元。'],
|
||||
] as const)('reads the prices of %s as words', (text, language, speech) => {
|
||||
expect(speakPrices(text, language)).toBe(speech)
|
||||
})
|
||||
})
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
import { codeSpans } from '../../markdown'
|
||||
import { moneyLocale } from './locales'
|
||||
|
||||
/** A dollar amount after `$`, with optional grouping commas and cents, for example `$1,200` or `$19.99`. */
|
||||
const priceSource = String.raw`\$(\d{1,3}(?:,\d{3})+|\d+)(?:\.\d+)?`
|
||||
/** A price ends at a space, punctuation, CJK text, or the end. `$5\%$` and `$5$` are formulas, not prices. */
|
||||
const priceEndSource = String.raw`(?=$|[\s,.;:!?)\]、，。！？）」』\p{Script=Han}\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Hangul}])`
|
||||
const price = new RegExp(`^${priceSource}${priceEndSource}`, 'u')
|
||||
const prices = new RegExp(`${priceSource}${priceEndSource}`, 'gu')
|
||||
|
||||
/**
|
||||
* Returns true when the `$` after `from` closes an inline formula by the Pandoc rule: no space before it, and no digit
|
||||
* after it. The search stops at a blank line, because a formula does not cross paragraphs.
|
||||
*/
|
||||
function closesFormula(text: string, from: number): boolean {
|
||||
for (let index = from; index < text.length; index++) {
|
||||
if (text[index] === '\n' && text[index + 1] === '\n')
|
||||
return false
|
||||
if (text[index] === '\\') {
|
||||
index++
|
||||
continue
|
||||
}
|
||||
if (text[index] === '$')
|
||||
return !/\s/.test(text[index - 1] ?? ' ') && !/\d/.test(text[index + 1] ?? '')
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Escapes the `$` of a price, outside code, so the markdown parser keeps it as text.
|
||||
*
|
||||
* remark-math pairs any two `$`, so `$5, and the area is $x^2$` became one formula from `$5` to `$x`. A `$` that starts
|
||||
* a price, and whose next `$` cannot close a formula, is a price. `$2 + 3$` stays a formula.
|
||||
*
|
||||
* @example
|
||||
* escapePrices('A ticket is $5, and the area is $x^2$.')
|
||||
* // => 'A ticket is \\$5, and the area is $x^2$.'
|
||||
*/
|
||||
export function escapePrices(markdown: string): string {
|
||||
return markdown
|
||||
.split(codeSpans)
|
||||
.map((part, index) => {
|
||||
if (index % 2 === 1)
|
||||
return part
|
||||
let result = ''
|
||||
let last = 0
|
||||
for (let at = 0; at < part.length; at++) {
|
||||
if (part[at] !== '$' || part[at - 1] === '\\')
|
||||
continue
|
||||
if (part[at + 1] === '$') {
|
||||
// `$$` opens or closes display math.
|
||||
at++
|
||||
continue
|
||||
}
|
||||
const match = price.exec(part.slice(at))
|
||||
if (!match || closesFormula(part, at + match[0].length))
|
||||
continue
|
||||
result += `${part.slice(last, at)}\\`
|
||||
last = at
|
||||
}
|
||||
return result + part.slice(last)
|
||||
})
|
||||
.join('')
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the dollar amounts of a text as words of one language.
|
||||
*
|
||||
* @example
|
||||
* speakPrices('Apples cost $3.', 'en')
|
||||
* // => 'Apples cost three dollars.'
|
||||
*/
|
||||
export function speakPrices(text: string, language: string): string {
|
||||
const locale = moneyLocale(language)
|
||||
return text.replace(prices, match => locale.dollars(match.slice(1).replaceAll(',', '')))
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user