torchaudio 2.9 moved save() onto TorchCodec, whose compatibility shim
converts non-float32 input with a bare src.float() and no rescaling, then
treats the result as [-1, 1] audio. Every save site passed a PCM-scale
int16 tensor, so on 2.9+ roughly 99.99% of frames clipped to full scale
with no exception and no warning.
Add save_pcm_wav() in indextts/utils/common.py, which normalizes PCM-scale
input to float32 [-1, 1] and pins encoding="PCM_S"/bits_per_sample=16 on
the torchaudio versions that still honour them. Pinning matters: below 2.9
the WAV subtype is derived from the input dtype, so normalizing alone would
silently switch output from 16-bit PCM to 32-bit float. Above 2.9 the two
arguments are ignored and warn, so they are omitted there.
Route all six save sites in indextts/ and backends/trt/ through it.
Verified on torchaudio 2.7.1, 2.8.0, 2.9.0 and 2.11.0: identical 16-bit
PCM output, 0% clipped frames. Before the change 2.9.0 and 2.11.0 produced
only 3 distinct sample values with 99.9875% of frames at full scale.
Fixes#724
The Space at IndexTeam/IndexTTS-2.5-Demo now runs 2.5, but the Demos
column only pointed at the demo page and the ModelScope studio. Add a
Space badge next to them in all five badge-style READMEs, and a plain
[HF Space] link in README2.5_ZH.md, which uses plain links instead.
The 2.5 inference path exposes `use_gpt_latent`, but enabling it has never
worked: the fusion at `S_infer + latent` raises
RuntimeError: The size of tensor a (1706) must match the size of tensor b (853)
The two tensors are on different time bases by construction.
`EnhancedCodec.decode()` upsamples 2x (`indextts/codec/models.py:226`,
`F.interpolate(x, scale_factor=2, mode="nearest")`, with `downsample_scale=2` as
the constructor default and no override in the released config), while the GPT
latent stays at the code-token rate. 853 -> 1706 is exactly that factor.
This is 2.5-specific: 2.0 hardcodes `use_gpt_latent=True`
(`indextts/infer_v2.py:145`) and works, because it uses the maskgct codec, which
has no such upsampling step. When 2.5 switched codecs the latent path was left
wired to the old assumption.
Rather than guess an alignment, remove the option. The released 2.5 checkpoint
does carry six `gpt_layer` tensors, but that only records what the training run
built, not that this is a supported inference path — and if 2.5 had actually been
trained through this fusion, the lengths would agree. Any alignment added now
would be inferred rather than recovered from the training recipe, and getting it
wrong yields quietly degraded audio instead of a crash, which is worse.
No behaviour change for anyone: the parameter defaulted to False, and neither
`webui.py` nor `indextts/cli_v2.py` exposes it. `load_checkpoint2` iterates
`for key in model.models`, so the now-unbuilt `gpt_layer` weights are skipped
silently with no warning. `MyModel` keeps its `use_gpt_latent` argument, which
`backends/trt/export/*.py` still passes.
Also drops the `gpt_forward_time` counter, whose only accumulation site was
inside the removed block — it would otherwise print a constant `0.00` and read as
"this stage was free" rather than "this stage does not exist". 2.0 keeps its own.
Verified: 153 passed / 22 deselected, and a real MPS inference produces audio
with the default path untouched.
Co-authored-by: nanaoto <10inspiral@gmail.com>
* test: resolve temp paths in the CLI tests so they pass off Linux
These assertions compare a path the CLI printed against one built from
`tempfile.TemporaryDirectory()`. On Linux those agree; on macOS they do not,
because `/var` is a symlink to `/private/var` and the CLI resolves it:
AssertionError: 'model_dir = "/var/folders/.../models"'
not found in 'model_dir = "/private/var/folders/.../models"'
18 of 153 non-GPU tests failed that way. Windows has its own version of the same
problem — `RUNNER~1` 8.3 short names and drive-letter case — so this blocks CI on
any non-Linux runner.
Resolve at the point the temp directory is wrapped, which is what
`test_cli_v2_download.py` already does at its seven sites and why that file was
the only one passing. 129 call sites across the other four files; no behaviour
changes, the comparison just uses the same canonical form on both sides.
153 passed, 0 failed afterwards.
* ci: run the lightweight checks on Windows too
Everything platform-specific in this repo was unverified. `pyproject.toml`
carries three `sys_platform` markers — triton-windows gated to win32, and the
wetext / WeTextProcessing split on linux — and `cli_tests` branches three ways on
`sys.platform` for config and model-cache locations. CI only ever exercised the
Linux side of any of it. The triton-windows marker bug (#742, #749) was exactly
this shape: a Windows-only resolution failure that no amount of green Linux CI
could catch.
No GPU is needed. `-m "not gpu"` deselects the 22 GPU tests, and the one
remaining case that passes `--device cuda` runs against a hand-built
`SimpleNamespace` whose `is_available` is `lambda: True`
(`cli_tests/test_cli_v2_check.py:93-98`), so nothing looks for a real device.
`fail-fast: false` so a Windows failure still lets the Linux result through, and
the uv cache is enabled — the Windows job resolves a cu128 torch and there is no
reason to refetch it every run.
Blocking rather than advisory, because the previous commit fixed the path
assertions that would otherwise have failed there: 153 passed, 0 failed.
---------
Co-authored-by: nanaoto <10inspiral@gmail.com>
* webui: adapt precision and QwenEmotion loading to available VRAM
The WebUI hardcoded use_qwen_emo=True and tied precision to --fp16, which
defaults off, so a small card got the most expensive combination available.
Measured on a 4090, peak reserved memory for a short utterance:
fp32 + QwenEmotion 8.15 GB (the previous default)
bf16 + QwenEmotion 6.54 GB
bf16, no QwenEmotion 5.48 GB
Re-run under a hard per-process cap via set_per_process_memory_fraction, the
old default does complete on an 8 GB budget -- but only by peaking at exactly
8.00 GB, i.e. with no headroom for the display output and driver overhead a
real 8 GB card also has to fund. The new path leaves ~2.5 GB spare and still
runs under a 6 GB cap (5.44 GB peak).
So below 10 GB, enable half precision and skip QwenEmotion, reusing the
threshold infer_v2_5.py already applies for text chunking. --qwen_emo forces it
back on for someone who knows their card can take it.
Skipping that model removes emotion-control-from-text, so gate the UI on it
too: mode 3 disappears from the experimental choices, its examples are filtered
out, and a preset saved with it resets to the default with a warning. Without
this gating, selecting mode 3 raises "use_emo_text=True requires QwenEmotion"
from infer_v2_5.py at inference time.
Verified all three configurations start and report the intended decision, with
process memory matching the table above.
* Gate use_bf16 on torch.cuda.is_bf16_supported() and reset is_experimental on fallback
Co-authored-by: nanaoto <19526637+nanaoto@users.noreply.github.com>
---------
Co-authored-by: nanaoto <10inspiral@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: nanaoto <19526637+nanaoto@users.noreply.github.com>
The recipe has landed, so the links no longer need to reference the pull
request. Verified https://recipes.vllm.ai/IndexTeam/IndexTTS-2.5 returns 200
and titles itself "IndexTeam/IndexTTS-2.5 | vLLM Recipes".
Seven occurrences across the English, Chinese, Japanese, Spanish and Arabic
READMEs.
Co-authored-by: nanaoto <10inspiral@gmail.com>
The 2.5 row's only Demos entry was index-tts2-5.github.io, which serves the
technical report rather than something you can talk to. Add the hosted studio
next to it so a reader can try the model without installing anything.
Verified https://modelscope.cn/studios/IndexTeam/IndexTTS-2.5 returns 200 and
titles itself "IndexTTS-2.5". English, Chinese, Japanese, Spanish and Arabic.
Co-authored-by: nanaoto <10inspiral@gmail.com>
* fix: repair Linux installs and README commands that fail on a clean machine
Verified on a Linux GPU box using a fresh .venv and a separate UV_CACHE_DIR,
since a warm cache hides every one of these.
pyproject.toml:
- triton-windows had no platform marker, so `uv sync --extra accel` (or
--all-extras) aborted on Linux: the package only publishes win_amd64 wheels.
Linux already gets `triton` as a torch dependency.
- Replace `no-build-isolation-package` with `extra-build-dependencies`. That
mode uses `.venv` itself as the build environment, and a fresh `.venv` has no
setuptools yet, so deepspeed failed with `ModuleNotFoundError: No module
named 'setuptools'` on any clean machine. Injecting torch with
`match-runtime` keeps the CUDA-enabled build inputs while leaving isolation
(and its setuptools) in place. It also lets flash-attn resolve a prebuilt
wheel from its own torch version and C++ ABI flag instead of invoking nvcc.
README (and the zh/ja/es/ar translations):
- `examples/09.wav` does not exist in the demo space and is not in
cases.jsonl; the on-demand download fetches `voice_09.wav`.
- `huggingface-hub` dropped its `cli` extra, so the install line warned.
- Example audio is fetched by the WebUI only, not by `infer_v2_5.py`, whose
default --prompt_wav is `examples/voice_01.wav`. Say so and give a one-liner
to fetch it.
- Drop `uv run indextts/infer_v2.py`: it is a benchmark loop hardcoded to
`checkpoints/`, so with the documented `checkpoints_2` layout it dies on
`vocab_file checkpoints/bpe.model does not exist`.
- `use_emo_text=True` needs `use_qwen_emo=True` at construction, which the
init example omits.
webui.py: title and arXiv badge were hardcoded to IndexTTS2, so the 2.5 UI
announced itself as 2 and linked the 2.0 paper.
* docs: install huggingface-hub without extras
`hf-xet` is an unconditional base dependency of huggingface-hub on x86_64,
arm64 and aarch64, not just an extra, so `[hf_xet]` selected nothing that the
plain package does not already pull in. The plain install still provides the
`hf`, `huggingface-cli` and `tiny-agents` executables.
Applies to the remaining seven translations and archived READMEs too, which
still carried the `[cli,hf_xet]` form whose `cli` extra no longer exists.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: nanaoto <10inspiral@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat: default webui to IndexTTS-2.5
- webui.py: --version default 2 -> 2.5. The v2 default existed only
because the empty-checkpoints auto-download fallback was implemented
for v2 alone; 2.5 just exited. Now both versions auto-download their
own repo (IndexTTS-2 / IndexTTS-2.5) when checkpoints are incomplete.
- webui.py: version-aware required_files. v2 checks bpe.model etc.;
2.5 checks codec.pth + multilingual_zh_ja_yue_char_del.tiktoken
(bpe.model is v2-only, 2.5 uses tiktoken).
- model_download.py: ensure_config_available takes version, downloads
config.yaml from the matching repo instead of hardcoded IndexTTS-2.
- README: 2.5 launch is now 'uv run webui.py' (zero flags).
* Fix ensure_config_available version validation and sync localized READMEs
Co-authored-by: nanaoto <19526637+nanaoto@users.noreply.github.com>
---------
Co-authored-by: nanaoto <10inspiral@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: nanaoto <19526637+nanaoto@users.noreply.github.com>
- Add translated READMEs: docs/README_ja.md, docs/README_es.md,
docs/README_ar.md (structure, code blocks and tables kept identical
to the English README)
- Add WebUI locales: ja_JP.json, es_ES.json, ar_SA.json (110 keys
each, covering every i18n key used in webui.py)
- Fill 3 missing keys in en_US.json (删除/情感向量/语言)
- Add language switcher links to the README headers
Co-authored-by: nanaoto <10inspiral@gmail.com>
- Add light/dark transparent logo variants, switch via <picture>
prefers-color-scheme so the header no longer shows a white box in
GitHub dark mode
- Add stars / arXiv / Discord badges under the language links
- Reorganize both READMEs into a clear section order: header intro,
model zoo, news, demos, getting started, usage, evaluation,
community, citation, acknowledgements, license
- Fix BibTeX author separators (use 'and') and complete the 2.5
author list in the zh README
- Add missing duration_factor and vLLM sections to the zh README
- Drop git-lfs steps (examples download on demand from HF/ModelScope)
- Remove uv advocacy block
Three checks on split_text_by_tokens, driven through a stub that bypasses
__init__ so they need no checkpoints and run in milliseconds:
- annotations stay paired across segment boundaries
- every segment fits text_pos_embedding's capacity
- short text is returned untouched
The annotation case is the one with teeth. Run against the pre-53dad14 splitter
it reports 4 unpaired markers; with the fix, 0. Worth recording why the earlier
manual cases were weak: single-annotation inputs pass either way, because the
splitter breaks on punctuation and then repacks chunks up to the budget, which
usually rejoins the two halves. Only when an annotation lands on a boundary does
the break survive -- so the fixture carries 10 annotations across 5 repeated
blocks to hit it.
That also corrects the trigger described in 53dad14 and e8d4383: it is not "any
annotated input long enough to segment", it is "an annotation landing on a
segment boundary", which is rarer.
Ports two changes from the internal tree (git.bilibili.co main, f8d2304) and
fixes an interaction between them and the token splitter from e8d4383.
From the internal tree, in front.py: TextNormalizer.normalize now swaps
<word|pronunciation> annotations for alphabetic placeholders before running the
zh/en normalizers and restores them afterwards, so the normalizer no longer
expands digits inside an annotation (XING2 -> XING二). Also picks up its
--text_normalization argparse fix (type=bool never produces False).
The splitter needed a matching fix. apply_pronunciation_annotations runs before
segmentation and rewrites <going|G OW1 . IH0 NG> as
<|SPECIAL_TOKEN_1|>G OW1 . IH0 NG<|SPECIAL_TOKEN_1|>, whose payload contains a
period -- one of the splitter's break characters -- so a boundary could land
inside the pair:
in: 他要<|SPECIAL_TOKEN_1|>G OW1 . IH0 NG<|SPECIAL_TOKEN_1|>去上学。
out: ['他要<|SPECIAL_TOKEN_1|>G OW1 .', ' IH0 NG<|SPECIAL_TOKEN_1|>去上学。']
leaving each half with an unpaired marker. Each
<|SPECIAL_TOKEN_n|>...<|SPECIAL_TOKEN_n|> span is now an atomic piece that
punctuation and per-character splitting skip over. This only bit inputs long
enough to segment, which is why the long-text testing in e8d4383 missed it --
those samples had no annotations.
Checked against annotated English (period inside the payload), annotated Chinese,
multiple annotations in one input, and unannotated text: markers stay paired in
all four, and unannotated input segments as before. Repo tests: 7 passed.
2.5 never segmented text. infer_generator encoded the whole input into one
tensor of shape (1, N) and then looped `for seg_idx, sent in enumerate(text_tokens)`,
which iterates the batch dimension -- so it ran exactly once no matter how long
the input was. `split_segments` appears zero times in infer_v2_5.py, versus once
in infer_v2.py.
text_pos_embedding holds max_text_tokens + 2 = 602 entries, so anything past
~600 tokens indexed out of bounds and surfaced as
`CUDA error: device-side assert triggered`. Measured on an RTX 4090 with the 2.5
weights: 450 chars worked (450 tokens), 750 and 1200 chars both crashed. 2.0 took
the same 450-char input fine because it segments.
webui's "分句最大Token数" slider (max_text_tokens_per_segment) reached 2.5 only as
a print statement.
Adds split_text_by_tokens, which counts real tokens rather than characters --
max_chars is unreliable for the multilingual tokenizer, where the character/token
ratio varies by language. It breaks on punctuation, falls back to per-character
accumulation for a clause that alone exceeds the budget, then packs clauses back
up to the limit. The budget is min(max_text_tokens_per_segment, capacity - 2)
minus the language prefix, which is re-applied to every segment.
The loop now iterates the segments, and reuses infer_generator's existing
multi-segment machinery (wavs accumulation, insert_interval_silence) plus the
per-segment progress callback 2.0 already had.
Verified on an RTX 4090: 750 chars -> 132.6s audio, 1200 chars -> 213.3s, both
previously a hard crash; short text unchanged at 2.00s. Splitter unit-checked at
1200 chars -> 11 segments, max 117 tokens each, all within the 602 capacity.
The existing low_vram path (split_text_by_punctuation, max_chars=40) is untouched.
Note it only engages below 10 GB VRAM, so larger cards were the ones hitting this.
infer_v2_5.py passed kv_cache=False while infer_v2.py passes True, so 2.5
recomputed the whole sequence every decode step. Its RTF degraded with length
(O(n^2)) instead of staying flat: 80-char text measured 0.334 with the cache off
versus 0.190 with it on, a 1.76x difference. GPT is ~80% of wall time here, which
is why one flag matters this much.
That flag arrived with the file's first version (internal commit a9ba7fd, "sync
from github"), not from a later fix, so it does not look deliberate.
Also replaces the README's "RTF 0.19-0.21" -- a single first-iteration
measurement, which runs ~55% slow -- with a table over four text lengths, median
of 3 reps, warmup discarded, all pinned to GPU 0:
overall median RTF: 2.0 fp16 0.3263 | 2.5 bf16 0.2035 | 2.0+TRT fp16 0.1365
TensorRT is 2.39x the 2.0 PyTorch path on the same weights.
Quality checked with whisper CER over 4 samples: 0-1.5% either way, so the cache
looks harmless. That sample is small, and note that "token counts differ between
kv on/off" is not a bug signal -- flipping the flag on 2.0, whose default is True
and is production-verified, also diverges, because the cached and uncached paths
use different position-embedding code and are not bit-equivalent.
Also reverts model_download.py to main's lazy _get_using_modelscope(). The 2.5
merge had taken the internal tree's version, which runs need_proxy() at import
time and drops that function, breaking
test_modelscope_single_file_download_matches_local_path. The lazy form also
avoids a network probe on import.
2.5 keeps its weights in checkpoints_25/, and the rule only covered
checkpoints/, so a 15G weight directory showed up as untracked -- easy to sweep
into a stash or commit by accident.
Uses /checkpoints*/* rather than /checkpoints*/ so the existing exception for
config.yaml still applies: git cannot re-include a file whose parent directory
is excluded outright.
Applies the 2.5 work from the internal tree (git.bilibili.co) onto current
main, which already carries the TensorRT backend from #733. The two histories
share no commits, so this is a file-level port rather than a merge.
New:
- indextts/infer_v2_5.py, indextts/gpt/model_v2_5.py
- indextts/codec/ (EnhancedCodec, amphion quantizers, vocos, maskgct codec)
- indextts/utils/tokenizer.py (multilingual), nemo_tn.py, ja_g2p.py
- configs/config_v2_5.yaml
- tests/regression_test.py, tests/cases.jsonl
- archive/README_INDEXTTS_2{,_ZH}.md (2.0 READMEs, superseded by 2.5 at root)
Changed:
- webui.py: --version {2,2.5} switch, language dropdown and duration_factor
slider for 2.5, build_tts() branching on use_bf16 + use_qwen_emo for 2.5 vs
use_fp16 for 2.0
- model_v2.py: spk_cond_mode ("conformer" default keeps 2.0 behaviour,
"campplus" for 2.5)
- infer_v2.py: use_qwen_emo flag so QwenEmotion can be skipped at load
- checkpoint.py: strict=False with missing/unexpected key reporting, needed by
2.5 checkpoints
- pyproject.toml: fugashi, unidic-lite (Japanese g2p), openai-whisper
Deliberately not carried over from the internal tree: its Tencent PyPI and
SJTU torch mirrors, .python-version 3.10 (main pins 3.11.13 per #720), the
.gitignore entry for .github (that tree is on GitLab), and examples/*.wav
(gitignored here and fetched on demand by examples_downloader.py). uv.lock
regenerated with `uv lock` rather than copied.
Verified on 1x RTX 4090: both paths load and synthesize from the merged tree --
2.5 (checkpoints_25) RTF 0.4946, 2.0 (checkpoints) RTF 0.4642, output audio
non-silent in both cases.
Souptik Chakraborty