Proposal
Add an opt-in FunASR annotation step for Fish Speech fine-tuning datasets. The tool would recursively find supported audio files and create the sibling .lab files expected by the existing fine-tuning pipeline.
The smallest useful first version would:
- process
.wav, .mp3, and .flac files under a dataset directory;
- write plain UTF-8 transcription text to
<audio-stem>.lab;
- skip existing
.lab files by default, with an explicit overwrite option;
- allow the model, device, language, and ITN behavior to be selected;
- keep the FunASR import optional and fail with a clear installation command;
- expose a dry-run mode and report processed, skipped, and failed files;
- cover file discovery, result parsing, overwrite behavior, and failure isolation with tests that do not download a model.
Suggested model scope
SenseVoiceSmall is a practical default for Mandarin, Cantonese, English, Japanese, and Korean. It also predicts utterance-level language, emotion, and audio-event tags. Plain transcription should remain the default output because Fish Speech currently documents .lab as transcription text.
An explicit option could preserve supported non-neutral emotion or non-speech event labels as Fish-style leading tags, for example [happy] or [laughter]. This should be opt-in: SenseVoice labels are utterance-level predictions, not word-level annotations, and should not be presented as fine-grained alignment.
For longer recordings, the recipe can compose SenseVoice with FunASR's separate FSMN-VAD model. Speaker diarization should remain out of scope for the first version; it requires a separate speaker model such as CAM++ and a policy for turning speaker segments into Fish Speech's per-speaker directory layout.
Integration choices
- Repository CLI (recommended): add a small
tools/data/annotate_funasr.py command and document it immediately before semantic-token extraction. This gives users reproducible skip/overwrite behavior and testable output.
- Documentation-only recipe: show a standalone FunASR script in the fine-tuning guide. This has less maintenance cost but leaves batching, retries, and safe file writing to each user.
- General ASR provider abstraction: support multiple annotation engines behind one interface. This is flexible but larger than the current request and probably unnecessary until a second backend is wanted.
I can prepare option 1 with lazy imports and model-free unit tests if that scope fits the maintainers' expectations. If keeping FunASR outside the main repository dependency surface is preferred, I can instead contribute option 2.
Capability and license notes
FunASR is a toolkit rather than a single checkpoint. The FunASR and SenseVoice repository source code is MIT-licensed; model weights follow their model cards. The released SenseVoiceSmall checkpoint supports the five ASR languages listed above, and its weights use the linked FunASR Model Open Source License Agreement. Language coverage, punctuation, timestamps, and performance depend on the selected model and runtime configuration.
References:
Proposal
Add an opt-in FunASR annotation step for Fish Speech fine-tuning datasets. The tool would recursively find supported audio files and create the sibling
.labfiles expected by the existing fine-tuning pipeline.The smallest useful first version would:
.wav,.mp3, and.flacfiles under a dataset directory;<audio-stem>.lab;.labfiles by default, with an explicit overwrite option;Suggested model scope
SenseVoiceSmallis a practical default for Mandarin, Cantonese, English, Japanese, and Korean. It also predicts utterance-level language, emotion, and audio-event tags. Plain transcription should remain the default output because Fish Speech currently documents.labas transcription text.An explicit option could preserve supported non-neutral emotion or non-speech event labels as Fish-style leading tags, for example
[happy]or[laughter]. This should be opt-in: SenseVoice labels are utterance-level predictions, not word-level annotations, and should not be presented as fine-grained alignment.For longer recordings, the recipe can compose SenseVoice with FunASR's separate FSMN-VAD model. Speaker diarization should remain out of scope for the first version; it requires a separate speaker model such as CAM++ and a policy for turning speaker segments into Fish Speech's per-speaker directory layout.
Integration choices
tools/data/annotate_funasr.pycommand and document it immediately before semantic-token extraction. This gives users reproducible skip/overwrite behavior and testable output.I can prepare option 1 with lazy imports and model-free unit tests if that scope fits the maintainers' expectations. If keeping FunASR outside the main repository dependency surface is preferred, I can instead contribute option 2.
Capability and license notes
FunASR is a toolkit rather than a single checkpoint. The FunASR and SenseVoice repository source code is MIT-licensed; model weights follow their model cards. The released
SenseVoiceSmallcheckpoint supports the five ASR languages listed above, and its weights use the linked FunASR Model Open Source License Agreement. Language coverage, punctuation, timestamps, and performance depend on the selected model and runtime configuration.References: