FireRedTTS3 — multilingual TTS (24 languages, incl. Ukrainian)

Local speech synthesis service based on FireRedTTS3 with an HTTP API and MCP server, on port :8020.

The model was released on 13.08.2026, Apache 2.0 license, weights are public.

What to know before starting

There are no preset voices. An empty library means there is nothing to synthesize with. First you add a voice from any speech recording (add_firered_voice), then use it by name. The model clones zero-shot right during synthesis.

A reference transcript is required. The model needs more than just the audio — it also needs the text of what is in it. If you don't provide it, we transcribe it with Whisper, but your own text is always more accurate.

No stress marks. No dictionary, no ByT5 fallback, no manual Му+дрого. The model places stress itself from context and will confuse homographs (за́мок/замо́к). This is a fundamental limitation of the multilingual model.

Related MCP server: STT2TTS MCP

Installation

cd /Users/admin/Projects/firered-tts
./setup.sh                    # venv + залежності + апстрім + патч + ваги

setup.sh does four things: creates .venv with Python 3.11 and torch 2.8.0 for MPS/CPU, clones upstream into vendor/FireRedTTS3 at the pinned commit 00570ad, patches it for Apple Silicon (see below), and pulls the base+redae weights (~11.4 GB) into pretrained_models/.

Weights separately (takes a while — better run detached):

nohup ./download-weights.sh base > data/download.log 2>&1 &
./download-weights.sh instruct   # +7.9 ГБ, для дизайну голосу й редагування

Why the patch

Upstream is written exclusively for NVIDIA and does not run on Mac at all: flash_attn cannot be built on Metal in principle, the device is hardcoded as cuda, autocast is nailed to CUDA. The patch replaces flash_attention_2 with sdpa (works everywhere, including on NVIDIA), makes the device dynamic, and adds optimizations from the speed section. The full list is in the header of src/patch_upstream.py.

The patch is idempotent and fails if the replacement did not land — if upstream has changed, you will find out immediately, not via a CUDA error on the first synthesis.

Running

./run.sh                      # http://localhost:8020

Running it separately is not required — the MCP server will start the backend itself. The backend shuts down after idle and frees memory.

HTTP API

# 1) завести голос
curl -X POST localhost:8020/clone_voice -H 'Content-Type: application/json' -d '{
  "audio": "prompts/зразок.wav", "name": "Богдан",
  "prompt_text": "Це зразок мого голосу для клонування.",
  "language": "Ukrainian", "gender": "male"}'

# 2) озвучити
curl -X POST localhost:8020/tts -H 'Content-Type: application/json' -d '{
  "text": "Сьогодні чудова погода, ходімо гуляти в парк.",
  "voice": "Богдан", "format": "mp3"}' -o out.mp3

# Або one-shot — без реєстрації голосу: передай reference_audio замість voice,
# транскрипт зробить Whisper (перший виклик +~30с, далі кешується)
curl -X POST localhost:8020/tts -H 'Content-Type: application/json' -d '{
  "text": "Сьогодні чудова погода.", "reference_audio": "prompts/зразок.mp3",
  "format": "mp3"}' -o out.mp3

MCP

See mcp_server/README.md. In short:

claude mcp add firered-tts -- /Users/admin/Projects/firered-tts/.venv/bin/python \
  /Users/admin/Projects/firered-tts/mcp_server/server.py

Tools: firered_backend_status, list_firered_voices, add_firered_voice, delete_firered_voice, synthesize_firered_speech, design_firered_voice, edit_firered_speech.

Memory

Mac mini M4, 16 GB. Weights on disk: base (7.9) + redae (3.5) = 11.4 GB. Less in memory — the LLM backbone lives in half (4.2 instead of 8.4 GB) — but on 16 GB it is still tight. The consequences are baked into the code:

• exactly one model lives in memory — base or instruct; switching = a full reload (minutes, logged loudly);

• synthesis is serialized by a global lock — two parallel requests on 16 GB cause OOM, not a speedup;

• auto-shutdown after 1800s of idle — deliberately long: a restart costs ~40s of shader compilation, so keeping the process alive is cheaper;

• Whisper for auto-transcription is int8 on CPU and is unloaded immediately after recognition.

Speed and four optimizations

A naive run of upstream on M4 gave ×186 real time — 3 seconds of Ukrainian took 9 minutes to compute. After four fixes it became ×3.0, i.e. about 60 times faster. What exactly was wrong (everything measured with tools/profile_steps.py):

1. Autocast on the AR-loop step — the main problem. Upstream attaches @torch.autocast as a decorator to _backbone_one_step, i.e. the region opens and closes on EVERY autoregression step. The weight recast cache in torch lives exactly inside the region, so 1.7B fp32 parameters were converted to half every step. Backbone step: 19000 → 2710 ms. Now autocast is disabled and the cast is done explicitly at the backbone boundary.

2. Backbone in fp32. The weights are stored in float32 (3.0B parameters), which on 16 GB means swapping. We convert only the LLM backbone to half (4.2 GB instead of 8.4), while redae and the flow decoder stay fp32 — half breaks MPS-matmul there. Step: 2710 → 1387 ms.

3. The reference was re-encoded for every sentence. generate() is called for every sentence of the text, and each call ran the redae encoder over the same reference recording: 5.6s each time, regardless of text length. Now there is a cache keyed by audio content. For 6.6 min of audio — 30 hits vs 2 misses, ~11% of the run time.

4. Metal shader compilation. The largest part of the "slowness" turned out to be one-time: Metal compiles kernels on first execution. Individual backbone step measurements: 9873, 2104, 120, 93, 96, 97… ms. That is why the service does a warm-up at startup (FIRERED_WARMUP=1) and has a long idle timeout.

Measured on Mac mini M4 / 16 GB, 3.0s of Ukrainian, warmed-up model:

Profile of warmed-up generation: flow decoder 57%, backbone 18%, redae 11% — the most effective lever is precisely n_timesteps. There is no real-time, but this is already a working tool, not a "run it overnight".

Text normalization

The built-in TN (wetext) only knows Chinese and English, so for Ukrainian it is useless and we do not install it. 19:30, 250 грн, 2026 р. the model will voice however it can. Two options: write the text in words right away, or enable LLM-TN — FIRERED_TN_API_URL / _API_KEY / _MODEL in .env. Any OpenAI-compatible endpoint works, including a local one (llama.cpp / vLLM / Ollama) — then everything stays offline.

License

The code and weights are Apache 2.0. The upstream README additionally states that zero-shot cloning is "solely for academic research purposes" — this contradicts Apache 2.0, so be careful about commercial use of cloned voices. For home use, the question does not arise.