---
language:
- en
license: mit
library_name: pytorch
pipeline_tag: automatic-speech-recognition
tags:
- automatic-speech-recognition
- asr
- speech
- zipformer
- ctc
- cr-ctc
- librispeech
- pytorch
- mps
datasets:
- librispeech_asr
---

# asr — a from-scratch ASR model

A 30.5M-parameter speech recognition model trained from scratch on Apple Silicon
(MPS backend) on LibriSpeech `train-clean-100`. No pretrained weights, no
fine-tuning — the encoder, decoder, training loop, and tokenizer are all in this
repo.

Code also at [github.com/shubhexists/asr](https://github.com/shubhexists/asr).

## What's here

| File | Size | What it is |
| --- | --- | --- |
| `checkpoints/model_weights.pt` | 112 MB | Weights only — use this for inference |
| `checkpoints/latest.pt` | 335 MB | Full checkpoint: weights + optimizer + scheduler, for resuming training |
| `configs/tokenizer.model` | 313 KB | SentencePiece BPE model (vocab 5000) — **required** for both |
| `configs/zipformer_s.yaml` | | Model + training config |
| `src/`, `scripts/` | | Model, training, evaluation, and transcription code |

Both checkpoints are epoch 40 / step 71320 and contain identical weights.

## Architecture

**Encoder** (`src/zipformer.py`) — Zipformer-inspired ([arXiv:2310.11230](https://arxiv.org/abs/2310.11230))
5-stage U-Net over time: full-res → half-res → quarter-res (bottleneck) →
half-res → full-res, with additive skip connections between matching
resolutions. Downsampling the middle stages fits more layers into the same
compute budget. Built from standard PyTorch primitives — not a byte-exact port
of icefall's custom ops.

**Frontend** (`src/model.py`) — log-mel fbank (80 mels) → per-utterance
mean/var normalization → SpecAugment, computed on the fly.

**Loss** (`src/model.py`) — hybrid CR-CTC + attention:
- Two SpecAugmented views of each utterance are stacked on the batch dimension
  for a single encoder pass, then split. Each gets its own CTC loss, plus a
  symmetric KL consistency loss between the two views
  (CR-CTC, [arXiv:2410.05101](https://arxiv.org/abs/2410.05101)).
- An attention decoder (`src/decoder.py`, standard Transformer decoder) consumes
  one view and is trained with cross-entropy.
- Total loss weights CTC (0.3), attention (0.7), and consistency (0.2) terms.

**Tokenizer** (`src/tokenizer.py`) — SentencePiece BPE, vocab 5000, fixed
special ids: `<pad>=0` (also the CTC blank), `<unk>=1`, `<bos>=2`, `<eos>=3`,
`<spk_change>=4`, regular pieces from 5.

**Shape** — `d_model=256`, 14 encoder layers across the 5 U-Net stages
(`[2, 3, 4, 3, 2]`), 4 attention heads, 4-layer attention decoder.

## Training

| | |
| --- | --- |
| Data | LibriSpeech `train-clean-100` (100 h), `dev-clean` for validation |
| Hardware | Apple Silicon, PyTorch MPS backend |
| Steps | 71,320 (40 epochs) |
| Batch size | 16 |
| Optimizer | AdamW, lr 3e-4, weight decay 0.01, 2500 warmup steps, grad clip 5.0 |
| Final val loss | 0.789 on `dev-clean` |

**WER has not been benchmarked.** Only losses were logged during training. To
measure it yourself, see the evaluation command below — `src/evaluate.py`
reports WER for both the greedy CTC and greedy attention-decoder paths.

## Usage

```bash
git clone https://huggingface.co/shubhexists/asr
cd asr
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
```

Transcribe a single file:

```bash
python -m scripts.transcribe --config configs/zipformer_s.yaml \
    --checkpoint checkpoints/model_weights.pt --audio /path/to/clip.wav
```

Evaluate WER on a LibriSpeech split (needs the data extracted under `data/`,
see below):

```bash
python -m src.evaluate --config configs/zipformer_s.yaml \
    --checkpoint checkpoints/model_weights.pt --split test-clean
```

Resume training from the full checkpoint:

```bash
python -m src.train --config configs/zipformer_s.yaml --resume checkpoints/latest.pt
```

Training and evaluation need LibriSpeech from
[openslr.org/resources/12](https://www.openslr.org/resources/12/), extracted
into `data/` in torchaudio's layout:

```
data/LibriSpeech/<split>/<speaker>/<chapter>/<speaker>-<chapter>-<utterance>.flac
data/LibriSpeech/<split>/<speaker>/<chapter>/<speaker>-<chapter>.trans.txt
```

> `torch.nn.functional.ctc_loss` has no MPS kernel, so the training and eval
> scripts set `PYTORCH_ENABLE_MPS_FALLBACK=1` to run that one op on CPU.

## Limitations

- Trained on 100 hours of read audiobook English only. Expect it to degrade on
  conversational speech, accents outside LibriSpeech's distribution, noisy
  audio, and any non-English input.
- Expects 16 kHz mono audio.
- No language model, no beam search — decoding is greedy.
- WER is unmeasured, so treat this as a working reference implementation rather
  than a model with a known quality bar.