FlaxSpeaker: univ_full_conformer_large_dr_vectors (conformer - large)

Disclaimer & Reproduction Notice

This is not an officially supported Google product.

This checkpoint and the accompanying FlaxSpeaker library are a from-scratch re-implementation in JAX/Flax, written using only the published papers (no proprietary code or data), of speaker recognition algorithms from:

• GE2E: Generalized End-to-End Loss for Speaker Verification (arXiv:1710.10467)
• Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognition (arXiv:2104.01989)
• ATP (pooling head repurposed from a language identification paper): Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech (arXiv:2202.12163)
• PFAS: Parameter-Free Attentive Scoring for Speaker Verification (arXiv:2203.05642)

The original papers were run on Google-internal infrastructure and data; this checkpoint is a full-data open-corpus universal flagship trained on 12,193 speakers (464,557 clean utterances + MUSAN/RIR/speed augmentation) across VoxCeleb2-dev, VoxCeleb1-dev, CN-Celeb 1+2, and LibriSpeech 960h with zero speaker overlap against official evaluation sets.

Training Budget (read this first)

This flagship model was trained for 2,400 gradient steps (warmup 150 steps, batch size 64) over the full multi-corpus universal training pool (VoxCeleb1/2 + CN-Celeb1/2 + LibriSpeech 960h) (12,193 base speakers, 464,557 clean utterances, expanded via 3-way speed perturbation + MUSAN/RIR).

Training-crop provenance. This checkpoint was trained with U(90, 180)-frame variable-length crops and an exact validity mask (the current pipeline default); the shipped config.json / YAML (variable_length_training: true) describes the released weights. Step-0 loss fingerprint at the released seed (scripts/fingerprint_released_checkpoints.py, pretrained_models/step0_fingerprints.json): archived 1.3912; current code with fixed 180-frame crops 1.1641, with U(90, 180) crops 1.3912 (match: variable). Evaluating the released weights with the current code reproduces the archived metrics in eval_results.json. At the converged S_safe budget, the seed-to-seed standard deviation of pooled 1-utt EER across seeds 0-2 is 0.08-0.41 % absolute (pretrained_models/r2_seed_and_control_results.json), while the 8,000-trial 95 % bootstrap CI half-width is ~0.8 %.

Model Summary

Property	Value
Model ID	univ_full_conformer_large_dr_vectors
Hugging Face Repository	wq2012/univ_full_conformer_large_dr_vectors
Hugging Face Collection	FlaxSpeaker Collection
GitHub Repository	wq2012/FlaxSpeaker
PyPI Package	flaxspeaker
Backbone Architecture	conformer (large)
Temporal Pooling	atp
Training Objective	Blended Extended-Set Softmax (FlaxSpeaker variant: 0.50 row-softmax + 0.50 extended-set with a -log N logit shift; not the paper's Eq. 2) (extended_set_softmax)
Verification Scorer	Down-scaled Dr-Vector variant (no base/side-info split, hidden (256, 128) leaky_relu + LayerNorm, fixed residual weight 0.25) (dr_vector)
Output Representation Dim	256
Inference Parameters (encoder + scoring head; excludes training-only classifier weights)	7,148,743
Training Data	VoxCeleb2-dev + VoxCeleb1-dev + CN-Celeb1/2 + LibriSpeech 960h (full multi-domain SOTA flagship, 12,193 speakers, 464,557 utterances)
Training Steps / Batch	2,400 steps x 64 utterances (~0.33 epochs), seed 42
Audio Frontend	80-dim natural-log Mel @ 16000 Hz + frame-level energy VAD + CMVN (flaxspeaker.AudioFrontend; not a Hugging Face AutoFeatureExtractor)

Model Files Included

• model.safetensors: model weights (encoder + GE2E w,b / Dr-Vector / PFAS heads) in SafeTensors format
• flax_model.msgpack: native Flax Msgpack serialized parameters
• config.json: FlaxSpeakerConfig (full ExperimentConfig, incl. the frontend/VAD settings)
• eval_results.json: multi-domain evaluation metrics (1-utt & 3-utt EER, minDCF, ROC-AUC, 95% bootstrap CIs, VoxCeleb1-O 10-speaker subset)

Quickstart Usage

Python API (from Hugging Face Hub or Local Directory)

Requires flaxspeaker >= 0.5.0 (pip install -U flaxspeaker).

from flaxspeaker import load_pretrained

model = load_pretrained("wq2012/univ_full_conformer_large_dr_vectors")
emb = model.extract_embedding("path/to/utterance.flac")
score = model.verify("path/to/enroll.flac", "path/to/test.flac")
print(f"Embedding shape: {emb.shape}, Verification score: {score:.4f}")

Command-Line Inference

python scripts/inference.py --model_dir wq2012/univ_full_conformer_large_dr_vectors \
    --enroll_audio path/to/enroll.flac --test_audio path/to/test.flac

Official Zero-Shot Speaker Verification Benchmarks

Benchmark	Trials	Raw EER (%)	AS-Norm EER (%)	AS-Norm + QMF EER (%)	[email protected]	ROC-AUC
VoxCeleb1-O (veri_test2.txt, 40 spks)	37,611	14.33%	13.87%	13.76%	0.9073	0.9408
CN-Celeb-E (eval/lists/trials.lst, 200 spks)	3,484,292	20.05%	19.88%	19.50%	0.9595	0.8935
CN-Celeb-E Multi-Enroll (enroll.map)	3,484,292	18.24%	17.87%	—	0.9560	0.9062
LibriSpeech test-clean (40 spks)	10,000	8.99%	8.36%	7.79%	0.7050	0.9696
LibriSpeech test-clean (3-Utt)	3,000	5.70%	6.27%	—	0.3113	0.9883

Multi-Domain Evaluation Results

1-Utterance Verification Across Evaluation Domains

Trial lists are 50/50 target/impostor over held-out speakers (LibriSpeech 40, CN-Celeb2 30, VoxCeleb1 10 speakers); [email protected] is close to its trivial ceiling of 1.0 for every model under this trial mix.

Evaluation Split	Trials	Raw EER (%)	AS-Norm EER (%)	95% Bootstrap CI (%)	[email protected]	[email protected]	ROC-AUC
LibriSpeech (English Read)	3,000	8.37%	7.80%	[7.27, 9.32]	0.6687	0.3800	0.9722
CN-Celeb2 (Chinese Multi-Genre)	3,000	16.60%	16.67%	[15.15, 17.82]	0.9213	0.6867	0.9180
VoxCeleb1 (In-the-Wild Celebrity)	2,000	15.50%	14.55%	[13.55, 17.23]	0.7520	0.7020	0.9322
Universal Pooled (All 3 Domains)	8,000	14.11%	14.39%	[13.34, 14.96]	0.8395	0.5945	0.9414
Universal Macro-Average (3 Domains)	—	13.49%	13.01%	—	0.7807	0.5896	0.9408

3-Utterance Multi-Enrollment Verification Across Evaluation Domains

Evaluation Split	Trials	3-Utt EER (%)	95% Bootstrap CI (%)	[email protected]	ROC-AUC
LibriSpeech (3-Utt)	1,000	6.00%	[4.80, 7.90]	0.2880	0.9871
CN-Celeb2 (3-Utt)	1,000	12.90%	[10.40, 14.45]	0.7420	0.9555
VoxCeleb1 (3-Utt)	1,000	10.60%	[8.75, 12.80]	0.7280	0.9616
Universal Pooled (3-Utt)	3,000	10.27%	[9.03, 11.22]	0.6573	0.9672
Universal Macro-Average (3-Utt)	—	9.83%	—	0.5860	0.9681

Single seed; the 95% bootstrap CIs resample trials (not speakers). Differences below about 1% absolute EER between FlaxSpeaker checkpoints are within the seed-to-seed variation measured in pretrained_models/r2_seed_and_control_results.json (see the GitHub README) and should not be interpreted as rankings.

Deviations from the Source Papers

• Training regime: 2,400 gradient steps x 64 utterances (~0.33 epochs) across 12,193 base speakers (464,557 clean utterances + 3x speed perturbation + MUSAN/RIR).
• Augmentation: corpus-backed MUSAN environmental noise, music, and multi-speaker babble + RIRS_NOISES room impulse responses + 3-way polyphase speed perturbation (0.9x, 1.0x, 1.1x) + SpecAugment.
• Objective: 50/50 blend of row-softmax and -log N-shifted extended-set softmax; the paper's pure Eq. 2 is available via pure_extended_set: true.
• Dr-Vector scorer: no 200/56 base/side-information split, hidden (256, 128) with leaky_relu + LayerNorm, linear read-out and a fixed residual weight 0.25 instead of the paper's 3 x 256 leaky-ReLU(0.2) layers, weighted-sum read-out and trainable affine.
• Conformer: parallel linear + Conv1D stem, no positional encoding, 4 blocks x 320-D, 8 heads, kernel 15, mask-aware GroupNorm(2), no dropout, multi-layer feature aggregation; the Google papers use 12 x 512 blocks, 4-8 heads, kernel 32, relative positional encoding and dropout.
• ATP pooling: 4 heads with a tanh bottleneck (128-D) and subspace_heads = True; the papers use a single head with a linear-sigmoid weight (attention_dim = 0). Note that arXiv:2202.12163 is a language-identification paper whose pooling head is repurposed here.
• Frontend: 80-D natural-log Mel + frame-level energy VAD (26 dB dynamic range) + CMVN; evaluation processes up to 1024 frames (10.2 s).

Citation

@inproceedings{wan2018generalized,
  title={Generalized End-to-End Loss for Speaker Verification},
  author={Wan, Li and Wang, Quan and Papir, Alan and Moreno, Ignacio Lopez},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={4879--4883},
  year={2018}
}

@inproceedings{pelecanos2021dr,
  title={Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognition},
  author={Pelecanos, Jason and Wang, Quan and Moreno, Ignacio Lopez},
  booktitle={Interspeech},
  pages={4603--4607},
  year={2021}
}

@inproceedings{wang2022attentive,
  title={Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech},
  author={Wang, Quan and Yu, Yang and Pelecanos, Jason and Huang, Yiling and Moreno, Ignacio Lopez},
  booktitle={The Speaker and Language Recognition Workshop (Odyssey)},
  year={2022}
}

@inproceedings{pelecanos2022parameter,
  title={Parameter-Free Attentive Scoring for Speaker Verification},
  author={Pelecanos, Jason and Wang, Quan and Huang, Yiling and Moreno, Ignacio Lopez},
  booktitle={The Speaker and Language Recognition Workshop (Odyssey)},
  year={2022}
}