IYRA Akshara Tokenizer (IAT)

Linguistically-correct tokenization for six Brahmic (Indic) scripts. The tokenizer treats the Akshara (orthographic syllable) as the atomic writing unit rather than the byte. It has two layers: a rule-based Akshara segmenter (pure Unicode rules, no model) that never splits an Akshara, and a SentencePiece model trained on the segmented stream. The Python library that provides the segmenter and the high-level tokenizer lives on GitHub and PyPI, linked below.

This repository holds two model versions:

Version	Vocab	Status	Files
v1.2	64,000	current	`akshara_tokenizer_v1_2.model`, `akshara_tokenizer_v1_2.map.json`
v1.1	16,000	legacy	`akshara_tokenizer_v1_1.model`, `akshara_tokenizer_v1_1.vocab`

Supported scripts

Script	Languages
Devanagari	Hindi, Sanskrit, Marathi
Gurmukhi	Punjabi
Tamil	Tamil
Telugu	Telugu
Bengali	Bengali
Kannada	Kannada

What is new in v1.2: lossless round-trip

In v1.2 each Akshara the map covers is replaced by a single private-use codepoint before SentencePiece, so no token boundary can fall inside a covered Akshara. decode(encode(text)) is byte-identical, reconstructed from the id stream alone with no side channel. This was verified on a 20-string probe set covering the six scripts, mixed script, nukta forms, conjuncts, and whitespace edges (20 of 20), and on 1,750 FLORES-200 devtest lines, 250 each from the six Indic scripts and English (1,750 of 1,750). v1.1 could not round-trip at all, because its pipeline space-joins aksharas and the model normalizes whitespace. 45,274 of the 64,000 v1.2 vocabulary pieces span two or more aksharas.

Fertility

FLORES-200 devtest, tokens per whitespace-delimited word (lower is better). Numbers match the GitHub README (benchmark_2026_07/results_v1_2.md).

Script (language)	v1.2 (64k)	v1.1 (16k)	Qwen3-14B
Devanagari (Hindi)	1.374	2.451	4.757
Gurmukhi (Punjabi)	1.445	2.684	7.758
Tamil (Tamil)	1.953	5.007	10.064
Telugu (Telugu)	2.001	3.710	11.406
Bengali (Bengali)	1.797	3.384	7.117
Kannada (Kannada)	2.058	4.166	11.876
Overall (six Indic)	1.717	3.411	8.398
English (info)	2.151	5.084	1.261

v1.2 roughly halves v1.1's token count on every Indic script and stays far below Qwen3-14B. On English, Qwen3 is more efficient, as expected for an English-centric byte-level BPE.

Akshara split rate

FLORES-200 devtest, percent of aksharas whose token boundaries fall inside them (lower is better).

Script	v1.2 (64k)	v1.1 (16k)
Devanagari	0.0945	1.6906
Gurmukhi	0.3345	0.9348
Tamil	0.0000	0.1467
Telugu	0.0643	3.6511
Bengali	0.0721	1.7465
Kannada	0.1154	3.6202
Overall	0.1109	1.8559

For v1.2, a covered akshara is one codepoint, so the residual is only the coverage-trim tail, not vocabulary pressure. v1.2 byte-fallback is near zero on all six scripts (0.00 to 0.04 percent).

The mapping table is required

The v1.2 ids are not decodable without the mapping table. Each piece is built from private-use codepoints, and the map (akshara_tokenizer_v1_2.map.json) translates those back to aksharas. The map ships next to the model and is bound to it by sha256, so loading refuses a mismatched pair. Always fetch the model and its map together, and do not store ids without both.

Versions and compatibility

v1.1 and v1.2 produce different, incompatible id streams. Nothing in the API records which model produced a stream, so decoding v1.1 ids with the v1.2 model returns silent garbage rather than an error. If you already have a corpus tokenized with v1.1, keep using v1.1 (the files remain in this repository, and PyPI 1.1.0 installs it) rather than upgrading in place.

Gurmukhi precomposed nukta letters

Precomposed Gurmukhi nukta letters (U+0A36 SHA, U+0A5E FA, U+0A59 KHHA) encode to 2 or 3 pieces rather than 1, because the map holds their decomposed equivalents. This costs tokens, never correctness: these aksharas carry zero byte-fallback and round-trip remains byte-identical. On FLORES-200 it affects 0.33 percent of Gurmukhi aksharas. Users who prefer efficiency over byte-identity may NFC-normalize their input before encoding.

Usage (v1.2, fetching files from this repo)

Install the library (it provides the segmenter and the AksharaTokenizer class), then download both v1.2 files and load them together:

pip install "akshara-tokenizer[model]"

from huggingface_hub import hf_hub_download
from akshara_tokenizer import AksharaTokenizer

repo = "GursimranSinghBasra/akshara-tokenizer"
model_path = hf_hub_download(repo, "akshara_tokenizer_v1_2.model")
map_path = hf_hub_download(repo, "akshara_tokenizer_v1_2.map.json")

tok = AksharaTokenizer.from_files(model_path, map_path)

tok.encode("न्याय दर्शन")            # [7645, 531, 7572]
tok.decode([7645, 531, 7572])        # "न्याय दर्शन"   (byte-identical)
tok.pieces("ਪੰਜਾਬ")                  # ["ਪੰਜਾਬ"]

The simpler path is PyPI: pip install "akshara-tokenizer[model]" already bundles the v1.2 model and map, so AksharaTokenizer.load() works with no download.

The rule-based segmenter alone needs no model:

from akshara_tokenizer import segment_aksharas, count_aksharas

segment_aksharas("ਨਿਆਯ ਦਰਸ਼ਨ")
# ['ਨਿ', 'ਆ', 'ਯ', ' ', 'ਦ', 'ਰ', 'ਸ਼', 'ਨ']

count_aksharas("ਪੰਜਾਬ")
# 3

Usage (v1.1, legacy)

v1.1 has no map and does not round-trip. Fetch the model and use the space-join pipeline:

from huggingface_hub import hf_hub_download
import sentencepiece as spm
from akshara_tokenizer import segment_aksharas

model_path = hf_hub_download("GursimranSinghBasra/akshara-tokenizer", "akshara_tokenizer_v1_1.model")
sp = spm.SentencePieceProcessor(model_file=model_path)
ids = sp.encode(" ".join(segment_aksharas("न्याय दर्शन")))

Patent and copyright

License

Apache License 2.0. See LICENSE and NOTICE.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

GursimranSinghBasra
/

akshara-tokenizer