Instructions to use chrishayuk/v11-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chrishayuk/v11-tokenizer with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("chrishayuk/v11-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
| { | |
| "schema": "v-tokenizers-provenance-1", | |
| "tokenizer_sha256": "10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae", | |
| "vocab_bin_sha256": "873f44dea58558abd355454b6a22281ba69ad5afe850d05e48c271536471905b", | |
| "vocab_size": 71260, | |
| "status": "adopted", | |
| "source_repo": "https://github.com/chrishayuk/v-tokenizers", | |
| "source_commit": "3ed5f3cb7aea16b150588f27f24a7f31121b1792", | |
| "corpus": { | |
| "catalog": "chuk-datasets", | |
| "dataset": "v11/tokenizer", | |
| "version_id": "v1", | |
| "content_sha": null, | |
| "resolve": "https://chuk-datasets.fly.dev/v1/datasets" | |
| }, | |
| "evaluation": { | |
| "ledger": "chuk-experiments", | |
| "run_id": null, | |
| "note": "Headline numbers belong in the card; the ledger is authoritative." | |
| } | |
| } |