# Model can generalize code better with programming-language-aware tokenization

**URL:** <https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957>\
**Category:** Research\
**Created:** [October 23, 2026, 5:51pm UTC](https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957 "2026-10-23T17:51:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![SamLegion888](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@SamLegion888](https://discuss.huggingface.co/u/SamLegion888)\
**Post date:** [October 23, 2026, 5:51pm UTC](https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957/1 "2026-10-23T17:51:11Z")

</div>

I have been developing a SWE model with my own code, chats, conversations for my personal agent. While tinkering with Nvidia Nemotron 3 and Qwen 3.5, I found that while their tokenizer preserves most of the operators and delimiters, they can sometimes break keywords, operators of the language in awkward places. For example it can break “\<\<=” operator to multiple different tokens as “\<\<”, “=” or “\<”, “\<=”.

Programming language has its own syntax, semantics, structure and “grammar” where the keywords, operators carry semantic meaning. A basic unit is called Lexical token which is recognized by the language lexer.

When lexical token is split to multiple tokens or merged together the meaning or behavior can change entirely, now the model has to learn that those separate pieces form a single unit or the merged vocab actually have separate meaning.

So to preserve the lexical token boundaries, I extended the vocab of Qwen3.5-4B-Base model with missing keywords from python, c++, golang, rust etc.. And finetuned on SWE dataset, then I finetuned the base model with its own tokenizer with the same recipe to compare the performance.

The extended-vocabulary model improved on HumanEval from 45.7% to 52.4% and HumanEval+ from 39.0% to 47.0%. MBPP scores slightly decreased from 65.3% to 64.6% on Evalplus.

I have published about the experiment, code, model weights and checkpoints here

> **[Lexical Boundaries for Code Tokenization](https://www.soumyaranjansahu.com/blog/qwen35-swe)**
>
> A note on exploring tokenizer boundaries, program syntax, and attention structure.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 26, 2026, 4:05am UTC](https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957/2 "2026-10-26T04:05:28Z")

</div>

For now, I tried a few things with the publicly available models:

* * *

## What I found

I was interested in two questions that seem related but are not identical: **can the extended-vocabulary fine-tune produce better code** , and **how much of a difference can be attributed specifically to lexical-token boundaries rather than the rest of the fine-tuning path?** I used the released checkpoints for an independent probe, then examined the actual tokenizer behavior and a small fixed-weight input-tokenization comparison. This is not an exact reproduction of your notebook.

The first result is a modest positive signal, with important limits. I generated one candidate for each of the **164 HumanEval tasks** using the standard model/tokenizer ( **A** ) and the extended model/tokenizer ( **B** ). I applied the core of EvalPlus’s sanitizer from a pinned revision, then used a **separate, non-official functional harness** :

| Independent functional check | A: standard | B: extended |
| --- | --- | --- |
| Tasks passing | **72/164 (43.9%)** | **81/164 (49.4%)** |
| Passes unique to the arm | 7 | 16 |

Both arms passed 65 tasks, and both failed 76.

That is **nine additional passes** for B in this run, directionally consistent with the HumanEval improvement in your [published experiment](https://www.soumyaranjansahu.com/blog/qwen35-swe). These are **not official EvalPlus pass@1 scores** : I used Transformers rather than your vLLM setup, did not run the official `evalplus.evaluate` pipeline, and had two reference-solution failures/timeouts under my own test harness. Of the 23 tasks with different A/B verdicts, B won 16 and A won seven (exact two-sided paired binomial p approximately **0.093** ). The result is suggestive, not a settled claim of superiority.

More revealing than the aggregate was where the difference appeared. In a separate tokenizer-only probe, the extended tokenizer actually emitted a newly added ID in only **8/164 HumanEval input prompts**. Among those eight tasks the independent pass counts were **A=3, B=2** ; among the other 156 they were **A=69, B=79**. So the improvement in this run was **not concentrated in prompts that directly used a new lexical ID**. That does not negate the retrofit idea: the extended checkpoint was fine-tuned under a different tokenization, and output-side tokens, new embeddings, and shared LoRA-updated weights can affect many other cases. It does weaken an explanation based only on cleaner or shorter _evaluation inputs_.

I also held the **extended checkpoint’s weights fixed** for those eight prompts and changed only their **input segmentation** : extended tokenizer versus the standard tokenizer’s existing IDs. The generated text differed for **4/8** prompts. Thus input segmentation can change a continuation without changing the weights; it does **not** establish that either segmentation gives better code in general. The output vocabulary/head remained that of the extended model, and these eight tasks were selected for tokenizer activation, not sampled to estimate general performance.

**My tentative interpretation:** the released extended checkpoint shows a positive direction on these HumanEval-style tasks under this one evaluation. The open question is _which paths contribute_: inference-time segmentation, the learned new token representations, other changes during SFT, output-side effects, stopping/length behavior, or a mixture. Separating them could develop the result further without taking anything away from the practical retrofit goal.

## The lowest-cost next check

Rather than starting with another training run, I would first **reuse saved generations and standardize the evaluator**. The [EvalPlus CLI](https://github.com/evalplus/evalplus/blob/master/docs/cli.md) supports evaluating candidate JSONL files independently of generation. I already have sanitized A/B outputs from this probe. Task-level outputs from your original vLLM run, if available, would allow a closer like-for-like comparison, but are not needed to evaluate these independent candidates.

A useful order would be:

1. **Evaluate both saved candidate sets in the same pinned official EvalPlus environment** , reporting **Base** and **Base + Extra** separately. The Plus suite adds tests to the same tasks; it is not an independent second set of 164 HumanEval problems.
2. **Join results by task ID.** Keep A-only, B-only, both, neither, timeout, invalid output, and length-cap outcomes instead of only two percentages.
3. **Overlay tokenizer measurements:** input-side new-ID hits, output-side new-ID usage, changed token counts, and actual stopping reasons. This locates interesting cases but does not by itself prove a mechanism.
4. **Choose the smallest discriminating comparison.** If improvements correlate with a particular lexical pattern, use a targeted fixed-weight test. If they occur without new IDs in the prompt, investigate output exposure and SFT/checkpoint adaptation. If the apparent gain changes with the evaluator, resolve that protocol first.

A small paired results table may already answer several questions; the larger training controls can wait until they are justified.

> **1. What I actually ran and what those numbers mean**
>
> ### Three distinct sources of evidence
> 
> To prevent an accidental apples-to-oranges comparison:
> 
> - **Your original report:** HumanEval **45.7% vs 52.4%** , HumanEval+ **39.0% vs 47.0%** , MBPP **65.3% vs 64.6%** , and MBPP+ **54.5% vs 53.7%** for standard versus extended variants. These are the published results, not my independent scores; I did not reproduce the training or original vLLM generation.
> - **Public tokenizer snapshots:** I compared retrieved files from [Qwen3.5-SWE](https://huggingface.co/SamLegion888/Qwen3.5-SWE) and ran CPU corpus tokenization; this does not certify the remote repository’s current revision.
> - **Independent model probe:** L4/Transformers raw generations, EvalPlus sanitizer **core** , and a non-official restricted functional harness. The methods and numbers should not be exchanged with your published results.
> 
> That distinction matters because agreement in _direction_ across evaluations is interesting, whereas treating the two percentages as directly exchangeable would overstate reproducibility.
> 
> ### A, B, and C: exactly what was changed
> 
> | Arm | Model weights | Input tokenizer | Evaluation prompts |
> | --- | --- | --- | --- |
> | **A** | Standard fine-tuned checkpoint | Standard | 164 |
> | **B** | Extended fine-tuned checkpoint | Extended | 164 |
> | **C** | **Same extended checkpoint as B** | **Standard input segmentation** | 8 selected |
> 
> A/B compares **whole fine-tuned systems** : tokenizer, new embeddings, and checkpoint weights all differ. B/C holds extended weights fixed and changes **input IDs only** for eight prompts selected because the extended tokenizer emits a new token. C keeps the extended output vocabulary/head and is not a new deployed model or an unbiased 164-task comparison; its 1/8 functional count cannot be compared directly with A/B rates.
> 
> The complete GPU pass produced **336 raw generations** : A164, B164, C8. It used an L4, **Transformers 5.20.2** , greedy decoding, up to **512 new tokens** , and a shared intended model-side EOS stopping criterion. Greedy is still not a promise of byte-identical continuations across inference frameworks or numeric kernels. The original report used vLLM.
> 
> | Observed generation behavior | A | B |
> | --- | --- | --- |
> | Stopped at accepted EOS | 120 | 104 |
> | Reached 512-token cap | 44 | 60 |
> | Mean generated tokens (in each arm’s IDs) | 194.3 | 238.6 |
> 
> B reached the cap more often; this might affect extraction but does **not** establish an EOS bug. Different token vocabularies also make token counts imperfect proxies for text length or runtime. Raw A/B text matched on 26/164 tasks; B/C text differed on 4/8 at fixed weights. Neither comparison alone measures correctness.
> 
> ### Functional checks and post-processing
> 
> Preserving the **raw generations** mattered: some continuations included extra definitions, repeated text or an incomplete tail, so raw-text testing and extracted-function testing answer different questions.
> 
> I first used an AST-oriented extraction approximation. I then ran the **core sanitization logic** from EvalPlus at pinned commit [`26d6d00`](https://github.com/evalplus/evalplus/blob/26d6d00bb1fd0fa37f39c99d5290da67891d1c5e/evalplus/sanitize.py). All 336 candidates were processed and the extracted source parsed in that probe. The official-core result agreed byte-for-byte with the approximate extraction for **330/336** cases. Of the six differences, five were B and one C; those six changed candidates were re-tested separately, with **no pass/fail change**. For the 330 identical sources, I carried forward the existing independent verdicts rather than re-executing them. The resulting A72/B81/C1 numbers therefore have an explicit provenance, but **this was neither a full `evalplus.sanitize` CLI run nor a full official `evalplus.evaluate` run**.
> 
> The contrast is substantial: under the independent harness, evaluating the **unprocessed raw generated source** gave **A52/164 and B53/164** , while evaluating the sanitized candidates gave **A72/164 and B81/164**. Neither raw nor sanitized independently defines the ‘true’ score; the point is that **candidate extraction must be fixed before scores are compared**. All-syntax-valid extracted code is not the same as all-tests-passing code.
> 
> The independent tests came from the `test` field in a retrieved Hugging Face HumanEval-related artifact; I have **not** demonstrated that field is identical to EvalPlus’s exact versioned **Base** tests. The reference solutions themselves passed only **162/164** in the test environment: `HumanEval/32` failed and `HumanEval/139` timed out. I cannot attribute those anomalies to the models, the benchmark, or my harness without further checking. The code was executed with separate user/namespaces, network isolation, chroot and resource limits, but this was **not an audited security sandbox**. I would not recommend directly executing untrusted generated Python in a notebook or on a host machine.
> 
> The paired outcome table from this _non-official_ test is:
> 
> | | B passes | B fails | Total |
> | --- | --- | --- | --- |
> | **A passes** | 65 | 7 | 72 |
> | **A fails** | 16 | 76 | 92 |
> | **Total** | 81 | 83 | 164 |
> 
> The 16 B-only versus seven A-only successes yield an exact two-sided paired binomial p-value of about **0.09314**. This single run cannot establish a stable advantage over prompts, inference backends or fine-tuning seeds. The per-task table is more informative than the nine-pass aggregate alone.

> **2. Tokenizer behavior: lexical matches, embeddings, and possible mechanisms**
>
> ### AddedToken extension versus a new code tokenizer
> 
> The public extended tokenizer I inspected keeps the underlying BPE vocabulary, merge rules, and pre-tokenizer rather than learning a new BPE model from scratch. It gives selected code-like strings added-token recognition priority. This is an attractive **retrofit** precisely because it leaves the old tokenizer’s ID space largely intact and can potentially reuse a pretrained model with comparatively modest adaptation.
> 
> The retrieved extended tokenizer contains **276 code-oriented AddedToken entries** , while the experiment note mentions **278 candidate lexemes** , and the model configurations differ by **40 rows of vocabulary capacity**. These are different quantities; their exact relationship is not established, and I would not infer a bug from the difference. Actual emitted IDs, selected lexemes and embedding capacity need separate accounting.
> 
> An [AddedToken](https://huggingface.co/docs/tokenizers/v0.20.3/en/api/added-tokens) can control properties such as `single_word`, whitespace matching and normalization, but it does **not** run a language grammar, track whether a character sequence is code versus a comment, or identify the active language. The same bytes may be recognized inside a string literal or comment. This is best described as **code-oriented prioritized literal matching** , not a full parser-aware or lexer-state-aware tokenizer. That does not undermine the lightweight approach; it tells us what intervention actually took place.
> 
> There are at least three separable hypotheses here:
> 
> 1. **Boundary:** a compound operator can become a single learned representation.
> 2. **Sequence geometry:** the new match may shorten or lengthen the surrounding tokenization.
> 3. **Adaptation:** the new embeddings and SFT-updated weights may alter model behavior beyond inputs containing new IDs.
> 
> ### What activates in real benchmark text?
> 
> I measured actual emitted token IDs, rather than searching strings from the proposed lexeme list. On the retrieved Python benchmark texts the new-ID activation was sparse:
> 
> | Corpus field | Texts using a new token | Total token-count change, extended minus standard |
> | --- | --- | --- |
> | HumanEval prompt | 8/164 | 0 |
> | HumanEval reference solution | 6/164 | 0 |
> | MBPP original prompts | 2/974 | +1 |
> | MBPP original reference solutions | 23/974 | +8 |
> 
> These are **snapshot-specific totals** , not estimates for the SFT training corpus. MBPP original 974 tasks differ from the EvalPlus MBPP evaluation subset. Round-trip encode/decode checks passed for the corpus-probe texts; some matches were in comments or strings.
> 
> The **8/164 input-hit rate** is a key limit on a prompt-only explanation, not a limit on all tokenizer effects. New output tokens can be generated later, and SFT may have changed shared weights even for prompts with no new IDs. The distinction between _directly activated at inference time_ and _exposed during training_ is more informative than a binary verdict.
> 
> ### Compression is neither guaranteed nor identical to correctness
> 
> An extra token can collapse a multi-piece operator but change the segmentation of adjacent whitespace or characters. My tiny spot checks included a Rust `macro_rules!` example that shortened from **19 to 17** tokens, a JavaScript `||=` example that lengthened from **16 to 17** , and a Python comment/string example that lengthened from **22 to 23**. These are illustrative _tokenization observations_, not measured code-quality gains or statistically representative corpus results. The author’s note already discusses this trade-off; I mention it only because it makes the measurement criterion clearer.
> 
> ### Why a pretrained embedding can matter
> 
> The pretrained network previously processed `<<=` through its old constituent IDs. A new single ID is a separate learned embedding, not automatically equivalent to that original computation. Its initialization, exposure during SFT and relation to the output head can matter; the old representation may already be effective in some contexts.
> 
> The experiment already uses selective training of added token embeddings with LoRA, so that is not a missing step to suggest. [PEFT’s `trainable_token_indices` documentation](https://huggingface.co/docs/peft/main/package_reference/lora) describes selective embedding adaptation, and [its troubleshooting guide](https://huggingface.co/docs/peft/en/developer_guides/troubleshooting) discusses embedding resizing, saving and weight tying. These are useful for reproducing the setup, not evidence of an issue in the released checkpoints.
> 
> For a later causal ablation, compare initialization strategies or inspect how frequently each added ID appears in the SFT corpus. A control with the same number of **unused** random tokens would be weak; token exposure and sequence-length effects matter. These are optional experiments, not prerequisites for checking saved evaluations.
> 
> ### Output-side exposure and the fixed-weight probe
> 
> The **8/164** figure describes evaluation **inputs** , not potential new IDs in generated outputs. Autoregressive generation can encounter added tokens after the prompt, changing later predictions. Even identical decoded text can arise from different token paths. Prompt-side coverage is informative but incomplete.
> 
> B/C provides evidence that **input segmentation can change generated text at fixed weights** on selected prompts. It cannot quantify which segmentation is more accurate. A stronger check would test functional results under the same generation and extraction setup, keeping the extended output vocabulary and the nonrandom task selection explicit.

> **3. Evaluation, safe rechecking, and what would count as a stronger replication**
>
> ### Separate generation, extraction and functional evaluation
> 
> I would treat the evaluation as a pipeline with individually inspectable inputs and outputs:
> 
> ```plaintext
> model checkpoint + tokenizer + prompt + generation settings
> | 
> raw generation (record exact text, IDs and stop reason)
> |
> fixed code extraction/sanitization
> |
> versioned functional tests in an isolated execution environment
> |
> per-task outcome + aggregate Base / Base + Extra score
> 
> ```
> 
> Extraction is part of the metric: a generated correct function plus extraneous text may fail if tested raw yet pass after a pinned extraction procedure. B’s larger number of length-cap cases makes an explicit extraction rule especially important.
> 
> [EvalPlus’s CLI documentation](https://github.com/evalplus/evalplus/blob/master/docs/cli.md) explicitly distinguishes `evalplus.codegen`, `evalplus.sanitize`, and `evalplus.evaluate`; it accepts saved `samples.jsonl` containing `task_id` and a full `solution`, or a completion in its expected form. One illustrative record is:
> 
> ```json
> {"task_id": "HumanEval/0", "solution": "def has_close_elements(numbers, threshold):\n ..."}
> 
> ```
> 
> For sanitized self-contained function code, `solution` is appropriate; `completion` means a function body and follows a different contract. Record the exact prompt/signature treatment to avoid duplicating the header.
> 
> ### Base and Base + Extra are different test strictness, not different prompts
> 
> EvalPlus **Base** tests the original HumanEval cases; **Base + Extra** applies additional checks to the same tasks. These are different test strictness levels, not independent replications. Re-evaluate the **same candidate files** for both, and examine problems that pass Base but fail Extra. The analogous distinction matters for MBPP.
> 
> ### Re-evaluate the saved A/B candidates first
> 
> My saved output consists of **A\_sanitized.jsonl** and **B\_sanitized.jsonl** , derived from the 164-task generations. Using a **pinned EvalPlus source version and dataset cache** , the next check is conceptually simple:
> 
> ```bash
> # Illustrative only: run inside a suitably restricted container/VM.
> # Use the exact EvalPlus release/commit selected for the comparison,
> # with immutable input JSONL and a versioned dataset snapshot.
> evalplus.evaluate --dataset humaneval --samples A_sanitized.jsonl
> evalplus.evaluate --dataset humaneval --samples B_sanitized.jsonl
> 
> ```
> 
> Follow the [EvalPlus safety guidance](https://github.com/evalplus/evalplus) and pin evaluator code, dataset and container-image digest rather than relying on `latest`. Restrict mounts, credentials, network access, privileges and resource limits. Containerization reduces risk, but does not make untrusted code safe by itself. Preserve task-level verdicts, exceptions and timeouts rather than silently treating harness problems as model failures.
> 
> My two anomalous reference cases deserve particular care:
> 
> - `HumanEval/32`: reference candidate failed under my harness.
> - `HumanEval/139`: reference candidate timed out.
> 
> These are **independent-harness anomalies** , not confirmed defects in EvalPlus or the author’s checkpoints. Potential causes include test interpretation, environmental mismatch and timeout settings. Keep the original task IDs, compare against a pinned official run, and document rather than silently exclude any unresolved cases.
> 
> ### Match the prompt and stop contract before requesting new scores
> 
> If an official re-evaluation of existing candidates leaves a meaningful discrepancy with the author’s results, check the **original generation contract** before changing the tokenizer or retraining. I would record:
> 
> | Item | Why it matters |
> | --- | --- |
> | Model/tokenizer revision and adapter | A vocabulary or weight mismatch changes the system |
> | vLLM/Transformers versions and decoding | Can produce different candidates |
> | Prompt format, BOS/EOS, stop rules | Base/chat mode, truncation and termination differ |
> | New-token cap and raw output | Overgeneration affects extraction |
> | Sanitizer/test revision, dataset hash, timeouts | Can change functional verdicts |
> 
> The [EvalPlus README](https://github.com/evalplus/evalplus/blob/master/README.md) notes that a base model with a tokenizer `chat_template` may require `--force-base-prompt` when EvalPlus would otherwise infer chat-style prompting. The experiment note already describes forcing a base prompt; I mention it as **a reproducibility condition for an independent comparison** , not as an overlooked step.
> 
> ### What would make the result reusable?
> 
> A reproducibility record can be small: model/tokenizer revision, framework versions, exact prompts/stopping settings, source JSONL, sanitizer/evaluator commit and dataset hash, plus per-task verdicts. One greedy sample per problem characterizes that specific run, not seed or sampling variability.

> **4. Choosing the next experiment according to the question**
>
> The next step depends on the intended goal:
> 
> - **Practical retrofit:** does a small vocabulary extension plus SFT improve code quality at acceptable cost? A reproducible systems-level gain may suffice.
> - **Mechanistic attribution:** did alignment with lexical units itself cause the gain? This needs more tightly controlled tokenizer and training contrasts, which are not mandatory for the first goal.
> 
> A decision tree for interpreting the existing data might look like this:
> 
> ```plaintext
> Does the observed A/B direction survive one pinned official EvalPlus run?
> 
> Yes:
> Inspect the paired wins and losses by task.
> New IDs often appear in the relevant input/output?
> Try a targeted fixed-weight lexical contrast.
> Improvements occur mostly without input new-ID hits?
> Inspect output-side exposure and checkpoint/SFT adaptation.
> Base improves but Base + Extra does not?
> Inspect edge-case failures on the same generated candidates.
> 
> No, or the magnitude changes greatly:
> First reconcile candidate extraction, test versions,
> timeouts, prompts and inference conditions.
> Do not immediately spend GPU time retraining.
> 
> After the basic evaluation is stable:
> Is the goal a better deployable SWE model?
> Compare quality, cost, long-context behavior and retention.
> Is the goal evidence for lexical-boundary causality?
> Design the smallest meaningful vocabulary/training ablation.
> 
> ```
> 
> ### What a matched SFT recipe does and does not control
> 
> Both variants use a comparable SFT recipe, which is a useful starting point. But when the tokenizer changes, ‘same recipe’ does not necessarily mean identical effective token exposure, embedding-update frequency, sequence packing, amount of information per token, or optimizer trajectory. A fixed number of optimizer steps may process different amounts of actual source text. Conversely, matching raw source bytes does not guarantee identical token budgets or wall time. Which match is fair depends on whether one is testing engineering cost, sample efficiency, or an isolated representation effect.
> 
> If the observed gain persists after standardized evaluation and it is important to attribute it to the **choice of lexemes** , one optional controlled set would compare:
> 
> 1. Standard-tokenizer SFT baseline.
> 2. Lexical AddedToken SFT under matched conditions.
> 3. If feasible, a **frequency/exposure-matched** alternative-token control.
> 4. Optionally, an embedding-initialization control or repeated fine-tuning seeds.
> 
> The same number of random extra IDs is a weak control if they are rarely used. Matched source text and matched optimizer steps are also not always the same as equal effective token exposure. Controlled retraining is useful **after** the saved-output comparison indicates what needs explaining.
> 
> ### Fixed weights, targeted prompts, and semantics-preserving rewrites
> 
> A targeted B/C extension would hold the **extended checkpoint fixed** , change only the input encoding where it genuinely differs, and evaluate correctness under identical decoding and extraction. Include executable operators as well as the same byte strings inside comments or literals, so string matching is not mistaken for lexer awareness.
> 
> One could also use **semantics-preserving rewrites** such as safe renaming or formatting changes, then compare tokenization and functional predictions. This resembles the question in [TokDrift](https://aclanthology.org/2026.acl-long.2199/). Every rewrite needs a language-aware validity check; macros, precedence and whitespace-sensitive constructs can make superficially similar programs inequivalent.

> **5. Related work, other languages, and the SWE-agent objective**
>
> ### Nearby papers and tooling
> 
> These are useful as **background or method references** , not as evidence that the reported improvement has been independently explained:
> 
> - [Dagan et al., _Getting the most out of your tokenizer for pre-training and domain adaptation_ (ICML 2024)](https://proceedings.mlr.press/v235/dagan24a.html): tokenizer design, throughput/context and switching within pretrained code LMs; a different scale/intervention.
> - [_TokDrift_ (ACL 2026)](https://aclanthology.org/2026.acl-long.2199/): behavior under semantics-preserving tokenization-changing rewrites; not a test of this retrofit.
> - [_Tokenization Is More Than Compression_ (EMNLP 2024)](https://aclanthology.org/2024.emnlp-main.40/): why compression and quality should not be conflated.
> - [AddedToken API](https://huggingface.co/docs/tokenizers/v0.20.3/en/api/added-tokens), [PEFT selective tokens](https://huggingface.co/docs/peft/main/package_reference/lora), [EvalPlus CLI](https://github.com/evalplus/evalplus/blob/master/docs/cli.md): implementation and evaluation contracts.
> - [MultiPL-E](https://github.com/nuprl/MultiPL-E): one possible route to multilingual functional tests, subject to pinned toolchains and validation.
> 
> None is a direct independent replication. Their value is making the next comparison easier to formulate and reproduce. A reading list should not substitute for the actual task-level records.
> 
> ### HumanEval, MBPP, and the meaning of generalization
> 
> HumanEval and MBPP mainly ask for relatively small Python functions; they differ in task distribution, difficulty and test design. The reported result improves on HumanEval while MBPP is slightly lower. That is a useful _boundary on the current observation_, not grounds to declare either the method or the model generally better or worse. It could reflect different problem mixes, different token exposure, learned specialization, noise or evaluation conditions. Task-level paired outcomes would help discriminate among these possibilities more cheaply than speculating from aggregate scores.
> 
> The lexeme list reaches beyond Python, but HumanEval/MBPP do not directly establish gains in Rust, C++ or Go. A language-specific evaluation should first measure whether new IDs activate in its test texts, then use pinned compilers/interpreters and **functional** checks. A broad test with almost no activation and a targeted lexical stress test answer different questions.
> 
> There is also a distinction between a tokenizer-friendly **synthetic diagnostic** and an actual **general-purpose benchmark**. A mini-suite intentionally packed with new operators can answer ‘can the model use this representation in favorable cases?’ It cannot estimate how often real repositories benefit. A broad functional benchmark answers a different question, but may activate almost none of the new tokens. Both can be useful if labeled honestly and compared with controls.
> 
> ### From function completion to a personal SWE agent
> 
> The stated goal is a personal SWE agent, not just isolated function completion. Later evaluations might cover repository edits, reading tests, repairing a failure and long-context multi-file work. Tool use, prompting and context management also affect those outcomes, so HumanEval should not be treated as proof of agent-level generalization.
> 
> For a practical retrofit, measure correctness together with actual latency, memory, token exposure, adapter/vocabulary compatibility and possible regressions outside the target code distribution. A useful post-hoc extension need not be the ideal vocabulary for pretraining from scratch; those are different engineering objectives.
> 
> The most reusable artifact for future readers would be a **per-task table** linked to frozen prompts, raw output, sanitized candidate, verdicts and revisions. It lets someone test alternate explanations without generating everything again, and keeps author-reported numbers distinct from this independent probe.

## Where this leaves the experiment

I would treat the independent A/B result as **a modest positive signal for the released extended checkpoint under one non-official functional check** , not as an official pass@1 reproduction or proof that lexical boundaries alone caused the difference. The sparse direct prompt-side activation and the fixed-weight B/C changes make the _mechanism_ particularly interesting: the training-time and output-side effects may matter just as much as the new prompt boundaries.

The clearest low-cost next step remains **one pinned official EvalPlus re-evaluation of the archived A/B candidates, followed by task-level analysis of wins, losses, lexical exposure and stopping behavior**. That can strengthen the practical finding without obliging anyone to start another expensive SFT. If the goal later becomes mechanistic attribution or multilingual SWE-agent performance, the matching ablations and evaluation choices can follow from what those saved results actually show.

---

<div class="post-metadata">

**Author:** ![SamLegion888](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@SamLegion888](https://discuss.huggingface.co/u/SamLegion888)\
**Post date:** [October 28, 2026, 8:45pm UTC](https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957/3 "2026-10-28T20:45:33Z")

</div>

It works [Albert Gu on X: "I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling… / X](https://x.com/_albertgu/status/2115502110578290900).

---

<div class="post-metadata">

**Author:** ![SamLegion888](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@SamLegion888](https://discuss.huggingface.co/u/SamLegion888)\
**Post date:** [October 28, 2026, 8:51pm UTC](https://discuss.huggingface.co/t/model-can-generalize-code-better-with-programming-language-aware-tokenization/190957/4 "2026-10-28T20:51:16Z")

</div>

To be clear, I did a study on how preserving lexical tokens can effect downstream coding performance. I was not claiming or providing proves.