Arm/neudecide-int4-executorch

NeuDecide Q4 - ExecuTorch

This package contains a per-channel Q4 conversion of neuphonic/neudecide for English spoken-request tool calling with ExecuTorch and XNNPACK. It generates JSON tool calls with arguments directly from audio, without an intermediate transcript.

✨ Key Highlights

• Incredibly fast tool calling from audio — on one Arm® CPU inference thread with structured tool-call output:
  • Median completed request: 77.6 ms on a MacBook Pro® computer with an M4 Pro chip and 164.4 ms on an Android™ Vivo X300 smartphone.
  • RTFx: 21.63× on the MacBook Pro® computer and 10.21× on the Vivo X300 smartphone.
• Tiny model for tool calling — the four ExecuTorch model files total 99.14 MiB.

📦 Model Details

Model Description

NeuDecide generates calls to configured assistant tools from an English spoken request. The package provides ExecuTorch .pte model files, a SentencePiece tokenizer, and a JSON file with eight assistant tool definitions, including the upstream example's cleanRoom action.

• Developed by: Neuphonic.
• Model type: English spoken-language understanding and audio tool calling.
• Base model: neuphonic/neudecide, revision 892237127c492b51d97a1824c459854bcd6a4519.
• License: Apache-2.0; see LICENSE.
• Package format: ExecuTorch .pte files and a SentencePiece tokenizer.
• Runtime model size: 99.14 MiB across four .pte files.

Model Sources

• Upstream model.
• Upstream implementation.

🚀 Get Started with the Model

Run with Compute Flow

Follow these steps to run NeuDecide Q4 - ExecuTorch on macOS® with Apple Silicon or on Linux® with an Arm®-based CPU, such as a Raspberry Pi® 5.

Step 1: Clone Compute Flow and set up the virtual environment

git clone https://github.com/ARM-software/ComputeFlow.git && cd ComputeFlow

uv sync && source .venv/bin/activate

Step 2: Install the brick and model

compute-flow update
compute-flow model install audio_tool_calling:neu-decide neudecide-en
compute-flow build native

Step 3: Run the Python CLI example

python -m cli_examples.audio_tool_calling_cli \
  -m @manifest/neu-decide/neudecide-en/neudecide_manifest.json \
  -a @sample/command.wav \
  -t 1

The example extracts a tool call from the bundled samples/command.wav recording. Replace -a with a path to your own WAV file.

📊 Quality evaluation

This package applies per-channel Q4 quantization to the released groupwise Q4 model. We compared its tool selections with the official Q4 implementation on spoken requests using the same audio and tool definitions. In these limited checks, we did not observe a substantial difference in overall tool calling. These checks do not establish general quality or evaluate tool arguments. We recommend evaluating the model on representative audio and tool definitions for the intended application.

🎯 Performance evaluation

The model artifacts were measured with structured tool-call output on a MacBook Pro® computer with an M4 Pro chip running macOS® 26.6.2 and a Vivo X300 smartphone running Android™ 16 under the following conditions:

• One Arm® CPU inference thread.
• 5 warmup requests.
• 30 measured requests with a 60-second pause between runs.
• The MacBook Pro® computer was AC powered and kept awake; the Vivo X300 smartphone was USB powered and its screen was kept on.

The following methodology and definitions were used:

• Input audio: The packaged upstream command.wav sample, a 1.679-second English vacuum-cleaning request at 16 kHz, with eight tool definitions and a 128-token output limit. The completed result was cleanRoom in every measured run on both devices.
• End-to-end latency: The summed execution time through a completed request. It includes preprocessing, tool tokenization, model execution, constrained decoding, and result collection; it excludes model setup, WAV file loading, and state reset.
• RTFx: Input audio duration divided by median latency; higher values are faster.
• Average memory: Mean process RSS sampled every 0.2 seconds across setup and inference.

Metric	MacBook Pro® (M4 Pro)	Android™ Vivo X300
RTFx	21.63×	10.21×
End-to-end latency	77.6 ms	164.4 ms
Average memory	347.9 MiB	419.7 MiB

The reported values describe this fixed utterance and tool list; other audio and tool schemas can change the execution time and output.

⚠️ Known Limitations

The model can produce a tool call for silence or noise. Applications should check inputs and generated arguments before executing tools.

🛠️ Technical Specifications

Objective

Generate structured calls to configured assistant tools from an English spoken request.

Runtime Architecture

Inference uses ExecuTorch with XNNPACK and SentencePiece tokenization.

Precision and Quantization

• Linear weights use signed per-channel Q4 reconstructed from the released groupwise Q4 weights; linear activations use dynamic INT8 quantization with FP32 channel scales.
• Embeddings preserve the source packed Q4 codes and FP32 group scales.
• Source Q8 convolution weights are dequantized to FP32. Remaining floating-point operations and public audio inputs use FP32.

Input Specification

Input	Format	Limits
Spoken request	Mono English audio at 16 kHz	The leading 30 seconds are processed; resample other rates to 16 kHz.

Output Specification

Output	Format	Meaning
calls	One UTF-8 JSON string	An array of tool calls in generation order. Each call has a name string and an arguments object.

An empty completed response produces []. If decoding stops before the full array is complete, the brick returns [] without exposing partial calls.

Manifest Configuration

The manifest's max_output_tokens setting defaults to 128 and accepts values from 2 to 128. A lower limit can truncate a response and produce [].

assets/tools.json defines the available tools. Changing this file requires reinitializing inference. Tool names must be unique and cannot be no_tool. The complete tool definitions must fit within 1,536 tokens.

Repository Contents

• *.pte — four ExecuTorch model artifacts.
• tokenizer.model — upstream SentencePiece tokenizer.
• neudecide_manifest.json — model and runtime asset configuration.
• assets/tools.json and assets/token_bytes.json — runtime assets.
• samples/command.wav — upstream vacuum-cleaning request; with the packaged tools, its expected tool name is cleanRoom.
• metadata.yaml — package metadata.
• SHA256SUMS — model-bundle checksums for reproducibility.

🗂️ Model and Asset Origin

• Model files: Converted from Neuphonic's Apache-2.0 Q4 release.
• Tokenizer: From the pinned upstream model revision.
• Sample audio: command.wav from the upstream implementation, distributed as samples/command.wav.
• Tool definitions: Prepared for this package; cleanRoom follows the pinned upstream example.

🔐 Checksums

SHA256SUMS was generated by recursively hashing every regular file in the model bundle, including files in subdirectories, except the generated root SHA256SUMS and paths with a dotfile component.

From the model bundle root, verify the checked-out files with:

shasum -a 256 -c SHA256SUMS