Model card: add Decision Index 0.3 public results (self-reported)

MODEL_CARD_zh.md

@@ -93,6 +93,76 @@ Jue 不适合用于自由文本生成、单独拦截或执行操作、法律意
 
 在 131 个外部任务的相同题目上，Jue-Flash 有 14 个任务高于 Jue，4 个持平（见技术报告 5.2 节）。
 
+### Decision Index
+
+[Decision Index](https://github.com/apolinario/decision-index) 是面向决策模型的通用基准，大部分内容和安全无关。我们在两个模型上跑了它的公开套件（0.3 版，五个领域共 37 个基准）。分数都按随机水平校正过，0 相当于瞎猜，100 是全对。
+
+| 基准 | Jue | Jue-Flash |
+|---|---|---|
+| 公开指数 | **32.52** | **6.51** |
+| 知识与推理 | 15.2 | 2.6 |
+| 语言理解 | 31.0 | 1.5 |
+| 检索与分类 | 40.1 | 11.7 |
+| 工具与自动化 | 61.0 | 15.0 |
+| 艺术与人类判断 | 14.1 | 3.7 |
+
+<details>
+<summary>全部 37 个基准</summary>
+
+| 基准 | Jue | Jue-Flash |
+|---|---|---|
+| 知识与推理: GPQA Diamond | 10.9 | 1.4 |
+| 知识与推理: GSM8K | 8.0 | 0.1 |
+| 知识与推理: ChessBench | 0.3 | 0.0 |
+| 知识与推理: MuSR | 29.4 | 17.5 |
+| 知识与推理: SATA-Bench | 16.6 | 0.0 |
+| 知识与推理: CRUXEval | 14.0 | 0.0 |
+| 知识与推理: CLadder | 15.9 | 0.2 |
+| 知识与推理: HLE | 0.0 | 0.0 |
+| 知识与推理: MMLU-Pro | 22.9 | 3.7 |
+| 知识与推理: BBH fixed-option tasks | 37.2 | 5.9 |
+| 知识与推理: WinoGrande | 10.7 | 0.0 |
+| 语言理解: ContractNLI | 41.9 | 0.1 |
+| 语言理解: ANLI | 20.1 | 0.0 |
+| 语言理解: HellaSwag | 40.6 | 11.8 |
+| 语言理解: ACOS | 3.8 | 0.0 |
+| 语言理解: FinEntity | 59.3 | 0.0 |
+| 语言理解: iSarcasmEval | 1.1 | 0.0 |
+| 语言理解: VAST | 31.2 | 0.0 |
+| 语言理解: NLI4CT | 47.1 | 0.0 |
+| 语言理解: RAGTruth | 33.9 | 0.0 |
+| 检索与分类: BANKING77 | 65.4 | 41.1 |
+| 检索与分类: CLINC150+OOS | 72.7 | 21.6 |
+| 检索与分类: BRIGHT | 33.0 | 0.0 |
+| 检索与分类: Amazon ESCI | 14.3 | 1.8 |
+| 检索与分类: PhishNChips | 18.4 | 0.0 |
+| 检索与分类: HoVer | 26.7 | 0.2 |
+| 工具与自动化: BFCL | 92.6 | 17.4 |
+| 工具与自动化: ToolRet | 60.9 | 8.6 |
+| 工具与自动化: API-Bank | 61.9 | 33.8 |
+| 工具与自动化: Home appliance simulator | 9.1 | 0.0 |
+| 工具与自动化: When2Call MCQ | 74.1 | 11.0 |
+| 艺术与人类判断: BPoMP | 34.2 | 0.0 |
+| 艺术与人类判断: Humicroedit | 8.0 | 0.0 |
+| 艺术与人类判断: POP909-CL | 0.8 | 0.6 |
+| 艺术与人类判断: cfcolor | 4.3 | 0.0 |
+| 艺术与人类判断: Habermas Machine | 6.9 | 0.0 |
+| 艺术与人类判断: New Yorker caption matching | 30.2 | 21.4 |
+
+</details>
+
+作为参照，工具包里公布的部分模型公开指数是：Torchcast Decision 27B 65.10，Jev 57.96，JPT-9B 46.96，JPT-0.8B 19.37。
+
+两个模型都不是为这套基准做的，分数也反映了这一点。DI 0.2 名单上的基准没有进入 Jue 的训练数据，不过 0.3 新增的基准我们没有逐条核对。开发 Jue 时我们一直在看 BFCL、When2Call 和 API-Bank 的分数，所以这三项不算盲测。Jue 在工具调用和意图分类上表现不错，知识类和艺术类的不少基准接近随机水平。Jue-Flash 在安全领域之外几乎没有判别能力：很多基准上它对所有题目都给出差不多的答案（HoVer 的 4,000 条陈述里有 3,996 条判为 SUPPORTED），所以很多分数是 0 或接近 0。不要把它当通用决策模型用。
+
+这些结果是我们自测的，没有上榜。关于这次运行，有几点需要说明：
+
+- 140,620 个请求全部得到回答，覆盖率 100%。
+- 运行环境是两张 NVIDIA T4，不是榜单的参考 GPU。延迟中位数：Jue（fp16）132 ms，Jue-Flash（fp32）72 ms。
+- 套件是我们从公开来源重建的，并按工具包 issue #274 的说明撤销了一次提交，因为它在冻结后改动了构建结果。撤销后所有校验值都与官方一致。
+- Jue 的运行中途因为我们自己脚本的 bug 停过一次，之后用相同的权重和设置续跑，没有请求被重复回答。
+- 榜单的主排名用 Full score，来自只有维护者能运行的私有测试，这个分数我们没有。
+
 ## 5. 局限
 
 1. 发布标准：最后一轮训练改善了工具调用判断和几个较新的安全领域，但事先定下的八条标准只达到三条。这是本项目的最终模型，并没有通过全部检查。

README.md

@@ -239,6 +239,16 @@ model-index:
     - type: f1
       value: 0.836
       name: macro-F1
+  - task:
+      type: text-classification
+      name: General decision
+    dataset:
+      type: decision-index
+      name: Decision Index 0.3 (public index)
+    metrics:
+    - type: score
+      value: 32.52
+      name: Public index, self-reported
 ---
 
 # Jue
@@ -353,6 +363,76 @@ We converted each benchmark into typed questions and took the highest-probabilit
 
 On the same items of 131 external tasks, Jue-Flash scores higher than Jue on 14 and ties on 4 (technical report, section 5.2).
 
+### Decision Index
+
+[Decision Index](https://github.com/apolinario/decision-index) is a general benchmark for decision models, and most of it has nothing to do with security. We ran its public suite (edition 0.3: 37 benchmarks in five areas) on both models. Each score is corrected for chance, so 0 means random guessing and 100 means perfect.
+
+| Benchmark | Jue | Jue-Flash |
+|---|---|---|
+| Public index | **32.52** | **6.51** |
+| Knowledge and reasoning | 15.2 | 2.6 |
+| Language understanding | 31.0 | 1.5 |
+| Retrieval and classification | 40.1 | 11.7 |
+| Tools and automation | 61.0 | 15.0 |
+| Arts and human judgment | 14.1 | 3.7 |
+
+<details>
+<summary>All 37 benchmarks</summary>
+
+| Benchmark | Jue | Jue-Flash |
+|---|---|---|
+| Knowledge and reasoning: GPQA Diamond | 10.9 | 1.4 |
+| Knowledge and reasoning: GSM8K | 8.0 | 0.1 |
+| Knowledge and reasoning: ChessBench | 0.3 | 0.0 |
+| Knowledge and reasoning: MuSR | 29.4 | 17.5 |
+| Knowledge and reasoning: SATA-Bench | 16.6 | 0.0 |
+| Knowledge and reasoning: CRUXEval | 14.0 | 0.0 |
+| Knowledge and reasoning: CLadder | 15.9 | 0.2 |
+| Knowledge and reasoning: HLE | 0.0 | 0.0 |
+| Knowledge and reasoning: MMLU-Pro | 22.9 | 3.7 |
+| Knowledge and reasoning: BBH fixed-option tasks | 37.2 | 5.9 |
+| Knowledge and reasoning: WinoGrande | 10.7 | 0.0 |
+| Language understanding: ContractNLI | 41.9 | 0.1 |
+| Language understanding: ANLI | 20.1 | 0.0 |
+| Language understanding: HellaSwag | 40.6 | 11.8 |
+| Language understanding: ACOS | 3.8 | 0.0 |
+| Language understanding: FinEntity | 59.3 | 0.0 |
+| Language understanding: iSarcasmEval | 1.1 | 0.0 |
+| Language understanding: VAST | 31.2 | 0.0 |
+| Language understanding: NLI4CT | 47.1 | 0.0 |
+| Language understanding: RAGTruth | 33.9 | 0.0 |
+| Retrieval and classification: BANKING77 | 65.4 | 41.1 |
+| Retrieval and classification: CLINC150+OOS | 72.7 | 21.6 |
+| Retrieval and classification: BRIGHT | 33.0 | 0.0 |
+| Retrieval and classification: Amazon ESCI | 14.3 | 1.8 |
+| Retrieval and classification: PhishNChips | 18.4 | 0.0 |
+| Retrieval and classification: HoVer | 26.7 | 0.2 |
+| Tools and automation: BFCL | 92.6 | 17.4 |
+| Tools and automation: ToolRet | 60.9 | 8.6 |
+| Tools and automation: API-Bank | 61.9 | 33.8 |
+| Tools and automation: Home appliance simulator | 9.1 | 0.0 |
+| Tools and automation: When2Call MCQ | 74.1 | 11.0 |
+| Arts and human judgment: BPoMP | 34.2 | 0.0 |
+| Arts and human judgment: Humicroedit | 8.0 | 0.0 |
+| Arts and human judgment: POP909-CL | 0.8 | 0.6 |
+| Arts and human judgment: cfcolor | 4.3 | 0.0 |
+| Arts and human judgment: Habermas Machine | 6.9 | 0.0 |
+| Arts and human judgment: New Yorker caption matching | 30.2 | 21.4 |
+
+</details>
+
+For scale, these are public indexes published with the kit: Torchcast Decision 27B 65.10, Jev 57.96, JPT-9B 46.96, JPT-0.8B 19.37.
+
+Neither model was built for this suite, and the scores show it. The benchmarks on the Decision Index 0.2 list were kept out of Jue's training data, though we didn't check the ones added in 0.3 item by item. We did track BFCL, When2Call and API-Bank while developing Jue, so its scores on those three aren't blind. Jue does well on tool use and intent classification and is close to chance on much of the knowledge and arts material. Jue-Flash has little skill outside security: on many benchmarks it gives nearly the same answer to every question (3,996 of 4,000 HoVer claims come back SUPPORTED), which is why so many of its scores are at or near 0. Don't use it as a general-purpose decision model.
+
+These results are self-reported and are not on the board. Things to know about the runs:
+
+- Every one of the 140,620 requests was answered, so coverage is 100%.
+- We ran on two NVIDIA T4s, not the board's reference GPU. Median latency was 132 ms for Jue (fp16) and 72 ms for Jue-Flash (fp32).
+- We rebuilt the suite from public sources and reverted one kit commit, as described in kit issue #274, because that commit changes the build after the freeze. With the revert, all checksums match the official ones.
+- The Jue run stopped partway because of a bug in our own script, and we resumed it with the same weights and settings. No request was answered twice.
+- The board's main ranking uses a Full score from private tests that only the maintainers can run. We don't have that number.
+
 ## 5. Limitations
 
 1. Release criteria: the final training run improved tool-use decisions and several of the newer security domains, but met only three of the eight criteria fixed beforehand. Jue is the last model from this project, and it did not pass every check.