seb129225/Lenormand-HumOmni-Track2

Lenormand HumOmni Track 2 System

This repository contains the reproducibility materials for Team Lenormand in HumOmni Track 2 ProactivEval.

Base Repository

Our system is built on top of the public MMDuet2 repository:

• MMDuet2: https://github.com/yellow-binary-tree/MMDuet2

We use MMDuet2 as the streaming video-language backbone and modify the inference and response-control pipeline for HumOmni Track 2.

Overview

The final system is a multi-stage agentic inference harness rather than a single-pass VLM generation pipeline. We found that the streaming VLM should not simultaneously act as the visual reader, memory controller, response gate, and final answer writer. Therefore, we decouple the system into several functional modules.

The core idea is to treat the VLM as a mostly stateless visual reader, while external modules handle timing, answer selection, memory/state tracking, and post-processing.

System Components

1. V36 Reader / Timing Agent

We use the MMDuet2-style streaming pipeline to propose candidate response times. The raw V36-style output is not directly used as the final submission because it may contain long responses, repeated content, and late-stage narrative contamination.

2. Causal Visual Answer Regeneration

For each candidate response time t, the system regenerates a concise answer using only visual frames up to t. No future frames are used when generating a response at time t.

3. QEA Selection Module

We include a lightweight Question–Evidence–Answer selection module to choose concise and question-relevant answers from candidate generations. This module compares the question, candidate answer, and causal visual evidence. It does not use hidden labels, ground-truth reply spans, or leaderboard feedback.

4. External Memory Agent

The external memory module stores only question-relevant textual states rather than raw visual history. It checks whether a candidate turn is new, relevant, non-redundant, and not contaminated by previous narrative outputs.

This module is designed to reduce answer self-contamination in streaming generation. The VLM is used mainly for visual reading, while memory and response control are handled outside the autoregressive dialogue context.

5. Clean-Light-Plus Post-processing

The final global post-processing step removes only obvious generation artifacts, such as:

• NO REPLY
• NS REPLY
• NSWYR
• REPLYING:
• no image displayed
• No active movement occurs
• No actions take place
• No description available
• No response available
• No additional context
• No additional elements

This filtering is global and rule-based. It does not use ground-truth labels, hidden reply spans, leaderboard feedback, or video-specific manual edits.

Final Submission Variant

The final prediction file submitted to the official Google Drive is generated by:

memory_agent_top6_clean_light_plus

Final sanity check:

N_records = 500
unique_question_ids = 500
duplicate_question_ids = 0
empty_outputs = 0
avg_turns_per_video = 3.03
max_turns_per_video = 6
time_min = 4.0
time_max = 30.0
chronological_per_video = True
all turns have time and content = True

Base Models and Checkpoints

We do not duplicate large public checkpoints in this repository. The system uses public base resources, including MMDuet2-style streaming components and Qwen2.5-VL-based vision-language inference components.

This repository provides the modified inference harness, post-processing code, configuration notes, and reproducibility notebook needed to regenerate the submitted system output.

Modified Components

Compared with the public MMDuet2-style baseline, our submitted system modifies the response-control pipeline in the following ways:

1. Deterministic decoding and concise generation

   • We use deterministic decoding for stable outputs.
   • We favor concise visual answers instead of long dialogue-style responses.

2. KV-cache / writeback control

   • We avoid feeding long generated answers back into the streaming context.
   • We use short summary/marker-style writeback to reduce narrative self-contamination.

3. Timestamp correction and causal frame filtering

   • Each response at time t is generated only from frames up to t.
   • The pipeline corrects timestamp alignment for frame sampling.

4. Per-turn answer regeneration

   • V36-style outputs are used as timing proposals.
   • Final answers are regenerated at candidate times using causal visual context.

5. QEA selection and targeted fallback

   • Candidate answers are selected using a lightweight QEA-style selection module.
   • Targeted fallback generation is used to rescue low-confidence or generic answers.

6. External Memory Agent

   • The memory/state module filters redundant, irrelevant, or contaminated turns.
   • It keeps only question-relevant textual states instead of storing the full visual history.

7. Clean-Light-Plus filtering

   • Obvious generation artifacts are removed using a global rule-based filter.
   • No test labels, hidden reply spans, leaderboard feedback, or sample-specific manual edits are used.

Repository Files

The repository includes:

README.md
requirements.txt
HumOmni_V4_FINAL_K1_MEMORY_AGENT_TOP6_CLEAN_PLUS_FULL500.ipynb
patch_mmd duet2_inference.py
clean_light_plus_cell.py
submission_generation_notes.md
export_modified_inference_notes.md

If a fully patched local inference.py is exported from the Colab runtime, it can also be uploaded as modified_inference.py for convenience.

Environment

The system was run on Google Colab Pro with an NVIDIA A100 GPU.

Main dependencies are listed in requirements.txt. The environment used the public MMDuet2 codebase with the provided inference patch and a Qwen2.5-VL-based visual-language inference setup.

Reproducibility

To reproduce the final submission:

1. Prepare the official HumOmni Track 2 Phase 1 test data.
2. Clone or prepare the public MMDuet2 repository.
3. Apply the provided inference patch / modified inference script.
4. Run HumOmni_V4_FINAL_K1_MEMORY_AGENT_TOP6_CLEAN_PLUS_FULL500.ipynb.
5. Generate the memory_agent_top6 output.
6. Run the Clean-Light-Plus post-processing cell.
7. Verify the final JSONL format using the final audit cell.

The official prediction JSONL is submitted separately through the official Google Drive prediction folder.

Notes on Fairness and Test Usage

The submitted system does not use hidden test labels, ground-truth reply spans, or manual video-specific edits. The final filtering rules are global, deterministic, and applied uniformly across all test samples.