Multimodal Data & Benchmarks: August 2026 Progress Brief

August 2026's multimodal benchmark literature is unusually convergent. Rather than announcing new tasks, it rebuilds evaluation around the specific step where current models break: an unanswerable clip a model will not flag, a second image it cannot keep straight, a polygon it cannot trace, a multi-hop chart computation it cannot finish, a fact it can retrieve but not deploy. Across the thirty benchmarks deep-read for this issue, the recurring result is that headline accuracy overstates competence as soon as the scoring is tightened at that step — and several papers make the tightening itself the contribution.

Window and coverage: 1–31 August 2026, by recorded v1 submission date. Sources: this library's arXiv mirror, categories cs.CV/CL/MM/AI/LG/SD. The corpus is a screened working set of 428 multimodal understanding and unified-model papers, of which 113 were classified as benchmarks or datasets; thirty of those were read in full for this issue, which is wider than a usual monthly brief. No earlier issue exists, so this report is the series baseline.

Figure 1. Theme map of the month's thirty benchmark deep reads, grouped into the five themes below; the numbers are reference citations.

1 Video and long-horizon understanding moves to agentic, object-centric and intent-level tests

The month's video benchmarks stop asking what happens and start testing how evidence is gathered, whether a chain of sub-claims holds, and what a film-maker meant. Two of them also tighten scoring in a way that makes partial competence uncreditable.

1.1 VideoGAIA: agentic video understanding under one shared harness

VideoGAIA targets agentic video understanding — using a video as one source of evidence inside a multi-turn, tool-using task — arguing that single-turn video QA is close to saturated while a model that must decide what to look up and then fetch it is not. Its 271 tasks are distilled from 100K public videos by a pipeline that deliberately hides the answer from the video, so the model must recognise an entity visually and retrieve the fact from the web (Figure 2). Twenty frontier MLLMs are run through one identical ReAct harness with three tools and a 40-step cap; no model reaches 60% pass@1, and the best (Seed2.0-Pro, 58.30%) leads the floor (Claude-Opus-4.7, 34.69%) by 24 points [1]. The diagnostic finding matters more than the ranking: across 615 wrong trajectories, video perception is the top error cause for every model (36.8–48.0%), ahead of web retrieval (20.3–31.0%) — and progress across model generations is not monotonic, with two Claude Opus releases scoring below an earlier one [1].

Figure 2. VideoGAIA's benchmark construction: data collection, task construction, five model-filtering gates, and human annotation, yielding 271 verified tasks (L1 62 / L2 209) over six domains (Figure 3 of [1]; reproduced from the original paper, CC BY 4.0).

1.2 STRAND: refusing to credit a right answer reached by the wrong path

STRAND's contribution is a scoring change rather than a harder task. Its 88 videos (mean 183 s, four domains) carry 977 target questions and 2,516 supporting sub-questions, all binary; the primary metric counts a target correct only if the target and every one of its sub-questions is correct, over a fixed denominator [2]. The authors' stated contrast is with long-video benchmarks that lengthen the horizon or chain evidence but still grade only the final answer. The two metrics diverge sharply in their own Table 1: no end-to-end model exceeds 68.4% target accuracy, and joint faithful accuracy sits far below it for every frontier model (Gemini-3.1-Pro 61.4 to 38.9; GPT-5 32.5 to 20.0; Claude-4.6-Sonnet 48.3 to 30.2) [2]. Two design choices are worth noting: questions are generated from a human-verified symbolic world model by a language model that never sees the video, and bounding boxes are deliberately excluded because the targeted failure is symbolic binding drift rather than localisation [2]. The videos come from Wikimedia Commons and are redistributed with the annotations under CC-BY-NC 4.0, which the paper presents as fixing the link-rot problem of URL-only benchmarks [2].

1.3 TAKE 85: directorial intent, measured through a lossy transcription

TAKE 85 asks what a film-maker intended, using 398 short films (about 86 hours) with twelve expert-checked intent questions each, 4,776 in total; the strongest model scores 58/100 on the fine-grained "Specific" subset [3]. The caveat is structural rather than incidental: models never receive the films. Inputs are captions from Llama-3.2-Vision-11B, audio descriptions from AudioFlamingo3 over three 30-second chunks, Whisper subtitles and twenty keyframes, so the benchmark measures reasoning over a shared lossy transcription, with upstream extraction errors held fixed across models [3]. The scored conclusion — that even the strongest MLLMs struggle with communicative purpose in film — is about grounded audio-visual perception only as far as that transcription preserves it.

Table 1 compares the four video-focused benchmarks.

Table 1. The month's video-facing benchmarks on shared axes (scale, novelty, headline result); settings as reported in each paper. Cell values are the authors' reported numbers unless marked otherwise.

Takeaways

• The competitive frontier in video evaluation has moved from task difficulty to what is credited: STRAND's joint metric and VideoGAIA's tool-necessity filtering both remove credit for right answers reached without the intended evidence path [1, 2].
• Video perception, not retrieval or answer formulation, is the dominant failure mode of agentic video systems (36.8–48.0% of errors for every model tested) [1].
• At least one flagship benchmark scores intent over captions and audio descriptions rather than pixels, so its numbers bound grounded perception only as far as that transcription survives [3].

2 Precise perception: documents, charts, handwriting and geometric grounding

A second cluster attacks output fidelity: can a model compute across charts, reason over OCR-grounded evidence, read real handwriting, and emit geometry that actually follows a boundary. All three featured papers find a large gap between recognising the referent and producing the required output.

2.1 LongChart VQA: multi-hop computation across many charts

LongChart VQA tests multi-chart question answering built from a latent-graph (DAG) synthesis pipeline: 557 images and 2,876 questions, averaging 6.5 images, 31.2 questions and 716.8 data points per instance under its main split [5]. Two settings isolate where the difficulty lives. With every intermediate chart visualised ("Extended Context") models do markedly better than when only a minimal chart set is supplied ("In-Depth"), and hops hurt more than length: In-Depth accuracy falls to 33.8% at five hops and 15.3% beyond six, against 53.7% and 46.1% in Extended Context [5]. Grounding fails outright — all models produce negative mean distance-IoU, and requiring a bounding box drops the best model from 85.5% to 57.6% answer accuracy, which the authors read as evidence against self-generated visual tools as intermediate steps [5]. The paper's own table also shows models slightly exceeding its human baseline on the synthetic set (85.76% vs 83.10%) [5].

Figure 3. LongChart VQA's curation pipeline and evaluation protocols: a latent logical graph defines the minimal set of charts from which questions and images are rendered, followed by human verification; evaluation contrasts minimal-set with full-set context and adds grounding/pointing outputs and image perturbations (reproduced from the original paper [5]).

2.2 OCR-MetaReasoning: deduction, induction and abduction over recognised text

OCR-MetaReasoning recasts text-rich image understanding as meta-reasoning over OCR-grounded evidence: 1,500 items balanced on a 3×5 grid of deduction, induction and abduction against five OCR-object categories, with a deterministic macro score for final answers and an LLM-judged process-compliance score [6]. The best model reaches 89.3, but the aggregate hides a bimodal distribution: deduction is the weakest axis across models (73.6) and layout semantics the weakest category (66.9), while a subset of items is solved by all eighteen evaluated models and a hard core of 214 falls below 0.5 [6]. A PhD human reference reaches 96.0 on a stratified 300-item subset, and a question-only control floors at 16.5, confirming the benchmark is genuinely OCR-dependent [6].

2.3 Groundbench: boxes are too forgiving for a geometry claim

Groundbench keeps the referring-expression data but changes the output contract: given an image, a referring expression and an announced vertex budget, the model must emit exactly that many integer points on one line, scored by a deterministic rasteriser with no LLM judge — 1,500 questions from RefCOCO, RefCOCO+ and RefCOCOg, each at five vertex budgets, for 7,500 target files [7]. The headline gap is between tasks: box-style performance reaches 88.2 IoU and 97.1 Acc@0.5, while direct polygon output falls to 57.7 IoU and 69.2 Acc@0.5 [7]. The authors are explicit that they do not claim saturation of the source distribution; their claim is narrower and sharper — that four-number rectangles leave little room to distinguish frontier systems and credit background interior [7].

Figure 4. Groundbench's exact-N protocol: the same image, expression and referent are frozen across matched tasks, a source polygon is resampled to an announced vertex budget N, and the parsed output is scored by mask IoU against the exact-N target; outputs that cannot be parsed score zero (reproduced from the original paper [7]).

Takeaways

• Across three very different output contracts — computed chart answers, OCR-grounded inferences and polygon vertices — the gap is between recognising the referent and emitting the required artefact [5, 6, 7].
• LongChart VQA's own numbers put its strongest model just above its human baseline on synthetic data, a reminder that self-constructed multi-hop sets can be easier than they look [5].
• Where the metric is deterministic (Groundbench's rasteriser, OCR-MetaReasoning's macro score) the reported differences are not confounded by judge behaviour; where a judge is used, the papers vary in how far they control for it [6, 7].

3 Trust and safety: hallucination, abstention, forgetting and jailbreaks

The largest cluster of the month is not about capability but about failure: benchmarks that isolate when a model invents, when it should have declined, what it forgets after an unlearning request, and how it can be talked into harm. The methodological thread is the same — a narrower, better-controlled probe instead of a broader benchmark.

3.1 MIOH: object hallucination is worse with more images

MIOH targets object hallucination in multi-image settings, arguing that existing hallucination benchmarks are single-image and binary, while general multi-image benchmarks do not control the visual factors that induce hallucination. Its design crosses four object-centric tasks (existence, counting, attribute, position) with three reasoning patterns (comprehensive, comparative, selective) and three pressures — the number of images, perceptual difficulty and contextual bias — giving 3,484 questions over 11,732 images drawn from COCO-ReM, PACO and SVG [11]. Across 29 model configurations the median picture is poor: mean accuracy 36.1%, with the best model at 64.4% (Gemini-2.5-Pro) and GPT-5 at 63.1% [11]. The paper's most useful result is a controlled contrast rather than a leaderboard: decomposing the same existence items into single-image questions raises every model's accuracy by roughly 5–26 points, with the weakest models gaining most (Figure 5) [11].

Figure 5. Multi-image versus decomposed single-image accuracy on MIOH's Existence task, per model (Figure 5 of [11]; reproduced from the original paper).

3.2 TRAPSBench: models encode restraint they do not express

TRAPSBench asks a question most hallucination benchmarks skip — will the model say "I don't know" when the video provably cannot resolve the outcome? It generates matched control/void pairs in MuJoCo (1,404 items across four splits) so that the void item is identical except the video stops before the outcome, and proposes a metric that multiplies control accuracy by Youden's J of abstention versus false abstention [12]. Sixteen VLMs score badly: the best standard-regime score is 0.292 (Gemini 2.5 Flash) and the best with guided prompting 0.568 (Gemini 3.1 Pro, low reasoning) [12]. The mechanistic half of the paper is the part worth carrying forward: linear probes on activations recover the needed distinction at up to 0.913 AUROC, so the information is present but not expressed in the output — a dissociation between representation and behaviour rather than a knowledge gap [12].

3.3 MMJailBench: factorising a jailbreak into what actually drives it

MMJailBench takes 272 harmful intents and crosses them with six prompt framings, five task-relevant visual-semantic conditions and two instruction carriers, for 16,320 instances per model — a design built to attribute risk to a factor rather than report one attack-success rate [13]. The attribution is clear: prompt framing dominates, with the spread between the most and least vulnerable framings exceeding 40 percentage points [13]. Two counter-intuitive results stand out. Task-relevant visual context raises attack success (an authorisation document adds about +13 points over no image), but control images barely move it, so the effect is semantic rather than the mere presence of an image [13]. And instructions rendered inside an image are weaker than the same instruction as text (40.59% versus 50.70%), contradicting the common assumption that OCR-borne prompts are the stronger carrier [13].

Figure 6. MMJailBench's factorised design: harmful intents crossed with six prompt framings, generated visual-semantic conditions and two instruction carriers, then scored by an LLM judge into attack-success and context-adjusted rates (reproduced from the original paper [13]).

Table 2. Trust-and-safety benchmarks on shared axes; settings as reported in each paper.

Takeaways

• Three independent lines converge on the same diagnosis: the capability is often present but not acted on — probes decode abstention (AUROC 0.913) that models fail to express [12], and multimodal jailbreak risk is driven by framing rather than by the image itself [13].
• Simply having more images is itself a pressure: MIOH's own decomposition raises accuracy substantially on identical items, so multi-image scores should not be read as single-image competence [11].
• Each of these benchmarks isolates one factor; none of them yet reports defence-side numbers, so they measure vulnerability rather than robustness [11, 12, 13].

4 Culture, language and pragmatics: benchmarks go local and figurative

A fourth cluster builds benchmarks for what general English-centric suites cannot see: a single national culture in one language, language-specific morphology and dialects, comics that only make sense in order, and meaning that depends on pragmatic context rather than cross-modal mismatch.

4.1 PUMA: one culture, three modalities, rule-based scoring

PUMA is a Polish benchmark with one hundred items in each of images, audio and documents, where a response counts only if all of a question's verification rules hold — a strict score reported alongside a fractional soft score [19]. The audio turn is deliberately hard (mean 82 s, a majority of low-quality recordings), and the document turn skews toward handwriting, degraded scans and perspective distortion, with OCR scored by word accuracy and table TEDS [19]. The methodological move worth noting is that Polish morphology motivates lemmatisation in the scoring normaliser, and the paper documents a residual model-label mismatch in its own OCR table [19].

4.2 PoVisLE: what "fluent Polish" hides

PoVisLE covers 1,117 images and 2,366 manually written Polish VQA pairs and evaluates 16 VLMs with a macro average over question formats, using circular option rotation so an item counts only if answered correctly under every rotation [20]. The best model scores 71.45% against a 16.79% random baseline, and reasoning variants gain up to 13.2 points [20]. The diagnostic finding is the useful one: a post-hoc analysis of open-ended answers labels 49.45% of predictions as hallucination overall, rising above 50% for weaker models — the paper's own framing is that fluent Polish output coexists with unreliable grounding [20].

4.3 CMPM: order is the task, and models lose it

CMPM is 1,214 Chinese multi-panel memes annotated with structure type, ordering dependency and optional comment context; the central task presents panels shuffled and requires the reading order to be recovered [21]. Performance collapses: Gemini 3.1 Pro recovers 75.2% and GPT-5.5 57.8%, while open models land between 6.1% and 27.4% — some below the 19.4% random-permutation reference — and providing comment context adds almost nothing (about +0.06) [21]. Because the correct answer is a valid linear extension of gold precedence constraints, this is one of the few benchmarks in the set where within-group permutations are tolerated rather than penalised [21].

Figure 7. The five structural types annotated in CMPM — narrative, progressive, conversational, comparison and parallel — each with an example; the order task presents panels shuffled and requires a valid reading order (Figure 2 of [21]; reproduced from the original paper).

Takeaways

• A single-language, single-culture benchmark is a distinct design choice, not a smaller version of a multilingual one: PUMA's rules are written for Polish morphology and its document and audio subsets are skewed toward degraded input by construction [19].
• Fluent generation in the target language is not evidence of grounded understanding (49.45% of PoVisLE's open-ended answers are labelled hallucination) [20].
• Structure and order can be a benchmark's whole object, and they remain a large gap: most open models score below chance on CMPM's order task [21].

5 Expert domains, hard reasoning and system-level evaluation

The last cluster measures competence where being approximately right is not useful — architecture, clinical diagnosis, multi-temporal Earth observation — and, in one case, whether a model can even return the modality it was asked for.

5.1 MMArch: composition, not perception, is the bottleneck

MMArch scores architecture-engineering reasoning over 1,212 items grounded in paper figures and expert-confirmed calculations [23]. The best model reaches 51.73% average (GPT-5.5) against a human panel at 94.57% — a gap of 42.84 points — while the strongest open-class model sits at 29.94% [23]. The error analysis is the contribution: pooled across models, composition errors (right intermediate values, wrong multi-step combination) are 35.8% of failures, ahead of principle 21.7% and perception 20.3%, and prompting interventions barely move the score, including a CoT change that hurts three of four tested models [23].

Figure 8. MMArch's construction pipeline: subdomain question types over an image database drawn from three sources (10k papers, 16k images), image selection, QA design, and quality control against accuracy, provenance and difficulty (reproduced from the original paper [23]).

5.2 MedReaMM: clinical synthesis, scored two ways

MedReaMM evaluates expert-level diagnostic synthesis over 625 cases, scoring predicted diagnoses both as ICD-11 code equivalence and as SapBERT embedding similarity at a stated threshold, precisely because the two regimes disagree [24]. Reasoning models lead on every metric (o1 at 61.3 Top-10 primary versus GPT-4o 55.7), and prompting is bounded in both directions: a CoT prompt lifts Llama-4-Maverick by about 10.5 points but costs Qwen2.5-VL-7B roughly 8.3 [24]. Two findings deserve caution. An input ablation shows models perform best with aligned captions and degrade below the image-plus-record baseline when captions are shuffled, so captions act as a bridge — a qualification of what the benchmark measures [24]. And in the paper's own comparison, three gastroenterologists score 10.87% strict accuracy on a 46-question instrument against 43.48% for Gemini-2.5-Flash, a gap the authors attribute to individual-versus-team capability [24].

5.3 LongEarth-R1: trajectories, not snapshots, in Earth observation

LongEarth-R1 argues that remote-sensing benchmarks test single observations or change endpoints, and builds a multi-temporal suite plus an RL-aligned model for long-horizon reasoning over image sequences [25]. The reported margins are large on the tasks that explicitly require sequencing — redundancy detection 90.28 versus 39.78 for Qwen2.5, chronological-violation detection 50.80 versus 15.47 — but narrowest on change direction (53.37 versus 45.80), and the paper does not label the metric per cell in its results table [25].

5.4 Modality Maturity Index: can a model return the modality it was asked for?

The Modality Maturity Index scores omni-capability prompts on whether a model can produce text, image, audio, video and documents, with a per-prompt macro average [26]. Its negative results are the interesting ones: audio output is returned zero times by any model, the document column is zero because no document-generation tool was provided, and the lenient default detection credits models that paste links to pre-existing assets — so the paper itself warns that rankings partly reflect willingness to link rather than synthesis ability [26]. The judge is a Gemini model grading a plurality of Gemini-produced assets, which the authors state cannot be read as judge-independent [26].

Figure 9. Modality Maturity Index coverage: counts of model responses by input modality (rows) and output modality (columns). The mass sits in the text-output column; non-text outputs are rare from any input, which is the paper's own caveat on reading the index as synthesis ability (reproduced from the original paper [26]).

Takeaways

• In expert domains the limiting factor is usually combination, not perception: MMArch's dominant error is composing correct intermediate values, and prompting does not fix it [23].
• Domain benchmarks increasingly report two scoring regimes side by side (ICD-11 codes versus embedding similarity; strict versus soft rules) because a single metric would hide the disagreement [19, 24].
• System-level evaluation is still immature: an omni-capability benchmark can report zero audio output and a zero document column, so its headline numbers describe infrastructure as much as capability [26].

Also noted

Grouped by theme; each entry is a one-line finding from the deep read.

Video and long-horizon. DVBench supplies the missing corpus for data videos — 300 real videos and 1,000 human-verified questions over narrative, animation, chart perception, chart reasoning and subtitle alignment, with a 91.93% human baseline on a stratified subset [4].

Precise perception. WildHandBench scores 500 real handwritten documents with a perplexity-based "prior-driven error" metric and finds the best model at 71.85% against a calibrated human 77.09%, while model error rates (63–91%) sit far above the human's 49.27% [8]. SpatialQuery supplies a million geometry-grounded multi-instance spatial queries with metric depth ground truth, and shows that widening tolerance by the model's own predicted uncertainty lifts tolerance accuracy from 58.50% to 87.50% while the authors explicitly decline to call it calibration [9]. Inter-3D VQA builds 407K roadside 3D spatio-temporal QA pairs from four cameras, a LiDAR point cloud and HD-map topology, with a LiDAR-enhanced baseline (Inter-Geo) beating image-only baselines [10].

Trust and safety. KnowHal adds a fourth, "knowledge" dimension to hallucination evaluation and finds it the hardest, with no model above 70% overall (best 67.9%) and Wikipedia as its only knowledge source [14]. SHROOM-Visions 2026 reuses the four-language SHEEP data as a shared task: 27 teams and 623 submissions produce best-average correlations of 0.58 (character) and 0.46 (label-conditioned) with 0.51 IoU, but the authors show the top rankings are not statistically separable, with rank intervals spanning up to 14–15 positions [15]. PRMU evaluates corpus-free person-centric unlearning — the method receives only a name and a reference image — across 91.9K probes, and reports that multiple-choice accuracy is nearly insensitive as a forgetting metric while the paper's own tables also contain a few wrong-direction movements it does not explain [16]. UNLINK-VL gives the first clean cross-modal unlearning result: forgetting a fact in the multimodal setting largely fails to transfer to a text-only query and vice versa, with the normalised text-to-multimodal penalty (14.0–16.2) roughly twice the reverse (6.3–10.3) [17]. MME-Safety scores safety jointly with helpfulness and refusal and reports that subtler harmful intents draw the highest harm rate (30.59%) while extreme ones rebound downward, a "defence saturation" pattern [18].

Culture and pragmatics. PragMatch separates pragmatic incongruity from cross-modal mismatch by controlled masking and injection, and the paper candidly reports that a caption-only text classifier reaches 99.6% item accuracy on the same paired metric — exceeding every evaluated LVLM — which bounds what the headline gap can mean [22].

Expert and system level. VIPER is an expert-curated veterinary-pathology benchmark with a deliberately calibrated judge (cross-model Pearson r = 0.98) and an explicit answer-position control via five cyclic option permutations [27]. TomaMMU covers 28,808 tomato-leaf images and 213,119 QA pairs; zero-shot Gemini 2.0 Flash reaches 70.3% accuracy, a fine-tuned domain model 96.09%, and a specialist leaf-disease model shows a highly irregular per-task profile [28]. OmniPhys evaluates Chinese physics understanding and diagram generation over 15,246 questions with a dual-track reason-and-result protocol, where human graders score model-generated diagrams at 0.10–0.29 while a model judge scores them at 0.51–0.75 — a disagreement the report leaves unresolved [29]. CAPEval decouples caption scoring across understanding and generation into 14,965 checklist items over 300 images, but rests entirely on one fixed LLM judge that never sees the image, with no human-agreement check reported [30].

Scope and method

Retrieval and screening

The window is 1–31 August 2026 by each item's recorded v1 submission date (not an announcement batch). Retrieval ran over this library's arXiv mirror within cs.CV, cs.CL, cs.MM, cs.AI, cs.LG and cs.SD, combining one semantic route with keyword routes for named families (unified understanding-and-generation, omni, MLLM/VLM); the union gave a candidate pool of 466 papers, which an AI screening pass reviewed against the boundary "multimodal understanding or unified understanding-generation, excluding pure generation and non-language multimodal work", leaving 428. A second classification pass sorted those into eleven sub-areas; the 113 judged to be benchmarks or datasets form the pool from which thirty were selected for this issue, spanning the sub-areas in proportion. Counts describe the checked routes and are not a census of the field.

Evidence depth and gaps

All thirty papers were read in full text by a per-paper reading task, and each entry above rests on a section or table named in those notes; the two figures were viewed at pixel level. Coverage limits worth stating: the semantic route was capped at its top-k and its tail had already drifted toward generation work, so a benchmark that shares no wording with the routes could be missing; a handful of included items carry arXiv identifiers in the September (2609) series despite August recorded dates, so a few may straddle the boundary, and for a few the library's mounted body is a later revision than the recorded v1. Several results quoted here are the authors' own headline numbers under their own settings and are not comparable across papers; where a paper's evaluation leaves something out — an unvalidated judge, a single model family, a benchmark built from its authors' own representation — that is said where it matters rather than collected here.

References

[1] Fan Zhang, Guangming Yao, Jinyang Wu, et al. VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding. 2026. arXiv preprint. arXiv:2608.14718

[2] Thong Nguyen, Tri Cao, Khoi Le, et al. STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models. 2026. arXiv preprint. arXiv:2609.29607

[3] Kaishuu Shinozaki-Conefrey, Olivier Pascaud, Robin Courant, Xi Wang, Dimitris Samaras, Vicky Kalogeiton. TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film. 2026. arXiv preprint. arXiv:2608.30068

[4] Bomiao Wang, Zekai Shao, Jiexiang Lan, Xiaoliang Fu, Xingchen Zeng, Siming Chen. DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos. 2026. arXiv preprint. arXiv:2608.29711

[5] Ziyan Xiao, Yinghao Zhu, Wenting Zhang, Heaju Kim, Lequan Yu. LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning. 2026. arXiv preprint. arXiv:2608.01328

[6] Gengxu Li, Yuan Wu, Yi Chang. OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding. 2026. arXiv preprint. arXiv:2608.30678

[7] Zhonghan Bian, Zhenran Wang, Jinsong Li, Zhangyang Qi. Groundbench: Multi-Resolution Polygon Grounding Exposes the Geometry Gap in Vision-Language Models. 2026. arXiv preprint. arXiv:2609.27821

[8] Jun Zhang, Qiao Zhao, Cheng Cui, et al. WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans. 2026. arXiv preprint. arXiv:2608.22959

[9] Hai Nguyen, Tung Vu, Cong Tran. SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models. 2026. arXiv preprint. arXiv:2608.01709

[10] Shaozu Ding, Linan Song, Dajiang Suo. Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering. 2026. arXiv preprint. arXiv:2608.28762

[11] Joonki Min, Chaeyun Kim, Hyungwook Choi, et al. Fine-Grained Multi Image Object Hallucination Benchmark. 2026. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 18295-18305. arXiv:2608.30653

[12] Fnu Pramono, John Cai, Sourabh Kulkarni. TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint. 2026. arXiv preprint. arXiv:2608.13167

[13] Tianshi Wang, Jingsong Wang, Yafei Huang, et al. MMJailBench: A Factorized Benchmark for Disentangling Multimodal Jailbreak Vulnerabilities. 2026. arXiv preprint. arXiv:2608.25490

[14] Ruihan Li, Jiyang Tan, Kailin Jiang, et al. KnowHal: A Knowledge-Driven Benchmark for Comprehensive Multimodal Hallucination Evaluation. 2026. arXiv preprint. arXiv:2608.03782

[15] Raul Vazquez, Aman Sinha, Chuyuan Li, et al. Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models. 2026. arXiv preprint. arXiv:2608.25662

[16] Huafeng Chen, Yueming Lyu, Ziyuan Chen, et al. PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models. 2026. arXiv preprint. arXiv:2608.11149

[17] Chunlin Liu, Junnian Chen, Haitong Jiang, et al. Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation. 2026. arXiv preprint. arXiv:2608.03791

[18] Yilian Shi, Yueming Lyu, Haoxiang Tan, et al. MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs. 2026. arXiv preprint. arXiv:2609.20850

[19] Slawomir Dadas, Michal Perelkiewicz, Rafal Poswiata, et al. PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding. 2026. arXiv preprint. arXiv:2608.21853

[20] Anna Kolos, Grzegorz Statkiewicz, Karolina Seweryn, Katarzyna Kowol, Karolina Piosek, Wojciech Kusa. Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation. 2026. arXiv preprint. arXiv:2608.07763

[21] Haihan Li, Haihao Li, Zhenfei Xu, Jize Qian, Yubo Xie. Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning. 2026. arXiv preprint. arXiv:2608.26866

[22] Zhanna Mukhametsharip, Vera Demberg, Varsha Suresh. PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models. 2026. arXiv preprint. arXiv:2608.09772

[23] Chenxu Du, Kang An, Tengyue Wang, et al. MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence. 2026. arXiv preprint. arXiv:2608.09281

[24] Lai Wei, Yuchao Chen, Zhenbiao Cao, et al. MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis. 2026. arXiv preprint. arXiv:2608.22323

[25] Yupan Ding, Jing Xiao, Zhenyuan Zhang, et al. LongEarth-R1: Benchmarking and Aligning Vision-Language Models for Long-Horizon Earth Observation Reasoning. 2026. arXiv preprint. arXiv:2608.13344

[26] Rohit Patel, Dieuwke Hupkes, Sloan Strader. Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models. 2026. arXiv preprint. arXiv:2608.26317

[27] Luca L. Weishaupt, Simone de Brot, Javier Asin, et al. VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology. 2026. arXiv preprint. arXiv:2608.26382

[28] Gia-Han Truong, Khang Nguyen Quoc, Luyl-Da Quach. TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases. 2026. arXiv preprint. arXiv:2608.08727

[29] Hao Chen, Yumin Lin, Nadila Yushanjiang, Xin Lin, Min Zhang. OmniPhys: A Unified Multimodal Benchmark for Physics Understanding and Generation from Chinese Educational Corpora. 2026. arXiv preprint. arXiv:2608.25398

[30] Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang. CAPEval: A Decoupled Caption Evaluation across Understanding and Generation. 2026. arXiv preprint. arXiv:2608.02589