Chart-EARL: Improving Chart Reasoning of Vision LLMs via Early Anchoring in Reinforcement Learning

Abstract

Chart reasoning poses a significant challenge for Vision Large Language Models (VLLMs). While Reinforcement Learning (RL) has emerged as a promising solution, existing outcome-based RL methods overlook intermediate errors, whereas process-based RL methods remain difficult in verifying abstract logical steps. We therefore analyze chart-reasoning failures to identify an intermediate supervision signal that is both relevant to the task and amenable to objective verification. Our pilot study suggests that early visual-perception errors constitute a prominent source of failure in the chart-reasoning setting. Guided by this insight, we propose Chart-EARL, which constructs verifiable early visual anchors for targeted chart-reasoning supervision. We first build a structured training corpus that explicitly decouples visual perception from logical deduction. Building upon this structure, our approach optimizes the model with rewards for both accurate extraction of descriptive visual features and final-answer correctness. This strategy provides direct and verifiable feedback on early visual grounding while keeping supervision on the final reasoning objective. Experiments show clear performance improvements on complex chart reasoning benchmarks.