Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code
175 human-curated data-analysis tasks that score plotting code by the picture it draws, judged by a vision model against the ground-truth plot.
Abstract
This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses on generating code for visualizing tabular data - such as a Pandas DataFrame - based on natural language instructions, complementing current evaluation tools and expanding their scope. The dataset includes 175 unique tasks. Our experiments assess several leading Large Language Models (LLMs) across three visualization libraries: Matplotlib, Seaborn, and Plotly. We show that the shortening of tasks has a minimal effect on plotting capabilities, allowing for the user interface that accommodates concise user input without sacrificing functionality or accuracy. Another of our findings reveals that while LLMs perform well with popular libraries like Matplotlib and Seaborn, challenges persist with Plotly, highlighting areas for improvement. We hope that the modular design of our benchmark will broaden the current studies on generating visualizations. Our dataset and benchmark code are available online: https://huggingface.co/datasets/JetBrains-Research/PandasPlotBench https://github.com/JetBrains-Research/PandasPlotBench.
- Type
- Benchmark
- Venue
- MSR '25
- First public
- 3 Dec 2024
- arXiv
- 2412.02764
- Authors
- Timur GalimzyanovSergey TitovYaroslav GolubevEgor Bogomolov
- Direction
More from Machine Learning
The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management
A study of context-management strategies for coding agents — context compression and observation masking over long horizons.
EnvBench: A Benchmark for Automated Environment Setup
Hard automated development-environment-setup cases across Python and JVM repositories, with dataset and code.
Long Code Arena: A Set of Benchmarks for Long-Context Code Models
Six benchmarks for long-context code tasks — library-based generation, CI build repair, project-level completion, commit message generation, bug localization, and module summarization.