BenchRoom
A pipeline that mines SWE-bench-like tasks out of real repositories — and validates that each one actually builds and fails as advertised.
Explore projectApplied Research
Prototypes, studies, and instruments that bridge research findings into the products millions of developers use every day.
What we focus on
Agent–environment interaction
How agents perceive, act, and coordinate inside a real development environment. That means exploring a repository properly before changing it — embeddings, context-aware retrieval, search delegated to a subagent — using the IDE, the terminal and CI as tools rather than generating text about them, and working out when several agents beat one and how they should divide the work.
Agent reliability
How to make agent behaviour measurable, predictable, and robust. Benchmarks mined from real repositories rather than written by hand; traces, visualization and anomaly detection so a failure can be explained after the fact; self-optimizing prompts and topologies instead of hand-tuning; and the testing and verification that decides whether generated code is actually correct.
The Benchmark Meaning Gap
By Mikhail Evtikhiev
It’s quite possible the tasks you care about are not covered well by any of the existing benchmarks.
Projects
Search Subagent
Delegating a coding agent's repository search to a small self-hosted model — and measuring what that costs in solved tasks.
Explore project
TestSpark
An IntelliJ IDEA plugin that generates unit tests for Java and Kotlin — LLM-based and search-based, in one place.
Explore project
Selected publications
May 2022
StudyApplied Research
A Large-Scale Comparison of Python Code in Jupyter Notebooks and Scripts
Python in notebooks against Python in scripts, structurally and stylistically: lower complexity, more entangled code, and 1.4 times the style issues.
May 2022
StudyApplied Research
Bus Factor in Practice
Asks 269 engineers whether bus factor is a problem they actually have, then estimates it from code reviews and meetings rather than from commits alone.
Other directions
Designing the interaction
Human–AI Experience
How AI reshapes the way developers think, collaborate, and get unstuck — what changes, what stays the same, and what to design around.
Beyond today's models
Machine Learning
Core ML research that powers verification, generation, and reasoning in our tools — papers, benchmarks, and the teams behind them.