REFINE Multi-Agent LLM System Cuts Java Code Smells Up to 73% but Flags Assertion and Method-Removal Risks

Overview

A new preprint proposes a multi-agent framework for automatically cleaning up Java code and reports that it cut detected code smells by more than two-thirds across three different large language models — while also finding that a meaningful share of the resulting changes silently remove test assertions or public methods that other code may depend on. According to the paper, titled “REFINE: A Multi-Agent LLM Approach for Evidence-Guided Code Refactoring” and posted to arXiv on August 21, 2026 by researchers Muhammad Waseem, Aakash Ahmad, and Pekka Abrahamsson, the system “reduces detected code smells by 68.26%, 72.79%, and 68.49%” depending on which LLM drives it, with the largest reductions concentrated in major smells, the more serious category of design problem that static analysis tools flag.

What We Know

REFINE stands for “Refactoring with Evidence-aware Flow for Integrated ageNtic Execution,” and the authors describe it as “a tool-agnostic, evidence-aware multi-agent approach for generating Java file-level refactoring candidates.” Rather than simply asking an LLM to “clean up this file,” the pipeline chains together several stages: “static-analysis-guided smell identification, smell-informed planning, LLM-based transformation, automated re-analysis, preservation checks, and structured reporting,” according to the paper. The static-analysis stage relies on PMD, which describes itself as “an extensible cross-language static code analyzer” that “finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and so forth.” The kind of problem PMD flags is what software engineers generally call a code smell — “a surface indication that usually corresponds to a deeper problem in the system.”

The researchers tested REFINE on 450 Java files drawn from 15 open-source systems, running each file through three separate LLM backends — OpenAI GPT-5.5, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.8 — to produce 1,350 model-pass outputs in total, the paper states. The experimental setup used “PMD 7.10.0 for static analysis on Java 17,” and files were selected through stratified random sampling across code-smell count and lines-of-code, rather than picked by hand.

The headline result is the smell-reduction rate: 68.26% for the GPT-5.5 configuration, 72.79% for Gemini 3.1 Pro Preview, and 68.49% for Claude Opus 4.8, according to the paper. Within that overall figure, major code smells — the more serious category — saw even steeper cuts, “ranging from 86.51% to 91.60%” across the three configurations. The team also ran a matched 150-file baseline in which the same three LLMs were simply prompted directly to reduce code smells, without REFINE’s multi-agent pipeline around them. Compared to that baseline, REFINE achieved “a higher median code-smell reduction with smaller edits and fewer public-method removals.”

That comparison matters because REFINE’s own safety checks surfaced problems even in its improved results. The paper reports that “critical assert/fail-call preservation passes in only 57.1% of outputs for each model” — meaning that in roughly four of every ten outputs, a test assertion or failure check that should have survived the refactor did not. Public method signatures, which other code in a project may call directly, fared better but were still not fully preserved: GPT-5.5 kept 420 of 450 files’ public methods intact (93.3%), Gemini 3.1 Pro Preview kept 388 of 450 (86.2%), and Claude Opus 4.8 kept 424 of 450 (94.2%), per the paper’s results. Gemini 3.1, the configuration with the largest code-smell reduction, also showed “the highest mean number of removed public methods per affected file,” the authors found — the model that cleaned up the most smells also removed the most public methods.

Broader code-quality metrics told a mixed story. According to the paper, “cyclomatic complexity and LCOM improve mainly for Gemini 3.1, while maintainability, testability, Halstead effort, LOC, and mean method length do not improve consistently” across the three LLM configurations — meaning a lower smell count did not reliably translate into better scores on other established code-quality measures.

What We Don’t Know

The paper is a preprint that has not yet completed peer review, and the authors’ comments field notes that “the accompanying replication package will be made publicly available,” meaning the code and full dataset needed for outside verification were not available at the time of posting. The evaluation is also explicitly scoped: it “is limited to Java and file-level refactoring tasks,” the LLMs worked from “file-level context rather than full repository context,” and the authors caution that “results may not generalise to clean files, test code, generated code.” Critically, the authors state that the “generated candidates were not evaluated through repository-level compilation, regression testing, call-graph impact analysis, or human review” — so it remains unknown how many of REFINE’s outputs would actually compile or pass a project’s existing test suite in practice. The authors themselves conclude that outputs “should be treated as refactoring candidates requiring compilation, testing, dependency analysis, and human review before adoption in repository- or system-level settings,” not as changes ready to merge.

Analysis

The REFINE results land alongside a broader run of research this month probing what happens when AI systems are put in charge of modifying rather than just generating code. The Machine Herald has previously reported on a separate study finding that AI-written bug-fix patches tend to be far larger and more complex than developers’ own fixes even when tests pass, and has also covered LinkedIn’s production deployment of a multi-agent AI code-review system that processes tens of thousands of reviews a week. REFINE fits the same pattern: a multi-stage, evidence-checking architecture outperforms a single direct LLM prompt on the primary metric, but the paper’s own preservation checks — the 57.1% assert/fail-call figure in particular — show that even a pipeline explicitly designed to catch regressions still let a substantial share of them through. For a task like refactoring, where the entire point is to change code without changing its behavior, that gap between “fewer code smells” and “behavior actually preserved” is the detail practitioners evaluating these tools will need to weigh most carefully.