Google Research 提出 Agile Deliberation，用人工审议构建主观图像分类器

Google Research on November 4, 2026 presented Agile Deliberation, an interactive framework for building image classifiers around subjective categories, accompanied by open-source code and a notebook demo. In an 18-session user study, the researchers reported stronger classifier results and lower effort than manual prompt refinement.

The framework addresses categories whose boundaries depend on personal judgment, cultural context, or operating policy: healthy food, inspiring artwork, and clickbait images that promise misleading rewards. Its central premise is that users often discover their own decision boundaries while examining difficult examples. A conventional labeling workflow treats those boundaries as fixed before annotation begins, leaving shifts in intent to surface as inconsistent labels or arbitrary model decisions.

Agile Deliberation turns the evolving definition into both a human-readable specification and the prompt that drives a vision-language model classifier. The design draws on interviews with five content-moderation experts and qualitative analysis of 20 definitions taken from their professional workflows, which showed experts scoping concepts first and then refining them through borderline images.

Two Stages of Concept Deliberation

Concept scoping divides an initial idea into candidate positive and negative subconcepts. A user building a healthy-food classifier, for example, can include salads and grain bowls while excluding processed foods or raw, unprepared ingredients. Representative images support each choice, and subconcepts that do not describe meaningful visual patterns can be discarded before the first classifier is created.

Concept iteration then searches for semantically borderline images instead of simply selecting cases where the model’s confidence approaches 0.5. The researchers argue that vision-language model confidence can be poorly calibrated to human ambiguity: a model may be confident about an image that a person finds difficult. Retrieved examples are deduplicated and clustered, with a multi-armed-bandit rule favoring clusters that previously corrected errors or produced useful feedback.

An ambiguity-mining step organizes a batch around one interpretable issue, such as creamy sauces, sugar content, or portion size, and generates a short explanation of each image’s ambiguity. Users mark images in or out of scope and can add natural-language rationales. The system generates alternative definitions, tests them against all labels collected so far, selects the candidate with the best F1 score, and allows manual edits.

The prototype uses Gemini 2.5 Pro for decomposition and Gemini 2.5 Flash for classification, borderline-query generation, and ambiguity mining. Those models were used without additional training or fine-tuning, and the interface runs through an interactive Colab notebook.

What the Live Study Found

Nine participants without professional content-moderation experience each completed two approximately 90-minute sessions, one with Agile Deliberation and one with manual deliberation, in randomized system order. The two test concepts were healthy food and pay-to-play clickbait moderation. Each session included about 45 minutes of definition work, annotation of 200 held-out test images, a survey, and a 15-minute interview. The comparisons also covered zero-shot classification and Modeling Collaborator, an automated decomposition baseline; manual participants could consult that system’s detailed prompts.

The paper reports that Agile Deliberation improved F1 by 11% over its zero-shot baseline for pay-to-play, while manual deliberation gained 8% over its own baseline. For healthy food, the changes were a 10% improvement for Agile Deliberation and a 3% decline for manual deliberation. The gains mainly reflected improved precision, accompanied by smaller reductions in recall. Accuracy was not used because participants’ personalized definitions produced imbalanced test-label distributions.

All participants preferred Agile Deliberation to manual deliberation. Participants reported mean effort of 3.11 with Agile Deliberation versus 4.67 with manual work; mean ratings for feeling insecure, stressed, or annoyed were 1.67 and 3.00, respectively. Both differences were marked statistically significant at p<.05. The interviews supplied a practical explanation: manual users had to invent searches for counterexamples, and some worried that a prompt edit fixing one error would damage behavior elsewhere.

Study Limits and Code Availability

The evaluation also has firm limits. The authors caution that nine participants restrict statistical power and generalizability, and each person used the two systems on different concepts to avoid carrying an established definition into the second session. Subjective experience could therefore be compared within participants, but classifier performance only between participant groups. The study did not isolate individual components or evaluate the full pipeline across other generative models.

The open-source repository contains the companion codebase and dataset plus a step-by-step Demo.ipynb notebook for Jupyter or Google Colab. Its demo builds a nearest-neighbor retrieval index over LAION400M. The default setup downloads two shards totaling roughly 260 MB to verify the pipeline; the documented 100-shard setup totals about 13 GB. Retrieval, external image downloads, and model calls can take seconds to minutes, a single API key may be rate-limited, and several stages support disk caching. The repository’s disclaimer says this is not an officially supported Google product.

Agile Deliberation’s practical contribution is to make subjective intent an explicit, revisable part of classifier development. Users can inspect positive and negative rules, examine the edge cases that exposed gaps, and supply rationales for revisions. For moderation and curation teams, that makes the category definition and the human judgments shaping it part of the classifier’s inspectable development process.