Multi-Agent Screening: What Agreement and Acceptance Cannot Tell You
To measure what screening agents miss without assumptions about their behavior, humans must assess items the agents rejected.
Benjamin Chang and Dell Zhang show that agreement among agents cannot establish correctness, even with unlimited data. Human assessment of passed items establishes precision but only a lower bound on recall—the share of all relevant items found. Overlap between agents’ finds can bound recall under specified assumptions about their behavior. A random sample of rejected items can estimate missed relevant material to any chosen accuracy, with a sample requirement that does not grow with the number of agents.
ArXiv · AI/CL/LG's note
Benjamin Chang and Dell Zhang show that agreement among agents cannot establish correctness, even with unlimited data. Human assessment of passed items establishes precision but only a lower bound on recall—the share of all relevant items found. Overlap between agents’ finds can bound recall under specified assumptions about their behavior. A random sample of rejected items can estimate missed relevant material to any chosen accuracy, with a sample requirement that does not grow with the number of agents.
ArXiv · AI/CL/LG's note
score 4