Who evaluates the evaluators? Money, access and the METR network
The finding is an institutional junction: evaluation research sits between philanthropic priorities, laboratory access and public authority. The relationships are visible enough to ask concrete questions about how that junction works.
A funded collaboration with RAND
METR’s October 2024 announcement describes about $38m catalyzed through Audacious for Canary, including about $17m for METR. Gates appears in the wider collective; a Gates-specific Canary contribution is not identified. The project profile describes evaluation capacity as the intended outcome.
Resources behind independence
METR’s August update announces about $71m in commitments over six months. Its acknowledgments cover multiple years. Its About page discloses an EU AI Office assistance contract and lab-supplied resources. The missing terms matter: publication rights, withdrawal of access and dispute procedures determine how independence operates.
The spin-off account describes separation from ARC, and its current text says Paul Christiano declined planned board and advisory involvement. Historical incubation is the relationship established here.
From methods to rules
The 2023 scaling-policy framework explicitly presents voluntary evaluations as groundwork for standards and regulation. This is a documented route by which technical criteria could become conditions on further development. Our inference is that the people designing those criteria deserve scrutiny alongside those formally empowered to enforce them.
RAND’s open-weight evaluation paper makes the asymmetry debate concrete: it proposes testing against post-release modifications while recognizing proportionality. A useful public argument should examine the tests, their costs and their consequences.
The next records with leverage
The EU contract and deliverables; lab-access agreements and publication clauses; Canary donor allocations; and appointment and appeal rules for any adopted regime. Those documents could establish who has discretion, who pays and who can contest the result.