Ka-Ping constructs a model voting machine, and decides to see how hard it would be to verify that we know it behaves correctly. To push that exploration to its furthest, Ka-Ping Yee and David Wagner try an interesting experiment:

David Wagner and I decided to insert three bugs into Pvote to see if the reviewers would find them. We inserted what we thought would be an “easy” bug, a “medium” bug, and a “hard bug” to find, and chose each bug individually in such a way that an insider could conceivably exploit the bug to influence the results of an election. […]

We decided to insert all of these bugs in a 100-line region of a single file, lines 11 to 109 of Navigator.py, and told the reviewers to look in this region. We did this both because the navigator was the most interesting in terms of the program logic and because we knew the reviewers would have limited time. The new version of the code that we gave the reviewers contained all three bugs, but we did not tell the reviewers how many bugs there were.

Yoshi Kohno, Mark Miller, and Dan Sandler participated as reviewers on the third day of the review. Dan was very familiar with Python and found the “easy” and “medium” bugs quickly, within about 70 minutes. Yoshi Kohno and Mark Miller found the “easy” bug after about four hours of reviewing. None of the reviewers found the “hard” bug.

The reviewers spent a total of about 20 reviewer-hours focused on the task of finding the bugs in this 100-line section of Navigator.py.

[…]

If some of the best programmers in the world struggle to find bugs they even know must be there within a 100 line program, there is simply no hope for humans to review the volume of output from LLMs.

And so there is only one thing to do: don’t bother. At each step, remove yourself. You tell yourself you won’t, but you do. You give in to the chute and the shape of the vibe tunnel, and down you go.