“A practical fix if you want me to keep iterating autonomously: add a post-failure step to e2e-tests.yml that posts the relevant Playwright output as a PR comment with a marker (e.g. ). The MCP pull_request_read get_comments method can read PR comments — so the next time CI fails, I’d see the failure verbatim without needing you to paste it. Lightweight, keeps the existing artifact upload as-is. Want me to add that?”

Me: Yes, add that with a secrets-scrubbing feature that leverages whichever OSS secrets detection tool you evaluate as most appropriate. 

It used gitleaks. I put this in my agents.md

“CI will exercise the new failure-summary bot on the next run; if the [e2e testing workflow] still fails, the failure blocks will appear as a PR comment so I can iterate without you pasting logs.”

How it works

Agents (currently a variety of Anthropic models set to various effort levels, driven by Claude Code) are given set of tasks they must automate using GitHub Actions, either using a particular scripting language or whichever they want.* They must use Test-Driven Development (TDD)– basically “write tests first, and don’t come back until they all pass”.**

A panel of judges (Google Gemini and Claude Haiku) then evaluates the comprehensiveness of the tests and the quality of the code.

Which model, effort level and scripting language should you use?

Adjust the sliders according to your priorities.

* When allowed to choose, the agents always choose Python.

** Agents run their tests locally in a container that leverages nektos act to emulate a GitHub-hosted runner.