It has to be legitimately hard
A competent engineer should have to stop and think. If a human would get it wrong for a silly reason, the failure is about the question, not the model.
Track record
Jun 2026 – Present · Remote · Freelance
Software Engineer, Handshake AI
Building the adversarial test surface used to find where frontier models break on real engineering work - the problems, the sandboxes they run in, and the harness that catches a fixed behaviour regressing.
- Role
- Software Engineer
- Period
- Jun 2026 - Present
- Basis
- Remote · Freelance
- Focus
- Adversarial evaluation
- Stack
- Python · C++ · Docker
01The company
Handshake AI is Handshake’s service for frontier AI development - in their words, “expert human model validation and technology systems for frontier AI development.” It sits on top of the careers platform Handshake has run for over a decade, which is where the expert network comes from.
The work it brokers is model training and validation done by people with real domain expertise: developing domain-specific prompts, assessing responses, and judging output quality. Their disciplines run from law and finance to virology and music theory. Mine is systems software.
- 1,500 university partners across the US and Europe.
- 18 million students and alumni in the network.
- 3 million graduate-level scholars, including over 500,000 PhDs - more than 50,000 of them in STEM.
- Close to 200 areas of specialty covered.
Company figures are from Handshake’s own announcement, joinhandshake.com. The work described below is mine.
02What I do there
My side of it is adversarial: build the problems that find where a model breaks on real engineering work, then make the result reproducible enough to be worth acting on. A problem that a model fails is only useful if you can show it failing the same way twice.
- Design multi-step Python problems and edge-case scenarios engineered to expose failures in model reasoning, context handling and logic execution.
- Containerise test environments and execution runtimes in Docker so model-generated code can be isolated, reproduced and benchmarked against strict unit-test suites.
- Evaluate, debug and rank model output in Python, C++ and C for time and space complexity, memory leaks, and production coding standards.
- Build local Python drivers inside Linux containers that automate prompt evaluation, regression testing and hallucination detection across model checkpoints.
- Audit chain-of-thought traces for subtle logical fallacies, edge-case failures and improper library usage.
03Why the problems are hard to write
Writing a problem a model fails is easy. Writing one that fails for a reason worth fixing is the actual skill. Trivia fails, obscure API corners fail, and neither tells you anything about whether the model can do engineering.
It has to need more than one step
Single-step problems test recall. The failures worth finding show up when a solution has to hold several constraints at once and the model loses one halfway.
It has to be gradeable without me
If pass or fail needs a human to arbitrate, it cannot run in a regression harness. The test suite has to decide.
It has to look like real work
Concurrency, memory ownership, protocol state - the places where production code actually goes wrong, not puzzle-book territory.
04The harness
Everything runs in containers. That is not ceremony: model-generated code is untrusted code, and the only way to compare two checkpoints honestly is to give them byte-identical environments. A result that depends on what happened to be installed on my machine is not a result.
- Docker images pinned per problem set, so a run in March and a run in August are comparable.
- Strict unit-test suites that decide pass or fail with no human in the loop.
- Python drivers inside the container that sweep prompts, collect output and record the verdict.
- Regression runs across checkpoints, so behaviour fixed in one release is caught if it comes back in the next.
The regression half matters more than it sounds. A single failing prompt is an anecdote. The same prompt failing again three checkpoints later, after it had been fixed, is a finding.
05What connects it to the rest
Building systems and judging them turn out to need the same instincts. Reviewing at volume taught me what to look for; building the runtime taught me what it costs to get those things right. Both sides feed the problem design - the failures I know how to provoke are the ones I have had to fix myself.