The reasoning has to be explicit
“This will break under load” is useless. Which line, which interleaving, which input - the chain has to be complete or it teaches nothing.
Track record
Jan 2025 – May 2025 · Remote · Freelance
AI Training Contributor, Outlier
Code review at volume for a model training pipeline - where reviewing first became a measurement discipline for me rather than an opinion, because the output had to work as training signal.
- Role
- AI Training Contributor
- Period
- Jan 2025 - May 2025
- Basis
- Remote · Freelance
- Reviewed
- 120+ C++ submissions
- Operated by
- Scale AI
01The platform
Outlier is operated by Scale AI. In their own description it is “a platform operated by Scale AI that connects experts with leading AI companies to provide human feedback that improves language learning models.” Contributors write prompts and problem/answer pairs, build grading rubrics, and rate and rank model answers.
It is a distributed contributor pool rather than a team, which shapes the work: your output has to stand on its own, because nobody is going to come and ask what you meant.
- 100,000+ experts active on the platform.
- $500M+ paid out to contributors.
- 700,000+ PhD and MS students onboarded, across 50 countries.
Platform figures are from outlier.ai. The work below is mine.
02What I did
Five months on the C++ side of a model training pipeline. The unit of work was a submission and a verdict: read the code, decide whether it is right, and write down the reasoning in a form that is useful as training signal rather than as a comment to a person.
- Reviewed and debugged 120+ C++ submissions for an AI model training platform, judging correctness, logic and adherence to coding standards.
- Fed structured technical assessments into fine-tuning and evaluation datasets, supporting model reasoning and response-quality improvements.
- Took on generalist evaluation tasks beyond code as part of the distributed contributor pool, contributing to broader training-data quality.
03Reviewing for a model, not a person
This is the thing that was genuinely different, and it took a while to adjust to. A code review written for a colleague can rely on shared context, gesture at a problem and trust them to follow. A review that becomes training data cannot do any of that.
The same defect needs the same verdict
Across a hundred submissions, judging one leak harshly and another leniently puts noise into the dataset. Consistency matters more than any single call.
Style and correctness are different axes
Conflating them makes the signal worse. A correct solution written unpleasantly and an elegant solution that is wrong are not the same thing.
Uncertainty has to be marked
Where I was not sure, saying so is more valuable than a confident guess - a wrong confident label is worse than no label.
04Where it led
This is where reviewing first became a measurement discipline for me rather than an opinion. Everything since - the volume review work, and now designing problems specifically to break models - started from the habits this role forced: be explicit, be consistent, and mark what you do not know.