• Pay: Not specified in the available role details.
• 100% remote
• Part-time independent contractor
• 20 hours per week with some overlap with PST
Senior engineers who know open-source codebases can help test large language models on real bug-fixing work, by preparing repositories and judging how a model performs.
AI training datasets now include whole repositories, not just short snippets. You will check that each repository and issue is a fair, useful test, run the code yourself, and help researchers choose the problems that challenge a model most.
## What you will do
• Analyze and triage issues in popular open-source libraries.
• Set up repositories and development environments, including Docker-based ones, and judge whether the unit tests are useful and reliable.
• Run and change codebases on your own machine to measure how well the model fixes bugs.
• Work with researchers to pick repositories and issues that make strong tests, and help coordinate junior engineers where needed.
## Why this is worth your time
• Use open-source and debugging experience to shape how AI models are tested.
• Hands-on work with real repositories, tests and Docker setups.
• Part-time contract work open to engineers worldwide.
• Fully remote, no commute; some hours overlap with Pacific time
## Who this is for
• Strong experience in at least one of Python, Go, Rust, Java, JavaScript, C, C++, C# or Ruby.
• Proficient with Git, Docker and setting up a basic software pipeline.
• You can understand complex codebases, and run, change and test real projects locally.
• Professional fluency in English.
• Helpful: open-source work as a contributor or evaluator, ideally on well-kept public repositories (for example ones with 500 or more stars), and experience with developer tools or automation agents.
• No AI experience needed; LLM research or evaluation experience is relevant and helpful.
## Pay and details
• Pay: Not specified in the available role details.
• Independent contractor.
• Part-time, flexible schedule; fully remote, Worldwide
• 20 hours per week with some overlap with PST
## How to apply
1. Apply on OpenTrain with your resume: https://www.opentrain.ai/jobs/llm-code-evaluation-engineer--cmrnidzfy001j04l5mxkpgan7/?ch=getaicareers&sub=p_3b165ebc9680
2. Complete the application on the hiring platform.
3. If shortlisted, take a conversational AI interview with technical questions and a coding challenge, whenever it suits you.
Your engineering and debugging experience can help AI models get tested more honestly.
## About OpenTrain AI
OpenTrain AI helps people find AI training and data-labeling work, create a profile, and apply to projects across the industry.
