Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

About the role

Mindrift is seeking a senior Python engineer to help build a high‑quality dataset for evaluating AI coding agents. This freelance, project‑based position lets you design realistic developer environments and craft challenging tasks that test the limits of current models.

Key responsibilities

• Create virtual company environments that include codebases, infrastructure, tickets, documentation, and conversation histories.
• Design intermediate‑state tasks, write clear prompts, define success criteria, and ensure they are solvable by an AI agent.
• Develop functional and integration tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback, analysing agent failures and refining evaluation criteria.

Required profile

• 8+ years of professional software development experience.
• Strong command of English (minimum B2 level).
• Hands‑on experience with Python (FastAPI) and JavaScript/TypeScript (React).
• Familiarity with Docker, PostgreSQL, Kafka, and Redis.
• Proven ability to write comprehensive functional and integration tests.

Required skills

• Python
• FastAPI
• JavaScript
• TypeScript
• React
• Docker
• PostgreSQL
• Kafka
• Redis
• Functional testing
• Integration testing

What we offer

• Freelance contract with flexible, remote work schedule.
• Compensation up to $200 per hour, based on experience and pace.
• Opportunity to shape cutting‑edge AI evaluation datasets.