AI Trainer

AI Trainer - Python Engineer, Agent Evaluation & Tooling (MCP)

Mindrift is partnering with Toloka to connect domain experts with cutting-edge AI projects. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe. We are seeking hands-on Python engineers for a project focused on developing Model Context Protocol (MCP) servers and internal tooling to run and evaluate agent behavior. You will implement base methods for agent action verification, integrate with internal and client infrastructures, and help fill tooling gaps across the team.

• Developing and maintaining MCP-compatible evaluation servers
• Implementing logic to check agent actions against scenario definitions
• Creating or extending tools that writers and QAs use to test agents
• Working closely with infrastructure engineers to ensure compatibility
• Occasionally helping with test writing or debug sessions when needed

• 4+ years of Python development experience, ideally in backend or tools
• Solid experience building APIs, testing frameworks, or protocol-based interfaces
• Understanding of Docker, Linux CLI, and HTTP-based communication
• Ability to integrate new tools into existing infrastructures
• Familiarity with how LLMs are prompted, executed, and evaluated
• Clear documentation and communication skills — you will work with QA and writers

• Experience with Model Context Protocol (MCP) or similar structured agent-server interfaces
• Knowledge of FastAPI or similar async web frameworks
• Experience working with LLM logs, scoring functions, or sandbox environments
• Ability to support development environments (devcontainers, CI configs, linters)
• JavaScript experience

• Competitive pay for your expertise, with rates up to $15/hour depending on skills and project needs
• Flexible, remote, freelance project that fits around your commitments
• Opportunity to participate in an advanced AI project and enhance your portfolio
• Influence how future AI models understand and communicate in your field

Note: This listing is for an ongoing project opportunity. If you submit high-quality work, you may be invited to collaborate on future projects.