Quality Assurance for AI Agent Evaluation Analyst

Há 5 dias


São Paulo, Brasil Mindrift Tempo inteiro
Overview

AI Agent Evaluation Analyst - AI Trainer at Mindrift. We connect domain experts with AI projects to ethically shape the future of GenAI. This is a flexible, project-based opportunity that is remote and part-time.

Responsibilities
  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism
  • Identifying inconsistencies, missing assumptions, or unclear decision points
  • Helping define clear expected behaviors (gold standards) for AI agents
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives
  • Thinking through complex systems and policies to ensure agents are tested properly
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage
Requirements
  • Excellent analytical thinking: can reason about complex systems, scenarios, and logical implications
  • Strong attention to detail: can spot contradictions, ambiguities, and vague requirements
  • Familiarity with structured data formats: can read, not necessarily write JSON/YAML
  • Can assess scenarios holistically: what’s missing, what’s unrealistic, what might break?
  • Good communication and clear writing (in English) to document findings
Bonus qualifications
  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design
  • Background in consulting, academia, olympiads (logic/math/informatics), or research
  • Exposure to LLMs, prompt engineering, or AI-generated content
  • Familiarity with QA or test-case thinking (edge cases, failure modes, what could go wrong)
  • Understanding of scoring or evaluation in agent testing (precision, coverage)
Benefits
  • Get paid for your expertise, with rates that can go up to $15/hour depending on your skills and project needs
  • Flexible, remote, freelance project that fits around your commitments
  • Participate in an advanced AI project and build relevant experience
  • Influence how future AI models understand and communicate in your field
Seniority level
  • Internship
Employment type
  • Part-time
Job function
  • Other
Industries
  • IT Services and IT Consulting
#J-18808-Ljbffr

  • Região Geográfica Intermediária de São Paulo, Brasil Mindrift Tempo inteiro

    Overview At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into...

  • Quality Assurance

    1 semana atrás


    São Paulo, Brasil Superbet Tempo inteiro

    Overview Join to apply for the Quality Assurance (QA) role at Superbet . It’s an exciting time to join Superbet as we enter new markets and develop new technologies. We’re looking for a new colleague to partner with our team to bring people initiatives to life and help deliver top-notch customer experiences. Job Summary We’re looking for a Quality...

  • Quality Assurance Analyst

    2 semanas atrás


    São Paulo, Brasil INDI Staffing Services Tempo inteiro

    At INDI, we're passionate about empowering individuals and businesses worldwide. Our cutting-edge recruiters connect leading companies with top talent, fostering a dynamic environment where innovation thrives. Join us in shaping the future of work.Overview of the role:We are looking for a Quality Assurance Analyst to join our Analyst team and participate in...

  • Quality Assurance Analyst

    3 semanas atrás


    São Paulo, Brasil Novartis Brasil Tempo inteiro

    OverviewQuality Assurance Analyst role at Novartis Brasil. ResponsibilitiesOversight of all production and testing activities, ensuring compliance with cGxP, data integrity, and eCompliance within the Novartis Quality Management System. Support exception investigations; review and approval of production, QC, and AS&T records; MBR review. Support OpEx...


  • Sao Paulo, Brasil Novo Nordisk Tempo inteiro

    Quality Assurance Analyst **Category**:Quality**Location**:São Paulo, São Paulo, BR- **The Position** - As a Quality Assurance Analyst at Novo Nordisk, you will play a pivotal role with the following key responsibilities: - Operationalize the Quality Management System (QMS) and uphold Good Warehousing and Distribution Practices (GWDP) with Sanitary...


  • São José dos Campos, Brasil Johnson & Johnson Tempo inteiro

    **Job Function**: Quality **Job Sub Function**: Customer/Commercial Quality **Job Category**: Professional **All Job Posting Locations**: São José dos Campos, São Paulo, Brazil, São Paulo, Brazil Our expertise in Innovative Medicine is informed and inspired by patients, whose insights fuel our science-based advancements. Visionaries like you work on...


  • São Paulo, São Paulo, Brasil Handoff Tempo inteiro US$600.000 - US$1.200.000 por ano

    Why join us?Handoff is the AI agent that runs a construction company.We help remodelers automate estimating, streamline operations, and win more work - backed by real-time cost data, intuitive design, and workflows that "speak contractor." With over 10,000 monthly active users and $6B in annualized project volume already flowing through our platform, we're...


  • São Paulo, Brasil Mindrift Tempo inteiro

    Freelance Software Developer (Rust) - Quality Assurance (AI Trainer)At Mindrift, innovation meets opportunity. We believe in using the power of collective intelligence to ethically shape the future of AI. The Mindrift platform connects specialists with AI projects from major tech innovators. Our mission is to unlock the potential of Generative AI by tapping...

  • Quality Assurance Analyst

    3 semanas atrás


    São Paulo, Brasil Fiserv Tempo inteiro

    You deserve to do what you love, and love what you do - a career that works as hard for you as you do. At Fiserv, we are more than 40,000 #FiservProud innovators delivering superior value for our clients through leading technology, targeted innovation and excellence in everything we do. You have choices - if you strive to be a part of a team driven to create...


  • Sao Paulo, Brasil Novo Nordisk Tempo inteiro

    Quality Assurance Analyst Jr **Category**:Quality**Location**:São Paulo, São Paulo, BR- **The Position** - As a Quality Assurance Analyst Jr. at Novo Nordisk, you will take on the following key responsibilities: - Operationalize the Quality Management System (QMS) and uphold Good Warehousing and Distribution Practices (GWDP) with Sanitary Technical...