Quality Assurance for AI Agent Evaluation Analyst

1 semana atrás


Brasília, Brasil Mindrift Tempo inteiro

Overview

AI Agent Evaluation Analyst - AI Trainer at Mindrift. We connect domain experts with AI projects to ethically shape the future of GenAI. This is a flexible, project-based opportunity that is remote and part-time.

Responsibilities
  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism
  • Identifying inconsistencies, missing assumptions, or unclear decision points
  • Helping define clear expected behaviors (gold standards) for AI agents
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives
  • Thinking through complex systems and policies to ensure agents are tested properly
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage
Requirements
  • Excellent analytical thinking: can reason about complex systems, scenarios, and logical implications
  • Strong attention to detail: can spot contradictions, ambiguities, and vague requirements
  • Familiarity with structured data formats: can read, not necessarily write JSON/YAML
  • Can assess scenarios holistically: what’s missing, what’s unrealistic, what might break?
  • Good communication and clear writing (in English) to document findings
Bonus qualifications
  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design
  • Background in consulting, academia, olympiads (logic/math/informatics), or research
  • Exposure to LLMs, prompt engineering, or AI-generated content
  • Familiarity with QA or test-case thinking (edge cases, failure modes, what could go wrong)
  • Understanding of scoring or evaluation in agent testing (precision, coverage)
Benefits
  • Get paid for your expertise, with rates that can go up to $15/hour depending on your skills and project needs
  • Flexible, remote, freelance project that fits around your commitments
  • Participate in an advanced AI project and build relevant experience
  • Influence how future AI models understand and communicate in your field
Seniority level
  • Internship
Employment type
  • Part-time
Job function
  • Other
Industries
  • IT Services and IT Consulting
#J-18808-Ljbffr

  • Brasília, Brasil Devio Tempo inteiro

    Aqui na DEVIO o QA é o guardião da qualidade dos produtos de software que a empresa entrega.Sua missão é assegurar que todos os lançamentos atendem aos mais altos padrões de qualidade, funcionando de forma eficaz e sem falhas para os usuários finais. É responsável por planejar, projetar e implementar estratégias de testes que abrangem desde testes...

  • Freelance Software Developer

    2 semanas atrás


    Brasília, Brasil Mindrift Tempo inteiro

    Freelance Software Developer (TypeScript) - Quality Assurance (AI Trainer) 3 weeks ago Be among the first 25 applicants At Mindrift, innovation meets opportunity. We believe in using the power of collective intelligence to ethically shape the future of AI. The Mindrift platform connects specialists with AI projects from major tech innovators. Our mission...

  • Agent Operator

    2 semanas atrás


    Brasília, Brasil OperationsArmy Tempo inteiro

    **Job Title**: Agent Operator **Schedule**: Full-Time, 45 hours/week **Work Days**: Monday to Friday **Work Hours**: Staggered shifts between 8:00 AM to 8:00 PM EST **Work Setup**: Fully Remote **About the Role**: **Key Responsibilities**: - ** Label and Tag AI Conversations**: Accurately annotate AI agent conversations using internal tools such as...

  • QA/Red Teaming Expert

    2 semanas atrás


    Brasília, Brasil Innodata Inc. Tempo inteiro

    Job Description: We are seeking highly analytical and detail-oriented professionals with hands-on experience in Red Teaming, Prompt Evaluation , and AI/LLM Quality Assurance . The ideal candidate will help us rigorously test and evaluate AI-generated content to identify vulnerabilities, assess risks, and ensure compliance with safety, ethical, and...

  • Freelance Chemistry

    1 semana atrás


    Brasília, Brasil Mindrift Tempo inteiro

    Overview Freelance Chemistry - Quality Assurance (AI Trainer) This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. At Mindrift, innovation meets opportunity. We believe in using the power of collective intelligence to ethically shape the future of AI. What We Do The Mindrift...

  • Content Writer

    2 semanas atrás


    Brasília, Brasil Innodata Inc. Tempo inteiro

    Job Description: We are seeking highly analytical and detail-oriented professionals with hands-on experience in Red Teaming, Prompt Evaluation , and AI/LLM Quality Assurance . The ideal candidate will help us rigorously test and evaluate AI-generated content to identify vulnerabilities, assess risks, and ensure compliance with safety, ethical, and...

  • Quality Assurance Engineer

    4 semanas atrás


    Brasília, Brasil BairesDev Tempo inteiro

    Quality Assurance Engineer - Remote Work | REF# Join to apply for the Quality Assurance Engineer - Remote Work | REF# role at BairesDev Quality Assurance Engineer - Remote Work | REF# 1 week ago Be among the first 25 applicants Join to apply for the Quality Assurance Engineer - Remote Work | REF# role at BairesDev At BairesDev, we've been leading the...

  • Quality Assurance Lead

    3 semanas atrás


    Brasília, Brasil MetaCTO Tempo inteiro

    MetaCTO is building next-generation developer tools powered by AI. Our mission is to streamline how technical teams design, build, and validate software—leveraging AI copilots, automation, and modern IDE extensions. We’re seeking a Backend Engineer to help build critical infrastructure, developer integrations, and intelligent workflows from the ground...

  • Quality Control Analyst

    3 semanas atrás


    Brasília, Brasil beBeeQuality Tempo inteiro

    Job Title: Quality Assurance Specialist We are seeking a detail-oriented individual to join our team as a Quality Assurance Specialist. The successful candidate will be responsible for ensuring the quality of translated documents and working closely with the project management team to clarify project parameters. This is an administrative position that...

  • Content Writer

    Há 7 dias


    Brasília, Brasil Innodata Inc. Tempo inteiro

    Job Description:We are seeking highly analytical and detail-oriented professionals with hands-on experience inRed Teaming, Prompt Evaluation, andAI/LLM Quality Assurance. The ideal candidate will help us rigorously test and evaluate AI-generated content to identify vulnerabilities, assess risks, and ensure compliance with safety, ethical, and quality...