AI Agent Evaluation Analyst

Há 2 dias

Região Geográfica Imediata de Criciúma, Brasil Mindrift Tempo inteiro

Overview Mindrift connects domain experts with cutting‑edge AI projects, unlocking the potential of GenAI by tapping real‑world expertise from across the globe. Who we’re looking for We seek curious, intellectually proactive contributors who double‑check assumptions, play devil’s advocate, and thrive amid ambiguity and complexity. About the project We need QAs for autonomous AI agents in a new project that validates and improves complex task structures, policy logic, and agent evaluation frameworks. The role balances quality assurance, research, and logical problem‑solving. Responsibilities Review evaluation tasks and scenarios for logic, completeness, and realism. Identify inconsistencies, missing assumptions, or unclear decision points. Help define clear expected behaviors (gold standards) for AI agents. Annotate cause‑effect relationships, reasoning paths, and plausible alternatives. Think through complex systems and policies as a human would to ensure agents are tested properly. Work closely with QA, writers, or developers to suggest refinements or edge‑case coverage. Requirements Excellent analytical thinking: reason about complex systems, scenarios, and logical implications. Strong attention to detail: spot contradictions, ambiguities, and vague requirements. Familiarity with structured data formats: can read, not necessarily write JSON/YAML. Ability to assess scenarios holistically: identify missing or unrealistic elements. Good communication and clear writing (in English) to document findings. Desired skills Experience with policy evaluation, logic puzzles, case studies, or structured scenario design. Background in consulting, academia, olympiads (e.g. logic/math/informatics), or research. Exposure to LLMs, prompt engineering, or AI‑generated content. Familiarity with QA or test‑case thinking (edge cases, failure modes, "what could go wrong"). Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.). Benefits Pay rates up to $15/hour depending on skills, experience, and project needs. Flexible, remote, freelance project that fits around primary professional or academic commitments. Participate in an advanced AI project and gain valuable experience to enhance your portfolio. Influence how future AI models understand and communicate in your field of expertise. Seniority level Internship Employment type Part‑time Job function Other Industries IT Services and IT Consulting Referrals increase your chances of interviewing at Mindrift by 2x. We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr

  • Quality Assurance for AI Agent Evaluation Analyst

    2 semanas atrás

    Região Geográfica Intermediária de São Paulo, Brasil Mindrift Tempo inteiro

    Overview At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into...

  • Quality Assurance for AI Agent Evaluation Analyst

    3 semanas atrás

    Rio de Janeiro, Brasil Mindrift Tempo inteiro

    OverviewAt Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into...

  • AI Trainer

    4 semanas atrás

    Região Geográfica Intermediária de Sorocaba, Brasil Mindrift Tempo inteiro

    AI Trainer - Python Engineer, Agent Evaluation & Tooling (MCP) 3 days ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. The Mindrift platform, launched and powered...

  • [EMEA] - Lead AI Developer

    Há 4 dias

    Região Geográfica Imediata de Criciúma, Brasil Ciandt Tempo inteiro

    We are tech transformation specialists, uniting human expertise with AI to create scalable tech solutions. With over 7,400 CI&Ters around the world, we’ve built partnerships with more than 1,000 clients during our 30-year history. Artificial Intelligence is our reality. We are looking for AI-first engineers who use Generative AI as a foundation of software...

  • AI Engineer

    1 semana atrás

    Região Geográfica Imediata de Criciúma, Brasil BairesDev Tempo inteiro

    Overview Join to apply for the AI Engineer - Remote Work role at BairesDev At BairesDev, we\'ve been leading the way in technology projects for over 15 years. We deliver cutting-edge solutions to giants like Google and the most innovative startups in Silicon Valley. Our diverse 4,000+ team, composed of the world\'s Top 1% of tech talent, works remotely on...

  • German Language Specialist – AI Trainer

    Há 2 horas

    Região Geográfica Imediata de Criciúma, Brasil Invisible Expert Marketplace Tempo inteiro

    German Language Specialist – AI Trainer Join Invisible Expert Marketplace and shape the future of AI by providing high‑quality German language training data. Your expertise will enrich AI systems that serve German speakers worldwide. Responsibilities Challenge advanced language models on German grammar, syntax, morphology, phonology, semantics, and...

  • Analista de Qualidade de Software

    2 semanas atrás

    Região Geográfica Imediata de Criciúma, Brasil SoluCX Tempo inteiro

    Analista de Qualidade de Software (QA) – com foco em AI SoluCX São José dos Campos, São Paulo, Brazil A SoluCX está em busca de um(a) Analista de Qualidade para fortalecer nosso Núcleo de Qualidade & Automação. Se você tem experiência em testes exploratórios ponta a ponta, gosta de mergulhar em jornadas de negócio e tem interesse em atuar com...

  • AI Red Team Engineer

    30 minutos atrás

    Região Geográfica Intermediária de Sorocaba, Brasil LILT AI Tempo inteiro

    AI Red Team Engineer - Brazilian Portuguese 1 day ago Be among the first 25 applicants About LILT AI is changing how the world communicates — and LILT is leading that transformation. We’re on a mission to make the world’s information accessible to everyone, regardless of the language they speak. We use cutting‑edge AI, machine translation, and...

  • Business Analyst II

    Há 4 dias

    Região Geográfica Imediata de Criciúma, Brasil Experian Group Tempo inteiro

    Overview We\'re looking for a Business Analyst with expertise in Generative AI, automation, and process improvement to join our Operational Excellence team supporting Compliance. You will focus on applying AI technologies like GPTs and bots to improve workflows and enhance productivity. Responsibilities Lead process discovery and identify automation...

  • Analyst – SAP IBP

    Há 4 dias

    Região Geográfica Imediata de Criciúma, Brasil HCLTech Tempo inteiro

    Get AI-powered advice on this job and more exclusive features. International Opportunity for Analyst – SAP IBP! HCLTech is hiring for a high-impact international project in Mexico City, and we’re looking for experienced talent in SAP IBP (Integrated Business Planning) to help drive digital transformation in supply chain operations. Role: Applications...