Ai QA

2 semanas atrás

Lauro de Freitas, Brasil Bloom Talent Partners Tempo inteiro

**Company Description**

**Role Description**

**Position: AI QA**

**Location: Brazil**

About the role: As the AI QA, you will be the analytical architect of the quality and performance ecosystem that powers Made. You will be responsible for answering the critical question, “How do we measure the quality of a generative AI’s output at scale?” This role is not about traditional QA, it is about creating a new discipline. You will design the frameworks, metrics, and systems that ensure our AI agents are not only responsive but also strategically valuable, creatively inspiring, and personality-aligned. You will be a key leader in a new function, working at the intersection of data science, product strategy, and creative insight to build a truly revolutionary product.

**Key Responsibilities**:

- Design and implement a comprehensive AI quality framework from the ground up, blending automated testing with expert human evaluation.
- Develop detailed Quality Scorecards to systematically assess the performance of our AI agents and their skills, such as content ideation or comment replies.
- Codify and maintain the distinct personalities and communication styles of our AI agents.
- Analyze and interpret the outputs from our Automated Quality Engine, identifying systemic trends, model weaknesses, and opportunities for improvement.
- Collaborate closely with our Creator-in-Residence (CIR) to translate their qualitative, expert feedback into actionable data and new test cases for the automated system.
- Partner with Product and Engineering teams to provide clear, data-driven feedback that directly informs model fine-tuning and the development of new AI skills.
- Own the curation and management of our "Golden Datasets", the benchmark collections of ideal prompts and responses that define excellence for our platform.

**Key Requirements**:

- Bachelor’s degree in Computer Science, Information Systems, Computer Engineering, or a related field.
- Minimum of 3 years of experience with QA
- Experience with automated tests in Python.
- Experience with LangChain, LangGraph or similar tools. LangSmith is a plus.
- Experience with RAG and fine-tuning of LLMs.
- Experience in Git and code reviews.
- Understanding of database systems (e.g., MySQL, PostgreSQL, MongoDB) and writing optimized SQL queries.
- Understanding of A/B testing, data analysis, and performance metrics.
- Exceptional problem-solving abilities and attention to detail.
- Effective communication skills and the ability to work in a fast-paced, collaborative environment. Plus requirements:

- Familiarity with evaluation metrics for generative AI (e.g., coherence, factuality, diversity, style alignment, and hallucination detection).
- Experience with machine learning libraries such as TensorFlow or PyTorch.
- Experience in Python for web development.
- Experience with FastAPI.

  • Ai QA

    2 semanas atrás

    Lauro de Freitas, Brasil Bloom Talent Partners Tempo inteiro

    **Company Description****Role Description****Position: AI QA****Location: Brazil**About the role: As the AI QA, you will be the analytical architect of the quality and performance ecosystem that powers Made. You will be responsible for answering the critical question, "How do we measure the quality of a generative AI's output at scale?" This role is not...

  • Ai Qa

    Há 6 dias

    Lauro de Freitas, Brasil Bloom Talent Partners Tempo inteiro

    **Company Description****Role Description****Position: AI QA****Location: Brazil**About the role: As the AI QA, you will be the analytical architect of the quality and performance ecosystem that powers Made.You will be responsible for answering the critical question, "How do we measure the quality of a generative AI's output at scale?"This role is not about...

  • AI QA

    2 semanas atrás

    Lauro de Freitas, Brasil Bloom Talent Partners Tempo inteiro US$104.000 - US$130.878 por ano

    Company DescriptionBloom Talent Partners specializes in IT recruitment, focusing primarily on the Latam market. We bridge the gap between employers and exceptional IT professionals, ensuring a perfect cultural fit. For companies, we provide tailored IT talent, tech expertise, efficient hiring, and a personalized approach. For candidates, we offer tailored...

  • QA Automation Lead

    3 semanas atrás

    Rio de Janeiro, Rio de Janeiro, Brasil Applaudo Tempo inteiro

    6 days ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. We are looking for an experienced QA Automation Lead with a keen eye for detail to design testing procedures and automation strategies for our customer's software applications. This is a temporary opportunity. You bring to Applaudo the following...

  • Qa Analyst

    3 semanas atrás

    Rio de Janeiro, Rio de Janeiro, Brasil Whirr Crew Tempo inteiro

    Get AI-powered advice on this job and more exclusive features. We are seeking a highly skilled QA Analyst to support the quality delivery of the company's internal and external applications. This role will require close collaboration with the development team and global users to ensure functionality, performance, and reliability across legacy and modern...

  • QA Automation Lead

    Há 6 dias

    Rio de Janeiro, Brasil Applaudo Tempo inteiro

    6 days ago Be among the first 25 applicants Get AI-powered advice on this job and more exclusive features. We are looking for an experienced QA Automation Lead with a keen eye for detail to design testing procedures and automation strategies for our customer’s software applications. This is a temporary opportunity. You bring to Applaudo the following...

  • AI/ML Engineer

    Há 2 dias

    Rio de Janeiro, Rio de Janeiro, Brasil Zyte Tempo inteiro R$120.000 - R$240.000 por ano

    About UsAt Zyte, we eat data for breakfast and you can eat your breakfast anywhere and work for Zyte. Founded in 2010, we are a globally distributed team of over 250 Zytans working from over 28 countries who are on a mission to enable our customers to extract the data they need to continue to innovate and grow their businesses. We believe that all businesses...

  • QA Engineer

    3 semanas atrás

    Rio de Janeiro, Rio de Janeiro, Brasil AgileEngine Tempo inteiro

    Join to apply for the QA Engineer (EST hours) ID35731 role at AgileEngine 2 weeks ago Be among the first 25 applicants Join to apply for the QA Engineer (EST hours) ID35731 role at AgileEngine Get AI-powered advice on this job and more exclusive features. AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and...

  • AI Quality Assurance Specialist

    1 semana atrás

    Rio de Janeiro, Rio de Janeiro, Brasil beBeeQuality Tempo inteiro US$70.000 - US$100.000

    Job OverviewA quality assurance professional plays a vital role in ensuring the quality of software products by designing and executing test strategies across all layers.Validate AI model performance, including accuracy, confidence thresholds, and fallback behavior.Automate quality processes from data ingestion to extraction and classification, with a focus...

  • Lead QA Automation Engineer

    3 semanas atrás

    Rio de Janeiro, Rio de Janeiro, Brasil Braintrust Tempo inteiro

    Join to apply for the Lead QA Automation Engineer role at Braintrust 1 week ago Be among the first 25 applicants Join to apply for the Lead QA Automation Engineer role at Braintrust Get AI-powered advice on this job and more exclusive features. About Pyyne Job DescriptionAbout Pyyne Pyyne is a modern technology consultancy engineering the next generation...