Data Engineer ID99990
Há 23 horas
Porto Alegre, Rio Grande do Sul, Brasil
AgileEngine
Tempo integral
Grátis com e-mail ou Google
Salve esta vaga e mantenha sua pesquisa organizada
Crie uma conta gratuita para salvar vagas, criar alertas e retornar a esta listagem a partir do seu painel.
Grátis com e-mail ou Google
Ao continuar, você concorda com nossos Termos & Política de Privacidade.
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you
ABOUT THE ROLE
We are looking for a Data Engineer to build the pipelines, storage, and retrieval systems behind a data platform that helps AI agents produce better forecasts. The mandatory requirements are 5+ years of experience with Python software engineering, strong SQL, database and data modeling skills, production ingestion pipelines, unstructured data processing and RAG, vector databases or vector search, and an upper-intermediate English level. WHAT YOU WILL DO
- Build reliable ingestion pipelines. Acquire and process diverse datasets, handling incremental updates, deduplication, schema changes, retries, backfills, and monitoring.
- Design the data platform. Develop storage and indexing architectures that support SQL queries, keyword and vector search, and graph-based retrieval.
- Make data useful to AI agents. Build document processing, chunking, embedding, and retrieval pipelines that deliver relevant context quickly and efficiently.
- Support historical reasoning and evaluation. Implement bitemporal models and point-in-time queries that distinguish when something happened from when it became known, preventing future information from leaking into historical tests.
- Own data quality and provenance. Define schemas, validation rules, source attribution, lineage, and timestamp semantics. Investigate inconsistencies and make data limitations visible to downstream users.
- Improve performance, cost, and access controls. Optimize queries and pipelines, measure retrieval quality, and provide secure access across datasets and agent workflows. MUST HAVES
- 5+ years of experience with Python software engineering, including testing, maintainable system design, and production debugging.
- Strong SQL, database, and data modeling skills, with experience optimizing queries and working with large datasets.
- Experience building and operating production ingestion and transformation pipelines, including monitoring and recovery from partial failures.
- Experience processing unstructured data, including document extraction, chunking, embeddings, and retrieval-augmented generation (RAG).
- Practical experience with vector databases or vector search systems, including indexing, metadata filtering, and retrieval performance. This includes methods to tune search quality and relevance.
- Careful judgment about data correctness and meaning: timestamps, units, identifiers, missing values, revisions, and source reliability.
- The ability to take an ambiguous requirement, make architectural tradeoffs, and deliver a working system.
- Upper-intermediate English level. NICE TO HAVES
- Databricks, Snowflake, or comparable lakehouse and warehouse platforms.
- AWS infrastructure, object storage, queues, and distributed processing.
- Financial, economic, news, or other frequently revised datasets.
- Bitemporal modeling, historical snapshots, and reproducible backtesting.
- Search relevance, hybrid retrieval, and RAG evaluation.
- Graph databases, entity resolution, and knowledge graphs.
- TurboPuffer or other vector databases. PERKS AND BENEFITS
- Professional growth: Mentorship, TechTalks, and personalized growth roadmaps.
- Competitive compensation: USD-based pay with education, fitness, and team activity budgets.
- Exciting projects: Modern solutions with Fortune 500 and top product companies.
- Flextime: Flexible schedule with remote and office options.
ABOUT THE ROLE
We are looking for a Data Engineer to build the pipelines, storage, and retrieval systems behind a data platform that helps AI agents produce better forecasts. The mandatory requirements are 5+ years of experience with Python software engineering, strong SQL, database and data modeling skills, production ingestion pipelines, unstructured data processing and RAG, vector databases or vector search, and an upper-intermediate English level. WHAT YOU WILL DO
- Build reliable ingestion pipelines. Acquire and process diverse datasets, handling incremental updates, deduplication, schema changes, retries, backfills, and monitoring.
- Design the data platform. Develop storage and indexing architectures that support SQL queries, keyword and vector search, and graph-based retrieval.
- Make data useful to AI agents. Build document processing, chunking, embedding, and retrieval pipelines that deliver relevant context quickly and efficiently.
- Support historical reasoning and evaluation. Implement bitemporal models and point-in-time queries that distinguish when something happened from when it became known, preventing future information from leaking into historical tests.
- Own data quality and provenance. Define schemas, validation rules, source attribution, lineage, and timestamp semantics. Investigate inconsistencies and make data limitations visible to downstream users.
- Improve performance, cost, and access controls. Optimize queries and pipelines, measure retrieval quality, and provide secure access across datasets and agent workflows. MUST HAVES
- 5+ years of experience with Python software engineering, including testing, maintainable system design, and production debugging.
- Strong SQL, database, and data modeling skills, with experience optimizing queries and working with large datasets.
- Experience building and operating production ingestion and transformation pipelines, including monitoring and recovery from partial failures.
- Experience processing unstructured data, including document extraction, chunking, embeddings, and retrieval-augmented generation (RAG).
- Practical experience with vector databases or vector search systems, including indexing, metadata filtering, and retrieval performance. This includes methods to tune search quality and relevance.
- Careful judgment about data correctness and meaning: timestamps, units, identifiers, missing values, revisions, and source reliability.
- The ability to take an ambiguous requirement, make architectural tradeoffs, and deliver a working system.
- Upper-intermediate English level. NICE TO HAVES
- Databricks, Snowflake, or comparable lakehouse and warehouse platforms.
- AWS infrastructure, object storage, queues, and distributed processing.
- Financial, economic, news, or other frequently revised datasets.
- Bitemporal modeling, historical snapshots, and reproducible backtesting.
- Search relevance, hybrid retrieval, and RAG evaluation.
- Graph databases, entity resolution, and knowledge graphs.
- TurboPuffer or other vector databases. PERKS AND BENEFITS
- Professional growth: Mentorship, TechTalks, and personalized growth roadmaps.
- Competitive compensation: USD-based pay with education, fitness, and team activity budgets.
- Exciting projects: Modern solutions with Fortune 500 and top product companies.
- Flextime: Flexible schedule with remote and office options.