Data Engineer – Python, PySpark
Há 16 horas
Brasília, Federal District, Brasil
Avenue Code
Tempo integral
Grátis com e-mail ou Google
Salve esta vaga e mantenha sua pesquisa organizada
Crie uma conta gratuita para salvar vagas, criar alertas e retornar a esta listagem a partir do seu painel.
Grátis com e-mail ou Google
Ao continuar, você concorda com nossos Termos & Política de Privacidade.
About The Opportunity
We are looking for a skilled and experienced
About The Opportunity
We are looking for a skilled and experienced Data Engineer to join our team and contribute to the design, development, and optimization of scalable data solutions in an AWS cloud environment. In this role, you will be responsible for building and maintaining robust ETL/ELT pipelines , integrating data from multiple sources, and developing efficient data-processing solutions that support business operations and data-driven decision-making. You will work with modern cloud-based data technologies, leveraging Python, SQL, PySpark, and AWS services to build reliable data pipelines, Data Lakes, and Data Warehouses. You will collaborate with cross-functional teams to understand business requirements, implement technical solutions, and ensure data quality, performance, and reliability. The ideal candidate is a hands‑on data engineer with strong expertise in Python, SQL, PySpark, AWS Glue, S3, Athena, Lambda, and Redshift , who is comfortable working across the entire data engineering lifecycle, from requirements gathering and solution design to implementation, testing, troubleshooting, documentation, and technical handoff.
Responsibilities
Design, develop, test, and maintain scalable ETL/ELT pipelines to support data integration, transformation, and processing across multiple data sources.
Build and optimize data-processing solutions using Python, SQL, and PySpark.
Develop and maintain cloud-native data solutions using AWS services, including S3, Glue, Athena, Lambda, and Redshift.
Design, implement, and support Data Lakes and Data Warehouses, ensuring efficient data storage, organization, accessibility, and processing.
Develop data models and transformation workflows to support analytics, reporting, and business intelligence requirements.
Integrate structured and semi-structured data from multiple sources, ensuring consistency, accuracy, and reliability.
Implement data-quality checks, validation processes, and monitoring mechanisms to maintain data integrity across pipelines.
Optimize data pipelines, SQL queries, and distributed processing workloads to improve performance, scalability, and cost efficiency.
Collaborate with business stakeholders, analysts, and engineering teams to understand data requirements and translate them into effective technical solutions.
Participate in technical design discussions, contributing to data architecture, integration patterns, and engineering best practices.
Develop and maintain reusable, well-structured, and maintainable code following data engineering best practices.
Work with Git, CI/CD pipelines, and Terraform to support version control, automated deployments, and infrastructure management.
Troubleshoot complex data-processing issues, investigate pipeline failures, and implement reliable, long-term solutions.
Perform testing, validation, and debugging to ensure the quality and reliability of data solutions.
Produce technical documentation covering pipeline architecture, data transformations, implementation decisions, and operational procedures.
Take end-to-end ownership of data engineering deliverables, from requirements analysis and development through testing, implementation, and technical handoff.
Required Qualifications
Solid professional experience in Data Engineering, with a strong background in developing and maintaining production data pipelines.
Strong proficiency in Python for data processing, transformation, and automation.
Advanced SQL skills, including complex queries, data transformations, performance optimization, and troubleshooting.
Hands‑on experience with PySpark and distributed data processing.
Strong experience designing, developing, and maintaining ETL/ELT pipelines.
Practical experience with AWS cloud services, particularly Amazon S3, AWS Glue, Amazon Athena, AWS Lambda, and Amazon Redshift.
Solid understanding of Data Lake and Data Warehouse architectures, including data storage, processing, and integration patterns.
Experience with data modeling, data transformation, and data integration across multiple sources.
Familiarity with data-quality validation, monitoring, and pipeline troubleshooting.
Experience with Git and collaborative development workflows.
Familiarity with CI/CD practices and Terraform for automated deployments and infrastructure management.
Strong understanding of data engineering principles, code quality, performance optimization, and maintainability.
Experience working across the full data engineering lifecycle, including requirements gathering, implementation, testing, troubleshooting, documentation, and technical handoff.
Strong analytical and problem‑solving skills, with the ability to independently investigate and resolve complex technical issues.
Excellent communication and collaboration skills.
Ability to work independently, manage priorities, and deliver high‑quality solutions in a collaborative en