Data Engineer

Há 8 horas

Manaus, Brasil Zunzun Solutions Tempo inteiro

Summary: We are seeking a highly skilled Data Engineer (Azure Databricks) to design, implement, and optimize enterprise-grade data pipelines. In this role, you will leverage Azure Databricks, Azure Data Factory, SQL Server, and Python to enable scalable, governed, and performant data solutions. You will play a key role in modernizing our data platform on the Azure Cloud, ensuring reliability, efficiency, and compliance across the full data lifecycle. Key Responsibilities: · Data Pipeline Development: Design, build, and optimize ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF). · Data Flows & Transformations: Develop pipelines, data flows, and complex transformations with ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading. · Data Processing: Develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation. · Database & Query Optimization: Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation. · SSIS & Migration Support: Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migrationand modernization into cloud-native pipelines. · Collaboration & DevOps: Work with cross-functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI/CD) for deployment. · Data Governance & Security: Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security. · API & External Integration: Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability. · Automation: Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability. · Advanced SQL Expertise: Craft and optimize complex T-SQL queries to support efficient data processing and analytical workloads. Required Califications: · 5+ years of hands-on expertise with Azure Databricks, Python, PySpark, and Delta Lake. · 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines. · Strong SQL Server / T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices. · Demonstrated experience in SSIS package design, deployment, and performance tuning. · Hands-on knowledge of Unity Catalog for governance. · Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects. Nice to Have: · Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables. · Experience with Microsoft Purview, Power BI, and Azure-native integrations. · Familiarity with Profisee Master Data Management (MDM). · Working in Agile/Scrum environments. Preferred Qualifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification Databricks Certified Data Engineer Associate or Professional Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise

  • Data Engineer

    1 semana atrás

    Manaus, Brasil Tata Consultancy Services Tempo inteiro

    Come to one of the biggest IT Services companies in the world!!Here you can transform your career!Why to join TCS?Here at TCS we believe that people make the difference, that's why we live a culture of unlimited learning full of opportunities for improvement and mutual development.The ideal scenario to expand ideas through the right tools, contributing to...

  • Data Engineer

    4 semanas atrás

    Manaus, Brasil Tata Consultancy Services Tempo inteiro

    Come to one of the biggest IT Services companies in the world!! Here you can transform your career! Why to join TCS? Here at TCS we believe that people make the difference, that's why we live a culture of unlimited learning full of opportunities for improvement and mutual development. The ideal scenario to expand ideas through the right tools, contributing...

  • Backend Data Engineer

    2 semanas atrás

    Manaus, Brasil Bebeedataengineer Tempo inteiro

    Backend Data Engineer RoleWe are seeking an experienced Backend Data Engineer to lead the development of our data pipelines and services.This role emphasizes Python-based architecture, AWS data workflows, and integration with AI/ML frameworks to enable advanced media processing capabilities.Key Responsibilities:Design and develop scalable backend services...

  • Senior Data Engineer

    Há 4 dias

    Manaus, Brasil Bebeedata Tempo inteiro

    About the RoleWe are seeking a highly skilled data engineer to join our team.This role emphasizes Python-based services, AWS data workflows, and integration with AI/LLM frameworks to enable advanced media processing capabilities.

  • Backend Data Engineer

    2 semanas atrás

    Manaus, Brasil beBeeDataEngineer Tempo inteiro

    Backend Data Engineer Role We are seeking an experienced Backend Data Engineer to lead the development of our data pipelines and services. This role emphasizes Python-based architecture, AWS data workflows, and integration with AI/ML frameworks to enable advanced media processing capabilities. Key Responsibilities: Design and develop scalable backend...

  • Senior Data Engineer

    3 semanas atrás

    Manaus, Brasil beBeeData Tempo inteiro

    About the Role We are seeking a highly skilled data engineer to join our team. This role emphasizes Python-based services, AWS data workflows, and integration with AI/LLM frameworks to enable advanced media processing capabilities.

  • Lead Data Engineer

    Há 4 dias

    Manaus, Brasil Bebeedataengineering Tempo inteiro

    Data Engineering Lead Position As a senior data engineer, you will be responsible for overseeing the development and optimization of large-scale data pipelines. Your expertise in building efficient ETL processes using Python will enable seamless data operations. Achieve end-to-end ownership of data processing workflows that transform and load datasets using...

  • Senior Data Engineer

    Há 6 dias

    Manaus, Brasil beBeeData Tempo inteiro

    Seeking a seasoned data engineer to spearhead end-to-end data pipeline design and optimization. Key Responsibilities Provide technical leadership and mentorship to junior engineers. Design scalable data solutions handling large datasets. Implement workflows using orchestration tools like Apache Airflow. Lead Snowflake and data lake technology implementation...

  • Gcp Cloud Data Engineer

    1 semana atrás

    Manaus, Brasil Bebeedataengineer Tempo inteiro

    Cloud Data Engineering Opportunity We are seeking a skilled Data Engineer to design and implement data architectures on GCP using services such as BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, Composer, and others. The ideal candidate will have expertise in building and optimizing scalable, high‑performance ETL/ELT pipelines, ensuring data quality,...

  • Senior Data Engineer

    2 semanas atrás

    Manaus, Brasil Breakthrough Talent Tempo inteiro

    Senior Data Engineers Breakthrough Talent is currently supporting a leading data business build a brand new data platform to support a modern data mesh organisation. They work with national and global clients across data, software and digital consulting, and they are now looking for a Data Infrastructure Engineer to help build and scale their next generation...