Building Scalable Data Pipelines

1 semana atrás

São Paulo, São Paulo, Brasil beBeeDataEngineer Tempo inteiro R$1.200.000 - R$2.000.000
Unlock Your Potential as a Data Engineer

A high-performance data engineer is needed to lead the development of scalable and efficient data processing pipelines.

About This Role
  • We are seeking a highly skilled individual with experience in AWS (API Gateway, Lambda/Fargate, S3, Glue, CloudWatch)
Main Skills Required
  • Strong knowledge of PySpark (partitioning, joins, optimization) and its applications in real-world scenarios.
  • Experience with data integration and manipulation in REST APIs using proven techniques.
  • Expertise in infrastructure as code (Terraform or CloudFormation) for streamlined deployment and management.
  • Advanced proficiency in English and excellent communication skills.
PREFERRED SKILLS
  • Hands-on experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg).
  • Familiarity with data quality tools (Great Expectations, Soda) for ensuring accuracy and reliability.
  • Knowledge of Step Functions, EventBridge, or Kinesis for building efficient workflow automation.
  • Good API security practices (Cognito, WAF, IAM Policies) to safeguard sensitive data.
  • Proactive problem-solving skills to address complex technical issues.
  • Ability to handle agile environments and rapid changes with ease.
Key Responsibilities
  • Design, develop, and maintain APIs using AWS API Gateway for seamless data exchange.
  • Implement integrations with Lambda or Fargate for efficient data ingestion.
  • Develop processing pipelines in PySpark (AWS Glue or EMR) to optimize data processing efficiency.
  • Manage data in S3 and Glue Data Catalog for streamlined data storage and retrieval.
  • Ensure data quality and integrity through regular data quality checks.
  • Optimize pipeline costs and performance through continuous monitoring and improvement.
  • Create and maintain technical documentation and development standards for team collaboration and knowledge sharing.

  • Scalable Data Architect

    Há 19 horas

    São Paulo, São Paulo, Brasil beBeeDataEngineer Tempo inteiro R$900.000 - R$1.200.000

    Job OverviewThe role of a Data Engineer involves designing, building and maintaining complex systems for storing and processing large amounts of data. This includes developing scalable architectures, ensuring high availability and performance.This encompasses designing and developing data processing systems including pipelines, APIs, and database schema as...

  • Building Efficient Data Pipelines

    2 semanas atrás

    São Paulo, São Paulo, Brasil beBeeDataEngineer Tempo inteiro US$100.000 - US$120.000

    Job OpportunityWe are seeking a Junior Data Engineer to join our team and play a key role in building, maintaining and supporting data pipelines from various platforms around the world.The ideal candidate will have a proven background in SQL, Python and ETL/ELT, with experience working with big data volumes. They should be able to proactively tackle problems...

  • Building Scalable Data Solutions

    2 semanas atrás

    São Paulo, São Paulo, Brasil beBeeDataEngineer Tempo inteiro US$125.000 - US$155.000

    Cloud Data EngineerOur cloud data engineering team is responsible for designing and implementing scalable, efficient, and reliable data pipelines. We work closely with cross-functional teams to ensure that our solutions meet the evolving needs of our organization.Key ResponsibilitiesDesign and implement data pipelines using Cloud-based technologies.Develop...

  • Data Architect for Scalable Pipelines

    1 semana atrás

    São Paulo, São Paulo, Brasil beBeeDataEngineer Tempo inteiro R$90.000 - R$120.000

    We're seeking a seasoned professional to spearhead the development and maintenance of scalable data pipelines using Apache Airflow.As a key member of our data team, you'll work closely with analysts, scientists, and engineers to ensure reliable data delivery and drive business outcomes through data-driven insights.The ideal candidate will possess hands-on...

  • Qualified Pipeline

    4 semanas atrás

    São Paulo, São Paulo, Brasil Data Meaning Tempo inteiro

    Qualified Pipeline - Senior Backend Python DeveloperLocation: remote, LATAMPosition Type: Contract, Long Term/Short TermPosition Summary:Data Meaning is Leading Business Intelligence consulting company, an Alteryx Premier Partner, Tableau Partner, and a growing organization. We provide top consulting services to companies throughout the US, generating...

  • Qualified Pipeline

    Há 7 dias

    São Paulo, São Paulo, Brasil Data Meaning Tempo inteiro US$90.000 - US$120.000 por ano

    Qualified Pipeline - Senior Backend Python DeveloperLocation: remote, LATAMPosition Type: Contract, Long Term/Short TermPosition Summary:Data Meaning is Leading Business Intelligence consulting company, an Alteryx Premier Partner, Tableau Partner, and a growing organization. We provide top consulting services to companies throughout the US, generating...

  • Qualified Pipeline

    3 semanas atrás

    São Paulo, São Paulo, Brasil Data Meaning Tempo inteiro

    Qualified Pipeline - Senior Backend Python DeveloperLocation: remote, LATAM Position Type: Contract, Long Term/Short Term Position Summary: Data Meaning is Leading Business Intelligence consulting company, an Alteryx Premier Partner, Tableau Partner, and a growing organization. We provide top consulting services to companies throughout the US, generating...

  • Qualified Pipeline

    Há 3 dias

    São Paulo, São Paulo, Brasil Data Meaning Tempo inteiro

    Overview Qualified Pipeline - Senior Backend Python Developer Location: remote, LATAM Position Type: Contract, Long Term/Short Term Data Meaning is a leading Business Intelligence consulting company, an Alteryx Premier Partner, Tableau Partner, and a growing organization. We provide top consulting services to companies throughout the US, generating...

  • Qualified Pipeline

    2 semanas atrás

    São Paulo, São Paulo, Brasil Data Meaning Tempo inteiro

    Qualified Pipeline - Senior Backend Python Developer Location: remote, LATAM Position Type: Contract, Long Term/Short Term Position Summary: Data Meaning is Leading Business Intelligence consulting company, an Alteryx Premier Partner, Tableau Partner, and a growing organization. We provide top consulting services to companies throughout the US,...

  • Data Engineer

    1 semana atrás

    São Paulo, São Paulo, Brasil beBeeDataEngineer Tempo inteiro US$120.000 - US$140.000

    Cloud Data Engineering RoleThis is an opportunity to contribute to our data engineering team, focusing on designing and implementing scalable data pipelines in the Google Cloud Platform ecosystem.We are looking for a skilled professional with expertise in cloud-native data engineering to join our team.The ideal candidate will have experience working with...