Data Pipeline Architect

Há 5 dias

Brasil beBeeDataEngineering Tempo inteiro
Data Engineering Specialist

Our organization is seeking a skilled and experienced Data Engineer to join our Threat Research team. The primary responsibility of this role will be to design, develop, and maintain data pipelines for threat intelligence ingestion, validation, and export automation flows.

Responsibilities:
  • Design, develop, and maintain data pipelines for ingesting threat intelligence data from various sources into our data ecosystem.
  • Implement data validation processes to ensure data accuracy, completeness, and consistency.
  • Collaborate with threat analysts to understand data requirements and design appropriate solutions.
  • Develop automation scripts and workflows for data export processes to external systems or partners.
  • Optimize and enhance existing data pipelines for improved performance and scalability.
  • Monitor data pipelines and troubleshoot issues as they arise, ensuring continuous data availability and integrity.
  • Document technical specifications, data flows, and procedures for data pipeline maintenance and support.
  • Stay updated on emerging technologies and best practices in data engineering and incorporate them into our data ecosystem.
  • Provide technical guidance and support to other team members on data engineering best practices and methodologies.
Requirements:
  • Proven experience as a Data Engineer or similar role, with a focus on data ingest, validation, and export automation.
  • Strong proficiency in Python.
  • Experience with data pipeline orchestration tools such as Apache Airflow, Apache NiFi, or similar.
  • Familiarity with cloud platforms such as Snowflake, AWS, Azure, or Google Cloud Platform.
  • Experience with data validation techniques and tools for ensuring data quality.
  • Experience building and deploying images using containerization technologies such as Docker and Kubernetes.
  • Excellent problem-solving skills and attention to detail.
  • Strong communication and collaboration skills, with the ability to work effectively in a team environment.

  • Data Architect

    Há 5 dias

    Brasil beBeeDataEngineering Tempo inteiro

    Job Title: Data ArchitectWe are seeking a skilled Data Architect to build and maintain our data pipelines. This role is critical in processing large-scale regulatory data worldwide, transforming usable datasets for downstream applications and APIs.

  • Data Pipeline Expert

    Há 5 dias

    Brasil beBeeData Tempo inteiro

    Expert Data Pipeline Architect We are seeking a seasoned expert to craft and maintain scalable data pipelines on AWS, ensuring optimal performance, quality, and security. Key Responsibilities: Develop and optimize Extract-Transform-Load (ETL) pipelines with AWS Glue. Collaborate with data scientists and analysts to integrate data from multiple sources and...

  • Data Architect

    2 semanas atrás

    Brasil beBeeDataArchitect Tempo inteiro R$90.000 - R$150.000

    About the PositionA skilled Data Architect with advanced expertise in Snowflake, Databricks, and Azure (Terraform) is required to design, develop, and maintain data solutions for efficient data processing.Key ResponsibilitiesDevelop scalable data pipelines to ensure seamless data flow across systems.Collaborate with cross-functional teams to drive technical...

  • Data Architect

    2 semanas atrás

    Brasil beBeeData Tempo inteiro R$119.000 - R$1.545.000

    Job Title: Data ArchitectWe are seeking a skilled Data Architect to design and build data engineering components and pipelines to meet our organization's data processing requirements.This role involves developing strong problem-solving skills and navigating solutions with minimal or no help.You will be responsible for executing tasks related to...

  • Lead Data Pipeline Architect

    2 semanas atrás

    Brasil beBeeData Tempo inteiro US$120.000 - US$160.000

    Job OverviewPrimary Responsibilities: Develop high-performance data pipelines for large-scale data sets, oversee the operation and maintenance of cloud-based data warehouses, collaborate with cross-functional teams to ensure seamless data engineering services.About You: 5+ years of hands-on experience deploying code in production environments, profound...

  • Chief Data Solutions Architect

    2 semanas atrás

    Brasil beBeeData Tempo inteiro R$80.000 - R$150.000

    Job Title: Data ArchitectA seasoned Data Architect with extensive expertise in Azure Databricks, Unity Catalog, PySpark, and Delta Live Tables is sought to lead the design, architecture, and delivery of cutting-edge data solutions.About the RoleWe are a leading provider of innovative data solutions, and we're looking for an experienced Data Architect to...

  • Databricks Data Architect

    2 semanas atrás

    Brasil Celebal Technologies Tempo inteiro

    Job Title: Solution Architect – Databricks Location: Remote Work 12 months About the Role Celebal Technologies is seeking an experienced Data Architect with deep expertise in Azure Databricks, Unity Catalog, PySpark, and Delta Live Tables . In this role, you will lead the design, architecture, and delivery of advanced data solutions,...

  • Data Architect

    3 semanas atrás

    Brasil dLocal Tempo inteiro

    Join to apply for the Data Architect role at dLocal Join to apply for the Data Architect role at dLocal Why should you join dLocal?dLocal enables the biggest companies in the world to collect payments in 40 countries in emerging markets. Global brands rely on us to increase conversion rates and simplify

  • Lead Data Architect

    3 semanas atrás

    Brasil WRS Health Tempo inteiro

    Join to apply for the Lead Data Architect role at WRS Health 2 days ago Be among the first 25 applicants Join to apply for the Lead Data Architect role at WRS Health Voted #1 EHR by PC Mag, WRS Health delivers a fully integrated cloud based EMR and practice management solutions to its clients. We bring solutions to physicians by providing constant...

  • Data Architect

    Há 5 dias

    Brasil beBeeData Tempo inteiro

    Job Title: Lead Data Architect Key Responsibilities: The project focuses on developing and implementing advanced data architectures to solve complex business problems. Designing scalable data systems for the ingestion, processing, and storage of large datasets in a cloud environment. Building performant data pipelines to transform raw data into actionable...