High-Performance Data Engineer

Há 4 dias

Brasil beBeeData Tempo inteiro R$80.000 - R$120.000
Job Description

We are seeking a skilled Data Engineer to lead the development and deployment of our high-performance data service module. This role focuses on implementing efficient data storage and retrieval systems using HDF5 or similar technologies, with parallel and concurrent I/O capabilities.

Key Responsibilities
  • Design and implement the data service module using HDF5 for optimal data storage and retrieval.
  • Develop and optimize parallel and concurrent I/O mechanisms to maximize performance for large-scale datasets.
  • Ensure seamless integration with HPC and visualization workflows.
Performance Optimization Strategies
  • Optimize I/O operations for CPU/GPU-based workflows to minimize bottlenecks.
  • Implement caching, compression, and other strategies to enhance performance.
Data Management and Integrity
  • Design suitable data structures and schemas for storing 3D grid data and other simulation outputs.
  • Ensure data integrity and consistency during concurrent read/write operations.
Testing and Validation
  • Develop and execute test cases to validate module performance and reliability under various load conditions.
  • Conduct benchmarking to ensure scalability across different hardware configurations.
Documentation and Support
  • Document the architecture, APIs, and usage guidelines for the data service module.
  • Provide technical support to the development and visualization teams for data integration.
RequirementsEducation

Bachelor's or Master's degree in Computer Science, Software Engineering, or related fields is required.

Experience
  • At least 3 years of experience in developing and deploying data services for HPC or similar systems is necessary.
  • Proven expertise with HDF5 or similar, in parallel I/O operations is essential.
Technical Skills
  • Programming: Strong proficiency in at least one programming language such as C++, Python, GoLang, or Fortran is required.
  • HDF5 Expertise: In-depth knowledge of HDF5 APIs and advanced features like parallel HDF5 is necessary.
  • Parallel I/O: Experience with MPI I/O, POSIX I/O, or similar frameworks for concurrent/parallel data access is essential.
  • Performance Optimization: Skills in profiling and optimizing I/O operations for large datasets are necessary.
  • Proficiency in SQL and experience with any RDMS is required.
  • Knowledge of at least one orchestration and scheduling tool is a plus.
Soft Skills
  • Strong problem-solving skills and ability to work in a multidisciplinary team is required.
  • Excellent communication skills for cross-team collaboration and documentation is essential.
Benefits
  • Flexible working format - remote, office-based or flexible.
  • A competitive salary and good compensation package.
  • Personalized career growth.
  • Professional development tools.
  • Active tech communities with regular knowledge sharing.
  • Education reimbursement.
  • Memorable anniversary presents.
  • Corporate events and team buildings.
  • Other location-specific benefits.
Employment Details
  • Seniority level: Not Applicable.
  • Employment type: Full-time.
  • Job function: Information Technology and Engineering.
  • Industries: Software Development.

  • High-Performance Data Engineer

    Há 6 dias

    Brasil beBeeDataEngineering Tempo inteiro US$90.000 - US$120.000

    Job OpportunityWe are seeking a Data Engineer to join our team in Portugal. As a leading provider of IT solutions, we offer a dynamic and challenging work environment that will allow you to grow professionally.The ideal candidate will have a strong background in data engineering, with experience in Scala programming language and cloud-based systems. Fluency...

  • High-Performing Data Engineer Wanted

    Há 4 dias

    Brasil beBeeDataEngineering Tempo inteiro US$120.000 - US$150.000

    Data Engineering SpecialistA high-performing data engineer with expertise in designing, building, and maintaining robust data platforms and pipelines is sought after to drive strategic decision-making. The ideal candidate will have 7+ years of experience in data engineering roles, specializing in technologies like Big Data (Spark, Hadoop, Hive, MapReduce),...

  • High-Performance Engineering Leader

    Há 5 dias

    Brasil beBeeEngineering Tempo inteiro US$80.000 - US$120.000

    Unlock Your Career PotentialWe are seeking a skilled Senior Reporting Engineer to join our dynamic team.This is an exciting opportunity for a talented engineer to build high-performance, scalable applications and drive business growth.As a Senior Reporting Engineer, you will be responsible for designing, developing, and delivering data analytics systems that...

  • Senior Data Engineer

    2 semanas atrás

    Brasil Pride Global Tempo inteiro

    We're Hiring: Senior Data Engineer (MLOps) | Remote from Brazil | Fluent English required | USD-Hourly pay Location : Remote – Brazil only Language : Fluent English required Are you passionate about building scalable data platforms and cutting-edge MLOps solutions? Do you want to work with a top-tier US company revolutionizing e-commerce and circular...

  • Data Engineer Specialist

    Há 3 dias

    Brasil beBeeData Tempo inteiro US$120.000 - US$150.000

    We're seeking a skilled professional to fill the role of Senior Data Engineer.This is a full-time opportunity, 100% remote, and exclusive to candidates located in Brazil. You'll be working with a fast-paced US-based company, leading innovations in sustainable commerce and data-driven operations.Job DescriptionThe ideal candidate will have strong expertise in...

  • Data Engineer

    1 semana atrás

    Brasil WebCreek Tempo inteiro

    Are you looking to join a highly regarded IT development firm? Well, look no further WebCreek is hiring a skilled Data Engineer with 3+ years of experience and a high level of English to work remotely from Latin America.The role focuses on building and maintaining scalable data pipelines and models using Azure Databricks, Spark, Python, and SQL. The ideal...

  • Data Engineer

    1 dia atrás

    Brasil Insight Global Tempo inteiro

    100% Remote Data Engineer1 year contract to start, high possibility of extensionsHiring this week - must be able to start a new opportunity in 2 weeks Must-Haves: 3+ Years of Data Engineering experience Advanced proficiency in SQL, with extensive experience writing complex queries. Intermediate level experience within Linux & Python.Linux

  • Data Engineer

    1 semana atrás

    Brasil WebCreek Tempo inteiro

    Are you looking to join a highly regarded IT development firm? Well, look no further WebCreek is hiring a skilled Data Engineer with 3+ years of experience and a high level of English to work remotely from Latin America. The role focuses on building and maintaining scalable data pipelines and models using Azure Databricks, Spark, Python, and SQL. The...

  • Data Engineer

    Há 3 dias

    Brasil Thaloz Tempo inteiro US$90.000 - US$120.000 por ano

    We are seeking a highly skilled and motivated Senior Data Engineer to join our Credit Platform Data team. In this pivotal role, you will be responsible for designing, building, and maintaining scalable and reliable data pipelines and ETL processes that empower our internal business units with efficient data processing and utilization. Your expertise will...

  • Data Engineer

    4 semanas atrás

    Brasil Launchpad Technologies Inc. Tempo inteiro

    Join to apply for the Data Engineer role at Launchpad Technologies Inc.Join to apply for the Data Engineer role at Launchpad Technologies Inc. Senior Data Engineer – Scalable Pipelines for AI Agent InfrastructureWe're looking for a Senior Data Engineer to design, build, and maintain the data foundation behind our next-generation AI agent platform. You'll...