ML Big Data Engineer

- Working Hours: 13h00 to 22h00 (BRT) / PST-aligned schedule

- Language: Advanced English - Mandatory

Position Overview

We are looking for an experienced ML Big Data Engineer to design, develop, and maintain scalable Big Data and Machine Learning data services, leveraging modern table formats, distributed processing frameworks, Java backend development, AWS cloud infrastructure, and Kubernetes.

The professional will work on distributed data processing environments, focusing on performance, scalability, reliable API development, and integration of data services supporting analytics and Machine Learning applications.

The position requires strong hands-on experience with Apache Iceberg and/or Lance, distributed processing technologies such as Apache Spark or Apache Flink, and Java backend engineering.

Key Responsibilities

• Design, develop, and maintain scalable Big Data services supporting analytics and Machine Learning workloads.
• Work with modern table formats, including Apache Iceberg and/or Lance, to support efficient data storage and processing.
• Develop and optimize distributed data processing solutions using Apache Spark, Apache Flink, or similar frameworks.
• Design, implement, and maintain backend services using Java.
• Develop robust REST APIs to support data services and application integrations.
• Deploy and maintain applications and services using AWS cloud infrastructure.
• Work with Kubernetes for container orchestration and management of distributed applications.
• Investigate and troubleshoot production issues involving backend services and distributed data processing.
• Optimize the performance, reliability, and scalability of distributed systems.
• Collaborate with engineering teams to support the development and evolution of Big Data and ML data services.

Mandatory Requirements

• Strong hands-on experience with Big Data engineering for analytics and Machine Learning environments.
• Experience with Apache Iceberg and/or Lance table formats.
• Practical experience with Apache Spark, Apache Flink, or similar distributed data processing frameworks.
• Strong Java backend development experience.
• Experience designing and implementing REST APIs and backend services.
• Experience troubleshooting Java applications and backend services in production environments.
• Hands-on experience with Amazon Web Services (AWS).
• Practical experience with Kubernetes.
• Strong understanding of distributed systems, performance optimization, and scalability.
• Experience designing and implementing robust APIs.
• Advanced English for professional communication.
• Availability to work from 1:00 PM to 10:00 PM Brazil time, aligned with the PST time zone.

Preferred Requirements

• Experience with Apache Beam.
• Experience with Apache Airflow.
• Knowledge of Data Lake and Lakehouse architectures.
• Experience with version control systems.
• Knowledge of CI/CD practices and automated testing frameworks.
• Experience with Scala and/or Python in addition to Java.

Technical Environment

• Programming Languages: Java (Mandatory); Python and Scala (Preferred).
• Data Formats: Apache Iceberg and/or Lance.
• Distributed Processing: Apache Spark, Apache Flink, or equivalent frameworks.
• Backend Development: REST APIs and Java services.
• Data Architecture: Big Data, Data Lakes, and Lakehouse.
• Additional Tools: Apache Beam, Apache Airflow, version control, CI/CD, and testing frameworks.

Soft Skills and Behavioral Competencies

• Strong analytical and technical problem-solving capabilities.
• Ability to investigate and resolve complex production issues.
• Effective collaboration with engineering teams.
• Ability to analyze system performance and identify scalability improvements.
• Clear technical communication in English.

Further information will be given along the technical interview.