Big Data Engineer

Job Opportunity

As a data engineer, you will play a crucial role in building innovative big data technologies and platforms to drive business impact. We are seeking a talented individual who can collaborate with partners and stakeholders to understand customer needs and develop effective solutions.

The data team is responsible for designing, testing, coding, and implementing new data products, systems, platforms, and pipelines of small to medium complexity. You will learn to measure, inspect, and drive decisions using data. In addition, you will develop and maintain CI/CD automation using tools like GitHub Actions and implement Infrastructure as Code (IaC) with tools like Terraform for cloud-based data infrastructure provisioning and management.

Key Responsibilities:

1. Collaborate with partners and stakeholders to understand customer business and key challenges.
2. Design, test, code, and implement new data products, systems, platforms, and pipelines of small to medium complexity.
3. Learn to measure, inspect, and drive decisions using data.
4. Develop and maintain CI/CD automation using tools like GitHub Actions.
5. Implement Infrastructure as Code (IaC) with tools like Terraform for cloud-based data infrastructure provisioning and management.
6. Practice software development with TDD and BDD, Microservice, and event-oriented architectures.
7. Analyze data to understand customer and business problems and develop effective solutions.
8. Explore new technologies and innovative approaches to improve efficiency and productivity.
9. Support live data products/systems, promote proactive monitoring, high data quality, rapid incident response, and continuous improvement through automation.
10. Identify bottlenecks and opportunities for improvement within systems and processes.
11. Learn from peers and foster continuous learning within the team.
12. Understand team processes and best practices, applying them to tasks to solve customer/business problems effectively and sustainably.
13. Partner with team members for development and problem-solving.
14. Take ownership and be proactive in your work.
15. Seek reviews from senior engineers to ensure quality.
16. Build reliable, secure, high-quality, efficient, and user-friendly big data platforms and tools at scale.
17. Develop systems, platforms, data pipelines, and tools for the entire data lifecycle, including ingestion, cleaning, processing, enrichment, and delivery, leveraging the Data platform.
18. Learn and master big data technologies, tools, and software, ensuring proper integration and execution.
19. Design and implement efficient and user-friendly data models and structures using data modeling techniques.

Required Qualifications:

• Bachelor's degree in Computer Science, Software Engineering, or a related field, or demonstrable equivalent experience and understanding.
• 1 year of software engineering experience is a plus.
• Strong problem-solving, communication, and collaboration skills.
• Highly self-motivated and eager to learn, with a proactive approach to exploring new technologies like GenAI for productivity and quality improvements.
• Ability to design solutions for small to medium problems or components.
• Strong Python skills, including coding, testing, and monitoring automation.
• Passionate about data, big data technology, and cloud computing.
• Interest in understanding and solving customer and business problems.
• Some knowledge of data ingestion, cleaning, processing, enrichment, serving, and quality assurance techniques, including data pipelines, SQL, relational databases, and ELT processes.
• Knowledge of data warehousing concepts and dimensional modeling is a plus.
• Experience with ETL tools like DAGs and Airflow is beneficial.