Lead Data Architect – Azure Databricks

Lead Data Architect – Azure Databricks & Advanced Analytics

Role Overview

We are seeking an experienced Lead Data Architect with deep expertise in Azure Databricks and advanced analytics to design and deliver scalable, cloud-native data solutions. This role requires strong hands-on technical skills combined with architectural leadership to drive end-to-end analytics platforms aligned with business objectives.

The ideal candidate will bring extensive experience in big data technologies, data engineering, and Azure cloud services, with a strong focus on Databricks, SQL, and modern data lakehouse architectures.

Key Responsibilities

- Lead the architecture, design, and implementation of advanced analytics solutions using Azure Databricks and Microsoft Fabric
- Collaborate closely with business stakeholders and cross-functional IT teams to gather requirements and translate them into effective data solutions
- Own end-to-end delivery of data platforms, ensuring alignment with business needs, architectural standards, and best practices
- Design and develop scalable data pipelines and ETL/ELT processes using Azure Databricks, PySpark, and related tools
- Integrate Azure Databricks with other Azure services, including Azure Data Lake, Azure Synapse, Azure Data Factory, and on-premise systems
- Provide technical leadership and mentorship to data engineering teams, fostering continuous learning and innovation
- Ensure comprehensive documentation of data architectures, pipelines, and data flows
- Enforce best practices for code quality, data security, governance, and scalability
- Stay current with evolving Databricks and Azure technologies to drive continuous improvement and innovation

Essential Technical Skills

- Strong hands-on experience with Azure Databricks, including cluster management, notebook development, and Delta Lake
- Proficiency in big data technologies such as Spark, Hadoop, and data processing frameworks like PySpark
- Deep understanding of Azure services including Azure Data Lake Storage (ADLS Gen2), Azure Synapse, and Azure Data Factory
- Experience designing and implementing ETL/ELT pipelines, data lakes, and data warehouses
- Advanced SQL skills and familiarity with NoSQL databases
- Experience with CI/CD pipelines and version control systems such as Git
- Strong knowledge of cloud security and governance best practices

Soft Skills

- Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders
- Strong problem-solving mindset with a proactive approach to identifying and resolving issues
- Proven leadership and mentoring skills, with experience leading and guiding data engineering teams

Nice to Have Skills

- Experience with Power BI for dashboarding and reporting
- Exposure to Microsoft Fabric for analytics and integration workloads
- Knowledge of Spark Streaming for real-time data processing
- Familiarity with Azure Resource Manager (ARM) templates and Infrastructure as Code (IaC) practices

Experience Requirements

- 12+ years of demonstrated experience in developing data ingestion and transformation pipelines using Databricks, Azure Synapse notebooks, and Azure Data Factory
- Strong hands-on experience with Delta Tables, Delta Lake, and Azure Data Lake Storage Gen2
- Proven expertise using Auto Loader and Delta Live Tables for efficient data ingestion and transformation
- Experience building and optimizing query layers using Databricks SQL
- Demonstrated experience integrating Databricks with Azure Synapse, ADLS Gen2, and Power BI for end-to-end analytics solutions
- Prior experience developing, optimizing, and deploying Power BI reports
- Solid understanding of modern CI/CD practices, particularly for Databricks and cloud-native architectures