Lead Data Architect – Azure Databricks

Lead Data Architect – Azure Databricks & Advanced Analytics

Role Overview

We are seeking an experienced Lead Data Architect with deep expertise in Azure Databricks and advanced analytics to design and deliver scalable, cloud-native data solutions. This role requires strong hands-on technical skills combined with architectural leadership to drive end-to-end analytics platforms aligned with business objectives.

The ideal candidate will bring extensive experience in big data technologies, data engineering, and Azure cloud services, with a strong focus on Databricks, SQL, and modern data lakehouse architectures.

Key Responsibilities

• Lead the architecture, design, and implementation of advanced analytics solutions using Azure Databricks and Microsoft Fabric
• Collaborate closely with business stakeholders and cross-functional IT teams to gather requirements and translate them into effective data solutions
• Own end-to-end delivery of data platforms, ensuring alignment with business needs, architectural standards, and best practices
• Design and develop scalable data pipelines and ETL/ELT processes using Azure Databricks, PySpark, and related tools
• Integrate Azure Databricks with other Azure services, including Azure Data Lake, Azure Synapse, Azure Data Factory, and on-premise systems
• Provide technical leadership and mentorship to data engineering teams, fostering continuous learning and innovation
• Ensure comprehensive documentation of data architectures, pipelines, and data flows
• Enforce best practices for code quality, data security, governance, and scalability
• Stay current with evolving Databricks and Azure technologies to drive continuous improvement and innovation

Essential Technical Skills

• Strong hands-on experience with Azure Databricks, including cluster management, notebook development, and Delta Lake
• Proficiency in big data technologies such as Spark, Hadoop, and data processing frameworks like PySpark
• Deep understanding of Azure services including Azure Data Lake Storage (ADLS Gen2), Azure Synapse, and Azure Data Factory
• Experience designing and implementing ETL/ELT pipelines, data lakes, and data warehouses
• Advanced SQL skills and familiarity with NoSQL databases
• Experience with CI/CD pipelines and version control systems such as Git
• Strong knowledge of cloud security and governance best practices

Soft Skills

• Excellent communication skills with the ability to explain complex technical concepts to non-technical stakeholders
• Strong problem-solving mindset with a proactive approach to identifying and resolving issues
• Proven leadership and mentoring skills, with experience leading and guiding data engineering teams

Nice to Have Skills

• Experience with Power BI for dashboarding and reporting
• Exposure to Microsoft Fabric for analytics and integration workloads
• Knowledge of Spark Streaming for real-time data processing
• Familiarity with Azure Resource Manager (ARM) templates and Infrastructure as Code (IaC) practices

Experience Requirements

• 12+ years of demonstrated experience in developing data ingestion and transformation pipelines using Databricks, Azure Synapse notebooks, and Azure Data Factory
• Strong hands-on experience with Delta Tables, Delta Lake, and Azure Data Lake Storage Gen2
• Proven expertise using Auto Loader and Delta Live Tables for efficient data ingestion and transformation
• Experience building and optimizing query layers using Databricks SQL
• Demonstrated experience integrating Databricks with Azure Synapse, ADLS Gen2, and Power BI for end-to-end analytics solutions
• Prior experience developing, optimizing, and deploying Power BI reports
• Solid understanding of modern CI/CD practices, particularly for Databricks and cloud-native architectures