Staff Security Data Engineer

About The Role

Adobe’s Security Data Platform team builds and operates a petabyte-scale security data lakehouse that turns enterprise telemetry into trusted data for threat detection, investigations, compliance, and security analytics.

As a Staff Security Data Engineer, you will own full-system designs and lead critical engineering initiatives across ingestion, storage, governance, orchestration, and data serving. This is a hands‑on individual contributor role: you will write and review production code, build infrastructure, define test strategy, fix complex failures, and work with other engineers to deliver the architecture you help shape.

You will partner with Security Operations, Threat Intelligence, Compliance, Security Engineering, and infrastructure teams to resolve technical tradeoffs and deliver reliable, cost‑effective platform capabilities. Your impact will come through engineering delivery, sound architectural decisions, and helping others become stronger technical leaders.

Help us evolve our team from petabyte scale toward exabyte scale

Core responsibilities

Platform architecture and engineering

• Own end‑to‑end designs that shape the Security Data Platform’s fundamental architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog. Translate ambiguous needs into design proposals, implementation plans, and production capabilities.
• Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and new security data sources. Design for throughput, fault tolerance, replay, schema evolution, and sub‑minute freshness where use cases require it.
• Write maintainable Python and SQL, develop complex platform features, and refactor unreliable code. Establish reusable engineering patterns, lead code and design reviews, and own test strategy and automation choices across pipelines and infrastructure.
• Build and maintain tested, reusable Terraform modules for AWS and Databricks infrastructure, including S3, IAM, VPC networking, and Amazon Managed Workflows for Apache Airflow (MWAA). Automate delivery through version control and CI/CD.

Data modeling governance and operations

• Design layered lakehouse data models and reusable normalization patterns, including Open Cybersecurity Schema Framework (OCSF) mappings where appropriate. Preserve source fidelity and make datasets useful for investigations, analytics, and detection workloads.
• Establish data contracts, quality checks, cataloging, classification, lineage, access controls, and retention policies with data owners. Use Unity Catalog to make data ownership and governance traceable from raw events to analyst‑facing datasets.
• Engineer and operate Airflow and MWAA orchestration, including dependencies, retries, backfills, alerting, and automated recovery. Define service level objectives with consumers and instrument data freshness, completeness, pipeline health, and query performance.
• Lead diagnosis and resolution of complex production issues across Spark, storage, orchestration, and cloud infrastructure. Improve incident response, runbooks, recovery testing, and operational readiness to prevent recurring failures.

Performance and technical leadership

• Profile workloads and lead measurable cost and performance improvements through query and Spark tuning, storage lifecycle policies, partitioning, and compute right‑sizing. Establish baselines and dashboards that show savings alongside reliability, latency, and capacity.
• Lead cross‑team technical discussions, reconcile competing requirements, and build agreement on architecture and delivery priorities without formal management authority. Communicate decisions, landmarks, risks, and business impact proactively to your manager and partners.
• Partner with detection and security analytics engineers to deliver the data interfaces, governance, and pipeline capabilities needed for SIEM modernization and automated detection deployment.
• Mentor engineers through pairing, technical reviews, documentation, and knowledge sharing. Help others lead technical work, promote inclusive debate, and contribute to hiring decisions through technical assessment.
• Evaluate and introduce useful platform innovations, including AI‑assisted engineering and operational automation. Validate outputs, protect sensitive security data, measure improvements, and help partner teams adopt proven approaches.

Required Qualifications

• Substantial experience building and operating production data platforms, typically 10 or more years in data or software engineering, with demonstrated Staff‑level scope and impact. Equivalent depth and outcomes are welcome; a prior Staff or Principal title is not required.
• A record of personally delivering complex distributed