Senior Data Engineer

Full Time

Job Description
Welcome to GEP Who We Are GEP, with over 30 offices internationally, is the fastestgrowing procurement and supply chain solutions firm
- consulting, software andmanaged services. GEP has succeeded by being smart and creative; solvingcomplex problems and finding opportunities for the world’s largestorganizations. How You Grow at GEP We recognize people who make a genuine difference, workhard and drive achievements. Results are noticed and rewarded. It’s how youwill grow a career at GEP, and in a much shorter time frame than at otherfirms. Celebrating Everyone GEP succeeds through the ideas and creativity of our teammembers so we embrace people of all experiences, nationalities, abilities,cultures, races, gender identities, sexual orientations and ages. What makesyou unique and different is celebrated and will help GEP stand out even more.
And we are a women-founded and -owned company so our foundation is making GEP agreat place to work for women, a place where women can learn, advance and giveback. What You Will Do Lead the design, development, and optimization of scalable data solutions on Databricks using Python, SQL, and PySpark. Architect lakehouse solutions based on Delta Lake and medallion architecture, defining data models, layer boundaries, and performance strategies. Define and govern data assets in Unity Catalog, including catalog and schema structure, access permissions, and lineage. Design and build batch and incremental pipelines with Lakeflow Jobs and Lakeflow Declarative Pipelines, including MERGE/CDC patterns and data-quality expectations. Own CI/CD for our Databricks solutions, packaging and deploying jobs and pipelines across environments with Databricks Asset Bundles (DABs), Git, and automated testing. Establish data-quality, observability, and data-contract practices that ensure the accuracy, consistency, and agreed service levels of business-critical datasets. Optimize Databricks workloads for performance and cost, including compute selection (classic and serverless), Spark tuning, and Delta table maintenance. Provide technical leadership in architectural decisions, code reviews, and the troubleshooting of critical pipeline and platform issues. Mentor junior and mid-level engineers, fostering a culture of learning and technical excellence, and take part in technical interviews and assessments. Collaborate with analysts, product managers, and business stakeholders to translate business needs into data solutions and communicate technical trade-offs. What Should You Bring Expert-level hands-on experience with Databricks, including Lakeflow Jobs, Lakeflow Declarative Pipelines, Spark DataFrames, and Delta Lake. Deep understanding of Spark and distributed data processing, including partitioning, joins, shuffles, and performance tuning. Expert-level Python and SQL skills for data engineering, including PySpark, Pandas, and writing testable, production-grade code. Strong experience with Unity Catalog governance, including catalog design, access control, and lineage. Strong experience with CI/CD for data solutions, including Databricks Asset Bundles (DABs), Git-based workflows, and automated testing (e.g., pytest). Solid knowledge of data modeling, lakehouse and medallion architecture, incremental processing (MERGE, CDC), and Delta Lake optimization. Knowledgedesigning data-quality frameworks, observability, and data contracts for production pipelines. Expertise with Azure Databricks is strongly preferred. Experience with AWS, Google Cloud, or another cloud platform is also valuable. Familiarity with AI-assisted development tools and Databricks AI Functions is a plus. Kkoeledge with web scraping tools and techniques (e.g., BeautifulSoup, Scrapy, Selenium) is a plus. Knowledge of procurement, public procurement, or foreign trade (import/export) data is a plus. Proven ability to lead projects in agile environments, balancing technical and business priorities. Excellent communication and mentorship skills to guide engineers and present solutions to stakeholders. Are You One of Us GEP is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, ethnicity, color, national origin, religion, sex, protected veteran status, disability status, or any other characteristics protected by federal, state or local law. We are committed to hiring and valuing a global diverse work team. GEP is proud to be an EEO/AA employer M/F/D/V. GEP's strength is its people. As a leader in procurement and supply chain solutions, we are committed to providing innovative solutions that create a competitive advantage for enterprises worldwide. Our leadership helps us maintain this focus and drives our commitment to deliver the highest value to our clients. GEP's leadership is a collective of some of the most experienced and accomplished professionals in the industry. Blending together strong business acumen with