Sr. Data Engineer – Databricks
Are you a highly skilled Data Engineer with expertise in Databricks ready to lead a major data platform modernization? Apply Now!
Working with one of our top financial clients, this role calls for a Sr. Data Engineer – Databricks specializing in Databricks and Data Platform Migration. This position offers a compelling opportunity to work on a large-scale initiative to transition trading and market data from legacy systems into modern, scalable data frameworks, supporting strategic modernization efforts within Capital Markets.
Responsibilities:
- Design and develop data ingestion pipelines, both batch and real-time, leveraging Databricks platform.
- Build and optimize Spark/Scala-based data processing workflows for high performance and reliability.
- Implement streaming ingestion solutions using Kafka, APIs, or similar technologies.
- Develop medallion architecture models (bronze, silver, gold layers) consistent with best practices.
- Create and maintain materialized views and other data structures to facilitate efficient data consumption.
- Ensure data quality through validation, deduplication, and reconciliation processes.
- Contribute to the development of reusable frameworks and engineering patterns for data ingestion and processing.
- Collaborate with internal technical teams to gather requirements and deliver scalable solutions.
- Support ongoing business-as-usual (BAU) data engineering tasks while participating in platform buildout initiatives.
Desired Skill-Set:
- Extensive hands-on experience with Databricks as a Subject Matter Expert (SME).
- Advanced proficiency in Apache Spark and Scala.
- Proven ability to develop data pipelines (batch and streaming).
- Knowledge of real-time ingestion frameworks such as Kafka or APIs.
- Strong understanding of data architecture, pipeline design, and data modeling best practices.
- Experience with data quality, validation, and reconciliation workflows.
- Ability to interpret complex technical requirements and translate into practical solutions.
- Effective communication skills to liaise with technical stakeholders.
- Delivery-focused mindset with pragmatic approach to engineering challenges.
Nice to Have:
- Capital Markets experience, particularly within FICC.
- Exposure to trading systems, market data, or risk management data environments.
BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability. BeachHead or our clients may use technology-enabled tools, including automation and artificial intelligence (AI), to support parts of the recruitment process such as resume screening, application management, and candidate matching. These tools assist our recruiters and our clients, and do not replace human decision-making. This job posting represents a current or anticipated vacancy. The position may be filled at any time, and the posting may be removed without notice once the role has been filled.