Azure Data Engineer-DataBricks
Are you an experienced Data Analyst ready to lead transformative projects? Apply Now!
Working with one of our top financial clients, this role calls for a Azure Data Engineer-DataBricks. This position involves working within a dynamic team to support the ongoing transformation journey, leveraging advanced Databricks functionalities to optimize data strategies and ensure regulatory compliance in a high-security environment.
Responsibilities
- Develop and orchestrate ADF pipelines integrating Databricks, ensuring seamless data flow across platforms.
- Design, implement, and optimize Databricks notebooks using Python, Spark, and pandas for large-scale data processing.
- Configure and manage Unity Catalog, including setup, permissions, and data governance policies.
- Migrate Databricks workspace from hive metastore to Unity Catalog with appropriate permission management.
- Create and maintain Delta Lake tables with proper schema design, security properties, and performance optimizations.
- Utilize Delta Live Tables for data processing, where applicable.
- Implement DevOps processes and CI/CD pipelines for ADF and Databricks using Azure DevOps and GitHub Enterprise.
- Configure Azure Entra ID for SSO and RBAC to ensure secure, role-based access.
- Manage deployment to Azure PaaS resources such as Azure SQL PaaS, ADLS, Synapse, and related services.
- Work independently and lead a small data team, providing technical guidance and problem-solving expertise.
- Ensure compliance with data governance, security standards, and industry regulations.
- Support the ongoing evaluation and enhancement of data functionality for advanced business needs.
Desired Skill-Set
- 7+ years experience with ADF, Databricks, and Azure Functions (C#).
- Strong Python/notebook expertise on Databricks/Spark/Panda.
- Expert knowledge of Unity Catalog and data governance setup.
- Experience loading datasets from various sources using ADF to Databricks with Medallion architecture.
- Proficiency in converting Databricks workspace from hive metastore to Unity Catalog and setting permissions.
- Experience creating delta tables with security and performance considerations.
- Familiarity with Delta Live Tables, DevOps, CI/CD pipelines, and security.
- Strong understanding of Azure Entra ID, SSO, and RBAC configurations.
- Hands-on experience with Azure SQL PaaS, ADLS, and Synapse.
- Ability to work in highly regulated environments, with minimal supervision.
- Proven team leadership and problem-solving skills.
- Financial services industry experience is an asset.
- Excellent communication skills.
Nice to Have
- Previous experience working with financial institutions.
- Knowledge of Medallion architecture and advanced performance tuning.
- Experience working within a highly secured, lockdown Windows environment.
- Prior exposure to advanced data governance tools and processes.
BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability. BeachHead or our clients may use technology-enabled tools, including automation and artificial intelligence (AI), to support parts of the recruitment process such as resume screening, application management, and candidate matching. These tools assist our recruiters and our clients, and do not replace human decision-making. This job posting represents a current or anticipated vacancy. The position may be filled at any time, and the posting may be removed without notice once the role has been filled.