Azure Data Engineer-Databricks /ADF
Are you a seasoned Data Analyst with expert-level skills in Azure and Databricks? Apply Now!
Working with one of our top financial clients, this role calls for an Azure Data Engineer-Databricks /ADF. This position involves working within a transformative data environment focused on advancing Databricks functionality and supporting large-scale investment transaction processes for institutional clients.
Pay rate range (CAD): $86.96/hr – $104.35/hr
Responsibilities:
- Develop and maintain pipelines using Azure Data Factory (ADF), Databricks, and Azure Functions (C#)
- Build and optimize Spark/Pandas notebooks for data processing and analysis
- Configure and manage Unity Catalog for data governance and security in Databricks
- Migrate Databricks workspace from hive metastore to Unity Catalog, handling permissions and security configurations
- Design delta tables with proper schema, properties, and security controls
- Implement Delta Live Tables and related features to enhance data workflows (nice to have)
- Manage DevOps and deployment pipelines using Azure DevOps and GitHub Enterprise for both ADF and Databricks
- Work with Azure Entra ID for SSO and RBAC setup
- Deploy and manage Azure PaaS resources such as Azure SQL, ADF/SHIR, ADLS, and Synapse SQL databases
- Collaborate with a team of Data Engineers, Business Analysts, and Project Managers to meet project goals
- Work independently and lead small technical team sessions to troubleshoot and resolve issues
- Ensure compliance with data governance and security standards for highly regulated environments
Desired Skill-Set:
- 7+ years of experience with ADF, Databricks, Azure Functions (C#)
- Extensive development experience with Python/notebooks on Databricks/Spark/Pandas
- Expert knowledge of Unity Catalog and data governance setup
- Experience loading datasets from various sources using ADF with Medallion architecture
- Proven track record migrating Databricks from hive metastore to Unity Catalog and managing permissions
- Ability to design delta tables with security considerations
- Strong DevOps skills using Azure DevOps and GitHub Enterprise
- In-depth experience with Azure Entra ID for SSO/RBAC
- Proficiency working with Azure SQL PaaS, ADLS, Synapse, and other Azure services
- Familiarity with working in a highly regulated environment
- Ability to work independently, lead a team, and solve complex technical problems
- Financial services industry experience (asset)
- Excellent communication skills
Nice to Have:
- Previous experience working with financial institutions
- Knowledge of Delta Live Tables
- Experience with Azure Synapse Analytics
- Familiarity with Windows environments in regulated industries
BeachHead is an equal opportunity agency and employer. We advocate for our candidates and welcome applicants regardless of race, color, religion, national origin, sex, age, or physical or mental disability. BeachHead or our clients may use technology-enabled tools, including automation and artificial intelligence (AI), to support parts of the recruitment process such as resume screening, application management, and candidate matching. These tools assist our recruiters and our clients, and do not replace human decision-making. This job posting represents a current or anticipated vacancy. The position may be filled at any time, and the posting may be removed without notice once the role has been filled.