Sr. Azure Data Engineer – ADF & Databricks
Are you a leader and learner? Would you like to see the impact of your work? Do you see yourself in a collaborative, challenging and result-oriented environment? Are you a peoples person? Then, Apply Now!
Working with one of the top financial clients, this role calls for a Sr. Azure Data Engineer – ADF & Databricks who will be responsible for migrating existing data sources from tactical setups to a strategic framework using ADF and Databricks, building ETL processes, and ensuring seamless integration with analytics platforms. This role demands deep expertise in Azure ecosystem services, Databricks management, data pipeline design, and strong problem-solving skills to enhance data accessibility and visualization.
Responsibilities
- Migrate current data sources from VM-based pulls to a strategic infrastructure leveraging ADF and Databricks.
- Develop and implement ETL processes to transfer data from SFTP, web services, and databases into Delta Lake.
- Establish secure connectivity between Plotly Dash and Databricks for real-time data visualization.
- Set up and manage Databricks environments, including Unity Catalog, clusters, schemas, and permissions.
- Create and maintain Delta table schemas with appropriate security and design standards.
- Manage Azure DevOps/GitHub pipelines, focusing on cluster/secret management and code deployments.
- Implement secure access using Azure Entra ID (SSO/RBAC).
- Collaborate with cross-functional teams to support data visualization and business intelligence needs.
Desired Skill Set
- 4–6 years’ experience with ADF, Databricks, and Azure Functions (C#).
- Strong development experience with Python/Notebooks on Databricks (Spark/Pandas).
- Expertise in loading datasets into Databricks using ADF with Medallion architecture.
- Experience converting Hive Metastore to Unity Catalog and configuring permissions.
- Knowledge of Azure PaaS services (Azure SQL PaaS, ADF/SHIR, ADLS, Synapse).
- Hands-on experience with DevOps & deployment pipelines (Azure DevOps/GitHub Enterprise).
- Strong skills in Delta table design, schema creation, and security setup.
- In-depth knowledge of Azure Entra ID for SSO/RBAC.
Nice to Haves
- Experience with Delta Live Tables.
- Exposure to highly regulated/locked-down Windows environments.
- Financial Services industry experience.
BeachHead is an equal opportunity agency and employer. We advocate for you and welcome anyone regardless of race, color, religion, national origin, sex, physical or mental disability, or age.