Data Engineer

Job Title: Senior Azure Data Engineer - Databricks Location: Toronto, ON Model: Hybrid (4 Days Onsite) Job Type: Full time/ Contract About the Role and Job: This job is for an existing, active vacancy and We are looking for a Senior Azure Data Engineer with strong expertise in Azure Databricks, Delta Lake, and Medallion Architecture to design and deliver scalable, high‑performance data platforms. This role will lead complex data engineering initiatives, influence architectural decisions, and partner closely with analytics, data science, and platform teams. Key Responsibilities Data Architecture & Leadership Design, implement, and evolve Medallion Architecture (Bronze, Silver, Gold) using Delta Lake on Azure Databricks.

Lead data architecture decisions, establish engineering standards, and define best practices. Review designs and code from team members, ensuring scalability, performance, and data quality. Data Engineering & Development Build and maintain high‑volume, fault‑tolerant ETL/ELT pipelines using Azure Databricks and Azure Data Factory.

Develop optimized PySpark transformations for large‑scale batch and streaming data. Implement reusable frameworks for ingestion, validation, transformation, and enrichment. Data Ingestion & Streaming Architect and develop ingestion pipelines for structured, semi‑structured, and unstructured data.

Implement batch and real‑time streaming pipelines using Azure Event Hubs / Kafka / Structured Streaming. Support incremental loads and CDC‑based data ingestion strategies. Performance, Optimization & Reliability Optimize Spark workloads using partitioning, caching, Z‑Ordering, and file sizing strategies.

Tune Delta Lake tables for performance and cost efficiency. Monitor pipeline health, troubleshoot failures, and improve data platform reliability. Cloud, Security & Governance Leverage Azure services including ADLS Gen2, Synapse Analytics, Key Vault, Azure Monitor, and networking/security patterns.

Implement role‑based access control, secrets management, and data protection best practices. Align data solutions with governance, compliance, and auditing requirements. Collaboration & Mentorship Work closely with data scientists, analysts, product owners, and business stakeholders.

Enable advanced analytics and machine learning workloads. Mentor junior and mid‑level data engineers and contribute to team skill growth. Required Skills & Qualifications 7+ years of data engineering experience with strong Azure cloud expertise.

Hands‑on expertise with Azure Databricks, Delta Lake, and Medallion Architecture. Advanced proficiency in PySpark and Python. Strong experience with Azure Data Factory for orchestration.

Deep understanding of data modeling, data lakes, and big data processing. Proven ability to design scalable, production‑grade data platforms. Strong communication skills and experience working in agile environments.

Preferred Qualifications Experience implementing CI/CD pipelines for data platforms (Azure DevOps, GitHub Actions). Exposure to streaming technologies (Kafka, Event Hubs, Structured Streaming). Familiarity with data governance, cataloging, and lineage tools (e.g., Azure Purview).

Experience supporting ML and advanced analytics workloads. Prior experience in financial services, healthcare, retail, or large‑scale enterprise environments.