Senior Databricks Engineer

Experience: 8–10 Years

Location: Remote

Employment Type: Full-Time

Job Summary

We are seeking a highly experienced Senior Databricks Engineer to design, build, and optimize large-scale data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate will have deep expertise in big data processing, cloud architecture, and end-to-end data pipeline development, along with strong leadership skills to guide technical teams. 

Key Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL/ELT workflows using Databricks (PySpark, Spark SQL, Delta Lake)

• Architect and implement Lakehouse solutions integrating data lakes and data warehouses

• Lead the migration of legacy data platforms to Databricks on Azure/AWS/GCP

• Optimize Spark jobs for performance, cost, and scalability

• Implement data governance, security, and access control using Unity Catalog

• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements

• Design and manage CI/CD pipelines for Databricks notebooks and jobs

• Mentor junior engineers and provide technical leadership on best practices

• Troubleshoot performance bottlenecks and ensure high availability of data pipelines

• Work with structured, semi-structured, and unstructured data at scale

Required Skills & Qualifications

• 8–10 years of overall experience in data engineering/big data, with strong hands-on expertise in Databricks

• Proficiency in PySpark, Spark SQL, and Delta Lake

• Strong experience with at least one major cloud platform (Azure Databricks, AWS Databricks, or GCP)

• Solid understanding of data warehousing concepts, data modeling, and ETL/ELT design patterns

• Experience with Unity Catalog, Delta Live Tables, and Databricks Workflows

• Proficiency in Python and SQL

• Knowledge of Azure Data Factory (ADF) and Medallion Architecture

• Experience with orchestration tools (Airflow, ADF, or similar)

• Familiarity with CI/CD tools (Git, Azure DevOps, Jenkins)

• Strong understanding of data governance, security, and compliance frameworks

• Excellent problem-solving, communication, and stakeholder management skills

Good to Have

• Databricks Certified Data Engineer Professional/Associate certification

• Experience with MLflow and machine learning pipelines

• Knowledge of Kafka or other streaming technologies

• Experience working in the US healthcare industry is a plus

Education

• Bachelor's/Master's degree in Computer Science, Information Technology, or a related field