Senior Data Engineer (Airflow Specialist)

Job title: Senior Data Engineer (Airflow Specialist) Company: Hvantage Tech Solutions Pvt. Ltd. Department: Enterprise Applications / Healthcare IT Location: Remote / Hybrid / Onsite (as applicable) Employment Type: Full-Time Experience Required: 5+ Years Work Mode: Must be comfortable working with global teams and US clients Methodology: SAFe Agile (Mandatory) About Hvantage Tech Solutions Pvt.

Ltd Hvantage Tech Solutions Pvt. Ltd. is a global technology and healthcare IT services provider delivering advanced digital solutions for healthcare payers, providers, and life sciences organizations. Our expertise spans enterprise platforms, healthcare interoperability, AI-enabled analytics, cloud-native architectures, and digital member engagement platforms .

Role Overvi ewWe are seeking a versatile Senior Data Engineer to lead the design, implementation, and scaling of our enterprise data orchestration platform using Apache Airflow. This role requires a \"builder\" mindset, combining the agility to create robust pipelines from the ground up with the maturity to ensure they are scalable, secure, and integrated into a complex cloud ecosystem. As the primary architect for workflow automation, you will move beyond simple task scheduling to build a resilient, self-healing data infrastructur e.

Key Responsibili tiesOrchestration & Pipeline Engineer ing: Author complex, modular, and idempotent DAGs using Python and the Airflow TaskFlow API.Framework Developm ent: Build custom Operators, Hooks, and Sensors to standardize data integrations across the organizat ion.High-Compute Integrat ion: Integrate Airflow with heavy-compute environments including Azure Databricks, Spark clusters, and cloud data warehou ses.Platform Managem ent: Optimize Airflow executors (Celery or Kubernetes) to manage high-concurrency workloads and handle backend database tuning (PostgreSQL/MyS QL).Security & Complia nce: Implement enterprise-grade security protocols, including Role-Based Access Control (RBAC), Secret Management (Azure Key Vault/HashiCorp Vault), and OAuth integrat ion.DevOps & Reliabil ity: Build and maintain automated CI/CD deployment pipelines for DAGs and infrastructure-as-code using Terraform or Bi cep.Observabil ity: Implement advanced monitoring and alerting using Prometheus, Grafana, or the ELK stack to track pipeline health and SLA breac hes.Leadership & Mentors hip: Establish data engineering best practices across the team and conduct rigorous code reviews to ensure architectural integr ity. Technical Requir ementsOrchestr ation: Expert knowledge of Apache Airflow 3.x+ (specifically XComs, Providers, and Dynamic Task Map ping).Lang uages: Advanced proficiency in Python and expert-leve l SQL.Cloud Plat forms: Proficiency in Azure (Data Factory, Blob Storage, AKS) or equivalent services in AW S/GCP.D evOps: Hands-on experience with CI/CD tools such as GitHub Actions, Azure DevOps, or Je nkins. Qualif icationsExp erience: 5+ years in Data Engineering, with a minimum of 2-3 years focused specifically on scaling Airflow in production envir onments.Ed ucation: Bachelor’s or Master’s degree in Computer Science, Information Systems, or a relate d field.Preferred Skills:Experience migrating legacy schedulers to Airflow.Background in software development (Java o r .NET).Knowledge of data quality frameworks like Great Expectations or dbt.

Co mpetenciesStrong focus on long-term maintenance and architectural integrity.Ability to move fast in an agile environment without compromising on cod e quality.Exceptional problem-solving skills for debugging complex data orchestrati on issues. Applicat i on Process -Interested candidates may apply at: or WhatsApp at +9 1 7024041115