Data Engineer

Role summary We are looking for an Expert Software Engineer to design, build, and scale our next-generation Data Platform and Data-Driven APIs. This role combines distributed data processing (Apache Spark) with platform and microservices engineering (Java) to enable reliable, scalable, and real-time data access. You will operate at the intersection of data engineering and backend platform engineering-building systems that not only process large volumes of data but also expose that data through robust, well-designed APIs and services.

This role goes beyond implementing requirements. We expect engineers to understand business context, challenge assumptions, and take end-to-end ownership of delivering meaningful outcomes. Key responsibilities Data Platform Engineering Design and develop scalable data pipelines using Apache Spark (batch and streaming) Build and maintain data platform layers: ingestion, transformation, and serving Optimize Spark jobs for performance, cost, and reliability (partitioning, skew handling, memory tuning) Implement data quality, observability, and lineage frameworks Contribute to data architecture decisions (Lakehouse, data mesh, storage formats, partition strategies) Define and enforce data contracts and schema evolution practices Platform APIs & Backend Engineering Design and build data-driven platform APIs using Java (preferred) Develop microservices that expose curated datasets for product and partner consumption Implement RESTful APIs and event-driven services for real-time and near real-time data access Ensure low-latency, high-availability data serving layers Integrate with upstream/downstream systems, including legacy APIs where required Cloud & Platform Integration Build and deploy solutions on Azure (preferred) / AWS / GCP Leverage cloud-native services for data storage, compute, and messaging Work with event streaming systems (Kafka/Event Hubs) for real-time pipelines Support containerized deployments and orchestration (Kubernetes) where applicable Quality, Observability & Engineering Excellence Champion unit tests across both data and service layers Build automated validation frameworks for data pipelines Implement end-to-end observability (metrics, logging, tracing) across pipelines and APIs Drive CI/CD practices for both data and application code Conduct code reviews and enforce engineering best practices Product Mindset & Ownership Engage deeply with product and business stakeholders to understand why, not just what Translate business problems into scalable data and platform solutions Take end-to-end ownership from design through production and support Proactively identify performance bottlenecks, data issues, and system gaps Required qualifications (Hard requirements) 8+ years of software engineering experience with strong focus on data platforms and/or distributed systems Hands-on expertise in Apache Spark or Scala or PySpark Strong programming skills in Java (preferred) / Scala / Python Experience building large-scale data pipelines (ETL/ELT) Experience developing backend services or APIs (REST/microservices) Deep understanding of: Distributed systems (partitioning, shuffle, fault tolerance) Data storage formats (Parquet, ORC, Avro) Data modeling and schema evolution Experience with cloud platforms (Azure/AWS/GCP) Familiarity with workflow orchestration tools (Airflow, Dagster, etc.) Strong system design and performance optimization skills Preferred qualifications Experience with Spark Structured Streaming Exposure to Lakehouse architectures (Delta Lake, Iceberg, Hudi) Experience with event-driven architectures (Kafka, Event Hubs) Knowledge of data governance, catalog, and lineage tools Experience with CI/CD for data and microservices Familiarity with Kubernetes and containerized workloads Experience designing low-latency data serving APIs What success looks like A successful engineer in this role will: Deliver high-quality, production-grade data pipelines and APIs that power real business outcomes Build systems that are scalable, observable, and resilient underload Take ownership end-to-end, ensuring data flows reliably from source to consumer Balance data correctness, performance, and cost efficiency Contribute to evolving a modern data platform integrated with product-facing services