Data Engineer

We are Hiring for an experienced Data Engineer to join our team in Sheffield, United Kingdom. The ideal candidate will have Strong Experience in Scala and/or Python for streaming apps, familiarity with testing frameworks and CI for stream processors. • Collaboration: Work closely with Kafka platform/admin teams while focusing on application-layer streaming logic; strong communication and documentation. Your responsibilities: Design and build Kafka-based streaming applications (Kafka Streams/ksqlDB) in Scala/Python for transformation, enrichment, and routing.

Implement end-to-end streaming pipelines: producers, stream processors, and consumers with strong data quality, idempotency, and DLQ patterns. Model topics, schemas, and contracts (Avro/Protobuf/JSON) and maintain backward/forward compatibility. Develop batch/stream interoperability: Spark/Structured Streaming jobs for aggregation, feature generation, and storage in Parquet/ORC.

Integrate processed data into analytics/observability platforms (e.g., Splunk) for dashboards, alerting, and proactive insights. Build automated validation, replay, and backfill mechanisms to ensure reliability and SLA adherence. Apply observability to the pipelines themselves (metrics, traces, structured logs) and tune performance/cost.

Collaborate with platform/infra teams who handle Kafka admin (brokers, security, ops) while owning application-side streaming logic. Ensure security and compliance for application data paths (authn/z, encryption in transit/at rest, secret management). Document data flows, schemas, and runbooks for streaming services.

Your Profile Essential skills/knowledge/experience: Kafka application development: Kafka Streams/ksqlDB, producer/consumer patterns, partitioning/serialization, exactly-once/at-least-once semantics. Languages: Strong in Scala and/or Python for streaming apps; familiarity with testing frameworks and CI for stream processors. Schema management: Avro/Protobuf/JSON, schema registry usage, compatibility strategies.

Stream/batch processing: Spark (including Structured Streaming), Parquet/ORC, partitioning/bucketing, performance tuning. Data quality and reliability: Idempotent processing, DLQs, replay/backfill, lineage, and SLA-aware designs. Observability: Metrics/tracing/logging for stream apps; integration with downstream dashboards/alerts.

Security/compliance: AuthN/Z in clients, TLS/SASL usage, secret management in code/services. Collaboration: Work closely with Kafka platform/admin teams while focusing on application-layer streaming logic; strong communication and documentation. #J-18808-Ljbffr