Software Engineer/ Data Engineer
Ports North
Job Description
Responsibilities Develop and support Databricks notebooks, jobs, and workflows. Write and optimize PySpark and Python code for data processing. Assist in designing scalable and reliable data pipelines.
Apply Spark best practices (partitioning, caching, joins, file sizing). Work with Delta Lake tables and data models. Perform data validation and quality checks during development.
Support cluster configuration and sizing for development workloads. Identify performance bottlenecks early and recommend improvements. Collaborate with Data Engineers to prepare solutions for future production rollout.
Document development standards, patterns, and best practices. Skills Required Databricks & Spark Hands‑on experience with Databricks. Strong knowledge of PySpark and Spark fundamentals.
Experience working with Delta Lake. Understanding of Spark execution concepts (jobs, stages, tasks). Programming Strong Python development skills.
Ability to write clean, modular, and reusable code. Cloud & Azure (Preferred) Experience working in Microsoft Azure. Familiarity with Azure Databricks.
Familiarity with Azure Data Lake Storage (ADLS). Familiarity with Azure Blob Storage. Basic understanding of cloud resource usage and cost awareness.
Development Practices Experience using Git and version control. Familiarity with Databricks Repos or similar workflows. Ability to perform basic testing and validation of data pipelines.
Education & Work Experience 3+ years of hands‑on experience. Bachelor's degree required (Masters preferred). Title Software Engineer/ Data Engineer Location Alpharetta, GA Client Industry Insurance #J-18808-Ljbffr