Job Summary
We are seeking an experienced Senior Databricks Developer with 5 to 8 yrs of experience in designing and implementing scalable data engineering solutions. The ideal candidate will have strong expertise in big data processing, data pipeline development, and cloud-based data platforms, with hands-on experience using Databricks and distributed data processing frameworks like Apache Spark.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks.
- Build and optimise large-scale data processing solutions using Apache Spark.
- Develop and manage ETL/ELT workflows for ingesting, transforming, and processing structured and unstructured data.
- Write data processing scripts using Python, Scala, and SQL.
- Implement and manage data lake and data warehouse architectures.
- Optimise performance of data pipelines and Spark jobs.
- Integrate Databricks with cloud data services and storage platforms.
- Collaborate with data scientists, analysts, and engineering teams to deliver reliable data solutions.
- Implement best practices for data governance, data quality, and security.
- Troubleshoot data pipeline issues and ensure system reliability and scalability.
- Document data architecture, workflows, and operational processes.
Required Skills & Qualifications
- 5 to 8 years of experience in data engineering or big data development.
- Good hands-on experience with Databricks.
- Extensive experience with distributed data processing using Apache Spark.
- Proficiency in Python, Scala, and SQL.
- Strong experience building scalable ETL/ELT data pipelines.
- Experience working with large datasets in data lake or data warehouse environments.
- Experience with cloud platforms such as Microsoft Azure, Amazon Web Services, or Google Cloud Platform.
- Strong knowledge of data modelling, data architecture, and data integration concepts.
- Excellent analytical, troubleshooting, and performance optimisation skills.