Cloud Data Engineer (Azure & Databricks)

This is your job:

We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The ideal candidate should have extensive experience in Azure Databricks, Data Lakehouse concepts, Python, SQL, and a strong background in working with various data sources and platforms.

Azure Databricks:

-Design, develop, and maintain data pipelines using Azure Databricks.

-Optimize and tune Spark jobs for performance and scalability.

-Implement Spark DataFrames and Pandas DataFrames for efficient data processin

Data Lakehouse Concept:

-Implement and manage data lakehouse concepts for effective data storage and processing.

-Work with Delta Tables and Delta Live Tables to maintain versioned and transactional data.

Azure Data Factory:

-Design and implement data workflows using Azure Data Factory.

-Handle data integration from various sources, both on-premises and in the cloud.

Data Sources:

-Integrate data from diverse sources such as Azure Data Lake, Azure SQL, Azure Synapse, APIs, and other relevant sources.

-Expertise in Copy Activity, Mapping Data Flow, and handling various data formats.

API Integration:

-Experience with making API calls and processing JSON responses.

-Ability to handle pagination and complex API interactions.

-Azure Analysis Services (SSAS)

-Experience with Azure SSAS for creating and managing tabular models.

-Azure Key Vault and Data Lake Concepts:

-Familiarity with Azure Key Vault for secure storage of sensitive information.

-Knowledge of Azure Data Lake concepts and best practices.

Additional Skills:

-The candidate should be proficient in SQL and understand the usage of functions and stored procedures for efficient data handling. He or she should possess a deep understanding of database design and normalization and have experience in database and query performance tuning and optimization.

-Familiarity with dbt (Data Build Tool) for analytics engineering (optional but advantageous).

Your Qualifications:

-Bachelor’s or higher degree in Computer Science, Data Science, or related field.

-Proven experience as a Data Engineer with a focus on Azure technologies.

-Strong proficiency in Python, SQL, and Spark DataFrames.

-Excellent understanding of data integration, data processing, and data modeling concepts.

-Experience with cloud and on-premises data sources and platforms.

All applications will be treated in strict confidentiality

Please note that only shortlisted candidates will be invited to an interview