Shadab Siddiqui

Data Engineer

Pune, Maharashtra, India5 yrs 1 mo experience

Most Likely To SwitchAI ML Practitioner

Key Highlights

Expert in Azure Data Engineering and Machine Learning.
Proven track record in optimizing data pipelines.
Strong leadership in cross-functional project delivery.

Stackforce AI infers this person is a Data Engineering expert with a focus on AI-driven solutions in the cloud.

Contact

shadab0799@gmail.com LinkedIn

Skills

Core Skills

Azure Data EngineeringMachine LearningData Science

Other Skills

Amazon Web Services (AWS)Apache SparkArtificial Intelligence (AI)Azure Data FactoryAzure Data Lake StorageBig DataChatGPTCommunicationData analysisData cleansingData standardizationDatabase Management System (DBMS)DatabricksDjangoETL pipelines

About

Results-driven Data Science and Engineering professional with 5 years of experience designing and optimizing scalable data pipelines, machine learning solutions, and advanced analytics workflows. A Databricks and 3× Microsoft Certified expert skilled in Machine Learning, Databricks, Azure Data Factory, and Azure cloud services. Proficient in ETL development, data modeling, and workflow automation, with end-to-end experience across the data science lifecycle from data acquisition and feature engineering to model deployment and monitoring. Adept at leading cross-functional projects and teams to deliver high-quality, business-driven solutions. Passionate about solving complex data challenges, enabling data-driven decision-making, and driving innovation through cloud and AI technologies. Core Competencies: Databricks | Azure Data Engineering | Azure Data Factory | Machine Learning Pipelines| Leadership | Technical Management | Requirements Gathering | Design & Architecture

Experience

Michelin

Assistant Manager - Azure Data Engineer

Sep 2023 – Present · 2 yrs 6 mos · Pune, Maharashtra, India · On-site

Collected structured and semi-structured data from diverse sources and developed ETL pipelines to cleanse, transform, and align raw data with business requirements.
Designed and optimized ETL pipelines using Databricks, PySpark, and Azure Data Factory, improving data ingestion speed by 80%.
Developed and implemented the architecture for access management across Power BI.
Automated daily and monthly data resets with quality checks and alerting, reducing manual intervention by 90%.
Designed and implemented an architecture for a data quality dashboard that monitors data issues.
Provided Level 3 support for a streaming data pipeline that ingests data from multiple Kafka topics, performing data cleaning, transformation, and aggregation to create dashboards.
Experienced in the full data science lifecycle, from data acquisition and feature engineering to model deployment and monitoring.
Contributed to a project focused on developing a Generative AI tool for analyzing research PDFs, enabling users to answer questions about extensive content.

ETL pipelinesDatabricksPySparkAzure Data FactoryPower BIKafka+2

Zapilio

Subject Matter Expert

Jul 2022 – Dec 2022 · 5 mos · Bengaluru, Karnataka, India · Remote

Accenture

Azure Data Engineer

Jan 2022 – Sep 2023 · 1 yr 8 mos · Mumbai, Maharashtra, India · On-site

Collecting the structured and semi-structured data from the different sources and after collecting the raw data, running the ETL pipelines to cleanse the data and convert it as per the business need.
Responsible for data ingestion, data cleansing, data standardization and data transformation.
Implementing various spark transformation and actions using Databricks on raw data as per the business requirement.
Involved in extracting data from traditional RDBMS to ADLS using Azure data factory.
Collaborated with infrastructure, network, database and BI teams to ensure data quality and availability.
Daily interaction with client stakeholders with the excellent interpersonal skills to gather and groom user story requirements and propose optimal technical solutions for the business, proactively covering all use cases.
Involved in the development of a well optimized ADF pipeline which has reduced the execution time by 30%.
Designing the workflow for sending the mail by using the logic apps in Azure.
Conducted KT sessions for new hires and ensured timeline oriented detailed documentation.

ETL pipelinesDatabricksAzure Data FactoryData cleansingData standardizationAzure Data Engineering+1

Web angel tech

Data Engineer

Jan 2021 – Dec 2021 · 11 mos · Bhopal Area, India · On-site

Developed Databricks notebooks and Job Compute.
Designed and implemented a data lake solution on Azure using Azure Data Lake Storage and Azure Databricks to enable real-time data processing and analysis.
Developed data ingestion pipelines and data transformation workflows using Python and Apache Spark.
Responsible for creating and maintaining data pipelines in Azure Data factory, Pipeline components such as Integration Runtime, Linked Services, Datasets and Activities.
Created nested pipelines with various triggering mechanisms. Transforming data with Azure Data Flows. Pipelines migration among different workspaces.
Created script to perform ETL operation on unstructured files in Azure Databricks.
Optimized data pipelines for performance and scalability.
Implemented data quality checks and data validation routines to ensure accuracy and consistency of data.
Collaborated with cross-functional teams to gather requirements and design data models for data warehousing projects.