Jul 2024 – Present · 1 yr 8 mos · Pune, Maharashtra, India

• Managed data pipelines using Azure Data Factory (ADF) and Databricks, improving the automation and scalability of data ingestion processes by 25%.
• Designed and implemented distributed data processing systems using PySpark, Azure Databricks and azure data lake storage, increasing efficiency by 30% through optimized Spark jobs.
• Created and executed a series of advanced SQL queries that aligned closely with business objectives; improved data accuracy by 30%, ensuring stakeholders received reliable insights for strategic planning and execution.
• Engineered robust data validation process within ADLS Gen2 using PySpark that identifies 99.99% of data anomalies and reduced downstream data errors.
• Implemented Lakehouse Architecture across Bronze, Silver, and Gold layers, streamlining data processing and enhancing data quality for daily Incremental data.
• Performed data cleaning tasks in PySpark, including handling missing values, standardizing date formats, and removing duplicate records, ensuring 100% data quality.
• Applied advanced optimization techniques such as Partitioning, Caching, Bucketing, and enabling Adaptive Query Execution (AQE), achieving a 5x improvement in data processing time.
• Integrated Delta Lake with Azure Data Lake to leverage ACID transactions, enforce schema, and enable time travel features, resulting in robust data management and zero data loss.
• Monitored Spark ETL pipelines on Azure Data Factory (ADF) by leveraging Spark optimization techniques, ensuring 99.9% accuracy and efficient data integration.

Jul 2022 – Present · 3 yrs 8 mos · Pune, Maharashtra, India

• Worked on a large-scale Big data projects for finance domain to use, manage and leverage amount of data to generate insights.
• Designed and implementing end to end data pipelines using Azure services.

Jan 2022 – Jun 2022 · 5 mos · Pune, Maharashtra, India