Aug 2024 – Present · 1 yr 10 mos · Gurugram, Haryana, India · On-site

• Currently working on a major banking project focused on migrating and optimizing SQL-based stored procedures into PySpark scripts, running efficiently in Databricks with Unity Catalog for secure data governance.
• Led a team of 8 engineers in designing and managing the Mart layer for optimized data access and reporting, ensuring high performance and scalability.
• Developed and orchestrated complex ETL pipelines using Azure Data Factory (ADF) to seamlessly ingest data from various sources, including databases, files, and APIs, into Databricks for processing.
• Performed extensive data transformation tasks, including data cleansing, filtering, and aggregations, leveraging PySpark's DataFrame and SQL APIs for efficient data processing.
• Implemented advanced transformations, such as joins, groupings, and window functions, to derive actionable insights from large datasets.
• Applied strong knowledge of OLTP, OLAP, and data warehousing principles to design and optimize data models that support analytical and transactional processing.
• Used Unity Catalog for centralized data governance and compliance, ensuring secure data access across multiple environments.
• Demonstrated expertise in handling large datasets (10TB+) and applying best practices in PySpark optimization to reduce processing times and enhance ETL performance.