Jan 2024 – Mar 2025 · 1 yr 2 mos · India · Remote

• ● Designed and developed scalable data pipelines using Airflow, Snowflake, and AWS, ensuring efficient data processing for ad exchange analytics.
• ● Collaborated with multiple third-party vendors to generate and deliver customized datasets via S3 and email, streamlining data sharing for external partners.
• ● Led the integration of Liveramp data into internal pipelines, enriching customer insights and improving decision-making for targeted ad placements.
• ● Optimized SQL queries and Snowflake workloads, reducing query times by 15% and improving data accessibility for analytics and reporting.
• ● Successfully backfilled a month's worth of critical client data within a day, ensuring zero downtime and data consistency.
• ● Implemented complex transformations in data pipelines, adding key fields to enhance reporting accuracy and business intelligence.
• ● Participated in a hackathon to optimize click-through rates (CTR) using Spark ML, demonstrating a potential uplift in CTR with predictive modeling.
• ● Provided production support, resolving 95% of critical data pipeline incidents promptly to ensure uninterrupted operations.

May 2023 – Nov 2023 · 6 mos · Remote

• ● Led the migration of disparate agricultural data from multiple sources into Trino, ensuring seamless integration and accessibility. Designed and implemented an optimal partitioning strategy, improving query performance and reducing processing time by 30% for large tables.
• ● Developed a robust Dagster-based orchestration pipeline to automate ETL workflows, improving data reliability and reducing manual intervention by 40%. Implemented monitoring and alerting mechanisms to enhance pipeline stability.
• ● Leveraged Python, SQL, and dbt to optimize data transformations and schema design, reducing data processing costs by 25%.

Apr 2022 – Dec 2023 · 1 yr 8 mos · Noida, Uttar Pradesh, India

• ● Engineered an event streaming solution using PySpark on Synapse, processing 10 million+ events per day from Azure Event Hub to Delta Lake, achieving a 40% increase in data processing efficiency.
• ● Innovated a data quality solution with LangChain and OpenAI, automating the generation of PySpark code from English prompts, resulting in a 60% reduction in data validation time.
• ● Designed and managed data pipelines in Azure Data Factory (ADF) to orchestrate data movement and transformations across cloud storage, Synapse, and Delta Lake.
• ● Orchestrated 15+ CI/CD pipelines via Azure DevOps, ensuring seamless deployment across development, testing, and production environments.
• ● Drove the creation and provisioning of essential Azure resources including Synapse workspaces, Key Vaults, and ADLS using ARM templates, contributing to a 60% faster project setup.
• ● Led the implementation of comprehensive logging practices using Azure Log Analytics, significantly improving resolution speed, resulting in a 25% decrease in incident response time.

Apr 2021 – Apr 2022 · 1 yr · Bengaluru, Karnataka, India

• ● Spearheaded the migration of semi-structured JSON data to Snowflake, leveraging Python, SQL, and AWS S3, ensuring efficient storage and retrieval.
• ● Automated ETL processes and email notifications with Dataiku triggers and AWS services, resulting in a 30% reduction in processing time.
• ● Managed the successful migration of 8,000 databases from SQL Server to Snowflake, optimizing schema design and query performance.
• ● Optimized processing speed using PySpark and Snowflake by 80%.

Jan 2021 – Mar 2021 · 2 mos

• Anomaly Detection :
• Anamoly detection in the logs of the wireless transmission and idetifying the KPIs.
• Technology : Python, Scikit Learn, MongoDB, Flask, Beautiful Soup

Aug 2016 – Apr 2021 · 4 yrs 8 mos · Gurgaon, India

• ● Developed Telecom business offerings, processing and analyzing data using Python, SQL, and Pandas.
• ● Reduced daily mapping process runtime from 20 to 2 hours using ETL optimization and automation, leading to a 90% efficiency improvement.
• ● Managed Ordering, Billing, and Migration activities for a user base of 100K+, leveraging SQL, Data Warehousing, and CRM systems.
• ● Successfully handled software defects using Agile methodologies and Jira, resolving an average of 15 issues per sprint.