May 2021 – Jun 2022 · 1 yr 1 mo · India

• Designed and build several data pipelines for data migration using Airflow to extract and cleanse data from source systems and ingest into different sinks via API endpoints.
• Collaborated with business stakeholders to convert the business requirements into technical design specifications.
• performed mapping and transformed legacy data to data format required for modern system.
• Developed validation framework using python to ensure data quality and reduce discrepancies in migrated data.
• Implemented operational improvements through automation, requiring ingenuity and creativity, within data pipelines
• Implemented standards and procedures, worked with the team to ensure conformity to standards and best practices.
• Technologies used: Python, Azure, Airflow, Azure SQL database, Mongo DB

May 2019 – Dec 2020 · 1 yr 7 mos

• Designed, developed, and implemented cloud-native architecture to ingest over 5TB/day of streaming and batch data(security logs) using Azure data services such as Azure Data Factory, Eventhubs, Databricks, Synapse.
• Integrated Service now with Azure and Implemented a robust monitoring system for ETL pipelines using Log analytics and logic apps
• Created Pyspark jobs in Azure Databricks to perform data standardization, business transformation on log files in ADLS Gen2, and load data to Azure Synapse and Implemented optimizations in Pyspark jobs to improve performance of data pipelines.
• developed event-based azure functions to perform transformations on streaming data.
• Developed event-driven python server-less application using Azure Functions app.
• Leveraged Iac Developed reusable Terraform templates to create services like Azure data lake, data - factory, event hubs, data bricks, functions apps reducing development effort and hence managing the entire development lifecycle in the unified template
• Advocated for implementing security best practices across projects.

Dec 2016 – Apr 2019 · 2 yrs 4 mos

• Worked on design and development of the automation script using python to generate daily critical commission data for the business.
• Developed Python scripts to submit mainframe job and pull daily batch commission data generated from it
• Developed Python scripts to extract data from on-premise sources (Oracle, SQL server), perform transformations on data, and load data google cloud storage.
• Developed Batch ETL pipelines with Stream sets to load data from on-prem to Bigquery and google cloud storage.
• Developed serverless Cloud functions using python to perform specific event-based tasks.
• Implemented optimal business rules on billing commission data to obtain curated information on billing counts.
• Developed python script to send consolidated data as excel to the business team as an email attachment with the body containing dynamic information for respective daily batch cycle and a copy of same data as excel to the shared path.

Aug 2016 – Nov 2016 · 3 mos

• Worked on design and development of the ETL process to automate the process of loading.
• Developed ETL scripts in Python to get data from one database table and insert, update the resultant data to another database table
• Implemented data wrangling: ingesting, transforming, merging, and reshaping data frames'
• Determined optimal business logic implementations, applying best design patterns