Nov 2022 – Present · 3 yrs 6 mos

• Pioneering the development of versatile, cloud-agnostic, and data-agnostic data pipelines by researching tools such as Airflow, Apache Beam, EMR on EKS, Delta Tables.
• Successfully Deployed over 20+ pipeline instances for internal customers, significantly enhancing team recognition for data infrastructure provider.
• Collaborating with AWS Solution Architect team to investigate the integration of Redshift Serverless with our system, resulting in successful conclusion & acceptance
• Facilitating effective testing for the team by establishing a dedicated staging environment for pipeline testing.
• Successfully mentored and led a team of three junior members, providing comprehensive training, guidance, and support. Led them to execute tasks with precision, fostering their growth and contributing to the team's overall success.

Jun 2020 – Nov 2022 · 2 yrs 5 mos

• Performed Cost analysis & resource optimization for AWS cloud, Saved approx $5K/month.
• Mentored a Junior Engineer.
• Performed Disaster recovery drill. AWS CDK implementation allowed us to deploy a complete data pipeline under 20 minutes which takes weeks or months if done manually.
• Solved a Major bottleneck in scaling due to IP starvation, Solved it by reimplementing VPC Infrastructure.
• Completely automated Cloud Infrastructure Deployment by adopting AWS CDK to achieve true "Infrastructure is code" status. Infrastructure includes a Data Pipeline that handles data of Petabytes scale ( Billion+ Devices ) along with all existing Infrastructure.
• Developed a Customized CI/CD from Scratch & integrated 3+ highly customized jobs that are active in production. Saved hours of manual work.
• Did Comprehensive comparison-based research on Big data file formate selection and got AVRO adopted as primary storage formate.
• Received appreciation for developing a Cloud Formation script for a notifier based on Lambda for a confidential gov entity.
• Single-handedly developed, deployed and maintained Fullstack of a web portal for more than 1 year with <1% downtime. The success of the project led to its adaptation to external customers.

Jan 2020 – Jun 2020 · 5 mos

• - Built and deployed the initial Phase of a web portal from concept to the production stage. Enabled it for Team wide usage.