Nov 2016 – May 2020 · 3 yrs 6 mos · Atlanta Metropolitan Area

• Built the Kabbage data lake using Databricks (Spark), Databricks Delta and AWS Athena
• Designed and built multiple data pipelines via Airflow and Databricks (using Spark) to ingest and process data from multiple sources into the Kabbage Data Lake
• Designed and built pipelines leveraging Spark and Airflow to unload data from SQL Server and Postgres into the Kabbage Data lake on a daily basis (300 Gb+ data per day)
• Designed and built pipelines leveraging Spark via AWS EMR and Airflow to ingest and process data from 3rd part REST APIs
• Designed and built a new stream-first data platform based on the Kappa Architecture, using MapR ecosystem (MaprFS, Mapr streams, MapR DB), DCOS (Mesos), Apache Flink, Apache Ignite, Druid, Streamsets etc
• Designed, built and automated the entire infrastructure using Chef and Terraform on AWS
• Designed a RESTful micro-service in Clojure for storing data in Apache Ignite
• Designed and Developed a Spark (Java) batch process to transfer data between S3 and MapR automated it via Python, Boto3 and AWS EMR
• Set up a highly available secure Apache Flink and Apache Spark standalone cluster on DCOS using Marathon
• Designed and setup monitoring and log aggregation using fluentd, ElasticSearch, DataDog and OpsGenie

Apr 2016 – Oct 2016 · 6 mos · New York City Metropolitan Area

• Worked with the Real-Time Events and Media team
• Refined the real time bidding pipeline by developing and testing clients using Python and Java
• Implemented and managed an HBase to Kafka pipeline using Python (with HappyBase and kafka-python) . Monitored the pipeline metrics through DataDog
• Developed an end-to-end integration test suite for the bidder pipeline using Python, Behave and Docker
• Designed a Spark job (Python) to analyze the performance of Avro vs Parquet
• Performed on-call duties as a Data Engineer including analyzing and solving bugs in the Data pipeline. This typically involved monitoring and fixing YARN and Luigi jobs. Also, DataDog was used for monitoring and tracking these pipelines

Jun 2015 – Aug 2015 · 2 mos · Cary, North Carolina

• Building a robust model for an intelligent semantic search server

Jul 2012 – Jun 2014 · 1 yr 11 mos · Pune/Pimpri-Chinchwad Area

• Responsible for Teamcenter customization through Integrated Teamcenter Kit(Server side C,C++ coding) and Teamcenter Rich Application Client(Client Side Java UI coding) for resolving issues related to Virtual Process Planning for Caterpillar Inc, the world's leading manufacturer of construction and mining equipment, diesel and natural gas engines, industrial gas turbines and diesel-electric locomotives.

Apr 2010 – Jun 2010 · 2 mos · Pune/Pimpri-Chinchwad Area

• Responsible for designing a mobile Java application using J2ME to track Loss Sales in Retail industry.