Mar 2021 – Jul 2021 · 4 mos · Berlin, Germany

Jan 2020 – Feb 2021 · 1 yr 1 mo · Berlin Area, Germany

• Building and improving a configuration-based format for managing our complex data pipelines​ ​in Apache Airflow - Declarative Data Pipeline Definition.
• Data Pipeline built using Apache Airflow, Databricks, Apache Spark, Amazon Deequ, AWS Redshift.

Sep 2019 – Dec 2019 · 3 mos · Berlin Area, Germany

Feb 2018 – Aug 2019 · 1 yr 6 mos · Munich Area, Germany

• Responsible for all the data pipelines, ETLs owned by the team. Develop and maintain workflow management using Apache airflow. Responsible for data warehouse - redshift along with related internal data warehouse. Worked on upgrading to Data-Lake, Amazon Athena, Glue.
• Deployed numerous docker containers in kubernetes using Helm, CircleCI used for continuous integration.
• Evangelized best practices of data processing as well as educating other developers within the company on the topics related to Data Platform products.

Jan 2017 – Jan 2018 · 1 yr · Chennai

• As a senior software engineer focused on BI and Data, responsible​ for analysing​ data​ and providing meaningful insights​ , develop reporting for cross functional use within the company; acted as default point of contact for internal stakeholders regarding data insights; Primary tools were Python (particularly Pandas, Numpy, Scipy, Matplotlib, internally-developed libraries, and the iPython Notebook framework), SQL.
• Worked closely with the Engineering (Product) team to create data sources for BI use, providing high-level architecture of the data necessary to efficiently query the information required by internal projects.
• Developed​ and maintained data pipeline, primary tools were Python, PySpark, Kafka, Presto, Kibana. Built recommendation​ ​ engine​ ​ to​ ​ enhance​ ​ user​ ​ experience using PySpark. Wrote ETL programs and services to consume data from databases.

Aug 2016 – Jan 2017 · 5 mos · Hyderabad Area, India

• - Responsible​ ​ for​ ​ developing​ ​ and​ ​ deploying​ various​ Data​ ​ Visualization​ ​ dashboards​ ​ for​ ​ different client​ s.​ ​ Python being​ ​ used​ ​ as​ ​ back-end,​ ​ visualizations​ ​ using​ ​ D3.js​ ​ and​ dashboards​ ​ made using​ Bootstrap, jQuery,​ Underscore.js. Assist in training and on-boarding of new team members. Developed web services and applications in a variety of frameworks such as Flask and Tornado.

Mar 2014 – Jul 2016 · 2 yrs 4 mos · Chennai Area, India

• Responsible for ensuring code-quality and performance of all the analytics related features in the Enterprise social​ collaboration​ platform​ ​ owned​ ​ by​ ​ the​ team. Involved in development of multiple analytic products and POCs that are built from time to time.
• Generation of automated and manual Reports for internal stockholders within the organisation, using python packages such as Pandas, Numpy, Matplotlib and R. Report generated in the form​ of​ ​ pivot​ ​ tables​ ​ in spreadsheets​ and​ ​ charts​ ​ in​ ​ PDF.
• Built​ a standalone​ one​ place​ analytics dashboard using Apache Spark for enterprise level​ communication and collaboration​ application.​ Technologies involved Apache Spark – Spark Streaming,​ Amazon​ Kinesis​ Streams, Redis.
• Developed tools in multiple languages such as Ruby, Python and SQL to automate ETL tasks, wrote internal libraries in Python & Ruby to assist ETL. Built various Visualization dashboards using D3.js, Ruby-on-Rails and using visualisation tool like Tableau.