Apr 2017 – Sep 2018 · 1 yr 5 mos · Bangaon Area, India

• Design and develop applications to handle PB of data.
• Designed streaming pipe lines for high availability data across the applications as per business needs.
• Involved in common designing frame works to support batch and streaming data pipelines, which will be
• going to be used by multiple applications.

Apr 2016 – Mar 2017 · 11 mos · Bangalore

• Developed data pipeline using Spark, Hive, Scala , Python, Kafka and HBase to ingest customer
• behavioral data and financial histories into Hadoop cluster for analysis.
• Involved in converting Hive/SQL queries into Spark transformations using Spark DataFrames, Python
• and Scala. Involved in defining job flows using Oozie for scheduling jobs to manage spark, hive and
• sqoop jobs.
• Used Hive to analyze the Partitioned and Bucketed data and compute various metrics for reporting.
• Developed the code for Importing and exporting data into HDFS and Hive using Sqoop.
• Automated Sqoop incremental imports by using Sqoop jobs and automated jobs using Oozie.
• Responsible for writing Hive Queries for analyzing data in Hive warehouse using HQL.
• Implemented the workflows using Oozie framework to automate tasks.
• Performed hive performance tuning aspects like Map join cost-based optimization and column level
• statistics.
• Extensively worked on NoSQL database Hbase and Cassandra and Postgres like HBase.
• Involved in migration of spark 1.6 applications to Spark 2.0.
• Developed near real time streaming pipe lines using spark and kafka.

Jun 2015 – Mar 2016 · 9 mos · Bengaluru Area, India

• Developed data pipeline using Spark, Hive, Scala , Python, and HBase to ingest customer
• behavioral data and financial histories into Hadoop cluster for analysis.
• Compared to the performance of Spark with Hive and SQL/Teradata.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and
• Scala. Involved in defining job flows using Oozie for scheduling jobs to manage Apache Hadoop jobs.
• Used Hive to analyze the Partitioned and Bucketed data and compute various metrics for reporting.
• Developed the code for Importing and exporting data into HDFS and Hive using Sqoop.
• Automated Sqoop incremental imports by using Sqoop jobs and automated jobs using Oozie.
• Responsible for writing Hive Queries for analyzing data in Hive warehouse using HQL.
• Implemented the workflows using Oozie framework to automate tasks.
• Performed hive performance tuning aspects like Map join cost-based optimization and column level
• statistics.
• Extensively worked on NoSQL database Hbase and Cassandra and Postgres like HBase.

Oct 2011 – May 2015 · 3 yrs 7 mos · Bangaon Area, India

• Incorporated SQOOP, Flume and Hadoop File System API’s for implementing data ingestion pipelines.
• Prepared metadata importing into Hive and migrating existing tables and applications to work on Hive.
• Involved in managing and reviewing Hadoop log files. Designed and implemented an ETL framework
• using Java and scala to load data from multiple sources into Hive and from Hive into Teradata.
• Imported data using Sqoop to load data from MySQL to HDFS on a regular basis.
• Written Hive queries for data analysis to meet business requirements.
• Developed a custom File system plugin for Hadoop to access files on Data Platform.
• Designed and implemented a Map Reduce-based large-scale parallel applications.