Mar 2017 – Present · 9 yrs 1 mo · London, England, United Kingdom

Jul 2016 – Mar 2017 · 8 mos · London, England, United Kingdom

• Design and development of software in Cloudera’s Hadoop MapReduce framework.
• Configured and ran all MapReduce programs on large node cluster using chef etc.
• Designed and developed incremental loads from legacy, involving several technologies like sqoop, impala, python.
• Used luigi to automate data loading into the Hadoop Distributed File System and Python to pre-process the data.
• Performing hands-on research and software development to explore how big data technologies such as Hadoop, Apache Spark (Scala) etc. can be leveraged to increase the scalability of company's data analysis.
• Developed Spark D-Stream’s to process data in micro batches using Spark Streaming.
• Used Spark for batch processing and ad-hoc analytics
• Used AVRO Serdes to handle AVRO format data in HIVE and IMPALA.
• Experienced in implementing Spark RDD transformations, actions to implement business analysis and Worked with Spark accumulators and broadcast variables
• Developed scripts, UDF's using both Spark SQL and Spark Core in scala for Data Aggregation, queries and verified its performance over MR jobs.
• Designed and implemented data staging methods and stress testing of ETL routines to make sure that they don’t break on heavy loads.

Apr 2015 – Dec 2015 · 8 mos · Edinburgh, Scotland, United Kingdom

• Developed MapReduce programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW.
• Design and development of software in Hadoop MapReduce framework using Amazon S3, Amazon EC2, Amazon Elastic MapReduce(EMR).
• Configured and ran all MapReduce programs on large node cluster (Amazon EC2 spot instances) with Apache Hadoop.
• Designed and developed a Star Schema modelled data warehouse in Amazon Redshift.
• Designed and developed incremental loads from legacy to Redshift, involving several technologies like snoop, pig, hive, python.
• Used luigi to automate data loading into the Hadoop Distributed File System and Python to pre-process the data.
• Successfully ran all Hadoop MapReduce programs on Amazon Elastic MapReduce framework by using Amazon S3 for Input and Output.
• Performing hands-on research and software development to explore how big data technologies such as Hadoop, Apache Spark (Scala), Amazon Web Services, etc. can be leveraged to increase the scalability of company's data analysis.
• Replaced Hive programs with Spark to leverage RDD's in-order to obtain lower latency for the end-user
• Used Spark for batch processing and ad-hoc analytics
• Created Hive tables, loading with data and writing hive queries which will run internally in map way
• Designed and implemented data staging methods and stress testing of ETL routines to make sure that they don’t break on heavy loads.

Dec 2014 – Apr 2015 · 4 mos · Hammersmith and Fulham, England, United Kingdom

• Developed MapReduce programs to parse the raw data, populate staging tables and store the refined data in partitioned tables in the EDW.
• Design and development of software in Hadoop MapReduce framework using Amazon S3, Amazon EC2, Amazon Elastic MapReduce(EMR).
• Configured and ran all MapReduce programs on 20-30 node cluster (Amazon EC2 spot instances) with Apache Hadoop to handle 1000GB/sample of WebLog data.
• Configured a 20-30 node (Amazon EC2 spot Instance) Hadoop cluster to transfer the data from Amazon S3 to HDFS and HDFS to Amazon S3 using distcp and also to direct input and output to the Hadoop MapReduce framework.
• Successfully ran all Hadoop MapReduce programs on Amazon Elastic MapReduce framework by using Amazon S3 for Input and Output.
• Performing hands-on research and software development to explore how big data technologies such as Hadoop, Apache Spark (Scala), Amazon Web Services, etc. can be leveraged to increase the scalability of company's data analysis.
• Built real-time distributed data access solutions around Apache spark
• Replaced MapReduce programs with Spark to leverage RDD's in-order to obtain lower latency for the end-user
• Used Spark for batch processing and ad-hoc analytics
• Good Knowledge in using Streaming based analytics using Apache Storm & Apache Spark
• Good Knowledge in using Messaging framework like Apache Kafka or RabbitMQ
• Designed and transmitted a RDBMS(Oracle) Database to Hive Database
• Created Hive queries that helped data analysts spot emerging trends by comparing fresh data with EDW reference tables and historical metrics.
• Used Azkaban to automate data loading into the Hadoop Distributed File System and PIG to pre-process the data.
• Created Hive tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map way

Nov 2009 – Dec 2014 · 5 yrs 1 mo · Staines-upon-Thames, England, United Kingdom

• During my tenure with Centrica, managed to work on several high profile data warehousing/ data migration and big data projects, using variety of tools like Hadoop, SAP Data Services, Business Objects etc.

Jan 2008 – Nov 2009 · 1 yr 10 mos · Galway, County Galway, Ireland

• Understand requirements, document and implement code to the specs.
• Data Modelling the data warehouse environment using Kimball Methodology
• Involved in the analysis, design and development of all the interface’s using Informatica Power mart tools in interface team and interfaced with all the other tracks for business related issues.
• Developed Informatica workflows/worklets required to extract, transform and load data from several legacy systems.
• Developed standard and re-usable mappings and mapplets using various transformations like expression, aggregator, joiner, source qualifier, router, lookup, and filter.
• Cleansed, Audited, Profiled, Validated and Reported ETLed data through Data Quality Management tools and transforms.
• Address, Name and custom standardizations using Data quality.
• Managed DI Metadata using metadata management tools.
• Job statistics reporting using metadata management tools.
• Written Oracle Packages, Procedures, MViews and Triggers required for Processing Batch Details, Inventory/Invoice Management.
• Was part of scripting team for shell scripts to automate and migrate data from ODS to Data warehouse.
• Worked on creating standard and adhoc reports, dashboard using Siebel Analytics.
• Build the Siebel Analytics repository/metadata and define/create the subject areas where reporting and analysis are centered around.
• Designed, Created and Tested Reports in the Dashboard and adhoc reports according to the client needs using Siebel Analytics/OBIEE.
• Designed, Developed and Tested Data security and Dashboard security in Siebel Analytics/OBIEE
• Writing Application Logic, Coding, Unit Testing and Support for Testing Personnel and Business Users.
• Participated in user meetings to resolve on-going implementation & performance issues.
• Created, updated and maintained ETL technical documentation.

Jan 2005 – Sep 2007 · 2 yrs 8 mos

Nov 2004 – Sep 2007 · 2 yrs 10 mos · Manchester, United Kingdom and Mumbai, IndiaManchester

• Extracted, transformed and loaded data from operational data sources into target data warehouse using BODI.
• Implemented profiling, extracting, cleansing, transforming, integrating and loading data into Data warehouse using BODI
• Written database Procedures, MViews and Triggers.
• Developed workflows required to extract, transform and load data from several GE Capital Solutions legacy systems.
• Administration and schedule DI jobs as per daily, weekly and monthly using job server.
• Involved in requirements gathering from the business users to build the Datawarehousing Environment.
• Installed Business objects created repository and distribute the BO Main Key across the enterprise.
• Created users and groups and applied row level and objects level securities using Central Management Console.
• Involved in migration and deployment of the universes and reports across multiple domains.
• Developed complex queries, procedures, custom functions using PL/SQL
• Resolving loops using contexts and aliases to remove the cyclic dependencies.
• Created reports using Business Objects functionalities like queries, slice and dice, drill down, cross tab, master detail reports, formulae, etc.
• Creating standard full client reports and Web I reports and scheduled full client reports and Web I reports with Broadcast Agent Server.