Dec 2020 – Jun 2021 · 6 mos

• Leading solution architecture team in Qubole for India/ME/Israel regions. With more than a decade data technologizes experience, I'm working with some of the data leaders in this region and helping them to achieve their business goals driven by data.
• Working on implementation and design of scalable big data architecture for companies in E-Commerce, Cab aggregator, Ad-tech, Logistics, Media etc.
• Participation in customer’s workshops and presentation of the proposed solution
• Design, implement, and deploy high-performance, custom applications at scale on big data stack
• Review and audit of existing solution, design and system architecture;
• Perform/Help troubleshooting of proposed solutions
• Have been working with some of the largest hot startups of India and APAC region
• As a lead, I'm responsible for managing/mentoring our great SA team in the APAC region
• Taking care of Pre/Post Sales tasks - Technical POCs, Expansion opportunities
• Working with cross functional teams - Product, Sales, Support, CSMs, Engineering

Feb 2020 – Dec 2020 · 10 mos

Feb 2019 – Jan 2020 · 11 mos

Jul 2017 – Jan 2019 · 1 yr 6 mos

Jan 2016 – Jul 2017 · 1 yr 6 mos

• Work highlights:
• Worked on multiple data pipelines to serve data for near real time and batch processing purposes
• Worked on a generic framework/product (Highly available, scalable and reliable) for real time streaming computation:
• This product is very scalable real-time distributed stream processing that connects systems to each other and enable creation of high level applications on top. The system is built on top of Kafka, Mesos and Marathon. It allows creation of topologies on the fly by drag-drop canned sources and processors through a multi-tenant secure console. It allows for deployment and scaling with one click. The system also provides support for delayed processing and joining events across topologies and sources. Along with properly schema tested, journaled and sidelined events, this system is the core for OLAs new data platform. A lot of high level apps are being written using this as the underlying transference layer.
• Big data warehouse using Qubole
• Data warehouse which enables 400+ business users to make Hive/Pig queries
• Created a data pipeline to enrich warehouse for 250+ tables coming from MySql, ES and Mongo
• Supporting and managing standards for the platform
• Worked on Hadoop, Hive (with MR/TEZ) and Hbase performance tuning
• Worked on Alluxio/Tachyon file system integration to accelerate performance
• Evaluating Presto, Phoenix and SparkSQL
• Using storage like HBase, Elasticsearch, Mongo, MySql and S3
• Worked on micro data services using Drop-wizard
• Using DCOS, Mesos, Marathon, Docker deployment environment
• Worked on real time event processing using Storm, RabbitMQ/Kafka
• Worked on Pig and Hadoop MR jobs for batch requirements

Dec 2014 – Dec 2015 · 1 yr

May 2013 – Dec 2014 · 1 yr 7 mos · Pune Area, India

• Technology Stack:
• Core Java, Ruby, Hadoop, Map-Reduce, HBase, Pig, Sqoop, Hive, Hue, Shell script, Jenkins, Jira, PigUnit, JUnit, Neo4J, Scala etc.
• Work highlights:
• Started Hadoop projects from scratch by creating complex analytic using Hadoop streaming and Pig scripts.
• Worked on data sourcing of more than 50 tables (Some of in TBs) from oracle to HDFS/HBASE environment using Sqoop, Oozie and de-normalization concept. Maintained delta refresh for big tables.
• Created big data files comparison tool
• Solved rule engine kind of requirement for health insurance data using map-reduce jobs.
• Developed Invoicing project using map-reduce and HBase
• Worked on a big data analytic platform to ease the process of running heterogeneous analytic
• Did POC on Neo4j, Cacading, Scalding, Akka, Spray etc.
• Worked on adjustment analytic in Scala using the same platform
• Worked on Spark

Aug 2011 – May 2013 · 1 yr 9 mos · Pune

• Technology Stack:
• Core Java, Ruby, Hadoop, Hadoop-Streaming, Map-Reduce, HBase, Pig, Sqoop, Hive, Hue, Bash, Jira, PigUnit, Control-M
• Work Highlights:
• Worked on scalable solution for big data processing using Hadoop echo system techs
• Created custom unix workflow to handle 500+ map-reduce jobs
• Worked on generic components (Compare tool, generic aggregation reducer, Fixed length file processing java classes for Pig, Map Reduce framework etc.) for Hadoop environment
• Coordinate with internal and external customers as necessary
• Familiarize the team with the customer needs, specifications, design targets, the development process, design standards, techniques and tools to support task performance
• Responsible for project architecture design, time estimations, coding, testing, bug fixing, optimisation, deployment, support etc.
• Coach and help team members to develop right approach for problem solving
• COSMOS project was a migration project from Mainframe to Hadoop system. It consists of six phases and I was the part of first three phases. It has moved the processing of certain finance applications from the Mainframe onto Hadoop system. During the course of these phases the output files which were being generated by Mainframe system are now generated by Hadoop, enabling MIPS reduction and decommissioning applications off Mainframe.
• Responsibilities and contributions:
• Leading team, Client interaction, Project design, Time estimation, Coding, code optimization, Bug fixing, deployment and support

Oct 2010 – Aug 2011 · 10 mos · Bangalore

• Technology Stack:
• Ruby, ROR, Web technologies, HTML, Java script, CSS, XML, MySql, MongoDB, HyperTable, Web services etc.
• Developed Online software and management project was all about on-line learning of real life works (e.g. How to batch or How to comb etc). It's basically conversion of a desktop application to a web application used for kids or mentally challenged people education
• Another project was Iceflower a resource management system which provides the organization investments and outcomes stats. It gives well featured reporting for the working resources on different projects with the time-sheet feature.
• Worked on NForm is a new financial application built for providing a better analysis and reporting to the customers / consumers about their financial data. Administration will be another important part of NForm application where, the administrator user will handle all the reports made available for the internal / External Users of any Firm, enabling / disabling Internal / External users, and all other general activities of the Administration. With NForm Internal / External Users will be able to get a better idea of what is the status of their assets, How effective is their investments, what is their investment structure / Asset allocation structure, how is it performing over a period of time, how did it Vary(upward trending / downward trending), etc

Jul 2007 – Oct 2010 · 3 yrs 3 mos · Noida

• Technology Stack:
• Core Java, PHP, Ruby, ROR, Web technologies like HTML, Java script, JQuery, CSS, XML, Ajax, MySql, Shell script,
• Worked on Online Intelligence provides proactive Web-crawling, spidering and brand protection throughout the entire advertising process, arming advertisers with transparency and traffic metrics. With this information, Online Intelligence examines sources of traffic through Web crawling. By monitoring ad campaigns in real time, OI quickly identifies dangers such as malware, incentivized traffic, negative keywords or off-brand ad placements. In this project, we used Ruby on Rails, PHP, Core Java, Nutch, Solr etc.
• Developed a SAAS application in Ruby on Rails, from scratch, as a part of four member team. This was basically an ecosystem between numerous casinos, where they share a common registration channel and/or with different types of authentication tokens.
• Developed a full functional Email-Engine in ruby. It is capable of sending mails purposefully to individual customer and can serve all the ECAD clients simultaneously. It can work with almost all types SMTP authentication.
• Developed a real time Petroleum Data Parser, in Java, for a hedge funding company. The full application comprised four major components. One crawler, one intelligent parser to fetch the necessary data from fully assorted source, one distributed file system for storage purpose, and finally a data getter. The parser was capable to tackle various types of text/html files as well as spreadsheets and rich text documents.
• An online whiteboard conferencing system, built with Ruby on Rails and PHP, named ‘VirtuaPedagogy’. It was an AJAX based online classroom coaching system. Key components comprised a full fledged real time whiteboard, equipped with standard drawing tools, distributed to all the participating members. A conferencing facility with three language support. The user could type in English, Hindi or in Bengali. Real time examination system