Prabhat Diwaker

CEO

Bengaluru, Karnataka, India14 yrs 8 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Expert in Big Data technologies and Google Cloud solutions.
Led successful data migration projects to cloud platforms.
Strong background in data governance and analytics.

Stackforce AI infers this person is a Big Data Engineer with expertise in cloud migration and data analytics.

Contact

prabhat.diwaker@gmail.com LinkedIn

Skills

Core Skills

Google Cloud PlatformBig Data TechnologiesData GovernanceData MigrationData Analytics

Other Skills

Agile & Waterfall MethodologiesApache KafkaApache OozieApache SparkApache SplineApache SqoopBig Data AnalyticsBigQueryBusiness IntelligenceCollibraData WarehousingData qualityDatabasesDruidETL

About

• Strong experience in Big data/Hadoop, Datalake and ETL developments. • Hands-on experience in developing analytics solutions using Big Data technologies like Hadoop, Hive, Spark, Spark-Streaming, Kafka, Druid, and Presto. • Experience in developing, optimizing, testing, and monitoring both batch and near real-time ETL applications. • Designing and Implementation of Google cloud infrastructure (GCP projects, GCS, IAM Policies, Ephemeral Dataproc clusters, BigQuery and Google Analytics). • Strong understanding of modern-day data lake and lakehouse patterns, architecture, and best practices like Data governance (Collibra), Metadata & Lineage management (Using Erwin and Apache Spline) and Transaction ELTs(using Apache HUDI) • Strong understanding of the principles of Data Warehousing using Fact, Dimensions, Slowly Changing Dimensions, Star & Snowflake schema modeling. • Implementing CI/CD process using Git, Jenkins, and Looper and Scheduling workflows using Apache Airflow and CA-Automic. • Creating Proof-Of-Technology/Concept (PoT& PoC) on emerging technologies in the Big Data landscape. • Experience in business stakeholder management, requirement gathering, planning, estimation, and other Agile practices. • Enabling self-serve and reporting solutions for the business using Google BigQuery, Looker, and Tableau.

Experience

14 yrs 8 mos

Total Experience

3 yrs 8 mos

Average Tenure

7 yrs 2 mos

Current Experience

Walmart global tech

3 roles

Principal Engineer

Promoted

May 2024 – Present · 2 yrs 1 mo

Staff Data Engineer

Promoted

Jul 2021 – May 2024 · 2 yrs 10 mos

Part of Data Strategy & Insight organization in Walmart-US (Supply Chain - Last Mile Team).
Tech lead for a team of 15+ members which works on Last-Mile data for Walmart E-comm.
Designing and implementing Google Cloud infrastruture for the team (GCP project, GCS, IAM policies, Ephemeral dataproc clusters, Big Query EDW).
Designing Data lake and ELT workflows with lambda architecture using Kafka, Spark, HUDI, Hive and Bigquery.
Enabling self-serve solutions for the business using Google BigQuery, Looker and Tableau.
Ensuring datalake best practices in the team.
Data governance and Cataloging Metadata using Erwin,Collibra and MITI (an in-House Metadata management tool)
Capture Lineage in spark jobs using Apache spline and send to a central repository for further lineage processing.
Applying Data quality rules and maintaining accuracy, completeness, consistency and conformity scores at lake tables.
Collaborating with business stakeholders, platform and Infra team, and other domain teams within the org.
Tech modernization initiative within the team and org (Ephemeral dataproc, Airflow, HUDI/DeltaLake, Lakehouse implementation)

Google Cloud PlatformGCSIAM PoliciesEphemeral Dataproc clustersBigQueryKafka+10

Senior Data Engineer

Apr 2019 – Jul 2021 · 2 yrs 3 mos

Part of Data Strategy & Insight organisation in Walmart-US (Merchant Datalake Team).
Maintained datalake using in-premise Hadoop infrastructure with data from various sources (Mainframe, Informix, Teradata, DB2, Azure data lake etc).
ELT workflows using latest Big data offerings like Spark (Scala), Hive, Imply Druid & Presto.
Worked on datasets in ranges of multiple Terabytes with focus on data quality and job performance.
Worked and led a migration project from On-premise Hadoop infrastructure to Google cloud platform.

HadoopSparkHiveDruidPrestoData quality+3

Adobe

Senior Data Engineer

Sep 2015 – Apr 2019 · 3 yrs 7 mos · Noida Area, India

Part of a data engineering and analytics team which work on Adobe data to develop business KPIs to bring useful insights.
Developed analytics solutions using Big Data technologies like Hadoop, HDFS, MapReduce, Hive,
Oozie, Sqoop, SQL, Spark , Kafka and Python.
Created applications to handle data in batch and near real-time using Hive/Spark , Kafka producer
and consumer APIs.
Used SnapLogic, Sqoop , Kafka and Python to create configurable batch and real-time ingestion
frameworks.
Created data lake and warehouse with Star and Snowflake schema models.
Written optimized transformations logic in hive/Spark/SQL to derive business KPIs to be consumed by
the dashboard.
Involved in OLAP cubes development on a distributed cluster using Kyvos and SSAS.

HadoopHDFSMapReduceHiveOozieSqoop+6

Cognizant technology solutions

Big Data Analyst For Apple iTunes Projects

May 2014 – Aug 2015 · 1 yr 3 mos · Bangalore

Worked for Apple Inc. as client and part of data analytics team.
Worked on Hadoop migration project from Oracle and Teradata to Cloudera distribution.
Written configurable automation scripts in python to test the codes and migration.
Worked on itunes and other source systems data to create a fact dimension model .
Written optimized transformation logic in hive and python to derive business KPIs.
Involved in creating a OLAP cube using SSAS on top of facts and dimensions.
Created oozie jobs to schedule the process.
Performed bug fixes and tracking them in Apple's Radar application.
Involved in project estimation and planning .