Pankaj Verma

Data Engineer

Bengaluru, Karnataka, India6 yrs 11 mos experience
Highly Stable

Key Highlights

  • Databricks certified Spark Developer with 6 years of experience.
  • Expertise in Snowflake and Data Vault modeling.
  • Proven track record of enhancing code execution speed.
Stackforce AI infers this person is a Data Engineering expert with a strong focus on Big Data and Cloud technologies.

Contact

LinkedIn

Skills

Core Skills

Data EngineeringSnowflakeBig Data Engineering

Other Skills

AWS S3Adobe PhotoshopAgile MethodologiesAmazon Elastic MapReduce (EMR)Amazon Relational Database Service (RDS)Amazon Web Services (AWS)Apache IcebergApache SqoopBig DataBig Data AnalyticsC (Programming Language)C++Core JavaDBTData Analysis

About

Databricks certified Spark Developer and Senior Data Engineer around 6 years of technical expertise spanning across Hadoop (HDFS, YARN), Spark (PySpark), Snowflake DWH, AWS (S3, Lambda, Glue, EMR), Data Build Tool (DBT) and SQL.

Experience

6 yrs 11 mos
Total Experience
1 yr 9 mos
Average Tenure
1 yr 7 mos
Current Experience

New relic

Senior Data Engineer

Nov 2024 – Present · 1 yr 7 mos · Bengaluru, Karnataka, India

Impetus

Senior Data Engineer

Nov 2021 – Nov 2024 · 3 yrs · Bengaluru, Karnataka, India

  • Designed a Data Vault model for ingesting transactional data into Snowflake Data Warehouse.
  • Engineered an XML Handling framework for efficient Data Vault ingestion.
  • Developed a Snowpark Framework for Data Reconciliation.
  • Enhanced code execution speed by 3x through native processing in Data Warehouse using Snowpark.
  • Conducted a POC to evaluate Databricks Spark vs Snowflake Data Warehouse for handling large-scale data.
  • Enhanced code maintainability by 50% using Data Build Tool for efficient SQL and Python model management.
  • Leveraged DBT to create SQL and Python models, enabling testing and incrementally loading data into Snowflake.
Data VaultSnowflakeXML HandlingSnowparkDBTSQL+2

Zs

Data Engineer

Feb 2021 – Nov 2021 · 9 mos · Pune, Maharashtra, India

  • Sanity-check the base layer for irregularities.
  • Create spark-sql queries to form a baseline for metrics.
  • Refactor the spark-sql queries into pyspark dataframes.
  • Create a framework around pyspark to avoid using manual scripts.
  • Analyse scripts to optimized the run-time of scripts.
  • POC of spark, snowflake and AWS for the initial stages of enhancement.
Spark-SQLPySparkData AnalysisData Engineering

Tata consultancy services

2 roles

Big Data Developer

Jun 2019 – Jan 2021 · 1 yr 7 mos · Pune, Maharashtra, India

  • Getting records through SQOOP into HDFS and AWS S3.
  • Loading data into Hive warehouse from HDFS & S3.
  • Transform and validate various metrics of the raw data using pyspark.
  • Perform data Load for further analytics using pyspark and hive.
  • Find and fix faults that arise during ETL daily.
SQOOPHDFSAWS S3HivePySparkBig Data Engineering

Project Trainee

Jul 2018 – Oct 2018 · 3 mos · Indore, Madhya Pradesh, India

Education

MEDICAPS INSTITUTE OF TECHNO MANAGEMENT, INDORE

Bachelor of Engineering - BE — Information Technology

Jan 2015 – Jan 2019

Army Public School, Mhow

Computer Science

Jan 2008 – Jan 2015

Stackforce found 100+ more professionals with Data Engineering & Snowflake

Explore similar profiles based on matching skills and experience

Ronan Collobert

Ronan Collobert

Research Scientist

at Apple

Menlo Park, United States25 yrs 1 mo exp
Machine LearningResearch
Mike Thacker-Cooke

Mike Thacker-Cooke

Director of Recruitment

at KDR Talent Solutions

Middlewich, United Kingdom13 yrs 11 mos exp
Data RecruitmentTechnology RecruitmentBusiness Intelligence RecruitmentData Management Recruitment
Deepak Gupta

Deepak Gupta

Talent Acquisition Group-Leadership Hiring (PAN India)

at EPAM Systems

Delhi, India16 yrs 3 mos exp
Talent AcquisitionLeadership Hiring
Mani Bhushan

Mani Bhushan

Engineering leader

at OrbitronAI.com

Bengaluru, India10 yrs 7 mos exp
System ArchitectureCross-functional Team LeadershipGeospatial DataGeospatial IntelligenceAmazon Web Services (AWS)
Cheng Niu

Cheng Niu

Distinguished Research Scientist

at News Break

San Francisco, United States20 yrs 5 mos exp
Machine LearningNatural Language ProcessingArtificial Intelligence
Rebecca Lotus

Rebecca Lotus

Author & Speaker

at The Human Work

Steilacoom, United States16 yrs 9 mos exp
LeadershipWritingSoftware ManagementEngineering ManagementAPI Development
Emma Hershberger (Cockram)

Emma Hershberger (Cockram)

Software Engineering Manager

at Ulteig

Wabasha, United States4 yrs 9 mos exp
Project ManagementSoftware ArchitectureContinuous Integration and Continuous Delivery (CI/CD)Full-Stack DevelopmentAzure DevOps
Jason Martin

Jason Martin

VP FDE

at Databricks

San Francisco, United States33 yrs 5 mos exp
Professional ServicesConsulting