Shubham Jain

Data Engineer

Meerut, Uttar Pradesh, India9 yrs 8 mos experience

Key Highlights

9 years of experience in Data Engineering and Analytics.
Expert in building scalable data pipelines using modern tools.
Proficient in both cloud and on-premise data solutions.

Stackforce AI infers this person is a Data Engineering expert in SaaS environments with strong cloud integration skills.

Contact

Skills

Core Skills

Data EngineeringCloud Data ServicesData AnalyticsBig Data ProcessingData QualityData Analysis

Other Skills

AWSAWS EC2Advanced ExcelAirflowApache AirflowAzure Data FactoryAzure Data LakeAzure DatabricksAzure SQLAzure SynapseBig DataC++CI/CDData LineageData Mapping

About

Senior Data Engineer with 9 years of experience in Data Engineering and Analytics. Experienced in Python, C++, SQL, ETL, Git, Hive, Hadoop, Sqoop, Cassandra, HDFS, Airflow, Snowflake, Jenkins, AWS, Databricks, dbt and Azure Cloud.

Experience

9 yrs 8 mos

Total Experience

1 yr 7 mos

Average Tenure

1 yr 11 mos

Current Experience

Trajector

Senior Data Engineer

Jul 2024 – Present · 1 yr 11 mos · Gurugram, Haryana, India · Hybrid

Worked on ELT data pipelines using Airflow, SQL, S3, dbt and Snowflake.
Ingested data from REST API’s, databases and other sources into Snowflake using Airflow, dbt and S3.
Ingested data into multiple Snowflake layers from data lake to dim/fact/aggregate tables using dbt.
Worked with Apache Hadoop, HDFS, Hive, PySpark, Spark SQL, Snowflake and Sqoop for big data pipelines.
Worked on Azure services like Azure Data Lake (ADLS Gen2), Azure Data Factory (ADF), Azure Synapse, Azure SQL, Unity Catalog and Azure Databricks.
Developed KPI transformation pipelines in dbt, used across enterprise-wide BI reports in Sigma.
Worked on production deployments using Git, GitHub and CI/CD.

Apache AirflowSQLS3dbtSnowflakeHadoop+15

Netradyne

Staff Engineer

Oct 2023 – Jun 2024 · 8 mos · Bengaluru, Karnataka, India · Hybrid

Worked on large datasets using Java, SQL, Salesforce, Spark and Snowflake.
Designed ETL/ELT pipelines to ingest data from multiple sources to Snowflake via S3.
Created analytical datasets using Spark, Snowflake and Python as per the requirement.
Deployed data pipelines on AWS EC2 instances using Git, GitHub and Jenkins.
Implemented Big Data pipelines using Spark, Hive, Spark SQL, Snowflake and Airflow.
Worked on AWS services like EC2, Athena, S3, Redshift, Glue and IAM.
Enhanced data quality by performing thorough cleaning, validation, and transformation tasks.

JavaSQLSalesforceSparkSnowflakeAWS EC2+7

Airtel digital

Senior Data Analyst

Mar 2022 – Oct 2023 · 1 yr 7 mos · Gurugram, Haryana, India · On-site

Designed Big Data ETL pipelines using Python, SQL, PySpark, Airflow, Hive, Databricks, Spark SQL.
Implemented query optimizations to improve the performance and efficiency of data pipelines.
Worked on Apache Sqoop to transfer data from RDBMS to HDFS or vice versa.
Automated the data quality framework to resolve data gaps and ensure data integrity.
Worked with different File Format (JSON, CSV, Parquet, ORC, Avro, Delta Lake) for big data processing.
Worked with REST API’s for Data Extraction as per business needs.

PythonSQLPySparkAirflowHiveDatabricks+4

Wells fargo

Risk Analytics Consultant

Nov 2020 – Mar 2022 · 1 yr 4 mos · Hyderabad, Telangana, India · Remote

Worked on large datasets using Python and SQL.
Designed an automated framework to perform ETL testing.
Automated Data Lineage report using Python and SQL.
Worked on code reviews using Github and provide sign offs.
Designed automated data pipelines as per analytics needs using Python, SQL and ETL.

PythonSQLETLData LineageGitHubData Engineering+1