shubham gaurav

Data Engineer

Noida, Uttar Pradesh, India9 yrs 9 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Expert in ETL processes and data engineering.
Proficient in AWS and Azure data solutions.
Strong background in telecommunications data management.

Stackforce AI infers this person is a Data Engineer specializing in telecommunications and cloud data solutions.

Contact

Skills

Core Skills

Data EngineeringEtlSoftware Development

Other Skills

ADLSAWSAWS Data EngineerAgile MethodologyAthenaAzure Data EngineerAzure Data FactoryAzure DataBricksAzure SynapseBash scriptingBilling EngineBlobBug investigationCDHCI/CD

About

Total 6 years of experience and 3 years in Data engineer. My core competency in ETL/ELT, bigdata, data engineer, AWS data engineer, data integration, data quality, data analysis, data cleansing, data models, database design development, data mining, visualization, reporting, Data ingestion, spark cluster, spark resource optimization and performance tuning, SQL query optimization. My tech stack: Python, Pyspark, SQL, hadoop, hive, spark, S3, Athena, Glue, Teradata, shell scripting, EMR, CI/CD, github, Data Structure and Algorithms, DSA, sqoop, cloudera, cdh, Oracle-ODI, Oracle, MySQL, Shell scripting, Databricks community edition, AWS, YARN, Debugging, NOSql, HBase, RDBMS, MapReduce. Handles files like csv, JSON, parquet, avro, ORC, structured and unstructured data Worked in Agile methodology using sprint, MTV, Jira, Jenkins, Nexus, Perforce, aws data engineer, aws, cloud data engineer, Blob, ADLS, Azure Data Factory, Azure DataBricks, Azure Synapse, Azure Data Engineer, snowflake

Experience

9 yrs 9 mos

Total Experience

9 yrs 9 mos

Average Tenure

9 yrs 9 mos

Current Experience

Amdocs

3 roles

Senior Data Engineer

Promoted

Apr 2019 – Present · 7 yrs 2 mos

Data ingestion and sync process:
Computed ingestion logic for telecom domain messages (via file and cross-application oracle DBs) containing millions of subscribers
into data hub hive staging entity.
Developed landing logic from hive staging to DWH and data-sync logic to populate existing related entities to meet SLA.
Automated a recon system that differs between hive staging and DWH to monitor the quantity of data loss.
Monitored error table/logs to identify corrupt data which helped to speed up bug investigation.
NRT pipeline:
Worked on Teradata based near real-time data pipeline to populate several entities in DWH following a snowflake schema.
Propagated data through 3 levels of staging tables via landing BTEQ, ODI-Tool, and Merge BTEQ in order to maintain data history with a data
volume ranges between 1k-239M.
Data Extraction:
Customized extraction logic to extract data from both the data hub and DWH as per SLA.
Made required configuration changes and bash scripted to FTP extracted files to internal systems as well as to 3rd parties.
Apart from FTP, also populated intermediate oracle DBs from extract files through CTL, and POST-Scripts to secure data.
D2D Anomalies:
Apart from development, I’m involved in fixing bugs from qc, remedy which includes hive queries, impala queries, Teradata queries,
E2E business knowledge, ODI-Tool query validation, Merge-BTEQ query verification, staging to target table data verification, Yarn-
Jobs monitoring, corrupt data patch.
Dealing with clients, Integration managers to triage bugs, provide ETAs, provide RCAs.

Data ingestionData syncHiveDWHETLTeradata+5

Software Developer

Promoted

Jan 2018 – Apr 2019 · 1 yr 3 mos

Collaborated in creating Billing Engine for AT&T cricket, to calculate charges in multifamily scenario introduced in 2017 with a complex discount mechanism.
Engineered on a *.pdf/*.xml generating tool which creates a replica of information at e-commerce web pages (Shopping cart, Order
summary, Contracts) It consists of challenges in gathering runtime data inputs, images, page layout.
Multiple Standardized templates from scratch for VF-Ireland and VF-Italy. These templates are provided as legal documentation to their
telecom subscribers.
Worked on Bio-Signature product. This system is used to capture an electronic signatures and copy the same on legal documents.
Integrated multiple internal Systems, and external 3rd parties via web services, file-transfer.

Billing EnginePDF/XML generationWeb servicesFile transferSoftware Development