Shubham Repe

Product Manager

Pune, Maharashtra, India3 yrs 4 mos experience

Key Highlights

Over 3 years of experience in Big Data Engineering.
Expert in building scalable data pipelines and optimizing data processing.
Proficient in Hadoop ecosystem and AWS cloud technologies.

Stackforce AI infers this person is a Big Data Engineer specializing in data pipeline development and optimization within the Data Engineering industry.

Contact

Skills

Core Skills

Big Data EngineeringHadoop Ecosystem

Other Skills

AWSAdvance JavaAirflowAmazon Web Services (AWS)AzkabanC#Cascading Style Sheets (CSS)CommunicationCore JavaData AnalysisData ScienceDigital MarketingDjangoDjango REST FrameworkETL

Experience

Tata consultancy services

Big Data Engineer

Nov 2022 – Present · 3 yrs 4 mos · India · Hybrid

As a Data Engineer , I have been responsible for designing and implementing scalable ETL pipelines that ingest data and file systems into HDFS, handling over each month. I’ve developed Spark SQL and Hive jobs with a focus on performance optimization through techniques such as partitioning, bucketing, and aggregation. My role also involves automating data workflows using based pipelines, streamlining deployments and minimizing manual effort. I’ve successfully migrated multiple legacy RDBMS systems to Hive using Sqoop, improving data accessibility and reducing storage costs. Additionally, I’ve selected efficient file formats like ORC and Parquet to enhance processing speed and ensured the stability of Hadoop operations across multi-node Cloudera clusters.

ETLSpark SQLHiveHDFSSqoopAWS+4