NIKHIL REDIJ

Data Engineer

Mumbai, Maharashtra, India11 yrs 11 mos experience

Most Likely To SwitchAI ML Practitioner

Key Highlights

8+ years of experience in Data Engineering.
Expert in designing scalable data architectures.
Proven track record in mentoring junior engineers.

Stackforce AI infers this person is a Data Engineering expert in Fintech and SaaS sectors.

Contact

nikhil.redij@trafigura.com LinkedIn

Skills

Core Skills

Data EngineeringAwsGcpAzureBig Data

Other Skills

AI AgentsAirflowAmazon RedshiftAmazon Web Services (AWS)Apache AirflowApache KafkaApache SparkAzure Data FactoryAzure DatabricksBigQueryCommodity MarketsData AnalysisData AnalyticsData MigrationData Mining

About

As a highly skilled and experienced Data Engineer with 8+ years of expertise in the industry, I have developed a deep understanding of the end-to-end data pipelines, REST API, from data ingestion to migration, transformation, storage, consumption and analysis as well as maintenance of the data. My experience includes working with large datasets and designing, implementing scalable solutions that allow for real-time data processing and analysis.I have a strong background in all cloud frameworks, including AWS, Azure, and GCP, and have designed and implemented complex data architectures in each of these environments. I have extensive experience with various data warehousing technologies such as Databricks, Redshift and BigQuery. Skilled in Technologies like Spark, Python, Kafka, SQL, Scala, Java etc. Throughout my career, I have collaborated with cross-functional teams to identify business requirements and translate them into effective data solutions. I am also constantly seeking to stay up-to-date with the latest industry trends and emerging technologies to continue delivering optimal solutions. Overall, my extensive experience and technical expertise make me a valuable asset to any organization seeking a skilled Data Engineer with a proven track record of delivering high-quality data solutions on the cloud. Experienced in hiring as well as mentoring junior engineers, interns for multiple projects Highest qualification, Master of Science in Computer Science

Experience

11 yrs 11 mos

Total Experience

1 yr 8 mos

Average Tenure

2 yrs 10 mos

Current Experience

Trafigura

Data Engineer

Aug 2023 – Present · 2 yrs 10 mos · Mumbai, Maharashtra, India · Hybrid

Amazon Web Services (AWS)Web ScrapingLarge Language Models (LLM)AI AgentsPythonRetrieval-Augmented Generation (RAG)+8

Simpl

2 roles

Staff Data Engineer

Nov 2022 – Jun 2023 · 7 mos · Mumbai, Maharashtra, India · Remote

Amazon Web Services (AWS)Amazon RedshiftApache SparkData WarehousingBig DataData Migration+13

Senior Data Engineer

Nov 2021 – Nov 2022 · 1 yr · Mumbai, Maharashtra, India · Remote

Amazon Web Services (AWS)Amazon RedshiftApache SparkPythonData WarehousingBig Data+13

Priceline

Data Engineer

Jun 2020 – Oct 2021 · 1 yr 4 mos · Mumbai, Maharashtra, India · Remote

(Python, Bigquery, GCP, Cloud Storage, Dataproc, Spark, Delta Lake, Airflow, Oracle)
Designed and implemented a framework to migrate data in daily batch jobs from Oracle to Delta Lake Storage and Bigquery tables using Airflow and PySpark.
Created, scheduled and deployed data ingestion pipeline to ingest daily data from Vendor API, transform and aggregate it and store the results in Bigquery tables using Python and Airflow.
Migrated and Automated old workflows from Oracle and Hadoop to Google Cloud Storage using Dataproc PySpark and Airflow.
Lead the design and implementation of Data lake architecture for Processing and Analytics layer for Data scientists and Analysts.
Successfully completed end to end projects by collaborating with Business Partners, Analysts, Engineering teams to translate Business requirement and developed highly functional and impactful products within tight timeline

Apache SparkPythonData WarehousingBig DataData MigrationData Mining+11

Gep worldwide

Data Engineer

Jun 2018 – May 2020 · 1 yr 11 mos · Mumbai Area, India · On-site

(Python, Scala, Spark, Azure DataBricks, HDInsight, Azure Data Factory, Flask API, Docker)
Architected, implemented, and automated highly scalable ETL pipelines to extract Data from Azure SQL Server / Data Lake using Azure Data Factory, store in Azure Data Lake storage, clean and normalize it using PySpark and load aggregated data in Azure SQL Server.
Developed various features of ETL Process by collaborating with cross teams for Quality Workbench project to perform sanity on data, Data Clustering used for data classification using DataBricks Scala-Spark.
Wrote a Web Crawler using Python-Scrapy to retrieve Supplier information from Google, Bing and Wikipedia web pages, which is used for enriching Supplier Master Lookup.
Developed a Python-Flask API to fetch data from Datalake and Azure SQL Server, run a Forecasting algorithm and send back output JSON back in response. Deployed the API using Docker on Azure App Service. Data was used for Analytical Platform
Mentored interns and junior engineers regarding best practices in data engineering, code reviews, testing, operations like CI/CD.

Web CrawlingMicrosoft SQL ServerApache SparkPythonBig DataData Migration+12

Loudcloud systems inc.

Associate Technology

Nov 2017 – Jun 2018 · 7 mos · Mumbai Area, India · On-site

(Java, Scala, Spark, HDFS, JDBC, RESTful Services, MySQL, MongoDB, PostgreSQL, Denodo)
Designing and developing features for data ingestion via REST API and JDBC services for Risk Analysis from different Learning Management Systems (LMS).
Developed a Denodo connector for getting LMS data from client's PostgreSQL to local application via REST API.
Ingestion of Student Information System (SIS) data from HDFS via different jobsets, storing it in MongoDB and in MySQL for predictive analysis.

Apache SparkHadoopBig DataData MigrationExtract, Transform, Load (ETL)Scala+2

Cerner corporation

Software Engineer

Jan 2014 – Mar 2017 · 3 yrs 2 mos · Kansas City, Missouri Area · On-site

(Java, MapReduce, Hive, Sqoop, Scala, Spark, REST Services)
Running ETL tasks on Hadoop using Java-MapReduce, Hive and Scala-Spark for transforming data.
Transferred EMR (Electronic Medical Record) data from various client storages to Hadoop using Sqoop.
Actively Participated in Requirement, Design and Code reviews with Team providing useful solutions to business issues.
Gained experience with Git, Crucible, JIRA following best industry coding and Unit testing practices.

Apache SparkHadoopBig DataData MigrationJavaExtract, Transform, Load (ETL)+3

Beplused.com

PHP Web Developer

Jun 2012 – Dec 2012 · 6 mos · Dallas/Fort Worth Area · On-site

(PHP, MySQL, Apache, Zend2, JOOMLA, JomSocial, HTML, CSS, JavaScript, MVC, JIRA, Crucible, SVN)
Developed features like friends, groups, and invite and register new user, design different city networks for social networking website using web technologies and MVC architecture.
Modeled the database according to Normalization principles and designed new SQL stored procedures.
Followed agile principles for development lifecycle while learning new technologies as Zend2-MVC, JOOMLA and JomSocial.