Arpit Solanki

Backend Engineer

Delhi, India8 yrs 2 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Expert in building data pipelines and quality platforms.
Proficient in open source technologies like Spark and Airflow.
Strong background in data engineering and ETL processes.

Stackforce AI infers this person is a Data Engineer specializing in SaaS and data integration technologies.

Contact

Skills

Core Skills

Data EngineeringEtl

Other Skills

AirflowAlgorithmsApache SparkCC++Cascading Style Sheets (CSS)CeleryDaskData QualityData StructuresDjangoExpress.jsGitGithubGoogle API

About

Currently building a Data Virtualization Platform to support querying over hetroginous data sources with central authorization and caching for faster querying with less cost. I am using open source projects such as Apache's Calcite, Ranger, Spark etc to build it. Most recent work: Built data quality platform at Atlan so users can setup checks and calculate metrics such as median, null values, sensitive data (PII detection credit card phone number etc) on their data to ensure only right data goes consumption. I used Spark as processing engine and kubernetes as infrastructure to build this. I am a seasoned Data Engineer, solving Data Enginering problems at Atlan. I am currently working on technologies such as Presto, Airflow, Spark, Kafka, Apache Hudi, Hive, k8s. At Atlan I have - Built data pipelines (ETL) using Airflow, - Built data lakes and data versioning in lakes using Spark, Presto and Hive. - Built developer tooling for Atlan data platform users. - Built plugin based generic data ingestion framework with change data capture (CDC). I am passionate about working on open source technologies and contributed to PyData's Dask and Presto. if you'd like to work together or just discuss anything, drop me an email on solankiarpit1997@gmail.com. Github: https://github.com/arpit1997/ Stackoverflow: https://stackoverflow.com/users/5250746/arpit-solanki Codementor: https://codementor.io/arpitsolanki My interview on Presto at Community broadcast: https://youtu.be/X77FAfIf1Qo

Experience

8 yrs 2 mos

Total Experience

1 yr 11 mos

Average Tenure

5 yrs 2 mos

Current Experience

Lightbeam.ai

Backend/Data Engineer

Apr 2021 – Present · 5 yrs 2 mos · Remote

Mate labs

Data Engineer

Oct 2020 – Mar 2021 · 5 mos · Remote

Atlan

3 roles

Data Engineer II

Oct 2019 – Sep 2020 · 11 mos

SparkAirflowData EngineeringETLData Quality

Data Engineer

Jun 2018 – Oct 2019 · 1 yr 4 mos

Led the development and deployment of Ministry of Rural Development India's project DISHA. Developed ETL pipelines using Airflow to power DISHA Dashboard.
Managed on-premise deployments on NIC's Meghraj Cloud. Deployed the entire stack Vue.js dashboard, ETL pipeline and dashboard modification APIs, ETL pipelines using debian packages.
Added Monitoring of application and infrastructure to alert stakeholders of critical scenarios like failing ETL pipelines, high resource usage, high traffic on dashboard etc.
Improved cataloging performance of Atlan Catalog by 90%, added versioning features such as remove a commit, rollback.
Added features like data deduplication, data ingestion from S3, running SQL queries in the workflow to Atlan Workflows

AirflowSparkPrestoHiveData EngineeringETL

Data Engineer Intern

Jan 2018 – May 2018 · 4 mos

Built a data integration framework using celery with support for state management and incremental data ingestion.
Worked with data scientists and Bussiness intelligence team to build ETL pipelines using Airflow to power analytics and dashboards
Worked on building system native packages (Debian and RPM) of data integration services and ETL pipelines written in Python.

CeleryAirflowPythonData Engineering

Codementor

Mentor

Jan 2019 – Jun 2020 · 1 yr 5 mos · Remote

Provided one to one live help on given problems/bugs to mentees
Taught and guided on new technologies, fields and approaches on problem-solving.
Have done over 100 sessions with 5 star rating with 40+ reviews.

Vstv

Backend Developer Intern

Sep 2017 – Nov 2017 · 2 mos · Remote

I worked as a Backend developer intern with MEAN stack. My responsibilities includes creating web services, handling data streams to Backend and collaboration with frontend team.

Mate labs

Backend Developer Intern

May 2017 – Jul 2017 · 2 mos · Bengaluru Area, India

I worked as a backend developer intern with python/Django stack. My responsibilities included writing web services, integrating machine learning pipelines to mateverse backend.

Brainplay learning solutions llp

Full Stack Developer and Code Reviewer Part time

May 2017 – Jul 2017 · 2 mos · Remote

I worked as a Code Reviewer with MEAN stack. My responsibilities included finding possible bugs, reviewing code quality and coverage and testing the functionalities.