S

Sonaiyakarthick P

Data Engineer

Bengaluru, Karnataka, India12 yrs experience
Most Likely To SwitchHighly Stable

Key Highlights

  • Over 10 years of experience in data engineering.
  • Expert in building scalable data architectures.
  • Contributed significantly to open-source projects.
Stackforce AI infers this person is a Data Engineering expert in SaaS with strong skills in ETL and big data technologies.

Contact

Skills

Core Skills

Data EngineeringBig DataEtlAwsBackend Development

Other Skills

Ad TechAerospikeAirflowAmazon AthenaAmazon Web Services (AWS)Apache SparkAthenaCore JavaDistributed SystemsDruidExtract, Transform, Load (ETL)HadoopHiveJavaJupyter

About

A seasoned Data Engineer with 10+ years of experience in big data platforms, cloud technologies, Backend systems, Low latency and High throughput systems, Devops, data pipeline development and Mid managment. Skilled in optimizing ETL processes, ensuring data reliability, and designing scalable architectures for data systems. Data Engineering, Real time system development, Java, Spark, Airflow, AWS, GCP, Devops, DBT, Snowflake, Databricks, EMR, Lambda, Netty Server, FastAPI, Aerospike, Mongo, Kafka. My contribution to Open source : https://github.com/zapr-oss/zapr-athena-client

Experience

Apollo.io

Senior Data Engineer

Nov 2022Present · 3 yrs 4 mos · Bengaluru, Karnataka, India

  • 1. Part of the development of Change Data Capture (CDC) processes, focusing on robust bootstrap methods and incremental updates for real-time data synchronization.
  • 2. Enhanced auto-scoring pipelines by designing and implementing a default scoring system, streamlining operational workflows.
  • 3. Led efforts in Snowflake data sharing for Panther, resolving schema cloning challenges and ensuring compliance with masking policies during blue-green deployments.
  • 4. Contributed to security hardening initiatives, including SIEM onboarding and compliance projects, enhancing the organization's security posture.
Apache SparkData EngineeringSnowflakedatabricksAirflowdbt+2

Samsung ads

Chief Engineer

May 2022Oct 2022 · 5 mos · Bengaluru, Karnataka, India

  • 1. Member of feature team in Samsung Ads Report & Insights platform
  • 2. Worked on the Reporting ETL pipeline to adopt the MMP conversion events.
AirflowOpen-Source SoftwareData EngineeringAWSAd TechExtract, Transform, Load (ETL)+1

Zapr media labs

4 roles

Technical Architect / Engineering Manager(acquired by Samsung Ads)

Promoted

Aug 2021Apr 2022 · 8 mos

  • 1. At ZAPR, I led a team of 4 individuals on the platform team and was also a member of the architecture team.
  • 2. I developed best practices for various big data frameworks such as Athena, Druid, and Hive clusters.
  • 3. I was responsible for designing and planning several platform services at ZAPR, including the meta service and geo service.
  • 4. I designed the entire migration process from Hive metastore to Glue metastore at ZAPR and provided guidance to my team members throughout the completion of the migration.
  • 5. Additionally, I served as an individual contributor and led a team of 3 individuals on the ZAPR voice analytics team. This team was instrumental in designing and spearheading the development of a voice SaaS product during its initial stages.

Technical Lead

Promoted

Feb 2019Jul 2021 · 2 yrs 5 mos

  • As the Technical Lead of the Platform Team at ZAPR, my responsibilities include collaborating across teams within the organization to equip them with essential tools and support. Here are some key contributions and achievements:
  • 1. I played a central role in the development of the ZAPR open-source Athena client, serving as the main contributor. The client is available at https://github.com/zapr-oss/zapr-athena-client.
  • 2. Collaborating with engineering teams, I spearheaded the development and maintenance of a data query engine to analyze data and generate reports efficiently.
  • 3. I led the migration of our in-house query engine to AWS Athena, streamlining data processing and report generation. This migration, detailed at https://github.com/zapr-oss/zapr-athena-client, significantly reduced execution time and optimized AWS costs.
  • 4. Throughout my tenure, I designed and implemented numerous low-latency, high-throughput systems at ZAPR to enhance performance and scalability.
  • 5. I developed an in-house adtech reporting system based on Druid for real-time analysis of campaign performance.
  • 6. As part of my role, I actively mentor team members, providing guidance and support to foster their professional growth and development.
JupyterAmazon Web Services (AWS)HiveJavaAmazon AthenaOpen-Source Software+3

Senior Software Engineer

Promoted

Jan 2017Feb 2019 · 2 yrs 1 mo

  • In my role, I've undertaken various responsibilities and projects:
  • I constructed an end-to-end pipeline for batch ad spot detection and viewership generation, ensuring seamless data flow and processing.
  • Managed the entire Extract, Transform, Load (ETL) process for data generation, ensuring accuracy and efficiency in data processing.
  • Successfully migrated the existing Aerospike cluster to new infrastructure to prevent data loss and ensure high availability of services.
  • Implemented ETL processes within the Oozie and EMR clusters, optimizing data workflows and enhancing system performance.
  • Held accountable for data generation and transformation using SPARK, including loading and warehousing data in the Data Lake, as well as visualizing data in PIVOT for both customer product and internal team needs.
  • Enforced DevOps practices across the team, ensuring adherence to processes and efficient utilization of infrastructure resources.
  • Took charge of re-architecting data and systems to improve scalability, reliability, and overall performance.
Extract, Transform, Load (ETL)Core JavaDistributed SystemsBig DataPython (Programming Language)ETL

Software Engineer

Aug 2015Dec 2016 · 1 yr 4 mos

  • During my tenure at ZAPR, I've been deeply involved in shaping and developing the adtech ecosystem. Here are some of my key contributions:
  • I played a pivotal role in designing and developing the adtech ecosystem at ZAPR, ensuring its alignment with company goals and industry standards.
  • From the ground up, I constructed a Real-time Bidding Demand-Side Platform (DSP), encompassing essential components such as ad servers, event servers, and reporting systems, among others.
  • To handle the substantial volume of queries per second (QPS) inherent in adtech operations, I architected and implemented multiple microservices tailored for scalability and performance.
  • I spearheaded the integration efforts, connecting various exchanges with the ZAPR DSP to facilitate seamless transactions and enhance marketplace dynamics.
  • As the focal point for adtech product development, I managed the entire lifecycle, from conceptualization to execution, ensuring the delivery of end-to-end solutions that meet customer needs and industry demands.
Apache SparkJavaAerospikeHadoopPython (Programming Language)Big Data+1

Tata consultancy services

Assistant System Engineer

Oct 2013Jul 2015 · 1 yr 9 mos · Chennai, Tamil Nadu, India

  • Role: Software Developer

Education

Thiagarajar College of Engineering

Bachelor of Engineering (BEng) — Computer Science and Engineering

Jan 2011Jan 2013

Jothi Higher and Secondary School

Tamilnadu Polytechnic College

Diploma in Computer Science and Engineering — CSE

Stackforce found 100+ more professionals with Data Engineering & Big Data

Explore similar profiles based on matching skills and experience