Vivek Kulkarni — Data Engineer

As an experienced data engineer with over 6 years of work experience, I have a deep understanding of building and optimizing end-to-end data pipelines using various technologies, including Python, SQL, Apache Spark, Airflow, dbt, Tableau, Power BI, and Apache Superset. I specialize in using Airflow to manage complex workflows and automate data processing tasks. I have extensive experience in creating and scheduling DAGs (Directed Acyclic Graphs) to automate ETL processes, and I am proficient in writing custom operators and sensors to integrate with various data sources and sinks. In addition, I have expertise in dbt, a popular open-source tool for building and maintaining data models in a data warehouse. I have worked extensively with dbt to create modular, reusable data models that enable efficient data analysis and visualization. I am also skilled in using dbt to test data quality and ensure data integrity across the entire data pipeline. I specialize in designing and implementing Spark-based data solutions that process vast amounts of data in real-time or near-real-time to enable faster and more accurate decision-making. I have extensive experience in developing Spark jobs to extract, transform, and load data from a variety of sources, including structured, semi-structured, and unstructured data. Throughout my career, I have collaborated with cross-functional teams to deliver data-driven solutions that meet the unique needs of organizations. I am committed to staying up-to-date with the latest technologies and best practices in data engineering to deliver cutting-edge solutions that drive business value.

Stackforce AI infers this person is a Data Engineer specializing in SaaS solutions with a focus on data pipeline optimization.

Location: San Francisco, California, United States

Experience: 9 yrs 7 mos

Skills

Data Engineering
Etl
Business Intelligence

Career Highlights

Over 6 years of experience in data engineering.
Expert in building and optimizing data pipelines.
Proficient in using Airflow for workflow management.

Work Experience

Fox Corporation

Data Engineer (2 yrs 8 mos)

Northeastern University

Graduate Teaching Assistant (4 mos)

Online Program Technician (1 yr 8 mos)

Tesla

Data Engineer (1 yr 5 mos)

Siemens PLM Software

Associate Software Engineer (2 yrs 2 mos)

Graduate Trainee Engineer (9 mos)

Education

Master's degree at Northeastern University

Bachelor of Engineering - BE at Savitribai Phule Pune University

Vivek Kulkarni

Data Engineer

San Francisco, California, United States9 yrs 7 mos experience

Most Likely To Switch

Key Highlights

Over 6 years of experience in data engineering.
Expert in building and optimizing data pipelines.
Proficient in using Airflow for workflow management.

Stackforce AI infers this person is a Data Engineer specializing in SaaS solutions with a focus on data pipeline optimization.

Contact

Skills

Core Skills

Data EngineeringEtlBusiness Intelligence

Other Skills

AWSAirflowSnowflakedbtApache KafkaApache SparkLookerApache SupersetTableauMicrosoft OfficeBusiness Intelligence (BI)Big DataMicrosoft SQL ServerExtract, Transform, Load (ETL)Databases

About

Experience

9 yrs 7 mos

Total Experience

1 yr 9 mos

Average Tenure

1 yr 11 mos

Current Experience

Fox corporation

Data Engineer

Oct 2021 – Jun 2024 · 2 yrs 8 mos · San Francisco Bay Area

Build configurable and scalable data pipeline around AWS, Airflow and Snowflake and deploy using Docker
containers and maintain them for smooth functioning and usability
Design data warehouse using dimensional modeling for insurance domain and build dbt models to transform the data before loading it
into snowflake and leverage Looker to build insightful dashboards
Improve performance and usability of snowflake warehouse by altering data models to save costs.
Support analysts and data scientists with all the ad-hoc requests for data ingestion and data retrieval from multiple sources.
Created PySpark scripts to interact with Hadoop data lake for data transformation, performed ETL to provide specific
insights from set of signal data and store the extracted data into data lake
Proficiently implemented and managed the Snowplow event data analytics platform, enabling real-time data processing with Apache
Kafka and Amazon Kinesis.
Customized and fine-tuned the platform for optimal data collection, enrichment, and analysis, resulting in data-driven insights that
informed strategic decision-making.
Consume Google Search Console API and Google Ads API to fetch keywords, clicks and ad related data using authentication tokens and use Airflow to build DAGs to do the analysis and cost estimations of Ads spend to help marketing team planning
Use CloudFormation to create IAM roles, S3, & DynamoDB & make DAG to load ML model data in DynamoDB to Snowflake
Update lender report data parser to process new files, fields, & status mapping & add dbt tests for data qualitychecks
Designed and constructed a comprehensive data warehouse from the ground up for a recently acquired insurance company, overseeing
the end-to-end migration of data and systems, ensuring a seamless transition and enabling streamlined data management and analysis for the entire organization

AWSAirflowSnowflakedbtApache KafkaApache Spark+3

Northeastern university

2 roles

Graduate Teaching Assistant

Jan 2021 – May 2021 · 4 mos

Microsoft Office

Online Program Technician

Sep 2019 – May 2021 · 1 yr 8 mos

Microsoft Office

Tesla

Data Engineer

May 2020 – Oct 2021 · 1 yr 5 mos · Boston, Massachusetts, United States

Created Spark scripts to interact with Hadoop data lake for data transformation, performed ETL to provide specific insights from a set of signal data, and store the extracted data into data lake
Developed interactive dashboard using Apache Superset and designed KPIs to monitor the performance of solar devices, also created dashboards in Tableau for specific requirements
Scheduled various scripts as DAGs to run via Apache Airflow to get daily and updated signal values from the devices for dashboards
Provided various Excel VBA reports for Account Managers to keep track of non-closed tickets to take appropriate actions
Consumed multiple APIs to get the required set of data from JSON objects and create useful dashboards for easier tracking of entities

Apache SupersetTableauAWSData EngineeringBusiness Intelligence

Siemens plm software

2 roles

Associate Software Engineer

May 2017 – Jul 2019 · 2 yrs 2 mos · Pune Area, India

Reviewed, developed, and designed data models, in conjunction with the application development team, created complex Stored
Procedures, Triggers, Tables, Cursors, Views and SQL Joins and made extensive use of Dynamic SQL scripting
Developed interactive dashboard and designed KPIs to monitor usage of Azure and draw insights resulting in 15% cost reduction
Prepare documents such as expected benefits, gap analysis, use cases, models, current and propose process, workflows, data flows,
implementation plans and end user guides in accordance with standards and methodologies
Established and enforce practice wide UX/UI standards and worked closely with the design team to evaluate and coordinate bringing
mock-ups to production while improving user experience for web solutions
Responsible for Data Sourcing, Data Cleansing, Data Integrity and solved errors by analyzing the linkages on the missing data

Business Intelligence (BI)AWSBusiness Intelligence

Graduate Trainee Engineer

Jul 2016 – Apr 2017 · 9 mos · Pune Area, India

Collaborated with scrum team for continuous product integration by sprint planning, backlog refinement & task management
Managed database integration issues including migration between disparate databases using SQL Server Integration Services
Improved the runtime of the database by 30% through analyzing the data and documenting the queries in SQL Server

AWSETL