Vijay Ashley Rodrigues

Data Engineer

Bengaluru, Karnataka, India7 yrs 2 mos experience

Most Likely To SwitchAI ML Practitioner

Key Highlights

Expert in modernizing cloud data platforms.
Proven track record in pipeline optimization.
Strong experience in data governance and lineage.

Stackforce AI infers this person is a Data Engineering expert with a focus on cloud-based solutions and pipeline optimization.

Contact

vijayrodrigues18@gmail.com LinkedIn

Skills

Core Skills

Data EngineeringPipeline OptimizationEtl DevelopmentBusiness Intelligence

Other Skills

Advanced SQLAirflowApache AirflowApache SparkAutomationBeautiful SoupBusiness AcumenCTAsCascading Style Sheets (CSS)Cloud ApplicationsCode DesignCommunicationComputer ScienceCritical ThinkingDAGs (Directed Acyclic Graphs)

About

Senior Data Engineer with 6+ years of experience building and modernizing cloud data platforms on Databricks, dbt, Spark, Airflow, and Azure. I focus on reliability, performance, governance, and simplifying complex legacy systems into scalable, maintainable pipelines. • Rebuilt legacy notebook pipelines into modular dbt models, reducing duplicate logic and improving refresh stability • Improved pipeline runtimes through incremental design, better partitioning, and Spark optimization • Built PySpark reconciliation frameworks comparing millions of rows, cutting manual QA effort • Automated recurring Databricks work using Airflow, removing manual steps and improving execution reliability • Migrated key datasets to Unity Catalog to strengthen lineage, governance, and access control • Earlier work also included migrating SSIS workflows to PySpark and building ETL/ELT processes that improved data quality and reduced manual effort • Earlier experience includes automating Python/SQL workflows to streamline reporting and improve accuracy Strong in dbt, Databricks, Spark tuning, SQL optimization, data modeling, Airflow orchestration, reconciliation frameworks, lineage design, and end-to-end pipeline ownership. Current focus: enhancing legacy pipelines and moving them toward more scalable, cloud-friendly designs using Databricks, dbt, and Airflow.

Experience

Chubb

3 roles

Senior Data Engineer - II

Apr 2025 – Present · 11 mos · Bengaluru, Karnataka, India · Hybrid

Rebuilt large sets of legacy Databricks notebook pipelines into modular dbt models, reducing duplicated SQL and improving overall maintainability.
Reduced pipeline refresh failures by improving dependency ordering, adding lineage awareness, and eliminating manual re-runs.
Improved end-to-end pipeline performance through incremental processing, field pruning, and removing unnecessary full-refresh logic.
Built a PySpark reconciliation framework to compare millions of vendor and legacy records, improving alignment with legacy transformations and reducing manual QA effort.
Automated recurring Databricks notebook runs by creating Airflow DAGs with clear scheduling and dependencies, reducing manual execution work.
Improved pipeline reliability by adding automated retries, alerts, and deterministic task ordering.
Reduced pipeline latency by enabling parallel execution of independent Databricks tasks.
Migrated key datasets from Hive Metastore to Unity Catalog, strengthening lineage visibility and access governance.

DatabricksdbtAirflowPySparkSQLData Engineering+1

Senior Data Engineer - I

Promoted

Apr 2023 – Apr 2025 · 2 yrs · Bengaluru, Karnataka, India · Hybrid

Designed and deployed high-performance ETL pipelines in Databricks using Spark, Delta, and Parquet, improving processing speed and optimizing storage usage.
Modernized legacy workflows by migrating SSIS and SQL Server logic to Spark, resulting in faster and more scalable processing.
Built automated ingestion and reconciliation systems using Python, improving validation reliability and reducing manual effort.
Developed optimized data models and transformation logic for Azure SQL to support faster reporting and cleaner downstream workflows.

DatabricksSparkETLSQLData EngineeringETL Development

Senior Associate - Data and BI (Data Engineer : Pyspark + SQL)

Dec 2021 – Apr 2023 · 1 yr 4 mos · Bengaluru, Karnataka, India · Hybrid

Collaborated with business stakeholders and DBAs to gather requirements and streamline post-ingestion workflows, improving overall implementation efficiency.
Migrated legacy SAS scripts to R using RODBC, improving data accessibility and supporting scalable reporting for users across North America.
Automated Price Monitoring Reports using Python, reducing manual reporting effort across business units.
Utilized GitHub and JIRA for documentation and tracking, improving project clarity and coordination; served as an SME during requirement gathering for the Clarity initiative.

PythonRSASSQLData EngineeringBusiness Intelligence

Flip robo technologies llc

Data Science Intern

Jun 2021 – Dec 2021 · 6 mos · Bengaluru, Karnataka, India · Remote

Leveraged Beautiful Soup and Selenium to perform web scraping for efficient data collection.
Developed machine learning models and documented the entire process with detailed project reports and Jupyter Notebooks.
Crafted project presentations to effectively showcase methodologies, findings, and insights for each project.

Beautiful SoupSeleniumMachine Learning

Proziod analytics pvt. ltd.

Data Engineer

Aug 2020 – Dec 2021 · 1 yr 4 mos · Bengaluru, Karnataka, India · Remote

Automated Python- and SQL-based data preparation workflows, reducing recurring manual effort across reporting cycles.
Built standardized ingestion and validation pipelines for Excel/CSV datasets, improving data accuracy and reducing downstream issues.
Designed reusable transformation logic and cleanup scripts, lowering maintenance overhead and enabling more consistent processing.

PythonSQLData Engineering

Teksystems

Technical Recruiter

Jan 2019 – Aug 2020 · 1 yr 7 mos · Bengaluru, Karnataka, India · On-site

Coordinated technical hiring workflows for engineering teams, improving requirement intake and candidate turnaround.
Supported documentation and tracking of hiring activities, enhancing internal process clarity and communication.