Data Engineer Lead

Data Engineer Lead AIG | Data Office · Data Engineering About The Role The Data Engineer Lead is a hands-on engineering role within AIG's Data Engineering organization. You will design, build, and operate production-grade data pipelines and platforms — and bring sound engineering judgment to every solution you deliver. We actively leverage GenAI tooling — including Claude Code and Snowflake Cortex — as a genuine productivity multiplier across the development lifecycle. We want someone who is motivated to learn, keeps up with a fast-moving space, and continuously finds new ways to put these tools to work. You will collaborate closely with the product owner, business stakeholders, and fellow engineers. Strong communication and analytical skills matter here as much as technical depth — the ability to ask the right questions, think through trade-offs clearly, and explain your reasoning is part of the job. Responsibilities Design, build, and operate production-grade PySpark and Python pipelines on AWS EMR, Glue, and S3 with integrated data quality checks and observability. Own end-to-end delivery of data products from requirements through production, working within an Agile/Scrum framework — including sprint planning, CI/CD, release management, and production readiness Translate business and product requirements into concrete technical designs, defining pipeline structure, data models, and SLAs. Demonstrate a sound understanding of architecture and design patterns, evaluate trade-offs of design decisions, and articulate clear rationale to peers and stakeholders. Set and enforce engineering standards — coding conventions, data quality frameworks, reusable pipeline patterns, and observability hooks — across the team. Conduct thorough code reviews and pair with engineers on hard problems, raising the technical quality of the whole team. Leverage GenAI tooling — Claude Code CLI, Snowflake Cortex — throughout the development lifecycle to write better code faster and improve solution quality. Collaborate with product owners to refine requirements, surface trade-offs, and push back constructively when scope is technically unworkable. Communicate design decisions and delivery updates clearly to both engineering peers and non-technical stakeholders. Mentor junior engineers through review, pairing, and knowledge sharing, building team capability alongside product delivery. Produce clear technical documentation — Architecture Decision Record (ADR), runbooks, data dictionaries, and pipeline lineage notes — as a standard part of delivery, not an afterthought. Evaluate and drive adoption of productivity tooling and best practices, setting team norms for safe and effective use. Experience/Skills Required Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field. 7+ years of hands-on data engineering experience with a consistent track record delivering production data pipelines end-to-end. Strong command of PySpark, Python, and SQL — including Spark optimization, partition management, and query performance tuning in Snowflake. Hands-on experience with AWS data services: S3, Glue, EMR, Aurora, Lambda, IAM, and CloudWatch. Working knowledge of architecture and design principles — Lakehouse patterns, data modeling, fault-tolerant ingestion — with demonstrated ability to apply them in practice. Strong analytical and problem-solving skills: able to decompose complex requirements, reason through design trade-offs, and arrive at practical, well-justified solutions. Clear communicator — able to explain technical decisions and design rationale in plain language to both engineering peers and business stakeholders. Self-motivated learner with a genuine interest in staying current with evolving tools, frameworks, and engineering best practices. Preferred/Nice To Have AWS certification — Solutions Architect Associate or equivalent. Domain familiarity with insurance or financial services data — claims, policy, risk, or actuarial structures. Experience writing ADRs, design notes, or technical documentation in a collaborative engineering team. Demonstrated use of GenAI tooling (e.g. Claude Code CLI or Snowflake Cortex) to improve productivity or solution quality. Our Culture AIG's Data Office is a team of builders, thinkers, and pragmatic problem-solvers. We move with the urgency of a technology company and the accountability of a global insurer. Great data engineering is a team sport here. We write documentation as carefully as we write code, review designs together, mentor across every level, and keep business stakeholders close to the work. We lean into GenAI not as a novelty but as a genuine force multiplier — and we want people who share that mindset. AIG is committed to a diverse and inclusive workforce and encourages candidates from all backgrounds to apply. We offer competitive compensation, comprehensive benefits, and hybrid flexibility. “The base salary range for this position is $125,000-135,000 and the position is eligible for a bonus in accordance with the terms of the applicable incentive plan. Your actual compensation will be dependent on your skills, experience, and qualifications. If your compensation expectations are above the posted range, we still encourage you to apply—your unique background matters to us. In addition, we’re proud to offer a range of competitive benefits, a summary of which can be viewed here: https://sprcdn-assets.sprinklr.com/248/7cae7257-96d1-4207-bc2c-e1b8447826e3-220371966/2026_AIG_Benefits_Overview.pdf . #LI-PS1 At AIG, we value in-person collaboration as a vital part of our culture, which is why we ask our team members to be primarily in the office. This approach helps us work together effectively and create a supportive, connected environment for our team and clients alike. Enjoy benefits that take care of what matters At AIG, our people are our greatest asset. We know how important it is to protect and invest in what’s most important to you. That is why we created our Total Rewards Program, a comprehensive benefits package that extends beyond time spent at work to offer benefits focused on your health, wellbeing and financial security—as well as your professional development—to bring peace of mind to you and your family. Reimagining insurance to make a bigger difference to the world American International Group, Inc. (AIG) is a global leader in commercial and personal insurance solutions; we are one of the world’s most far-reaching property casualty networks. It is an exciting time to join us — across our operations, we are thinking in new and innovative ways to deliver ever-better solutions to our customers. At AIG, you can go further to support individuals, businesses, and communities, helping them to manage risk, respond to times of uncertainty and discover new potential. We invest in our largest asset, our people, through continuous learning and development, in a culture that celebrates everyone for who they are and what they want to become. Welcome to a culture of inclusion We’re committed to creating a culture that truly respects and celebrates each other’s talents, backgrounds, cultures, opinions and goals. We foster a culture of inclusion and belonging through learning, cultural awareness activities and Employee Resource Groups (ERGs). With global chapters, ERGs are a cornerstone for our culture of inclusion. The talent of our people is one of AIG’s greatest assets, and we are honored that our drive for positive change has been recognized by numerous recent awards and accreditations. AIG provides equal opportunity to all qualified individuals regardless of race, color, religion, age, gender, gender expression, national origin, veteran status, disability or any other legally protected categories. AIG is committed to working with and providing reasonable accommodations to job applicants and employees with disabilities. If you believe you need a reasonable accommodation, please send an