Senior Software Engineer - SRE, Retail and Pharmacy

We’re building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you’ll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time. Requisition Job Description Position Summary: About the Team Our Site Reliability Engineering team is the execution engine behind the reliability, availability, and performance of distributed store technology powering thousands of retail and pharmacy locations nationwide. We operate across pharmacy platforms, Point of Sale (POS) systems, handheld devices, store servers, dispensing systems, and edge computing infrastructure — spanning hybrid cloud and on-premises environments at massive fleet scale. Our engineering philosophy is grounded in five pillars: Detection , Prevention , Recovery , Learning Loops , and Developer Experience (DevX) . Our operating principle is the reliability covenant : our success is not measured by incident response volume — it is measured by the reliability capability we transfer to the engineering teams we serve. Your success in this role is measured by what the engineering teams in your domain can do independently after working with you , not by how indispensable you become to them. An SSE who has enabled a development team to detect, respond to, and learn from production failures without SRE involvement has delivered the highest-value outcome this role can produce. We track operational toil as an engineering metric . Engineers at this level are expected to identify recurring manual work, eliminate it through automation, document the reduction, and treat toil accumulation as a reliability risk — not as a sign of operational expertise. About the Role As a Senior Software Engineer — SRE , you independently own the reliability posture of an assigned engineering domain. You are not waiting for direction — you are setting it for your domain. You design the alerting strategy, own the SLO health, lead incident command for production issues, facilitate postmortems, tune anomaly detection models, and partner directly with engineering domain owners to shift reliability left into design. You are a technical mentor to SE-level engineers and an escalation resource during active incidents. You have the technical depth to diagnose complex distributed system failures, the data instincts to distinguish genuine anomalies from noise in ML-generated signals, and the organizational skills to drive reliability practice adoption in teams that did not necessarily ask for SRE involvement. Scope: Domain ownership — you operate independently and influence adjacent engineering teams. The Environment You Are Joining This role exists inside an active SRE transformation. The domain-based SRE ownership model you will operate within is in its early stages. Some of the toolchains you will work with are being built in parallel with the operational work. Engineering domain owners are simultaneously learning what SRE can offer them. This is an honest description of the role, not a caveat. The opportunity is to shape a domain's reliability posture from the ground up in a large-scale, consequential technology environment. The engineering decisions you make will affect pharmacy dispensing, prescription fill workflows, and store operations across thousands of locations. If you are energized by the combination of technical depth and organizational building, the scope here is significant. If you are looking for a mature, fully-defined SRE environment where the frameworks and toolchains already exist, this role will feel like a different challenge. Success in this role requires patience alongside technical rigor : you will demonstrate value before you demand process change, build credibility before you expect adoption, and earn trust with engineering domain owners through partnership rather than mandate. The operating environment includes an edge computing fleet deployed directly inside store locations — unattended nodes where deployment blast radius is geographic and fleet-wide, not functional and service-scoped. You will develop a fleet operations mindset: the primary failure mode in this environment is deployment and configuration propagation, not service logic. What You Will Do Detection & Observability Own SLI/SLO health for your assigned domain end-to-end; monitor error budget burn rates and drive proactive burn-down actions before incidents reach end users or pharmacy patients Design multi-signal alerting strategies that go well beyond threshold alerts: burn rate alerting, composite health signals, and anomaly-based detection using time-series models — and validate their output against production ground truth Tune and maintain ML-based anomaly detection models in production: adjust sensitivity thresholds, evaluate false positive rates against alert fatigue metrics, and iterate on model configurations based on incident data Build Critical User Journey (CUJ)-anchored dashboards that surface end-to-end business flow health — not just individual service metrics Drive alert signal-to-noise improvement across the domain; own suppression policies during deployments and maintenance windows to protect on-call quality Identify and instrument unmonitored external dependencies — third-party APIs, downstream services, data providers — using tools like Prometheus Blackbox Exporter , OpenTelemetry , or custom health probes Contribute to the observability platform design: understand the hot/warm/cold tier architecture ( Apache Kafka → ClickHouse → cold storage) and contribute domain-specific data models and SLI schemas Prevention & Reliability Engineering Lead Production Readiness Reviews (PRR) for services in your assigned domain; own the readiness gate sign-off and be accountable for what makes it into production on your watch Design and execute fault injection experiments at service level using tools such as LitmusChaos , Chaos Toolkit , or Gremlin ; validate blast radius assumptions before rollouts reach production Partner with engineering domain owners on reliability requirements during architecture design and sprint planning — reliability is designed in, not bolted on Own dependency risk mapping for your domain: catalog third-party API failure modes, shared infrastructure failure paths, and chain-wide blast radius scenarios Apply a fleet operations mindset to change management: for any configuration or deployment change touching the edge fleet, assess deployment blast radius by node cohort, review the rollback procedure, and contribute to the go/no-go decision on high-risk changes Understand progressive rollout strategies — canary cohorts, staged fleet expansion, blast radius budgets — and apply them in high-risk deployment reviews Incident Response & Recovery Serve as the primary on-call Technical Incident Commander (IC) for domain incidents; drive structured bridge calls from detection to resolution using established incident command frameworks Author and maintain P0/P1 runbooks with validated, step-by-step remediation procedures; own a quarterly review and dry-run testing cadence to ensure runbooks are accurate when they are actually needed Lead post-incident reviews using structured root cause analysis: causal chain documentation, origin layer classification, and contributing factor identification — not just a timeline of what happened Serve as the real-time escalation point and technical decision support for SE engineers during active incidents Pursue Technical Incident Commander (TIC) certification; qualify as a cross-domain IC candidate available to lead major incidents beyond your assigned domain Learning Loops & Continuous Improvement Learning Loops at this level is systems engineering for organizational memory — not postmor