AI Engineer - Monitoring, Observability, and Governance

Job Description: At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. We do this by driving Responsible Growth and delivering for our clients, teammates, communities and shareholders every day. Being a Great Place to Work and providing a culture of caring is core to how we drive Responsible Growth. We are intentional about fostering an inclusive workplace where every teammate has the opportunity to succeed, build a career and contribute to our shared success. This includes attracting and developing exceptional talent, recognizing and rewarding performance, and supporting our teammates’ physical, emotional, and financial wellness through affordable, competitive and flexible benefits. We value the unique perspectives individuals bring from all backgrounds and career paths - whether shaped by military service, community college education, or a wide range of work and life experiences. These journeys foster resilience, leadership and innovation, strengthening our workforce and positively impact the communities we serve. Bank of America is committed to an in-office culture that supports collaboration, engagement, and career development. Our approach includes clear in-office expectations, while providing an appropriate level of flexibility based on role-specific responsibilities and business needs. At Bank of America, you can build a successful career with opportunities to learn, grow, and make an impact. Join us! Job Description: This role is responsible for engineering and operating third-party applications within an on-premises OpenShift environment supporting enterprise GenAI monitoring, observability, and governance. The position will develop and integrate application components, deploy and support containerized services, and maintain AI guardrail and observability platforms such as Galileo and LangSmith. The engineer will monitor application quality, safety, and performance while helping operationalize model-based evaluation workflows and supporting enterprise AI governance initiatives. The domain is GenAI operations: the applications you run provide guardrails and observability for LLM- and RAG-powered systems across the enterprise. You will work with these vendor tools to monitor application quality, safety, and performance—tracking metrics such as attribution, grounding, prompt-injection detection, tone, bias, and PII handling—and surface them through dashboards and alerting. Where the platform uses model-based evaluation (LLMs-as-Judges and specialized/small language models), you will help operationalize, calibrate, and maintain those workflows. Because the AI landscape changes quickly, the ideal engineer learns new tools and environments fast, brings a pragmatic build-and-operate mindset, and can lead cross-functional work to move initiatives forward. Responsibilities: Develop, deploy, and support vendor applications in a fully on-premises OpenShift environment. Build APIs and integrations connecting guardrail and observability tools with enterprise GenAI applications. Implement and maintain operational monitoring for application performance, safety, latency, and quality metrics. Develop and calibrate model-based evaluation workflows including annotation and feedback processes. Utilize orchestration frameworks such as LangChain for guardrails, prompt management, and scoring workflows. Create observability dashboards and alerting capabilities. Troubleshoot production issues and improve scalability, reliability, and performance. Support emerging agentic and multi-step AI workflows. Lead and mentor team members while driving engineering standards and cross-functional delivery. Required Qualifications: Bachelor's or Master's degree in Computer Science, Data Science, MIS, related field, or equivalent experience. 10+ years of software engineering experience designing, building, and scaling production applications. Strong Python development experience, including data and machine learning libraries. Experience deploying and supporting containerized workloads on OpenShift or Kubernetes. Experience taking LLM or RAG-based applications into production. Understanding of AI risks, agent behavior, and governance frameworks. Experience building APIs and integrating third-party platforms. Experience with OpenAI APIs and orchestration frameworks such as LangChain. Understanding of inference performance optimization and observability. Experience implementing AI guardrails and model-based evaluation frameworks. Experience working in Agile environments. Demonstrated technical leadership and stakeholder influence. Desired Qualifications: Experience delivering technology solutions in large, complex, regulated organizations, preferably financial services. Experience with annotation pipelines, feedback loops, fine-tuning, and alignment techniques. Familiarity with prompt versioning and lifecycle management. Experience with Galileo, LangSmith, Splunk, or similar observability platforms. Knowledge of additional GenAI frameworks such as LlamaIndex or Haystack. Experience supporting evaluation and guardrails for agentic AI workflows. Strong collaboration skills and ability to quickly learn emerging technologies. Skills: Application Development Automation Influence Solution Design Technical Strategy Development Architecture Business Acumen DevOps Practices Result Orientation Solution Delivery Process Analytical Thinking Collaboration Data Management Risk Management Test Engineering Shift: 1st shift (United States of America) Hours Per Week: 40 Pay Transparency details US - NJ - Jersey City - 101 Hudson St - 101 Hudson (NJ2101), US - NJ - Pennington - 1600 American Blvd - Hopewell Bldg 6 (NJ2600), US - VA - Richmond - 8011 VILLA PARK DR - VILLA PARK OPS CENTER BLDG B (VA2125) Pay and benefits information Pay range $102,900.00 - $179,900.00 annualized salary, offers to be determined based on experience, education and skill set. Discretionary incentive eligible This role is eligible to participate in the annual discretionary plan. Employees are eligible for an annual discretionary award based on their overall individual performance results and behaviors, the performance and contributions of their line of business and/or group; and the overall success of the Company. Benefits This role is currently benefits eligible. We provide industry-leading benefits, access to paid time off, resources and support to our employees so they can make a genuine impact and contribute to the sustainable growth of our business and the communities we serve.