Salwa Usmani, Austin Spencer Miller, Devin Kekoa Lane & Jincheng Zhou

Executive Summary

     As artificial intelligence becomes increasingly integrated into the areas of modern financial services, balancing operational efficiency with important ethical standards has become a huge concern. This report assesses the deployment of a Generative AI-powered customer service chatbot by Chase Bank. As it is initially designed to optimize 24/7 accessibility and focus on customer service inquiries, the current design of the system highlights significant issues. These issues primarily revolve around AI hallucinations and the communication of inaccurate financial data and financial information. The chatbot can relay incorrect information that Chase customers rely on. Because banking is a high stakes industry that is subject to strict regulatory oversight, the distribution of unverified financial guidance introduces high legal, financial, and reputational risks for Chase Bank. 

     Chase Bank’s customer service chatbot is focused on trust, transparency, and accountability. When looking over the chatbot, it shows that the initial design of the chatbot prioritized speed over accuracy, which lacks the important responsible AI safeguards it needs. However, rather than getting rid of the chatbot, which would get rid of its accessibility benefits, this consulting report recommends a comprehensive redesign and redeployment strategy for Chase Bank’s customer service chatbot. By shifting the system from efficiency to trust and accuracy, implementing continuous human oversight, and ensuring that all AI generated responses are within the official Chase policy, the bank can change this issue into a strategic advantage, establishing itself as a leader in responsible AI.

Description of the Client and AI System

     Our client for this assessment is Chase Bank, which is a major national financial services institution. Chase operates as a large national bank subject to strict financial regulation and oversight. Customer trust and reliability are critical in the banking industry, especially when providing financial services and information. Chase has already taken steps to integrate AI-driven customer service tools within its digital application and website. These tools are designed to allow customers to access their credit journeys and provide them with tailored tips, as well as personalized credit score guidance.

     The specific product under review is a Generative AI-powered customer service chatbot that is used within Chase’s online banking system. This system is an extensive conversational tool that is used in high stakes financial contexts. Its main function is to answer questions that customers may have and provide financial information. The system operates by taking customer questions as inputs and combining them with internal bank policy and data sources. The resulting outputs consist of financial guidance, such as explanations of interest rates, fee structures, and specific product information.

     The deployment and potential errors of this system impact a broad network of stakeholders. The Chief Risk Officer is directly responsible for managing the legal, regulatory, and reputational risks associated with the bank's AI use. Customer service agents are also heavily impacted, as they rely on accurate chatbot outputs but are currently burdened with handling disputes and correcting customer issues when the AI makes mistakes. Ultimately, the customers themselves are the most affected stakeholders, as they depend on the chatbot to provide accurate financial information to make informed decisions for their financials. Furthermore, regulators and compliance teams are tasked with ensuring the AI system meets financial regulations, while society in general must be able to maintain trust in the AI systems utilized in the banking industry. Chase Bank’s brand and reputation depends on how accurate AI information from the chatbot is.

Key Responsible AI Risks and Opportunities

     The primary Responsible AI risk in this system is hallucination (data inaccuracy), AI chatbot generates inaccurate or misleading information. For example, AI Chatbot may present incorrect interest rates, loan terms, or fees. Inaccurate financial information can mislead clients and lead to future customer trust issues and violate certain policies.

If inaccurate information in such a case happens the issue may be pretty serious, since it's affecting customers making decisions regarding their financial information. Financial regulations require accurate and compliant information. In the end, banks will have to pay responsibility for the misinformation.

     Some real-world examples are, Air Canada case in 2024: Airline ordered to pay damages after its chatbot came up with a fake bereavement fare policy and misled a customer. Consumer Financial Protection Bureau warning in 2023: US regulator warned banks that inaccurate AI Chatbot information may violate consumer protection laws. NYC Government Chatbot in 2024:

In addition to hallucinations, the system lacks sufficient transparency. Customers are not always aware of how responses are generated. This reduces trust with the customers.

The opportunities that these chatbots can provide, when properly designed, are AI chatbots that can significantly improve customer service accessibility, reduce operational costs for the bank, and provide support for large customer bases.

The challenge is not whether to use AI, but how to use it responsibly and minimize as many risks as possible so it may last for the future.

Design–Develop–Deploy Analysis

     For the current state, the chatbot is designed primarily to focus on efficiency and accessibility. It may respond to a large number of customers. This design is more about speed than accuracy, so there is a trade-off happening.

Responsible AI elements such as transparency and accountability are not fully considered. Customers are not consistently informed that responses are AI-generated, nor are responses always linked to reliability.
In the future state the system’s design should be focused on trust and accuracy. Transparency should be integrated into the user experience by linking each response to official Chase policy. Customers should also be clearly informed when they are interacting with an AI system. This shift reframes the system’s purpose from simply providing fast answers to delivering reliable financial guidance.

Recommendations

     In order to mitigate the identified risks, Chase should consider the following actions: - Adjust the functionality of the chatbot so that it refers only to approved policy and procedures which have been checked by personnel and are stored within the internal policy database. - Include wording to the effect that the responses provided by the chatbot are the product of an AI process and including a link to the relevant policy.

     Chase needs to implement KPIs to monitor the performance of the system. Metrics that could be measured include: - Accuracy rate - Hallucination rate - Source citation rate The model should not be released until it reaches the minimum required level of accuracy (as determined by Chase). - Model audits need to occur regularly; weekly model audits are recommended with a full model review occurring every 3 months (i.e. on a quarterly basis).

     To address the governance perspective we will need to ensure that there is some level of human oversight of the chatbot. This will mean that we will be constantly monitoring and reviewing the output of the chatbot and will potentially include high risk questions such as loan terms and interest rates in a secondary approval step that requires a human to review and approve the chatbot’s response to these questions. This will greatly reduce our legal and compliance risk as well as customer complaints. We at Chase want to be at the forefront of implementing responsible practices for the use of Artificial Intelligence. We feel that by doing so we will be positioned as a leader in the trustworthy financial technology space and will position Chase for long term competitive advantage.

Conclusion

     These are good safety guidelines for this chatbot because they address a number of the risk factors mentioned. The company only trains the chatbot on a knowledge base that it has validated. This will minimize the potential for hallucinations. It discloses the source of the information to the user and verifies that information before providing it to the user. This will help increase customer trust. A live person checks the output to make sure it wasn’t marked as incorrect before it’s actually published to the public.

     AI-related errors and incidents have been the subject of many academic papers and court cases, from the Air Canada chatbot case where a passenger was awarded $7,500 after an error was held responsible for a flight cancellation. It is recommended to add evaluation metrics and governance rules to these systems, even if it is impossible to guarantee that no errors will occur. Rather than being unavoidable, these errors should be less frequent, more predictable, and easier to detect and correct, in order to improve customer satisfaction, as well as the regulatory compliance of organizations.
 

Works Cited

“CFPB Issue Spotlight Analyzes Artificial Intelligence Chatbots in Banking.” Consumer Financial Protection Bureau,
https://www.consumerfinance.gov/about-us/newsroom/cfpb-issue-spotlight-analyzes-artificial-intelligence-chatbots-in-banking/.
Accessed 10 Mar. 2026.

“Chase.” JPMorgan Chase & Co.,
https://www.chase.com/.
Accessed 10 Mar. 2026.

“Customer Service.” Chase,
https://www.chase.com/digital/customer-service.
Accessed 10 Mar. 2026.

“The Business Case for Responsible AI.” EY (Ernst & Young),
https://www.ey.com/en_us/insights/ai/the-business-case-for-responsible-ai.
Accessed 10 Mar. 2026.

“Malfunctioning NYC AI Chatbot Still Active Despite Widespread Evidence It’s Encouraging Illegal Behavior.” The Markup, 2 Apr. 2024,
https://themarkup.org/artificial-intelligence/2024/04/02/malfunctioning-nyc-ai-chatbot-still-active-despite-widespread-evidence-its-encouraging-illegal-behavior.
Accessed 10 Mar. 2026.