How SMBC-GAI Learned to Cross-Search 1.3 Million Internal Documents: Behind the Scenes of Japan’s Largest Class RAG Implementation

When we first launched in July 2023, the system handled around 6,000 messages per day. Since then, we’ve responded to internal feedback and progressively expanded its capabilities, including providing multimodal support and adding features such as generating meeting minutes from audio files, and drafting research reports. Today, SMBC-GAI processes around 70,000 to 80,000 messages per day, and that just continues to grow.

The response has been amazing. As more people have started to use the system, requests have begun to pour in from the various departments wanting to use AI to boost their operational efficiency. Some departments have even been developing AI tools trained solely on their own data.

In reality, it should be enough to have one single AI capable of comprehensively understanding internal information—if individual units build their own separate tools, the result is weaker AI. That worried us so much that we started to look at creating an organization-wide AI.

One longstanding headache was the amount of time it took to search for internal information. With so much information across multiple categories—from internal regulations and notices to operation manuals—searches took forever. A questionnaire we sent out after trialing the enhanced system in head office departments showed that the new system could improve efficiency by around eight hours per month.

RAG works by retrieving documents in response to a user’s query and providing a summary of the relevant results. We set out to consolidate approximately 1.3 million internal files into a single searchable repository, but preparing the data turned out to be a real challenge.

We limited the scope to information that all employees are authorized to access, and even then the number of files reached 1.3 million.

At the same time, because some departments wanted to train AI models on data accessible only within their own units, we also instituted access controls, creating a mechanism whereby the AI could be trained in an environment accessible only by a particular department. Overall, we have built a security framework that can also withstand external cyberattacks and other threats.

Because this wasn’t our only project and we had to tackle it alongside all our other regular responsibilities, resource constraints were a constant headache.

We had nowhere near enough hands on deck for this kind of large-scale RAG implementation—so we decided to bring in Yoshida and Okayasu from the Japan Research Institute (JRI; the SMBC Group thinktank), which served the additional purpose of developing internal talent.

The SMBC-GAI rollout marked the starting point for broader generative AI efforts within the organization. There wasn’t a lot of information on AI around then, either internally or externally, so we were playing a trial-and-error catchup game. Today, Mr. Hyodo has us also handling RAG architecture design and the application of cutting-edge AI technologies.

We’re approaching AI development at the kind of pace where we’ll be testing today a technology that just emerged yesterday. We’ve studied the latest developments in generative AI and RAG from English-language sources and once even took part in Microsoft events in the United States so that we could bring back the latest insights and build them into SMBC-GAI. I think that looking not just domestically but globally has been a key factor in the project’s success.

Banking operations are complex and use a lot of specialized terminology, so a detailed understanding of the banking business is essential. Explaining the nuances from scratch to an external vendor just wasn’t realistic.

There were also many situations—like dealing with hallucinations—where only the developers actually writing the code could find an answer. Close, ongoing dialogue between developers and users was essential. We felt that observing operations with our own eyes and tailoring our AI development accordingly was the only way to create a tool that truly adds value.

Our testing showed that hallucinations occur in roughly ten percent of all cases regardless of the AI model used, and that increases along with the level of complexity of the scenario. Right now, if it’s something where even a human feels that they should double-check just in case, the AI too is highly likely to be wrong. That’s just the technological limit at this point in time.

We consistently remind users that because generative AI outputs may contain hallucinations, they need to be fully aware that the responsibility for the generated content lies with the user and to use GenAI outputs at their own risk. In addition, SMBC-GAI is designed to show the sources for its answers so that users can check images of the original documents. Presenting the images providing the basis for the answer could be paying off, because there have been zero operational incidents so far. When information is provided just as text, some users don’t check the reference links, whereas visualizing the supporting material as images makes it easier for them to spot any issues.

Yes, we converted all the files from their original formats—PowerPoint, text documents, PDFs, etc.—into PDFs and image data optimized for AI learning. The AI extracts and learns information from these converted files, and users get responses along with the original page images. In the end, to train SMBC-GAI on all 1.3 million files, we designed a system to run 2,000 virtual machines in parallel so that the processing could be completed within 48 hours.

I’ve been told by Microsoft and other companies that this is the first case they’ve heard in Japan of training on a scale of 1.3 million files, so I think it’s fair to say that this represents the largest RAG implementation in Japan.

I think it’s because SMBC Group has such a strong pool of talent able to drive GenAI use. You need skilled, hands-on practitioners who are deeply familiar with your data and operational environment and who can build solutions in the cloud, along with an environment where they can experiment freely without being bound by procedures and restrictions. Given that level of autonomy, people with a sense of curiosity naturally absorb new technologies and continue to grow.

At the same time, there are very few organizations with the capacity to define and nurture talent, create the right environment, and manage expectations appropriately around outcomes.

SMBC Group has a strong culture of embracing new initiatives and encouraging people to give something a go. The value that SMBC places on employees’ individual strengths was another factor supporting this project.

SMBC Group is proud to have been first among the megabanks to introduce Microsoft 365 and directly leverage cloud services. That early spirit of challenge became the driving force that has positioned us to continue leading at the cutting edge even in the current AI era.

People are loving being able to reference both external and internal information. Typically, when new features are released, we get a lot of grumbles about usability at first. This time, we’ve heard a lot of positive comments from relatively early on. Even departments with limited previous interaction with the tool have been telling us that now they couldn’t operate without it. The response has really exceeded our expectations.

Before the update, SMBC-GAI was used primarily by head office departments, but recently we’ve seen steady uptake in branch offices too. We think it might be because SMBC-GAI makes branches more self-sufficient—in other words, they can now find their own answers to queries they previously had to direct to head office.

When we talk to users, we’re often surprised at how they’re using SMBC-GAI, and we incorporate the best ideas as new features. It’s great to see user creativity now driving the tool beyond our original vision.

It’s amazing how SMBC-GAI has evolved from an idea that was being kicked around in conversation by the team into a highly professional AI tool.

As a JRI employee, I’d like to leverage the technical insights and AI utilization know-how gained through our experience so far to benefit the entire SMBC Group. I want to be a kind of messenger connecting the operational frontlines with the technology.

Our next goal is to have AI handle roughly 70-80% of the materials currently produced by humans. Now that we have an AI familiar with the workings of the SMBC Group, we want to create AI agents capable of taking over some knowledge work. We’re also considering integration with low-code development tools in response to requests from within the organization to achieve specific objectives using SMBC-GAI.

Looking ahead, we plan to implement various upgrades that combine multiple LLMs to meet employees’ wide-ranging needs and take operational efficiency to the next level.

We’re also introducing Copilot to handle routine tasks like general information searches—news and financial reports, that kind of thing—while SMBC-GAI handles specialized tasks like searches for SMBC-specific information. Our goal is to continue delivering surprises, such as firsts among Japanese banks or internationally, at a pace of roughly one to three per year.

There are so many GenAI tools available outside our own SMBC-GAI that it’s impossible to pick just one as the best and only solution for us in the coming years. It will be important to track the evolution of multiple tools, apply them broadly, then identify the best solutions from among them.