Internal Chat System Development
European Consulting Firm | Internal Project
Overview
Developed and implemented the internal inquiry chat as RAG (retrieval-augmented generation).
RAG first searches for documents related to the question, and an LLM then writes the answer from their content.
What I Was Responsible For
Handled the work below. Search-quality work and CI/CD are covered in later sections.
- Azure environment: set up the Azure environment.
- RAG environment: built the environment that runs the chat as RAG.
- Japanese support: added Japanese support to the chat.
How It Works
Internal documents are the search target: Azure AI Search retrieves them and Azure OpenAI Service generates the answer, combined with other Azure services.
Improving Search Quality
To improve search quality, tried and implemented the following.
- Search methods: compared keyword search, vector search, hybrid search, and the semantic ranker. In general, keyword search finds documents by matching words, vector search finds them by closeness in meaning, and hybrid search uses both. The semantic ranker reorders the retrieved results by how close they are in meaning to the question.
- Document count and chunk size: tuned how many documents go into an answer and how large the document chunks are. In general, more documents or larger chunks pass more information to the LLM, but also let in more text unrelated to the question.
- Document Intelligence: introduced Azure AI Document Intelligence, a service that reads text and table structure from documents such as PDFs.
- Newer models: switched the embedding model and the LLM to newer versions.
Delivery and Outcome
Managed tickets and branches in Azure DevOps. Proposed and implemented CI/CD so that changes are deployed after passing tests.
The system was used firm-wide and won the FY2024 Corporate Award.
Technologies
Azure AI SearchAzure OpenAI ServiceAzure DevOpsCI/CDRAGAzure AI Document IntelligenceHybrid Search