Central Government Ministry: Research on the Structural Similarity of Government Programs
European Consulting Firm
Context and Problem
Handled verification in a research study that uses AI to find government programs related to a given policy theme.
The previous year's approach had left practical issues, so this study examined whether generative AI and LLMs could be used.
From Prototype to Retrieval Method
First prototyped RAG by modifying an Azure public sample (Azure-Samples/azure-search-openai-demo): added processing that reads a table of program descriptions as text and splits it by sentence for the index, and tuned search for Japanese (the ja.microsoft analyzer, ja-JP query language, spell correction turned off, and a prompt that rewrites questions into Japanese).
The prototype could return only a few programs per answer, and it sometimes output programs that do not exist. In general, an LLM can handle only a limited amount of text at once, and it can write content that is not in the retrieved documents.
So the search was switched: devised and implemented a retrieval method that embeds the detailed program descriptions and ranks programs by vector similarity to the policy theme. The LLM was used only to write, for each program, why it resembles the policy theme.
Embedding turns text into a vector of numbers, so that texts with closer meanings get closer values.
What I Designed and Built
- Analysis environment: built a verification environment on Azure that used JupyterLab, and made communication with the environment work over https.
- Security: worked out the security requirements from the procurement's reference documents and implemented them; the measures include user management, encrypted transport, and restricting inbound access.
How It Was Evaluated
Metric: the retrieval was evaluated by calculating recall and precision over the top 100 results by similarity, against the programs the client labeled as matches.
Recall is the share of the correct programs that appear in the results, so it shows how few are missed. Precision is the share of the listed programs that are correct, so it shows how little noise there is.
As a policy, the figures are not published.