Education Company: 24/7 AI Tutor PoC Development
Commissioned Work | Forward Deployed Engineer
Context and Problem
A PoC for an AI tutor that answers students' photographed questions around the clock. Answers had to follow the cram school's teaching policy (lesson plans and other PDFs).
Designed and built the PoC single-handedly (every commit in the repository is mine or my AI coding agent's).
How It Works
A 17-node LangGraph state graph. One question flows through these steps:
- Reading and topic unit: the photo is read, and the Jev classifier decides which unit the question belongs to.
- Teaching-policy selection: the policy sections whose tags match that unit are selected; this step is deterministic and calls no LLM.
- Solving: the writer model drafts a skeleton (steps, equations, one final answer per sub-question), and the skeleton's consistency is checked.
- Cross-check: a checker model from a different vendor reads the photo directly and returns only final answers. If the writer and checker come from the same vendor, start-up fails with an error.
- Gate: the tutor answers only when the two vendors agree, the writer's arithmetic recomputes, the skeleton check passes, and the photo shows the whole question. Otherwise it asks once more; if that still fails, it does not answer and suggests asking a teacher.
- Display: the answer and equations on screen are copied by code from the verified skeleton. The explanation model writes prose only, so no unverified number reaches the screen.
- Hard-question lane: a stronger solver runs on a separate thread for every question and must agree with a third vendor's model; it never slows the fast pair.
How It Was Verified
- Measured: 0 wrong answers shown in 82 test questions with reference answers (questions held out from the received material plus questions I wrote); the Wilson 95% confidence interval for 0/82 is 0 to 4.5%. This was a provisional-criteria check of the framework before tuning; 5 questions whose value matched but whose notation differed could not be graded automatically and were flagged for human review.
- Tests (counted): 148 test files with 3,953 test functions (a static count of the repository, not a run result), including a test that pins what each side of the cross-check is shown. It was added because existing tests barely caught two mistakes imagined in a review (showing the checker the reading, showing the consistency check the checker's answers).
Handling the Teaching Policy
- OKF: wrote the teaching policy one section per file in Google Cloud's Open Knowledge Format (OKF).
- Failure handling: the Jev call never raises. It waits up to 0.6 seconds by default and otherwise falls back to the broader category's policy, so a failure only makes the guidance less specific and never loses an answer.
- Independence: the policy goes to the writer and the explanation, never to the checker or the consistency check, so the two sides being compared do not share a starting point.
Technical Decisions and Rejected Alternatives
- No RAG: for low latency, RAG (vector search over the policy) was ruled out of scope, and so was letting the model read files through tool calls; Python reads the bundle. The same material and unit always yield the same sections, so runs are comparable.
- Difficulty prediction rejected: tried routing questions predicted to be hard to a stronger model. No predictor met the adoption bar set in advance, and the "hard" labels were unstable from run to run, so the hard-question lane runs on every question instead.
- Classification confidence: replaced confidence tiers with using every tag above a score threshold, because a score split between two units fell back to the broad category.
- Removed the rule that passes a question if either of two checker draws agrees: redrawing a checker can only turn a refusal into a pass.
Problems Found, and Fixes
- Two models can agree on a cropped photo → completeness is checked directly, and a dedicated result asks the student to retake the photo.
- A page with several problems → the run stops and the operator selects the problem box.
- A slow figure node held up the whole explanation → figures are built on a separate thread and collected after the answer is assembled.
- At low reasoning effort the checker answered with almost no reasoning and produced wrong answers → its reasoning effort was raised to medium.
Technologies
Python 3.12uvLangGraphFastAPIPydanticJevOpen Knowledge Format (OKF)OpenRouterMulti-vendor LLMpytest