Case study
Faultline
A real production-readiness review of a live multi-tenant RAG app: a critical cross-tenant leak, six ranked findings, and a 90-day plan, the literal deliverable of the Architecture & AI Review offer.
Role: Architecture & AI Review, self-initiated
Stack: Automated and manual code review, zero live compute; report in TypeScript, React, Vite, Tailwind
Why it exists
The problem this was built to solve, and the approach behind it.
Most AI applications ship with no injection defense, no spend ceiling, no eval set, and no defined failure behavior, and there is usually no way to know that from the outside.
A real review of one live, permissively licensed AI application, performed as a genuine demonstration of the Architecture and AI Review deliverable: every finding read in the real source and grounded in it, not padded with generic best practices.
What it does
- A full production-readiness pass across security, cost, scale, and eval quality, on one real public AI application
- Six findings, ranked by severity, each read in the real source and stated precisely rather than as a general concern
- A critical finding: debug endpoints bypassing the app's own access control and leaking cross-tenant data
- Cost and scale reasoned from the code's own call sites and constants, no load testing or live probing
- A prioritized 90-day plan across three horizons: stop the bleeding, structural fixes, maturity work
- An honest scope section naming exactly what was not reviewed
In the product
Swipe or click through a few screens from the working app.
Finding F-01 in Faultline's report: debug endpoints bypass the app's own access control and leak cross-tenant data, with what was found, why it matters, the fix, and the effort to fix it
Architecture decisions
The calls that shaped the build, written the same way the product itself records them.
Automated scanning to be complete, manual reading to be right
The review paired automated scanning across the whole codebase, every API route, every model call site, the auth middleware, the test suite, with close manual reading of the handlers that actually mattered. Automation is what makes a review of an unfamiliar system complete rather than a spot check, and fast enough to be worth doing; judgment is what turns a search hit into a finding worth a client's Monday. Neither half is the deliverable on its own.
No paid compute, no live probing, not a cent spent
Cost and scale findings are reasoned from the code's own call sites and hardcoded constants, not measured by running the target's live service. The app was not load-tested, and no LLM calls were run against the maintainer's own keys or bill. Reading code is free; running up someone else's bill to prove a point is not a demonstration, it is an imposition.
Credit the real strengths before the findings
The review opens with what the target got right, and means it: real multi-tenant authentication, role-based access control correctly implemented where it is used, provider abstraction, usage tracking, an audit log. A review that only attacks reads as a junior showing off. The findings are more useful for being specific about where a genuinely capable system's controls stop being applied, not for pretending the whole thing is weak.
Every finding is cut if it cannot be pointed at in the code
No 'it probably does not scale.' Each finding was read in the real source and states a concrete consequence, not a hypothetical one. Ten grounded findings beat thirty generic ones, and generic findings are exactly what exposes a review as shallow.
Want a review like this on your own system?
A focused 30-minute call to scope an Architecture & AI Review and see if it is a fit. No pitch.