Selected work · Case studyAI-assisted test design and analysis system
Role
Initiative lead (product-owner capacity)
Context
150-person software services firm
Sector
Quality-critical software
Stage
PoC → client-data pilot
45–50%
less test-case preparation time
Documented · internal pilot
30–35%
less total testing time (measured scenario)
Documented · internal pilot
70–80%
usable-output rate
Internal evaluation
4
person cross-functional team
Project fact
01 · Executive summary
A hidden process cost, made visible and materially reduced
A documentation-heavy testing process required testers to research Jira tickets, requirements, system specifications, architecture material and controlled templates before producing a single reviewed test case. In the target healthcare environment, preparing one complex test case could require roughly 4–12 hours or more. A substantial part of that effort was hidden process cost created by fragmented knowledge, manual research and documentation overhead.
I initiated and led a four-person cross-functional team to build an AI-assisted test design and analysis system. It used Jira, a RAG knowledge base, the OpenAI API, a Python backend, PostgreSQL with vector-search capability and Xray integration to generate test cases in the required format, return related requirements and source references, and surface potential contradictions, open questions and assumptions for human review.
A timed parallel comparison on a real internal company application — the same test scope prepared manually and AI-assisted — documented a 45–50% reduction in test-case preparation time and a 30–35% reduction in total testing time for the measured scenario. A later healthcare client-data pilot and an AWS refactoring generated strong organisational interest and repeated prospective-client demand for demonstrations.
03 · Business problem
Complexity had accumulated where nobody could see it
The objective was never to remove human review. It was to reduce research and documentation effort while preserving traceability, controlled wording and reviewer accountability.
Knowledge was scattered
Requirements and specifications were distributed across Jira, PRD, SRS, architecture and UI documentation.
Strict controlled output
Test cases had to follow strict templates, language rules and review expectations.
Tacit knowledge
Testers relied on product knowledge that did not exist in any formal documentation.
Cost visible, cause invisible
Management could see testing cost, but not the process complexity creating it. One complex case: 4–12 hours.
04 · BPMN 2.0 · Business process
From a Jira ticket to an execution-ready test case
A single process pool with three lanes representing the tester, the AI test design system, and review and approval. Jira, requirements and architecture appear as data objects the AI reads from, not as actors.
Swipe or scroll to explore the diagram →
05 · Solution architecture summary
Generation and approval, deliberately kept apart
A locally hosted, browser-based application backed by Python services. The tester picked a Jira issue and test type; the system retrieved the ticket via the Jira API, searched a locally hosted knowledge base, assembled a constrained few-shot prompt, and called the OpenAI API. The result was reviewed by a tester and could be pushed into Jira / Xray.
The AI could propose and analyse — but the tester and peer reviewer retained responsibility for the final content.
Knowledge preparation
- 01
Text extraction from PRD, SRS, architecture & UI docs
- 02
Fixed-size chunking → structure-aware section chunking
- 03
Metadata tagging & source references
- 04
Full re-embedding when documentation changed
- 05
Tables & diagrams represented as text
06 · Solution architecture
How the pieces fit together
Four layers — the tester's browser client, the Python application services, the retrieval & generation core, and the Jira / Xray system of record — with an offline knowledge-preparation pipeline feeding the vector store.
Swipe or scroll to explore the diagram →
07 · AI components
What the AI actually did — and where it stopped
Generation and retrieval were paired with deliberate guardrails: a constrained prompt, exposed assumptions, and potential contradiction and ambiguity detection — then handed to human gates. The model produced not only the draft, but related requirements, source references, assumptions and open questions.
01 · Grounding & retrieval
- OpenAI embeddings — vector representation for semantic retrieval
- RAG retrieval — grounded on documented knowledge only
- Sources: PRD, SRS, architecture material, UI-navigation knowledge
02 · Generation
- Few-shot prompt — examples, output template
- Terminology restrictions and prohibited words
- OpenAI API — generation & analysis
03 · Guardrails & human control
- Assumptions surfaced — unsupported inferences made explicit
- Potential contradiction & ambiguity detection — flags conflicting, unclear or unsupported information
- Human & peer review — reviewer keeps accountability for the final content
08 · Technology stack
The build
Language & app
Python
backend services & orchestration
Local web app
browser UI, locally hosted
AI & retrieval
OpenAI API
test-case generation
OpenAI embeddings
RAG semantic retrieval
Data
PostgreSQL
with vector-search capability
Preprocess + chunk
fixed → section-aware
Metadata + refs
source traceability
Integration
Jira REST API
ticket & issue retrieval
Jira Xray
test-case system of record
09 · Evaluation method
Measured by timed parallel comparison
The same test scope was prepared twice on a real internal company application — once manually, once AI-assisted, by testers of comparable experience — with start / finish timing and, in some cases, blinded quality evaluation. The strongest figures came from this internal pilot; a later healthcare pilot showed interest but produced no comparable timing record before handover.
01
Same test scope
one ticket, held constant
02
Manual & AI-assisted preparation
comparable experience levels
03
Timed comparison
start / finish per case
04
Independent quality assessment
blinded, in some cases
10 · Results and evidence
What the numbers show — and what they do not
All figures come from the internal pilot's timed parallel comparison. The measured scenario included test analysis, preparation, review, execution and result recording; it excluded clarification delays with external stakeholders, retesting and consolidated reporting.
Test-case preparation time
Documented timed parallel comparison in an internal pilot.
Measured testing scenario
Included analysis, preparation, review, execution and result recording; excluded clarification delays, retesting and consolidated reporting.
Usable generated output
Internal evaluation; correction was more efficient than restarting manually.
Broadly comparable quality
In blind comparison, AI-assisted vs. manual cases judged comparable — AI-assisted preferred in some.
Directional · blinded evaluator
Organisational interest
Shown to CTO-level client leadership; repeated prospective-client demonstrations.
Client-reported
Healthcare client-data pilot
Handed off for AWS refactoring. Final results are unknown.
Directional
11 · Limitations & evidence boundaries
What we are — and are not — claiming
- The 4–12-hour preparation figure describes complex healthcare test cases; it is not the pilot baseline.
- The 45–50% reduction came from the simpler internal pilot, not the healthcare environment.
- The 30–35% figure applies only to the measured testing scenario, with the exclusions noted above.
- The evaluation is a timed parallel comparison, not a conventional before-and-after study.
- The later healthcare-client pilot occurred, but final results are unknown.
- We do not claim full production deployment. We do not claim signed commercial revenue from this system.
- We describe the AI as flagging potential contradictions and ambiguities — not as deterministic contradiction detection.
- Human review and approval remained mandatory throughout.
12 · Lessons learned
What I would carry into the next build
Start with an expensive problem, not with AI
The strongest opportunities are attached to measurable friction people have normalised.
A proof of concept is not a product
Technical viability, deployment, support ownership and commercial readiness are separate questions.
AI needs visible uncertainty
Surfacing assumptions and contradictions beat pretending hallucinations could be eliminated.
Tacit knowledge must be externalised
The AI could only use what was documented — which drove a dedicated UI-navigation knowledge source.
Informal innovation needs formal ownership
A small team moves fast, but continuity is fragile without owned budget and succession.
13 · The Neurocroft method
The same six moves we apply to any organisational friction
This engagement predated Neurocroft as a business, but it followed the same underlying pattern we now use with every client.
Notice
Recognised the hidden documentation cost testers had normalised.
Understand
Mapped where tester time was actually consumed.
Simplify
Created a grounded, structured generation workflow.
Connect
Linked Jira, requirements, architecture and test management.
Amplify
Used AI to increase analysis capacity, not replace judgement.
Evolve
Measured the result and prepared the concept for wider deployment.
14 · Related services
Where this pattern fits in how we help
Primary service
Automation & AI Implementation
Automation and AI applied only where they create measurable value, with human governance kept intact.
Read the serviceAlso demonstrated
Data & Decision Intelligence
Connecting fragmented information sources so the decisions people actually make are backed by trusted, traceable data.
Read the service15 · Talk it throughHave a process cost you can feel but can't see?
Start with a short fit call. 20–30 minutes. You describe the situation; we tell you honestly whether AI belongs anywhere near it.
Get in touch