Abstract Body

Background

Comparable measures of provider quality are needed to improve HIV-related service delivery; however, many tools overlook the relational aspects of care. We investigated whether large language models (LLMs) can be used to extract valid and scalable measures of quality from standardized-patient (SP) transcripts in Tanzania.

Methods

We analyzed English translations of 139 audio-recorded SP encounters with shopkeepers conducted from a cluster-randomized trial of a youth-friendly loyalty program (“Malkia Klabu”) in drug shops. Two SP profiles (younger woman, unmarried; older woman, married) requested contraception counseling, HIV self-testing, and a price discount on antifungal medicine. A GPT-4.1 agent, constrained by a structured instruction set, read all transcripts to: (1) answer 12 validation questions spanning protocol comprehension, cross-transcript aggregation, and single-transcript retrieval; (2) inductively propose themes of provider quality-of-care; (3) generate a 17-item instrument; and (4) score each encounter (1-85) and rank them. We compared agent outputs with manual review for selected transcripts.

Results

The agent accurately reproduced the SP protocol and actor profiles, correctly counted 139 encounters in 88 shops, and identified transcripts with contraceptive counseling (n=63). Single-transcript accuracy varied (65% for who introduced Malkia Klabu; 50% for interruption detection; 67% correct transcript citation). The LLM-derived index ranged 4–83 (mean 55.3, SD 15.6). Score distribution (n=139): 0–10, 1 (0.7%); 11–20, 0 (0.0%); 21–30, 4 (2.9%); 31–40, 29 (21.0%); 41–50, 12 (8.7%); 51–60, 35 (25.4%); 61–70, 31 (22.5%); 71–80, 24 (17.4%); 81–85, 2 (1.4%). At the sub-index level, 45.7% of encounters met ≥5 criteria, 21.7% met ≥6, and 10.9% met ≥7. Manual spot-checks preserved rank-order across quartiles, with one second-quartile discrepancy due to yes/no scaling. Strengths: parallel review and theme generation; weaknesses: limited subtext reasoning and occasional cross-document errors.

Conclusions

An LLM can transform SP transcripts into quantitative, face-valid measures of provider quality relevant to youth-friendly HIV prevention and contraception services in drug shops in Tanzania. This approach could enable routine quality surveillance and feedback at scale. Human validation, construct/prompt refinements, and integration of audio features are needed to improve detection of bias/subtext and scoring calibration, alongside safeguards for privacy and data security.

Table or Figure