Evaluation of multimodal large language models as image classifiers to detect growth on agar media

Background Traditional computer vision models require training on thousands of reference images to achieve acceptable accuracies regarding the detection of microbial growth on agar plates. Recent multi-modal large language models (LLMs) accept binary (including images) in addition to textual data and could potentially be used as image classifiers without additional training. The current project examines the validity of this assumption.

Methods Images of agar plates processed in a clinical microbiology laboratory were acquired using a hand-held camera (iPhone 14). To adjust for differences in angle and distance, images were segmented, de-skewed, and cropped using custom-made software “arucutter”, version 0.1.3 [1]. Media used to screen for vancomycin-resistant Enterococcus spp. (Chromid® VRE), cephalosporin-resistant Enterobacterales (Chromid® ESBL, bioMérieux), and Staphylococcus aureus (BBL™ CHROMagar™ MRSAII/SA bi-plates, BD) were included. Processed images and media-specific prompts were forwarded to large language models (LLMs) Gemma 3 (4b), Gemma 4 (e4b), Qwen 3.5 (4b), and Qwen 3-VL (8b) served over “Ollama” in a “zero-shot” fashion using custom library “oneshot”, version 0.1.5 [2].

Results The sample consisted of 109 bi-plates, 96 ESBL plates, and 47 VRE plates, which were categorized by a human reader. Proportions of media without any growth (labelled “negative”) were 55.0%, 30.2%, and 59.6%, respectively. Accuracies of LLMs ranged from 47.7% to 96.9% and were lowest regarding the correct classification of both halves of bi-plates. Nevertheless, the separate classification of bi-plate halves was higher and particularly pronounced for MRSA, where accuracies ranged from 69.7% (Gemma 3) to 95.4% (Gemma 4). The highest accuracies of 96.9% were recorded in ESBL-plates both by models Gemma 4 and Qwen 3.5. Further results will be presented at the conference.

Conclusions LLMs called in a zero-shot fashion perform surprisingly well regarding the classification of agar media into “positive” (i.e. bacterial growth present) and “negative” (i.e. bacterial growth absent). Bi-plates are visually more challenging than entire plates (e.g. ESBL or VRE agar) but can nevertheless be queried in a meaningful fashion without resorting to additional training or fine-tuning.

References 1. https://github.com/joheli/arucutter 2. https://github.com/joheli/oneshot