A Multimodal Swarm-Based Multi-Agent Retrieval-Augmented Generation (RAG) For Domain-Specific Applications

Abstract

The increasing complexity of medical data, encompassing text, images, and documents, demands advanced systems capable of integrating and analyzing multimodal inputs to support clinical decision-making. This study presents a novel multimodal AI system designed for medical information retrieval and decision support, utilizing swarm-based agents and retrieval-augmented generation (RAG) to process diverse data types concurrently. The system employs collective intelligence-inspired swarm agents to distribute tasks efficiently and RAG to enhance contextual relevance. It achieved strong semantic understanding on dental health queries with BERTScore F1 values of 0.8338 in query with image upload scenario, though it exhibited challenges in text-image alignment with MM-Align score of 0.3103 and factual consistency of 0.4737. In query-only scenario, dental-related queries have high BERTScore F1 values of 0.9548 and mostly consistent facts of 0.950. While for non-dental queries, the system leveraged web search and give current information as result. The comparative analysis results highlight its adaptability across medical domains, despite limitations in aligning multimodal inputs and ensuring complete factual accuracy. These findings underscore the potential of swarm-based multimodal AI to improve healthcare outcomes by providing contextually relevant analyses while identifying key areas for further refinement.