Clinical AI faces a fundamental problem: patient data spans multiple modalities-PDFs, radiology images, lab results, pathology slides, clinical notes-yet most AI systems process each in isolation. Context is lost between encounters. Hallucinations emerge when reasoning across complex medical ontologies. The patient's complete clinical picture remains fragmented across disconnected vector stores and stateless LLM calls.
G.A.M.E.R.S (Graph Agents with Multimodal Entities and Reasoning Schemas) solves this through graph-based context memory.
In this session, you will witness graph agents that maintain persistent context across conversations, patient encounters, and multimodal inputs. The speakers will demonstrate how open-source LLMs, VLMs, hybrid RAG pipelines, and multimodal embeddings-extracted from text, images, and video are fed into a graph-based memory layer where relationships between clinical entities are preserved, queried, and reasoned over, while preserving privacy. This is built upon FHIR in the WHOLE framework ( presented at NODES 2025 )
The session will cover:
1. Graph-Based Context Memory - How agents store, retrieve, and reason over conversation history, clinical entities, and cross-session context using graph structures instead of flat vector stores
2 .Multimodal Entity Extraction - Processing clinical PDFs, radiology images, pathology slides, and unstructured text into interconnected graph nodes with typed relationships
3. Reasoning Schemas - Defining traversal patterns that mirror clinical thinking: disease hierarchies, comorbidity chains, temporal progressions, and causal relationships
4. Hybrid Retrieval Architecture - Combining dense embeddings, sparse search (BM25), and graph traversal for retrieval that understands both semantic similarity and structural relationships
5. Agent Memory Patterns - Implementing short-term (conversation), working (session), and long-term (persistent) memory layers within a unified graph structure

With all these, imagine a graph agent that doesn't just retrieve relevant chunks-but traverses patient biomarkers, radiology findings, histopathology observations, MRI interpretations, and ultrasound reports as connected entities. Context flows through relationships. Memory persists across sessions. Reasoning follows the graph.
You will walk away with architectural patterns, live examples, and graph schemas for building agents that see the complete patient, remember every encounter, and reason across modalities-all using open-source models in a completely decentralised private compute ecosystem.

Co-Presented with Dr. Julia Hitzbleck [ CEO, TrialBridge, Founder HI10x ]