Not everyone needs to send data to the public cloud to benefit from large language models. In this session, I demonstrate how to embed private LLM inference directly into existing Java-based systems using Ollama and Gemma3.

We will walk through an offline demo that uses a JSON “sales” dataset, generates prompts in Java, and returns results locally no internet connection and no data leaving your environment. I will explain why Java remains a practical choice for mission-critical backends across Africa (and beyond) and how Gemma3’s architecture (including grouped-query attention and RMSNorm) balances speed with accuracy for real-time queries.

This approach is especially relevant for regulated industries that must keep sensitive data private such as healthcare, banking, insurance, and retail loyalty programs handling personally identifiable information. Many teams in these sectors face strict data residency rules, confidentiality requirements, and audit obligations. By running inference locally, organisations can preserve privacy, reduce exposure risk, and maintain control over logs and model inputs while still gaining AI-level capabilities.

Who it is for:

Enterprise developers, solution architects, and product leaders in regulated or privacy-sensitive environments who need AI capabilities on existing stacks and must keep data private.

What attendees will learn

-- How to run private LLM inference locally with Java, Ollama, and Gemma3.

-- A repeatable offline workflow: JSON data → Java prompts → local inference → outputs that can feed terminal, reports, or a user interface.

-- Where private inference fits today in regulated industries, and how to extend it with Retrieval-Augmented Generation (RAG) tomorrow.

-- Practical considerations for latency, reliability, auditability, AI ethics and data protection in real-world environments.

I have delivered this talk for GDG Harare's Build with AI Workshops and written an article that outlines what the workshop will include. You can access the article here: https://www.linkedin.com/pulse/private-llm-inference-enterprises-ruvimbo-delia-hakata-tvrjf