Today, organizations are increasingly using LLM to allow users to query data stored within the enterprise domain using natural language. There are a number of advantages to doing so, but a number of security risks are involved. These include prompt injection, attempts to jailbreak the application, unintended access to database schema, and SQL injection. These risks cannot be addressed by conventional means and thus emphasize the need to use a Zero Trust approach, wherein all data is validated prior to accepting it.

This session will demonstrate a proven approach to securing a natural language to SQL application.

Architecture Overview

The solution will use a multi-layered approach, wherein each layer will have a specific role to play. The layers are as follows:

UI Layer (React):
This is the frontend of the system. The system is designed to use natural language to accept the query entered by the user. The query is then forwarded to the backend services.

API Layer (.NET Core):
This is the second layer of the system. The API layer handles the security of the system. It is responsible for authenticating and authorizing users. Only users who are authorized can access the system through this layer.

LLM Function App (Python 3.10 – Azure Functions):
This is the main component of the system. The component interacts with the LLM. The LLM is responsible for generating SQL queries. The security validations of the system are performed in this layer. The system then runs the query on the database and presents the output to the users.

Security Design (Zero Trust in Practice):
Instead of having a single point of security check, the system has security checks at multiple points. The security checks of the system occur at the following points:

1. Input Validation (Pre-LLM):
Before sending the query to the LLM, the system validates the user input. The system utilizes the following tools to validate the user input:

* Azure AI Content Safety Prompt Shields
* Custom Python Validation Logic

2. Controlled SQL Generation
The system generates the SQL queries using LangChain’s SQldBToolkit, which is integrated with the ChatGPT model. However, the system does not execute the queries directly.

Instead, the system performs the following operations on the query:

* Query Rewriting
* Allow-list validation of tables, columns, and operations

3. Secure Query Execution
The system only allows the execution of validated queries on the SQL database. The system does not execute the query directly from the LLM.

4. Output Validation (Post-Execution)
After executing the query, the system scans the output set for sensitive data, including:

* Personally identifiable information
* Salary or financial data
* Internal schema data

If the system detects any sensitive data, it blocks the output.

5. Structured Output Parsing
The system utilizes a custom output parser, which parses the output into the following formats:

* JSON
* CSV
* HTML

This ensures the system does not expose the output of the database.

6. Logging and Monitoring
The system utilizes tools like Azure Monitor or Logspace to log all the actions, including security-related actions. The system logs the following data:

* Blocked Queries
* Detected Threats
* Policy Violations

Key Takeaway
If we connect the LLM model to the database directly without proper security measures, it is dangerous. To make it safe, we need to ensure that the model is validated at every stage of input, processing, execution, and output.
This architecture shows how to safely allow natural language access to enterprise data and how to control security and data exposure.