Deteksi Ill-Formed Natural Language Query Berbasis Rule Pada Model Llm Text-to-SQL Berbahasa Indonesia

Abstract

The performance of a Large Language Model (LLM)-based Text-to-SQL system is heavily influenced by the quality of the user-supplied Natural Language Queries (NLQs). Ill-formed NLQs, such as incomplete, ambiguous, or inconsistent information within the database schema, can generate syntactically valid but semantically incorrect SQL queries. This study aims to develop a rule-based ill-formed NLQ detection system as a pre-processing stage in an Indonesian Text-to-SQL pipeline. The system is built using the Experta library with 15 IF-THEN rules that identify six categories of ill-formed NLQs: insufficient information, multiple interpretations, incomplete conditions, illogical, out-of-schema reference, and unsafe/non-SELECT. After rule-only filtering, a LLaMA 3.2 3B Instruct-based validator is optionally used to examine cases requiring deeper semantic understanding. An evaluation of 202 Indonesian NLQs showed that the Rule-only mode achieved 89.60% Recall, 78.87% Precision, 83.90% F1-Score, and 78.71% Accuracy. False positives were primarily influenced by overly strict rules, while false negatives were dominated by semantic ambiguity. These results demonstrate that the rule-based approach provides a consistent and transparent initial filtering mechanism and can be combined with an LLM-based validator to handle cases requiring more complex semantic interpretation.