SQL Structure Embedding Berbasis Transformer untuk Perankingan Few-Shot Example pada Sistem Text-To-SQL

Abstract

This research develops and evaluates a Transformer encoder-based SQL Structure Embedding (SSE) to improve SQL structure representation quality in Text-to-SQL systems. Static embedding approaches such as GloVe, commonly used in few-shot learning-based Large Language Model (LLM) systems, have limited ability to capture inter-clause contextual relationships in SQL, thereby reducing the quality of few-shot example selection. The dataset used is the BIRD dev set with 1,534 Natural Language Query (NLQ) and SQL query pairs, annotated with five SQL latent intent categories and 44 SQL component labels. The SSE model is built using an encoder-only Transformer architecture initialized with 100-dimensional pretrained GloVe weights, two encoder layers with 4 attention heads, and mean pooling. Evaluation results show the SSE model achieves a Micro F1 of 0.8671, Macro F1 of 0.7920, and Samples F1 of 0.8312, improving by 9.17%, 10.31%, and 9.20% respectively over the GloVe baseline. The combination of SSE with a downstream classifier yields the best performance with a Micro F1 of 0.8821. These results confirm that the self-attention mechanism in Transformer more effectively captures relational context between SQL clauses to support few-shot example ranking in Text-to-SQL systems.