Benchmarking Embedding Models for POJK Regulatory Retrieval in Indonesian Insurance Documents
DOI:
https://doi.org/10.63158/journalisi.v8i5.1781Keywords:
regulatory retrieval, legal information retrieval, embedding benchmark, POJK, BGE-M3, BM25Abstract
Indonesian insurance regulations are clause-dense and terminologically specific, making manual retrieval of applicable provisions difficult. This study presents a retrieval-only benchmark comparing embedding and lexical methods on POJK 26/2025 and POJK 36/2025. The corpus contains 181 context chunks and 181 synthetic Indonesian queries, each assigned one primary relevant chunk. Queries were generated in JSON mode using Gemma 4 E4B via Ollama (ollama:gemma4:e4b), with malformed outputs retried and heuristic fallbacks prohibited. Evaluation comprised a four-model local dense comparison and a separate native BGE-M3/BM25 ablation using exact exhaustive ranking. The local comparison reports MRR and Hit rates, while the native ablation additionally reports NDCG@10, MAP@10, bootstrap confidence intervals, paired tests, and rank bins. Qwen3 Embedding achieved the highest local MRR (0.8075). Under a prespecified descriptive composite rule combining MRR, Hit rates, and measured latency, BGE-M3 ranked first. In the native ablation, BGE-M3 combined retrieval achieved NDCG@10 of 0.8810. A practitioner-validated subset audit and artifact-backed qualitative analysis supplemented the quantitative results. Findings are limited to this two-document benchmark and do not evaluate answer generation, citation correctness, or end-to-end RAG faithfulness.
Downloads
References
[1] Z. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 1–38, Dec. 2023, doi: 10.1145/3571730.
[2] P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” in Adv. Neural Inf. Process. Syst., H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., Curran Associates, Inc., 2020, pp. 9459–9474. doi: 10.48550/arXiv.2005.11401.
[3] H.-T. Nguyen, M.-K. Phi, X.-B. Ngo, V. Tran, L.-M. Nguyen, and M.-P. Tu, “Attentive deep neural networks for legal document retrieval,” Artif. Intell. Law, vol. 32, no. 1, pp. 57–86, Mar. 2024, doi: 10.1007/s10506-022-09341-8.
[4] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang, “Retrieval Augmented Language Model Pre-Training,” in Proc. 37th Int. Conf. Mach. Learn. (ICML), Proc. Mach. Learn. Res., vol. 119. PMLR, Jul. 2020, pp. 3929–3938. doi: 10.48550/arXiv.2002.08909.
[5] V. Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering,” in Proc. 2020 Conf. Empirical Methods Natural Lang. Process. (EMNLP), Online: Association for Computational Linguistics, 2020, pp. 6769–6781. doi: 10.18653/v1/2020.emnlp-main.550.
[6] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in Proc. 2019 Conf. Empirical Methods Natural Lang. Process. and 9th Int. Joint Conf. Natural Lang. Process. (EMNLP-IJCNLP), Hong Kong, China: Association for Computational Linguistics, 2019, pp. 3980–3990. doi: 10.18653/v1/D19-1410.
[7] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models,” in Proc. 35th Conf. Neural Inf. Process. Syst. Datasets Benchmarks Track (Round 2), 2021. doi: 10.48550/arXiv.2104.08663.
[8] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “MTEB: Massive Text Embedding Benchmark,” in Proc. 17th Conf. Eur. Chapter Assoc. Comput. Linguist. (EACL), 2023, pp. 2014–2037. doi: 10.18653/v1/2023.eacl-main.148.
[9] S. Es, J. James, L. Espinosa Anke, and S. Schockaert, “RAGAs: Automated Evaluation of Retrieval Augmented Generation,” in Proc. 18th Conf. Eur. Chapter Assoc. Comput. Linguist. Syst. Demonstrations (EACL), St. Julians, Malta: Association for Computational Linguistics, 2024, pp. 150–158. doi: 10.18653/v1/2024.eacl-demo.16.
[10] Y. Huang and J. X. Huang, “A Survey on Retrieval-Augmented Text Generation for Large Language Models,” ACM Comput. Surv., vol. 58, no. 12, pp. 1–38, Sep. 2026, doi: 10.1145/3805774.
[11] L. Wang et al., “Text Embeddings by Weakly-Supervised Contrastive Pre-training,” arXiv preprint arXiv:2212.03533, 2022, doi: 10.48550/arXiv.2212.03533.
[12] T. Formal, B. Piwowarski, and S. Clinchant, “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking,” in Proc. 44th Int. ACM SIGIR Conf. Res. Dev. Inf. Retrieval (SIGIR), Virtual Event Canada: ACM, Jul. 2021, pp. 2288–2292. doi: 10.1145/3404835.3463098.
[13] K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia, “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction,” in Proc. 2022 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. (NAACL), Seattle, United States: Association for Computational Linguistics, 2022, pp. 3715–3734. doi: 10.18653/v1/2022.naacl-main.272.
[14] J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation,” arXiv preprint arXiv:2402.03216, 2024, doi: 10.48550/arXiv.2402.03216.
[15] J. Alsubhi et al., “Optimizing RAG Pipelines for Arabic: A Systematic Analysis of Core Components,” arXiv preprint arXiv:2506.06339, 2025, doi: 10.48550/arXiv.2506.06339.
[16] V.-H. Le, D.-V. Nguyen, V. K. Nguyen, and N. L.-T. Nguyen, “Optimizing Legal Document Retrieval in Vietnamese with Semi-hard Negative Mining,” in Computational Collective Intelligence, Lect. Notes Comput. Sci., vol. 16138, Cham: Springer Nature Switzerland, 2026, pp. 269–283. doi: 10.1007/978-3-032-09318-9_19.
[17] M.-Y. Kim, J. Rabelo, K. Okeke, and R. Goebel, “Legal Information Retrieval and Entailment Based on BM25, Transformer and Semantic Thesaurus Methods,” Rev. Socionetwork Strateg., vol. 16, no. 1, pp. 157–174, Apr. 2022, doi: 10.1007/s12626-022-00103-1.
[18] E. M. Voorhees, D. Samarov, and I. Soboroff, “Using Replicates in Information Retrieval Evaluation,” ACM Trans. Inf. Syst., vol. 36, no. 2, pp. 1–21, Apr. 2018, doi: 10.1145/3086701.
[19] Y. Luan, J. Eisenstein, K. Toutanova, and M. Collins, “Sparse, Dense, and Attentional Representations for Text Retrieval,” Trans. Assoc. Comput. Linguist., vol. 9, pp. 329–345, Apr. 2021, doi: 10.1162/tacl_a_00369.
[20] S. Hofstätter, S.-C. Lin, J.-H. Yang, J. Lin, and A. Hanbury, “Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling,” in Proc. 44th Int. ACM SIGIR Conf. Res. Dev. Inf. Retrieval (SIGIR), Virtual Event Canada: ACM, Jul. 2021, pp. 113–122. doi: 10.1145/3404835.3462891.
[21] Y. Zhang et al., “Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models,” arXiv preprint arXiv:2506.05176, 2025, doi: 10.48550/arXiv.2506.05176.
[22] Z. Nussbaum and B. Duderstadt, “Training Sparse Mixture Of Experts Text Embedding Models,” arXiv preprint arXiv:2502.07972, 2025, doi: 10.48550/arXiv.2502.07972.
[23] A. Kusupati et al., “Matryoshka Representation Learning,” in Adv. Neural Inf. Process. Syst., vol. 35, 2022, pp. 30233–30249, doi: 10.48550/arXiv.2205.13147.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Author's Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
