LLM 기반 RAG 시스템 성능 평가 및 실증 분석:리크루팅 상담서비스를 중심으로

A Study on Performance Evaluation and Empirical Analysis of LLM-based RAG Systems: Focusing on Recruitment Counseling Services

This study comprehensively evaluated and empirically analyzed the performance of Retrieval-Augmented Generation (RAG) systems based on Large Language Models (LLMs), with a specific focus on recruitment counseling services. Specifically, it systematically compared the performance of LLaMA 3.1 70B and GPT-4o models using the comprehensive RAGAS evaluation framework. The research methodology involved conducting extensive experiments based on various employment-related data sources, including job application documents, vocational psychological tests, detailed job care reports, vocational training courses, and National Tomorrow Learning Card operating regulations. Performance evaluation utilized five key indicators: Context Precision, Faithfulness, Answer Similarity, Context Recall, and Answer Relevancy. The experimental results showed that both models demonstrated excellent performance in Context Precision (0.97), but LLaMA 3.1 significantly outperformed GPT-4o in terms of Faithfulness metrics. Conversely, GPT-4o showed superior performance in Answer Similarity and Answer Relevancy compared to LLaMA 3.1. Data type-specific analysis revealed that Context Precision was perfect (1.0) for both standard text and meta data, but notable differences existed in other performance indicators. This study emphasizes the critical importance of model selection in designing LLM-based RAG systems and suggests appropriate model selection according to specific service objectives. The research findings can contribute to improving the quality of AI-based recruitment counseling services and serve as valuable reference material for designing and implementing RAG systems across various industries in the future.