In-context learning (ICL) enables LLMs to adapt to new tasks by including a few examples directly in the input without updating their parameters. However, selecting appropriate in-context examples (ICEs) is critical, especially for functions like math and logic that require multi-step reasoning. Traditional text-based embeddings often prioritize shallow semantic similarities, which may not align with the deeper reasoning structures necessary for such tasks. Recent research suggests that graph-based representations mirror human cognitive processes and can better model multi-step reasoning and improve ICE selection by capturing transferable thought patterns.
Existing techniques for selecting ICEs fall into two categories: training-free and training-based. Training-free methods typically use heuristic criteria like similarity, diversity, or complexity or rely on feedback from LLMs, such as probability distributions or model outputs, to guide selection. While these approaches are computationally efficient, they often need to perform better compared to training-based methods. Training-based approaches focus on selecting individual or group examples but are resource-intensive.
A team of researchers from Southeast University, Beijing Institute of Mathematical Sciences, Yale, and UC San Diego introduced GraphIC, a graph-based ICE retrieval method. GraphIC uses graph representations and Bayesian Networks (BNs) to capture reasoning processes and select ICEs, filtering irrelevant semantics while preserving core reasoning. It mirrors human cognition by modeling thought dependencies. GraphIC’s retrieval system aligns examples with the reasoning structure of a query, even if they’re not semantically similar. Experiments on tasks like math reasoning and code generation show GraphIC surpasses both training-free and training-based models in effectiveness and efficiency.
The proposed GraphIC model uses graph-based representations to enhance example selection for reasoning tasks. It introduces “thought graphs,” which represent reasoning steps as nodes, and employs a probabilistic model based on BNs to capture dependencies between thoughts. The retrieval system selects examples that maximize the probability density of reasoning processes. A personalized PageRank mechanism refines the thought graph, simulating how humans revisit earlier steps when solving problems. Through bilinear form optimization, GraphIC efficiently selects examples with the highest potential for solving multi-step reasoning tasks, outperforming traditional graph similarity-based methods.
The GraphIC model is evaluated on four reasoning benchmarks: GSM8K and AQUA (mathematical reasoning), MBPP (code generation), and ProofWriter (logical reasoning). Using GPT-4o-mini and Llama-3.1-8B-Instruct, GraphIC outperforms training-free and training-based retrieval baselines, with an average 2.57% and 4.29% gain respectively. It excels in complex reasoning tasks, particularly in mathematical and logical datasets like GSM8K and AQUA. Ablation studies highlight the importance of thought graphs, Personalized PageRank (PPR), and BN-based retrieval in improving performance. GraphIC consistently shows robust performance improvements across all datasets as the number of ICE examples increases.
In conclusion, GraphIC is a graph-based method for ICE retrieval designed to improve LLMs on multi-step reasoning tasks. By representing reasoning as “thought graphs” and employing BNs and personalized PageRank, GraphIC selects ICEs that align with cognitive reasoning structures. It surpasses text-based embedding methods, which need help with complex reasoning tasks. Experimental results across mathematical, logical, and code generation functions show GraphIC consistently outperforms both training-free and training-based models. Although its training-free framework has limitations in capturing intricate thought patterns, it offers a way to represent and enhance LLM reasoning processes.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 50k+ ML SubReddit
Interested in promoting your company, product, service, or event to over 1 Million AI developers and researchers? Let’s collaborate!
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
