In this first milestone we will write a generator for logic grid puzzles which assembles them into standardized descriptions and corresponding PLN Atomspaces. Logic grid puzzles are a simple to understand problem domain are easy to vary in complexity and can be solved using PLNs. In order to baseline the PLN part of the guidance process we will create a dataset of logic grid puzzles and check that the associated PLNs are able to solve them.

Software to generate the logic grid puzzles and turn them into PLN Atomspaces. A dataset of such puzzles and associated Atomspaces will be made available for others to work with. Experimental results showing that PLNs can solve such puzzles and that the reasoning steps can be adequately captured.

$8,000 USD

1. The program is able to generate logic grid puzzles 2. The program is able to convert them into MeTTa Atomspaces

In this milestone we will feed the logic grid puzzles from the dataset produced in Milestone 1 to graphRAG. We will then use the original problem statement of the puzzle to prompt a Llama instance augmented by graphRAG. We are interested in determining: - at what complexity was the retrieval-augmented LLM able to solve the puzzle? - did the LLM combined with the graphRAG instance generate any reasoning steps in reaching its answer? The behavior of the LLM during these tests will form the baseline for our forthcoming prompt engineering work. During this experimental phase we will settle on a set of metrics which can be objectively recovered from the query-reponse sequences. These will include but are not limited to; solution accuracy solution efficiency elucidation of optimal reasoning steps and propensity to make inappropriate steps.

The raw graph store of graphRAG the raw query and output data from the LLMs (including the retrieved data from graphRAG) and a summary of our findings and insights regarding which presentation approaches yield best results.

$8,000 USD

1. We can feed the raw text of the logic grid puzzles into graphRAG 2. We can augment queries to the LLM with data from graphRAG 3. We are able to establish a baseline to compare all other experiments with

We will use the PLNs to generate a line of reasoning and develop an automatic process for engineering Chain-of-Thought prompts for the LLM. We will experiment with the following methods for giving the LLM information from the PLN: - use the output of the PLN directly as input to the LLM - transliterate the output of the PLN into a prompt using a set of transformation rules The LLM is prompted at each stage by the requisite step in the reasoning process. In this set of tasks we are trying to discover the optimum lingua franca for communicating PLN reasoning to LLMs.

The raw query and output data from the LLMs and a summary of our findings and insights regarding which presentation approaches yield best results.

$19,000 USD

The LLM is able to generate meaningful responses based on the query augmented with data from the PLN

In this milestone we will use the PLN to generate examples of reasoning using the logic-grid Atomspaces. These exemplars are fed into the LLM as examples of query/answer pairs and the context is used to allow the LLM to perform few-shot learning on the domain. The LLM is then presented with the test puzzle and its own line of reasoning is examined.

The raw query and output data from the LLMs and a summary of our findings and insights regarding which presentation approaches yield best results.

$18,000 USD

1. We are able to generate solid exemplars: some that are simple, some that are tricky, and some that yield surprising results from the LLM (i.e., it struggles with an easy prompt or succeeds with a difficult one) 2. We are able to determine how well the LLM was able to explain its own reasoning (Will be discussed at length in the paper in milestone 6)

We make a hybrid of the exemplar and chain-of-thought approaches by giving the LLM a set of exemplars for each step in the chain-of-thought. We are here determining a good balance between the few-shot learning from exemplars and the step by step prompting from chain-of-thought. Our aim is to provide an interface definition for enabling PLNs to provide guidance to LLMs.

The raw query and output data from the LLMs and a summary of our findings and insights regarding which presentation approaches yield best results.

$19,000 USD

We are able to determine which approach yielded the best results

We examine the standard benchmarks for LLM reasoning tests and determine which (if any) correspond to existing (or easy to develop) Atomspaces for PLNs. Once we have settled on a benchmark we will evaluate the CoT and exemplar approaches to LLM guidance. We will collate all of our findings and assemble them into a research paper for dissemination to the SingularityNET community. We will further explore the potential research directions for CoT exemplar and hybrid prompt engineering for PLN guidance to LLMs and perform a preliminary survey of how PLNs and LLMs might be more tightly integrated. Furthermore we will recommend ways in which PLNs could be integrated into systems beyond LLMs. We expect this final task to be predominantly paper-based research rather than experimental.

Benchmark dataset domain Atomspace and research paper.

$8,000 USD

We are able to analyze our results and synthesize meaningful conclusions that help lay the groundwork for future AGI research