Company Name (if applicable)

Project details

Probabilistic Soft Logic (PSL) is a highly scalable, declarative framework used for probabilistic reasoning over large sets of relational data. It provides a foundation for reasoning over continuous, soft-truth values between 0 and 1, representing varying degrees of certainty in real-world information. Unlike traditional logic systems where truth values are binary (true or false), PSL assigns a "soft" truth value to each logical proposition. This capability allows PSL to naturally handle uncertainty, incomplete data, and noisy inputs. PSL is particularly well-suited for complex relational domains like knowledge graphs, entity resolution, and link prediction tasks.

PSL’s strength lies in its ability to combine the efficiency of convex optimization with the flexibility of first-order logic. A PSL model is composed of weighted, first-order logic rules that describe the relationships between different entities. These weights represent the importance or confidence of each rule, and during inference, PSL uses these weights to compute the most probable truth values for unknown relationships. Importantly, the inference process is highly scalable due to PSL’s convex optimization formulation, enabling it to process millions of facts efficiently. This makes PSL an excellent choice for KG enrichment, as it can handle the scale and complexity of real-world datasets, providing an efficient way to infer missing information and correct inconsistencies.

In this proposal, we will use PSL as a probabilistic logic framework to enhance Atomspace knowledge graphs using batch-mode PLN. By probabilistically enriching these graphs, we aim to improve the memory and reasoning capabilities of large language models (LLMs). This approach will demonstrate the utility and efficieny of PSL in improving the retrieval accuracy of graph-based memory systems, particularly in comparison to existing methods like graphRAG.

Before describing the specific steps, and detailing corresponding milestones in the next section, we provide some details on why PSL is such an efficient framework, especially compared to alternative frameworks such as Markov Logic Networks (MLNs). MLNs which handle uncertainty with probabilistic rules over binary truth values, but they face significant computational challenges because inference in MLNs involves solving large combinatorial optimization problems. This often results in slow performance, especially on large datasets.

In contrast, PSL uses continuous-valued truth values rather than binary ones, allowing for more flexible reasoning under uncertainty. This continuous nature simplifies the optimization problem, transforming it into a convex optimization task rather than a combinatorial one, which makes it highly scalable and computationally efficient. PSL’s inference process, which relies on convex optimization, ensures that the system can efficiently handle millions of data points without the prohibitive complexity often encountered with MLNs.

Additionally, PSL allows for soft truth values to represent the varying confidence in each rule, making it particularly useful in real-world applications like knowledge graphs, where data can be noisy and incomplete. This flexibility, combined with the ability to define probabilistic dependencies between facts via first-order logic rules, makes PSL a perfect fit for enhancing LLM memory systems that rely on enriched, probabilistic knowledge graphs. Its scalability and efficiency allow for the processing of large, relational datasets, critical for systems like Atomspace that manage complex knowledge graphs.

The first step of the project involves setting up the architectural foundation for the integration of PSL into Atomspace within the OpenCog Hyperon framework. This stage will focus on designing the pipeline through which probabilistic reasoning will operate on Atomspace knowledge graphs. We will define how batch-mode processing of PLN will function within this architecture and how it interfaces with MeTTa for seamless integration into the larger system. The initial architecture will also outline how graphRAG comparison benchmarks will be set up to evaluate the system’s performance.

Key aspects include the definition of interfaces between PSL, Atomspace, and the MeTTa scripting language, ensuring compatibility with the OpenCog Hyperon framework. The architecture must meet functional requirements, such as enabling scalable batch processing, and non-functional requirements like moderate dataset handling and extensibility for future improvements.

Next, we will focus on the development and implementation of the batch-mode PLN system, applying PSL to enrich Atomspace knowledge graphs. Using the rules and methods from the PSL framework, we will probabilistically enrich knowledge graphs, creating a more informative structure. This enriched graph will include nodes with weighted relations that support memory retrieval tasks in LLMs. By focusing on batch mode rather than real-time inference, the system will operate in the background to continuously improve the graph structure.

The key challenge here is to ensure that the PSL-enriched knowledge graph can improve the retrieval accuracy of LLM memory while working within the constraints of Atomspace and the OpenCog Hyperon framework. The enriched graph will be benchmarked against traditional graphRAG systems, with the goal of demonstrating superior performance in retrieving relevant information.

In the third phase, we will test the PSL-enhanced batch-mode system to evaluate its performance across several key metrics. These tests will focus on reasoning accuracy, retrieval speed, and overall memory improvement in LLMs. The system’s probabilistic reasoning capabilities will be compared against traditional graphRAG systems, with a focus on improvements in retrieval and reasoning.

The tests will address both functional requirements, such as quantifiable improvements in LLM reasoning, and non-functional requirements, such as efficient processing of moderate-sized datasets. The results will be used to refine the system and guide further optimizations in subsequent milestones.

After testing the batch-mode PSL system, we will conduct a comparative study between the PSL-enhanced knowledge graphs and traditional graphRAG methods. This milestone involves further optimization of the system, with a focus on improving probabilistic reasoning accuracy and the quality of memory retrieval in LLMs. Optimization will focus on fine-tuning the PSL rules, adjusting the weights assigned to different relations, and improving the overall system’s efficiency in batch-mode processing.

We will also explore hybrid approaches that may combine batch-mode processing with elements of real-time PLN queries, as suggested by the RFP. The goal is to identify the most efficient and effective strategy for leveraging PSL in LLM memory systems.

The final milestone involves completing the project by producing a fully functioning system, accompanied by comprehensive documentation. This will include user guides, technical documentation of the architecture, and instructions for reproducing the results and extending the system for future applications. We will also deliver visualizations and reports showcasing how PSL-enhanced memory and reasoning processes function in practice.

The documentation will highlight both functional and non-functional achievements, such as scalability, moderate dataset handling, and system extensibility. The final deliverable will be a ready-to-use system integrated with OpenCog Hyperon, capable of serving as a benchmark for future research in probabilistic logic and LLM memory enhancement.

The team proposed here is eminently qualified to complete these ambitious tasks within the stated budget and timeframe. Prof. Mayank Kejriwal, who is leading the project, is an expert on knowledge graphs, having applied them to projects funded by DARPA on problems ranging from human trafficking to financial fraud. These are difficult problems requiring advanced knowledge representation. Kejriwal has also presented in the AGI conference, and his recent papers have been applying LLMs to problems like healthcare and uncertainty estimation. His work has appeared in numerous press outlets like The BBC and CNN Indonesia, and published in prestigious journals like Nature Machine Intelligence. He also has extensive systems experience: he runs a company called ACI Solutions LLC, which provide software consulting services to major corporations, by applying AGI principles to practical problems. He has taught advanced courses on text analytics and KGs, and written four books on the subject. His PhD students and engineers, who are part of his group, have applied experience in combining KGs and LLMs, allowing the team to get off to a fast start to the project. Additionally, resources available to Kejriwal's group at USC will also be an asset in implementing this project.

Open Source Licensing

We will release all outputs of this work under a permissive MIT license, with full documentation. 

Links and references

Principal Investigator's (PI) group website: https://aicomplex.github.io/ 

PI's Substack: https://aiscientist.substack.com/ 

PI's MIT Press Textbook on knowledge graphs: https://mitpress.mit.edu/9780262045094/knowledge-graphs/ 

Primer on Probabilistic Soft Logic: https://psl.linqs.org/ 

Additional videos

https://www.youtube.com/watch?v=I9tic-AeReQ&pp=ygUUbWF5YW5rIGtlanJpd2FsIEFBQUk%3D

https://www.youtube.com/watch?v=1H6gMX4smzM&pp=ygUUbWF5YW5rIGtlanJpd2FsIEFBQUk%3D