Question of Semantics

chevron-icon
RFP Proposals
Top
chevron-icon
project-presentation-img
Expert Rating 3.6
evolveai
Project Owner

Question of Semantics

Expert Rating

3.6

Overview

Assessment of solution quality at a semantic level is limited in current MOSES approach. In order to support the long-term vision of AGI, the proposed effort will enhance MOSES to use LLMs to integrate qualitative assessments of individual solutions and solution sets. Several methods will be explored to augment classic fitness assessment and meta-optimization with search suggestions that will enable MOSES to identify and explore high-value areas in a complex, multi-objective and subjective space.

RFP Guidelines

Utilize LLMs for modeling within MOSES

Complete & Awarded
  • Type SingularityNET RFP
  • Total RFP Funding $150,000 USD
  • Proposals 10
  • Awarded Projects 1
author-img
SingularityNET
Oct. 9, 2024

This RFP invites proposals to explore the integration of LLMs into the MOSES evolutionary algorithm. Researchers can pursue one of several approaches, including generation modeling, fitness function learning, fitness estimation, investigation into domain-independent “cognitively motivated” fitness functions, or propose new innovative ways to leverage LLMs to enhance MOSES's capabilities within the OpenCog Hyperon framework.

Proposal Description

Project details

Enhancing Fitness Evaluation and Meta-Optimization in MOSES Using LLMs for Domain-Specific Insights

Objective

This research aims to enhance the fitness evaluation and meta-optimization processes in the MOSES evolutionary algorithm by integrating Large Language Models (LLMs) to incorporate domain-specific factors, expand the range of objectives that MOSES can assess (e.g., creativity, interpretability, robustness), and provide semantic guidance for multi-objective search optimization. The goal is to ensure that competing objectives (e.g., accuracy vs. simplicity, efficiency vs. creativity) are balanced in a way that incorporates both quantitative and domain-specific qualitative factors.

The objective is to improve the quality of evolved solutions by guiding the search process toward more meaningful, domain-relevant features and interactions. It is assumed that a Hyperon implementation of MOSES is near completion, (as mentioned in a forum chat in late October). Interactions with the LLM, coordination with MOSES, and new fitness evaluation methods will be implemented within MeTTa using Python. However, the specific implementation approach will be iteratively adjusted as necessary for effective prototyping.


Month 1-3: Integration of LLM for Domain-Specific Fitness Evaluation

The first phase of the research will focus on investigating and prototyping methods for integrating the LLM with MOSES to enhance the fitness evaluation function. The aim is to incorporate domain-specific and subjective factors into the evaluation process. This will involve an iterative, automated prompt engineering approach, with the following steps:

  1. Informing the LLM
    The LLM will be provided with problem-specific context, such as dataset characteristics, feature sets, goal descriptions, and the desired qualities of the final solutions. The LLM may also be informed about the evolutionary algorithm, including the fitness function in use, genetic manipulations, and related aspects of the MOSES process.

  2. Identifying Domain-Specific Factors
    The LLM, through carefully crafted prompts, will identify key factors to assess. These may include domain-specific relationships and interactions that are critical for solution evaluation but not captured by traditional fitness functions. Potential factors could include feature interactions, behavioral changes over time, solution efficiency, and solution clarity.

  3. Creating an Evaluation Rubric
    Using the identified factors and information about the state of the evolutionary search, the LLM will generate a rubric to assess solutions beyond the traditional MOSES fitness function. The rubric will be multi-objective, incorporating qualitative insights, and may provide a specific numerical evaluation or categorical flags (e.g., high value, medium value, low value).

  4. Applying the Rubric
    Once the rubric is generated, it will be used to augment the fitness evaluation performed by MOSES. Several approaches will be explored to integrate the rubric into the fitness evaluation process:

    • Basic integration: The rubric’s evaluation will be combined with the traditional MOSES fitness evaluation.
    • Subjective analysis: The rubric’s categorization (e.g., high/medium/low value) will influence selection and evolutionary manipulations.

Expected Outcome:
The result of this first phase will be a preliminary integrated system where MOSES’ fitness evaluation is enhanced by LLM-generated insights, guiding the evolutionary search toward more effective solutions.


Month 4-6: Meta-Optimization with LLM Feedback

The second phase will focus on refining the meta-optimization process by integrating LLM feedback directly into the evolutionary search. This will ensure that the search process is steered toward more meaningful solutions, leveraging domain-specific insights from the LLM at multiple levels:

  1. LLM-Powered Semantic Guidance for Adaptation
    The LLM will provide semantic guidance for adapting individuals within the evolutionary search process. This guidance will help MOSES navigate complex, multi-objective spaces with competing goals. For example:

    • If a solution is highly accurate but overly complex, the LLM might suggest simplifications without significant loss in accuracy.
    • If creativity is a key goal, the LLM could suggest areas of the solution space to explore where more novel solutions are likely to emerge.

This will be achieved through generalized prompts that analyze solutions and sub-populations.  Methods that track and leverage changes in population composition over time as well as changes in complexity of individuals and other factors over time will be explored.  The intent here is to identify and use the types of patterns that an LLM can meaningfully analyze to provide semantic guidance, as well as the types of manipulations that an LLM can meanginfully perform. 

  1. Balancing Multi-Objective Trade-offs
    A key challenge in many complex domains is balancing competing objectives (e.g., accuracy vs. simplicity, robustness vs. creativity). LLMs can help MOSES dynamically balance these objectives based on the state of the evolutionary algorithm.  For example,

    • If the search is favoring accuracy over simplicity, the LLM might suggest adjusting search parameters to explore simpler models while still maintaining performance.
    • The LLM can also propose adaptive weighting for different objectives, helping MOSES shift its priorities throughout the search based on ongoing evaluations and solution progress.

This will be achieved through parameterized prompts that capture performance data on individuals and identify trends.

Expected Outcome:
At the end of these first two phase, the MOSES system will be enhanced by LLM-guided fitness evaluation and meta-optimization. This will allow the system to better capture domain-specific nuances in the search process. The final prototype will be assessed by measuring the following:

  • Predictive accuracy
  • Domain relevance (Do solutions reflect meaningful, real-world factors?)
  • Computational cost

Deliverables

  1. Recommendations for LLM Integration
    A set of recommendations for integrating LLMs into future MOSES applications, with strategies for incorporating domain-specific factors and optimizing multi-objective search.

  2. Performance Benchmarks
    Comparative performance benchmarks assessing the impact of LLM guidance on MOSES’ performance, with respect to:

    • Solution quality (accuracy, interpretability, robustness)
    • Computational costs (evaluation time, API costs)
    • Domain relevance (alignment with real-world factors)
  3. Best Practices for Multi-Objective Optimization
    A demonstration of best practices for balancing competing objectives using LLM-driven insights, focusing on effective integration into the evolutionary process.

  4. Open-Source Code Release
    The release of the Python implementation that integrates LLMs within MOSES via MeTTa, along with detailed documentation and usage examples.

Open Source Licensing

BSD - Berkeley Software Distribution License

Results of this effort will be made available to the community for further extension.

Links and references

Proposal Video

Not Avaliable Yet

Check back later during the Feedback & Selection period for the RFP that is proposal is applied to.

  • Total Milestones

    2

  • Total Budget

    $80,000 USD

  • Last Updated

    8 Dec 2024

Milestone 1 - Integrate LLM for Domain-Specific Fitness Eval

Description

Integrate the LLM to enhance the fitness evaluation function by incorporating domain-specific factors into the evaluation process.

Deliverables

Demonstration of integration

Budget

$40,000 USD

Success Criterion

Evolutionary process produces solutions that incorporate desired quantitative and qualitative characteristics.

Milestone 2 - Meta-Optimization with LLM Feedback

Description

Refine the meta-optimization process by integrating LLM feedback into the evolutionary search. The aim is to ensure that the search process is guided toward more meaningful solutions leveraging domain-specific insights from the LLM.

Deliverables

Recommendations for making targeted use of LLMs in future MOSES applications with strategies for incorporating domain-specific factors and optimizing multi-objective search. Performance benchmarks comparing MOSES’s performance with and without LLM guidance including computational costs and solution quality. Demonstration of best practices for balancing competing objectives using LLM-driven insights with an emphasis on effective integration. Open-source release of functional code for methods developed.

Budget

$40,000 USD

Success Criterion

Evolutionary process adapts to produce solutions that effectively balance multiple quantitative and qualitative factors. Ideally, solutions are more effective, more appropriate qualitatively, and achieved with low additional computational cost.

Join the Discussion (0)

Expert Ratings

Reviews & Ratings

Group Expert Rating (Final)

Overall

3.6

  • Feasibility 4.0
  • Desirabilty 3.3
  • Usefulness 4.0
  • Expert Review 1

    Overall

    3.0

    • Compliance with RFP requirements 4.0
    • Solution details and team expertise 3.0
    • Value for money 4.0
    The proposal is to use LLMs for fitness evaluation. It makes sense but is kinda simplistic/unambitious

    Using LLMs as fitness estimators or evaluators for MOSES is a very simple thing to do and in itself doesn't need $80K. On the other hand, just setting up MOSES + an LLM to work on some application project will take a lot of work given the state of Hyperon, so in that sense this would be a worthy project.

  • Expert Review 2

    Overall

    4.0

    • Compliance with RFP requirements 4.0
    • Solution details and team expertise 4.0
    • Value for money 4.0
    Good and on topic.

    The proposal is consice, clear, and on topic. The methodology of the work is clear as well.

  • Expert Review 3

    Overall

    4.0

    • Compliance with RFP requirements 4.0
    • Solution details and team expertise 3.0
    • Value for money 4.0

    Using domain specific semantic search from LLMs to aid in multi-objective optimization. Solid proposal.

feedback_icon