Completed 👍

Almalgo_Labs

Project Owner

DFR4 Best contributors experiment

Project for Round 4 - Beta
Funding Awarded $5,000 USD
Funding Pools Ideation-phase
Milestones 4 / 4 Completed

Status

Overall Status
🥳 Completed & Paid
Funding Transfered
$5,000_USD
Max Funding Amount
$5,000_USD

Funding Schedule

View Milestones

Milestone Release 1	$1,000 USD	Transfer Complete	25 Apr 2024
Milestone Release 2	$2,000 USD	Transfer Complete	30 May 2024
Milestone Release 3	$1,000 USD	Transfer Complete	27 Jun 2024
Milestone Release 4	$1,000 USD	Transfer Complete	25 Jul 2024

Status Reports

May. 7, 2024

Status

😀 Excellent

Summary

We are very close to finishing milestone 2 and 3, which means 75% of the project will be done!

Full Report

Link Name 1

Overview

We aim to compare LLM's scoring of the contributions (i.e. Comments) from DFR3 with the scores given by reviewers. We also aim to start the first steps of a web app for the best contributor's review process that will remove most friction in the process while adding an AI agent as one of the reviewers to get insights into how LLMs score contributions vs humans.

Proposal Description

How Our Project Will Contribute To The Growth Of The Decentralized AI Platform

This project aims to start developing a web app through an MVP that aims to foster the best contributor's review process. The main innovation of this project is to add AI agents as reviewers alongside humans. The project aims to contribute in at a later stage to the AI platform by training AI agents and hosting them in the AI platform. This will give access to the whole community to use this agent to judge textual contributions from others and assign a point system to it.

Company Name (if applicable)

Almalgo

The core problem we are aiming to solve

From our experience in the best contributors review process, we can assert that the process around this review is faced with many challenges, the main ones are Technical difficulty in distributing and assigning tasks, Human bias when reviewing textual comments, Ensuring randomness and fairness in the process, Smooth experience for reviewers.

Our specific solution to this problem

Our solution is a platform that randomizes the input given to each reviewer and has limited access per reviewer to keep the whole process fair, The platform will also randomize the contributions to be evaluated and choose a subset of them based on parameters provided by the admins. The platform will also have AI agents who will be joining as reviewers in the process. We will do a statistical analysis of the difference between AI agents and human evaluation alongside some explanations for the results we will get. This will provide good material to start the discussion in the community about automated AI-based reviewing systems.

The AI agent and the code used are to be implemented at a later stage on the platform to be called via API.

Project details

The work to be done in this ideation phase is:

Providing an action plan and the architecture to be implemented to get community feedback.
Developing an MVP of the Webapp solution to test the solution concept.
Simulating contributions in a round to test our solution.
Re-running DFR3 Best Contributor's Rewards With AI Agents

The aim of this project is to develop a proof of concept and a valid architecture to be fully implemented later on in the New Projects pool. This project will aim to:

Decreasing Technical difficulty in distributing and assigning tasks.
Reduce Human bias when reviewing textual comments.
Ensuring randomness and fairness in the process.
Providing a smooth experience for reviewers.

Our project also aims to engage the community at each stage of the work to be done through community calls and presentations in townhalls.

Additional links

Example of the work previously done here: Tale of Two Funds – Deepfunding

Activity Summary

Milestones

4

Total

Discussion

4

Total Comments

Reviews

3

Total Posted

Project Team

3

Total People

Group Expert Rating (Final)

Overall

4.0

Feasibility 4.0
Desirabilty 5.0
Usefulness 5.0

New reviews and ratings are disabled for Awarded Projects

Overall Community

4

from 3 reviews

5

1
4

1
3

1
2

0
1

0

Feasibility

3.3

from 3 reviews

Viability

3.3

from 3 reviews

Desirabilty

4.3

from 3 reviews

Usefulness

4

from 3 reviews

Sort by

3 ratings

Expert Review 1
Overall

5.0
- Feasibility 3.0
- Desirabilty 5.0
- Usefulness 5.0
Directly applicable and necessary for progress

The topic of this investigation does need advancing to make the DF program yet more fair and equitable. Gross analysis of the voting patterns and outcomes per funding round is being done already, but without digging into specific aspects of behaviors of individual reviewers, furthers insights will be limited.

The proposal is overly ambitious for the expectations at this phase and for the budget, but it shows a clear vision toward a meaningful set of methods and tooling.

There may be overlap with other projects within the test round, particularly the one on Reputation, thus opportunities for collaboration.

Once fully built out, with more substantial work delivered in further funding rounds, such a system would naturally fit into the marketplace.

Expert Review 2
Overall

3.0
- Feasibility 3.0
- Desirabilty 4.0
- Usefulness 4.0
A desirable project for consideration

The proposal outlines a project aimed at improving the best contributor's review process by incorporating AI agents alongside human reviewers. The project seeks to address various challenges such as technical difficulties in task distribution, human bias in reviewing, ensuring fairness, and providing a smooth experience for reviewers. Through the development of a web app MVP and the inclusion of AI agents, the project aims to gather community feedback and compare AI-generated scores with human evaluations.

Positives of this project include its innovative approach to integrating AI into the review process, its focus on community engagement, and the potential to improve the efficiency and fairness of the review process. The team demonstrates an understanding of the challenges faced in the review process and proposes feasible solutions to address them. Also, the project milestones and budget seem realistic and achievable, with a clear plan outlined for each phase of development.

However, some potential drawbacks include the complexity of integrating AI agents into the review process and the need for careful consideration of ethical implications and biases in AI-generated scores. Furthermore, the success of the project may depend on the willingness of the community to adopt and trust AI-based review systems.

In terms of feasibility, the project team has the necessary skills and experience to carry out the project successfully, as evidenced by their proposed action plan, technical expertise, and previous experience in similar projects. The project also aligns with SingularityNET's goals of advancing AI technology and fostering community participation.

In terms of viability, the project has the potential to provide valuable insights into the effectiveness of AI agents in the review process and could lead to the development of more efficient and fair review systems in the future. By leveraging AI technology, the project contributes to the advancement of AI capabilities within the SingularityNET platform.

This project is desirable to SingularityNET AI as it addresses important challenges in the review process and has the potential to improve the efficiency and fairness of community-driven initiatives. However, careful consideration of ethical and practical considerations will be essential to ensuring the success and acceptance of AI-based review systems

Expert Review 3
Overall

4.0
- Feasibility 4.0
- Desirabilty 4.0
- Usefulness 3.0
An Interesting solution to improve Deep Funding

Feasibility -> This proposal seems overall feasible, however I have a concern that we might not yet have enough data to reach significant conclusions from this proposal which decreases a bit this metric.

Viability -> The Team seems experienced and capable enough to deliver on this proposal. However, they might face some difficulties with the data available to train this ai model and the data available for comparisson and reaching a decision. In addition, the proposal seems to need contributions from the community, but to make sure people really put in the effort and have the necessary availability to contribute it would have been good to add some kind of incentives for the community, even if just a prize for the 3 strongest contributors.

Desirability -> This project can actually help streamline the process of reviewing ratings and defining the best contributors, it would also remove some of the workload and cost with reviewers.

Usefulness -> The proposal is not directly useful for the AI marketplace, but is useful for the Deep Funding Operations, not so much right away where the need for this is limited, but as Deep Funding grows and we start to have more reviews to be analyzed and more individuals participating. So it definitely addresses a future need.

Posting publicaly as

edit profile

Overall

Confidence level that the project is possible at all Feasibility

0
Confidence in a successful outcome considering team, time, and budget Viability

0
Market fit - Balancing needs and benefits against competition Desirabilty

0
To what extent will the project help the AI platform grow? Usefulness

0

Review Headline

0 /50 chars

Review Summary

0 /5000 chars

Expert Review

Overall

Confidence level that the project is possible at all Feasibility

0
Confidence in a successful outcome considering team, time, and budget Viability

0
Market fit - Balancing needs and benefits against competition Desirabilty

0
To what extent will the project help the AI platform grow? Usefulness

0

Review Headline

0 /50 chars

Review Summary

0 /5000 chars

Reviews and Ratings in Deep Funding are structured in 4 categories. This will ensure that the reviewer takes all these perspectives into account in their assessment and it will make it easier to compare different projects on their strengths and weaknesses.

Overall (Primary)
This is an average of the 4 perspectives. At the start of this new process, we are assigning an equal weight to all categories, but over time we might change this and make some categories more important than others in the overall score. (This may even be done retroactively).

Feasibility (secondary)

This represents the user's assessment of whether the proposed project is theoretically possible and if it is deemed feasible. E.g. A proposal for nuclear fission might be theoretically possible, but it doesn’t look very feasible in the context of Deep Funding.

Viability (secondary)

This category is somewhat similar to Feasibility, but it interprets the feasibility against factors such as the size and experience of the team, the budget requested, and the estimated timelines. We could frame this as: “What is your level of confidence that this team will be able to complete this project and its milestones in a reasonable time, and successfully deploy it?”
Examples:

A proposal that promises the development of a personal assistant that outperforms existing solutions might be feasible, but if there is no AI expertise in the team the viability rating might be low.
A proposal that promises a new Carbon Emission Compensation scheme might be technically feasible, but the viability could be estimated low due to challenges around market penetration and widespread adoption.

Desirability (secondary)

Even if the project team succeeds in creating a product, there is the question of market fit. Is this a project that fulfills an actual need? Is there a lot of competition already? Are the USPs of the project sufficient to make a difference?

Example:

Creating a translation service from, say Spanish to English might be possible, but it's questionable if such a service would be able to get a significant share of the market

Usefulness (secondary)

This is a crucial category that aligns with the main goal of the Deep Funding program. The question to be asked here is: “To what extent will this proposal help to grow the Decentralized AI Platform?”

For proposals that develop or utilize an AI service on the platform, the question could be “How many API calls do we expect it to generate” (and how important / high-valued are these calls?).
For a marketing proposal, the question could be “How large and well-aligned is the target audience?” Another question is related to how the budget is spent. Are the funds mainly used for value creation for the platform or on other things?
Examples:

A metaverse project that spends 95% of its budget on the development of the game and only 5 % on the development of an AI service for the platform might expect a low ‘usefulness’ rating here.

A marketing proposal that creates t-shirts for a local high school, would get a lower ‘usefulness’ rating than a marketing proposal that has a viable plan for targeting highly esteemed universities in a scaleable way.
An AI service that is fully dedicated to a single product, does not take advantage of the purpose of the platform. When the same service would be offered and useful for other parties, this should increase the ‘usefulness’ rating.

This represents the user\'s assessment of whether the proposed project is theoretically possible and if it is deemed feasible. E.g. A proposal for nuclear fission might be theoretically possible, but it doesn’t look very feasible in the context of Deep Funding.

Viability (secondary)

A proposal that promises the development of a personal assistant that outperforms existing solutions might be feasible, but if there is no AI expertise in the team the viability rating might be low.
A proposal that promises a new Carbon Emission Compensation scheme might be technically feasible, but the viability could be estimated low due to challenges around market penetration and widespread adoption.

Desirability (secondary)

Example:

Creating a translation service from, say Spanish to English might be possible, but it\'s questionable if such a service would be able to get a significant share of the market

Usefulness (secondary)

A metaverse project that spends 95% of its budget on the development of the game and only 5 % on the development of an AI service for the platform might expect a low ‘usefulness’ rating here.

A marketing proposal that creates t-shirts for a local high school, would get a lower ‘usefulness’ rating than a marketing proposal that has a viable plan for targeting highly esteemed universities in a scaleable way.
An AI service that is fully dedicated to a single product, does not take advantage of the purpose of the platform. When the same service would be offered and useful for other parties, this should increase the ‘usefulness’ rating.

Total Milestones
4
Total Budget
$5,000_USD
Last Updated
5 Aug 2024

Milestone 1 - Action plan