Patrick Nercessian
Project OwnerLead the delegation of tasks and oversight of progress on each milestone. Utilize prior experience with evolutionary methods to guide and inform on structural decisions of the evolution.
The goal of this project is to create a framework that uses evolutionary computation to design new neural network architectures for natural language prediction. The idea is to build on the success of transformers while exploring entirely new design possibilities that could outperform them. By simulating evolution—introducing variations, selecting the best-performing models, and iterating—the framework aims to uncover architectures that go beyond what human researchers have created. We aim to represent neural networks as directed acyclic graphs - and perform mutations and crossover on those representations. We also plan to allow mutations/crossover on model hyperparameters as well.
Explore and demonstrate the use of evolutionary methods (EMs) for training various DNNs including transformer networks. Such exploration could include using EMs to determine model node weights, and/or using EMs to evolve DNN/LLM architectures. Covariance Matrix Adaptation Evolution Strategy (CMA-ES) is an example of one very promising evolutionary method among others.
In order to protect this proposal from being copied, all details are hidden until the end of the submission period. Please come back later to see all details.
1. Develop a mechanism to represent neural networks architectures (including transformers) as DAGs. 2. Implement hyperparameter mutation and recombination operators. 3. Implement architectural mutation and recombination operators which ensure valid architectures.
The creation of a framework which can create artificial neural networks from DAGs. By training on very small amounts of data we will also confirm the framework’s ability to generate valid architectures using mutation and recombination.
$12,000 USD
1. Integrate evaluation and selection mechanisms using standard NLP datasets and fitness metrics. 2. Run initial evolutionary experiments. Try different neural network sizes and dataset sizes to generate data about how these experiments and architectures differ across scale.
An updated framework which is feature-complete for evolutionary runs. The output of the evolutionary experiments including perplexity on next-word prediction and other related evolutionary output metrics.
$8,000 USD
1. Develop advanced mutation and recombination operators incorporating biases toward favorable traits such as modularity or sparsity. 2. Run further experiments using these updated operators including creative population initialization (such as portions of the population having varying subsets or magnitudes of these new biases).
An updated framework which includes these advanced variation operators aimed to improve the search of the evolutionary algorithm.. The output of the evolutionary experiments including perplexity on next-word prediction and other related evolutionary output metrics.
$8,000 USD
1. Scale up the best-performing architectures for larger-scale training runs. 2. Compare the scaled models’ performance against known transformers of similar scale using training/test metrics and benchmark results. 3. Document findings and release codebase with comprehensive instructions and supporting documentation.
Larger-scale training runs to demonstrate the scalability of the best output model architectures from the evolutionary runs. Also the final codebase and report of our findings over the entire project.
$12,000 USD
Reviews & Ratings
Please create account or login to write a review and rate.
Check back later by refreshing the page.
Join the Discussion (0)
Please create account or login to post comments.