Company Name (if applicable)

Project details

When building an LLM (or indeed, any DNN), one important task is the tuning of hyperparameters: settings that define and configure the learning process and architecture of the network itself. These range from choices about how data is presented (for example, context window size) to decisions around the structure of the network itself (for example, vocabulary size). These parameters are inter-dependent, both with each other and with the data, and the desired learning outcomes, and determining their optimal (or even satisfactory) values is a difficult process, with little to guide the data scientist. Often, these must be tuned by trial and error, a process which is tedious and repetitive, especially when the number of hyperparameters under consideration is large, as is often the case with transformer-based architectures more generally, and LLMs specifically.

The difficulty of this problem stems from the complex relationship between hyperparameters, which can often have unpredictable results when any of them are adjusted. Furthermore, while certain natural bounds exist (for example, vocabulary size is ultimately limited by the size of the token dictionary of the input data), these can be uselessly large, giving little idea of where to begin or explore. While we can certainly assess what outcome any given choice of hyperparameter(s) will have, we have no easy way of knowing which to tune and in which way. To make matters even worse, it is not always the case that we want to straightforwardly minimize or maximize any given hyperparameter: continuing the example of the vocabulary size, there likely exists an 'ideal' or 'good enough' size, which is not necessarily the largest possible (as it would include many tokens that might not be relevant or useful) nor the smallest possible (as this would discard too many relevant or useful tokens). This makes it challenging to approach this problem in a systematic way.

In situations similar to ours, evolutionary algorithms (or EAs) are an often-used metaheuristic. In particular, we want a multi-objective EA, not only because we want the ability to tune as many (or as few) hyperparameters as we want, we may assess the outcome of a particular hyperparameter, or set of hyperparameters, differently to others. Together with suitable mutation and crossover operations, this would allow both the automation of the search for better hyperparameter combinations and settings, but also a more directed search than a purely trial-and-error one. Lastly, the generic nature of multi-objective EAs, and their ability to balance any number of objectives, would make them suitable regardless of what kind of tuning is desired, giving data scientists flexibility in their use.

At the same time, however, not just any multi-objective EA will do for this problem. In particular, any successful approach must address the following two problems:

1. As every measure of the fitness function requires building, and training, a neural network, we must not do this unless absolutely necessary (the rebuilding problem);
2. As the cost of the fitness function is thus high, we must make use of parallel resources as part of the EAs execution as far as possible (the parallelism problem).

Many, if not most, 'off-the-shelf' EAs, multi-objective or not, fall afoul of one, or both, of these problems: the general assumption is that fitness function invocation is cheap, and most EA designs function like simulations, creating a large critical path that resists parallelism. Thus, a careful, problem-specific design is needed here, which avoids both of these problems, but also retains enough flexibility and power to be usable for tuning any number of hyperparameters, while balancing a range of fitness measures, to suit whichever application any given neural network should be trained for.

We will design, implement, and test an EA-based solution to the hyperparameter tuning problem, extensible to suit various choices of hyperparameter(s) and measurable objectives relating to these. Our solution will avoid the rebuilding and parallelism problems, and will be 'blind' to exactly what is being used to train the neural network whose hyperparameters are being tuned. To demonstrate our solution's efficacy, we will use NanoGPT as a test bed, with the combined works of Shakespeare as input. As a benchmark, we will use two hyperparameters:

• Context window size; and
• Vocabulary size.

We will measure the efficacy of the tuning by comparing loss, which we aim to minimize. We will also use these measurements to improve the performance of our solution, particularly focusing on better utilization of parallel resources, fewer rebuilds of the neural network architecture, improving the quality of the outcome with fewer generations, and lastly, any overall runtime or memory use improvements. We will demonstrate the efficacy of these improvements bycomparing against the measurements mentioned previously.

Team

Koz Ross, Data Engineer - Project Leader

Koz Ross is a seasoned Data Engineer and Software Developer. With a background in both software engineering and academia in evolutionary computation and data structures, Koz combines familiarity with current literature on the topic of EAs with an understanding of how these can be implemented, and engineered, appropriately. He has also led several projects in a range of areas, ranging from eDSLs to libraries for code generation to a whole compiler, making him familiar with the organizational principles and practices required to organize and run this project to successful completion. He will be responsible for the overall direction of the project, as well as providing expert advice on EAs.

Gergely Szabo, Data Engineer - Project Scientist

Gergely Szabo is a seasoned Data Engineer and Software Developer. He is an experienced team leader and avid contributor to open-source libraries. With a robust background in software engineering, Gergely blends his passion for data with practical expertise in creating scalable, efficient, and maintainable solutions. He is a versatile team player and an adept communicator, making him a valuable asset to the team. He will assist Koz in all aspects of data engineering.

Jared Pon, Senior Software Engineer - Project Software Architect

Jared Pon is a skilled Software Developer with deep expertise in compiler development including tools for translating algebraic data types across languages and designing algorithms for type inference, lambda lifting, and instruction set generation. He is proficient in multiple software languages and has a strong command of backend development, database design, and performance optimization. He comes to the team with a rigorous academic background in Computer Science and Mathematics, Jared combines a passion for innovation with exceptional problem-solving skills, making him a versatile and impactful member of the team.

Banashree Sarma, AI Engineer - Project LLM Engineer

Dr Sarma received her Doctorate in Data Science and Artificial Intelligence from the Indian Institute of Technology in Madras, India. She specializes in natural language processing and reinforcement learning. She has built context embedding models, and LLMs for a number of languages, and is currently a member of the NLP team at MLabs AI, as well as handling our reinforcement learning activities. She will be responsible for the coordinating the LLMs built during the project.

References

https://github.com/karpathy/nanoGPT

Open Source Licensing

Links and references

Cost: $12.5K

References

https://github.com/karpathy/nanoGPT

Website

https://www.mlabs.city/