How LLaMA, MPT, Falcon, and LLaMA-2 put open-source LLMs on the map… — Open-source research on large language models (LLMs) is incredibly valuable, as it aims to democratize a powerful and influential technology. Although open-source LLMs are now commonly used and widely studied, this area of research saw some initial struggles that were difficult to overcome. Namely, open-source LLMs performed poorly at first…

Understanding GPT-Neo, GPT-J, GLM, OPT, BLOOM, and more… — Research on language modeling has a long history that dates back to models like GTP and GPT-2 or even RNN-based techniques (e.g., ULMFit) that predate modern, transformer-based language models. Despite this long history, however, language models have only become popular relatively recently. The first rise in popularity came with the…

How high-quality data impacts every aspect of the LLM training pipeline… — Large Language Models (LLMs) have been around for quite some time, but only recently has their impressive performance warranted significant attention from the broader AI community. With this in mind, we might begin to question the origin of the current LLM movement. What was it that actually made recent models…

The gap between open-source and proprietary LLMs continues to shrink… — Recent research in open-source large language models (LLMs) has mostly focused upon two areas: imitation learning and pre-training open-source base models. Though both approaches are viable, the creation of high-quality, open-source base models is especially enticing, as these models can be further fine-tuned (at a lower cost) and used in…

How high-quality base models unlock new possibilities for an entire industry… — Recently, we have overviewed a lot of current research on the creation of open-source large language models (LLMs). Across all of this work, models are created using a common framework with a few simple components; see below.

Leveraging imitation to create high-quality, open-source LLMs… — As research progresses on large language models (LLMs), one key question that remains unanswered is whether an existing, high-quality LLM can be used to effectively train another LLM. Currently, there is a lot of debate and contention around this topic. The recent explosion of open-source imitation models initially indicated that…

Are proprietary LLMs like ChatGPT and GPT-4 actually easy to replicate? — The proposal of the LLaMA suite [2] of large language models (LLMs) led to a surge in publications on the topic of open-source LLMs. In many cases, the goal of these works was to cheaply produce smaller, opens-source LLMs (for research purposes) that have comparable quality to proprietary models like…

LaTM, CREATOR, and other closed-loop frameworks for LLM tool usage… — In recent overviews, we have explored the utility of augmenting large language models (LLMs) with external tools. These models can be taught to leverage tools in a variety of ways. However, we should realize that existing tool-following LLMs leverage only a limited set of potential tools [3], whereas the range…

What happens when we give LLMs access to thousands of deep learning models? — Recently, we have witnessed a rise of foundation models to popularity within deep learning research. Pre-trained large language models (LLMs) have led to a new paradigm, in which a single model can be used — with surprising success — to solve many different problems. Despite the popularity of generic LLMs…