ACACIA Associate-Team

Generative AI has rapidly become unavoidable, with widely used applications such as code assistants, text analysis, information retrieval, and translation. The core architecture of generative AI, relying on the transformer framework, inherently requires sequential token generation, leading to high computing times. Furthermore, the increasing complexity of models, requires a very large memory capacity in order to store them. Recent usages such as “reasoning” models even further push the memory demand because of their very large context requirements. When model weights and context exceed the available memory, frequent I/O operations become necessary, degrading performance and efficiency. In the end, a substantial and growing energy consumption is associated with the use of these models, raising concerns about both environmental impact and operational feasibility.

Most recent improvements in LLM quality have been mainly driven by scaling up the model. While it has led to remarkable results, this calls for even larger computing power and memory size. To ensure that generative AI remains accessible, efficient, and environmentally responsible, there is an urgent need to optimize resource usage. This can be achieved through architectural innovations that reduce the computational load, smarter memory management, and more efficient data loading strategies.

Our proposal builds upon existing techniques from the literature designed to accelerate the inference and/or training of large language models. We aim to adapt and enhance three of these approaches for our specific objectives: the use of multiple models of varying cost and quality, organized in “cascades”; the application of low-rank adapters to specialize models for particular tasks; and the adoption of the mixture-of-experts paradigm to reduce the memory requirement during inference.

More information to come.