Cerebras Unlocks AI Potential: Cutting-Edge GPT Models & Research Advancements
Cerebras Systems Unveils Seven Novel GPT Models, Setting New Standards in Training and Open Access
Cerebras Systems, a trailblazer in AI-driven generative computing, has announced the development and release of seven cutting-edge GPT-based large language models (LLMs) for the research community’s benefit. This groundbreaking move marks the first instance of a company utilizing non-GPU-based AI systems to train LLMs with up to 13 billion parameters, while sharing the models, weights, and training methodology under the widely recognized Apache 2.0 license. All seven models were developed using the 16 CS-2 systems within Cerebras’ Andromeda AI supercomputer.
Cerebras’ release of these seven GPT models not only showcases the impressive capabilities of its CS-2 systems and Andromeda supercomputer as top-tier training platforms, but also positions the company’s researchers among the elite in the AI realm. The rapid expansion of LLMs, led by OpenAI’s ChatGPT, has ignited a competitive race to develop more potent and specialized AI chips. Although numerous companies have claimed to offer alternatives to Nvidia® GPUs, none have proven both capable of training large-scale models and open to sharing their results with flexible licenses. In contrast, Cerebras directly addresses these issues by training a series of seven GPT models on the Andromeda supercomputer, with 111M, 256M, 590M, 1.3B, 2.7B, 6.7B, and 13B parameters each.
Cerebras’ unprecedented speed in completing this typically multi-month project in just a few weeks is thanks to the extraordinary capabilities of the CS-2 systems within Andromeda and the company’s weight streaming architecture that eliminates the challenges of distributed computing. These accomplishments demonstrate that Cerebras’ systems are fully equipped to handle the most demanding and intricate AI workloads today.
For the first time, a series of GPT models trained with state-of-the-art efficiency techniques has been made publicly available. These models, developed using the Chinchilla recipe for optimal compute budget efficiency, boast reduced training time, lower training costs, and less energy consumption compared to existing public models.
Only a handful of organizations have the capacity to train truly large-scale models, and even fewer have achieved this on dedicated AI hardware, said Sean Lie, Cerebras’ co-founder and Chief Software Architect. By releasing seven fully-trained GPT models to the open-source community, Cerebras demonstrates the remarkable efficiency and ability of CS-2 systems to quickly tackle the most extensive AI challenges—tasks that would normally require hundreds or thousands of GPUs. Cerebras eagerly shares these models and insights with the AI community.
As LLM development becomes increasingly costly and complex, companies have halted public releases of their models. In contrast, Cerebras promotes open research and access by releasing all seven models, the training methodology, and training weights under the permissive Apache 2.0 license. This offers several advantages:
The training weights supply a highly accurate pre-trained model for fine-tuning, enabling users to create powerful, industry-specific applications with minimal effort by using a small amount of custom data.
The varying sizes of the models and their respective checkpoints allow AI researchers to develop and test new optimizations and workflows that benefit the community at large.
The use of the industry-standard Apache 2.0 license allows these models to be used for research or commercial ventures without royalties.
Today’s release, which builds on the GPT architecture, offers several technical contributions:
The derivation of a new scaling law based on an open dataset: Scaling laws, as crucial to AI as Moore’s Law is to semiconductors, enable researchers to predict the relationship between a given compute training budget and model performance. Cerebras’ scaling law builds upon previous work by OpenAI and DeepMind and is the first scaling law derived from an open dataset, making it reproducible by the AI community.
The demonstration of a simple, data-parallel-only approach to training: Traditional LLM training on GPUs necessitates a complex combination of pipeline, model, and data parallelism techniques. In contrast, Cerebras’ weight streaming architecture is a data-parallel-only model that requires no code or model modification to scale to arbitrarily large models.
The introduction of Cerebras-GPT, the first family of GPT models that are compute-efficient at every size: Existing open GPT models are trained on a fixed number of data tokens. By applying the Chinchilla training recipe across all model sizes, Cerebras-GPT establishes a new high-accuracy baseline for widespread use.
Scaling large language models is a formidable technical challenge. Cerebras now joins an exclusive group of organizations that have successfully trained and open-sourced such vast model suites. Stella Biderman, Executive Director at EleutherAI, expressed excitement about Cerebras building upon EleutherAI’s work, such as the Pile and Eval Harness, to create a range of open models that will benefit researchers globally.
Karl Freund, founder and principal analyst at Cambrian AI, noted that Cerebras’ release of seven GPT models not only highlights the prowess of its CS-2 systems and Andromeda supercomputer as elite training platforms but also positions Cerebras researchers among the best in the AI field. There are only a few companies worldwide capable of deploying end-to-end AI training infrastructure and achieving state-of-the-art accuracy with the largest LLMs. Cerebras now stands among them. By sharing these models with the open-source community under the Apache 2.0 license, Cerebras demonstrates its commitment to ensuring AI remains an open technology that benefits humanity on a broad scale.
All seven Cerebras-GPT models can be accessed immediately on Hugging Face and Cerebras Model Zoo on GitHub. The Andromeda AI supercomputer used to train these models is available on-demand at https://www.cerebras.net/andromeda/.
Cerebras may not be as proficient as heavyweights like LLaMA, ChatGPT, or GPT-4, but it’s got a special edge: it’s available under the Apache 2.0 license, a fully permissive open-source license. This makes its weights accessible for anyone to download and explore. Contrast this with models like LLaMA, which restrict usage to non-commercial endeavors, such as academic research or personal experimentation.
To try out LLaMA, you’d need a powerful GPU or access to a volunteer-run service like KoboldAI. You can’t simply visit a website like ChatGPT and start feeding it prompts without risking a DMCA takedown notice from Meta.
Cerebras’ real purpose is to demonstrate the revolutionary silicon technology the company has spent years developing. Their chips employ a novel silicon architecture, networking together chips at the die-level, as opposed to using multiple NVIDIA GPU-laden computers. By releasing Cerebras-GPT and proving its competitiveness, Cerebras positions itself as a strong rival to NVIDIA and AMD, which benefits everyone through healthy competition.
In simpler terms, Cerebras isn’t as advanced as LLaMA or ChatGPT. It’s a smaller, 13B-parameter model that has been intentionally “undertrained” to reach a “training compute optimal” state. However, it’s still a valuable addition to open-source models like GPT-2, GPT-J, and GPT NeoX. The community might be able to enhance the model through fine-tuning or creating LORAs for it.
Cerebras-GPT’s availability is particularly important for those who can’t rely on OpenAI’s monopoly, such as enterprises with strict security requirements, foreign governments, or individuals seeking control over their infrastructure.
It’s challenging to make direct comparisons between Cerebras-GPT, GPT-J, and GPT NeoX, but a quick look at their performance on tasks like OpenBookQA and ARC-c reveals that Cerebras-GPT holds its own. Although these tasks require a certain level of “common sense” knowledge, even the formidable ChatGPT sometimes struggles with basic arithmetic.
And let’s not forget GPT-4, which dominates the competition across the board!
Is Cerebras-GPT the go-to choice? Based on the data, it doesn’t outperform existing open-source models. However, with some fine-tuning, it might just give GPT-J, GPT NeoX, and others a run for their money. But we’ll leave that to the experts to decide!
{
"prompt": "Cerebras Unlocks AI Potential: Cutting-Edge GPT Models & Research Advancements, deepleaps.com, best quality, 4k, 8k, ultra highres, raw photo in hdr, sharp focus, intricate texture, skin imperfections, photograph of",
"seed": 8091925,
"used_random_seed": true,
"negative_prompt": "worst quality, low quality, normal quality, child, painting, drawing, sketch, cartoon, anime, render, 3d, blurry, deformed, disfigured, morbid, mutated, bad anatomy, bad art",
"num_outputs": 1,
"num_inference_steps": 25,
"guidance_scale": 7.5,
"width": 512,
"height": 512,
"vram_usage_level": "high",
"sampler_name": "euler",
"use_stable_diffusion_model": "liberty_main",
"use_vae_model": "vae-ft-mse-840000-ema-pruned",
"stream_progress_updates": true,
"stream_image_progress": false,
"show_only_filtered_image": true,
"block_nsfw": false,
"output_format": "jpeg",
"output_quality": 75,
"metadata_output_format": "json",
"original_prompt": "Cerebras Unlocks AI Potential: Cutting-Edge GPT Models & Research Advancements, deepleaps.com, best quality, 4k, 8k, ultra highres, raw photo in hdr, sharp focus, intricate texture, skin imperfections, photograph of",
"active_tags": [],
"inactive_tags": [],
"save_to_disk_path": "",
"use_upscale": "RealESRGAN_x4plus",
"upscale_amount": "4",
"use_lora_model": ""
}
