Boosting LLaMA Performance with mmap(): Unlocking the Potential of Meta’s Language Model

Meta’s LLaMA Language Model Gets a Major Boost with Llama.cpp Project and its use of mmap()

When Meta released LLaMA, its groundbreaking Large Language Model (LLM), in February, it generated considerable excitement within the AI community. However, users quickly encountered challenges when trying to run LLaMA on edge devices and personal computers. Enter Georgi Gerganov’s innovative llama.cpp project, which has since taken GitHub by storm, amassing over 19,000 stars. Over the past few weeks, the project has made remarkable advancements in enhancing LLaMA’s performance.

Llama.cpp now employs mmap() for loading weights instead of C++ standard I/O, resulting in a staggering 100x faster load time and a 50% reduction in memory usage. This optimization has led to three game-changing benefits:

More Processes: Users can now operate multiple LLaMA processes simultaneously on their computers. This allows LLaMA to not only serve as a single AI companion, but also as an entire circle of artificial friends. The utilization of mmap() with MAP_SHARED, a technique traditionally used for loading executable software, enables this groundbreaking capability.

Bigger Models: With llama.cpp, models twice as large can be loaded without compromising system stability. This optimization reduces memory requirements, enabling users to run LLaMA-13B on older Android phones and LLaMA-30B on PCs with 32GB RAM comfortably. The elimination of the need to copy pages prevents copied memory from competing with the kernel file cache, avoiding slow loading from disk each time.

Faster Loading: Linux users can now enjoy a 100x improvement in load time, while Windows and MacOS users can expect a 10x improvement. Tokens will be generated almost instantaneously when running LLaMA, providing a user experience akin to ChatGPT on the shell. It is important to note that these improvements come from an amortized cost, meaning the first load after rebooting will still be slow. However, each subsequent load will be significantly faster, making it an ideal tool for generating text from shell scripts.

The integration of mmap() into the llama.cpp project has successfully lowered barriers to entry for running large language models, granting wider public access to the benefits of these models and helping businesses cut costs. Additionally, the reduction in user-visible latency has made the tool more user-friendly and enjoyable to use.

The new mmap()-based loader is now available in the llama.cpp project under the MIT license on GitHub in both source code and binary forms: https://github.com/ggerganov/llama.cpp

source with full tech details: https://justine.lol/mmap/

{
"prompt": "Boosting Meta's LLaMA Performance with mmap, deepleaps.com, 4k, 8k, ultra highres, raw photo in hdr, sharp focus, intricate texture",
"seed": 9456568,
"used_random_seed": true,
"negative_prompt": "worst quality, low quality, normal quality, child, painting, drawing, sketch, cartoon, anime, render, 3d, blurry, deformed, disfigured, morbid, mutated, bad anatomy, bad art",
"num_outputs": 1,
"num_inference_steps": 25,
"guidance_scale": 7.5,
"width": 512,
"height": 512,
"vram_usage_level": "high",
"sampler_name": "euler",
"use_stable_diffusion_model": "neverendingDreamNED_bakedVae",
"use_vae_model": "vae-ft-mse-840000-ema-pruned",
"stream_progress_updates": true,
"stream_image_progress": false,
"show_only_filtered_image": true,
"block_nsfw": false,
"output_format": "jpeg",
"output_quality": 75,
"metadata_output_format": "json",
"original_prompt": "Boosting Meta's LLaMA Performance with mmap, deepleaps.com, 4k, 8k, ultra highres, raw photo in hdr, sharp focus, intricate texture",
"active_tags": [],
"inactive_tags": [],
"use_upscale": "RealESRGAN_x4plus",
"upscale_amount": "4",
"use_lora_model": ""
}