Today, we are thrilled to introduce the initial two models of the new iteration of Llama, known as Meta Llama 3, now available for widespread use. This release showcases pre-trained and finely-tuned language models equipped with 8B and 70B parameters, designed to accommodate a diverse range of applications. The next generation of Llama exhibits cutting-edge performance across various industry benchmarks and introduces enhanced reasoning capabilities.
We firmly believe that these models represent the pinnacle of open-source offerings in their category. In line with our longstanding commitment to openness, we are placing Llama 3 in the hands of the community. Our goal is to ignite the next phase of AI innovation across all facets—from applications and developer tools to evaluations, inference optimizations, and beyond. We are eager to witness the creations that emerge and eagerly anticipate your valuable feedback.
Meta Llama 3 Model Architecture
In keeping with our design ethos, we chose Llama 3 a rather conventional decoder-only transformer architecture. We were able to make several significant improvements over Llama 2. With a vocabulary of 128K tokens, Llama 3’s tokenizer encodes language much more effectively, significantly enhancing model performance. In both the 8B and 70B sizes, we have implemented grouped query attention (GQA) to increase the inference efficiency of Llama 3 models. To ensure that self-attention does not cross document boundaries, we used a mask during the 8,192 token sequences used to train the models.
Training Data
Curating a sizable, excellent training dataset is essential to training the best language model. Following our design tenets, we made significant investments in pretraining data. Pre-trained on more than 15T tokens, all of which were gathered from publicly accessible sources, is Llama 3. Compared to Llama 2, our training dataset has four times more code and is seven times larger. More than 5 percent of the Llama 3 pretraining dataset is composed of high-quality non-English data covering more than 30 languages, in anticipation of future multilingual use cases. But, we do not anticipate these languages to perform at the same level as English.
We have developed a series of data-filtering pipelines to guarantee that Llama 3 is trained on the best possible data. To predict data quality, these pipelines use text classifiers, NSFW filters, heuristic filters, and semantic deduplication techniques. Because Llama’s earlier generations are surprisingly adept at spotting high-quality data, we trained Llama 2 to produce the training data for Llama 3’s text-quality classifiers.
To determine the most effective methods of combining data from various sources in our final pretraining dataset, we also conducted several in-depth experiments. Through these tests, we were able to determine the right combination of data that will guarantee Llama 3’s performance in a variety of use cases, such as trivia, STEM, coding, historical knowledge, etc.
What’s next for Meta Llama 3?
The Llama 3 8B and 70B models represent the start of what we intend to release for Llama 3. And there’s plenty more to come. Our largest models have over 400B parameters, and while they are still training, our team is excited to see how they are trending. Over the next few months, we’ll release several models with new features such as multimodality, the ability to converse in multiple languages, a much longer context window, and improved overall capabilities. We will also publish a detailed research paper once we have completed training Llama 3.
To give you a sneak peek at where these models are today as they continue to train, we thought we’d share some snapshots of how our largest LLM model is performing. Please keep in mind that this data is based on an early checkpoint of Llama 3, which is still in the process of being trained, and these capabilities are not supported by the models released today.
Building with Llama 3
Our vision is to allow developers to customize Llama 3 to support relevant use cases while also making it easier to adopt best practices and improve the open ecosystem. With this release, we introduce new trust and safety tools, including updated components for Llama Guard 2 and Cybersec Eval 2, as well as Code Shield, an inference time guardrail for filtering insecure code generated by LLMs.
Additionally, torch tune, a brand-new PyTorch-native library for quickly authoring, adjusting, and experimenting with LLMs, and I worked together to co-develop Llama 3. Torch tune offers fully PyTorch-written training recipes that are both memory-efficient and hackable. Hugging Face, Weights & Biases, EleutherAI, and other well-known platforms are integrated with the library. It also supports Executorch, which makes it possible to run effective inference on a range of mobile and edge devices. We have a comprehensive getting started guide that walks you through the entire process of downloading Llama 3, from prompt engineering to using Llama 3 with LangChain, to deploying Llama 3 at scale within your generative AI application.
How good is Llama 3?
The 8B and 70B parameter Llama 3 models, according to Meta, are a huge improvement over Llama 2. The advancements in pre- and post-training have made this feasible. According to the company’s website, “Our trained and instruction-fine-tuned models are the best models existing today at the 8B and 70B parameter scale.” The company claims that by implementing post-training procedures, Llama 3’s reasoning, code generation, and instruction have all significantly improved, making it more steerable.
According to Meta, Llama 3 8B outperformed other open-source AIs such as Mistral 7B and Gemma 7B in benchmark evaluations. In benchmarks like MMLU 5-shot (Massive Multitask Language Understanding), GPQA 0-shot (A Graduate-Level Google-Proof Q&A Benchmark), HumanEval 0-shot (a benchmark for evaluating the multilingual ability of code generative models), GSM-8K 8-shot and Math 4-shot, CoT (maths and word problems), Llama 3 outperformed Google’s Gemma 7B and Mistral’s Mistral 7B, and Anthropic’s Claude 3 Sonnet.
Though Llama 3’s use cases have not been formally announced by Meta. Since Llama 3 is comparable to current AI chatbots, it can be used to generate various types of texts, including scripts, code, poems, and musical compositions. It can be used to translate languages and summarize factual subjects.
How to try Llama 3?
Meta announced that it has integrated Llama 3 into Meta AI, which is available on Facebook, Instagram, WhatsApp, Messenger, and the web. It is readily available to developers because Meta has integrated the LLM into the Hugging Face ecosystem. It is also available through Perplexity Labs, Fireworks AI, and cloud-based platforms like Azure ML and Vertex AI.
Llama 3 models will soon be available on AWS, Google Cloud, Hugging Face, Databricks, Kaggle, IBM Watson, Microsoft Azure, NVIDIA NIM, and Snowflake.
Meta AI is currently available in English across the United States via WhatsApp. Meta is also expanding into new markets, including Australia, Canada, Ghana, Jamaica, Malawi, New Zealand, Nigeria, Pakistan, Singapore, South Africa, Uganda, Zimbabwe, and Zambia.
Conclusion
Meta Llama 3 represents a revolutionary development in open-source language models. In many applications, models with 8B and 70B parameters show state-of-the-art performance. Meta’s commitment to openness and community engagement heralds a new era of AI innovation and invites thoughtful commentary and intriguing advancements.
