Zyphra is a company that focuses on full-stack probabilistic computing within the artificial intelligence and information processing sectors. They were founded in 2020 and are based in Palo Alto, California. The company offers solutions that enhance the capabilities of AI through advanced probabilistic models and algorithms.
Zyphra recently released their latest model, Zamba2-2.7B, which is a state-of-the-art small language model that achieves twice the speed and 27% reduced memory overhead compared to other models. This model is trained on a substantial dataset of approximately 3 trillion tokens derived from Zyphra’s proprietary datasets, allowing it to match the performance of larger models like Zamba1-7B and other leading 7B models.
Small Language Models offer a powerful combination of advanced AI capabilities with reduced computational demands, making them an attractive option for businesses, researchers and developers. With their ability to perform well on specific tasks, SLMs have the potential to revolutionize a wide range of industries and applications.
SLMs provide tailored efficiency and precision, cost-effectiveness, improved security and privacy and adaptability. They have lower computational demands, use less energy and deploy faster, making them more accessible and feasible for organizations or researchers with limited resources. SLMs also perform better, have higher accuracy and are more relevant due to their focused training and adaptability to specific tasks or domains. They also support on-device AI, allowing for faster and more efficient processing and are more environmentally friendly due to their lower power consumption.
The Zamba2-2.7B model utilizes advanced techniques such as the interleaved shared attention scheme with LoRA projectors on shared MLP blocks, which enables it to handle complex tasks more efficiently. The model also features a lower generation latency, delivering 1.29 times lower latency compared to Microsoft’s state-of-the-art SLM, Phi3. This makes it a suitable option for deployment on devices with limited memory resources and for applications that require seamless and uninterrupted communication.
Zyphra trained Zamba2-2.7B on about 3T tokens from a dataset that included Zyda and open-access pre-training datasets. This dataset was then refined using 100 billion of the best tokens after being aggressively filtered and deduplicated to ensure quality.
Zamba2-2.7B is released under an open source license, allowing researchers, developer and companies to leverage its capabilities. Zyphra made the model available on Huggingface along with a pure-pytorch implementation.
In addition to Zamba2-2.7B, Zyphra has ealier released an open-source 7B SSM-hybrid foundation model called Zamba, which is designed to bring AI to more devices while requiring a lower inference cost. This model is built using Zyphra’s Mamba blocks and a global shared attention layer and it outperforms fellow open-source models LLaMA 1, LLaMA 2 7B and OLMo-7B across a wide range of standard benchmarks.
The launch of Zyphra’s Zamba2-2.7B and the 7B SSM-hybrid model underscores the growing prominence of SLMs in the AI landscape. SLMs are emerging as vital tools, offering a balance between performance and efficiency, making advanced AI capabilities more accessible and sustainable. These models not only reduce computational costs and energy consumption but also provide specialized, high-accuracy solutions for targeted applications.
Zyphra’s advancements highlight the potential of SLMs to transform industries by enabling on-device AI that enhances privacy while lowering barriers to entry for businesses and researchers.
