As Chief Scientist (Neural Networks) at Databricks, I lead our analysis workforce towards the objective of giving everybody the power to construct and fine-tune AI fashions with their very own information. In 2020, I used to be a part of a small group of machine studying teachers and trade veterans that based MosaicML. We’ve got all the time been dedicated to supporting open scientific inquiry, each by sharing our information and offering instruments to the group. Since becoming a member of Databricks, which shares related educational roots, we now have solely deepened that dedication.
With that spirit in thoughts, we now have been collaborating with scientists from the nonprofit Allen Institute for AI (AI2) on every part from technical knowledge-sharing to at this time’s large announcement: OLMo. For my part, AI2 is among the finest NLP labs on the earth, much more so as a result of they conduct their cutting-edge analysis with the unrestrained creativity, dedication to integrity, and sources of a non-profit. We’ve discovered frequent floor in a perception in openness, a ardour for doing rigorous science, and a love of constructing artifacts that we put into the arms of the group.
In the present day AI2 is releasing OLMo 7B, an open supply, state-of-the-art giant language mannequin. Databricks is proud to have supported their work: OLMo (quick for Open-source Giant Language Mannequin) was educated utilizing our Mosaic AI Mannequin Coaching Platform. The AI2 workforce can also be sharing the pre-training information and coaching code used to develop this mannequin (which is a by-product of the MosaicML LLM Foundry).
We’re thrilled to have performed a component within the success of the OLMo undertaking, however I wish to give credit score the place credit score is due. We shared our instruments, however they did the exhausting work of constructing the fashions. Pete Walsh, Senior Software program Engineer at AI2, stated, “Mosaic was a game-changer for growing OLMo. Their platform allowed us to effortlessly scale up coaching and ablations when wanted, whereas their command-line interface lets us iterate shortly by launching multi-node jobs proper from our laptops.” AI2’s seamless expertise utilizing our coaching platform validated the work we’ve carried out to make constructing and fine-tuning giant fashions as easy as potential. To be taught extra in regards to the OLMo 7B mannequin and its variants, take a look at AI2’s weblog submit or the mannequin card on Hugging Face.
