Orca: How Microsoft’s 13B Model Challenges the Dominance of Large Foundation Models through Reasoning Imitation

For a long time, the prevailing narrative in the AI community was that open-source models could only mimic the 'style' of large foundation models (LFMs) like GPT-4 without capturing their core reasoning capabilities. This phenomenon, often called the 'False Promise of Imitation,' suggested that smaller models were merely becoming better at sounding smart rather than being smart. However, the release of Orca, a 13 billion parameter model by Microsoft, has fundamentally challenged this assumption. Unlike its predecessors such as Vicuna or Alpaca, Orca was designed to internalize the logic and thought processes of its teachers.

While models like Vicuna were fine-tuned on simple query-response pairs, Orca was exposed to the internal reasoning of its larger counterparts. This shift from 'what' the model says to 'how' the model thinks represents a tectonic shift in AI training methodologies. Orca proves that parameter count is not the only metric for intelligence; the quality and depth of the training data are equally, if not more, critical for achieving high-level performance.

The developers of Orca identified that previous open-source models lacked rigorous evaluation, leading to an overestimation of their true capabilities. By focusing on complex zero-shot reasoning benchmarks, the Microsoft team showed that Orca could outperform its peers by over 100% in tasks requiring logical deduction and causal judgment. This is particularly impressive given that Orca is roughly 1% to 2% the size of the estimated parameter count of GPT-4.

One of the most innovative aspects of Orca's development is the use of Progressive Learning. The researchers found that jumping straight from a base model to learning from GPT-4 was too difficult for a 13B parameter architecture—much like a primary school student trying to attend a PhD seminar. To solve this, they introduced an intermediate teacher: GPT-3.5 (ChatGPT). This 'teaching assistant' provided a bridge, allowing Orca to master foundational concepts before tackling the more complex logic of GPT-4.

By leveraging system instructions like 'think step-by-step' or 'explain like I am five,' the researchers forced the teacher models to generate detailed, logical explanations. This resulted in a dataset of 5 million examples from GPT-3.5 and 1 million examples from GPT-4. This volume of data is an order of magnitude larger than what was used for previous models, providing Orca with a much richer educational environment.