explain this to me in simple terms "48 hours ago Microsoft research released a paper describing a new language model for code called NAME_1 despite using few parameters and little training data by modern standards NAME_2 1 gets very strong results in this video I'll describe the key findings of the paper and explain why together with the Orca work from Microsoft This research could represent a step change in the cost of training large language models with significant implications for the machine learning ecosystem but first let's take a step back to get a handle on y1 we need to understand the Journey of large language models so far it helps to begin by looking at what the community has learned over the last few years perhaps the best known empirical result has been the so-called scaling laws finding that performance improves somewhat predictably as one scales up either the amount of compute or the size of the network in this work the authors explore a different axis the quality of the data they draw inspiration from the recent tiny stories research on small-scale synthetic data that highlighted the importance of data quality the goal of this work is to show that high quality data can even improve the state of the art of large language models while dramatically reducing the data set size and training compute this is a result that can significantly reduce the environmental cost of llms as noted by the authors I think it may also have big implications for the ability of teams with fewer resources to build powerful models the headline results in this paper are reported on human eval which is a data set of handwritten problems curated by openai as part of their codex work and the mostly basic python programs Benchmark of entry-level python programs introduced by Google a few things jump out first fire one gets very strong results 50.6 percent on human eval the most widely reported Benchmark only wizard coder and gpt4 score higher in this table and 51 outperforms wizard coder on MBP the second is the scale while wizard coder has 16 billion parameters and a trillion tokens of training data NAME_2 1 is just 1.3 billion parameters and is trained on 7 billion tokens of course gpt4 is playing its cards close to its chest how are these NAME_2 1 results possible what is the magic ingredient the answer according to the authors is very much the data quality they say that roughly speaking we pre-train on textbook quality data both synthetically generated with GPT 3.5 and filtered from web sources and we fine-tune on textbook exercise like d Limit your response to 550 words.
user