given paper:
abstract: Weintroduceanewlarge-scaleNLIbenchmarkdataset,collectedviaaniterative,adversarialhuman-and-model-in-the-loopprocedure.Weshowthattrainingmodelsonthis newdatasetleadstostate-of-the-artperformanceonavarietyofpopularNLIbenchmarks,whileposingamoredifficultchallenge withitsnewtestset.Ouranalysisshedslight ontheshortcomingsofcurrentstate-of-theartmodels,andshowsthatnon-expertannotatorsaresuccessfulatfindingtheirweaknesses. Thedatacollectionmethodcanbeappliedin anever-endinglearningscenario,becominga movingtargetforNLU,ratherthanastatic benchmarkthatwillquicklysaturate.

Introduction: ProgressinAIhasbeendrivenby,amongother things,thedevelopmentofchallenginglarge-scale benchmarkslikeImageNet(Russakovskyetal., 2015)incomputervision,andSNLI(NAME_1 etal.,2015),SQuAD(Rajpurkaretal.,2016),and othersinnaturallanguageprocessing(NLP).Recently,fornaturallanguageunderstanding(NLU) inparticular,thefocushasshiftedtocombined benchmarkslikeSentEval(NAME_2, 2018)andGLUE(Wangetal.,2018),whichtrack modelperformanceonmultipletasksandprovide aunifiedplatformforanalysis. WiththerapidpaceofadvancementinAI,however,NLUbenchmarksstruggletokeepupwith modelimprovement.Whereasittookaround15 yearstoachieve“near-humanperformance”on MNIST(LeCunetal.,1998;Cires¸anetal.,2012; Wanetal.,2013)andapproximately7yearsto surpasshumansonImageNet(Dengetal.,2009; Russakovskyetal.,2015;Heetal.,2016),the GLUEbenchmarkdidnotlastaslongaswewould havehopedaftertheadventofBERT(Devlinetal.,2018),andrapidlyhadtobeextendedintoSuperGLUE(Wangetal.,2019).Thisraisesanimportant question:Canwecollectalargebenchmarkdataset thatcanlastlonger? Thespeedwithwhichbenchmarksbecomeobsoleteraisesanotherimportantquestion:arecurrentNLUmodelsgenuinelyasgoodastheirhigh performanceonbenchmarkssuggests?Agrowingbodyofevidenceshowsthatstate-of-the-art modelslearntoexploitspuriousstatisticalpatterns indatasets(Gururanganetal.,2018;Poliaketal., 2018;Tsuchiya,2018;Glockneretal.,2018;Geva etal.,2019;McCoyetal.,2019),insteadoflearningmeaningintheflexibleandgeneralizableway thathumansdo.Giventhis,humanannotators—be theyseasonedNLPresearchersornon-experts— mighteasilybeabletoconstructexamplesthat exposemodelbrittleness.Weproposeaniterative,adversarialhuman-andmodel-in-the-loopsolutionforNLUdatasetcollectionthataddressesbothbenchmarklongevityand robustnessissues.Inthefirststage,humanannotatorsdeviseexamplesthatourcurrentbestmodelscannotdeterminethecorrectlabelfor.These resultinghardexamples—whichshouldexposeadditionalmodelweaknesses—canbeaddedtothe trainingsetandusedtotrain Provide exactly two different responses that are separated with 6 asterisk symbols: ******.
user