LL\" aMmlein: Compact and Competitive German-Only Language Models from Scratch

J Pfister, J Wunderle, A Hotho - arXiv preprint arXiv:2411.11171, 2024

We create two German-only decoder models, LL\"aMmlein 120M and 1B, transparently from scratch and publish them, along with the training data, for the German NLP research community to use. The model training involved several key …


Llama (LLM)

… Data was mostly from CommonCrawl and C4. Llama-2-70B saw 2T tokens but Llama-3-70B saw 15T tokens. Llama-3-8B and Llama-3-70B took 1.3M and 6.4M GPU hours for pretraining. …


LLaMA Beyond English: An Empirical Study on Language Capability Transfer

J Zhao, Z Zhang, Q Zhang, T Gui, X Huang - arXiv preprint arXiv:2401.01055, 2024

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (eg LLaMA) are …


LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

T Zhu, X Qu, D Dong, J Ruan, J Tong, C He, Y Cheng - arXiv preprint arXiv …, 2024

… As to the Dev-to-Train differences in Figure 9b, we find HellaSwag and ARC-c share the most similar expert preferences with CommonCrawl and C4, and GSM-8K is similar to arXiv. This may provide some insights for continual pre-training to further …


LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

Y Lu, W Zhu, L Li, Y Qiao, F Yuan - arXiv preprint arXiv:2407.05975, 2024

Large Language Models~(LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we …


LLM and RAG-Based Question Answering Assistant for Enterprise Knowledge Management

G Şahin, K Varol, BK Pak - 2024 9th International Conference on Computer …, 2024

Large language models (LLM) have become integral to many natural language processing applications, particularly in the area of automatic question answering. In this study, a question answering system was developed to enable Adesso Turkiye …


LLM-Assistance for Quality Control of LLM Output

K Sandkuhl - Perspectives in Business Informatics Research: 23rd …

… GPT-3, the model behind ChatGPT, operates with 175 billion parameters and utilizes datasets that include nearly a trillion words sourced from various corpora, such as the Common Crawl, WebText2, internet-based books, and the English …


LLM-Based Edge Intelligence: A Comprehensive Survey on Architectures, Applications, Security and Trustworthiness

O Friha, MA Ferrag, B Kantarci, B Cakmak, A Ozgun… - IEEE Open Journal of the …, 2024

The integration of Large Language Models (LLMs) and Edge Intelligence (EI) introduces a groundbreaking paradigm for intelligent edge devices. With their capacity for human-like language processing and generation, LLMs empower edge …


LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models

M Ostendorff, PO Suarez, LF Lage, G Rehm

… All of the numbers are given by Common Crawl dump. … 2 show the number of documents by both date of the CommonCrawl dump and language, the percentage of documents removed at each filtering step, and the final document and token …


LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

A Aizawa, E Aramaki, B Chen, F Cheng, H Deguchi… - arXiv preprint arXiv …, 2024

… The statistics of the Japanese corpus from Common Crawl dumps are presented in Figure 2. We used the publicly available Common Crawl dumps from 2013 to the middle of 2023. We merged the Common Crawl dumps from 2013 to 2016 because …


LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

N Lee, T Wattanawong, S Kim, K Mangalam, S Shen… - arXiv preprint arXiv …, 2024

… We assume that we are given an LLM model M (eg, GPT-3.5 or LLaMA2-7B) that is pre-trained on some source dataset (eg, Common Crawl). The goal is to adapt M (hereon called the student model) to a new target domain by using a small seed dataset D …


LLM4Schema. org: Generating Schema. org Markups with Large Language Models

MH Dang, THT Pham, P Molli, H Skaf-Molli, A Gaignard

The integration of Schema. org markup in web pages has resulted in the creation of billions of RDF triples, yet approximately 75% of all web pages still lack these essential markups. Large Language Models (LLMs) offer a potential solution by …


LLMs achieve adult human performance on higher-order theory of mind tasks

W Street, JO Siy, G Keeling, A Baranes, B Barnett… - arXiv preprint arXiv …, 2024

This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (eg I think that you …


LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings

F Alam, SA Chowdhury, S Boughorbel, M Hasanain - … of the 18th Conference of the …, 2024

The recent breakthroughs in Artificial Intelligence (AI) can be attributed to the remarkable performance of Large Language Models (LLMs) across a spectrum of research areas (eg, machine translation, question-answering, automatic speech …


LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

BP Cipriano, P Alves - arXiv preprint arXiv:2403.06254, 2024

… are high-level descriptions of the training datasets (eg a filtered version of Common Crawl 14, English Wikipedia, among others) [3], it was not possible to perform this analysis for those models. 13https:github.com/camilaabrantes/CursoJava …


LLMs: A Comprehensive Survey of Applications, Challenges, Datasets, Models, Limitations, and Future Prospects

MU Hadi, Q Al-Tashi, R Qureshi, A Shah, A Muneer…

Within the vast expanse of computerized language processing, a revolutionary entity known as Large Language Models (LLMs) has emerged, wielding immense power in its capacity to comprehend intricate linguistic patterns and conjure coherent and …


LMaaS: Exploring Pricing Strategy of Large Model as a Service for Communication

P Wu, Q Liu, Y Dong, F Wang - arXiv preprint arXiv:2401.02675, 2024

The next generation of communication is envisioned to be intelligent communication, that can replace traditional symbolic communication, where highly condensed semantic information considering both source and channel will be extracted and …


LMCK: pre-trained language models enhanced with contextual knowledge for Vietnamese natural language inference

NLT Nguyen, KTK Phan, TV Huynh, KV Nguyen - Multimedia Tools and Applications, 2024

… trained with a cross-lingual masked language modeling objective on data in 100 languages, including Vietnamese from Common Crawl. … a combination of span masking and sentence shuffling objectives on a subset of 25 languages, including …


LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP

D Chen, F Shi, A Agarwal, J Myerston… - arXiv preprint arXiv …, 2024

… 2023), a pixel-based language model pre-trained on the Common Crawl dataset with a masked image modeling objective, to encode the … Among the aforementioned models, T5, ByT5, and PIXEL are pre-trained on large-scale text …


Long-input summarization using Large Language Models

… the colossal and cleaned version of common crawl CNN … T5 was pre-trained on the C4 (The Colossal and Cleaned version of Common Crawl) dataset. C4 contains text from 350M web pages; the dataset is 750 GB. A maximum sequence length of …


Long-Term Ad Memorability: Understanding & Generating Memorable Ads

SI Harini, S Singh, Y Kumar, A Bhattacharyya, V Baths…

Marketers spend billions of dollars on advertisements, but to what end? At purchase time, if customers cannot recognize the brand for which they saw an ad, the money spent on the ad is essentially wasted. Despite its importance in marketing, until now …


LongWanjuan: Towards Systematic Measurement for Long Text Quality

K Lv, X Liu, Q Guo, H Yan, C He, X Qiu, D Lin - arXiv preprint arXiv:2402.13583, 2024

… In the English data, the CommonCrawl domain predominates, accounting for over 50% of the data. Apart from a significant amount of aggregated texts in the CommonCrawl domain, the majority of data in other domains consists of holistic texts …


Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Q Zhang, T Du, H Huang, Y Wang, Y Wang - Forty-first International Conference on Machine …

In recent years, the rise of generative self-supervised learning (SSL) paradigms has exhibited impressive performance across visual, language, and multi-modal domains. While the varied designs of generative SSL objectives lead to distinct properties in …


Look Hear: Gaze Prediction for Speech-directed Human Attention

S Mondal, S Ahn, Z Yang, N Balasubramanian… - arXiv preprint arXiv …, 2024

For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is …


LoQT: Low-Rank Adapters for Quantized Pretraining

SB Loeschcke, M Toftrup, M Kastoryano, S Belongie… - The Thirty-eighth Annual …

Despite advances using low-rank adapters and quantization, pretraining of large models on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations …


LoRA-Enhanced Language Alignments for Robust Code-Mixed Text Representation

X Ran, Y Zhang, J Wang, D Xu, X Zhang - 2024 International Joint Conference on …, 2024

… • XLM-RoBERTa-Large [3] is a multilingual version of RoBERTa, which is pre-trained on a filtered Common Crawl dataset containing 100 languages. • XMFGM [31] applies adversarial training to construct some adversarial samples via adding …


LoRA-Guard: Parameter-Efficient Guardrail Adaptation for Content Moderation of Large Language Models

H Elesedy, PM Esperança, SV Oprea, M Ozay - arXiv preprint arXiv:2407.02987, 2024

Guardrails have emerged as an alternative to safety alignment for content moderation of large language models (LLMs). Existing model-based guardrails have not been designed for resource-constrained computational portable devices, such …


Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

Z Zhong, M Xia, D Chen, M Lewis - arXiv preprint arXiv:2405.03133, 2024

… We suspect that segment-level routing may be particularly effective at handling out-of-domain evaluation data (we assume that there is only a very small part of Python code in Commoncrawl). Our analysis in Section 5.4 shows that segment-level routing …


Low-Complexity and Secure Clustering-Based Similarity Detection for Private Files

Detection of the similarity between files is a requirement for many practical applications, such as copyright protection, file management, plagiarism detection, and detecting duplicate submissions of scientific articles to multiple journals or …


LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

Y Kim, H Ha, S Yang, S Lee, J Kim, C Park - arXiv preprint arXiv:2411.11289, 2024

… To validate the feasibility of the LP Data Pipeline with large-scale datasets, we conducted experiments using actual CommonCrawl dumps, focusing on key metrics such as processing time and estimated cost. The results demonstrate the pipeline’s …


LSHBloom: Memory-efficient, Extreme-scale Document Deduplication

A Khan, R Underwood, C Siebenschuh, Y Babuji… - arXiv preprint arXiv …, 2024

… 2020) (CCNET) is an automated preprocessing pipeline that aims to extract high-quality monolingual language datasets from the Common Crawl corpus of Web content. CCNet performs dataset sharding, deduplication, language identification, and …


M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

G Thakkar, S Hakimov, M Tadić - arXiv preprint arXiv:2404.01753, 2024

… 2020b) is a large multilingual language model trained on 2.5 TB of filtered Common Crawl data containing 100 languages. The model was trained with the Masked Language Modelling (MLM) objective, with 15% of the input words masked …


Machine learning and natural language processing to assess the emotional impact of influencers' mental health content on Instagram

N Merayo, A Ayuso-Lanchares, C González-Sanguino - PeerJ Computer Science, 2024

Background This study aims to examine, through artificial intelligence, specifically machine learning, the emotional impact generated by disclosures about mental health on social media. In contrast to previous research, which primarily focused on …


Machine Learning and Neural NLG

E Reiter - Natural Language Generation, 2024

This chapter gives an overview of machine learning and neural approaches to NLG, including language models. Because this area is changing very rapidly, the focus is on giving a high-level overview of basic concepts behind models and then …


Machine Learning and the Analysis of Culture

The focus of this chapter is on how machine learning (ML) impacts the analysis of culture in sociology. It shows how ML has greatly advanced the analysis of culture, with new tools enabling a massive and fine-grained extraction of information from …


Machine Translation that Peeks at the Reference

Abstract Machine translation with lexical constraints is a popular research topic, especially for terminology translation. Existing approaches for lexical control in MT are usually complex and not easily applicable to all existing MT toolkits. We propose …


Machine-Learning Phishing Detection Model Used in the E-Banking Environment

M Manala, J Jansen van Vuuren - IFIP International Conference on Human Choice …, 2024

… Rashid, Mahmood [20] proposed a machine-learning approach that uses Alexa and the Common Crawl archive dataset with 5000 URLs. … Geyik, Erensoy [21] used the PhishTank Alexa Common-Crawl datasets. The performance attained …


Machines Do See Color: A Guideline to Classify Different Forms of Racist Discourse in Large Corpora

DD Gordillo, J Timoneda, SV Vera - arXiv preprint arXiv:2401.09333, 2024

Current methods to identify and classify racist language in text rely on small-n qualitative approaches or large-n approaches focusing exclusively on overt forms of racist discourse. This article provides a step-by-step generalizable guideline to …


MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions

K Zhang, Y Luan, H Hu, K Lee, S Qiao, W Chen, Y Su… - arXiv preprint arXiv …, 2024

… We collect all images with the same URL from Common Crawl2 as a group of images from the same web page for potential pairing. Due to the inevitable noisy images introduced by simple grouping, we remove duplicated, low-resolution, and …


MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

O Ignat, X Xu, R Mihalcea - arXiv preprint arXiv:2404.12938, 2024

Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and …


MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness

S Zhou, H Shan, B Plank, R Litschko - arXiv preprint arXiv:2404.02570, 2024

This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness (STR), on Track C: Cross-lingual. The task aims to detect semantic relatedness of two sentences in a given target language without access to …


MAKG: A maritime accident knowledge graph for intelligent accident analysis and management

D Liu, L Cheng - Ocean Engineering, 2024

With the increasing frequency of human activities at sea, maritime accidents are occurring more often. Analyzing and mining maritime accident cases can help uncover the causal mechanisms behind these incidents, thereby enhancing …


Making Javascript Render Decisions to Optimize Security-Oriented Crawler Process

… They indicate that if their method had applied to the July 2015 Common Crawl dataset, a web-scale archival crawler would have discovered an additional 7.17 PB (5.12 times more) of information per year. However, rendering JavaScript and using client-side …


MambaByte: Token-free Selective State Space Model

J Wang, T Gangavarapu, JN Yan, AM Rush - arXiv preprint arXiv:2401.13660, 2024

Token-free language models learn directly from raw bytes and remove the bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences, and standard autoregressive Transformers scale poorly in such settings …


MAmmoTH2: Scaling Instructions from the Web

X Yue, T Zheng, G Zhang, W Chen - arXiv preprint arXiv:2405.03548, 2024

… We argue that the pre-training corpus (eg, Common Crawl) already contains a vast amount of high-quality instruction data for LLM reasoning. For example, the web corpus contains a large amount of educational materials in the form of instruction-following …


MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

G Zhang, S Qu, J Liu, C Zhang, C Lin, CL Yu, D Pan… - arXiv preprint arXiv …, 2024

… Inspired by [89], which adopts an iterative pipeline to facilitate the acquisition of large-scale, high-quality data from Common Crawl, we propose to select high-quality data for mathematics, scientific exam synthetic data, and wiki-based content in our Matrix. …


Mapping the media landscape: predicting factual reporting and political bias through web interactions

D Sánchez-Cortés, S Burdisso, E Villatoro-Tello… - International Conference of …, 2024

… In this work we used the graph G built in [8] consisting of 17K news sources obtained after processing 100M news articles from Common Crawl News. Following [3, 8] we report 5-fold cross-validation evaluation results on our MBFC datasets …


Mapping the Positive Self-Bias Embedded in Human Languages

… Here, we employed the fastText algorithm 62, pre-trained on the Common Crawl and Wikipedia corpora, for the analyses across Studies 1a-c. We opted for this model due to its leading performance, widespread acceptance, and multilingual …


MapPool-Bubbling up an extremely large corpus of maps for AI

R Schnürer - 2024 ICA Workshop on AI, Geovisualization, and …, 2024

MapPool is a dataset of 75 million potential maps and textual captions. It has been derived from CommonPool, a dataset consisting of 12 billion text-image pairs from the Internet. The images have been encoded by a vision transformer and classified …


Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement

L Ming, B Zeng, C Lyu, T Shi, Y Zhao, X Yang, Y Liu… - arXiv preprint arXiv …, 2024

… First of all, we extract the main contents from the raw Common Crawl (WARC) files. Meanwhile, we perform URL filter and language identification to avoid subsequent computationally expensive processing. The former targets fraudulent and/or adult …


Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking

Y Fung, R Zhao, J Doo, C Sun, H Ji - arXiv preprint arXiv:2402.09369, 2024

Pretrained large language models have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions …


MASTER THESIS Large-scale product classification for efficient matching in procurement systems

… Peeters, and Bizer, 2019 is compiled based on open data from the Common Crawl … 1https://commoncrawl.org/ … available Common Crawl and Wikipedia datasets. For our experiments, we have …


MASTERARBEIT/MASTER'S THESIS

Today, a huge amount of unstructured data from various sources is available. Natural language processing (NLP) techniques using such data are applied to several tasks such as named entity recognition (NER). This study is designed to …


MASTERARBEIT| MASTER'S THESIS

The exponential growth of the social media user base has led to an alarming rise in the use of offensive language and profanity, necessitating effective detection mechanisms. This thesis explores the challenges and advancements in automatic …


Mastering Transformers: The Journey from BERT to Large Language Models and Stable Diffusion

Explore transformer-based language models from BERT to GPT, delving into NLP and computer vision tasks, while tackling challenges effectively Key Features Understand the complexity of deep learning architecture and transformers …


MaTElDa: Multi-Table Error Detection

F Ahmadi, M Speckmann, MF Kuhlmann, Z Abedjan - 2025

… WDC Web Table Corpus: We randomly picked 100 Englishlanguage relational webtables out of the massive collection of over 233 million tables derived from the July 2015 version of the CommonCrawl [30] to measure the effectiveness of our …


Materials science in the era of large language models: a perspective

G Lei, R Docherty, SJ Cooper - arXiv preprint arXiv:2403.06949, 2024

Large Language Models (LLMs) have garnered considerable interest due to their impressive natural language capabilities, which in conjunction with various emergent properties make them versatile tools in workflows ranging from complex …


MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code

Z Lu, A Zhou, K Wang, H Ren, W Shi, J Pan, M Zhan… - arXiv preprint arXiv …, 2024

… We first use a fastText classifier to filter the Common Crawl corpus, resulting in the initial filtered math texts. Then, we annotate part of the filtered texts to train a new fastText classifier, and conduct a second filtering, resulting in the finer filtered math …


MathPile: A Billion-Token-Scale Pretraining Corpus for Math

Z Wang, X Li, R Xia, P Liu - The Thirty-eight Conference on Neural Information …

… , especially given the vast size of the Common Crawl corpus . We also experimented with more intricate … -CommonCrawl, totaling approximately 633 million tokens. We acknowledge the potential for more efficient methods to sift …


Matthew Coscia and

… URL: https://github.com/commoncrawl/ia-web-commons. [9] Image Details - Container Registry - Google Cloud Platform. 2022. …


Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining

S Feng, S Prabhumoye, K Kong, D Su, M Patwary… - arXiv preprint arXiv …, 2024

… 2024a) mentions that web crawl documents like Common Crawl (CC) form a large majority of their pretraining data, none of them share a recipe on how to mix different slices of CC. … The overall percentage of the Common Crawl Snapshots in …


McGill NLP Group Submission to the MRL 2024 Shared Task: Ensembling Enhances Effectiveness of Multilingual Small LMs

S Li, H Yu, J Ojo, D Adelani - Proceedings of the Fourth Workshop on Multilingual …, 2024

… mT5 is pre-trained on a massive multilingual dataset covering 101 languages from the Common Crawl corpus, which enables it to perform a wide range of natural language processing tasks. It operates using a text-to-text framework, where all …


MDSAA

In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

M Song, SH Sim, R Bhardwaj, HL Chieu, N Majumder… - arXiv preprint arXiv …, 2024

… To do this, we query Wikipedia and Common Crawl to retrieve the 100 most relevant documents. We filter seed question for which the retriever fails to retrieve relevant documents. Furthermore, we identify 5 documents that are equally effective …


Measuring and Improving the Energy Efficiency of Large Language Models Inference

MF Argerich, M Patiño-Martínez - IEEE Access, 2024

… 4) RedPajama [42]: another family of models based on Pythia with 7B and 3B parameters, fine-tuned on a dataset with the same name as the model, with over 100B text documents coming from 84 CommonCrawl snapshots. Similar to Dolly …


Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training

Y Mansour, R Heckel - arXiv preprint arXiv:2412.02857, 2024

… Thus all of the datasets we consider for LLM training are obtained by the dataset creators by extracting text from CommonCrawl data… CommonCrawl. We consider the CommonCrawl part, which was obtained by first downloading about a quarter of …


Measuring complex psychological and sociological constructs in large-scale text

A Herderich, J Lasser, S Aroyehun, D Garcia…

In recent years, there has been an increasing exchange between social and computational sciences. Methods from natural language processing enable social scientists to systematically process large amounts of text. Rich psycho-sociological …


Measuring Contextual Informativeness in Child-Directed Text

M Valentini, T Wright, A Marashian, J Weber… - arXiv preprint arXiv …, 2024

To address an important gap in creating children's stories for vocabulary enrichment, we investigate the automatic evaluation of how well stories convey the semantics of target vocabulary words, a task with substantial implications for generating …


Measuring individual semantic networks: A simulation study

S Aeschbach, R Mata, DU Wulff - arXiv preprint arXiv:2410.18326, 2024

Accurately capturing individual differences in semantic networks is fundamental to advancing our mechanistic understanding of semantic memory. Past empirical attempts to construct individual-level semantic networks from behavioral paradigms …


Medical Concept Normalization in a Low-Resource Setting

T Patzelt - arXiv preprint arXiv:2409.14579, 2024

… The multilingual RoBERTa model, RoBERTaXLM, by conneau_unsupervised_2020 was trained on more than two terabytes of filtered Common Crawl (wenzek_ccnet_2020) data. The Common Crawl dataset is a …


Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

I García-Ferrero, R Agerri, AA Salazar, E Cabrio… - arXiv preprint arXiv …, 2024

… mT5 was trained using mC4, a 1 Trillion token Common Crawl-based dataset covering 101 languages. The pre-training is based on a masked language modeling “spancorruption” objective, where consecutive spans of input tokens are replaced …


MedQAS: A Medical Question Answering System Based on Finetuning Large Language Models

BT Duong, TH Le - International Conference on Future Data and Security …, 2024

… Specifically, LLaMA was trained on 1.0 trillion tokens from publicly accessible data sources such as CommonCrawl and arXiv [6]. They fine-tuned the LLaMA model using conversations from HealthCareMagic-100k [15] following the Stanford …


Meeting the challenge: A benchmark corpus for automated Urdu meeting summarization

B Sadia, F Adeeba, S Shams, K Javed - Information Processing & Management, 2024

Meeting summarization has become crucial as the world is gradually shifting towards remote work. Nowadays, automation of meeting summary generation is really needed in order to minimize the time and effort. The surge in online meetings …


MELTing point: Mobile Evaluation of Language Transformers

S Laskaridis, K Kateveas, L Minto, H Haddadi - arXiv preprint arXiv:2403.12844, 2024

Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with ``sparks of intelligence''. However, their runtime requirements have prevented them from being …


Memorization and Privacy Risks in Domain-Specific Large Language Models

X Yang, Z Wen, W Qu, Z Chen, Z Xiang, B Chen, H Yao - ICLR 2024 Workshop on Reliable …

… Common Crawl3 is a nonprofit organization that provides a large and open web crawl data repository for public use. It collects web pages from the internet every month and stores them on Amazon Web Services. Common Crawl’s … C4 cleanses …


Mentions of prejudice in news media–an international comparison

D Rozado - Journal of Computational Social Science, 2024

Previous research has identified a post-2010 sharp increase of terms used to denounce prejudice (ie racism, sexism, homophobia, Islamophobia, anti-Semitism, etc.) in US and UK news media content. Here, we extend previous analysis to an …


MeSH2Matrix: Combining MeSH keywords and machine learning for biomedical relation classification based on PubMed

H Turki, BFP Dossou, CC Emezue, A Toluwase…

Biomedical relation classification has been significantly improved by the application of advanced machine learning techniques on the raw texts of scholarly publications. Despite this improvement, the reliance on large chunks of raw text makes these …


MetaphorShare: A Dynamic Collaborative Repository of Open Metaphor Datasets

J Boisson, A Mehmood, J Camacho-Collados - arXiv preprint arXiv:2411.18260, 2024

The metaphor studies community has developed numerous valuable labelled corpora in various languages over the years. Many of these resources are not only unknown to the NLP community, but are also often not easily shared among the …


Methodological and practical challenges of doing transnational studies

N Brügger - The Routledge Companion to Transnational Web …, 2024

… collections such as the Internet Archive or Common Crawl, or holdings of other national web … establishment of the web archive, whereas using the Common Crawl is uncharted territory. … for both the Internet Archive and the Common Crawl …


Methods and Applications for Probing Deep Neural Networks

In recent years, the impressive abilities shown by deep neural network (DNN)-based systems have led to the curiosity towards the intrinsic mechanisms. The query towards these mechanisms has led to an emerging field, model interpretation. In this …


Methods Futures Report–Sea, Sky, and Land: Engaging in Solidarity in Endangered Ecologies–4S Conference Nov 2023

This is a report of the 4S 2023 conference. The 4S (Society for the Social Studies of Science) conference 2023 was a large, disciplinarily diverse gathering with over 400 sessions spread over four days. Navigating a particular route through the conference …


Methods to Assess the UK Government's Current Role as a Data Provider for AI

N Majithia, E Simperl - arXiv preprint arXiv:2412.09632, 2024

… Each website was present in CommonCrawl datasets before April 2024, therefore sitting in the training window for up-to-date models like Llama 3.1 [13]. The plaintext of each website was requested from the CommonCrawl index server, cleaned …


MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

AH Kargaran, A Modarressi, N Nikeghbal, J Diesner… - arXiv preprint arXiv …, 2024

English-centric large language models (LLMs) often show strong multilingual capabilities. However, the multilingual performance of these models remains unclear and is not thoroughly evaluated for many languages. Most benchmarks for …


mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval

X Zhang, Y Zhang, D Long, W Xie, Z Dai, J Tang, H Lin… - arXiv preprint arXiv …, 2024

We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained …


mHumanEval--A Multilingual Benchmark to Evaluate Large Language Models for Code Generation

N Raihan, A Anastasopoulos, M Zampieri - arXiv preprint arXiv:2410.15037, 2024

Recent advancements in large language models (LLMs) have significantly enhanced code generation from natural language prompts. The HumanEval Benchmark, developed by OpenAI, remains the most widely used code generation …


Milestones in Bengali Sentiment Analysis leveraging Transformer-models: Fundamentals, Challenges and Future Directions

S Sengupta, S Ghosh, P Mitra, TI Tamiti - arXiv preprint arXiv:2401.07847, 2024

Sentiment Analysis (SA) refers to the task of associating a view polarity (usually, positive, negative, or neutral; or even fine-grained such as slightly angry, sad, etc.) to a given text, essentially breaking it down to a supervised (since we have the view …


MIND: Math Informed syNthetic Dialogues for Pretraining LLMs

SN Akter, S Prabhumoye, J Kamalu, S Satheesh… - arXiv preprint arXiv …, 2024

… OpenWebMath contains 14.7B tokens of mathematical web pages filtered from CommonCrawl based on mathematical strings, … 2016) classifier to further extract mathematical documents from CommonCrawl. They cluster the extracted documents …


MindLLM: Lightweight large language model pre-training, evaluation and domain application

Y Yang, H Sun, J Li, R Liu, Y Li, Y Liu, Y Gao, H Huang - AI Open, 2024

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing …


Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance

Z Gao, Z Chen, E Cui, Y Ren, W Wang, J Zhu, H Tian… - arXiv preprint arXiv …, 2024

Multimodal large language models (MLLMs) have demonstrated impressive performance in vision-language tasks across a broad spectrum of domains. However, the large model scale and associated high computational costs pose significant …


MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

S Hu, Y Tu, X Han, C He, G Cui, X Long, Z Zheng… - arXiv preprint arXiv …, 2024

The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This …


MINNESOTA JOURNAL OF LAW, SCIENCE & TECHNOLOGY

Z Niesel, J Villasenor, J Perl, DE Ho, A Vallebueno

In administrative law, there is perhaps no more misunderstood phrase than the" arbitrary and capricious" standard of judicial review. American administrative law's cornerstone is the judicial review of agency decision-making-a review which has …


MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens

A Awadalla, L Xue, O Lo, M Shu, H Lee, EK Guha… - arXiv preprint arXiv …, 2024

… We follow OBELICS’s method for extracting interleaved multimodal documents from CommonCrawl WARC files by parsing each WARC entry’s DOM tree. While OBELICS only processed documents from February 2020 to February 2023 …


Misinformation Resilient Search Rankings with Webgraph-based Interventions

P Carragher, EM Williams, KM Carley - arXiv preprint arXiv:2404.08869, 2024

… To this end, we investigate the distribution of changes in CommonCrawl PageRank as a result of our interventions for all domains in the webgraph, not just news domains. We then take the domains whose PR score changes significantly ( 50%) …


Mission Critical--Satellite Data is a Distinct Modality in Machine Learning

E Rolf, K Klemmer, C Robinson, H Kerner - arXiv preprint arXiv:2402.01444, 2024

Satellite data has the potential to inspire a seismic shift for machine learning -- one in which we rethink existing practices designed for traditional data modalities. As machine learning for satellite data (SatML) gains traction for its real-world impact …


Mitigating Catastrophic Forgetting in Language Transfer via Model Merging

A Alexandrov, V Raychev, MN Müller, C Zhang… - arXiv preprint arXiv …, 2024

As open-weight large language models (LLMs) achieve ever more impressive performances across a wide range of tasks in English, practitioners aim to adapt these models to different languages. However, such language adaptation is often …


Mitigating Downstream Model Risks via Model Provenance

K Wang, AN Iranzad, S Schaffter, D Precup… - arXiv preprint arXiv …, 2024

Research and industry are rapidly advancing the innovation and adoption of foundation model-based systems, yet the tools for managing these models have not kept pace. Understanding the provenance and lineage of models is critical for …


Mitigating the Impact of Adversarial Fine-tuning on Large Language Models through Perturbation Attenuation

Z Jejesi, A McBride, F Kovacic, L O'Connor - 2024

The increasing complexity and scale of machine learning models have rendered them vulnerable to adversarial manipulation, particularly during fine-tuning processes where subtle perturbations can lead to significant degradation in model …


Mitigating the negative impact of over-association for conversational query production

A Wang, L Song, Z Min, G Xu, X Wang, J Yao, J Su - Information Processing & …, 2025

Conversational query generation aims at producing search queries from dialogue histories, which are then used to retrieve relevant knowledge from a search engine to help knowledge-based dialogue systems. Trained to maximize the likelihood of …


Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data

S Zeng, J Zhang, P He, J Ren, T Zheng, H Lu, H Xu… - arXiv preprint arXiv …, 2024

… 2021), we randomly select segments from the Common Crawl dataset to function as the information component. However, the randomness of the input may affect the command component. To mitigate this issue, we limit the maximum length of the …


MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures

J Ni, Y Song, D Ghosal, B Li, DJ Zhang, X Yue, F Xue… - arXiv preprint arXiv …, 2024

… In this phase, we detected around 20k queries using Vicuna 33B over a subset of Common Crawl. We then used GPT-4 to more accurately … 98%) on our benchmarks and detected 2M user queries from the Common Crawl subset. Finally …


MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures

J Ni, F Xue, X Yue, Y Deng, M Shah, K Jain, G Neubig… - arXiv preprint arXiv …, 2024

… In the detection phase, we train open-source LLMs on self-collected data to detect queries in Common Crawl splits. During filtering, we utilize GPT-4 Turbo to exclude non-query sentences. In classification, we categorize the filtered queries by input …


Mixture of Hidden-Dimensions Transformer

Y Chen, J Shang, Z Zhang, J Sheng, T Liu, S Wang… - arXiv preprint arXiv …, 2024

Transformer models encounter challenges in scaling hidden dimensions efficiently, as uniformly increasing them inflates computational and memory costs while failing to emphasize the most relevant features for each token. For further understanding …


ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science

R Wolfe, A Hiniker, B Howe - arXiv preprint arXiv:2408.01966, 2024

This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in …


MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

B Xue, H Wang, R Wang, S Wang, Z Wang, Y Du… - arXiv preprint arXiv …, 2024

… 2024) which determines the resource levels for each language by utilizing the data ratio from the CommonCrawl corpus 7, we select three languages (Chinese, Japanese, and French) in high-resource category whose data ratio exceeds 1 …


MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models

G Son, D Yoon, J Suk, J Aula-Blasco, M Aslan, VT Kim… - arXiv preprint arXiv …, 2024

Large language models (LLMs) are commonly used as evaluators in tasks (eg, reward modeling, LLM-as-a-judge), where they act as proxies for human preferences or judgments. This leads to the need for meta-evaluation: evaluating the …


MMRA: A Benchmark for Multi-granularity Multi-image Relational Association

S Wu, K Zhu, Y Bai, Y Liang, Y Li, H Wu, J Liu, R Liu… - arXiv preprint arXiv …, 2024

Given the remarkable success that large visual language models (LVLMs) have achieved in image perception tasks, the endeavor to make LVMLs perceive the world like humans is drawing increasing attention. Current multi-modal benchmarks …


Mobile Generative AI: Opportunities and Challenges

Y Zhang, J Zhang, S Yue, W Lu, J Ren, X Shen - IEEE Wireless Communications, 2024

Recently, generative artificial intelligence (GenAI) has gained significant interest on a global scale, particularly with the explosion of some killer GenAl applications, like ChatGPT. However, due to the excessively large sizes of generative models, most …


MobileVLM: A Fast, Reproducible and Strong Vision Language Assistant for Mobile Devices

X Chu, L Qiao, X Lin, S Xu, Y Yang, Y Hu, F Wei… - arXiv preprint arXiv …, 2023

We present MobileVLM, a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It is an amalgamation of a myriad of architectural designs and techniques that are mobile-oriented, which comprises a set of language …


Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation

Ensuring the verifiability of model answers is a fundamental challenge for retrieval-augmented generation (RAG) in the question answering (QA) domain. Recently, self-citation prompting was proposed to make large language models (LLMs) generate citations …


Modeling Caption Diversity in Contrastive Vision-Language Pretraining

S Lavoie, P Kirichenko, M Ibrahim, M Assran… - arXiv preprint arXiv …, 2024

There are a thousand ways to caption an image. Contrastive Language Pretraining (CLIP) on the other hand, works by mapping an image and its caption to a single vector -- limiting how well CLIP-like models can represent the diverse ways to describe an …


Modeling Nonnative Sentence Processing with L2 Language Models

T Aoyama, N Schneider - Proceedings of the 2024 Conference on Empirical …, 2024

We study LMs pretrained sequentially on two languages (“L2LMs”) for modeling nonnative sentence processing. In particular, we pretrain GPT2 on 6 different first languages (L1s), followed by English as the second language (L2). We examine the …


Modeling the Sacred: Considerations when Using Considerations when Using Religious Texts in Natural Language Processing

B Hutchinson - arXiv preprint arXiv:2404.14740, 2024

This position paper concerns the use of religious texts in Natural Language Processing (NLP), which is of special interest to the Ethics of NLP. Religious texts are expressions of culturally important values, and machine learned models have a …


Modelling cross-lingual transfer for semantic parsing

This thesis has been submitted in fulfilment of the requirements for a postgraduate degree (eg PhD, MPhil, DClinPsychol) at t Page 1 This thesis has been submitted in fulfilment of the requirements for a postgraduate degree (eg PhD, MPhil …


Modelling Multimodal Integration in Human Concept Processing with Vision-and-Language Models

A Bavaresco, MH Kloots, S Pezzelle, R Fernández - arXiv preprint arXiv:2407.17914, 2024

Representations from deep neural networks (DNNs) have proven remarkably predictive of neural activity involved in both visual and linguistic processing. Despite these successes, most studies to date concern unimodal DNNs, encoding either …


Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal …

S Aggarwal, A Pandey, DK Vishwakarma - arXiv preprint arXiv:2408.02595, 2024

… With over 2 terabytes of cleaned CommonCrawl data [45], the masked language model [44] has already been pre-trained on over a hundred languages, including Hindi. We chose this particular variant of [46] for our investigation since the majority …


Models for analyzing the complexity of English words in the text on the scale from A1 to C2

М Бєліков, Т Ліхоузова, Ю Олійник - Адаптивні системи автоматичного управління, 2024

… This model was trained on data from Common Crawl (web archives with data from web pages on various topics), OntoNotes 5 (corpus of annotated texts together with annotations of parts of speech, word dependencies and named entities) and …


MoH: Multi-Head Attention as Mixture-of-Head Attention

P Jin, B Zhu, L Yuan, S Yan - arXiv preprint arXiv:2410.11842, 2024

In this work, we upgrade the multi-head attention mechanism, the core of the Transformer model, to improve efficiency while maintaining or surpassing the previous accuracy level. We show that multi-head attention can be expressed in the …


MojoBench: Language Modeling and Benchmarks for Mojo

N Raihan, J Santos, M Zampieri - arXiv preprint arXiv:2410.17736, 2024

The recently introduced Mojo programming language (PL) by Modular, has received significant attention in the scientific community due to its claimed significant speed boost over Python. Despite advancements in code Large Language Models (LLMs) …


Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

G Luo, X Yang, W Dou, Z Wang, J Dai, Y Qiao, X Zhu - arXiv preprint arXiv …, 2024

The rapid advancement of Large Language Models (LLMs) has led to an influx of efforts to extend their capabilities to multimodal tasks. Among them, growing attention has been focused on monolithic Multimodal Large Language Models (MLLMs) …


Mono-lingual text reuse detection for the Urdu language at lexical level

A Noreen, I Muneer, RMA Nawab - Engineering Applications of Artificial Intelligence, 2024

Text reuse is the process of creating new texts from pre-existing ones. In recent years, Urdu Text Reuse Detection (U-TRD) has garnered the attention of researchers due to the ready availability of digital text all over the internet, which can be copied or …


mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus

M Futeral, A Zebaze, PO Suarez, J Abadji, R Lacroix… - arXiv preprint arXiv …, 2024

… This motivated us to collect and release the first large-scale multilingual and multimodal document dataset derived from Common Crawl.Our dataset, multimodal OSCAR (mOSCAR), follows the OSCAR initiative [Ortiz Suárez et al., 2019, Abadji et …


Moshi: a speech-text foundation model for real-time dialogue

A Défossez, L Mazaré, M Orsini, A Royer, P Pérez…

… Our training dataset is made of a mix of high-quality data sources and filtered web data from CommonCrawl. More specifically, 12.5% of our … The remaining 87.5% of our dataset is from CommonCrawl, and was filtered with the pipeline described in …


MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Z Zong, B Ma, D Shen, G Song, H Shao, D Jiang, H Li… - arXiv preprint arXiv …, 2024

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in …


MPDS: A Movie Posters Dataset for Image Generation with Diffusion Model

M Xu, T Zhang, F Wang, Y Lei, X Liu, Z Cui - arXiv preprint arXiv:2410.16840, 2024

… This dataset is sourced from Common Crawl (Crawl 2024), a non-profit organization that regularly releases HTML data from billions of … Some Image-Text Datasets, such as LAION-400M, obtain HTML-like data from Common Crawl …


mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

P Lin, C Hu, Z Zhang, AFT Martins, H Schütze - Findings of the Association for …, 2024

Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to …


mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

A Hu, H Xu, J Ye, M Yan, L Zhang, B Zhang, C Li… - arXiv preprint arXiv …, 2024

Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition …


MQM-Chat: Multidimensional Quality Metrics for Chat Translation

Y Li, J Suzuki, M Morishita, K Abe, K Inui - arXiv preprint arXiv:2408.16390, 2024

The complexities of chats pose significant challenges for machine translation models. Recognizing the need for a precise evaluation metric to address the issues of chat translation, this study introduces Multidimensional Quality Metrics for Chat …


MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels

Q Chen, X Geng, C Rosset, C Buractaon, J Lu, T Shen… - Companion Proceedings of …, 2024

Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked …


Mult-IT Multiple Choice Questions on Multiple Topics in Italian: A CALAMITA Challenge

M Rinaldi, J Gili, M Francis, M Goffetti, V Patti, M Nissim - Proceedings of the 10th …, 2024

… For example, the Common Crawl dataset2, often used as a base for more refined datasets to be employed in the pre-training of LLMs, is composed of 45% English content, while the data for languages such as Spanish, French, Italian, and Chinese …


Multi Class Depression Detection Through Tweets using Artificial Intelligence

MO Nusrat, W Shahzad, SA Jamal - arXiv preprint arXiv:2404.13104, 2024

… Typically, GloVe embeddings were pretrained on extensive corpora like Wikipedia or Common Crawl and fine-tuned for specific tasks. 300 LSTM units were used with a dropout rate of 0.4. For depression classification, the dense layer was …


Multi-Agent Collaborative Data Selection for Efficient LLM Pretraining

T Bai, L Yang, ZH Wong, J Peng, X Zhuang, C Zhang… - arXiv preprint arXiv …, 2024

… For the topic agent, due to the absence of a suitable pretrained model for topic classification and labeling, we designed a classification schema using 1.44 billion documents collected by the Common Crawl project (Project, 2007) and fine-tuned a …


Multi-class hate speech detection in the Norwegian language using FAST-RNN and multilingual fine-tuned transformers

E Hashmi, SY Yayilgan - Complex & Intelligent Systems, 2024

… FastText, a word representation tool developed by Facebook’s research division, offers both unsupervised and supervised modes and features a comprehensive database of 2 million words from Common Crawl, each represented by 300 …


Multi-Head Mixture-of-Experts

X Wu, S Huang, W Wang, F Wei - arXiv preprint arXiv:2404.15045, 2024

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for …


Multi-modal Comparative Analysis on Execution of Phishing Detection using Artificial Intelligence

DJ Dsouza, AP Rodrigues, R Fernandes - IEEE Access, 2024

Phishing is the process of deceiving or stealing private or confidential information through illicit means. This could lead to financial loss, loss of reputation, and identity theft. Hence, identifying and preventing the use of such phishing sites becomes …


Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation

J Choi, J Yun, K Jin, YB Kim - arXiv preprint arXiv:2404.09682, 2024

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Multi-Reference Benchmarks for Russian Grammatical Error Correction

FP Gomez, A Rozovskaya - Proceedings of the 18th Conference of the European …, 2024

This paper presents multi-reference benchmarks for the Grammatical Error Correction (GEC) of Russian, based on two existing single-reference datasets, for a total of 7,444 learner sentences from a variety of first language backgrounds. Each …


Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation

M Guan, Y Wang, G Ma, J Liu, M Sun - arXiv preprint arXiv:2405.05672, 2024

Sign language serves as a non-vocal means of communication, transmitting information and significance through gestures, facial expressions, and bodily movements. The majority of current approaches for sign language recognition (SLR) …


Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings

I Mohr, M Krimmel, S Sturua, MK Akram, A Koukounas… - arXiv preprint arXiv …, 2024

… Seeking further diversity and volume, we utilize Common Crawl data to create two types of pairs: one from web page titles and their contents, and another by mining questionanswer pairs from FAQ and support-related pages. Additionally, we pair …


Multidimensional Text Feature Analysis: Unveiling the Veil of Automatically Generated Text

… Given the widespread use of transformer-based pre-trained models in various downstream tasks and the availability of multilingual datasets in official task data, XLM-RoBERTa[19] is a multilingual version of Roberta that has been pre-trained on …


Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture

J Song, Y Huang, Z Zhou, L Ma - arXiv preprint arXiv:2407.07342, 2024

… We consider most state-of-the-art LLMs to have trained on these 55 source languages, as these languages are enclosed in the CommonCrawl corpus. All multilingual translations are conducted using Google Translate API3. All codes for …


Multilingual Detection of Cyberbullying in Mixed Urdu, Roman Urdu, and English Social Media Conversations

… The pre-training data is carefully selected from Common Crawl, a vast collection of web pages, to ensure that it’s both high-quality and diverse. This diverse training data allows m-BERT to learn language-agnostic representations, making it easier to …


Multilingual Diversity Improves Vision-Language Representations

T Nguyen, M Wallingford, S Santy, WC Ma, S Oh… - arXiv preprint arXiv …, 2024

… Data We experiment with the medium pool of the DataComp benchmark [14], which consists of 128M image-text pairs randomly sampled from Common Crawl dumps between 2014 and 2022, and deduplicated. Unlike other heavily filtered …


Multilingual E5 Text Embeddings: A Technical Report

L Wang, N Yang, X Huang, L Yang, R Majumder, F Wei - arXiv preprint arXiv …, 2024

This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a …


Multilingual Fake News Detection: A Study on Various Models and Training

R Chalehchaleh, R Farahbakhsh, N Crespi - … Systems and Applications: Proceedings of the …

… XLM-RoBERTa model is pre-trained on 2.5 TB of filtered CommonCrawl data containing 100 languages it was introduced in [8]. LASER (Language-Agnostic SEntence Representations)[3] is trained to capture the semantic meaning of sentences …


Multilingual Hallucination Gaps in Large Language Models

C Chataigner, A Taïk, G Farnadi - arXiv preprint arXiv:2410.18270, 2024

Large language models (LLMs) are increasingly used as alternatives to traditional search engines given their capacity to generate text that resembles human language. However, this shift is concerning, as LLMs often generate hallucinations, misleading …


Multilingual Hate Speech Detection: Comparison of Transfer Learning Methods to Classify German, Italian, and Spanish Posts

J Fillies, MP Hoffmann, A Paschke - 2023 IEEE International Conference on Big Data …, 2023

… This datasetincludes Wikipedia dumps for all supported languages anda substantial 2.5 terabytes of Common Crawl Data collectedfrom the Internet. The ”RoBERTa” component in the model’sname indicates the adherence to the training procedure …


Multilingual Language Models: Analysis and Algorithms

While large language models (LLMs) continue to grow in scale and gain new zero-shot capabilities, their performance for languages beyond English increasingly lags behind. This gap is due to the curse of multilinguality, where multilingual language …


Multilingual Large Language Models and Curse of Multilinguality

D Gurgurov, T Bäumel, T Anikina - arXiv preprint arXiv:2406.10602, 2024

… Common Crawl The Common Crawl corpus is a vast collection of web data accumulated over more than 10 years through web crawling. It includes raw web page data, metadata extracts, and text extracts. This noncurated dataset includes …


Multilingual Large Language Models: A Systematic Survey

S Zhu, S Xu, H Sun, L Pan, M Cui, J Du, R Jin… - arXiv preprint arXiv …, 2024

… CommonCrawl9, a large-scale collection of web dumps. It crawls billions of web pages every month and releases a snapshot, now containing snapshots from 2008 to the present, and it is continuously being updated. CommonCrawl … 12 snapshots …


Multilingual Model Fine-tuning for Sentiment Analysis

ML Elrefai, MI Khalil, HM Abbas - … on Intelligent Computing and Information Systems …, 2023

… A sizable multilingual language model was created using 2.5 TB of filtered CommonCrawl data. One of the outstanding XLMRoBERTa in low-resource languages, Low-resource languages are languages with limited available data for …


Multilingual Models in Neural Machine Translation

… XGLM [19] uses the decoder-only model architecture similar to that of GPT-3 [6], and was pre-trained on CC100-XL that covers 68 Common Crawl (CC) snapshots and 134 languages. Experiment results show that XGLM (7.5B) outperforms GPT-3 (6.7B) …


Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language

J Wang, Y Lu, M Weber, M Ryabinin, Y Chen, R Tang… - arXiv preprint arXiv …, 2024

English, as a very high-resource language, enables the pretraining of high-quality large language models (LLMs). The same cannot be said for most other languages, as leading LLMs still underperform for non-English languages, likely due to a gap in …


Multilingual Sarcasm Detection for Enhancing Sentiment Analysis using Deep Learning Algorithms

Recent years have seen a notable rise in online opinion-sharing, underscoring the demand for automated sentiment analysis tools. Addressing sarcasm in text is crucial, as it can significantly influence the effectiveness of sentiment analysis models. This …


Multilingual Speech and Language Analysis for the Assessment of Mild Cognitive Impairment: Outcomes from the Taukadial Challenge

PA Perez-Toro, T Arias-Vergara, P Klumpp, T Weise…

… Our analysis utilized the BERTBase and RoBERTa-Base models, trained on Wikicorpus data spanning 102 languages and CommonCrawl data filtered from 100 languages [16], respectively. For both models, the final layer, comprising 768 units …


MultiLS: A Multi-task Lexical Simplification Framework

K North, T Ranasinghe, M Shardlow, M Zampieri - arXiv preprint arXiv:2402.14972, 2024

Lexical Simplification (LS) automatically replaces difficult to read words for easier alternatives while preserving a sentence's original meaning. LS is a precursor to Text Simplification with the aim of improving text accessibility to various target …


Multimodal Alignment and Fusion: A Survey

S Li, H Tang - arXiv preprint arXiv:2411.17040, 2024

This survey offers a comprehensive review of recent advancements in multimodal alignment and fusion within machine learning, spurred by the growing diversity of data types such as text, images, audio, and video. Multimodal integration enables …


Multimodal Approaches to Fair Image Classification: An Ethical Perspective

J Hickmon - arXiv preprint arXiv:2412.12165, 2024

In the rapidly advancing field of artificial intelligence, machine perception is becoming paramount to achieving increased performance. Image classification systems are becoming increasingly integral to various applications, ranging from …


Multimodal Large Language Models for Bioimage Analysis

S Zhang, G Dai, T Huang, J Chen - arXiv preprint arXiv:2407.19778, 2024

Rapid advancements in imaging techniques and analytical methods over the past decade have revolutionized our ability to comprehensively probe the biological world at multiple scales, pinpointing the type, quantity, location, and even temporal …


Multimodal Latent Language Modeling with Next-Token Diffusion

Y Sun, H Bao, W Wang, Z Peng, L Dong, S Huang… - arXiv preprint arXiv …, 2024

Multimodal generative models require a unified approach to handle both discrete data (eg, text and code) and continuous data (eg, image, audio, video). In this work, we propose Latent Language Modeling (LatentLM), which seamlessly integrates …


Multimodal Learning for Visual Question Answering using World Knowledge

Navigating the frontier of the Visual Turing Test, this research delves into multimodal learning to bridge the gap between visual perception and linguistic interpretation, a foundational challenge in artificial intelligence. It scrutinizes the integration of visual …


Multimodal predictive model for analyzing news data

The aim of this thesis is to create a predictive tool utilising a multimodal machine-learning model for the combined textual and visual data from a dataset of news articles to predict the political orientation of news articles. A thorough exploration of multimodal …


MULTINET: ABENCHMARK FOR GENERALIST ACTION MODELS

… OBELICS OBELICS [27] is an open web-scale filtered dataset of interleaved image-text documents extracted from Common Crawl [1]. It … The dataset was curated from approximately 10 billion initial pairs collected from CommonCrawl …


Multiscale cascaded domain-based approach for Arabic fake reviews detection in e-commerce platforms

N Qandos, G Hamad, M Alharbi, S Alturki, W Alharbi… - Journal of King Saud …, 2024

Fake reviews in e-commerce can lead to customer deception and financial losses. Despite the importance of fake reviews detection, studies for Arabic language are scarce due to the lack of comprehensive datasets. This study addresses this gap by …


Multivariate Log-based Anomaly Detection for Distributed Database

L Zhang, T Jia, M Jia, Y Li, Y Yang, Z Wu - arXiv preprint arXiv:2406.07976, 2024

… Following this, MultiLog utilizes pre-trained word vectors based on the Common Crawl Corpus with the FastText algorithm[45], which is adept at capturing the inherent relationships among words in natural language. This means each word in a …


Musical Word Embedding for Music Tagging and Retrieval

SH Doh, J Lee, D Jeong, J Nam - arXiv preprint arXiv:2404.13569, 2024

… We used two publicly available embeddings trained on Common Crawl, a large-scale general word corpus, with the GloVe and skipgram … It demonstrates that word embeddings trained with a general word corpus, such as Common Crawl and Wiki …


Myanmar XNLI: Building a Dataset and Exploring Low-resource Approaches to Natural Language Inference with Myanmar

Despite dramatic recent progress in NLP, it is still a major challenge to apply Large Language Models (LLM) to low-resource languages. This is made visible in benchmarks such as Cross-Lingual Natural Language Inference (XNLI), a key task …


Named Entity Recognition in Italian Lung Cancer Clinical Reports using Transformers

D Paolo, A Bria, C Greco, M Russano, S Ramella… - 2023 IEEE International …, 2023

The widespread adoption of electronic health records (EHRs) offers a valuable opportunity to support clinical research by containing crucial patient information, including diagnoses, symptoms, medications, lab tests, and more. Despite the …


NATALITY AT RISK? RAISING DOUBTS ON THE EDUCATIONAL IMPORTANCE OF ChatGPT

Digital tools are part of our daily life. Thus, they also enter educational contexts. With reference to Hannah Arendt’s writings this paper explores if, and to what extent, digital tools can be considered to be helpful and desirable in education. For this …


Native Design Bias: Studying the Impact of English Nativeness on Language Model Performance

M Reusens, P Borchert, J De Weerdt, B Baesens - arXiv preprint arXiv:2406.17385, 2024

Large Language Models (LLMs) excel at providing information acquired during pretraining on large-scale corpora and following instructions through user prompts. This study investigates whether the quality of LLM responses varies depending on …


Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning

A Das - arXiv preprint arXiv:2410.16029, 2024

Training LLMs presents significant memory challenges due to growing size of data, weights, and optimizer states. Techniques such as data and model parallelism, gradient checkpointing, and offloading strategies address this issue but are often …


Natural Language Processing Advancements: Breaking Barriers in Human-Computer Interaction

JGC Ramírez - Journal of Artificial Intelligence General science (JAIGS …, 2024

Natural Language Processing (NLP) advancements have revolutionized human-computer interaction, breaking barriers and opening new frontiers in technology. NLP techniques enable machines to understand, interpret, and generate human …


Natural Language Processing Approaches for Monitoring Health Activities

Monitoring health activities is vital for individual and public health. It identifies trends, detects risks early, and tailors interventions. At the individual level, it informs customized treatments. At the population level, monitoring broad health patterns …


Natural Language Processing for Dialects of a Language: A Survey

A Joshi, R Dabre, D Kanojia, Z Li, H Zhan, G Haffari… - arXiv preprint arXiv …, 2024

State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language …


Natural Language Processing for Slang

… Existing NLP approaches rely on large training corpora derived from sources such as Wikipedia8 and Common Crawl9 which are composed of formal documents such as news articles and wiki pages. Because of this, models observe formal …


Natural Language Processing Journal

MN Hoque, U Salma, MJ Uddin, MM Ahamad, S Aktar

S Gopali, S Siami-Namini, F Abri, AS Namin

In the sentiment analysis (SA) task, we can obtain a positive or negative-typed comment or feedback from an online user or a customer about any object, such as a movie, drama, food, and others. This user’s sentiment may positively impact various …


Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey

DVT Le, L Bigo, M Keller, D Herremans - arXiv preprint arXiv:2402.17467, 2024

Several adaptations of Transformers models have been developed in various domains since its breakthrough in Natural Language Processing (NLP). This trend has spread into the field of Music Information Retrieval (MIR), including studies …


Natural Language Processing Tools for Romanian–Going Beyond a Low-Resource Language

… Oscar (Open Super-large Crawled Aggregated coRpus) [64] is a multilingual dataset obtained via filtering the common crawl corpus. It encompasses 151 languages. The Romanian sub-corpus contains over 4,5 million documents …


Natural Language Processing

CH Focke, RM Sylvester - Statistical Methods in Epilepsy, 2024

In this chapter, we introduce modern natural language processing (NLP) libraries, techniques and their applications. This chapter focuses on deep learning methods and less on computational linguistics that require nuanced knowledge of linguistics …


Natural Language, Legal Hurdles: Navigating the Complexities in Natural Language Processing Development and Application

I Ilin, A Kelli - Journal of the University of Latvia. Law, 2024

This article delves into the legal challenges faced in developing and deploying Natural Language Processing (NLP) technologies, focusing particularly on the European Union’s legal framework, especially the DSM Directive, the InfoSoc …


Natural

I have been working on natural language generation (NLG), that is using artificial intelligence techniques to produce texts in English and other human languages, since I got my PhD in this area in 1990. In late 2022, NLG became much more …


Navigating Challenges and Technical Debt in Large Language Models Deployment

A Menshawy, Z Nawaz, M Fahmy - Proceedings of the 4th Workshop on Machine …, 2024

… These models, trained on an internet scale data including common crawl, books, and articles, have acquired an extensive understanding of language patterns, nuances, and contexts. This profound comprehension allows LLMs to perform a …


Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

S Woerner, CF Baumgartner - arXiv preprint arXiv:2408.08058, 2024

… LAION-2B contains 2 billion image-text pairs extracted from common crawl [3] and is the English language subset of the larger LAION-5B [12] … https://commoncrawl.org. [4] Alexey Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image …


Navigating Multimodal Complexity: Advances in Model Design, Dataset Creation, and Evaluation Techniques

Ibn Sina, a philosopher of 11th-century Persia, wrote of a `Floating Man'. This man is floating through a void, without the use of his sight or touch or any of the senses which make us human. Yet as he has a human brain this man, according to Ibn Sina …


Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents

B Gou, R Wang, B Zheng, Y Xie, C Chang, Y Shu… - arXiv preprint arXiv …, 2024

… We apply few filtering methods, excluding non-webpage files based on URLs, and removing non-English pages based on labels of language provided by Common Crawl. We use Playwright to load and render the webpages, capturing …


Navigating the ethical landscape behind ChatGPT

L Peng, B Zhao - Big Data & Society, 2024

… While ChatGPT and similar text-generating AIs claim to be trained on mostly public data such as the Common Crawl archives, rendering … content in public datasets used for AI training, such as the Common Crawl, which is beyond the …


Navigating the Legal Landscape: Technical Implementation of Copyright Reservations for Text and Data Mining in the Era of AI Language Models

L Löbling, C Handschigl, K Hofmann, J Schwedhelm - JIPITEC–Journal of Intellectual …, 2023

… However, limitations arose from the potential omission of specific pages of a website in individual monthly crawls by Common Crawl. Through manual investigation, a section containing the terms of use could be identified for 40 …


Ncl-uor at semeval-2024 task 8: Fine-tuning large language models for multigenerator, multidomain, and multilingual machine-generated text detection

F Xiong, T Markchom, Z Zheng, S Jung, V Ojha… - Proceedings of the 18th …, 2024

SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual …


Near to Mid-term Risks and Opportunities of Open Source Generative AI

F Eiras, A Petrov, B Vidgen, CS de Witt, F Pizzati… - arXiv preprint arXiv …, 2024

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential …


Need a Programming Exercise Generated in Your Native Language? ChatGPT's Got Your Back: Automatic Generation of Non-English Programming Exercises Using …

M Jordan, K Ly, AG Soosai Raj - Proceedings of the 55th ACM Technical Symposium …, 2024

Large language models (LLMs) like ChatGPT are changing computing education and may create additional barriers to those already faced by non-native English speakers (NNES) learning computing. We investigate an opportunity for a positive …


Negotiating becoming: a Nietzschean critique of large language models

SWS Fischer, B Boer - Ethics and Information Technology, 2024

Large language models (LLMs) structure the linguistic landscape by reflecting certain beliefs and assumptions. In this paper, we address the risk of people unthinkingly adopting and being determined by the values or worldviews embedded …


Nemotron-4 15B Technical Report

J Parmar, S Prabhumoye, J Jennings, M Patwary… - arXiv preprint arXiv …, 2024

We introduce Nemotron-4 15B, a 15-billion-parameter large multilingual language model trained on 8 trillion text tokens. Nemotron-4 15B demonstrates strong performance when assessed on English, multilingual, and coding tasks: it …


Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

D Su, K Kong, Y Lin, J Jennings, B Norick, M Kliegl… - arXiv preprint arXiv …, 2024

Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such …


NER-RoBERTa: Fine-Tuning RoBERTa for Named Entity Recognition (NER) within low-resource languages

AA Abdullah, SH Abdulla, DM Toufiq, HS Maghdid… - arXiv preprint arXiv …, 2024

… In the same vein, a new version of RoBERTa is fine-tuned in 2024 and it is re-trained using various text-datasets within 100 different languages using the CommonCrawl dataset [11]. However, the RoBERTa for NER purposes using Central-Kurdish …


Neural Approaches to Decode Semantic Similarities in Spanish Song Lyrics for Enhanced Recommendation Systems

… In this model, it was found that traditional methods of pre-training, which rely on large amounts of data from sources such as Common Crawl… The mC4 dataset is a multilingual variant of C4, containing natural text in 101 languages from the public …


Neural Chat Translation as Online Document-to-Document Translation

M Lyu, H Dou, J Li, M Zhu, G Zhou - … on Natural Language Processing and Chinese …, 2024

Neural chat translation (NCT) aims to translate utterances in a bilingual dialogue, where participants speak different languages, into their respective native languages. Recent advances on NCT translate chat utterances one by one using document-to-sentence …


Neural Information Retrieval

N Tonellotto - Information Retrieval: Advanced Topics and …, 2024

… As illustrated in Figure 2.3, pre-training typically requires a huge generalpurpose training corpus, such as Wikipedia or Common Crawl web pages, expensive computation resources, and long training times, spanning several days or weeks. On …


NEURAL MACHINE TRANSLATION BETWEEN MYANMAR AND ENGLISH LANGUAGES

This study aims to present a thorough overview of cutting-edge machine translation solutions and evaluate their effectiveness in translating between Myanmar and English language pairs. Translating Myanmar poses challenges due to its unique …


Neural Methods for Aligning Large-Scale Parallel Corpora from the Web for South and East Asian Languages

We introduce neural methods and a toxicity filtering step to the hierarchical web mining approach of Paracrawl (Bañón et al., 2020), showing large improvements. We apply these methods to web-scale parallel corpus mining for 9 South and East …


NEURAL MODELS FOR ONTOLOGY ANNOTATIONS-NEMO

Words cannot express my gratitude to my advisors Dr. Somya D. Mohanty and Dr. Prashanti Manda for their invaluable supervision and guidance. I am extremely grateful to them for their continuous support, constructive suggestions, and …


Neural Network-based Language Modelling for Code-Switching in South African Languages

Language models, specifically those based on neural network architectures, have become an attractive alternative to classical statistical language models for natural language processing and speech recognition. Naturally, therefore, the field of neural …


Neural Networks with Model Compression

B Zhang, T Wang, S Xu, D Doermann - 2024

… Firstly, the availability of large-scale datasets, such as ImageNet for computer vision or the Common Crawl dataset for natural language processing, has enabled the training of deep neural networks with millions or even billions of parameters …


Neural Pathway Embedding Through Hierarchical Interchange Networks in Large Language Models

L Eamen, A Phillips, J Mitchell, B Parker, S Bennett - 2024

… Dataset Description The model was trained and tested on the Common Crawl dataset, a comprehensive corpus of web pages that provided … The selection of the Common Crawl dataset was motivated through its extensive coverage of linguistic …


Neural Speech Tracking in EEG: Integrating Acoustics and Linguistics for Hearing Aid Users

This master thesis explores the neural encoding of speech features for hearing aid users. The study utilizes electroencephalography (EEG) and audio data from an experiment that stimulates a Cocktail Party scenario. This is a complex auditory …


Neutral residues: revisiting adapters for model extension

FS Talla, H Jegou, E Grave - arXiv preprint arXiv:2410.02744, 2024

… For the French finetuning dataset, we use a dataset extracted from CommonCrawl, which was pre-processed with the following steps. First, the … Finally, we filtered document using a fastText linear classifier, which was trained to discriminate …


Next word prediction for newspaper article formulation

Next word prediction describes the process of predicting the next word after every written word in the authoring of texts. It refers to machine learning techniques, which are part of the language modelling domain, a sub-domain of natural language …


NLLB-Based Uzbek NMT: Leveraging Multisource Data

N Abdurakhmonova, AZ Mohirdev, M Salokhiddinov… - 2024 9th International …, 2024

Recent advancements in Neural Machine Translation (NMT) have been largely propelled by the development of transformer-based architectures, exemplified by the NLLB (No Language Left Behind) model. This study extends previous work on low-resource …


NLP Essentials

H Assoudi - Natural Language Processing on Oracle Cloud …, 2024

In the digital era, there has been a significant increase in text data, ranging from social media content to corporate documents. To effectively utilize this data, a thorough grasp of Natural Language Processing (NLP) is essential. This chapter …


NLP for The Greek Language: A Longer Survey

K Papantoniou, Y Tzitzikas - arXiv preprint arXiv:2408.10962, 2024

English language is in the spotlight of the Natural Language Processing (NLP) community with other languages, like Greek, lagging behind in terms of offered methods, tools and resources. Due to the increasing interest in NLP, in this paper we …


NLP-ML Hybrid: Identifying Signs of Suicidal Thoughts in Social Media Content

RJR Samuel, A Jenefa, P Santhiya, GJL Paulraj… - 2024 2nd International …, 2024

… Model Used: cc.en.300.bin represents the implementation of pre-trained embeddings on Common Crawl data with 300-dimensional vectors for the English language. 2) TF-IDF: The Term Frequency-Inverse Document Frequency (TF-IDF) is …


No Filter: Cultural and Socioeconomic Diversityin Contrastive Vision-Language Models

A Pouget, L Beyer, E Bugliarello, X Wang, AP Steiner… - arXiv preprint arXiv …, 2024

We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training …


No Need to Talk: Asynchronous Mixture of Language Models

A Filippova, A Katharopoulos, D Grangier, R Collobert - arXiv preprint arXiv …, 2024

We introduce SmallTalk LM, an innovative method for training a mixture of language models in an almost asynchronous manner. Each model of the mixture specializes in distinct parts of the data distribution, without the need of high-bandwidth …


Non-governmental Governance of Trust on the Internet: WebPKI as Public Good

… The population of sites we sampled from was indexed by the non-profit foundation Common Crawl. We then employed a script to pull certificate information from our list of URLs. Slightly more than half the sample did not return certificate …


Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models

This paper identifies a cultural dominance issue within large language models (LLMs) due to the predominant use of English data in model training (eg, ChatGPT). LLMs often provide inappropriate English-culture-related answers that are not relevant to …


Not All Tokens Are What You Need for Pretraining

Z Lin, Z Gou, Y Gong, X Liu, R Xu, C Lin, Y Yang, J Jiao… - The Thirty-eighth Annual …

Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that ''Not all tokens in a corpus are equally important for language model training''. Our initial …


Notes towards infrastructure governance for large language models

This paper draws on information infrastructures (IIs) in science and technology studies (STS), as well as on feminist STS scholarship and contemporary critical accounts of digital technologies, to build an initial mapping of the infrastructural …


nowhash at SemEval-2024 Task 4: Exploiting Fusion of Transformers for Detecting Persuasion Techniques in Multilingual Memes

AN Chowdhury, M Ptaszynski - Proceedings of the 18th International Workshop on …, 2024

Nowadays, memes are considered one of the most prominent forms of medium to disseminate information on social media. Memes are typically constructed in multilingual settings using visuals with texts. Sometimes people use memes to …


NRK at SemEval-2024 Task 1: Semantic Textual Relatedness through Domain Adaptation and Ensemble Learning on BERT-based models

NT Kiet, D Van Thin - Proceedings of the 18th International Workshop on …, 2024

This paper describes the system of the team NRK for Task A in the SemEval-2024 Task 1: Semantic Textual Relatedness (STR). We focus on exploring the performance of ensemble architectures based on the voting technique and different …


NSINA: A News Corpus for Sinhala

H Hettiarachchi, D Premasiri, L Uyangodage… - arXiv preprint arXiv …, 2024

The introduction of large language models (LLMs) has advanced natural language processing (NLP), but their effectiveness is largely dependent on pre-training resources. This is especially evident in low-resource languages, such as Sinhala …


Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis

A Shah, A Hiray, P Shah, A Banerjee, A Singh… - arXiv preprint arXiv …, 2024

In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a …


NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

W Wongso, DS Setiawan, S Limcorn, A Joyoadikusumo - arXiv preprint arXiv …, 2024

… We also note that NusaBERT was pre-trained on Wikipedia and common crawl corpora, which explains its effectiveness on and closeness to NusaX and NusaTranslation source domains, but not so for NusaParagraph. Thus, due to their high linguistic and …


Nyonic Technical Report

J Tian, R Wang, C Li, Y Zhou, J Liu, J Wang - arXiv preprint arXiv:2404.15702, 2024

… Our curated dataset is specifically designed to meet these criteria, incorporating sources such as the Common Crawl, books, Wikipedia, code, and … Our data sources encompass Common Crawl, Wiki, and various coding repositories. The final …


Ocean Decade Vision 2030 White Papers–Challenge 8: Create a digital representation of the ocean.

JB Calewaert, PC Sierra-Correa, O McMeel… - 2024

… The Common crawl and WebImageText are examples, with communities such as Hugging Face and KAGGLE creating new communities of practice16F … 17 https://commoncrawl.org, https://github.com/google-research-datasets/wit, https://huggingface.co, https://www.kaggle.com …


OF CLOSED NATIONAL WEB ARCHIVES THROUGH SUMMARIES

J Nielsen, Y Maurer, E Zierau - The Routledge Companion to Transnational Web …, 2024

… are the national web archives placed at the national libraries of Denmark, France, Hungary, and Luxembourg as well as what has been preserved of these countries’ national country code top-level domains (ccTLDs) in the collections of the Internet …


OF. LU DOMAIN NAMES THROUGH A TRANSNATIONAL COMPARISON

C Noguera, J Nielsen, Y Maurer, B Els - The Routledge Companion to Transnational …, 2024

This chapter aims to study the development of the. lu domain by analysing the domain names from 1992 to 2022 and putting the results in perspective using a transnational approach. Part of the objective is to evaluate whether a method used …


Offensive language detection in low resource languages: A use case of Persian language

M Mozafari, K Mnassri, R Farahbakhsh, N Crespi - PLOS ONE, 2024

… XLM-RoBERTa [27]: is a transformer-based multilingual masked language model pre-trained on 100 languages, including Persian, using more than two terabytes of filtered CommonCrawl data. To fine-tune XLM-RoBERTa model on our target …


OLMo: Accelerating the Science of Language Models

D Groeneveld, I Beltagy, P Walsh, A Bhagia, R Kinney… - arXiv preprint arXiv …, 2024

… well on evaluations predominated by Common Crawl, such as C4, though different ways of postprocessing Common Crawl are best fit by … The RedPajama evaluation shows a similar pattern, perhaps as only 2 of its 7 domains are from …


OLMoE: Open Mixture-of-Experts Language Models

N Muennighoff, L Soldaini, D Groeneveld, K Lo… - arXiv preprint arXiv …, 2024

We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to …


OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations

L Ouyang, Y Qu, H Zhou, J Zhu, R Zhang, Q Lin… - arXiv preprint arXiv …, 2024

… Specifically, we collected 200,000 initial PDF documents from Common Crawl, Google, Baidu search engines, and internal data. Subsequently, we extracted visual features from these document pages using ResNet-50 and performed clustering …


Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models

S Ruan, Y Dong, H Liu, Y Huang, H Su, X Wei - arXiv preprint arXiv:2404.12139, 2024

Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint …


On humans'(explicit) intuitions about the meaning of novel words

D Gatti, F Rodio, L Rinaldi, M Marelli - Cognition, 2024

Pseudowords offer a unique opportunity to investigate how humans deal with new (verbal) information. Within this framework, previous studies have shown that, at the implicit level, humans exploit systematic associations in the form-meaning interface to …


On Initializing Transformers with Pre-trained Embeddings

HY Kim, N Balasubramanian, B Kang - arXiv preprint arXiv:2407.12514, 2024

It has become common practice now to use random initialization schemes, rather than the pre-trained embeddings, when training transformer based models from scratch. Indeed, we find that pre-trained word embeddings from GloVe, and some …


On Instruction-Finetuning Neural Machine Translation Models

V Raunak, R Grundkiewicz, M Junczys-Dowmunt - arXiv preprint arXiv:2410.05553, 2024

In this work, we introduce instruction finetuning for Neural Machine Translation (NMT) models, which distills instruction following capabilities from Large Language Models (LLMs) into orders-of-magnitude smaller NMT models. Our instruction-finetuning …


On Labs and Fabs: Mapping How Alliances, Acquisitions, and Antitrust are Shaping the Frontier AI Industry

T Aguirre - arXiv preprint arXiv:2406.01722, 2024

As frontier AI models progress, policy proposals for safe AI development are gaining increasing attention from researchers and policymakers. This paper evaluates the present landscape of integration within the AI supply chain, emphasizing vertical …


On Protecting the Data Privacy of Large Language Models (LLMs): A Survey

B Yan, K Li, M Xu, Y Dong, Y Zhang, Z Ren, X Cheng - arXiv preprint arXiv …, 2024

… For example, GPT-3, developed by OpenAI, was pre-trained using CommonCrawl, constituting 45TB of compressed plaintext before filtering [37]. Regarding multimodal LLMs, CLIP’s training dataset encompasses 400 million pairs of images and text …


On the data scarcity problem of neural-based named entity recognition

… The initial release trained on Wikipedia includes word representations for nine languages, namely Arabic, Czech, German, English, Spanish, French, Italian, Romanian and Russian, and has quickly expanded to 157 languages by training on larger …


On the evaluation and application of neural language models for grammatical error detection

Neural language models (NLM) have become a core component in many downstream applications within the field of natural language processing, including the task of data-driven automatic grammatical error detection (GED). This thesis …


On the importance of Data Scale in Pretraining Arabic Language Models

A Ghaddar, P Langlais, M Rezagholizadeh, B Chen - arXiv preprint arXiv:2401.07760, 2024

Pretraining monolingual language models have been proven to be vital for performance in Arabic Natural Language Processing (NLP) tasks. In this paper, we conduct a comprehensive study on the role of data in Arabic Pretrained Language …


On the In-context Generation of Language Models

Z Jiang, Y Zhang, K Luo, X Yuan, J Zhao, K Liu - … of the 2024 Conference on Empirical …, 2024

… As known, typical pretrained corpora contain (eg CommonCrawl1) internet data which has an unneglectable portion of array-page data such as IMDB2 review pages. After preprocessing, these pages are converted to repetition mode data, as shown in …


On the Mathematical Relationship Between Contextual Probability and N400 Amplitude

JA Michaelov, BK Bergen - Open Mind, 2024

Accounts of human language comprehension propose different mathematical relationships between the contextual probability of a word and how difficult it is to process, including linear, logarithmic, and super-logarithmic ones. However, the …


On the Question of Authorship in Large Language Models

C Soos, L Haroutunian - KO KNOWLEDGE ORGANIZATION, 2024

… CommonCrawl exemplifies the problem of documentation debt. The dataset is an effort by The Common Crawl Foundation to “[democratize] access to web information by producing and maintaining an open repository of web crawl data” (Common …


On the Relationship of Social Gender Equality and Grammatical Gender in Pre-trained Large Language Models

M Biesialska, D Solans, J Luque, C Segura - 2024

Large Language Models pre-trained on vast amounts of text have demonstrated remarkable capabilities in modeling and generating human language, finding applications across a wide range of Natural Language Processing tasks. However …


On the Robustness of Neural Networks Quantization against Data Poisoning Attacks

Y Lu, Y Wang, G Zhang, Y Yu - ICML 2024 Next Generation of AI Safety Workshop

… For example, practitioners may continuously collect user data online (eg, using Common Crawl1), which may be maliciously contaminated by an attacker. Such a threat is called data poisoning attacks (Biggio et al. 2012), … https://commoncrawl.org/ …


On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe

N Xu, Q Zhang, M Zhang, P Qian, X Huang - arXiv preprint arXiv:2402.14404, 2024

Probing and enhancing large language models' reasoning capacity remains a crucial open question. Here we re-purpose the reverse dictionary task as a case study to probe LLMs' capacity for conceptual inference. We use in-context learning …


One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

V Rasiah, R Stern, V Matoshi, M Stürmer, I Chalkidis…

Recent strides in Large Language Models (LLMs) have saturated many NLP benchmarks (even professional domain-specific ones), emphasizing the need for more challenging ones to properly assess LLM capabilities. Domain-specific and …


Online health search via multi-dimensional information quality assessment based on deep language models

B Zhang, N Naderi, R Mishra, D Teodoro - 2024

Background Widespread misinformation in Web resources can lead to serious implications for individuals seeking health advice. Despite that, information retrieval models are often focused only on the query-document relevance dimension to rank …


Online Health Search Via Multidimensional Information Quality Assessment Based on Deep Language Models: Algorithm Development and Validation

B Zhang, N Naderi, R Mishra, D Teodoro - JMIR AI, 2024

… In this study, we simulated online health information search scenarios with a topic set of 32 different health-related inquiries and a corpus containing 1 billion web documents from the April 2019 snapshot of Common Crawl. Using state-of-the-art …


OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters

Z Chen, C Li, X Xie, P Dube - arXiv preprint arXiv:2409.00286, 2024

This paper explores the potential of a small, domain-specific language model trained exclusively on sports-related data. We investigate whether extensive training data with specially designed small model structures can overcome model size …


Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysis

N Li, T Bailleux, Z Bouraoui, S Schockaert - arXiv preprint arXiv:2403.17216, 2024

We consider the problem of finding plausible knowledge that is missing from a given ontology, as a generalisation of the well-studied taxonomy expansion task. One line of work treats this task as a Natural Language Inference (NLI) problem, thus relying …


Open Assistant Toolkit--version 2

S Fischer, F Rossetto, C Gemmell, A Ramsay, I Mackie… - arXiv preprint arXiv …, 2024

… To source knowledge and tasks, we release a new offline pipeline to parse and augment task data from CommonCrawl. The offline pipeline supports using LLMs and additional open-source multimodal data sources to enhance tasks and make …


Open foundation models for Azerbaijani language

J Isbarov, K Huseynova, E Mammadov, M Hajili - arXiv preprint arXiv:2407.02337, 2024

The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-grade systems rely on cloud solutions, such as GPT-4 …


Open Science at the Generative AI Turn: An Exploratory Analysis of Challenges and Opportunities

Technology influences Open Science (OS) practices, because conducting science in transparent, accessible, and participatory ways requires tools/platforms for collaborative research and sharing results. Due to this direct relationship …


Open Source Software Policy in Industry Equilibrium

… I start with homepages of the top 10,000 most visited websites, archived by Common Crawl (Appendix B) in each month from 2014 to 2022… When Common Crawl fails to archive a homepage, I assume its software remains unchanged from …


Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Q Xie, D Li, M Xiao, Z Jiang, R Xiang, X Zhang, Z Chen… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs} …


OpenAI ChatGPT and Biased Information in Higher Education

Motivated by the appearance of large language models and their sudden societal impacts—both beneficial and harmful, realized and potential—we evaluated several of them with respect to bias in its myriad forms. Bias in machine-learning models …


OpenForge: Probabilistic Metadata Integration

T Cong, F Nargesian, J Xing, HV Jagadish - arXiv preprint arXiv:2412.09788, 2024

Modern data stores increasingly rely on metadata for enabling diverse activities such as data cataloging and search. However, metadata curation remains a labor-intensive task, and the broader challenge of metadata maintenance -- ensuring its consistency …


OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

F Xue, Z Zheng, Y Fu, J Ni, Z Zheng, W Zhou, Y You - arXiv preprint arXiv:2402.01739, 2024

To help the open-source community have a better understanding of Mixture-of-Experts (MoE) based large language models (LLMs), we train and release OpenMoE, a series of fully open-sourced and reproducible decoder-only MoE LLMs, ranging from …


OPI: An Open Instruction Dataset for Adapting Large Language Models to Protein-Related Tasks

H Xiao, W Lin, H Wang, Z Liu, Q Ye - Neurips 2024 Workshop Foundation Models for …

Large language models (LLMs) pretrained on extensive general corpora, such as GPT-4 and Llama series, have demonstrated remarkable excellence in a large variety of natural language processing (NLP) tasks. These models provide a user-friendly …


OptiComm-GPT: a GPT-based versatile research assistant for optical fiber communication systems

X Jiang, M Zhang, Y Song, Y Zhang, Y Wang, C Ju… - Optics Express, 2024

… Generally, LLMs are pre-trained on publicly available datasets such as BooksCorpus, Common Crawl, and Wikipedia, thereby ensuring comprehensive coverage of knowledge. Meanwhile, with the support of powerful analysis, reasoning …


Optimization and Evaluation in Machine Learning Challenges

To develop new machine learning methods, it is necessary to evaluate them reliably. This doctoral thesis discusses some aspects of preparing machine learning challenges and techniques for developing their solutions. The work consists of …


Optimization Hyper-parameter Laws for Large Language Models

X Xie, K Ding, S Yan, KC Toh, T Wei - arXiv preprint arXiv:2409.04777, 2024

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they …