Harnessing Webpage UIs for Text-Rich Visual Understanding

J Liu, T Ou, Y Song, Y Qu, W Lam, C Xiong, W Chen… - arXiv preprint arXiv …, 2024

Text-rich visual understanding-the ability to process environments where dense textual content is integrated with visuals-is crucial for multimodal large language models (MLLMs) to interact effectively with structured environments. To enhance this …


Hate speech detection in low-resourced Indian languages: An analysis of transformer-based monolingual and multilingual models with cross-lingual experiments

K Ghosh, A Senapati - Natural Language Processing

Warning: This paper is based on hate speech detection and may contain examples of abusive/ offensive phrases.Cyberbullying, online harassment, etc., via offensive comments are pervasive across different social media platforms like ™Twitter, ™Facebook …


HCI International 2024 Posters: 26th International Conference on Human-Computer Interaction, HCII 2024, Washington, DC, USA, June 29–July 4, 2024, Proceedings …

Preliminary scientific results, professional news, or work in progress, described in the form of short research papers (4–11 pages long), constitute a popular submission type among the International Conference on Human-Computer …


HDeFC: Hierarchical Secure Fuzzy Deduplication Based on Fog Computing

Z Tang, S Zeng, S Han, J Liu, M He - IEEE Internet of Things Journal, 2024

Secure deduplication not only optimizes cloud storage but also prevents data leakage. However, traditional schemes are with high computation and communication costs to deal with large-scale multimedia data. To address this …


Headline Generation for Indian Languages

… mT5: We conducted experiments on our dataset by fine-tuning the pre-trained mT5 [35] model, a multilingual variant of T5 [25], which was originally trained on the common crawl dataset, encompassing 101 languages. We use mT5-small model for …


Health Misinformation in Social Networks: A Survey of IT Approaches

V Papanikou, P Papadakos, T Karamanidou… - arXiv preprint arXiv …, 2024

In this paper, we present a comprehensive survey on the pervasive issue of medical misinformation in social networks from the perspective of information technology. The survey aims at providing a systematic review of related research and helping …


HGT: Leveraging Heterogeneous Graph-enhanced Large Language Models for Few-shot Complex Table Understanding

R Jin, Y Li, G Qi, N Hu, YF Li, J Chen, J Wang, Y Chen… - arXiv preprint arXiv …, 2024

Table understanding (TU) has achieved promising advancements, but it faces the challenges of the scarcity of manually labeled tables and the presence of complex table structures.To address these challenges, we propose HGT, a framework with a …


Hidden Capabilities and Counterintuitive Limits in Large Language Models

As massive language models like GPT-4 dominate NLP and AI, extreme-scale has become a clear and frequent theme for success. My research envisions a world where alternative approaches, efficient methods working on small to medium-scale …


Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

H Xu, L Chen, Z Zhao, D Ma, R Cao, Z Zhu, K Yu - arXiv preprint arXiv:2402.18262, 2024

… Common Crawl is a publicly available web crawl dataset that collects webpages from the internet. Instead of using the source code, we collect webpage links from Common Crawl. We traverse all links in a dataset snapshot 4 and categorize and …


High Performance Computing Web Search System Based on Computerized Big Data

… In addition, this paper builds an experimental environment on Microsoft Azure cloud platform and tests it with Common Crawl dataset. Finally, this paper evaluates the performance of the system by three indicators: response time, accuracy and …


Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania

J Rudolph, MF Ismail, S Popenici - Journal of University Teaching and Learning …, 2024

Higher education is currently under a significant transformation due to the emergence of generative artificial intelligence (GenAI) technologies, the hype surrounding GenAI and the increasing influence of educational technology business …


HindiLLM: Large Language Model for Hindi

S Chouhan, SB Nath, A Dutta - International Conference on Pattern Recognition, 2025

The advancements in the Large Language Model (LLM) have helped in solving several problems related to language processing. Most of the researches have focused on the English language only, because of its popularity and abundance on …


Historical Dutch Spelling Normalization with Pretrained Language Models

A Wolters, A Van Cranenburgh - Computational Linguistics in the Netherlands …, 2024

… These texts are from the common crawl corpus, which contains petabytes of data scraped from the internet. Further pretraining on in-domain data can make the model more suited for the downstream task it will be finetuned for, and since our task …


Historical German Text Normalization Using Type-and Token-Based Language Modeling

A Ehrmanntraut - arXiv preprint arXiv:2409.02841, 2024

Historic variations of spelling poses a challenge for full-text search or natural language processing on historical digitized texts. To minimize the gap between the historic orthography and contemporary spelling, usually an automatic orthographic …


History, Development, and Principles of Large Language Models-An Introductory Survey

Z Chu, S Ni, Z Wang, X Feng, C Li, X Hu, R Xu, M Yang… - arXiv preprint arXiv …, 2024

Language models serve as a cornerstone in natural language processing (NLP), utilizing mathematical methods to generalize language laws and knowledge for prediction and generation. Over extensive research spanning decades, language …


History, development, and principles of large language models: an introductory survey

Z Wang, Z Chu, TV Doan, S Ni, M Yang, W Zhang - AI and Ethics, 2024

Abstract Language models serve as a cornerstone in natural language processing, utilizing mathematical methods to generalize language laws and knowledge for prediction and generation. Over extensive research spanning decades, language …


HJ-Ky-0.1: an Evaluation Dataset for Kyrgyz Word Embeddings

A Alekseev, G Kabaeva - arXiv preprint arXiv:2411.10724, 2024

… Our experiments also showed that fastText embeddings trained on the Leipzig Corpus outperform those trained on CommonCrawl, highlighting the importance of clean, high-quality text data for training. These findings suggest that the HJ-Ky-0.1 …


HMoE: Heterogeneous Mixture of Experts for Language Modeling

A Wang, X Sun, R Xie, S Li, J Zhu, Z Yang, P Zhao… - arXiv preprint arXiv …, 2024

Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying …


Ho to Read a Generative AI Image System: Diffusion models as a techno-social entanglement

… The first approach to resource extraction is the data commons of the world wide web, which is extracted through Common Crawl into datasets. However, there is also an environmental extraction: it is estimated that 700,000 litres of water were …


Holographic Embeddings for Text and Graphs A Master's Thesis Presented to The Faculty of the Graduate School of Arts and Sciences

… Static word embeddings such as word2vec, Global Vectors for Word Representation (GloVe), and fastText are trained on large amounts of natural language text sources like Common Crawl and Wikipedia dumps. These models …


Homonym Sense Disambiguation in the Georgian Language

D Melikidze, A Gamkrelidze - arXiv preprint arXiv:2405.00710, 2024

This research proposes a novel approach to the Word Sense Disambiguation (WSD) task in the Georgian language, based on supervised fine-tuning of a pre-trained Large Language Model (LLM) on a dataset formed by filtering the Georgian …


HOPE: Your Mental Health Companion

DP Gawade, NA Tank - Recent Advances in Artificial Intelligence and Smart …

RA Borgalli, JP Suryawanshi, DP Gawade, NA Tank… - Recent Advances in Artificial …

… Common Crawl dataset [11] now containing almost a trillion words. It has been discovered, however, that unfiltered or minimally filtered copies of Common Crawl … train language models, including filtering a version of Common Crawl based on …


How Chinese are Chinese Language Models? The Puzzling Lack of Language Policy in China's LLMs

AW Wen-Yi, UES Jo, LJ Lin, D Mimno - arXiv preprint arXiv:2407.09652, 2024

… Additionally, similar to the training of international models, sources of pretrained data rely heavily on web pages, with three reports specifically mentioning Common Crawl as their primary data source… Common Crawl 8 (80%), encyclopedia, books …


How Do We Learn What We Cannot Say?

The contributions of this thesis are two-fold. First, this thesis presents UDTube, an easily usable software developed to perform morphological analysis in a multi-task fashion. This work shows the strong performance of UDTube versus the current state-of-the-art …


How Does Code Pretraining Affect Language Model Task Performance?

J Petty, S van Steenkiste, T Linzen - arXiv preprint arXiv:2409.04556, 2024

Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may …


How Effective are State Space Models for Machine Translation?

H Pitorro, P Vasylenko, M Treviso, AFT Martins - arXiv preprint arXiv:2407.05489, 2024

… In order to obtain a small high-quality subset for training, we exclude ParaCrawl and CommonCrawl samples from the original dataset and clean the remaining data. Our cleaning process includes three steps. First, we identify and remove samples in …


How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

Z Chen, W Wang, H Tian, S Ye, Z Gao, E Cui, W Tong… - arXiv preprint arXiv …, 2024

In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple …


How far can bias go?--Tracing bias from pretraining data to alignment

M Thaler, A Köksal, A Leidinger, A Korhonen… - arXiv preprint arXiv …, 2024

As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much work has gone into measuring or mitigating biases in these models, fewer studies have investigated …


How Gender Interacts with Political Values: A Case Study on Czech BERT Models

AA Ali, J Libovický - arXiv preprint arXiv:2403.13514, 2024

Neural language models, which reach state-of-the-art results on most natural language processing tasks, are trained on large text corpora that inevitably contain value-burdened content and often capture undesirable biases, which the models …


How Large Corpora Sizes Influence the Distribution of Low Frequency Text n-grams

JF Silva, JC Cunha - Pacific-Asia Conference on Knowledge Discovery and …, 2024

The prediction of the numbers of distinct word n-grams and their frequency distributions in text corpora is important in domains like information processing and language modelling. With big data corpora, there is an increased application …


How Lexical is Bilingual Lexicon Induction?

H Kohli, H Feng, N Dronen, C McCarter, S Moeini… - arXiv preprint arXiv …, 2024

… derived from Common Crawl and Wikipedia. The plot visualizes the Spearman’s Rank correlation of term frequencies between each of the source (by row) and target (by column) language pairs in the 5k vocabularies in the XLING corpus derived from …


How many news websites block AI crawlers?

In this factsheet we describe the proportion of news websites in ten countries that block AI (artificial intelligence) crawlers. We find that, (i) by the end 2023, 48% of the most widely used news websites across ten countries were blocking OpenAI’s (ChatGPT) …


How Many Van Goghs Does It Take to Van Gogh? Finding the Imitation Threshold

S Verma, R Rassin, AM Das, G Bhatt, P Seshadri… - Neurips Safe Generative AI …

… These datasets consist of semi-curated image-text pairs scraped from Common Crawl, 22 which leads to the inclusion of explicit, copyrighted, and licensed material [4, 9, 17, 20, 56]. Training 23 models on such images may be problematic because text-to-image …


How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library

M Ravaut, B Ding, F Jiao, H Chen, X Li, R Zhao, C Qin… - arXiv preprint arXiv …, 2024

… This includes policies and protocols for data privacy, consent, and use that help prevent the incorporation of contaminated data from unethical sources and the contamination of widely used pre-training data sources (eg, CommonCrawl) …


How much semantic information is available in large language model tokens?

… We first regressed MMLU score on the percentage of the Common Crawl that a language makes up and whether it uses the Latin alphabet… is a significant predictor when added to the model that includes Common Crawl percentage and …


How to Synthesize Text Data without Model Collapse?

X Zhu, D Cheng, H Li, K Zhang, E Hua, X Lv, N Ding… - arXiv preprint arXiv …, 2024

Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic data will fundamentally reshape the web data ecosystem. Future GPT-$\{n\} …


How to Train Data-Efficient LLMs

N Sachdeva, B Coleman, WC Kang, J Ni, L Hong… - arXiv preprint arXiv …, 2024

The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, ie, techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We …


How to Train Long-Context Language Models (Effectively)

T Gao, A Wettig, H Yen, D Chen - arXiv preprint arXiv:2410.02660, 2024

We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. We first establish a reliable evaluation protocol to guide model development -- Instead of perplexity or simple …


HPLT's First Release of Data and Models

N Arefyev, M Aulamo, P Chen, O de Gibert, B Haddow… - 2024

… Datasets For the first release, we processed 1.85 petabytes of the Internet Archive and CommonCrawl to create monolingual and parallel corpora. We release them under the permissive CC0 licence1 through our project website2, OPUS3, and Hugging …


Human vs. Machine: A Comparative Study on the Detection of AI-Generated Content

A Boutadjine, F Harrag, K Shaalan - ACM Transactions on Asian and Low-Resource …, 2024

… It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl data on one hundred languages including Arabic. The transfer learning process involves a meticulous fine-tuning procedure that was diligently …


Human-Centered Interaction in Virtual Worlds: A New Era of Generative Artificial Intelligence and Metaverse

Y Wang, L Wang, KL Siau - International Journal of Human–Computer Interaction, 2024

The metaverse has emerged as an exciting new paradigm for human-computer interaction (HCI) and virtual collaboration. This paper presents a comprehensive review of the metaverse to address the gap in the existing literature where there is a …


Human-Centered Natural Language Processing for Countering Misinformation

As curbing the spread of online misinformation has proven to be challenging, we look to artificial intelligence (AI) and natural language technology for helping individuals and society counter and limit it. Despite current advances, state-of-the-art …


Humkinar: Construction of a Large Scale Web Repository and Information System for Low Resource Urdu Language

… first Common Crawl Corpus (CCC) filter leverages the open-source Common Crawl repository to filter Urdu language webpages. Common Crawl… From August 2018 onward releases, Common Crawl also provides the meta information …


Hybrid Bilingual Question-Answering System with Human-Level Naturalness Text-to-Speech Extension

This project introduces a robust bilingual question-answering and natural Text-to-Speech (TTS) system, primarily designed to support both Vietnamese and English languages. Leveraging pre-trained Large Language Models (LLMs), the system enables …


Hybrid semantic models for building smart and robust home robots

The creation of home robots that can aid human beings in daily mundane chores has been a long-standing goal of robotics research. Some common indoor tasks that service robots can help with include retrieving objects from different locations …


Hybrid Tokenization Strategy for Turkish Abstractive Text Summarization

NZ Kayalı, Sİ Omurca - 2024 8th International Artificial Intelligence and Data …, 2024

Text summarization is a significant topic in natural language processing. Tokenization approaches are important in this regard as they underpin text recognition and processing. The aim of this paper is to research the efficiency of …


Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers

S Hwang, A Lahoti, T Dao, A Gu - arXiv preprint arXiv:2407.09941, 2024

A wide array of sequence models are built on a framework modeled after Transformers, comprising alternating sequence mixer and channel mixer layers. This paper studies a unifying matrix mixer view of sequence mixers that can be …


HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts

W Kim, S Chun, T Kim, D Han, S Yun - arXiv preprint arXiv:2404.17507, 2024

In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a …


Hypertext Entity Extraction in Webpage

Y Yang, T Liu, B Shao, H Zhao, L Shou, M Gong… - arXiv preprint arXiv …, 2024

… Common Crawl5 is a widely used large-scale dataset for IE. However, they all overlook rich hypertext features and contain too much noise. Moreover, their annotations are often determined by rules or automated tools, lacking accurate manual annotations. …


I'm In! A Comparative Analysis of Business Pitch Competition

E Barron, I Bernatovic, LE Ruiz - DIANA INTERNATIONAL RESEARCH …, 2023

Methods Based on a perceived theoretical gap in research, entrepreneurship scholars have called for increased research to understand how institutions (including governments and academia), influence the construction of intersectional …


Identification of patients' smoking status using an explainable AI approach: a Danish electronic health records case study

A Ebrahimi, MBH Henriksen, CL Brasen, O Hilberg… - BMC Medical Research …, 2024

Smoking is a critical risk factor responsible for over eight million annual deaths worldwide. It is essential to obtain information on smoking habits to advance research and implement preventive measures such as screening of high-risk …


Identification of Trolling Memes in Kannada and Tulu-Under-resourced Dravidian Languages

… Trained on a diverse dataset comprising Common Crawl and Wikipedia content across 157 languages, fastText employs Continuous Bag-of-Words (CBOW) with position weights, generating embeddings in a low-dimensional space of size 300 [44] …


Identifying DNS Scanners from a TLD Perspective

… Presumably popular ones include extracting domain names from public TLS certificate registrations using CT logs as well as using any kind of public list, for example from Common Crawl. CT logs are not directly related to the DNS ecosystem …


Identifying the sources of ideological bias in GPT models through linguistic variation in output

C Walker, JC Timoneda - arXiv preprint arXiv:2409.06043, 2024

Extant work shows that generative AI models such as GPT-3.5 and 4 perpetuate social stereotypes and biases. One concerning but less explored source of bias is ideology. Do GPT models take ideological stances on politically sensitive topics? In …


Identifying Threats on Social Media to Spot Offensive Behavior

AT Hridi, S Abdullah, MAI Hasnath, RH Adiba, S Proma… - 2024 IEEE 12th …, 2024

In the modern age, with the help of social media, communication has become available for everyone. Offensive text is broadly used in social media to humiliate or threaten someone. Offensive text like a bully, trolling, threats, and sexual …


iDRAMA-rumble-2024: A Dataset of Podcasts from Rumble Spanning 2020 to 2022

U Balci, J Patel, B Balci, J Blackburn - 2024

Rumble has emerged as a prominent platform hosting controversial figures facing restrictions on YouTube. Despite this, the academic community’s engagement with Rumble has been minimal. To help researchers address this gap, we introduce a …


If You Don't Understand It, Don't Use It: Eliminating Trojans with Filters Between Layers

A Hernandez - arXiv preprint arXiv:2407.06411, 2024

… In this model an LLM is pre-trained on a large web corpus such as Common Crawl, the model server tunes the model and inserts it into a (business) mission-critical system, and the model user uses this system. This system can be a chat-bot, code-completion …


IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining

D Feng, Y Zhang, Z Xu - arXiv preprint arXiv:2405.09857, 2024

Pretrained Large Language Models (LLM) such as ChatGPT, Claude, etc. have demonstrated strong capabilities in various fields of natural language generation. However, there are still many problems when using LLM in specialized domain-specific …


Imitation Attacks: Extracting and Exploiting Model Capabilities

… • Nepali→English: We query the Nepali Language Wikipedia (approximately 100,000 sentences) and around two million sentences from Nepali Common Crawl We train Transformer Big models on both datasets. In this attack, we replace certain …


Impact of COVID-19 Pandemic on Social Determinants of Health Issues of Marginalized Black and Asian Communities: A Social Media Analysis Empowered by …

C Whitfield, Y Liu, M Anwar - Journal of Racial and Ethnic Health Disparities, 2024

Purpose This study aims to understand the impact of the COVID-19 pandemic on social determinants of health (SDOH) of marginalized racial/ethnic US population groups, specifically African Americans and Asians, by leveraging natural language …


Improve the Accuracy and Efficiency of Large Language Models via Dynamic Token Compression and Adaptive Layer Pruning

F Potkins, O Langton, V Smirnov, C Gallacher… - 2024

… Experimental Setup The experiments utilized diverse and widely adopted datasets, such as WikiText and Common Crawl, chosen for their extensive coverage of linguistic features and varied textual content. These datasets allowed for a rigorous …


Improved methodology for longitudinal Web analytics using Common Crawl

HS Thompson - 16th ACM Web Science Conference 2024, 2024

Common Crawl is a multi-petabyte longitudinal dataset containing over 100 billion web pages which is widely used as a source of language data for sequence model training and in web science research. Each of its constituent archives is on the order …


Improving and Understanding Clarifying Question Generation in Conversational Search

D Ortega, S Söhnel, NT Vu - International Conference on Text, Speech, and …, 2024

Conversational information-seeking systems (CISs), such as chatbots and virtual personal assistants, encounter difficulty when processing ambiguous user requests (URs) and generate an accurate response, especially when multiple search results match …


Improving Arithmetical Reasoning of Language Models

This work focuses on improving language models for solving math word problems. First, we clean and transform existing datasets to create Calc-X, a standardized collection of 300,000 math problems with step-by-step solutions. We demonstrate …


Improving Bengali and Hindi Large Language Models

A Shahriar, D Barbosa - Proceedings of the 2024 Joint International Conference …, 2024

Despite being widely spoken worldwide, Bengali and Hindi are low-resource languages. The state-of-the-art in modeling such languages uses BERT and the Wordpiece tokenizer. We observed that the Wordpiece tokenizer often breaks words …


Improving Consumer Health Search with Field-Level Learning-to-Rank Techniques

In the area of consumer health search (CHS), there is an increasing concern about returning topically relevant and understandable health information to the user. Besides being used to rank topically relevant documents, Learning to Rank (LTR) …


Improving Cross-lingual Representation for Semantic Retrieval with Code-switching

M Maimaiti, Y Zheng, J Zhang, F Huang, Y Zhang… - arXiv preprint arXiv …, 2024

Semantic Retrieval (SR) has become an indispensable part of the FAQ system in the task-oriented question-answering (QA) dialogue scenario. The demands for a cross-lingual smart-customer-service system for an e-commerce platform or some particular …


Improving Parameter-Efficient Cross-Lingual Transfer for Low-Resource Languages

… The success of the XLM-R can largely be attributed to the quality and size of its training corpora the model is pretrained on more than two terabytes of the filtered Common Crawl data. Notable differences in comparison with mBERT (beyond much …


Improving Pretraining Data Using Perplexity Correlations

T Thrush, C Potts, T Hashimoto - arXiv preprint arXiv:2409.05816, 2024

Quality pretraining data is often seen as the key to high-performance language models. However, progress in understanding pretraining data has been slow due to the costly pretraining runs required for data selection experiments. We present a …


Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning

Z Nie, R Zhang, Z Feng, H Huang, X Liu - arXiv preprint arXiv:2406.18254, 2024

… For instance, XLM-R, CCR𝑘 ’s cross-lingual encoder, is trained on the 2.5TB CommonCrawl Corpus encompassing 100 languages. Discrepancies in data sizes between high-resource and low-resource languages within this corpus, like the …


INCLURE: a Dataset and Toolkit for Inclusive French Translation

P Lerner, C Grouin - The 17th Workshop on Building and Using …, 2024

… Indeed, this corpus mostly contains transcripts of political speeches, whose oral style differs from the text typically found in OSCAR/CommonCrawl. Exceptions are six examples used to illustrate the use of the inclusive neutralization process …


Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognition

K Buettner, S Malakouti, XL Li, A Kovashka - arXiv preprint arXiv:2401.01482, 2024

Existing object recognition models have been shown to lack robustness in diverse geographical scenarios due to significant domain shifts in design and context. Class representations need to be adapted to more accurately reflect an object concept …


Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic

G Verhulsdonck, J Weible, DM Stambler, T Howard… - Technical Communication, 2024

Purpose: As technical and professional communicators (TPCs) use AI to develop content, inaccuracies due to AI limitations are introduced; it is vital TPCs evaluate AI-generated content to improve accuracy and human-centeredness. In this article, we present a …


Incremental and Flexible Extraction of Parallel Corpus from the Web

X Liu, Y Zhen, D Li - 2024 3rd International Conference on Artificial …, 2024

… To avoid duplication of work and to prove the need for incremental extraction, we perform parallel sentence extraction based on the most recent archives of Common Crawl in 2023, and select Chinese and other representative languages as …


IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context

NR Sahoo, PP Kulkarni, N Asad, A Ahmad, T Goyal… - arXiv preprint arXiv …, 2024

The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the …


INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages

AK Singh, R Murthy, J Sen, G Ramakrishnan - arXiv preprint arXiv:2407.13522, 2024

… To address this, we sampled numerous Wikipedia and Common Crawl pages, focusing on paragraphs rich in cultural nuances and domain diversity. Using these paragraphs, we employed large language models (LLMs) to generate QA pairs, thus …


IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages

MSUR Khan, P Mehta, A Sankar, U Kumaravelan… - arXiv preprint arXiv …, 2024

Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive …


Indiscriminate Data Poisoning Attacks on Pre-trained Feature Extractors

Y Lu, MYR Yang, G Kamath, Y Yu - arXiv preprint arXiv:2402.12626, 2024

… To collect such large-scale datasets, practitioners usually extract the desired data by crawling on the internet (eg, web pages using Common Crawl1). However, using outsourced data raises an imminent security risk (Kumar et al. 2020; Nelson et al …


Individual-vs. Multiple-Objective Strategies for Targeted Sentiment Analysis in Finances Using the Spanish MTSA 2023 Corpus

R Pan, JA García-Díaz, R Valencia-García - Electronics, 2024

… complexity; (vi) RoBERTuito [22], a pretrained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets; and (vii) XLM-RoBERTa [17], a multilingual version of RoBERTa, trained with data …


IndoBerea: Evolving Semantic Search in Theological Context

FVP Samosir, S Mendrofa - 2023 Eighth International Conference on Informatics …, 2023

This paper presents IndoBerea, a semantic search model pre-trained on an Indonesian Bible dataset and based on SentenceTransformers and IndoBERT. It aims to enhance theological research by providing contextually relevant verses in …


Indonesian Health Question Multi-Class Classification Based on Deep Learning

WO Vihikan, INP Trisna - Journal of Information Systems and Informatics, 2024

… It has been trained on Indonesian Common Crawl and Wikipedia with a vector dimension of 300. A combination of RNN and CNN methods are also implemented, it is called BiLSTM-CNN and BiGRU-CNN. These two methods are connecting …


INDUS: Effective and Efficient Language Models for Scientific Applications

B Bhattacharjee, A Trivedi, M Muraoka… - arXiv preprint arXiv …, 2024

Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better …


Inference optimization of Large Language Models on RISC-V HPC platforms

Over the past decade, there have been significant improvements in Artificial Intelligence (AI), more particularly in the area of natural language processing (NLP) thanks to the emergence of Large Language Models (LLMs). These models are …


Inference-Based No-Learning Approach on Pre-Trained BERT Model Retrieval

HL Pham, R Mibayashi, T Yamamoto, MP Kato… - 2024 IEEE International …, 2024

In recent years, the practice of leveraging pre-trained machine learning models for specific tasks has gained traction. Instead of training models from the ground up, it is now common to fine-tune existing pre-trained models. However, users have the …


Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

N Yax, PY Oudeyer, S Palminteri - arXiv preprint arXiv:2404.04671, 2024

This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we …


InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning

X Han, Y Jian, X Hu, H Liu, Y Wang, Q Fan, Y Ai… - arXiv preprint arXiv …, 2024

… CommonCrawl [15] repository, preserving full multimodal web documents with interleaved images and text. We start with 44 snapshots of CommonCrawl … To achieve this, we apply language filtering to the CommonCrawl repositories, which …


Information Retrieval Chatbot on Military Policies and Standards

C Gunasekara, A Sharafeldin, M Triff, Z Kabir…

In the Canadian Armed Forces (CAF), navigating through extensive policies and standards can be a challenging task. To address the need for streamlined access to these vital documents, this paper explores the usage of artificial intelligence (AI) and …


Information Retrieval with Dense and Sparse Representations

… However, they use 1TB of the training data from Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining. We put their scores in Table 2.2 for reference. In SimCSE, the authors propose to use MLM as an auxiliary …


Infusing Prompts with Syntax and Semantics

AB Labate, FG Cozman - arXiv preprint arXiv:2412.06107, 2024

Despite impressive success, language models often generate outputs with flawed linguistic structure. We analyze the effect of directly infusing various kinds of syntactic and semantic information into large language models. To demonstrate the …


Initial exploration into sarcasm and irony through machine translation

ZL Chia, M Ptaszynski, M Karpinska, J Eronen, F Masui - Natural Language …, 2024

In this paper, we investigate sarcasm and irony as seen through a novel perspective of machine translation. We employ various techniques for translation, comparing both manually and automatically translated datasets of irony and sarcasm. We first …


InjectBench: An Indirect Prompt Injection Benchmarking Framework

… Many datasets such as CommonCrawl cover a much wider distribution of the types of website online compared to the few datasets we have selected. However, we have chosen to use the following datasets to better understand and emphasize …


Innovative Use of Self-Attention-Based Ensemble Deep Learning for Suicide Risk Detection in Social Media Posts

HS Choi, J Yang - Applied Sciences, 2024

… Specifically, we employed the extensive corpus variant, known as “Common Crawl (840 B tokens, 2.2 M vocab, cased, 300 d vectors)”, which encompasses an impressive 840 billion tokens, a vocabulary of 2.2 million cased terms, and provides …


Inquiring the'oracle'. An empirical study on how artificial intelligence literacy and prompt engineering influence the use of LLMs and GAI in higher education

The release of ChatGPT by OpenAl on November 2022 marked a significant paradigm shift in the landscape of human-artificial intelligence interaction (HAII); for it signified the accessibility of generative artificial intelligence (GAI) and, in particular …


Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoders

Y Xin, Z Li, N Yu, D Chen, M Fritz, M Backes, Y Zhang - arXiv preprint arXiv …, 2024

Despite being prevalent in the general field of Natural Language Processing (NLP), pre-trained language models inherently carry privacy and copyright concerns due to their nature of training on large-scale web-scraped data. In this paper, we pioneer a …


Inspecting and Measuring Fairness of unlabeled Image Datasets

R Görge, M Mock, M Akila - 2024 IEEE 40th International Conference on Data …, 2024

Bias in training data can lead to algorithmic unfairness in machine learning tasks. Therefore, a general requirement for trustworthy AI is that data should be representative and free of bias. There are several approaches to measure fairness …


Institutional challenges

F Geeraert, K de Wild, M Rockembach, J Winters… - The Routledge Companion …, 2024

Collections of archived web content can be a connecting factor between GLAM institutions. There is a large variety in the kinds of collections these institutions manage. However, with the increasing awareness about the importance of web …


Instruction-tuned Language Models are Better Knowledge Learners

Z Jiang, Z Sun, W Shi, P Rodriguez, C Zhou, G Neubig… - arXiv preprint arXiv …, 2024

… However, its scope is limited to Wikipedia, which restricts the trained models’ adaptability to other sources like web pages from Common Crawl or scientific documents from arXiv. We focus on eliciting factual knowledge with instruction-tuning …


Integrating Generative AI Literacy into the Information Retrieval Course at a university in Canada: towards critical evaluation of online search results

… • Trained on information (billions of words) on the open web prior to 2021 • Among other open sources, dataset comes from Common Crawl (crawls the web) and Wikipedia • Length of answers limited - between 500 and 700 words, “leaving …


Intelligent Pharmacy

The modern language generation model ChatGPT, created by Open Artificial Intelligence (AI), is recognised for its capacity to comprehend context and produce pertinent content. This model is built on the transformer architecture, which enables …


Intelligent Phishing Website Detection Model Powered by Deep Learning Techniques

U Chetachi, O Henry, OA Agbugba - Asian Journal of Research in Computer …, 2024

… Approximately one million authentic and fraudulent URLs were employed in the dataset gathered from PhishTank and Common Crawl. Over 10,000 images and one million URLs were used in training for the CNN classifier and LSTM to create the …


Intelligent question answering for water conservancy project inspection driven by knowledge graph and large language model collaboration

Y Yang, S Chen, Y Zhu, X Liu, S Pan, X Wang - LHB, 2024

Engineering inspection is of great significance to ensure the safe operation of the project. However, the unclear query statements of the detectors pose a challenge to the intelligent question answering task. Existing knowledge graph-based question-answering …


Interacting Large Language Model Agents. Interpretable Models and Social Learning

A Jain, V Krishnamurthy - arXiv preprint arXiv:2411.01271, 2024

This paper develops theory and algorithms for interacting large language model agents (LLMAs) using methods from statistical signal processing and microeconomics. While both fields are mature, their application to decision-making …


Intercity relationships between 293 Chinese cities quantified based on toponym co-occurrence

W Tongjing, Z Yin, Z Bao, E Meijers - Cybergeo: European Journal of Geography, 2024

… 14The primary dataset of this study was obtained from the Common Crawl, a web archive that has periodically crawled the Internet since … We used the entire Common Crawl text archive from April 2019 for processing and conducting …


International Journal of Cognitive Computing in Engineering

… MC4 dataset contains 101 language variants drawn from the Common Crawl web scrape. MT5 did not deviates much from the original T5 model and attempted to extend the capacities of the T5 to multilingual settings. T5 is a pre-trained language …


Internet-scale Topic Modeling using Large Language Models

… This proportion is estimated from the Common Crawl dataset’s statistics [28]. In short, the model should have an excellent understanding of English, an adequate understanding of languages such as German and Spanish, and be satisfactory at …


InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

H Ying, S Zhang, L Li, Z Zhou, Y Shao, Z Fei, Y Ma… - arXiv preprint arXiv …, 2024

The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We unify chain-of-thought …


InternLM2 Technical Report

Z Cai, M Cao, H Chen, K Chen, K Chen, X Chen… - arXiv preprint arXiv …, 2024

… Our web page data mainly comes from Common Crawl1. Firstly, we need to decompress the original Warc format files and use Trafilatura (… The entire filtering process eliminates a large proportion of Web page data (Common Crawl) and Patent …


Interpretable Syntactic Representations Enable Hierarchical Word Vectors

B Silwal - arXiv preprint arXiv:2411.08384, 2024

The distributed representations currently used are dense and uninterpretable, leading to interpretations that themselves are relative, overcomplete, and hard to interpret. We propose a method that transforms these word vectors into reduced …


Interpretable Tensor Fusion

S Varshneya, A Ledent, P Liznerski, A Balinskyy… - arXiv preprint arXiv …, 2024

Conventional machine learning methods are predominantly designed to predict outcomes based on a single data type. However, practical applications may encompass data of diverse types, such as text, images, and audio. We introduce …


Interpreting Themes from Educational Stories

Y Zhang, FA González, T Solorio - arXiv preprint arXiv:2404.05250, 2024

Reading comprehension continues to be a crucial research focus in the NLP community. Recent advances in Machine Reading Comprehension (MRC) have mostly centered on literal comprehension, referring to the surface-level …


Interrupted time series analysis of clickbait on worldwide news websites, 2016-2023

A McCutcheon, C Brogly - arXiv preprint arXiv:2408.06376, 2024

… This research provides an analysis of clickbait at the scale of the web on primarily English-language news sites from the Common Crawl. In this analysis, 3 of 5 selected newsgenerating events produced significant terms that suggest major …


Intersecting Register and Genre: Understanding the Contents of Web-Crawled Corpora

A Myntti, L Repo, E Freyermuth, A Kanner, V Laippala… - Proceedings of the 4th …, 2024

Web-scale corpora present valuable research opportunities but often lack detailed metadata, making them challenging to use in linguistics and social sciences. This study tackles this problem by exploring automatic methods to classify web corpora …


Intersectional Male-Centric and White-Centric Biases in Collective Concepts

AH Bailey, A Williams, A Poddar, A Cimpian - 2024

… Here we used NLP tools (specifically, static word embeddings trained on the Common Crawl corpus) to investigate collective understanding of three fundamental concepts: PERSON, WOMAN, and MAN.These three concepts organize much of …


Introducing the Djinni Recruitment Dataset: A Corpus of Anonymized CVs and Job Postings

N Drushchak, M Romanyshyn - Proceedings of the Third Ukrainian Natural …, 2024

This paper introduces the Djinni Recruitment Dataset, a large-scale open-source corpus of candidate profiles and job descriptions. With over 150,000 jobs and 230,000 candidates, the dataset includes samples in English and Ukrainian, thereby …


Invasion of ChatGPT and LLMs in Review and Writing Spaces: A Revolution or Radical

A Hadap, V Khatri - Scientific Publishing Ecosystem

Though AI flourished before 1970s but in the last five years, the invasion of artificial intelligence (AI) tools, such as ChatGPT and Large Language Model (LLMs), has received much attention in scientific research writing. These AI tools assist the …


Inverting Gradient Attacks Naturally Makes Data Poisons: An Availability Attack on Neural Networks

W Bouaziz, EM El-Mhamdi, N Usunier - arXiv preprint arXiv:2410.21453, 2024

… an analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Investigating Complex Answer Attribution Approaches with Large Language Models

This master’s thesis explores the attribution of answers in complex question-answering scenarios utilizing large language models (LLMs). The research aims to assess and enhance the traceability of answers back to their source documents, a critical aspect …


Investigating Conversational Agents to Support Secondary School Computer Science Exploratory Search

Online educational resources (eg, curricula, tutorials, documentation, Q&A sites) increasingly serve as key sources for secondary school students learning Computer Science Principles (CSP). A big obstacle to using these resources is finding …


Investigating Gender Bias in Turkish Language Models

O Caglidil, M Ostendorff, G Rehm - arXiv preprint arXiv:2404.11726, 2024


Investigating the Classification of radicalisation phases of Dutch social media messages upon physical threats

The increase of social media platforms such as Twitter and Telegram has transformed communication, enhanced free speech, but also facilitated the spread of harmful content. This rise in online threats against Dutch politicians highlights the …


Investigating the impact of pretraining corpora on the performance of Arabic BERT models

AS Alammary - The Journal of Supercomputing, 2025

… Akin to Multilingual BERT, the model underwent pretraining using 2.5 terabytes of data from Wikipedia and Common Crawl. The Arabic pretraining corpus is in MSA and comprises 2869 tokens, totaling approximately 28 gigabytes of data. The base …


Investigating the pre-training bias in low-resource abstractive summarization

D Chernyshev, B Dobrov - IEEE Access, 2024

Recent advances in low-resource abstractive summarization were largely made through the adoption of specialized pre-training, pseudo-summarization, that integrates the content selection knowledge through various centrality-based …


Investigating Word Embedding Techniques for Extracting Disease, Gene, and Chemical Relationships from Biomedical Texts

This thesis investigates word embedding models, including PubMedBERT, BioBERT, SkipGram, CBOW, and GloVe, in the context of Literature-Based Discovery (LBD) within biomedical research, with a specific focus on cancer-related entities. Firstly, I …


IPA Transcription of Bengali Texts

K Fatema, FD Haider, NF Turpa, T Azmal, S Ahmed… - arXiv preprint arXiv …, 2024

The International Phonetic Alphabet (IPA) serves to systematize phonemes in language, enabling precise textual representation of pronunciation. In Bengali phonology and phonetics, ongoing scholarly deliberations persist concerning the …


IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

DI Adelani, J Ojo, IA Azime, JY Zhuang, JO Alabi, X He… - arXiv preprint arXiv …, 2024

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (eg African languages) are often …


Is AI changing learning and assessment as we know it? Evidence from a ChatGPT experiment and a conceptual framework

O Kolade, A Owoseni, A Egbetokun - Heliyon, 2024

ChatGPT, a state-of-the-art chatbot built upon Open AI's generative pre-trained transformer, has generated a major public interest and caused quite a stir in the higher education sector, where reactions have ranged from excitement to …


Is Artificial Intelligence the Future of Collective Memory?

S Gensburger, F Clavert - Memory Studies Review, 2024

… ai tools are first and foremost memory products, as, at least for connectionist ai, they rely on the notion of training which implies the use of datasets – such as CommonCrawl, a sort of archive of the web – which are a product of human activities …


Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning

A Bandari, L Yin, CY Hsieh, AK Jaiswal, T Chen… - arXiv preprint arXiv …, 2024

Network pruning has emerged as a potential solution to make LLMs cheaper to deploy. However, existing LLM pruning approaches universally rely on the C4 dataset as the calibration data for calculating pruning scores, leaving its optimality …


Is ChatGPT taking over the language classroom? How language ideologies of large language models impact teaching and learning

M Lau - Working papers in Applied Linguistics and Linguistics …, 2024

… Thus, using datasets such as the Common Crawl can lead to the over-representation of English and the perspectives of English-speaking content … This means that LLMs that use Common Crawl for training, such as ChatGPT, may perform with …


Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search Engines

J Bevendorff, M Wiegmann, M Potthast, B Stein

Many users of web search engines have been complaining in recent years about the supposedly decreasing quality of search results. This is often attributed to an increasing amount of search-engineoptimized but low-quality content. Evidence for …


Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

C Liu, W Zhang, Y Zhao, AT Luu, L Bing - arXiv preprint arXiv:2403.10258, 2024

… We categorize languages larger than 1% frequency in Common Crawl2 as high-resource languages (ie, de, ru, fr, zh, es, ja, it and vi), and the rest as low-resource languages. We exclude English since we want to evaluate the efficient prompting strategy for …


It is Time to Develop an Auditing Framework to Promote Value Aware Chatbots

Y Wang, L Singh - arXiv preprint arXiv:2409.01539, 2024

… According to OpenAI, 60% of the training data come from Common Crawl, a large data set consisting of web pages, extracted metadata and text since 2008. Another 22% of data are from WebText2, containing all Reddit posts until December 2017 …


It's About Time: Incorporating Temporality in Retrieval Augmented Language Models

A Gade, J Jetcheva - arXiv preprint arXiv:2401.13222, 2024

… Additionally both models are fine-tuned jointly as a Fusion-in-Decoder [3] on common-crawl data using Wikipedia as a document index. The authors showed that Atlas performs well out-of-box and can be adapted to different tasks such as question-answering …


IUST at ClimateActivism 2024: Towards Optimal Stance Detection: A Systematic Study of Architectural Choices and Data Cleaning Techniques

G Mahmoudi, S Eetemadi - Proceedings of the 7th Workshop on Challenges and …, 2024

This work presents a systematic search of various model architecture configurations and data cleaning methods. The study evaluates the impact of data cleaning methods on the obtained results. Additionally, we demonstrate that a combination of …


J. Mitrović, M. Granitzer, University of Passau, Passau, Germany

M Dinzinger, S Zerhoudi - 6th International Open Search Symposium# …, 2024

… In addition to crawling, OWLer has ingested parts of publicly available dumps of Common Crawl to fill our crawl space with a broad range of seed URLs and provide a continuous output of crawled web documents despite the discontinuous …


Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent

Q Gallouédec, E Beeching, C Romac, E Dellandréa - arXiv preprint arXiv:2402.09844, 2024

The search for a general model that can operate seamlessly across multiple domains remains a key goal in machine learning research. The prevailing methodology in Reinforcement Learning (RL) typically limits models to a single task …


Japanese Pointer Network based Entity Linker for Wikidata

Y Sawamura, T Morita, S Egami, T Ugai, K Fukuda - 2023

… Pre-trained fastText is a model trained on Common Crawl and Wikipedia and provides pre-computed vectors for a word. The length of the POS Tags was changed from 36 to 18. The other five embeddings, namely Entity Embedding, Text Match …


JetMoE: Reaching Llama2 Performance with 0.1 M Dollars

Y Shen, Z Guo, T Cai, Z Qin - arXiv preprint arXiv:2404.07413, 2024

Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence. This report introduces JetMoE-8B …


Jina CLIP: Your CLIP Model Is Also Your Text Retriever

A Koukounas, G Mastrapas, M Günther, B Wang… - arXiv preprint arXiv …, 2024

H Xiao, G Mastrapas, B Wang - Multi-modal Foundation Model meets Embodied AI …

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks …


JiuZhang3. 0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models

K Zhou, B Zhang, J Wang, Z Chen, WX Zhao, J Sha… - arXiv preprint arXiv …, 2024

Mathematical reasoning is an important capability of large language models~(LLMs) for real-world applications. To enhance this capability, existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs (\eg GPT-4) …


JLBert: Japanese Light BERT for Cross-Domain Short Text Classification

C Kayal, S Chattopadhyay, A Gupta, S Abrol, A Gugol - Proceedings of the 2024 Joint …, 2024

… Second, there are limited models that have been trained on diverse datasets beyond Wikipedia and Common Crawl, limiting experimentation with other forms of text data. Moreover, with the English language being the focus of research interest …


JOANNA ZYLINSKA: Diffused Seeing

This article examines the transformation of the relationship between seeing and understanding in humans and machines by the technologies of machine learning known as ‘generative AI’. Taking Stable Diffusion as the main case study, while also …


Journal of Business Academy

This study examines in detail the potential of artificial intelligence (AI)-assisted ChatGPT in the field of accounting education and the opportunities and challenges that this utilisation may bring. ChatGPT can benefit by reducing error rates and …


Juru: Legal Brazilian Large Language Model from Reputable Sources

RM Junior, R Pires, R Romero, R Nogueira - arXiv preprint arXiv:2403.18140, 2024

The high computational cost associated with pretraining large language models limits their research. Two strategies have emerged to address this issue: domain specialization and pretraining with high-quality data. To explore these strategies, we …


Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text

S Meisenbacher, F Matthes - arXiv preprint arXiv:2405.19831, 2024

The study of Differential Privacy (DP) in Natural Language Processing often views the task of text privatization as a $\textit{rewriting}$ task, in which sensitive input texts are rewritten to hide explicit or implicit private information. In order to evaluate the …


JusticeAI: A Large Language Models Inspired Collaborative & Cross-Domain Multimodal System for Automatic Judicial Rulings in Smart Courts

NA Samee, M Alabdulhafith, SMAH Shah, A Rizwan - IEEE Access, 2024

… For GloVe embedding’s, which are static, we employed embedding’s pre-trained on large-scale datasets such as the Common Crawl or Wikipedia + Gig word corpus. For the dynamic embedding’s from BERT, ALBERT, RoBERTa, and Distilled BERT …


KazQAD: Kazakh Open-Domain Question Answering Dataset

R Yeshpanov, P Efimov, L Boytsov, A Shalkarbayuli… - arXiv preprint arXiv …, 2024

… Kaz-RoBERTa6 is a monolingual model trained on a collection of Kazakh texts from Common Crawl, the Leipzig Corpora Collection, the … XLM-R was pre-trained on a larger multilingual corpus derived from Common Crawl, in which Kazakh is …


KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes

R Yeshpanov, HA Varol - arXiv preprint arXiv:2403.19335, 2024

This paper presents KazSAnDRA, a dataset developed for Kazakh sentiment analysis that is the first and largest publicly available dataset of its kind. KazSAnDRA comprises an extensive collection of 180,064 reviews obtained from various sources …


KDPII: A New Korean Dialogic Dataset for the Deidentification of Personally Identifiable Information

L Fei, Y Kang, S Park, Y Jang, J Lee, H Kim - IEEE Access, 2024

… Using this framework, they examined the "Common Crawl's web crawl corpus," a frequently utilized resource for training LMs, to determine whether it contains high-risk personal information. They also compared various detection methods, including …


Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI

N Pangakis, S Wolken - arXiv preprint arXiv:2409.09467, 2024

… Because LLMs are trained on huge volumes of text data, using digital resources such as the Common Crawl, they are able to “memorize” publicly available information during the training process. This issue is compounded due to the fact …


KEPA-CRF: Knowledge expansion prototypical amortized conditional random field for few-shot event detection

R Wu, L Yu, S Tian, J Long, T Zhou, B Wang - Journal of Intelligent & Fuzzy Systems

Event Detection (ED) has long struggled with the ambiguous definition of event categories, making it challenging to accurately classify events. Previous endeavors aimed to tackle this problem by constructing prototypes for specific event categories …


Kernel Least Squares Transformations for Cross-Lingual Semantic Spaces

A Mištera, T Brychcín - International Conference on Text, Speech, and …, 2024

The rapid development in the field of natural language processing (NLP) and the increasing complexity of linguistic tasks demand the use of efficient and effective methods. Cross-lingual linear transformations between semantic spaces play a …


Key Algorithms for Keyphrase Generation: Instruction-Based LLMs for Russian Scientific Keyphrases

A Glazkova, D Morozov, T Garipov - arXiv preprint arXiv:2410.18040, 2024

Keyphrase selection is a challenging task in natural language processing that has a wide range of applications. Adapting existing supervised and unsupervised solutions for the Russian language faces several limitations due to the rich …


Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

N Chirkova, V Nikoulina - arXiv preprint arXiv:2402.12279, 2024

… For Intermediate tuning we finetune models for 100k steps on the CommonCrawl data uniformly sampled across all target languages and English, with the batch size of 5k tokens and the LR chosen to optimize fluency of model generations, inspected …


Knowledge Distillation with Applications to Interpretable Arabic Sentiment Analysis

A Diwali, K Saeedi, K Dashtipour, M Gogate, A Hussain - 2024

… These embeddings, trained on Common Crawl and Wikipedia data, were constructed using the CBOW model with position weights. The embeddings consist of 300-dimensional vectors featuring character n-grams of length 5, a context …


Knowledge Graph as Pre-training Corpus for Structural Reasoning via Multi-hop Linearization

W Kim, H Jung, W Kim - IEEE Access, 2024

Large language models have demonstrated exceptional performance across various natural language processing tasks. However, their reliance on unstructured text corpora for pre-training limits their effectiveness in tasks requiring structured …


Knowledge Induced Transformer Network for Causality Prediction

T Dasgupta, M Sinha, A Naskar - Companion Proceedings of the ACM on Web …, 2024

… The CausalBank dataset3 contains 314 million pairs of cause-effect statements scraped from the Common Crawl corpus using causal lexical patterns. Baselines: Most of the earlier work on causality detection is based on identifying cause, effect …


Knowledge Sources

M Jiang, BY Lin, S Wang, Y Xu, W Yu, C Zhu - Knowledge-augmented Methods for …, 2024

… To be more specific, the dataset includes Pile-CommonCrawl (227.12G, web crawling data), PubMed Central (90.27G, biomedical articles), Books (100.96G, a mix of fiction and nonfiction books), OpenWebText (262.77G, content from Reddit) …


Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models

R Vemulapalli, H Pouransari, F Faghri, S Mehta… - Forty-first International Conference …

… 2023) which contains 1.28B images filtered from Common Crawl (web-crawled data). We take 10 randomly augmented crops from each image and create a gallery of 12.8B images. See Appendix B.2 for additional details about this gallery set and …


Knowledge-augmented Methods for Natural Language Processing

Ever since language was invented and used, humans have leveraged this powerful tool to pass down experience over generations. These kinds of knowledge summarize precious findings and inspiring ideas, which constitute the human …


Knowledge-Enhanced Neural Networks for Machine Reading Comprehension

Machine Reading Comprehension is a language understanding task where a system is expected to read a given passage of text and typically answer questions about it. When humans assess the task of reading comprehension, in addition to the …


KOMPYUTER LINGVISTIKASIDA TA'MINOT MASALASI VA SINONIM BIRLIKLARNING LINGVISTIK TA'MINOTI

N SOTVOLDIYEVA - News of UzMU journal, 2024

… Lingvistik ma'lumotlar konsorsiumi (LDC) va Common Crawl[8] loyihasi kabi ochiq hamkorlik tashabbuslari lingvistik resurslarning mavjudligiga sezilarli hissa qo'shdi. Bu hamkorlik resurslar almashinuvini targ‘ib qiladi, ortiqchalikni kamaytiradi va …


Krishiq-BERT: A Few-Shot Setting BERT Model to Answer Agricultural-Related Questions in the Kannada Language

P Ajawan, V Desai, S Kale, S Patil - Journal of The Institution of Engineers (India) …, 2024

… corpora for 11 languages from two language families obtained primarily by sourcing articles from news crawls [3], and OSCAR (Open Super-large Crawled ALMAnaCH coRpus), a huge multilingual corpus procured by language …


KVP10k: A Comprehensive Dataset for Key-Value Pair Extraction in Business Documents

O Naparstek, R Pony, I Shapira, FA Dahood, O Azulai… - arXiv preprint arXiv …, 2024

… : extensive web data from Common Crawl and a collection of images from publicfiles.fcc.gov. From Common Crawl, we employed a systematic … A schematic describing the data acquisition process for the common crawl data is shown in Fig.2 …


L3i++ at SemEval-2024 Task 8: Can Fine-tuned Large Language Model Detect Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text?

HTH Tran, TN Nguyen, A Doucet, S Pollak - … of the 18th International Workshop on …, 2024

… Meanwhile, XLM-R is a multilingual version of RoBERTa that was pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. These models are also suggested as the baseline methods from SemEval2024 Task 8 organizers. …


Labadain-30k+: A Monolingual Tetun Document-Level Audited Dataset

G de Jesus, S Nunes - Proceedings of the 3rd Annual Meeting of the Special …, 2024

This paper introduces Labadain-30k+, a monolingual dataset comprising 33.6 k documents in Tetun, a low-resource language spoken in Timor-Leste. The dataset was acquired through web crawling and augmented with Wikipedia documents …


Label-dependent semantic associations for existing concepts

In a series of studies, we examined whether a label assigned to a cue concept (political event or its actor) modulates(a) its semantic associations (semantic effects) and (b) valence of the associations and attitude judgments (valence effects). We introduced …


Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Level

Z Feng, R Chen, Y Zhang, Z Meng, Z Liu - arXiv preprint arXiv:2406.15741, 2024

General-purpose Large Language Models (LLMs) like GPT-4 have achieved remarkable advancements in machine translation (MT) by leveraging extensive web content. On the other hand, translation-specific LLMs are built by pre-training on …


Language as a Lens: A Hybrid Text Summarization and Sentiment Analysis Approach for Multiclass Stock Return Prediction

F Balaneji - Intelligent Systems Conference, 2024

This research explores the application of text summarization and sentiment analysis techniques in the multiclass classification of hourly stock price returns, studying six companies from the Dow Jones Index between 2017 and the first quarter of 2020 …


Language as a Lens: A Hybrid Text Summarization and Sentiment Analysis Approach for Multiclass Stock Return

F Balaneji - Intelligent Systems and Applications: Proceedings of …


Language Model Frame Filling for Low Resource Languages

Acceptability judgment tasks are key tools used in the understanding of the grammar of languages by linguists. A typical way to construct these tasks is to use semantic frames. We provide evidence that the fill-mask objective for language models (LMs) …


Language Model-Driven Data Pruning Enables Efficient Active Learning

AH Azeemi, IA Qazi, AA Raza - arXiv preprint arXiv:2410.04275, 2024

Active learning (AL) optimizes data labeling efficiency by selecting the most informative instances for annotation. A key component in this procedure is an acquisition function that guides the selection process and identifies the suitable …


Language Models Fine-Tuning for Automatic Format Reconstruction of SEC Financial Filings

G Lombardo, G Trimigno, M Pellegrino, S Cagnoni - IEEE Access, 2024

The analysis of financial reports is a crucial task for investors and regulators, especially the mandatory annual reports (10-K) required by the SEC (Securities and Exchange Commission) that provide crucial information about a public company in …


Language Models for Online Depression Detection: A Review and Benchmark Analysis on Remote Interviews

R Qin, R Cook, K Yang, A Abbasi, D Dobolyi, S Seyedi… - ACM Transactions on …, 2024

… The project primarily uses data from the English CommonCrawl (67%) but also includes other public data sources including Wikipedia, GitHub, and ArXiv. LLaMA’s ability to outperform GPT-3 on the RACE Reading Comprehension benchmark …


Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

N Ljubešić, V Suchomel, P Rupnik, T Kuzman… - arXiv preprint arXiv …, 2024

The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters …


Language Models Pre-training

U Kamath, K Keenan, G Somers, S Sorenson - Large Language Models: A Deep Dive …, 2024

… The Common Crawl data is the most notable publicly available web scrape. … They demonstrated that these sources improved downstream performance over models trained on less diverse corpora such as Common Crawl. Taking this idea …


Language models scale reliably with over-training and on downstream tasks

SY Gadre, G Smyrnis, V Shankar, S Gururangan… - arXiv preprint arXiv …, 2024

Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal …


Large Concept Models: Language Modeling in a Sentence Representation Space

LCM The, L Barrault, PA Duquenne, M Elbayad… - arXiv preprint arXiv …, 2024

LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of LLMs is to process input and generate output at the token level. This is in sharp contrast to …


Large Language Model Powered In-House Question Answering Assistant

G Şahin, K Varol, BK Pak - 2024 Innovations in Intelligent Systems and …, 2024

Large language models are widely used in many natural language processing applications today. Automatic question answering is one of the areas where language models are frequently used. In this study, a question answering system …


Large Language Model-Based Chatbots in Higher Education

D Yigci, M Eryilmaz, AK Yetisen, S Tasoglu, A Ozcan - Advanced Intelligent Systems, 2024

Large language models (LLMs) are artificial intelligence (AI) platforms capable of analyzing and mimicking natural language processing. Leveraging deep learning, LLM capabilities have been advanced significantly, giving rise to generative …


Large Language Models (LLMs) on Tabular Data: Predic-tion, Generation, and Understanding-A Survey

X Fang, W Xu, FA Tan, J Zhang, Z Hu, Y Qi… - arXiv preprint arXiv …, 2024

Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


Large Language Models and Rule-Based Approaches in Domain-Specific Communication

D Halvoník, J Kapusta - IEEE Access, 2024

Currently, we are once again experiencing a frenzy related to artificial intelligence. Generative Pre-trained Transformers (GPT) models are highly effective at various natural language processing tasks. Different varieties of GPT models are widely …


Large Language Models and Sentiment Analysis in Financial Markets: A Review, Datasets and Case Study

C Liu, A Arulappan, R Naha, A Mahanti… - IEEE Access, 2024

This paper comprehensively examines Large Language Models (LLMs) in sentiment analysis, specifically focusing on financial markets and exploring the correlation between news sentiment and Bitcoin prices. We systematically categorize various …


Large Language Models and the Disappearing Private Sphere

A Cappello, M Dada, V Grigoreva, R Khan, C Stinson… - 2024

… The higher quality datasets are sampled more often during training, but the Reddit approved contents of CommonCrawl still … of CommonCrawl contains fan fiction, video game chats, conspiracy theories, pornography, junk advertising, and …


Large Language Models aren't all that you need

KV Holla, C Kumar, A Singh - arXiv preprint arXiv:2401.00698, 2024

This paper describes the architecture and systems built towards solving the SemEval 2023 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) [1]. We evaluate two approaches (a) a traditional Conditional Random …


LARGE LANGUAGE MODELS AS AN AID FOR SOFTWARE DEVELOPMENT USING GRAPHICAL PROGRAMMING LANGUAGE.

… In pre-training, the model is trained with the Common Crawl dataset, internet books, and Python code, summing up to 400B tokens. After the model is pre-trained, the phase preference model pre-training comes, and after that, the preference model …


Large Language Models As Genetic Counselors: AI-DRIVEN TOOLS FOR HEALTH MANAGEMENT

This thesis, in collaboration with Arkus-AI, explores the development of an AI-driven genetic counselor designed to enhance genetic counseling services at Eiira. By utilizing advanced large language models, this project aims to improve the quality …


Large Language Models As Genetic Counselors: Personalized AI Driven Tool for Health Management


Large language models as oracles for instantiating ontologies with domain-specific knowledge

G Ciatto, A Agiollo, M Magnini, A Omicini - arXiv preprint arXiv:2404.04108, 2024

… Models following the RoBERTa [30] approach were trained on (i) the same datasets as BERT, plus (ii) a sample of the CommonCrawl4 News dataset (a dump of news … 4https://commoncrawl.org 5https://mistral.ai/news/mixtral-of-experts …


Large Language Models Can Not Perform Well in Understanding and Manipulating Natural Language at Both Character and Word Levels?

Y Zhang, Z He - Findings of the Association for Computational …, 2024

Despite their promising performance across various tasks, recent studies reveal that Large language models (LLMs) still exhibit significant deficiencies in handling several word-level and character-level tasks, eg, word unscrambling and sentence …


Large Language Models for Biomedical Text Simplification: Promising But Not There Yet

Z Li, S Belkadi, N Micheletti, L Han, MSG Nenadic

… the common crawl corpus and filtered to keep only natural text and de-duplication processing. They extracted 750GB of clean English data to feed into the model for multi-task pretraining. Different masking strategies are integrated into the T5 model …


Large language models for biomedicine: foundations, opportunities, challenges, and best practices

SS Sahoo, JM Plasek, H Xu, Ö Uzuner, T Cohen… - Journal of the American …, 2024

… In the pre-training phase, generic training data, such as the Common Crawl dataset or social media posts, are used for unsupervised learning.With further training on prompt-response pairs (such as the Alpaca dataset developed at Stanford23) …


Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review

J Su, C Jiang, X Jin, Y Qiao, T Xiao, H Ma, R Wei… - arXiv preprint arXiv …, 2024

… The training data for GPT-3 includes the Common Crawl dataset for lower quality, as well as the WebText2 dataset for higher quality, as well as the Books1, Books2, and Wikipedia datasets for higher quality [148]. GPT-3 assigns different weights to …


Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language

J Kaiser, A Eichler, A Lauscher - arXiv preprint arXiv:2405.08888, 2024

Autonomous tuning of particle accelerators is an active and challenging field of research with the goal of enabling novel accelerator technologies cutting-edge high-impact applications, such as physics discovery, cancer research and material sciences. A …


Large Language Models for identification of medical data in unstructured records

… Sources for the training data include web pages extracted by CommonCrawl, opensource GitHub repositories, Wikipedia in twenty languages, Project Gutenberg books in the public domain, LaTeX source code of scientific articles in ArXiv and …


Large Language Models for Relevance Judgement in Table Retrieval

… The tables come from the English subset of the Web Data Commons (WDC) table corpus 2015, which contains 50.8M HTML tables from the July 2015 Common Crawl The creators of the WTR collection further filtered the collection by identifying entities …


Large Language Models for Reliable Information Extraction

This project addresses the ongoing challenge of achieving reliable Information Extraction (IE), particularly in domains which require near-perfect precision. ELICIT (Butcher et al., 2023) introduced a novel approach that combines the processing speed of …


Large Language Models for Software Engineering: A Systematic Mapping Study

MK Görmez, M Yılmaz, PM Clarke - European Conference on Software Process …, 2024

In this research, we aim to conduct a systematic mapping study on Large Language Models (LLMs) for Software Engineering (SE). The significantly enhanced capabilities of LLMs have led to their use in many fields, including the important …


LARGE LANGUAGE MODELS FOR TEXT CLASSIFICATION: FROM ZERO-SHOT LEARNING TO INSTRUCTION-TUNING

Advances in large language models (LLMs) have transformed the field of natural language processing and have enormous potential for social scientific analysis. We explore the application of LLMs to supervised text classification. As a case study, we …


LARGE LANGUAGE MODELS IN ECONOMICS

This chapter explores the transformative impact of large language models (LLMs) on text analysis in economics. We trace the evolution from traditional methods like bag-of-words to advanced models such as BERT and GPT, highlighting how these models …


Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

A Moglia, K Georgiou, P Cerveri, L Mainardi… - Artificial Intelligence Review, 2024

Large language models (LLMs) have the intrinsic potential to acquire medical knowledge. Several studies assessing LLMs on medical examinations have been published. However, there is no reported evidence on tests related to robot-assisted …


Large Language Models on Tabular Data--A Survey

Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table …


Large Language Models Understand Layouts

W Li, M Duan, D An, Y Shao - arXiv preprint arXiv:2407.05750, 2024

… We find that GLM, Llama, and GPT-3 all use datasets such as CommonCrawl, Wikipedia, and Books (Pile includes CommonCrawl, Wikipedia, and Books) in their pretraining. CommonCrawl is a large-scale, unstructured, multilingual web dataset …


Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation

M Wysocka, O Wysocki, M Delmas, V Mutel, A Freitas - Journal of Biomedical …, 2024

Objective: The paper introduces a framework for the evaluation of the encoding of factual scientific knowledge, designed to streamline the manual evaluation process typically conducted by domain experts. Inferring over and extracting information from …


Large Language Models: A Case Study on Corporate Values and Transformative Technologies

Design (VSD) a technique that provides theories and methods to systematically “account for human values… throughout the design process”(Value Sensitive Design Lab, nd). This paper employs VSD as a theoretical lens to dissect the dual challenges of …


Large Language Models: A Deep Dive

U Kamath, K Keenan, G Somers, S Sorenson

In the panorama of technological evolution, Large Language Models (LLMs) have emerged as a cornerstone, transforming our interaction with information, reshaping industries, and redefining the boundaries of artificial intelligence. As we stand on the …


Large Language Models: A Survey

S Minaee, T Mikolov, N Nikzad, M Chenaghlu… - arXiv preprint arXiv …, 2024

… Common Crawlbased dataset consisting of texts in 101 languages. … CommonCrawl. They also released an extract of 600 billion tokens from our REFINEDWEB dataset, and 1.3/7.5B parameters language models trained on it. 27 …


Large Language Models: An Introduction

U Kamath, K Keenan, G Somers, S Sorenson - Large Language Models: A Deep Dive …, 2024

This chapter begins with a discussion of the historical context and progression of natural language processing. Beginning with the origins of human linguistic capabilities, this chapter explains the gradual transition to computational language …


Large Language Models: Ethics and Norms in the European Union

M Ciullo - 2024 IEEE International Conference on Metrology for …, 2024

This paper investigates the ethical-legal issues underlying the implementation of Large Language Models, in the aftermath of the European Union Parliament's approval of the Artificial Intelligence Regulation. In very recent years, there has been …


Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding

H Zolkepli, A Razak, K Adha, A Nazhan - arXiv preprint arXiv:2401.13565, 2024

… An analysis of the widely utilized Common Crawl dataset reveals a mere 0.0742% contribution from the Malay language based on CCMAIN-… We also replicated the same technique for Malaysia Hansard and MS CommonCrawl samples. All synthetic …


Large Scale Transfer Learning for Tabular Data via Language Modeling

J Gardner, JC Perdomo, L Schmidt - arXiv preprint arXiv:2406.12031, 2024

… Given that T4 consists of 3.1Mtables sourced from public data sources (Common Crawl, Github) and that our evaluations are also comprised of public benchmarks, we investigate the extent and possible impact of data contamination – that is …


Large Vocabulary Size Improves Large Language Models

S Takase, R Ri, S Kiyono, T Kato - arXiv preprint arXiv:2406.16508, 2024

This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary …


Large-language models facilitate discovery of the molecular signatures regulating sleep and activity

D Peng, L Zheng, D Liu, C Han, X Wang, Y Yang… - Nature Communications, 2024

Sleep, locomotor and social activities are essential animal behaviors, but their reciprocal relationships and underlying mechanisms remain poorly understood. Here, we elicit information from a cutting-edge large-language model (LLM), generative pre-trained …


Large-Language-Models (LLM)-Based AI Chatbots: Architecture, In-Depth Analysis and Their Performance Evaluation

V Kumar, P Srivastava, A Dwivedi, I Budhiraja… - International Conference on …, 2023

… LLaMA models span from 7 billion to 65 billion parameters and were trained on trillions of tokens derived from publicly accessible datasets, including Common Crawl and Wikipedia. The primary objective of LLaMA models is to serve as …


Large-scale machine unlearning in transformer-based pre-trained language models: Evaluation and adaptation of existing machine unlearning frameworks KGA and …

In the past few years, Machine Learning (ML) models such as GPT-4, utilized by the prominent chatbot ChatGPT, have faced data breaches, raising concerns about data privacy and awareness about the "right to be forgotten." Consequently, the concept …


Large-scale Summarization of Chat Transcripts in the Absence of Annotated Summaries

PK Biswas - Proceedings of the 7th International Conference on …, 2024

Text summarization is the process of condensing a piece of text to fewer sentences, while still preserving its content. Chat transcript, in this context, is a textual copy of a digital/web conversation between a customer (caller) and agent (s). This paper …


LASERBEAK: Evolving Website Fingerprinting Attacks with Attention and Multi-Channel Feature Representation

N Mathews, JK Holland, N Hopper, M Wright - IEEE Transactions on Information …, 2024

In this paper, we present LASERBEAK, a new state-of-the-art website fingerprinting attack for Tor that achieves nearly 96% accuracy against FRONT-defended traffic by combining two innovations: (i) multi-channel traffic representations and (ii) advanced …


Latent mechanisms of language disorganization relate to specific dimensions of psychopathology

I Fradkin, RA Adams, N Siegelman, R Moran, RJ Dolan - Nature Mental Health, 2024

Comprehensible communication is critical for social functioning and well-being. In psychopathology, incoherent discourse is assumed to reflect disorganized thinking, which is classically linked to psychotic disorders. However, people do not express …


Latxa: An Open Language Model and Evaluation Suite for Basque

J Etxaniz, O Sainz, N Perez, I Aldabe, G Rigau, E Agirre… - arXiv preprint arXiv …, 2024

We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque corpus comprising 4.3M documents and 4.2B tokens. Addressing the …


Law and the Political Economy of AI Production

The governance of Artificial Intelligence (AI) is at a historical juncture. Legislative acts, global treaties, export controls, and technical standards are now dominating the discourse over what used to be a predominantly market-driven space. Amidst all this …


Law, Technology and Humans

… Around 60% of GPT-3 training data comes from a filtered version of Common Crawl—that is, a non-profit organisation that archives and provides open-access petabytes of data collected since 2011. [30] The remaining 40% comes from diverse …


Law/Juridiskā zinātne: The Journal of the University of Latvia, N 17

The goal of this research is to provide recommendations for the improvement of both national legislation and the practice of public prosecution in the suppression of corruption. In the paper, authors started from the assumption that the effectiveness of …


LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdicts

L Ragazzi, G Moro, S Guidi, G Frisoni - Artificial Intelligence and Law, 2024

Large-scale public datasets are vital for driving the progress of abstractive summarization, especially in law, where documents have highly specialized jargon. However, the available resources are English-centered, limiting research …


LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics

T Robert, M Safaryan, IV Modoranu, D Alistarh - arXiv preprint arXiv:2410.16103, 2024

We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during training. This strategy keeps …


Learning entity and relation representation for low-resource medical language processing

Recent advancements in natural language processing have led to growing interest in critical domains such as legal, finance, and healthcare. In particular, medical language processing has emerged as a focus of its own. As the number of …


Learning Fine-Grained Grounded Citations for Attributed Large Language Models

L Huang, X Feng, W Ma, Y Gu, W Zhong, X Feng, W Yu… - arXiv preprint arXiv …, 2024

… 2021), a pre-processed and cleaned version of the Common Crawl corpus, serving as a proxy web search index. In particular, for a given user query sampled from the AQuAMuSe dataset, we initially retrieve the top 100 relevant documents …


Learning from Noisy Data for Cross Lingual Text Generation in Low-Resource Languages

With Large Language Models (LLMs) and Language models in general becoming a more significant part of our daily content consumption, it is paramount to ensure that languages with fewer resources do not get excluded. As most language models are …


Learning to Generate Answers with Citations via Factual Consistency Models

R Aly, Z Tang, S Tan, G Karypis - arXiv preprint arXiv:2406.13124, 2024

Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations. One approach to address this issue is to provide citations to relevant sources alongside generated content, enhancing the verifiability of …


Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA

N Roy, LFR Ribeiro, R Blloshmi, K Small - arXiv preprint arXiv:2409.15515, 2024

… In particular, we use 54M passages from Wikipedia and Common Crawl as the knowledge base.We use an off-the-shelf Contriever model trained on MS-MARCO as the retriever.. The retrieved passages are used by SELFmulti-RAG to generate …


Learning with Noisy Foundation Models

H Chen, J Wang, Z Wang, R Tao, H Wei, X Xie… - arXiv preprint arXiv …, 2024

… GPT2 is pre-trained on WebText [29], a scraped web dataset from Common Crawl that contains low-quality raw texts. We also leverage OpenAI’s API service “text-ada-002”. We cannot use larger and more recent language models such as LLaMA [30], since …


Leveraging AI to Deliver Culturally Responsive Mental Health Care at Scale

A Cerezo, D Cooper, V Palat, A Jolley, S Peregrine

… One of the most commonly used datasets to train LLMs is The Common Crawl dataset, a large collection of web pages scraped from the internet since 2014. These pages contain the gamut from high-quality news, science, and literature to the …


Leveraging distant supervision and deep learning for twitter sentiment and emotion classification

M Kastrati, Z Kastrati, A Shariq Imran, M Biba - Journal of Intelligent Information …, 2024

Nowadays, various applications across industries, healthcare, and security have begun adopting automatic sentiment analysis and emotion detection in short texts, such as posts from social media. Twitter stands out as one of the most popular …


Leveraging investments, promoting transparency and mobilising communities: a qualitative analysis of news articles about how the Ebola outbreak informed COVID …

LP Courtney, M Billaud, A Paulenich, R Chew… - BMJ Global Health, 2024

… Articles were compiled from the Common Crawl archives,24 an open-source database of historical web pages across local, regional, and international news outlets, extracted using the Python ‘news-please’ library25 and translated into …


Leveraging Large Language Models for Code-Mixed Data Augmentation in Sentiment Analysis

L Zeng - arXiv preprint arXiv:2411.00691, 2024

Code-mixing (CM), where speakers blend languages within a single expression, is prevalent in multilingual societies but poses challenges for natural language processing due to its complexity and limited data. We propose using a large …


Leveraging Large Language Models for Fact Verification in Italian

A Scaiella, S Costanzo, E Passone, D Croce… - 2024

… For the automatic translation process, we utilized MADLAD400 [20], a machine translation system based on the Transformer architecture3, trained on MADLAD, a manually audited, general domain 3T token multilingual dataset based on …


Leveraging large language models for word sense disambiguation

JH Yae, NC Skelly, NC Ranly, PM LaCasse - Neural Computing and Applications, 2024

Natural language processing (NLP) is difficult because human language contains ambiguity. The same word can have a different meaning depending on the context and may result in different interpretations given biases held by a NLP technique …


Leveraging Large Language Models to Measure Gender Bias in Gendered Languages

E Derner, SS de la Fuente, Y Gutiérrez, P Moreda… - arXiv preprint arXiv …, 2024

Gender bias in text corpora used in various natural language processing (NLP) contexts, such as for training large language models (LLMs), can lead to the perpetuation and amplification of societal inequalities. This is particularly …


Leveraging LLMs for Collaborative Ontology Engineering in Parkinson Disease Monitoring and Alerting

G Bouchouras, D Doumanas, A Soularidis, K Kotis…

This paper explores the integration of Large Language Models (LLMs) in the engineering of a Parkinson’s Disease (PD) monitoring and alerting ontology through four key methodologies: One Shot (OS) prompt techniques, Chain of Thought (CoT) …


Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach

M Sebők, Á Máté, O Ring, V Kovács, R Lehoczki - Social Science Computer Review, 2024

The article presents an open-source and freely available natural language processing system for comparative policy studies. The CAP Babel Machine allows for the automated classification of input files based on the 21 major policy topics of …


Leveraging Parameter Efficient Training Methods for Low Resource Text Classification: A case study in Marathi

P Deshmukh, N Kulkarni, S Kulkarni, K Manghani… - 2024 IEEE 9th International …, 2024

With the surge in digital content in low-resource languages, there is an escalating demand for advanced Natural Language Processing (NLP) techniques tailored to these languages. BERT (Bidirectional Encoder Representations from Transformers) …


Leveraging Social Media and Deep Learning for Sentiment Analysis for Smart Governance: A Case Study of Public Reactions to Educational Reforms in Saudi Arabia

The Saudi government’s educational reforms aim to align the system with market needs and promote economic opportunities. However, a lack of credible data makes assessing public sentiment towards these reforms challenging. This research …


Leveraging Social Media Data For More Comprehensive Traffic Load Prediction

Traffic congestion is a major problem in developing countries, where urban space is often limited. With the growing urban population and increasing number of vehicles, it has become essential to manage the existing road networks efficiently. In the past …


Leveraging Universal Speech Representations for Detecting and Assessing the Severity of Mild Cognitive Impairment Across Languages

A Favaro, T Cao, N Dehak, L Moro-Velazquez

This study examines the suitability of language-agnostic features for automatically detecting Mild Cognitive Impairment (MCI) and predicting Mini-Mental State Examination (MMSE) scores in a multilingual framework. We explored two methods …


Leveraging Zero-Shot Prompting for Efficient Language Model Distillation

L Vöge, V Gurgul, S Lessmann - arXiv preprint arXiv:2403.15886, 2024

… Specifically, we use the T5 v1.1 models which were only pretrained on the C4 Common Crawl dataset for general language understanding without additional task-specific finetuning. The use of text-to-text models allows for straightforward processing, as …


LexEval: A Scalable LLM Evaluation Framework

As Large Language Models (LLMs) become more powerful and their parameter counts increase, it is crucial for them to exhibit consistent recall capabilities for terms that appear less frequently in their pre-training data. The importance of accurate …


Lib2Life–Digital Library Services Empowered with Advanced Natural Language Processing Techniques

M Nitu, M Dascalu, MD Dascalu, LM Neagu…

Educational institutions are struggling to keep up with the accelerated technological advancements; hence, sustainable and supportive tools have become essential to reshape traditional models into intelligent learning systems. This paper introduces …


LiBERTa: Advancing Ukrainian Language Modeling through Pre-training from Scratch

M Haltiuk, A Smywiński-Pohl - Proceedings of the Third Ukrainian Natural Language …, 2024

… CC-100, a multilingual text corpus sourced from Wikipedia and CommonCrawl, was processed following the CCNet1 methodology. The Ukrainian segment of CC-100 encompasses 6.5 billion tokens, equivalent to 84 GiB of data2. This corpus primarily …


LIBRARIES, ARCHIVES, AND THE BORN-DIGITAL

P Gooding - The Routledge Companion to Libraries, Archives, and …, 2024

The material record of contemporary culture is increasingly being created in digital form. While decades in the making, this shift has accelerated in recent years as communications, work, commerce, and creativity are mediated by computers. We …


Likelihood as a Performance Gauge for Retrieval-Augmented Generation

T Liu, J Qi, P He, A Bisazza, M Sachan, R Cotterell - arXiv preprint arXiv:2411.07773, 2024

… 2021)—a filtered version of Common Crawl9, where k is taken to be {5,10,20} to avoid truncation due to the long questions and LMs responses for the long-form QA task. In contrast to NQ-Open, ELI5 does not provide the annotations of gold …


Limitations of Language Models in The Oil & Gas Upstream Operations

A Alsultan, FA Razak - International Petroleum Technology Conference, 2024

… Although the process is simple in principle, it is done on very large training corpra which can be as big as dataset provided by the Common Crawl project (428.81 TB as of the date of writing this paper). Thus, training these models using these large …


Limits to Predicting Online Speech Using Large Language Models

M Remeli, M Hardt, RC Williamson - arXiv preprint arXiv:2407.12850, 2024

We study the predictability of online speech on social media, and whether predictability improves with information outside a user's own posts. Recent work suggests that the predictive information contained in posts written by a user's peers …


LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages

AM Bean, S Hellsten, H Mayne, J Magomere, EA Chi… - arXiv preprint arXiv …, 2024

… ], while the second and fourth rows are sorted by Common Crawl Share [16]. The upper two … of pages tagged with a given language in the Common Crawl dataset [16]. Figure 12 show the … LLMs are trained on, using the Common Crawl share and …


Linguistic Profiling of Deepfakes: An Open Database for Next-Generation Deepfake Detection

Y Wang, Z Huang, Z Ma, X Hong - arXiv preprint arXiv:2401.02335, 2024

The emergence of text-to-image generative models has revolutionized the field of deepfakes, enabling the creation of realistic and convincing visual content directly from textual descriptions. However, this advancement presents considerably greater …


Linguistic Structure from a Bottleneck on Sequential Information Processing

R Futrell, M Hahn - arXiv preprint arXiv:2405.12109, 2024

… Table 3: English expressions for the given meanings, along with frequencies from the English Common Crawl web corpus [64]. Example … n-gram counts and language models from the common crawl. In Language Resources and Evaluation …


Linguistic variation beyond the Indo-European web: Analyzing Turkish web registers in TurCORE

S Erten-Johansson, V Skantsi, S Pyysalo, V Laippala - Register Studies, 2024

… To ensure that the documents in TurCORE do not retain any structure from the original CommonCrawl datasets, we extracted the TurCORE … CommonCrawl data are typically very noisy due to the automatic processes involved in crawling and …


Linguistic_Hygenist at PAN 2024 TextDetox: HybridDetox-A Combination of Supervised and Unsupervised Methods for Effective Multilingual Text Detoxification

S Gangopadhyay, MT Khan, H Jabeen - 2024

… To handle these, we used the XLM-RoBERTa large model [25], which is pretrained in a self-supervised manner on 2.5TB of filtered CommonCrawl data spanning 100 languages, including all languages featured in the competition. The model employs a …


Linguistically-Informed Multilingual Instruction Tuning: Is There an Optimal Set of Languages to Tune?

G Soykan, GG Şahin - arXiv preprint arXiv:2410.07809, 2024

Multilingual language models often perform unevenly across different languages due to limited generalization capabilities for some languages. This issue is significant because of the growing interest in making universal language models …


Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval

DY Hwang, B Taha, H Pande, Y Nechaev - arXiv preprint arXiv:2410.18385, 2024

Despite the recent advancements in information retrieval (IR), zero-shot IR remains a significant challenge, especially when dealing with new domains, languages, and newly-released use cases that lack historical query traffic from existing users. For …


LITS: An Optimized Learned Index for Strings (An Extended Version)

Y Yang, S Chen - arXiv preprint arXiv:2407.11556, 2024

Index is an important component in database systems. Learned indexes have been shown to outperform traditional tree-based index structures for fixed-sized integer or floating point keys. However, the application of the learned solution to variable-length …


LITS: An Optimized Learned Index for Strings

Y Yang, S Chen - Proceedings of the VLDB Endowment, 2024


Living in Digital Ecosystems: Are We Aware of This?

S Cozzini, M de Luca - Digital Environments and Human Relations, 2024

… , we briefly discuss the Common Crawl case, which has already been extensively analysed (Baack and Insights, 2024). Common Crawl is a … Recently, Common Crawl dataset has become one of the most important sources of training for …


Living in Digital Ecosystems: Are

S Cozzini, M de Luca - Digital Environments and Human Relations: Ethical …