Asking GPT for the Ordinary Meaning of Statutory Terms

C Engel, RH McAdams - MPI Collective Goods Discussion Paper, 2024

We report on our test of the Large Language Model (LLM) ChatGPT (GPT) as a tool for generating evidence of the ordinary meaning of statutory terms. We explain why the most useful evidence for interpretation involves a distribution of replies rather …


Asking Questions Framework for Oral History Archives

J Švec, M Bulín, A Frémund, F Polák - European Conference on Information Retrieval, 2024

… The resulting Wav2Vec 2.0 end-to-end model was combined with a lower-case four-gram language model estimated from CommonCrawl data. To decrease the model’s … The model was pre-trained on a self-supervised text-restoration task …


Assessing and Optimizing Large Language Models on Spondyloarthritis Multi-Choice Question Answering: Protocol for Enhancement and Assessment

A Wang, Y Wu, X Ji, X Wang, J Hu, F Zhang, Z Zhang… - JMIR Research Protocols, 2024

Background Spondyloarthritis (SpA), a chronic inflammatory disorder, predominantly impacts the sacroiliac joints and spine, significantly escalating the risk of disability. SpA’s complexity, as evidenced by its diverse clinical presentations and symptoms …


Assessing Task-Specific Performance Gains from Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models

This study investigates the effectiveness and economic feasibility of fine-tuning Large Language Models (LLMs). With a focus on the open-source LLaMA-2 model of Touvron, Martin et al.(2023), Parameter-Efficient Fine-Tuning (PEFT) is used in …


ASSESSING THE IMPACT OF PROMPTING TECHNIQUES ON SHORT-TERM CURRICULUM DESIGN: A COMPARATIVE APPROACH

L Paz, M Berg, S Kreibich, D Werth - ICERI2024 Proceedings, 2024

Since the public release of Large Language Models (LLMs) such as ChatGPT, the education landscape has witnessed groundbreaking innovations in personalized learning, automated content creation, and improved accessibility. Although previous …


Assessing the potential of leveraging LLaMA-2 to create an institute-specific online chatbot assistant.

With the increasing popularity of chatbots such as ChatGPT, when faced with questions, people are getting more and more inclined to opt for a chatbot instead of browsing, calling, or travelling and waiting in line at service points. 8 To address this …


Assessing the research landscape and clinical utility of large language models: a scoping

YJ Park, A Pillai, J Deng, E Guo, M Gupta, M Paget… - 2024

… Introduced in November 2022, ChatGPT was trained using a large corpora of unlabelled text, including CommonCrawl, WebText, and Wikipedia, as well as internet-based book corpora spanning multiple languages [3]. GPT, along with other …


Assessing the Role of Imagery in Multimodal Machine Translation

NK Motlagh, J Davis, T Anderson, J Gwinnup…

… 2023), which comprises 12.8 million image-text pairs harvested from the Common Crawl (Common Crawl). The DataComp dataset serves as a foundation dataset for enhancing the training of CLIP models. We employ a CLIP ViT-B/32 model …


Assessment of beliefs and attitudes towards benzodiazepines using machine learning based on social media posts: an observational study

L de Anta, MÁ Alvarez-Mon, V Pereira-Sanchez… - BMC Psychiatry, 2024

Benzodiazepines are frequently prescribed drugs; however, their prolonged use can lead to tolerance, dependence, and other adverse effects. Despite these risks, long-term use remains common, presenting a public health concern. This study aims to …


ASSESSMENT OF HARMFULNESS OF INSTRUCTION-TUNED LARGE LANGUAGE MODEL LLAMA 2 BY SELF-EVALUATION

The relatively recent introduction of public large language models like ChatGPT has raised numerous concerns regarding the ethical issues surrounding their application. Their general proficiency in performing natural language tasks, rivaling that of …


Assisting Quality Assurance of Examination Tasks: Using a GPT Model and Bayesian Testing for Formative Assessment

N Willert, PK Würz - Computers and Education: Artificial Intelligence, 2024

Formative quality assurance in the creation of examination tasks has always been an extremely time-consuming process. Especially due to the changing and short-lived content of computer science, new questions have to be created regularly, which in …


Association of Celtic Students

Technology is a vital part of language revitalisation and conversation. While certain languages have usable speech-to-text (STT; sometimes also called automatic speech recognition or ASR) models, this is not the case for most Celtic languages …


Asymmetric Polysemous Reasoning for Image-Text Matching

H Zhang, M Yang - 2023 IEEE International Conference on Data Mining …, 2023

Image-text matching has received growing interest since it bridges vision and language. The key challenge lies in how to learn correspondence between image and text. Upon observation, we find existing works suffer from two limitations. Firstly …


Asynchronous Local-SGD Training for Language Modeling

B Liu, R Chhaparia, A Douillard, S Kale, AA Rusu… - arXiv preprint arXiv …, 2024

Local stochastic gradient descent (Local-SGD), also referred to as federated averaging, is an approach to distributed optimization where each device performs more than one SGD update per communication. This work presents an empirical …


Atmospheric Limitations for High-frequency Ground-based Very Long Baseline Interferometry

DW Pesce, L Blackburn, R Chaves, SS Doeleman… - The Astrophysical Journal, 2024

Very long baseline interferometry (VLBI) provides the highest-resolution images in astronomy. The sharpest resolution is nominally achieved at the highest frequencies, but as the observing frequency increases, so too does the atmospheric contribution …


AtomGPT: Atomistic Generative Pre-trained Transformer for Forward and Inverse Materials Design

K Choudhary - arXiv preprint arXiv:2405.03680, 2024

Large language models (LLMs) such as generative pretrained transformers (GPTs) have shown potential for various commercial applications, but their applicability for materials design remains underexplored. In this article, we introduce AtomGPT, a …


Auditing Large Language Models for Enhanced Text-Based Stereotype Detection and Probing-Based Bias Evaluation

Z Wu, S Bulathwela, M Perez-Ortiz, AS Koshiyama - arXiv preprint arXiv:2404.01768, 2024

Recent advancements in Large Language Models (LLMs) have significantly increased their presence in human-facing Artificial Intelligence (AI) applications. However, LLMs could reproduce and even exacerbate stereotypical outputs from …


Augmenting a Spanish clinical dataset for transformer-based linking of negations and their out-of-scope references

AJ Tamayo-Herrera, DA Burgos, A Gelbukh - Natural Language Processing

A negated statement consists of three main components: the negation cue, the negation scope, and the negation reference. The negation cue is the indicator of negation, while the negation scope defines the extent of the negation. The negation …


Augmenting Knowledge-Based Conversational Search Systems With Large Language Models

Conversational interfaces are increasingly used by websites and apps, including office software as well as creativity tools. This trend reflects a shift towards natural language in human-computer interaction, which is also facilitated by utilizing Large …


Automated Data Analytics: Combining Human Creativity and AI Power Using ChatGPT

The human mind is endowed with a remarkable capacity for creative synthesis between intuition and reason; this mental alchemy is the source of genius. A new synergy is emerging between human ingenuity and the computational capacity of …


Automated Risk Analysis for Construction Contracts Using Neural Networks

K Hamdy, I AbdelRasheed, YAS Essawy… - Journal of Legal Affairs and …, 2024

… for requiring less training time compared to previous recurrent neural architectures, such as long short-term memory (LSTM), and has been prevalently adopted for training large language models on large (language) data sets, such as the …


Automated Text Identification on Languages of the Iberian Peninsula: LLM and BERT-based Models Aggregation

This paper describes our solution approach for the IberAuTexTification (Automated Text Identification on Languages of the Iberian Peninsula) competition held as part of the IberLEF 2024 conference. Machinegenerated text fragments can be spotted in …


AutoMathText: Autonomous Data Selection with Language Models for Mathematical Texts

Y Zhang, Y Luo, Y Yuan, ACC Yao - arXiv preprint arXiv:2402.07625, 2024

To improve language models' proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


Automatic Coding of Contingency in Child-Caregiver Conversations

A Agrawal, M Nikolaus, B Favre, A Fourtassi - 2024

One of the most important communicative skills children have to learn is to engage in meaningful conversations with people around them. At the heart of this learning lies the mastery of contingency, ie, the ability to contribute to an ongoing exchange …


Automatic Control With Human-Like Reasoning: Exploring Language Model Embodied Air Traffic Agents

J Andriuškevičius, J Sun - arXiv preprint arXiv:2409.09717, 2024

… Most of the data used to train leading-edge large language models, such as the latest Common Crawl dataset [5], which comprises over 250 billion web pages, sources information from publicly accessible internet sites. This extensive training …


Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

HV Vo, V Khalidov, T Darcet, T Moutakanni… - arXiv preprint arXiv …, 2024

… By filtering Common Crawl data, the authors managed to obtain large datasets to train reliable … To this end, we apply our curation pipeline to two text pools based on Common Crawl. The first … Doing so keeps the original data distribution closer to …


Automatic detection of hate speech in code-mixed Indian languages in twitter social media interaction using DConvBLSTM-MuRIL ensemble method

P Kakati, D Dandotiya - Social Network Analysis and Mining, 2024

… During the pre-training process, 2.5 terabytes of Common Crawl data in 100 languages are used. This model gives a much better performance than multilingual BERT when code-switching, especially for languages with little resources, because …


Automatic Ellipsis Reconstruction in Coordinated German Sentences Based on Text-to-Text Transfer Transformers

M Schmidt, K Harbusch, D Memmesheimer - International Conference on Text …, 2024

Ellipsis reconstruction, ie, revealing omitted syntactically obligatory words in a sentence, is still a challenging task in Natural Language Processing (NLP) technologies, even though this information is essential for advanced human-computer …


Automatic Extraction of User-Centric Aspects for Tourist Spot Recommender Systems Using Reviews in Japanese

F Uwano, R Kobayashi, M Ohta - International Conference on Human-Computer …, 2024

In tourist reviews, various pieces of information are described to confirm the characteristics of tourist spots. This paper proposes a method to extract tourist spot aspects using Japanese tourist reviews automatically. The aspects of tourist spots …


Automatic language ability assessment method based on natural language processing

N Nnamoko, T Karaminis, J Procter, J Barrowclough… - Natural Language …, 2024

Background and Objectives: The Wechsler Abbreviated Scales of Intelligence second edition (WASI-II) is a standardised assessment tool that is widely used to assess cognitive ability in clinical, research, and educational settings. In one of the …


Automatic Personality Recognition via XLNet with Refined Highway and Switching Module for Chatbot

OTC Chen, CH Tsai, MH Ha - 2024 IEEE International Symposium on Circuits and …, 2024

This study introduces an Automatic Personality Recognition (APR) model, named XLNet refined-highway-switch network, comprising XLNet, refined highway units, a switching module, and a fully-connected layer. The pretrained XLNet is employed to …


Automatic Pull Request Description Generation Using LLMs: A T5 Model Approach

MN Sakib, MA Islam, MM Arifin - arXiv preprint arXiv:2408.00921, 2024

… One reason T5 excels at producing high-quality PR descriptions is its training on the diverse ”Colossal Clean Crawled Corpus” (C4), a cleaned and structured version of the Common Crawl web dump. The C4 dataset, which encompasses 250 …


Automatic Simplification of Lithuanian Administrative Texts

J Mandravickaitė, E Rimkienė, DK Kapkan… - Algorithms, 2024

… These data were drawn from the public Common Crawl web scrape. Being pre-trained on multilingual data made mT5 model particularly … Multilingual Corpus: The model was pre-trained on CC25, which is a subset of the Common Crawl (CC) corpus that …


Automatic software product features extraction from software vendor documents

This project focuses on the automatic software features extraction from vendors' webpages and the consequent creation of a knowledge graph to accommodate these features. The use of LLMs for the classification of features is explored, through …


Automatic Text Classification With Large Language Models: A Review of openai for Zero-and Few-Shot Classification

KL Anglin, C Ventura - Journal of Educational and Behavioral Statistics, 2024

While natural language documents, such as intervention transcripts and participant writing samples, can provide highly nuanced insights into educational and psychological constructs, researchers often find these materials difficult and …


Automatické vytváření souhrnů historických dokumentů

… The dataset is comprised of a large amount of cleaned English text extracted from Common Crawl … The results show that the pre-training has 1https://commoncrawl.org/ … It uses the same pretraining objective as BART and it uses a multilingual Common …


Automating Code Adaptation for MLOps--A Benchmarking Study on LLMs

H Patel, BA Ramanan, MA Khan, T Williams… - arXiv preprint arXiv …, 2024

This paper explores the possibilities of the current generation of Large Language Models for incorporating Machine Learning Operations (MLOps) functionalities into ML training code bases. We evaluate the performance of OpenAI (gpt-3.5-turbo) and …


Automating String Encoding in AutoML

Automated Machine Learning (AutoML) systems often struggle with data sets with string data. Due to their unstructured nature and dependence on domain knowledge, string data are a frequent cause of errors and coming up with generalized ways of …


Automation and Optimisation of Seo Content Gap Analysis Task with Semantic Clustering of Portuguese Search Keywords

In the field of digital marketing, efficient content gap analysis is crucial for developing effective SEO strategies. Traditional approaches to this task are time-consuming, representing a challenge for Organic Performance teams, who must process large …


Autonomous Data Selection with Language Models for Mathematical Texts

Y Zhang, Y Luo, Y Yuan, AC Yao - ICLR 2024 Workshop on Navigating and …, 2024

To improve language models’ proficiency in mathematical reasoning via continual pretraining, we introduce a novel strategy that leverages base language models for autonomous data selection. Departing from conventional supervised fine-tuning or …


Autoregressive multimodal transformer for zero-shot sales forecasting of fashion products with exogenous data

M Rajendran, B Hong - Applied Intelligence, 2025

Predicting future sales volumes of fashion industry products is challenging due to rapid market changes and limited historical sales data for recent products. As traditional forecasting methods and machine learning models often fail to address …


Autoregressive Pre-Training on Pixels and Texts

Y Chai, Q Liu, J Xiao, S Wang, Y Sun, H Wu - Proceedings of the 2024 Conference on …, 2024

… a substantial refinement of the Common Crawl corpus. This dataset, derived from the extensive Common Crawl web scrape, undergoes … Common Crawl Common Crawl is a comprehensive web corpus that collects data from a variety of web pages …


AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs

F Kang, Y Sun, B Wen, S Chen, D Song, R Mahmood… - arXiv preprint arXiv …, 2024

… With more compute, data sources with diverse examples, such as CommonCrawl, demonstrate continued reductions in training loss even at … With more compute, data sources with diverse examples, such as CommonCrawl, continue to reduce …


BA-LORA: BIAS-ALLEVIATING LOW-RANK ADAPTA-TION TO MITIGATE CATASTROPHIC INHERITANCE IN LARGE LANGUAGE MODELS

Large language models (LLMs) have demonstrated remarkable proficiency across various natural language processing (NLP) tasks. However, adapting LLMs to downstream applications requires computationally intensive and …


Bachelor's Thesis

… We ran its script dedicated to gather articles from the Common Crawl news archive, limiting the set to only a few major German publishers, ranging in dates from 2020-2023. This should return more search results that might at first glance be …


BACK TO THE ROOTS: TRACING SOURCE LANGUAGES IN WIKIPEDIA WITH LABSE

Our objective is to try and ascertain the plausible source language of Wikipedia articles on two domains, gastronomy and science, across five languages: Italian, English, Spanish, German, and French. Drawing upon the concept of text similarity …


BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models

J Xue, M Zheng, Y Hu, F Liu, X Chen, Q Lou - arXiv preprint arXiv:2406.00083, 2024

Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data, commonly referred to as "hallucinations." Retrieval-Augmented Generation (RAG) addresses these limitations by combining …


BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

G Dong, D Pan, Y Sun, S Zhang, Z Liang, X Wu… - arXiv preprint arXiv …, 2024

… We collect 94 publicly available batches of CommonCrawl spanning the past decade. We construct our WARC extraction and processing pipeline, considering the effectiveness and cost of the WET and WARC formats. … https://commoncrawl.org …


Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding

LC Chen, ZR Li - arXiv preprint arXiv:2404.00862, 2024

… Owing to computational constraints, we further conducted additional sampling of the OSCAR dataset and the Common Crawl dataset to mitigate the issue related to dataset size. In the long run, there are around 13 billion tokens left in the corpus. The …


Balanced Data Sampling for Language Model Training with Clustering

Y Shao, L Li, Z Fei, H Yan, D Lin, X Qiu - arXiv preprint arXiv:2402.14526, 2024

Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most …


Balancing Multilingual Model Training Data Using Exponential Smoothing

… The XLM-R [9] model is pre-trained over a larger common crawl corpus similarly to MBERT, using language sampling according to exponentially-smoothed language probabilities. Again, performance on low-resources tasks seems to benefit …


BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

Q Zhang, N Gritsch, D Gnaneshwar, S Guo, D Cairuz… - Workshop on Efficient Systems for …

Training Mixture of Experts (MoEs) from scratch in a large-scale regime is expensive. Previous work addresses this challenge by independently training multiple dense expert models and using them to initialize an MoE. In particular, initializing MoE …


Bangla Emergency Post Classification on Social Media using Transformer Based BERT Models

AA Nabil, D Das, MS Salim, S Arifeen, HMA Fattah - 2023 6th International …, 2023

… It is a large multi-lingual language model trained on 2.5TB of filtered Common Crawl data. XLMRoBERTa is a multilingual model trained in 100 different languages. Unlike some XLM multilingual models, it does not require lang tensors to understand …


Banned Books: Analysis of Censorship on Amazon. com

J Knockel, J Dałek, N Aljizawi, M Ahmed, L Meletti… - 2024

… Common Crawl dataset. This sampling of products may be more likely to include hightrafficandlonger-livedproducts.Whilethismayevenbeadesirableproperty, there may also exist other biases introduced by sampling from the Common Crawl dataset …


Basic Overview of the Components of the LLM Architectures

D Grigorov - Introduction to Python and Large Language Models: A …, 2024

This chapter delves into the intricate components that constitute large language model (LLM) architectures. Understanding these elements is crucial for appreciating how LLMs transform raw textual data into meaningful, context-aware outputs. The …


BeanCounter: A low-toxicity, large-scale, and open dataset of business-oriented text

S Wang, B Levy - arXiv preprint arXiv:2409.17827, 2024

… We show that this data is indeed novel: less than 0.1% of BeanCounter appears in Common Crawl-based datasets and it is an order of magnitude larger than datasets relying on similar sources. Given the data’s provenance, we hypothesize …


Beauty Contest in Equity-Based Crowdfunding Campaigns

AA Trzebiński, Ł Kołodziejczyk - Journal of Alternative Finance, 2024

Purpose This study investigates how investor sentiment influences equity-based crowdfunding campaigns outcomes, with a particular focus on sentiment related to specific industries. The motivation stems from the need to understand the …


BeeManc at the PLABA Track of TAC-2023: Investigating LLMs and Controllable Attributes for Improving Biomedical Text Readability

Z Li, S Belkadi, N Micheletti, L Han, M Shardlow… - arXiv preprint arXiv …, 2024

In this system report, we describe the models and methods we used for our participation in the PLABA2023 task on biomedical abstract simplification, part of the TAC 2023 tracks. The system outputs we submitted come from the following three …


Bella Turca: A Large-Scale Dataset of Diverse Text Sources for Turkish Language Modeling

D Altinok - International Conference on Text, Speech, and …, 2024

… To meet the data requirements for language modeling, earlier large language models (LLMs) heavily relied on the Common Crawl dataset as their primary or sole source of training data [5, 26]. While training on the Common Crawl has proven …


Benchmarking Defeasible Reasoning with Large Language Models--Initial Experiments and Future Directions

I Tachmazidis, S Batsakis, G Antoniou - arXiv preprint arXiv:2410.12509, 2024

Large Language Models (LLMs) have gained prominence in the AI landscape due to their exceptional performance. Thus, it is essential to gain a better understanding of their capabilities and limitations, among others in terms of nonmonotonic …


Benchmarking Large Language Models

D Ellison, C Van Buren, X Jiang, A Dholakia - … TPCTC 2023, Vancouver, BC, Canada, August …

… It draws from an enormous training dataset including webpages (CommonCrawl), open-source repositories (GitHub), Wikipedia in 20 languages, books (Project Gutenberg), scientific papers (ArXiv), and questions and answers (Stack Exchange)[6] …


Benchmarking LLM for Zero-day Vulnerabilities

M Lisha, V Agarwal, S Kamthania, P Vutkur, M Chari - 2024 IEEE International …, 2024

Large Language Models (LLMs) have emerged as powerful tools for natural language processing tasks. This paper investigates the efficacy of various LLMs in detecting zero-day vulnerabilities, crucial for preemptive cybersecurity measures …


Benchmarking LLM Guardrails in Handling Multilingual Toxicity

Y Yang, S Dan, D Roth, I Lee - arXiv preprint arXiv:2410.22153, 2024

With the ubiquity of Large Language Models (LLMs), guardrails have become crucial to detect and defend against toxic content. However, with the increasing pervasiveness of LLMs in multilingual scenarios, their effectiveness in handling …


Benchmarking PathCLIP for Pathology Image Analysis

S Zheng, X Cui, Y Sun, J Li, H Li, Y Zhang, P Chen… - arXiv preprint arXiv …, 2024

… All these versions are trained using masked language modeling and next-word prediction techniques with training data sourced from publicly available repositories like Wikipedia, Common Crawl. The results of LLaMA demonstrate that cuttingedge …


Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

R Aiyappa, S Senthilmani, J An, H Kwak, YY Ahn - arXiv preprint arXiv:2403.00236, 2024

We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations …


BengaliLCP: A Dataset for Lexical Complexity Prediction in the Bengali Texts

N Ayman, MA Hossain, A Aziz, RU Faruqui, AN Chy - Proceedings of the 2024 Joint …, 2024

Encountering intricate or ambiguous terms within a sentence produces distress for the reader during comprehension. Lexical Complexity Prediction (LCP) deals with predicting the complexity score of a word or a phrase considering its context. This …


BERGEN: A Benchmarking Library for Retrieval-Augmented Generation

D Rau, H Déjean, N Chirkova, T Formal, S Wang… - arXiv preprint arXiv …, 2024

Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of …


BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformer Models in Political Science Text

Transformer models such as BERT, RoBERTa, and DeBERTa have revolutionized the field of Natural Language Processing in recent years with substantial improvements in the contextual understanding of text. While political scientists have …


BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformers Models in Political Science Text


BERTicelli at HaSpeeDe 3: Fine-tuning and Cross-validating Large Language Models for Hate Speech Detection Leonardo Grotti¹², Patrick Quick¹ ¹Universiteit …

L Grotti¹ - EVALITA Proceedings of the Eighth Evaluation …, 2024

The present paper describes the results from the experiments carried out for the HaSpeeDe 3 shared task, an Italian-language Hate Speech (HS) detection task, at EVALITA 2023. Two BERT-based language models were selected: UmBERTO (cased) …


BERTs are Generative In-Context Learners

D Samuel - arXiv preprint arXiv:2406.04823, 2024

This paper explores the in-context learning capabilities of masked language models, challenging the common view that this ability does not 'emerge' in them. We present an embarrassingly simple inference technique that enables DeBERTa to operate as …


BEThiz: Precise and complete BERT model to fill out questions with correct answers, understanding the context for the Spanish language

This article provides a research viewpoint on the training procedure of a Bidirectional Encoder Representations from Transformers (BERT) model starting from the beginning, employing the Tensor Processing Unit (TPU), with a particular …


Better Alignment with Instruction Back-and-Forth Translation

T Nguyen, J Li, S Oh, L Schmidt, J Weston… - arXiv preprint arXiv …, 2024

We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). Given documents from a web corpus, we generate and curate synthetic instructions …


Beware of Calibration Data for Pruning Large Language Models

Y Ji, Y Xiang, J Li, Q Xia, P Li, X Duan, Z Wang… - arXiv preprint arXiv …, 2024

… It includes 2.6T tokens extracted from Common Crawl. We sample from a subset5 of the DCLM. … C4 and Slimpajama, which are also extracted from Common Crawl, perform slightly worse. In contrast, the source of Wikipedia differs significantly from …


Beyond Lexical Boundaries: LLM-Generated Text Detection for Romanian Digital Libraries

M Nitu, M Dascalu - Future Internet, 2024

Machine-generated content reshapes the landscape of digital information; hence, ensuring the authenticity of texts within digital libraries has become a paramount concern. This work introduces a corpus of approximately 60 k Romanian documents …


Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

M Ochieng, V Gumma, S Sitaram, J Wang, K Ronen… - arXiv preprint arXiv …, 2024

The deployment of Large Language Models (LLMs) in real-world applications presents both opportunities and challenges, particularly in multilingual and code-mixed communication settings. This research evaluates the performance of seven leading …


Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression

Z Xu, A Gupta, T Li, O Bentham, V Srikumar - arXiv preprint arXiv:2407.04965, 2024

Large language models (LLMs) are increasingly deployed in real-world scenarios with the help of recent model compression techniques. Such momentum towards local deployment means the use of compressed LLMs will widely impact a large …


Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data

A Lee, B Miranda, S Sundar, A Casasola, S Koyeyo

… Since many datasets are publicly available (eg Common Crawl, Wikipedia), data used to train new models may be curated from such … We evaluate on Pile-CC (Pile Common Crawl) and OpenWebText2 since those datasets align with intuitively …


Beyond Technology. Our New Life with IT

… A publicly available data repository measured in many petabytes (1 petabyte = 1 million gigabytes) is Common Crawl. It is provided free of charge by Amazon Web Services to anyone who needs data for training purposes for AI. In the context of very …


Beyond the Black Box: Optimization Within Latent Spaces

… We compute b using Common Crawl monolingual datasets.For each language and contextual embedding model, we create 1M candidate-reference pairs by grouping two random sentences. Because of the random pairing and the corpus …


Beyond Words: Sentiment Analysis of Croatian Language Attitudes

Sociolinguistic research can benefit heavily from natural language processing (NLP) tools. Especially language attitude research can be facilitated by using sentiment analysis. By using existing tools for Bosnian-Croatian-Montenegrin-Serbian (BCMS) …


BharatBhasaNet-A unified framework to identify Indian code mix Languages

S Dey, S Thakur, A Kandwal, R Kumar, S Dasgupta… - IEEE Access, 2024

In the rapidly globalizing digital communication sphere, the imperative for advanced multilingual text recognition and identification is increasingly evident. Contrasting the previous works, which were predominantly constrained to 2-3 languages, this …


BhashaVerse: Translation Ecosystem for Indian Subcontinent Languages

V Mujadia, DM Sharma - arXiv preprint arXiv:2412.04351, 2024

… 2019b) mined from Wikipedia and CommonCrawl offer valuable data for pre-processing pipelines. Other notable resources include the Nepali National Corpus (Yadavaetal. 2008), the Kathmandu University English-Nepali Parallel Corpus (Duwal and Bal …


Bi-directional GRU-Based Approach for Multi-Class Text Error Identification System

S Reeha, BV Nikith, GMP Reddy, PB Pati - 2024 IEEE 9th International Conference …, 2024

… The C4 dataset, utilized in this study, is an extensively cleaned and curated version of Common Crawl's web crawl corpus. Common Crawl is a publicly available dataset accessible at commoncrawl.org. and boasting over 700 million …


BIAS ASSESSMENT IN LARGE LANGUAGE MODELS

The introduction of Large Language Models (LLMs) such as Chat-GPT has transformed the field of natural language processing, providing exceptional capabilities in text generation and comprehension. However, these models have …


Bias in LLMs for High-Stakes Recommendations: An Analysis of BERT-Family Architectures with Varied Fine-Tuning Configurations

… This chapter explained the mechanisms behind LLMs, including their training with extensive data sources like Common Crawl and the role of tokenization. It covered how fine-tuning improves LLMs for specific tasks using methods such as sequence …


Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models

Y Chang, Y Chang, Y Wu - arXiv preprint arXiv:2408.04556, 2024

Large language models (LLMs) have exhibited remarkable proficiency across a diverse array of natural language processing (NLP) tasks. However, adapting LLMs to downstream applications typically necessitates computationally intensive and …


BiasMirror: Towards Mitigating Implicit Bias

K Brohman, A Khan, T Fu, R Mukkamala, A Baiyere - 2024

This research-in-progress study attends to the issue of implicit bias, which underlies discriminatory attitudes and behaviors that people may not be aware that they hold. Specifically in the context of performance evaluations, attending to such bias is …


Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models

C Campanella, R van der Goot - arXiv preprint arXiv:2403.08046, 2024

Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known …


BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks

JA Rodriguez, X Jian, SS Panigrahi, T Zhang, A Feizi… - Workshop on Responsibly …

Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long-structured …


Bilingual Adaptation of Monolingual Foundation Models

G Gosal, Y Xu, G Ramakrishnan, R Joshi, A Sheinin… - … on Foundation Models in the Wild

We present an efficient method for adapting a monolingual Large Language Model (LLM) to another language, addressing challenges of catastrophic forgetting and tokenizer limitations. We focus this study on adapting Llama 2 to Arabic. Our two-stage …


Bilingual Propaganda Detection in Diplomats' Tweets Using Language Models and Linguistic Features

A Modzelewski, P Golik, A Wierzbicki - IberLEF@ SEPLN, 2024

Our study presents an approach to a shared task of propaganda identification and characterization at the DIPROMATS 2024 hosted by the Iberian Languages Evaluation Forum. As the DSHacker team, we participated in the propaganda …


Binary Battle: Leveraging Machine Learning and Transfer Learning Models to Distinguish between Conspiracy Theories and Critical Thinking

S Mahesh, S Divakaran, K Girish, HL Shashirekha - 2024

In the context of automatic content moderation Natural Language Processing (NLP) has a complex task when it comes to distinguishing between conspiracy theories and critical thinking. While conspiracy theories present complex narratives …


BioBridge: Unified Bio-Embedding with Bridging Modality in Code-Switched EMR.

J Jeon, S Cho, D Lee, C Lee, J Kim - IEEE Access, 2024

Pediatric Emergency Department (PED) overcrowding presents a significant global challenge, prompting the need for efficient solutions. This paper introduces the BioBridge framework, a novel approach that applies Natural Language Processing (NLP) …


Bioeconomy firms and where to find them

… This dataset uses the open-source web repository CommonCrawl to identify German company websites and has proven to be a valuable database for spatial research. From this data, we identify bioeconomy firms using a combination of …


BioMedLM: A 2.7 B Parameter Language Model Trained On Biomedical Text

E Bolton, A Venigalla, M Yasunaga, D Hall, B Xiong… - arXiv preprint arXiv …, 2024

… The Pile contains multiple sub-corpora, including general content from Common Crawl and specialized content like PubMed, Github, and FreeLaw. This model serves as an important baseline for comparison with our domain-specific models; …


BIS Papers

I Aldasoro, S Doerr, L Gambacorta, S Notra, T Oliviero… - 2024

The rapid advancement of artificial intelligence (AI) relies on a complex supply chain comprising five key layers: hardware, cloud infrastructure, training data, foundation models and AI applications. This paper examines the market structure of each layer …


BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

J Myung, N Lee, Y Zhou, J Jin, RA Putri, D Antypas… - arXiv preprint arXiv …, 2024

… [21] introduce a methodology to extract large-scale cultural commonsense knowledge from the Common Crawl corpus on geography, religion, and occupations. CREHate [16] is a cross-cultural English hate speech dataset covering annotations …


Blind Baselines Beat Membership Inference Attacks for Foundation Models

D Das, J Zhang, F Tramèr - arXiv preprint arXiv:2406.16201, 2024

… More precisely, these benchmarks introduce two datasets, CommonPool and DCLM-POOL, which contain respectively 12.8 billion image-text pairs and 240 trillion text tokens sampled from Common Crawl. To enable experiments at …


BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks

AV Ramesh, V Ganapathiraman, IH Laradji, M Schmidt - arXiv preprint arXiv …, 2024

… The C4 dataset is a large-scale, cleaned version of the Common Crawl web corpus used for pre-training language models, featuring diverse and high-quality text from the internet. Our experiment setup is similar to [34], following the setup from …


Boost Large Language Model Performance through Self-Training with Reward Guided Tree Search

J Hargreaves, E Fairweather, O Bellingham - 2024

Large language models have changed the field of natural language processing by enabling sophisticated language generation tasks, yet there remains a persistent challenge in enhancing their performance through autonomous learning …


Botlitica: A generative AI-based tool to assist journalists in navigating political propaganda campaigns

E Musi, EEG Aguilar, L Federico - Studies in Communication Sciences, 2024

The hype on generative AI has raised concerns about the spread of disinformation but also opened up new opportunities for hybrid journalism. The proliferation of political propaganda campaigns spread across digital media during election periods …


BOTS-LM: Training Large Language Models for Setswana

N Brown, V Marivate - arXiv preprint arXiv:2408.02239, 2024

In this work we present BOTS-LM, a series of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, BOTS-LM achieves performance similar to models …


Brain Rush

… LLM developers could tap free data libraries – such as the Common Crawl repository of web data – supplemented by licensing foreign-language …


Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

S Sukhbaatar, O Golovneva, V Sharma, H Xu, XV Lin… - arXiv preprint arXiv …, 2024

We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts …


Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling

YC Yu, CC Kuo, Z Ye, YC Chang, YS Li - arXiv preprint arXiv:2406.12585, 2024

Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve …


Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models

T Blevins, T Limisiewicz, S Gururangan, M Li, H Gonen… - arXiv preprint arXiv …, 2024

Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters. We propose Cross-lingual Expert Language Models (X-ELM), which …


Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners

J Prather, BN Reeves, P Denny, J Leinonen, S MacNeil… - arXiv preprint arXiv …, 2024

… than 1% of the language distribution available in Common Crawl2 (an open repository of web crawl data that makes up a significant portion of the training data for many LLMs), whereas Chinese is one of the most common languages in the …


Brevity is the soul of wit: Pruning long files for code generation

AK Singh, Y Yang, K Tirumala, M Elhoushi, AS Morcos - arXiv preprint arXiv …, 2024

Data curation is commonly considered a "secret-sauce" for LLM training, with higher quality data usually leading to better LLM performance. Given the scale of internet-scraped corpora, data pruning has become a larger and larger focus. Specifically, many have …


Bridging large language model disparities: Skill tagging of multilingual educational content

Y Kwak, ZA Pardos - British Journal of Educational Technology, 2024

The adoption of large language models (LLMs) in education holds much promise. However, like many technological innovations before them, adoption and access can often be inequitable from the outset, creating more divides than they bridge. In …


Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

EC Acikgoz, M Erdogan, D Yuret - arXiv preprint arXiv:2405.04685, 2024

… This dataset was generated by extracting content from 71 monthly snapshots of the internet via Common Crawl (CC). CulturaX contains version … However, the diversity of these datasets should be sufficient, including common crawl, book …


Bridging the Data Provenance Gap Across Text, Speech and Video

S Longpre, N Singh, M Cherep, K Tiwary… - arXiv preprint arXiv …, 2024

Progress in AI is driven largely by the scale and quality of training data. Despite this, there is a deficit of empirical analysis examining the attributes of well-established datasets beyond text. In this work we conduct the largest and first-of-its-kind …


Browsing and Searching Metadata of TREC

T Breuer, EM Voorhees, I Soboroff - Proceedings of the 47th International ACM SIGIR …, 2024

Information Retrieval (IR) research is deeply rooted in experimentation and evaluation, and the Text REtrieval Conference (TREC) has been playing a central role in making that possible since its inauguration in 1992. TREC's mission centers …


Build a Large Language Model (From Scratch)

… For context, consider the size of the CommonCrawl dataset, which alone consists of 410 billion tokens and requires about 570 GB of storage. In comparison, later iterations of models like GPT-3, such as Meta’s LLaMA, have expanded their training scope to …


Building a Large Japanese Web Corpus for Large Language Models

N Okazaki, K Hattori, H Shota, H Iida, M Ohi, K Fujii… - arXiv preprint arXiv …, 2024

… The corpus developed in this study is made from 21 snapshots of Common Crawl (from CC-MAIN-2020-40 to CC-MAIN-2023-23), and the size of the corpus after cleaning is 173,350,375 pages and 312,093,428,689 characters. To confirm the …


Building a Specialized ChatGpt-Like System

… be examined, which was sampled from a mix primarily made up of the Common Crawl dataset [11]. This dataset is large enough to train the … This means that Common Crawl and Books2 datasets are sampled less than once during training …


Building and AI Chatbot using LLM

In our digital world, where communication is key, the emergence of AI-powered chatbots has revolutionized the way we interact with technology. This project delves into the development of an AI chatbot utilizing Large Language Models (LLM), a …


Building and better understanding vision-language models: insights and future directions

H Laurençon, A Marafioti, V Sanh, L Tronchon - arXiv preprint arXiv:2408.12637, 2024

The field of vision-language models (VLMs), which take images and texts as inputs and output texts, is rapidly evolving and has yet to reach consensus on several key aspects of the development pipeline, including data, architecture, and training …


Building and Running Large-Scale Language Models: The Infrastructure and Techniques Behind GPT-4

… o Common Crawl: One of the primary sources for large-scale training datasets is Common Crawl, which is an open repository of web data collected by crawling the internet. Common Crawl provides a vast amount of diverse text from websites, blogs …


Building Better Datasets: Seven Recommendations for Responsible Design from Dataset Creators

W Orr, K Crawford - arXiv preprint arXiv:2409.00252, 2024

The increasing demand for high-quality datasets in machine learning has raised concerns about the ethical and responsible creation of these datasets. Dataset creators play a crucial role in developing responsible practices, yet their …


Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers

X Wei, S Moalla, R Pascanu, C Gulcehre - arXiv preprint arXiv:2406.16450, 2024

… Dataset We use the RefinedWeb dataset [44], a carefully curated subset of CommonCrawl, processed for optimal filtering and deduplication. It provides 600B tokens available for public use. As it is a substantial dataset, we shuffle, extract, and …


Building Question-Answer Data Using Web Register Identification

A Eskelinen, A Myntti, E Henriksson, S Pyysalo… - Proceedings of the 2024 …, 2024

… (2015), is a corpus of Finnish collected between 2015 and 2016 from Common Crawl and by crawling .fi domains. As all of these resources are Common Crawl based, in the post-processing stage we make sure that duplicate QA pairs are …


Building Test Collections for Japanese Dense Information Retrieval Technologies and Beyond

H Joho, A Keyaki, Y Tachioka, S Yamamoto - 2024

This paper presents the NTCIR Transfer Task, a series of evaluations aimed at advancing dense information retrieval technologies for non-English languages, particularly Japanese. While dense retrieval methods have shown significant …


By Senior Judge Stephanie Domitrovich and Judge Herbert B. Dixon Jr.

CJJ Federal Judiciary - Judges Journal, 2024

… For example, the datasets used to train ChatGPT 3 included text from the following sources: (1) Common Crawl, containing petabytes of data collected from the internet since 2008; (2) WebText2, text from webpages with highly ranked Reddit …


Bye-bye Bias: What to Consider When Training Generative AI Models on Subjective Marketing Metrics

… However, these models are trained on vast data sets scraped from the Internet, such as Common Crawl or LAION, that contain little information on the types of perceptual measures marketing is often interested in, such as perceived brand …


Byte Latent Transformer: Patches Scale Better Than Tokens

A Pagnoni, R Pasunuru, P Rodriguez, J Nguyen… - arXiv preprint arXiv …, 2024

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes …


Cache Me If You Can: The Case For Retrieval Augmentation in Federated Learning

A Muhamed, P Thaker, MT Diab, V Smith - Privacy Regulation and Protection in Machine …

We propose retrieval augmentation (RA) as an enhancement to federated learning (FL) that can improve privacy protection and ensure regulatory compliance. FL, primarily designed for data privacy preservation, faces challenges with conventional …


CAM 2.0: End-to-End Open Domain Comparative Question Answering System

A Shallouf, H Herasimchyk, M Salnikov, RG Veliz…

… After extracting objects and aspects, we look for their matches in the Common Crawl text. It is also important to mention, that for the final … (lemmatized) corpus containing 14.3 billion English sentences from the Common Crawl. To retrieve …


CamelEval: Advancing Culturally Aligned Arabic Language Models and Benchmarks

Z Qian, F Altam, MSS Alqurishi, R Souissi - arXiv preprint arXiv:2409.12623, 2024

Large Language Models (LLMs) are the cornerstones of modern artificial intelligence systems. This paper introduces Juhaina, a Arabic-English bilingual LLM specifically designed to align with the values and preferences of Arabic speakers …


Can AI Help with Your Personal Finances?

In recent years, Large Language Models (LLMs) have emerged as a transformative development in artificial intelligence (AI), drawing significant attention from industry and academia. Trained on vast datasets, these sophisticated AI systems exhibit …


Can Bias in LLMs be Used for Good?

It is a well-known fact that LLMs express harmful biases in their predictions. The main source comes from the training datasets, which are too large and expensive to check thoroughly. In my submission, I explore how we can leverage the bias in LLMs …


Can ChatGPT learn Chinese or Swahili? Considering how large language models might act differently if trained in different languages.

… Thien Nguyen, a professor of computer science at the University of Oregon, looked at the CommonCrawl corpus, a selection of text scraped … Then, because only a small percentage of the CommonCrawl corpus is in Korean, the company …


Can ChatGPT Plan Your Retirement?: Generative AI andFinancial Advice

We identify some of the most pressing issues facing the adoption of large language models (LLMs) in practical settings, and propose a research agenda to reach the next technological inflection point in generative AI. We focus on three challenges …


Can cross-domain term extraction benefit from cross-lingual transfer and nested term labeling?

HTH Tran, M Martinc, A Repar, N Ljubešić, A Doucet… - Machine Learning, 2024

Automatic term extraction (ATE) is a natural language processing task that eases the effort of manually identifying terms from domain-specific corpora by providing a list of candidate terms. In this paper, we treat ATE as a sequence-labeling task and …


Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration

M Husain, P Wach, TG Topcu - The Proceedings of the 2024 Conference on Systems …

The old saga of complex system development continues as the vast majority of government and industry programs continue to result in cost and schedule overruns [1–7]. While numerous researchers and government reports attribute this trend to the ever-increasing …


Can large language models understand molecules?

S Sadeghi, A Bui, A Forooghi, J Lu, A Ngom - BMC bioinformatics, 2024

Purpose Large Language Models (LLMs) like Generative Pre-trained Transformer (GPT) from OpenAI and LLaMA (Large Language Model Meta AI) from Meta AI are increasingly recognized for their potential in the field of cheminformatics, particularly …


Can Models Learn Skill Composition from Examples?

H Zhao, S Kaur, D Yu, A Goyal, S Arora - ICML 2024 Workshop on LLMs and Cognition

As large language models (LLMs) become increasingly capable, their ability to exhibit *compositional generalization* of skills has garnered significant attention. Yu et al. 2023 recently introduced Skill-Mix evaluation, where models are tasked with …


Cascaded transformer-based networks for wikipedia large-scale image-caption matching

N Messina, DA Coccomini, A Esuli, F Falchi - Multimedia Tools and Applications, 2024

… It uses an XLM-RoBERTa masked language model pre-trained on the CommonCrawl dataset and fine-tuned on the image URL-caption match classification task. The CLS token in output from XLM-RoBERTa is attached to a feed-forward …


Causality-driven multivariate stock movement forecasting

A Díaz Berenguer, Y Da, MN Bossa, MC Oveneke… - PloS one, 2024

Our study aims to investigate the interdependence between international stock markets and sentiments from financial news in stock forecasting. We adopt the Temporal Fusion Transformers (TFT) to incorporate intra and inter-market …


CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

I Ilyankou, J Haworth, S Cavazzi - arXiv preprint arXiv:2405.11039, 2024

The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content …


CCI3. 0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models

L Wang, BW Zhang, C Wu, H Zhao, X Shi, S Gu, J Li… - arXiv preprint arXiv …, 2024

… Among them, the Open-source datasets, such as The Pile[3] and Common Crawl[4], have been instrumental in propelling LLM … such as FineWeb(15TB)[7], which draw extensively from Common Crawl. Simultaneously, the focus has shifted from rule-based …


CCSUM: A large-scale and high-quality dataset for abstractive news summarization

… Accordingly, among 35 million CommonCrawl News articles, we identify pairs of articles about the same news story and use one article’s first sentence as the summary for the other article. To ensure high quality, we apply strict filters whose …


CERM: Context-aware Literature-based Discovery via Sentiment Analysis

JC Young, U Akujuobi - arXiv preprint arXiv:2402.01724, 2024

Driven by the abundance of biomedical publications, we introduce a sentiment analysis task to understand food-health relationship. Prior attempts to incorporate health into recipe recommendation and analysis systems have primarily focused on …


Challenge design roadmap

HJE Balderas, I Guyon, A Howard, W Reade, S Treguer - AI Competitions and …, 2023

Challenges can be seen as a type of game that motivates participants to solve serious tasks. As a result, competition organizers must develop effective game rules. However, these rules have multiple objectives beyond making the game enjoyable …


Challenges and Contributions in Table-to-text Generation: A Survey

… The pre-training of T5 involves an unsupervised “span masking” objective applied to Common Crawl data, where spans of text are masked and the model learns to predict them, thus gaining a deep understanding of language patterns and …


Changes and challenges of legal education in the era of generative artificial intelligence: Chinese experience

W Wang, Z Xu, Z Xu - Journal of Infrastructure, Policy and Development, 2024

Using generative artificial intelligence systems in the classroom for law case analysis teaching can enhance the efficiency and accuracy of knowledge delivery. They can create interactive learning environments that are appropriate, immersive …


Chatbots in Airport Customer Service—Exploring Use Cases and Technology Acceptance

I Auer, S Schlögl, G Glowka - Future Internet, 2024

… Those pre-trained transformer systems like BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) have been trained on vast language corpora such as Wikipedia and Common Crawl …


Chatbots to Cyber Threats: The Evolution of Generative AI in Digital Security

Generative AI (GenAI) technologies like models such as ChatGPT and Google Bard are racing into an era of rapid incursion of the digital interaction space, especially in the areas of cybersecurity and privacy. I pore on in this paper such duality of the …


Chatgpt & Google Bard AI: A Comparative Study From Student's Perspective.

The motivation behind choosing this research topic is that rising generation will be based on AI ie Artificial Intelligence. Artificial intelligence is a specialty within computer science that is concerned with creating systems that can replicate human …


ChatGPT Alternative Solutions: Large Language Models Survey

H Alipour, N Pendar, K Roy - arXiv preprint arXiv:2403.14469, 2024

In recent times, the grandeur of Large Language Models (LLMs) has not only shone in the realm of natural language processing but has also cast its brilliance across a vast array of applications. This remarkable display of LLM capabilities has ignited a …


ChatGPT and GPT-4: utilities in the legal sector

Artificial intelligence systems such as ChatGPT, the OpenAI chatbot, based on the language model family GPT (generative pre-trained transformers), as well as other solutions built on this technology and fine-tuned for specific tasks, have generated …


ChatGPT and Language Translation: A Small Case Study Evaluating English–Mandarin Translation

C Woodrum - International Conference on Human-Computer …, 2024

… has been informed speculation that it relies heavily on common crawl [13], a compendium of publicly available … common crawl. The \(15^{th}\) most common language, Indonesian, makes up less than 1% of the data set [14, 15]. It is important …


ChatGPT as the Marketplace of Ideas: Should Truth-Seeking Be the Goal of AI Content Governance?

J Zhang - arXiv preprint arXiv:2405.18636, 2024

As one of the most enduring metaphors within legal discourse, the marketplace of ideas has wielded considerable influence over the jurisprudential landscape for decades. A century after the inception of this theory, ChatGPT emerged as a …


ChatGPT in Cyber Onslaught and Fortification: Past, Present, and Future

M Gunda, V Manda, P Naradasu, S Mekala… - 2024 IEEE 9th International …, 2024

… GPT-1 was unveiled in 2018 and trained on a pair of data sets: over 11,000 books from Book Corpus and the enormous Common Crawl … With 1.5 billion features and a much more extensive and diversified training dataset comprising Common Crawl …


ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis

W Ahmed, M Saturno, R Rajjoub, AH Duey, B Zaidat… - European Spine Journal, 2024

… Furthermore, the largest dataset used to train both ChatGPT 3.5 and 4.0 is CommonCrawl, a platform that does not include PubMed and consists predominantly of open access articles, some of which may be of less rigorous if not …


ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis

A Buscemi, D Proverbio - arXiv preprint arXiv:2402.01715, 2024

… It relied on a dataset referred to as the Common Crawl [30], a publicly accessible repository that comprises billions of web pages, making it one of the most extensive text databases available. It is important to note that the choice of dataset significantly …


ChatGPT's applications in marketing: a topic modeling approach

W Tafesse, A Wien - Marketing Intelligence & Planning, 2024

Purpose ChatGPT is a versatile technology with practical use cases spanning many professional disciplines including marketing. Being a recent innovation, however, there is a lack of academic insight into its tangible applications in the marketing …


ChatGPT, the perfect virtual teaching assistant? Ideological bias in learner-chatbot interactions

M Van Poucke - Computers and Composition, 2024

This paper examines ChatGPT's use of evaluative language and engagement strategies while addressing information-seeking queries. It assesses the chatbot's role as a virtual teaching assistant (VTA) across various educational settings. By …


ChatGPT-3.5 can create and justify short Afrikaans poems: Implications for practice

CJ van Staden - Suid-Afrikaanse Tydskrif vir Natuurwetenskap en …, 2024

ChatGPT-3.5 can create poems but the implications thereof has not yet been sufficiently investigated. The purpose of this explorative case study was to determine to which extent ChatGPT-3.5 can create and justify Afrikaans poems. Because …


ChatQA: Building GPT-4 Level Conversational QA Models

Z Liu, W Ping, R Roy, P Xu, M Shoeybi, B Catanzaro - arXiv preprint arXiv …, 2024

… We collect 7k documents (average ∼1k words per document) from common crawl, which cover a wide range of domains. Each document is used for generation of a single conversational QA sample, which leads to a total of 7k multi-turn QA dialogues …


Chatqa: Surpassing gpt-4 on conversational qa and rag

Z Liu, W Ping, R Roy, P Xu, C Lee, M Shoeybi… - The Thirty-eighth Annual …, 2024

In this work, we introduce ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answering (QA). To enhance generation, we propose a two-stage instruction tuning method that …


Chatting Over Course Material: The Role of Retrieval Augmented Generation Systems in Enhancing Academic Chatbots.

Large Language Models (LLMs) have the potential to enhance learning among students. These tools can be used in chatbot systems allowing students to ask questions about course material, in particular when plugged with the so-called …


Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data

L Castro-Gonzalez, YL Chung, HR Kirk, J Francis… - arXiv preprint arXiv …, 2024

The field of machine learning has recently made significant progress in reducing the requirements for labelled training data when building new models. These `cheaper' learning techniques hold significant potential for the social sciences, where …


Check for Extending the Comparative Argumentative Machine: Multilingualism and Stance Detection

I Nikishina¹, A Bondarenko, S Zaczek¹, OL Haag… - Robust Argumentation Machines …

The comparative argumentative machine CAM can retrieve arguments that answer comparative questions questions that ask which of several to-be-compared options should be favored in some scenario. In this paper, we describe how we equipped …


Check for updates A Study of Transductive Graph-Based Regression

ADB Valejo - Intelligent Systems Design and Applications: Deep …

Regression methods play an important role in many realworld applications such as econometric, pattern recognition, and prediction of protein chains to cite a few tasks. In the semi-supervised transductive learning problem, some studies present graph-based …


Check for updates Digilog: Enhancing Website Embedding on Local Governments-A Comparative Analysis

J Gerber, B Kreiner, J Saxer, A Weiler - Foundations of Intelligent Systems: 27th International …

The ability to understand and process websites, known as website embedding, is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local …


CheckIT!: A Corpus of Expert Fact-checked Claims for Italian Jacopo Gili', Lucia Passaro² and Tommaso Caselli³ ¹Department of Computer Science, University of Turin …

LP Jacopo Gili, T Caselli - Proceedings of the 9th Italian Conference on …, 2024

This paper introduces CheckIT!, a resource of expert fact-checked claims, filling a gap for the development of fact-checking pipelines in Italian. We further investigate the use of three state-of-the-art generative text models to create variations of claims …


ChemTEB: Chemical Text Embedding Benchmark, an Overview of Embedding Models Performance & Efficiency on a Specific Domain

AS Kasmaee, M Khodadad, MA Saloot, N Sherck… - arXiv preprint arXiv …, 2024

… Unlike models that rely on limited labeled data or low-quality synthetic text pairs, E5 is contrastively trained on CCPairs, a curated web-scale dataset that incorporates diverse data sources such as CommunityQA, Common Crawl, and …


Chimera: Improving Generalist Model with Domain-Specific Experts

T Peng, M Li, H Zhou, R Xia, R Zhang, L Bai, S Mao… - arXiv preprint arXiv …, 2024

… We constructed our benchmark by systematically sampling from an initial pool of 200, 000 PDF documents obtained from Common Crawl, Google, Baidu search engines, and internal repositories, which is consistent with the approach of …


Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model

X Du, Z Yu, S Gao, D Pan, Y Cheng, Z Ma, R Yuan… - arXiv preprint arXiv …, 2024

In this study, we introduce CT-LLM, a 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional …


ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information

W Zhang, Z Li, W Yang, C Leng, Y Bai, Q Du, C Zong… - arXiv preprint arXiv …, 2024

… Given the relatively high noise levels in Common Crawl data, we directly sample from Common Crawl to construct the negative samples. Figure 2a provides a detailed breakdown of the composition and quantity of the training data …


ChocoLlama: Lessons Learned From Teaching Llamas Dutch

M Meeus, A Rathé, F Remy, P Delobelle, JJ Decorte… - arXiv preprint arXiv …, 2024

While Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, their performance often lags in lower-resource, non-English languages due to biases in the training data. In this …


Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task

Z Wu, D Wei, Z Li, H Shang, J Guo, S Li, Z Rao, Y Luo… - arXiv preprint arXiv …, 2024

… and Common Crawl data sources. The amount of data we used for training NMT and LLM-based MT models is shown in Table 1. It should be noted that in order to obtain better translation performance in the general domain, we mix the monolingual …


CHORDONOMICON: A Dataset of 666,000 Songs and their Chord Progressions

S Kantarelis, K Thomas, V Lyberatos, E Dervakos… - arXiv preprint arXiv …, 2024

… A relevant search on https://index.commoncrawl.org/ will show all the harvested UG URLs, and the relevant Common Crawl web archive (.warc) files will show the raw scrapped pages, along with the lyrics, song names, bands, and all information …


CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models

S Wu, Y Wang, R Yuan, Z Guo, X Tan, G Zhang… - arXiv preprint arXiv …, 2024

Challenges in managing linguistic diversity and integrating various musical modalities are faced by current music information retrieval systems. These limitations reduce their effectiveness in a global, multimodal music environment. To …


CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systems

S Rosenthal, A Sil, R Florian, S Roukos - arXiv preprint arXiv:2404.02103, 2024

… However, they use sentencelevel matching (by encoding sentences for semantic similarity comparisons) to retrieve up to top 7 documents from Common Crawl while avoiding exact matches as the abstractive dataset. In the extractive version, the …


CLASSLA-web: Comparable Web Corpora of South Slavic Languages Enriched with Linguistic and Genre Annotation

N Ljubešić, T Kuzman - arXiv preprint arXiv:2403.12721, 2024

This paper presents a collection of highly comparable web corpora of Slovenian, Croatian, Bosnian, Montenegrin, Serbian, Macedonian, and Bulgarian, covering thereby the whole spectrum of official languages in the South Slavic language space …


Cleaner Pretraining Corpus Curation with Neural Web Scraping

Z Xu, Z Liu, Y Yan, Z Liu, C Xiong, G Yu - arXiv preprint arXiv:2402.14652, 2024

… The web-crawled datasets, such as Common Crawl, have been widely used for pretraining, facilitating the development of language … htmlparser serves as the text pre-extraction tool for CommonCrawl6 WET file (containing pre-extracted text), while …


CLEARNESS: Coreference Resolution for Generating and Ranking Arguments Extracted from Debate Portals for Queries

… Refining Arg-CTRL with data from Common Crawl leads to a higher quality of generated arguments compared to using user discussions from Reddit comments. In the mentioned studies, new arguments are generated through the use of knowledge …


CLIP and the City: Addressing the Artificial Encoding of Cities in Multimodal Foundation Deep Learning Models

In this project, we propose and explore a computational pipeline to examine urban cultural landscapes through the lens of artificial intelligence, and for questioning modes of embedding culture in machine learning models. By employing machine …


CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination

K Yang, T Gu, X An, H Jiang, X Dai, Z Feng, W Cai… - arXiv preprint arXiv …, 2024

Contrastive Language-Image Pre-training (CLIP) has achieved excellent performance over a wide range of tasks. However, the effectiveness of CLIP heavily relies on a substantial corpus of pre-training data, resulting in notable consumption …


Clusters and Copies: An Analysis of Cryptocurrency Investment Scam Websites

Cryptocurrency investment scams have become an increasingly prevalent threat, leveraging sophisticated methods to deceive and exploit victims. The phenomenon of pig butchering has gained prominence, but victims rarely see the perpetrators …


Co-constructing AI Authoring Using Ethical Theories

B Jones - 88th Annual International Conference

We must show ethical humility, historically contextualizing ethical standards, as we implement AI tools in courses. The pace of change requires agile approaches to defining standards for the ethical use of AI. We invited students to co-construct a …


CoastTerm: a Corpus for Multidisciplinary Term Extraction in Coastal Scientific Literature

J Delaunay, HTH Tran, CE González-Gallardo… - arXiv preprint arXiv …, 2024

… – Multilingual pre-trained model: We opt for XLMR [9], a transformer-based model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. This multilingual version of RoBERTa, achieves benchmark performance in ATE for rich-resourced …


Code LLMs: A Taxonomy-based Survey

N Raihan, C Newman, M Zampieri - arXiv preprint arXiv:2412.08291, 2024

Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks and have recently expanded their impact to coding tasks, bridging the gap between natural languages (NL) and programming languages (PL). This …


Cognitive Development as a Model for the Effects of Psychedelics: Do Changes in Cognitive Flexibility Underly the Clinical Benefits of Psychedelic Therapy?

… A set of pre-trained word vectors was used which contained 840 billion tokens gathered via Common Crawl. The original set of word vectors was filtered to include only a subset of words originally used by Contexto, then further filtered to remove …


Combatting Dimensional Collapse in LLM Pre-Training Data via Submodular File Selection

Selecting high-quality pre-training data for large language models (LLMs) is crucial for enhancing their overall performance under limited computation budget, improving both training and sample efficiency. Recent advancements in file …


ComiCap: A VLMs pipeline for dense captioning of Comic Panels

E Vivoli, N Biondi, M Bertini, D Karatzas - arXiv preprint arXiv:2409.16159, 2024

… It was trained using a unique set of training data that includes CommonCrawl and Code Pretrain datasets, C4, and smaller contributions from sources such as Arxiv and Open Web Math, among others. For handling high-resolution images, the model …


Community OSCAR: A Community Effort for Multilingual Web Data

M Brack, M Ostendorff, PO Suarez, JJ Saiz, IL Castilla…

The development of large language models (LLMs) relies heavily on extensive, highquality datasets. Publicly available datasets focus predominantly on English, leaving other language communities behind. To address this issue, we introduce …


Comparative Analysis of Cognitive Agreement between Human Analysts and Generative AI in Construction Safety Risk Assessment

U Ray, C Arteaga, JW Park - ISARC. Proceedings of the International Symposium on …, 2024

The construction industry struggles with safety risk assessment complexities due to evolving work environments, diverse labor forces, time constraints, regulatory intricacies, and inconsistent practices. While previous studies have highlighted the …


Comparative Analysis of Generative AI Tools in Enhancing Educational Engagement

F Borović, K Aleksić-Maslać, P Vranešić - 2024 47th MIPRO ICT and Electronics …, 2024

Technological development is continuously reshaping the traditional educational setting, presenting various possibilities for educational refinement. It is pivotal to motivate active engagement among all participants, and generative AI tools emerge …


Comparative evaluation of machine learning algorithms for phishing site detection

NF Almujahid, MA Haq, M Alshehri - PeerJ Computer Science, 2024

The advent of Internet technologies has resulted in the proliferation of electronic trading and the use of the Internet for electronic transactions, leading to a rise in unauthorized access to sensitive user information and the depletion of resources for …


Comparative Performance of Advanced NLP Models and LLMs in Multilingual Geo-Entity Detection

K Kopanov - Proceedings of the Cognitive Models and Artificial …, 2024

The integration of advanced Natural Language Processing (NLP) methodologies and Large Language Models (LLMs) has significantly enhanced the extraction and analysis of geospatial data from multilingual texts, impacting sectors such as …


Comparative Study on Synthetic and Natural Error Analysis with BART & MarianMT

R Rohit, SA Gandheesh, GS Sannala, PB Pati - 2024 IEEE 9th International …, 2024

Text is essential for communication, information sharing, knowledge acquisition, and analysis. It shapes public opinion, supports education, and drives online content, making it crucial in various domains. While there are various language models …


Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

H Bansal, A Suvarna, G Bhatt, N Peng, KW Chang… - arXiv preprint arXiv …, 2024

A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This only leverages the pairwise comparisons when the generations are …


Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models

Z Zhao, N Aletras - arXiv preprint arXiv:2403.12809, 2024

In many real natural language processing application scenarios, practitioners not only aim to maximize predictive performance but also seek faithful explanations for the model predictions. Rationales and importance distribution given by feature …


Comparing Fine-Tuning, Zero and Few-Shot Strategies with Large Language Models in Hate Speech Detection in English

R Pan, JA García-Díaz, R Valencia-García

… This is a multilingual model based on RoBERTa, pre-trained with a 2.5 TB filtered CommonCrawl dataset containing 100 languages, using an MLM goal, where it randomly masks 15% of the words in the input, runs the entire masked sentence …


Comparing Foundations: Insights into the Construction of Financial Causal Knowledge Graphs with and without Ontology

Z Xu, R Ichise - 人工知能学会全国大会論文集 第 38 回 (2024), 2024

Simplicity in information navigation, interpretation, and reasoning has positioned mono-relation knowledge graphs (KGs) as a focus of attention, particularly in targeted scenarios. However, their deficiency in hierarchical interactions between …


Comparing generative and retrieval-based chatbots in answering patient questions regarding age-related macular degeneration and diabetic retinopathy

KX Cheong, C Zhang, TE Tan, BJ Fenner, WM Wong… - British Journal of …, 2024

… ChatGPT was trained on multiple extremely large datasets, of which Common Crawl (open repository of web crawl data) constituted the bulk of the datasets.30 31 Google Bard was trained on Infiniset (data comprising public forum dialogue …


Comparison of Common Crawl News & GDELT

A El Ouadi, D Beskow - 2024 IEEE International Systems Conference (SysCon), 2024

… Common Crawl data has proven valuable for research and training artificial intelligence, noting that 82% of raw tokens used to train GPT-3 come from the Common Crawl dataset. CC-News, a derived corpus of news data from Common …


Comparison of Machine Learning Algorithms and Large Language Models for Product Categorization

A İhsanoğlu, M Zaval, OT Yıldız - 2024 32nd Signal Processing and Communications …, 2024

This study explores the efficacy of traditional machine learning algorithms and Large Language Models (LLMs) in automating product categorization for online e-commerce platforms. By comparing these methodologies, we assess their performance in …


Compass: Large Multilingual Language Model for South-east Asia

S Maria - arXiv preprint arXiv:2404.09220, 2024

… We categorized these sources into seven types: CommonCrawl, C4, Wikipedia, WebText, Academic, Books, and Code. We upsampled and … We expect that the introduction of biases may originate from CommonCrawl, as this substantial dataset …


Competition concerns with foundation models: a new feast for big tech?

S Mitra - European Competition Journal, 2024

… Some well-known FMs such as GPT-3 and LLaMA have been trained on data from public sources (such as Common Crawl). However, there are concerns that publicly available data would eventually have been completely saturated for training …


Compilation of a Synthetic Judeo-French Corpus

I Nikolova-Stoupak, G Lejeune, E Schaeffer-Lacroix - Proceedings of the 8th Joint …, 2024

This is a short paper describing the process of derivation of synthetic Judeo-French text. Judeo-French is one of a number of rare languages used in speaking and writing by Jewish communities as confined to a particular temporal and …


Complex Word Identification for Italian Language: a dictionary-based approach

L Occhipinti - Sixth International Conference, 2024

… The model used in our study was trained on Wikipedia and Common Crawl datasets6. These embeddings provided vector representations for each word in our dataset, primarily comprising isolated items, allowing us to incorporate contextual …


Compositional Text-to-Image Generation with Dense Blob Representations

W Nie, S Liu, M Mardani, C Liu, B Eckart, A Vahdat - arXiv preprint arXiv:2405.08246, 2024

Existing text-to-image models struggle to follow complex text prompts, raising the need for extra grounding inputs for better controllability. In this work, we propose to decompose a scene into visual primitives - denoted as dense blob representations …


Comprehensive Analysis of Falcon 7B: A State-of-the-Art Generative Large Language Model

M Aridoss, KS Bisht, AK Natarajan - Generative AI: Current Trends and Applications, 2024

This study provides an in-depth analysis of Falcon 7B, a state-of-theart generative language model, emphasizing its complex performance metrics, flexible features, and multifaceted attributes. This chapter thoroughly reviews Falcon 7B’s …


Comprehensive Analysis of Falcon 7B: A State-of-the-Art Generative Large

This study provides an in-depth analysis of Falcon 7B, a state-of-the-art generative language model, emphasizing its complex performance metrics, flexible features, and multifaceted attributes. This chapter thoroughly reviews Falcon 7B's …


Comprehensive analysis of natural language processing

RK Yadav, A Madaan - Global Journal of Engineering and Technology …, 2024

… Huge datasets like the Penn Treebank and the Common Crawl drove the development of more sophisticated NLP models. Transfer learning techniques like BERT enabled finetuning of pre-trained models for specific tasks. TensorFlow and …


Comprehensive review and comparative analysis of transformer models in sentiment analysis

H Bashiri, H Naderi - Knowledge and Information Systems, 2024

Sentiment analysis has become an important task in natural language processing because it is used in many different areas. This paper gives a detailed review of sentiment analysis, including its definition, challenges, and uses. Different …


COMPREHENSIVE STUDY OF CLINICAL ENTITY EXTRACTION AND CLASSIFICATION USING LARGE LANGUAGE MODELS

M Faedi, P Torroni, DA Galassi, DG Grundler…

This project aims to evaluate various techniques for extraction and categorization of clinical terms in unstructured documents. The purpose of this study is to assist automated systems operating in the biomedical domain by highlighting relevant …


Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

A Saxena, AK Bishwas, AA Mishra, R Armstrong - arXiv preprint arXiv:2407.15904, 2024

… For pruning language models, we used XLM-RoBERTA [30] trained on 2.5Tb of filtered CommonCrawl data containing 100 languages, BERTsmall [31] trained on disaster tweets dataset [32] and GPT-2 small [33] finetuned on E2E NLG [34]. …


Compressing Lengthy Context With UltraGist

P Zhang, Z Liu, S Xiao, N Shao, Q Ye, Z Dou - arXiv preprint arXiv:2405.16635, 2024

Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality compression of lengthy context due to the innovative design of the compression and …


Compression Represents Intelligence Linearly

Y Huang, J Zhang, Z Shan, J He - arXiv preprint arXiv:2404.09937, 2024

… Concretely, for assessing knowledge and commonsense, we have compiled texts from the latest Common Crawl dataset. To evaluate coding ability, we have sourced data from GitHub repositories mainly on the Python language since the downstream …


Computational Approaches to Lexical Complexity Prediction and Simplification

Lexical Simplification (LS) automatically replaces difficult to understand words with easier alternatives whilst maintaining the original meaning of a sentence. LS serves as a preliminary step to Text Simplification with the aim of enhancing a text's …


Computational Collective Intelligence: 16th International Conference, ICCCI 2024, Leipzig, Germany, September 9–11, 2024, Proceedings, Part II

This volume contains the first part of the proceedings of the 16th International Conference on Computational Collective Intelligence (ICCCI 2024), held in Leipzig, Germany from 9–11 September 2024. The conference was organized in a hybrid …


Computational criminology: at-scale quantitative analysis of the evolution of cybercrime forums

Cybercrime forums and marketplaces are used by members to share hacking techniques, general community-building discussions, and trade hacking tools. While there is a large corpus of literature studying these platforms, from a cross-forum …


Computational Measures of Language Variation in Textual Utterances

The use of language as a tool for self-expression and social communication is marked by extensive variation in the way that language is used by different people and communities. The computational modeling of this variation studies the social …


Computerized diagnostic decision support systems–a comparative performance study of Isabel Pro vs. ChatGPT4

… ChatGPT4 is trained on Common Crawl, a publicly available dataset and one of the most extensive text datasets. While not trained explicitly for medical diagnosis, ChatGPT4 accesses medical textbooks, medical websites, medical papers, and …


Comquest: an Adaptive Crawler for User Comments on the Web

This thesis introduces Comquest, an adaptive framework designed for the large-scale collection and integration of user comments from the Web. User comments are featured on many websites and there is growing interest in mining and studying user …


Comquest: Large Scale User Comment Crawling and Integration

Z Chen, L He, A Mukherjee, E Dragut - 2024

… To train the model, we collect an extensive dataset from the Common Crawl Project. We filter the webpages that show apparent code signals (eg, loading the commenting system libraries) of the supported commenting systems. …


Confounders in Instance Variation for the Analysis of Data Contamination

B Mehrbakhsha, D Garigliottid, F Martínez-Plumeda…

… 2020) the same approach has been taken and all data points from the evaluation sets that had a 13-gram collision in the pre-training Common Crawl (C4) dataset were removed to tackle contamination. As contamination can involve minor …


Consent in Crisis: The Rapid Decline of the AI Data Commons

S Longpre, R Mahari, A Lee, C Lund, H Oderinwale… - arXiv preprint arXiv …, 2024

… In our study, we focus on three popular, open-source, and permissively licensed data sources which are derived from Common Crawl, the largest publicly available crawl of the web, which has collected and stored hundreds of billions of web pages …


Considering the Role of Fairness in Copyright Fair Use

Moral intuitions regarding when it is "fair" to engage in unauthorized copying or derivation from a copyrighted work may often take into account factors that the fair use statute, 17 USC § 107, does not enumerate among fair use considerations. This …


Conspiracy Theory Detection using Transformers with Multi-task and Multilingual Approaches

The COVID-19 pandemic sparked a new age of conspiracy theories in society. This has become an issue, especially since these theories are mixed in with reasonable arguments that criticize the measures taken by governments and their effects. A way …


Constructing Multimodal Datasets from Scratch for Rapid Development of a Japanese Visual Language Model

K Sasagawa, K Maeda, I Sugiura, S Kurita, N Okazaki… - arXiv preprint arXiv …, 2024

To develop high-performing Visual Language Models (VLMs), it is essential to prepare multimodal resources, such as image-text pairs, interleaved data, and instruction data. While multimodal resources for English are abundant, there is a …


Construction of Text Summarization Corpus in Economics Domain and Baseline Models

S Jumpathong, A Takhom, P Boonkwan… - 2024

… It was pre-trained with a data collection of 101 languages from Common Crawl Due to time and resource constraints, we chose mT5-small… It was pre-trained with a data collection of 101 languages from Common Crawl. Due to time and resource …


Context-aware embeddings for robust multiclass fraudulent URL detection in online social platforms

S Afzal, M Asim, MO Beg, T Baker, AI Awad, N Shamim - Computers and Electrical …, 2024

… Their dataset comprised 2 million URLs from Common Crawl and PhishTank, and they applied three-fold cross-validation with 90% training data. Feature-ranking techniques were used to identify the most important among 14 features. They …


Context-aware Transliteration of Romanized South Asian Languages

C Kirov, C Johny, A Katanova, A Gutkin, B Roark - Computational Linguistics, 2024

While most transliteration research is focused on single tokens such as named entities – eg, transliteration of “અમદાવાદ” from the Gujarati script to the Latin script “Ahmedabad” – the informal romanization prevalent in South Asia and elsewhere …


Context-Relevant Denoising for Unsupervised Domain-Adapted Sentence Embeddings

M Lowe, JD Prusa, JL Leevy, TM Khoshgoftaar - … and Integration for Data Science (IRI …, 2024

… Over the course of this case study, we utilized two information archives as our primary data sources: CommonCrawl News (CC-News), a repository of archived worldwide news events in various languages; and MIMIC-III/IV, a publicly available …


Contextual Chart Generation for Cyber Deception

DD Nguyen, D Liebowitz, S Nepal, SS Kanhere… - arXiv preprint arXiv …, 2024

… (1) HPN-T5 is based on Text-to-Text Transfer Transformer (T5) [42], a highly scalable sequence-to-sequence Transformer pre-trained on a large corpus harvested from Common Crawl. T5 introduced a novel pre-training objective that …


Contextual Word Embedding for Biomedical Knowledge Extraction: a Rapid Review and Case Study

D Vithanage, P Yu, L Wang, C Deng - Journal of Healthcare Informatics Research, 2024

Recent advancements in natural language processing (NLP), particularly contextual word embedding models, have improved knowledge extraction from biomedical and healthcare texts. However, limited comprehensive research compares these models …


Contextualized Word Embeddings in Azerbaijani Language

T Alizada, U Suleymanov, Z Rustamov - 2024 IEEE 18th International Conference on …, 2024

This study delves into the implementation of Contextualized Word Embeddings in Azerbaijani. To this end, the project is capitalizing on the recent advancements in NLP, in particular, transformer models such as RoBERTa and GPT-2, to derive the …


Contextually Enriched Meta-Learning Ensemble Model for Urdu Sentiment Analysis Symmetry 2023, 15, 645

K Ahmed, MI Nadeem, D Li, Z Zheng, N Al-Kahtani… - 2023

… On the other hand, this model makes use of pre-trained word vectors that were trained on “common crawl” and “Wikipedia” through the use of the fastText model [82]. The CBOW algorithm with position-weighting is used to train this word vector. After …


Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

K Fujii, T Nakamura, M Loem, H Iida, M Ohi, K Hattori… - arXiv preprint arXiv …, 2024

Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an …


Continuous or Discrete, That Is the Question: A Survey on Large Multi-Modal Models from the Perspective of Input-Output Space Extension

Z Li, J Zhang, D Wang, Y Wang, X Huang, Z Wei - 2024

… OBELICS [260] is constructed from HTML files obtained from Common Crawl dumps, and the resulting documents maintain the original linearity of images and texts as they appeared on the websites, while removing spam and ads. Exposing the …


CONTOR: Benchmarking strategies for completing ontologies with plausible missing rules

L Na, T Bailleux, Z Bouraoui, S Schockaert - 2024

We consider the problem of finding plausible rules that are missing from a given ontology. A number of strategies for this problem have already been considered in the literature. Little is known about the relative performance of these strategies …


Controllable Sentence Simplification in Dutch

T Seidl, V Vandeghinste - Computational Linguistics in the Netherlands Journal, 2024

Text simplification aims to reduce complexity in vocabulary and syntax, enhancing the readability and comprehension of text. This paper presents a supervised sentence simplification approach for Dutch using a pre-trained large language …


Controllable Sentence Simplification in Swedish Using Control Prefixes and Mined Paraphrases

J Monsen, A Jönsson - Proceedings of the 2024 Joint International Conference …, 2024

… 2020), an opensource repository with tools to download and clean Common Crawl snapshots. Each file in the CC-100 corpus contains … The Common Crawl snapshots contain a large portion of low-quality data, including offensive and …


Cook Smarter Not Harder: Enhancing Learning Capacity in Smart Ovens with Supplementary Data

AM MIGEA, VA NEGRU, T Sebastian-Antonio… - 2024 IEEE 20th …, 2024

This paper presents a novel approach to improve cooking experiences by integrating advanced natural language processing and machine learning techniques within the food industry. The research focuses on developing a …


Copilot for Microsoft 365: A Comprehensive End-user Training Plan for Organizations

This thesis presents a comprehensive end user training plan for Copilot for Microsoft 365, a generative AI tool that integrates with Microsoft’s productivity suite of applications. The research is conducted within the context of Sulava Ltd., a Finnish …


Copyleft for Alleviating AIGC Copyright Dilemma: What-if Analysis, Public Perception and Implications

X Guo, Y Li, Y Peng, X Wei - arXiv preprint arXiv:2402.12216, 2024

As AIGC has impacted our society profoundly in the past years, ethical issues have received tremendous attention. The most urgent one is the AIGC copyright dilemma, which can immensely stifle the development of AIGC and greatly cost the entire …


CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation

Y Cheng, K Mao, Z Zhao, G Dong, H Qian, Y Wu… - arXiv preprint arXiv …, 2024

Retrieval-Augmented Generation (RAG) has become a powerful paradigm for enhancing large language models (LLMs) through external knowledge retrieval. Despite its widespread attention, existing academic research predominantly focuses …


CoRePooL—Corpus for Resource‐Poor Languages: Badaga Speech Corpus

HB Barathi Ganesh, G Jyothish Lal, R Jairam… - … Speech Recognition and …, 2024

This chapter presents a corpus named CoRePooL that stands for Corpus for Resource‐Poor Languages. As voice‐specific human‐machine interaction applications are accelerated by deep learning algorithms, the lack of resources …


Corpus Considerations for Annotator Modeling and Scaling

OO Sarumi, B Neuendorf, J Plepi, L Flek, J Schlötterer… - arXiv preprint arXiv …, 2024

Recent trends in natural language processing research and annotation tasks affirm a paradigm shift from the traditional reliance on a single ground truth to a focus on individual perspectives, particularly in subjective tasks. In scenarios where …


CorpusNÓS: A massive Galician corpus for training large language models

I de-Dios-Flores, SP Suárez, CC Pérez, DB Outeiriño… - Proceedings of the 16th …, 2024

CorpusNÓS is a massive Galician corpus made up of 2.1 B words primarily devised for training large language models. The corpus sources are varied and represent a relatively wide range of genres. CorpusNÓS is, to the best of our knowledge, the …


Correcting Auditory Spelling Mistakes in Jordanian Dialect Using Machine Learning Techniques

M Smadi, G Abandah - 2024 15th International Conference on Information and …, 2024

This paper explores the application of machine learning techniques, specifically leveraging an efficient transformer model known as Byt5, to rectify Arabic auditory spelling mistakes. The model is trained using datasets with synthetic errors …


Cost-Effective Big Data Orchestration Using Dagster: A Multi-Platform Approach

H Picatto, G Heiler, P Klimek - arXiv preprint arXiv:2408.11635, 2024

… The Common Crawl dataset is a particular promising avenue for this kind of research, as it also contains historic data which allows one to … The temporal partitioning aligns with the Common Crawl1 dataset used, facilitating efficient data …


Cost-Effective Event Mining on the Web via Event Source Page Discovery and Data API Construction

YH Lin, CH Chang, HM Chuang, XS Lin, T Yeh… - IEEE Access, 2024

… • Compared to the less than 1% of event pages found in the Common Crawl and Clueweb datasets when crawling the entire web, a significantly higher proportion of events, 13.3%, were identified when specifically targeting the websites of potential …


Cost-efficient prompt engineering for unsupervised entity resolution in the product matching domain

N Nananukul, K Sisaengsuwanchai, M Kejriwal - Discover Artificial Intelligence, 2024

… Web Data Commons (WDC) [52]: The full version of this dataset contains 26 million products and descriptions from different e-commerce websites scraped from the Web as part of the Web Data Commons (Common Crawl). The products are …


Count information: retrieving and estimating cardinality of entity sets from the web

… (2007)], and massive crawls of the Web released regularly by the Common Crawl since 2008 [Crawl(2008)]. Wikipedia, for instance, is a … While C4 exclusively comes from pre-processing and filtering the Common Crawl data, the Pile dataset …


Counter (media) Intelligence and Visioning: An Interview with Adam Harvey

… Among the largest and most popular datasets used in AI research projects are Common Crawl (for NLP), and ImageNet and COCO (for computer vision), which are all derived from user-generated content. In my research project Exposing.ai, I’ve …


Course-Skill Atlas: A national longitudinal dataset of skills taught in US higher education curricula

A Javadian Sabet, SH Bana, R Yu, MR Frank - Scientific Data, 2024

Higher education plays a critical role in driving an innovative economy by equipping students with knowledge and skills demanded by the workforce. While researchers and practitioners have developed data systems to track detailed occupational skills …


Crafting clarity: Leveraging large language models to decode consumer reviews

SV Praveen, P Gajjar, RK Ray, A Dutt - Journal of Retailing and Consumer Services, 2024

Large Language Models (LLMs) have emerged as powerful tools for understanding consumer perceptions and extracting insights from unstructured textual data. This study investigates the effectiveness of LLMs in comprehending consumer opinions …


Creating Ad Campaigns Using Generative AI Check for updates

A Bulut, B Arslan - Applications of Generative AI

Search campaigns consist of ad groups. An ad group contains a related set of keywords and ads. During an online campaign, search advertisers experiment with different marketing messages such as subtle vs. strong being used in ad copies, with …


Creating Ad Campaigns Using Generative AI

A Bulut, B Arslan - Applications of Generative AI, 2024

… In particular, T5 and PEGASUS models were trained on 750 GB of English-language text from the public Common Crawl web scrape while the BART model was trained on the CNN/Daily Mail dataset, which contains roughly 300K unique news articles …


Creating Parallel Corpora for Ukrainian: a German-Ukrainian Parallel Corpus (ParaRook|| DE-UK)

M Shvedova, A Lukashevskyi - Proceedings of the Third Ukrainian Natural Language …, 2024

Parallel corpora are currently a popular and vibrantly developing category of linguistic resources, used both in literature and translation studies, as well as in the field of NLP. For Ukrainian, though, there are still not enough significant parallel …


Creation of AI-driven Smart Spaces for Enhanced Indoor Environments--A Survey

A Varol, NH Motlagh, M Leino, S Tarkoma, J Virkki - arXiv preprint arXiv:2412.14708, 2024

Smart spaces are ubiquitous computing environments that integrate diverse sensing and communication technologies to enhance space functionality, optimize energy utilization, and improve user comfort and well-being. The integration of emerging AI …


Creative Writers' Attitudes on Writing as Training Data for Large Language Models

KI Gero, M Desai, C Schnitzler, N Eom, J Cushman… - arXiv preprint arXiv …, 2024

The use of creative writing as training data for large language models (LLMS) is highly contentious. While some argue that such use constitutes "fair use" and therefore does not require consent or compensation, others argue that consent and …


CroissantLLM: A Truly Bilingual French-English Language Model

M Faysse, P Fernandes, N Guerreiro, A Loison, D Alves… - arXiv preprint arXiv …, 2024

We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local …