Topology Only Pre-Training: Towards Generalised Multi-Domain Graph Models

AO Davies, R Green, NS Ajmeri, TMS Filho

The principal benefit of unsupervised representation learning is that a pre-trained model can be fine-tuned where data or labels are scarce. Existing approaches for graph representation learning are domain specific, maintaining consistent node and …


TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training

W Liang, T Liu, L Wright, W Constable, A Gu… - arXiv preprint arXiv …, 2024

The development of large language models (LLMs) has been instrumental in advancing state-of-the-art natural language processing applications. Training LLMs with billions of parameters and trillions of tokens require sophisticated distributed …


Toward a Holistic Evaluation of Robustness in CLIP Models

W Tu, W Deng, T Gedeon - arXiv preprint arXiv:2410.01534, 2024

Contrastive Language-Image Pre-training (CLIP) models have shown significant potential, particularly in zero-shot classification across diverse distribution shifts. Building on existing evaluations of overall classification robustness, this work aims …


Toward Better Education Quality through Students' Sentiment Analysis Using AutoML

C Simionescu, D Marcu, MS Măciucă - … . Broad Research in Artificial Intelligence and …, 2024

Sentiment analysis from students' interactions with learning environments is a topic of interest for researchers in the field of education because it can make important contributions to improving the quality of instructional processes through …


Toward Inference-optimal Mixture-of-Expert Large Language Models

L Yun, Y Zhuang, Y Fu, EP Xing, H Zhang - arXiv preprint arXiv:2404.02852, 2024

Mixture-of-Expert (MoE) based large language models (LLMs), such as the recent Mixtral and DeepSeek-MoE, have shown great promise in scaling model size without suffering from the quadratic growth of training cost of dense transformers …


Toward Open-Source AI Systems as Digital Public Goods: Definitions, Hopes and Challenges

L Gimpel - New Frontiers in Science in the Era of AI

… Training data might be proprietary, free but not open as in Google Trends, have unclear provenance or feature copyright-protected data (including the massive “open” datasets scrapped from the web used to train generative AI models such as …


Toward Robust Arabic AI-Generated Text Detection: Tackling Diacritics Challenges

H Alshammari, K Elleithy - Information, 2024

Current AI detection systems often struggle to distinguish between Arabic human-written text (HWT) and AI-generated text (AIGT) due to the small marks present above and below the Arabic text called diacritics. This study introduces robust Arabic text …


Toward Robust, Reliable, and Generalizable Models for Tabular Data

Tabular data–spreadsheet-type data with rows and columns–is widely used across many domains and real-world applications, from finance to healthcare to natural and social sciences. However, tabular data modeling has received far less attention, and …


Toward Understanding Training Data Attacks In Artificial Intelligence And Machine Learning Models

… With all of these changes, the most accessible source of training data continues to be open source data, such as the Common Crawl datasets(Tom B. Brown, 2020) and others that have been made publicly available. Only extremely large technology …


Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence

A Basdevant, C François, V Storchan, K Bankston… - arXiv preprint arXiv …, 2024

Over the past year, there has been a robust debate about the benefits and risks of open sourcing foundation models. However, this discussion has often taken place at a high level of generality or with a narrow focus on specific technical attributes. In …


Towards a Scalable Geoparsing Approach for the Web

The ongoing surge in web data generation and storage, coupled with embedded geographic information, holds immense potential for enhancing search applications across diverse domains. However, extracting geographic information for further …


Towards a small language model powered chain‐of‐reasoning for open‐domain question answering

J Roh, M Kim, K Bae - ETRI Journal, 2024

… Unlike the ONUS approach [10], which extracts simple sub-question examples from a Common Crawl, our method emphasizes the autonomous generation of a dataset without human intervention facilitated by the use of LLMs. Consequently, we …


Towards a sociology of recurrent events. Constellations of cultural change around Eurovision in 18 countries (1981–2021)

L Carbone, J Mijs, T van Dooremalen, S Daenekindt - Poetics, 2024

… First, we used CMD with a corpus that was pre-trained on Common Crawl data using fastText English embeddings. Common Crawl is a vast and heterogeneous source of information that crawls web archives and freely provides their data to the …


Towards a Unified Terminology: A study on Siemens Energy Technical Information Systems

… The dataset includes the Common Crawl archive, an extensive collection of web-extracted text [5], as well as the WebText dataset, books … GPT-3’s training includes a staggering 400 billion tokens from Common Crawl, 19 billion from WebText, 12 …


Towards adaptive support for self‐regulated learning of causal relations: Evaluating four Dutch word vector models

HJ Pijeira‐Díaz, S Braumann, J van de Pol, T van Gog… - British Journal of …, 2024

Advances in computational language models increasingly enable adaptive support for self‐regulated learning (SRL) in digital learning environments (DLEs; eg, via automated feedback). However, the accuracy of those models is a common concern …


Towards Automated Solution Recipe Generation for Industrial Asset Management with LLM

N Zhou, D Patel, S Lin, F O'Donncha - arXiv preprint arXiv:2407.18992, 2024

This study introduces a novel approach to Industrial Asset Management (IAM) by incorporating Conditional-Based Management (CBM) principles with the latest advancements in Large Language Models (LLMs). Our research introduces an …


Towards automatic question generation using pre-trained model in academic field for Bahasa Indonesia

D Suhartono, MRN Majiid, R Fredyan - Education and Information Technologies, 2024

… This model is a potent tool for transforming our text data into numerical representations because it was previously trained on the sizable Common Crawl Dataset in Indonesian. These embeddings are essential for later modeling and …


Towards Better Language Models: Algorithms, Architectures, and Applications

This thesis explores the advancement of language models by focusing on three important perspectives: Algorithms, Architectures, and Applications. We aim to improve the performance, efficiency, and practical usage of these language models …


Towards Better Statistical Understanding of Watermarking LLMs

Z Cai, S Liu, H Wang, H Zhong, X Li - arXiv preprint arXiv:2403.13027, 2024

In this paper, we study the problem of watermarking large language models (LLMs). We consider the trade-off between model distortion and detection ability and formulate it as a constrained optimization problem based on the green-red algorithm …


Towards Building Multilingual Language Model for Medicine

P Qiu, C Wu, X Zhang, W Lin, H Wang, Y Zhang… - arXiv preprint arXiv …, 2024

In this paper, we aim to develop an open-source, multilingual language model for medicine, that the benefits a wider, linguistically diverse audience from different regions. In general, we present the contribution from the following aspects: first, for …


Towards Chapterisation of Podcasts Detection of Host and Structuring Questions in Radio Transcripts

This Master thesis investigates the application of Bidirectional Encoder Representations from Transformers (BERT) on podcast to identify the host and detect structuring questions within each episode. This research is conducted on an …


Towards Cross-Lingual LLM Evaluation for European Languages

K Thellmann, B Stadler, M Fromm, JS Buschhoff… - arXiv preprint arXiv …, 2024

… Section 4.2 shows the connection between the proportion of languages in the CommonCrawl dataset and the performance of the current best open weights models. Lastly, Section 4.3 highlights how the performance of the models depend on the …


Towards Data Governance of Frontier AI Models

Data is essential to train and fine-tune today’s frontier artificial intelligence (AI) models and to develop future ones. To date, academic, legal, and regulatory work has primarily addressed how data can directly harm consumers and creators, such …


Towards Data-efficient Machine Learning Systems

The amount of data available to train modern machine learning systems has been increasing rapidly, so much so that we're using, eg, entirety of the publicly available text data to train state-of-the-art (SoTA) large language models (LLMs), interaction …


Towards detection and explanation of factual inconsistencies

Factual inconsistencies in text, which include a range of errors from minor inaccuracies to substantial distortions, present a significant challenge in the realm of information dissemination. These inconsistencies, whether unintentional or the …


Towards Disfluency Annotated Corpora for Indian Languages

C Kochar, V Mujadia, P Mishra, DM Sharma

In the natural course of spoken language, individuals often engage in thinking and self-correction during speech production. These instances of interruption or correction are commonly referred to as disfluencies. When preparing data for …


Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models

J Wang, A Jatowt, Y Cai - arXiv preprint arXiv:2406.01863, 2024

In the evolving field of Natural Language Processing, understanding the temporal context of text is increasingly crucial. This study investigates methods to incorporate temporal information during pre-training, aiming to achieve effective time-aware …


Towards Enhanced Information Access in Finance: A Dataset for Table Structure Understanding in Annual Securities Reports

K Kadowaki, Y Kimura, H Ototake - … for Financial Engineering and Economics (CIFEr), 2024

… The WDC Web Table Corpus 2015 [8] structured 230 million tables in web pages from CommonCrawl by classifying them into categories such as relational, entity, matrix, and other, as well as identifying the presence and location of headers and …


Towards Estimating Personal Values in Song Lyrics

AM Demetriou, J Kim, S Manolios, C Liem - arXiv preprint arXiv:2408.12694, 2024

Most music widely consumed in Western Countries contains song lyrics, with US samples reporting almost all of their song libraries contain lyrics. In parallel, social science theory suggests that personal values - the abstract goals that guide our …


Towards Federated Large Language Models: Motivations, Methods, and Future Directions

Y Cheng, W Zhang, Z Zhang, C Zhang, S Wang, S Mao - … Communications Surveys & …, 2024

Large Language Models (LLMs), such as LLaMA and GPT-4, have transformed the paradigm of natural language comprehension and generation. Despite their impressive performance, these models still face certain challenges, including the …


Towards Green AI. A methodological survey of the scientific literature

E Barbierato, A Gatti - IEEE Access, 2024

The pervasive deployment of Deep Learning models has recently prompted apprehensions regarding their ecological footprint, owing to the exorbitant levels of energy consumption necessitated by the training and inference processes. The term …


Towards interactive information seeking: conversational question answering

… To further advance the field, additional conversational search benchmarks have been introduced, such as ORQuAC [93], which extends CAsT and QuAC with more conversations constructed from Natural Questions and additional passages from …


Towards Large Language Models for Everyone: Instruction Following, Knowledge Retrieval and Multilingualism

Large language models (LLMs) have significantly advanced the field of Natural Language Processing and demonstrated the potential to fuel a variety of AI applications. Nonetheless, building them in a way that maximally benefits the very …


Towards Machine Translation Based on Monolingual Texts

… The CommonCrawl1 project carries out periodic web crawls and publishes the crawled data in an open … the CommonCrawl corpus. For example, the open source OSCAR2 project compiled a large multilingual corpus by language …


Towards More Factual Large Language Models: Parametric and Non-parametric Approaches

Large language models (LLMs) are increasingly important in assisting people to access information, ranging from simple factoid questions such as “where is the world’s largest ice sheet located” to complex questions that require accessing real-time …


Towards Privacy Preserving LLMs Training

B Buesser - Large Language Models in Cybersecurity: Threats …, 2024

Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) …


Towards quantifying politicization in foreign aid project reports

S Wang, G Eggers, ART Georgiadis, TA Đo, L Gontard… - Proceedings of the Second …, 2024

We aim to develop a metric of politicization by investigating whether this concept can be operationalized computationally using document embeddings. We are interested in measuring the extent to which foreign aid is politicized. Textual reports of foreign …


Towards Region-aware Bias Evaluation Metrics

A Borah, A Garimella, R Mihalcea - arXiv preprint arXiv:2406.16152, 2024

… To measure biases in different data domains and regions, we extract region-aware topics using the GeoWAC dataset which spans Common Crawl separated by regions, and create a WEAT-style evaluation setup using these topics. Data. We consider two …


Towards reliability and interactive debugging for large language models

Large language models (LLMs) have permeated our everyday lives and are used in critical decision-making scenarios that can affect millions of people. Despite their impressive progress, model deficiencies may result in exacerbating harmful biases …


Towards Robust Evaluation: A Comprehensive Taxonomy of Datasets and Metrics for Open Domain Question Answering in the Era of Large Language Models

A Srivastava, A Memon - arXiv preprint arXiv:2406.13232, 2024

Open Domain Question Answering (ODQA) within natural language processing involves building systems that answer factual questions using large-scale knowledge corpora. Recent advances stem from the confluence of several factors …


Towards Safe Multilingual Frontier AI

A Kanepajs, V Ivanov, R Moulange - arXiv preprint arXiv:2409.13708, 2024

… Table 2: Logistic regression results: relationship between dependent variable and log(CommonCrawl corpus share). 2400 observations across 24 languages (100 per language) for each regression, with Holm–Bonferroni adjusted p-values. Adjusted …


Towards Systematic Monolingual NLP Surveys: GenA of Greek NLP

J Bakagianni, K Pouli, M Gavriilidou, J Pavlopoulos - arXiv preprint arXiv:2407.09861, 2024

Natural Language Processing (NLP) research has traditionally been predominantly focused on English, driven by the availability of resources, the size of the research community, and market demands. Recently, there has been a noticeable shift …


Towards the automatic creation of NER systems for new domains

E Matos, M Rodrigues, A Teixeira - Proceedings of the 16th International Conference …, 2024

… ner-roberta (ROBERTA): xlm-roberta-base-trimmed-pt-600002 model based in xlm-roberta-base, pretrained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper Unsupervised Crosslingual …


Towards the Machine Translation of Scientific Neologisms

… OSCAR-fr 22.01 is a French cleaned subset of Common Crawl, which was also partly used to train BLOOM. It consists of 42 billion words (382 GB). Figure 4 shows that 15.8% of FranceTerme target (French) terms do not appear even a single time in …


Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference

M Shen, M Umar, K Maeng, GE Suh, U Gupta - arXiv preprint arXiv:2412.11854, 2024

… We construct our retrieval indices using a 100M document chunk subset of Common Crawl [18]. Each document chunk is made up of 100 tokens, encoded into a vector. We explore three different dense retriever indices: an accuracyoriented one …


Towards Video Understanding through Language in Real-life Settings

Videos have become an integral part of our daily lives, with a rapidly growing number on YouTube, Netflix, and TikTok serving as testimony to their widespread popularity. Behind the simplicity of their interfaces and user experiences, the …


Towards Website X-Ray for Europe's Municipalities: Unveiling Digital Transformation with Multimodal Embeddings

J Gerber, B Kreiner, J Saxer, A Weiler - International Conference on Information …, 2025

Local governments are going digital! To track their progress, researchers want to analyze their websites. This study explores finding these websites and understanding their content. Three methods for website discovery were compared …


Toxic Memes Recognition Through with Multimodal Bidirectional Cross-Attention

… Common Crawl (CC) Common Crawl is a large-scale web corpus that provides diverse and extensive data, capturing a wide range of topics and domains. Models pre-trained on Common Crawl benefit from exposure to varied language usages …


Toxicity Detection for Free

Z Hu, J Piet, G Zhao, J Jiao, D Wagner - arXiv preprint arXiv:2405.18822, 2024

… An analysis of undesirable content in the common crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2 …


Toxicity Mitigation with Retrieval-Augmented Language Models applied to English and Portuguese Text Generation

Grandes modelos de linguagem alcançaram capacidades impressionantes de geração e compreensão textual através de treino autosupervisionado ao longo dos anos. Enquanto aprendem a prever a próxima palavra, esses modelos mapeiam o …


Toxicity of the Commons: Curating Open-Source Pre-Training Data

C Arnett, E Jones, IP Yamshchikov, PC Langlais - arXiv preprint arXiv:2410.22587, 2024

Open-source large language models are becoming increasingly available and popular among researchers and practitioners. While significant progress has been made on open-weight models, open training data is a practice yet to be adopted by …


Training and evaluation of vector models for Galician

M Garcia - Language Resources and Evaluation, 2024

This paper presents a large and systematic assessment of distributional models for Galician. To this end, we have first trained and evaluated static word embeddings (eg, word2vec, GloVe), and then compared their performance with that of current …


Training BERT Models to Carry over a Coding System Developed on One Corpus to Another

D Galambos, P Zsamboki - Proceedings of the 2024 Joint International Conference …, 2024

This paper describes how we train BERT models to carry over a coding system developed on the paragraphs of a Hungarian literary journal to another. The aim of the coding system is to track trends in the perception of literary translation around …


Training Bilingual LMs with Data Constraints in the Targeted Language

S Seto, M ter Hoeve, H Bai, N Schluter, D Grangier - arXiv preprint arXiv:2411.12986, 2024

… on combinations of German (target) and English (auxiliary) with varying existing English datasets based on the common crawl: mC4 (Xue… different snapshots of the common crawl. They are all aimed at general language model pretraining …


Training Data for Large Language Model

Y Ju, H Ma - arXiv preprint arXiv:2411.07715, 2024

… 通过提供这些数据,Common Crawl支持了各种研究和开发活 动,包括但不限于搜索 引擎优化,语言模型训练,社会数据研究等.由于其数据量… ,Common Crawl成为了许多 数据驱动项目的重要基础,在GPT-3的训练中,Common Crawl贡献 了其中约60%数据的 …


Training Data for the Price of a Sandwich1

… , this report sheds light on the implications of Common Crawl’s popularity by exploring the values that guide Common Crawl itself in-depth. … ) whether Common Crawl’s data had been used and b) what specific version of Common Crawl was …


Training Task Experts through Retrieval Based Distillation

J Ge, X Jia, V Viswanathan, H Luo, G Neubig - arXiv preprint arXiv:2407.05463, 2024

One of the most reliable ways to create deployable models for specialized tasks is to obtain an adequate amount of high-quality task-specific data. However, for specialized tasks, often such datasets do not exist. Existing methods address this by …


Transfer Learning in the Era of Large Language Models

Natural language processing has made great strides in recent years, largely driven by developments in large-scale neural networks trained on large quantities of textual data. This thesis is a collection of work studying transfer learning—the transfer of …


Transformer verbatim in-context retrieval across time and scale

K Armeni, M Pranjić, S Pollak - Proceedings of the 28th Conference on …, 2024

To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) …


Transformer-Based Named Entity Recognition Model-Tamil Language Check for updates

K Dhayalan, N Sultanova, J Mustafina, P Daud - Data Science and Emerging Technologies …

This work presents different transformer-based approaches to the named entity recognition problem in the Tamil language. The WikiANN-ta dataset is used for training, validating and testing the model. Person, organization and location tags are …


Transformer-Driven Models for Language, Vision, and Multimodality

M Luo, T Gokhale, N Varshney, Y Yang, C Baral - Advances in Multimodal Information …, 2024

In this chapter, we will learn about the modeling and learning techniques that drive multimodal applications. We will focus specifically on the recent advances in transformer-based modeling for natural language understanding, and image …


Transformers and large language models in healthcare: A review

S Nerella, S Bandyopadhyay, J Zhang, M Contreras… - Artificial Intelligence in …, 2024

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning …


Transformers and tradition: using Generative AI and Deep Learning for financial markets prediction

Artificial intelligence has revolutionized numerous industries, and financial markets are no exception. With the ability to process vast amounts of data quickly and accurately, AI algorithms have been increasingly used in finance to predict stock …


Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures

K Singhal, J Bedi - Proceedings of the 7th Workshop on Challenges and …, 2024

Over the past years, researchers across the globe have made significant efforts to develop systems capable of identifying the presence of hate speech in different languages. This paper describes the team Transformers’ submission to the subtasks …


Transformers for Detection of Distressed Cardiac Patients with an ICD Based on Danish Text Messages

JDW Andersen, ML Jensen, UK Wiil, S Skovbakke… - 2023 IEEE International …, 2023

Cardiac patients with implantable cardioverter defibrillator devices frequently exhibit signs of anxiety and depression (termed "Distressed"). Early detection of these patients is vital for evaluation, intervention, and prevention against relapse …


Transformers on Dynamical Systems-An Exploration of In-context Learning

Large Language Models (LLMs) have shown to be highly effective at performing in-context learning, where, given a prompt, the model can learn from the prompt and complete the sequence without needing to perform additional gradient steps or fine-tuning. In …


Transformers@ DravidianLangTech-EACL2024: Sentiment Analysis of Code-Mixed Tamil Using RoBERTa

K Singhal, J Bedi - Proceedings of the Fourth Workshop on Speech, Vision …, 2024

In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team Transformers’ submission to the Caste/Immigration …


Transformers@ LT-EDI-EACL2024: Caste and Migration Hate Speech Detection in Tamil Using Ensembling on Transformers

K Singhal, J Bedi - Proceedings of the Fourth Workshop on Language …, 2024

In recent years, there has been a persistent focus on developing systems that can automatically identify the hate speech content circulating on diverse social media platforms. This paper describes the team “Transformers” submission to the Caste …


Transforming Conversational AI

We were motivated to write this book by the launch in November 2022 of ChatGPT and by the ensuing excitement and disruption across the world of Conversational AI. The book is written for a broad audience who are already working, starting to work …


Transforming Dutch: Debiasing Dutch Corefence Resolution Systems for Non-binary Pronouns

Gender-neutral pronouns are increasingly being introduced across Western languages, and are continuously more frequently being adopted by non-binary individuals. Recent evaluations have however demonstrated that English language …


TransGPT: Multi-modal Generative Pre-trained Transformer for Transportation

P Wang, X Wei, F Hu, W Han - arXiv preprint arXiv:2402.07233, 2024

Natural language processing (NLP) is a key component of intelligent transportation systems (ITS), but it faces many challenges in the transportation domain, such as domain-specific knowledge and data, and multi-modal inputs and outputs. This …


Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation

E Yang, D Lawrie, J Mayfield, DW Oard, S Miller - arXiv preprint arXiv:2401.04810, 2024

Prior work on English monolingual retrieval has shown that a cross-encoder trained using a large number of relevance judgments for query-document pairs can be used as a teacher to train more efficient, but similarly effective, dual-encoder student …


Translating and predicting document structure for medical domain scientific abstracts

SA Rauf, F Yvon - Computer Speech & Language, 2024

Abstract Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies …


Translating QA is Enough: A Key to Unlocking In-context Cross-lingual Performance

S Kim, D Ki, Y Kim, J Lee - ICML 2024 Workshop on In-Context Learning

Multilingual large language models (MLLMs) have demonstrated significant cross-lingual capabilities through in-context learning. Existing approaches typically construct monolingual few-shot examples, either in the source or target language. However …


Translating scientific abstracts in the bio-medical domain with structure-aware models

SA Rauf, F Yvon - Computer Speech and Language, 2024

Machine Translation (MT) technologies have improved in many ways and generate usable outputs for a growing number of domains and language pairs. Yet, most sentence based MT systems struggle with contextual dependencies, processing …


Translation from Historical to Contemporary Japanese Using Japanese T5

H Usui, K Komiya - Proceedings of the Joint 3rd International Conference …, 2023

This paper presents machine translation from historical Japanese to contemporary Japanese using a Text-to-Text Transfer Transformer (T5). The result of the previous study that used neural machine translation (NMT), Long Short Term Memory (LSTM) …


TransLSTM: A hybrid LSTM-Transformer model for fine-grained suggestion mining

S Riaz, A Saghir, MJ Khan, H Khan, HS Khan, MJ Khan - Natural Language …, 2024

… Each model, excluding BERT, which employs BERT-BASE-Uncased, undergoes testing using simple text vectorization, GloVe, and Common Crawl. Notably, the GloVe word embedding surpasses common text vectorization and Common Crawl in …


TraQuLA: Transparent Question Answering Over RDF Through Linguistic Analysis

E Zimina, K Järvelin, J Peltonen, A Ranta… - International Conference on …, 2024

Answering complex questions over knowledge graphs has gained popularity recently. Systems based on large language models seem to achieve top performance. However, these models may generate content that looks reasonable …


Treading water: new data on the impact of AI ethics information sessions in classics and ancient language pedagogy

EAS Ross, J Baines - Journal of Classics Teaching, 2024

Over 2023, many universities and policy organisations in the higher education (HE) sector are working to create guiding principles and guidelines for the use of generative artificial intelligence (AI) in HE Teaching and Learning (T&L). Despite …


TriLM vs FloatLM: Ternary LLMs are more Performant than Quantized FP16 LLMs

A Kaushal, T Vaidhya, T Pandey, A Bhagat, I Rish - … 2024 Workshop on Foundation Models in …

Ternary LLMs offer significantly better performance for their size (measured in bits) than the models trained and deployed in FP16/BF16. Given the widespread usage of quantization before deployment and advancements in Post Training Quantization …


Trustworthy and Responsible AI for Human-Centric Autonomous Decision-Making Systems

F Dehghani, M Dibaji, F Anzum, L Dey, A Basdemir… - arXiv preprint arXiv …, 2024

Artificial Intelligence (AI) has paved the way for revolutionary decision-making processes, which if harnessed appropriately, can contribute to advancements in various sectors, from healthcare to economics. However, its black box nature …


Tucano: Advancing Neural Text Generation for Portuguese

NK Corrêa, A Sen, S Falk, S Fatimah - arXiv preprint arXiv:2411.07854, 2024

Significant advances have been made in natural language processing in recent years. However, our current deep learning approach to language modeling requires substantial resources in terms of data and computation. One of the side effects of this …


Turing Representational Similarity Analysis (RSA): A Flexible Method for Measuring Alignment Between Human and Artificial Intelligence

M Ogg, R Bose, J Scharf, C Ratto, M Wolmetz - arXiv preprint arXiv:2412.00577, 2024

As we consider entrusting Large Language Models (LLMs) with key societal and decision-making roles, measuring their alignment with human cognition becomes critical. This requires methods that can assess how these systems represent …


Turning Point

Does this headline serve to characterize the enormously rapid and profound changes that information technology (IT) is currently causing in all areas of our lives? Is the comparison to an eighth day of creation exaggerated? After reading the …


Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models

K Pipatanakul, P Manakul, N Nitarach… - arXiv preprint arXiv …, 2024

… In our study, we begin with a total of 40 packs of Thai CommonCrawl data with a cut-off date of September 2023 and perform HTML extraction from scratch using Trafilatura2. This process results in a significantly larger dataset, comprising …


Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages

E Bayes, IA Azime, JO Alabi, J Kgomo, T Eloundou… - arXiv preprint arXiv …, 2024

Evaluations of Large Language Models (LLMs) on knowledge-intensive tasks and factual accuracy often focus on high-resource languages primarily because datasets for low-resource languages (LRLs) are scarce. In this paper, we present Uhura -- a …


Ultra-Sparse Memory Network

Z Huang, Q Min, H Huang, D Zhu, Y Zeng, R Guo… - arXiv preprint arXiv …, 2024

It is widely acknowledged that the performance of Transformer models is exponentially related to their number of parameters and computational complexity. While approaches like Mixture of Experts (MoE) decouple parameter count from …


Unboxing Occupational Bias: Grounded Debiasing LLMs with US Labor Data

A Gorti, M Gaur, A Chadha - arXiv preprint arXiv:2408.11247, 2024

Large Language Models (LLMs) are prone to inheriting and amplifying societal biases embedded within their training data, potentially reinforcing harmful stereotypes related to gender, occupation, and other sensitive categories. This issue …


Uncovering Inverse Generativity: An Exploratory Prompt Analysis in LLM Platforms

Extended generativity theory states that while generativity lead to more users, more users also affect product boundaries of a platform. We seek to uncover the complex relationship by turning to large language model (LLM) platforms, such as ChatGPT …


Understanding College Students' Satisfaction With ChatGPT: An Exploratory And Predictive Machine Learning Approach Using Feature Engineering

K Pabreja, N Pabreja - MIER Journal of Educational Studies Trends and …, 2024

Artificial Intelligence (AI) technologies are continually improving and becoming more pervasive in many facets of our lives. ChatGPT is one such cutting-edge artificial intelligence application, and it has received a lot of worldwide media attention …


Understanding Emergent Abilities of Language Models from the Loss Perspective

Z Du, A Zeng, Y Dong, J Tang - arXiv preprint arXiv:2403.15796, 2024

Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) …


Understanding Large Language Models

Today, finding someone who hasn’t heard of ChatGPT, the AI chatbot that took the world by storm, is hard. ChatGPT—and its competitors such as Google Bard, Microsoft Bing Chat, etc.—are part of a broader area in AI known as large language …


Understanding latent affective bias in large pre-trained neural language models

A Kadan, P Deepak, S Bhadra, MP Gangan, VL Lajish - Natural Language …, 2024

Groundbreaking inventions and highly significant performance improvements in deep learning based Natural Language Processing are witnessed through the development of transformer based large Pre-trained Language Models (PLMs). The …


Understanding LLM's: Comprehensive Overview

Y Liua, H Hea, T Hana, X Zhanga, M Liua, J Tiana…

The introduction of ChatGPT has led to a significant increase in the utilization of Large Language Models (LLMs) for addressing downstream tasks. There’s an increasing focus on cost-efficient training and deployment within this context. Low-cost …


Understanding LLMs: A Comprehensive Overview from Training to Inference

Y Liu, H He, T Han, X Zhang, M Liu, J Tian, Y Zhang… - arXiv preprint arXiv …, 2024

… from the CommonCrawl. However, due to the presence of a substantial amount of low-quality data in web archives, preprocessing is essential when working with CommonCrawl data. Currently, four commonly used filtered datasets based on …


Understanding New Machine Learning Architectures: Practical Generative Artificial Intelligence for Anesthesiologists

Recent advances in neural networks have given rise to generative artificial intelligence, systems able to produce fluent responses to natural questions or attractive and even photorealistic images from text prompts. These systems were …


Understanding Privacy Risks of Embeddings Induced by Large Language Models

Z Zhu, N Shao, D Lian, C Wu, Z Liu, Y Yang, E Chen - arXiv preprint arXiv …, 2024

… CC-News (CommonCrawl News dataset) is a dataset containing news articles from international news websites. It contains 708241 English-language news articles published between January 2017 and December 2019. Pile-PubMed is a …


Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection

HT Lau, A Zubiaga - arXiv preprint arXiv:2411.03806, 2024

Natural Language Generation has been rapidly developing with the advent of large language models (LLMs). While their usage has sparked significant attention from the general public, it is important for readers to be aware when a piece of text is LLM-generated …


Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

M Ali, S Panda, Q Shen, M Wick, A Kobren - arXiv preprint arXiv:2407.21058, 2024

… We experiment with English Wikipedia and the CC-100 English subset of Common Crawl. For each model size and type of training data, we compute biases in the representations upstream and performance disparities downstream …


UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages

B Melesse Tessema, A Kedia, TS Chung - arXiv e-prints, 2024

BM Tessema, A Kedia, TS Chung - arXiv preprint arXiv:2411.14343, 2024

… In this study, we extract all the available textual data from every archive within the common crawl for a specific low resource language. Our choice of the common crawl dataset is driven by our objective to maximize the acquisition of the available …


UNIVERSAL IMITATION GAMES

In 1950, Alan Turing proposed a framework called an imitation game in which the participants are to be classified Human or Machine solely from natural language interactions. Using mathematics largely developed since Turing–category theory–we …


University Students' Attitudes and Perceptions on Learning English with ChatGPT

M Atakuziev, AM Gultom - Formosa Journal of Sustainable Research, 2024

This study examines 43 master's students' perceptions of ChatGPT in English language learning at Universitas Negeri Yogyakarta. Using a descriptive quantitative methodology, data was gathered via questionnaires and interviews …


Unlock the Power of Frozen LLMs in Knowledge Graph Completion

B Xue, Y Xu, Y Song, Y Pang, Y Ren, J Ding, L Fu… - arXiv preprint arXiv …, 2024

Classical knowledge graph completion (KGC) methods rely solely on structural information, struggling with the inherent sparsity of knowledge graphs (KGs). Large Language Models (LLMs) learn extensive knowledge from large corpora with …


Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

H Laurençon, L Tronchon, V Sanh - arXiv preprint arXiv:2403.09029, 2024

… available online (and often captured in web crawls like Common Crawl) presents a tempting resource for generating pairs of screenshots and corresponding HTML codes by simply rendering the HTML and capturing the output. However, this …


Unlocking the Marketing and Sales Potential: Bridging Generative AI with Domain-Specific Data using Retrieval Augmented Generation Architecture

V Lac, D Leschik - Generative Künstliche Intelligenz in Marketing und …, 2024

The landscape of artificial intelligence (AI) has undergone a paradigm shift due to the transformative impact of Large Language Models (LLMs), facilitated by their capacity to generate text closely simulating human language. Amidst the notable …


Unlocking the Potential of Generative Artificial Intelligence: A Case Study in Software Development

To unlock the potential of GenAI, software developers are facing a key tension. On the one hand, GenAI, as a general-purpose technology, offers vast possibilities for software development. On the other hand, developing software in enterprises …


UNLOCKING THE POWER OF LANGUAGE: IMPROVING MULTILINGUAL CAPABILITIES IN GENERATIVE AI FOR GLOBAL ACCESSIBILITY

Generative AI has emerged as a transformative technology with the potential to revolutionize global communication and accessibility. However, the current state of language support in generative AI applications presents both opportunities and …


Unmasking Memorization: Assessing Dutch Language Memorization in mT5 Models

This study investigates the memorization of Dutch language content in mT5 models, a multilingual variant of the T5 Transformer-based models. A fill-mask evaluation technique is used to assess memorization and how it varies across different model …


Unraveling the Mystery of Scaling Laws: Part I

H Su, Z Tian, X Shen, X Cai - arXiv preprint arXiv:2403.06563, 2024

Scaling law principles indicate a power-law correlation between loss and variables such as model size, dataset size, and computational resources utilized during training. These principles play a vital role in optimizing various aspects of model pre-training …


Unseen but influential associates: Properties of words' associates influence lexical and semantic processing

EJ Muraki, PM Pexman - Psychonomic Bulletin & Review, 2024

In many models of lexical and semantic processing, it is assumed that single word processing is a function of the characteristics of the words presented and the distributional properties of the words’ networks. Recent research suggests that …


Unsupervised approach for misinformation detection in Russia-Ukraine war news

N Khairova, A Galassi, FL Scudo, B Ivasiuk, I Redozub - 2024

The Russian-Ukrainian war has attracted considerable global attention; however, fake news often obstructs the formation of public opinion and disseminates false information. To address this issue, we have curated the RUWA dataset, comprising …


Unsupervised LLM Adaptation for Question Answering

K Saito, K Sohn, CY Lee, Y Ushiku - arXiv preprint arXiv:2402.12170, 2024

… Commoncrawl 2. The topics cover politics, sports, business, etc. To reduce the number of tokens, we ask ChatGPT to generate summaries. … We employ NewsFetch7 to crawl the news from Commoncrawl. The published date of the article …


Unsupervised Outlier Detection for Language-Independent Text Quality Filtering

J Daðason, H Loftsson - Proceedings of the 3rd Annual Meeting of the Special …, 2024

… Common Crawl (CC) is an organization that maintains a massive repository of data crawled … corpora, all of which are derived from Common Crawl. It consists of 6.3 trillion tokens in 167 … crawled sources, such as Common Crawl. The total size …


Unsupervised Video Moment Retrieval with Knowledge-based Pseudo Supervision Construction

G Wang, X Wu, X Tu, Z Liu, J Yan - ACM Transactions on Information Systems

Video moment retrieval locates a specified moment by a sentence query. Recent approaches have made remarkable advancements with large-scale video-sentence annotations. These annotations require extensive human labor and expertise …


Untangling the Unrestricted Web: Automatic Identification of Multilingual Registers

E Henriksson, A Myntti, A Eskelinen… - arXiv preprint arXiv …, 2024

This article explores deep learning models for the automatic identification of registers - text varieties such as news reports and discussion forums - in web-based datasets across 16 languages. Web register (or genre) identification would provide a …


Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

J Tian, D Zheng, Y Cheng, R Wang, C Zhang, D Zhang - arXiv preprint arXiv …, 2024

… (2024), who emphasize the influence of data quality and diversity in training models, our curated dataset incorporates sources such as Common Crawl, books, Wikipedia, code, and academic papers. Additionally, our dataset is multilingual, with …


Unveiling Al's Evolution and Algorithms

JK Jayabarathan - Responsible Implementations of Generative AI for …, 2024

… LLMs are trained on massive datasets such as Common Crawl and the Pile. The terabytes of data comprising these datasets were extracted from billions of publicly accessible websites. On the other hand, Small language models are trained on …


Unveiling Gender Bias in Large Language Models: Using Teacher's Evaluation in Higher Education As an Example

Y Huang - arXiv preprint arXiv:2409.09652, 2024

This paper investigates gender bias in Large Language Model (LLM)-generated teacher evaluations in higher education setting, focusing on evaluations produced by GPT-4 across six academic subjects. By applying a comprehensive analytical …


UNVEILING THE COGNITIVE CAPACITY OF CHATGPT: ASSESSING ITS HUMAN-LIKE REASONING ABILITIES

… OpenAI utilizes a dataset termed Common Crawl, which is a collection of publicly available websites. This dataset includes billions of web pages and is one of the largest text datasets available. Meanwhile, this dataset is just a starting line. …


USA: Copyright Management Information and Training Material for Generative AI

M Lejeune - Computer Law Review International, 2024

… Plaintiffs allege that “thousands” of their copyrighted works of journalism were caught in this “scrape,” stripped of their author, title, and copyright information, and input into at least three of OpenAI’s training sets (WebText, WebText2, and Common …


USDC: A Dataset of $\underline {U} $ ser $\underline {S} $ tance and $\underline {D} $ ogmatism in Long $\underline {C} $ onversations

M Marreddy, SR Oota, VC Chinni, M Gupta, L Flek - arXiv preprint arXiv:2406.16833, 2024

Identifying user's opinions and stances in long conversation threads on various topics can be extremely critical for enhanced personalization, market research, political campaigns, customer service, conflict resolution, targeted advertising, and …


Use of Large Language Models to Aid Analysis of Textual Data

LR Bentley, X Xia, JM Sitt, SC Fankhauser…

… The use of the data generated by the Common Crawl discussed earlier in this paper mitigates this limitation, drawing from the vast and growing body of content generated through the internet. The quality of the input data also plays a vital role in …


User Story Classification with Machine Learning and LLMs

P Chuor, A Ittoo, S Heng - … Conference on Knowledge Science, Engineering and …, 2024

We address the problem of classifying Capability, Task, Hard-goal, and Soft-goal in user stories. Such a classification is essential for generating Rationale Tree. Several articles have attempted to classify different aspects of user stories in the past …


Using Generative AI in Research

The slides accompany a workshop that is intended for graduate students to learn more about generative AI in the context of the research lifecycle. This work is licensed under a Creative Commons license so that others may share and adapt the …


Using Images to Find Context-Independent Word Representations in Vector Space

H Kumar - arXiv preprint arXiv:2412.03592, 2024

… • We use the CommonCrawl dump image links and DuckDuckGo’s image search API to search for openly available images for terms in the final vocabulary. The image set for each term consists of the images of the given term and images for the …


Using Large Language Models to Probe Cognitive Constructs, Augment Data, and Design Instructional Materials

F Kieser, P Wulff - … in Educational Sciences: Approaches, Applications and …, 2024

… It was shown that transformer-based large language models (LLMs could be pretrained on large language data sets such as the Common Crawl of the Internet, the Books Corpus, or Wikipedia, and then further fine-tuned in specific tasks [9, 10] …


Using Large Language Models to Understand Leadership Perception and Expectation

Y Zhang, X Wang, Z Zhang, X Wang, X Ma, Y Wu… - 2024 IEEE International …, 2024

Using multimodal approaches for measuring propositions in natural language (the Fill-Mask Association Test, FMAT) and ChatGPT 4.0, based on the Five-Factor Model of personality traits, we examined how leaders are perceived and expected in …


Using Large Language Models to Understand Telecom Standards

A Karapantelakis, M Shakur, A Nikou, F Moradi… - arXiv preprint arXiv …, 2024

… They have billions of parameters and are trained on corpora such as Common Crawl [5]. For foundation models to adapt to application domains, such as telecommunications, prompt engineering and fine-tuning approaches are used …


Using LLMs for the Extraction and Normalization of Product Attribute Values

A Brinkmann, N Baumann, C Bizer - … Conference on Advances in Databases and …, 2024

Product offers on e-commerce websites often consist of a product title and a textual product description. In order to enable features such as faceted product search or to generate product comparison tables, it is necessary to extract structured attribute-value …


Using LLMs for the Extraction

ABDN Baumann, C Bizerb - Advances in Databases and Information Systems: 28th …


Using rhetorical strategies to design prompts: a human-in-the-loop approach to make AI useful

N Ranade, M Saravia, A Johri - AI & SOCIETY, 2024

… It consumed the text content of the popular CommonCrawl dataset of web pages from 2016 to 2019. Although OpenAI curated it to eliminate redundancy and improve quality, it is nominally 45-terabytes worth of compressed text data. …


Utilizing Large Language Models for Non-trained Binary Sentiment Classification

Prompt Return a VALID JSON object with key “sentiment”. Return sentiment must be “positive” or “negative”. Do not say anything other than JSON object result. There is no “neutral”,“mixed” answer. Answer must start with ‘1’. Classify the sentiment of the …


Utilizing Large Language Models in Ophthalmology: The Current Landscape and Challenges

P Chotcomwongse, P Ruamviboonsuk, A Grzybowski - Ophthalmology and Therapy, 2024

A large language model (LLM) is an artificial intelligence (AI) model that uses natural language processing (NLP) to understand, interpret, and generate human-like language responses from unstructured text input. Its real-time response capabilities …


Uzbek news corpus for named entity recognition

A Yusufu, K Aziz, A Yusufu, A Ainiwaer, F Li, D Ji - Language Resources and …, 2024

We have presented a corpus of Uzbek news articles containing manually annotated named entities. The corpus comprises 500 articles (222,536 tokens) and three entity classes (person, location, organization) sourced from Qalampir, an online news …


UzRoberta: A pre-trained language model for Uzbek

R Davronov, F Adilova - AIP Conference Proceedings, 2024

… Common Crawl datasets, was generated distributed word representations for 157 different languages [8]. Author’s Uzbek Wikipedia model has 110K words, compared to -830K words in the Common Crawl … It is pre-trained on 2.5TB of filtered …


UZROBERTA: AN UZBEK LANGUAGE PRE-TRAINED MODEL

F Adilova, R Davronov, R Safarov - Universum: технические науки, 2023

… Distributed word representations for 157 distinct languages were created using the Wikipedia and Common Crawl databases [8]. The author's Uzbek Wikipedia model has 110K words, but the Common Crawl model only has 830K words. They …


V-GlórIA-Customizing Large Vision and Language Models to European Portuguese

A Simplício, D Semedo, J Magalhães - Proceedings of the 1st Workshop on …, 2024

Generative Vision and Language models have obtained remarkable results recently, thanks to the use of robust pre-trained Visual encoders and Large Language Models (LLMs), together with efficient model adaptation training strategies, requiring …


Validating and Exploring Large Geographic Corpora

J Dunn - arXiv preprint arXiv:2403.08198, 2024

… Beginning with a 427 billion word corpus derived from the Common Crawl, three methods are used to improve the quality of sub-corpora representing specific language-country pairs … This paper starts with an existing web corpus derived …


Validating pretrained language models for content quality classification with semantic-preserving metamorphic relations

PYP Chan, J Keung - Natural Language Processing Journal, 2024

Context: Utilizing pretrained language models (PLMs) has become common practice in maintaining the content quality of question-answering (Q&A) websites. However, evaluating the effectiveness of PLMs poses a challenge as they tend to provide local …


Value Residual Learning For Alleviating Attention Concentration In Transformers

Z Zhou, T Wu, Z Jiang, Z Lan - arXiv preprint arXiv:2410.17897, 2024

Transformers can capture long-range dependencies using self-attention, allowing tokens to attend to all others directly. However, stacking multiple attention layers leads to attention concentration. One natural way to address this issue is to use cross-layer …


VERB: Visualizing and Interpreting Bias Mitigation Techniques Geometrically for Word Representations

YAN ZHENG, CCM YEH, J WANG, WEI ZHANG - 2024

Complicated and massive datasets are becoming more and more commonly represented as vectorized embeddings. They are part of a de facto model for words in word vector embeddings such as Word2Vec [51] and GloVe [54], and are …


Verifying Claims About Metaphors with Large-Scale Automatic Metaphor Identification

K Aono, R Sasano, K Takeda - arXiv preprint arXiv:2404.01029, 2024

… Third, in this study, the examples were collected from CommonCrawl, which is a corpus of texts on the Web, and thus contains mostly written language. It is unknown whether similar results can be obtained when analyzing examples collected from a …


VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation

Y Song, Y Kim, M Iyyer - arXiv preprint arXiv:2406.19276, 2024

Existing metrics for evaluating the factuality of long-form text, such as FACTSCORE (Min et al., 2023) and SAFE (Wei et al., 2024), decompose an input text into "atomic claims" and verify each against a knowledge base like Wikipedia. These metrics are not …


VetMedGPT: Generative Pre-trained Transformer for Veteri-nary Medicine Healthcare

P Lin, SR Hussain, T Kadari, V Biyyala, S Bennur…

… This open dataset is sourced from 84 CommonCrawl snapshots and refined through the CCNet pipeline, presenting a vast pool of over 100 billion text documents. Notably, it includes a corpus of 30 billion documents that are enhanced with quality …


Viewpoints Detection in Political Speeches

Politics has significant impacts on our daily lives. It influences laws, policies, and resources allocation. The Internet and social media confront readers with an overwhelming amount of political texts. Readers struggle to analyze the information …


ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese

TT Hoa, TQ Duy, KQ Tran, K Van Nguyen - arXiv preprint arXiv:2412.15308, 2024

The rapid spread of information in the digital age highlights the critical need for effective fact-checking tools, particularly for languages with limited resources, such as Vietnamese. In response to this challenge, we introduce ViFactCheck, the first …


ViHateT5: Enhancing Hate Speech Detection in Vietnamese With A Unified Text-to-Text Transformer Model

LT Nguyen - arXiv preprint arXiv:2405.14141, 2024

… Leveraging a Transformer architecture and trained on a massive dataset of 2.5TB filtered CommonCrawl text across 100 languages, it employs a masked language modeling (MLM) objective to learn crosslingual representations. This approach …


ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

TH Nguyen, AC Le, VC Nguyen - arXiv preprint arXiv:2404.11086, 2024

The rapid advancement of large language models (LLMs) necessitates the development of new benchmarks to accurately assess their capabilities. To address this need for Vietnamese, this work aims to introduce ViLLM-Eval, the …


Virtual reality in automotive digital twins: a large language model-assisted content analysis of an expert evaluation

KSD Ka, IV Bosman, D Smit - Proceedings of the 27th International Academic …, 2024

Digital twins are gaining significant traction within the automotive engineering industry. Within this environment of data interchange that may reach very high levels of complexity, virtual reality can ease understanding and provide intuitive interaction …


VISION2UI: A Real-World Dataset with Layout for Code Generation from UI Designs

Y Gui, Z Li, Y Wan, Y Shi, H Zhang, Y Su, S Dong… - arXiv preprint arXiv …, 2024

… we opt to create our dataset on top of the Common Crawl3 dataset. The Common Crawl dataset is a vast collection of Web page data from … This partial dataset from the original Common Crawl dataset includes 3.35 billion pages, amounting to a total …


Visual Artists, Technological Shock, and Generative AI

The impact of generative AI (GenAI) programs on visual art is comparable to earlier historical moments of technological shock, when literary and visual artists grappled with unprecedented reproductive tools such as the printing press, photography, and …


Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey

J Huang, J Zhang, K Jiang, H Qiu, S Lu - arXiv preprint arXiv:2312.16602, 2023

Traditional computer vision generally solves each single task independently by a dedicated model with the task instruction implicitly designed in the model architecture, arising two limitations: (1) it leads to task-specific models, which require …


VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding

PNT Do, SQ Tran, PG Hoang, K Van Nguyen… - arXiv preprint arXiv …, 2024

… 250002 multilingual CommonCrawl XLM-Robertalarge … This model was trained on a Transformers-based masked language task using two terabytes of CommonCrawl data across more than a hundred languages. The model has two …


Vocabulary Expansion of Chat Models with Unlabeled Target Language Data

A Yamaguchi, T Morishita, A Villavicencio, N Aletras - arXiv preprint arXiv:2412.11704, 2024

Chat models (ie language models trained to follow instructions through conversation with humans) outperform base models (ie trained solely on unlabeled data) in both conversation and general task-solving abilities. These models are generally English-centric …


Volume 2024 Fall Issue 2

U Benoliel, C Engel, RH McAdams, AHC Hung

Disclosure laws are a fundamental regulatory mechanism in the United States, mandating firms to provide comprehensive and intricate data that is of significant value to the public. Unfortunately, this data is often complex and voluminous, making …


W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering

J Nian, Z Peng, Q Wang, Y Fang - arXiv preprint arXiv:2408.08444, 2024

… The corpus may comprise chuncked Wikipedia articles or from online sources such as Reddit and Common Crawl, while the question-answer pairs could be sourced from OpenQA platforms such as Quora and StackOverflow. The simple first-stage …


WaCadie: Towards an Acadian French Corpus

J Robichaud, P Cook - Proceedings of the 2024 Joint International Conference …, 2024

… We treat each post as one document, similar to one website’s HTML from the Common Crawl corpus. We refer to this corpus as r/acadie. … We used CommonCrawl’s extensive database and scraped New Brunswick (.nb.ca) websites …


WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset

J Qiu, H Lv, Z Jin, R Wang, W Ning, J Yu, CB Zhang… - arXiv preprint arXiv …, 2024

… Moreover, Common Crawl contains a large amount of unsafe content, such as toxic and pornographic … Common Crawl data, which includes data extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality …


WaterPark: A Robustness Assessment of Language Model Watermarking

J Liang, Z Wang, L Hong, S Ji, T Wang - arXiv preprint arXiv:2411.13425, 2024

… Specifically, WATERPARK integrates a variety of LLMs including OPT-1.3B [74], Llama2-7B [58], Mistral-7B [22], and Gemma-2B/7B [15]; it also includes a variety of datasets including: C4 is a cleaned version of Common Crawl’s web crawl corpus …


WCC-EC 2.0: Enhancing Neural Machine Translation with a 1.6 M+ Web-Crawled English-Chinese Parallel Corpus

J Zhang, K Su, Y Tian, T Matsumoto - Electronics, 2024

This research introduces WCC-EC 2.0 (Web-Crawled Corpus—English and Chinese), a comprehensive parallel corpus designed for enhancing Neural Machine Translation (NMT), featuring over 1.6 million English-Chinese sentence pairs …


We have no idea what we are walking into: AI and ethical considerations

KB Forrest - Annals of the New York Academy of Sciences, 2024

… called “Common Crawl” provide data, “C4” is a curated (de-duplicated and some nonsensical junk eliminated) set of the Common Crawl data, then there is a data set well named “The Pile”, and others that are compilations of books, music …


Weakly supervised learning for an effective focused web crawler

PRJ Dhanith, K Saeed, G Rohith, SP Raja - Engineering Applications of Artificial …, 2024

Focused crawler traverses the Web to only collect pages that are relevant to a particular topic, and is increasingly considered as a way to get around the scalability issues with current general-purpose search engines. But the data diversity in the …


WEATHER-5K: A Large-scale Global Station Weather Dataset Towards Comprehensive Time-series Forecasting Benchmark

T Han, S Guo, Z Chen, W Xu, L Bai - arXiv preprint arXiv:2406.14399, 2024

Global Station Weather Forecasting (GSWF) is crucial for various sectors, including aviation, agriculture, energy, and disaster preparedness. Recent advancements in deep learning have significantly improved the accuracy of weather predictions by …


Web Archives Metadata Generation with GPT-4o: Challenges and Insights

A Nair, ZR Goh, T Liu, AYP Huang - Information Technology and Libraries, 2025

AY Huang, A Nair, ZR Goh, T Liu - arXiv preprint arXiv:2411.05409, 2024

Current metadata creation for web archives is time consuming and costly due to reliance on human effort. This paper explores the use of gpt-4o for metadata generation within the Web Archive Singapore, focusing on scalability, efficiency, and …


Web Crawl Refusals: Insights From Common Crawl

… Abstract: This study investigates server-side blocks encountered by Common Crawl, a major … that approximately 1.68% of websites in a Common Crawl snapshot exhibit some form of explicit … of these blocks and the effectiveness of …


Web Data Commons-RDFa, Microdata, Embedded JSON-LD, and Microformats Data Sets-October 2023

The Web Data Commons RDFa, Microdata and Microformats data sets has been extracted from the September/October 2023 release of the Common Crawl. In summary, we found structured data within 1.7 billion HTML pages out of the 3.4 …


Web Data Commons-Schema. org Table Corpus 2023

The WDC schema.org Table Corpus consists of ~5 million relational tables which were generated by extracting schema.org data from the Common Crawl and grouping the data into seperate tables for each class/host combination, eg all …


Web scraping and Generative Models training in the Directive 790/19

… The companies that created GM never disclosed the exact corpus of training data they employed to train their models; however, it has become clear sources that many copyrighted materials were employed as part of larger datasets such as the …


Web scraping: Jurisprudence and legal doctrines

AG Fontana - The Journal of World Intellectual Property

… In the case of ChatGPT, for example, during the pretraining phase large quantities of textual data are used, extracted from archives like Common Crawl, Wikipedia or Google Books but also from platforms like Reddit. Without web scraping no AI or …


WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations

H Deng, C Wang, X Li, D Yuan, J Zhan, T Zhou, J Ma… - arXiv preprint arXiv …, 2024

Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still …


WebGraph: The Next Generation (Is in Rust)

T Fontana, S Vigna, S Zacchiroli - ACM Web Conference 2024, 2024

We report the results of a yearlong effort to port the WebGraph framework from Java to Rust. For two decades WebGraph has been instrumental in the analysis and distribution of large graphs for the research community of TheWebConf, but the …


What are Large Language Models Made of?

C Stinson - Leading the Way: Envisioning the Future of Higher …, 2024

… CommonCrawl is the lowest quality but largest of these datasets, consisting of text scraped from all over the web. Their data from the years 2016 to 2019 were used to train GPT-3. OpenAI’s quality control measure for filtering CommonCrawl was to …


What can be learned about color from

… Word embedding models trained on COCA-fiction performed better than models trained on much larger corpora: COCA-fiction (at 120 million tokens) outperformed OpenSubtitles (750 million tokens) and Common Crawl ( 600 billion tokens) though …


What do BERT word embeddings learn about the French language?

E Goliakova, D Langlois - Sixth International Conference, 2024

Pre-trained word embeddings (for example, BERT-like) have been successfully used in a variety of downstream tasks. However, do all embeddings, obtained from the models of the same architecture, encode information in the same way? Does the …


What Does “Palliative” Mean? Sentiment, Knowledge, and Public Perception Concerning Palliative Care on the Internet since the COVID-19 Pandemic

J Peters, M Heckel, E Breindl, C Ostgathe - Palliative Medicine Reports, 2024

… data from the CommonCrawl corpus, 25 consisting of English-language websites with the ccTLD-domains .com, .org and .uk. Twenty-three, 219 texts with relevant thematic focus remained in the corpus. (3) We created a large X corpus because the …


What is the authentic internet register before & after the Russian invasion in Ukraine? Polish and Czech YouTube comments from 2021–2023

A Tikhonov - Lodz Papers in Pragmatics, 2024

Over one million tokens of comments were collected for the study using data mining methods. The videos under which the comments were dug out were not chosen arbitrarily but according to the current official national YouTube trends in Poland and …


What's in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT

DM Kaplan, R Palitsky, SJ Arconada Alvarez… - Journal of Medical Internet …, 2024

… In the case of GPT-3.5 (the current version at the time of this study), the model contains 175 billion parameters and was trained with commonly available text training sets that represent a broad swath of data on the internet (Common Crawl) …


What's New in My Data? Novelty Exploration via Contrastive Generation

M Isonuma, I Titov - arXiv preprint arXiv:2410.14765, 2024

Fine-tuning is widely used to adapt language models for specific goals, often leveraging real-world data such as patient records, customer-service interactions, or web content in languages not covered in pre-training. These datasets are typically …


When Context Leads but Parametric Memory Follows in Large Language Models

Y Tao, A Hiatt, E Haake, AJ Jetter, A Agrawal - arXiv preprint arXiv:2409.08435, 2024

Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when …


When Every Token Counts: Optimal Segmentation for Low-Resource Language Models

G Suri, V Dewangan, R Sonavane - arXiv preprint arXiv:2412.06926, 2024

V Dewangan, G Suri, R Sonavane - Proceedings of the First Workshop on …, 2025

… The CC100 dataset consists of monolingual data of 116 languages extracted from the January-December 2018 Commoncrawl snapshots. We benchmark on the English language using the Wikipedia corpus readily accessible on Kaggle Datasets2 …


When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

H Wang, Q Liu, C Du, T Zhu, C Du, K Kawaguchi… - arXiv preprint arXiv …, 2024

… This suggests that CommonCrawl sequences in our dataset are not only more numerous but also longer, contributing significantly to the … Overall, our up-sampled data mixture places greater emphasis on longer sequences from CommonCrawl …


When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

J Wang, C Huang, S Yan, W Xie, D He - International Journal of Human–Computer …, 2024

As a novel AI-powered conversational system, large language models (LLMs) have the potential to be used in various applications. Recent advances in LLMs like ChatGPT have made LLM-based academic tools possible. However, most of the …


Where does In-context Learning\\Happen in Large Language Models?

S Sia, D Mueller, K Duh - The Thirty-eighth Annual Conference on Neural …

Self-supervised large language models have demonstrated the ability to perform various tasks via in-context learning, but little is known about where the model locates the task with respect to prompt instructions and demonstration examples. In …


Where does In-context Translation Happen in Large Language Models

S Sia, D Mueller, K Duh - arXiv preprint arXiv:2403.04510, 2024

Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration …


Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance?

F Uchiyama, T Kojima, A Gambardella, Q Cao… - arXiv preprint arXiv …, 2024

Recent large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and logical reasoning tasks. Prior research indicates that LLMs pre-trained with programming language data exhibit high …


Which Word Embeddings for Modeling Web Search Queries? Application to the Study of Search Strategies

In order to represent the global strategies deployed by a user during an information retrieval session on the Web, we compare different pretrained vector models capable of representing the queries submitted to a search engine. More precisely …


Who Gets Paid (for) What? The Cultural Political Economy of News Content in Generative AI

One of the key controversies that generative artificial intelligence (AI) has recently stirred was whether compensation is due for the copyrighted materials used to train AI models. This article explores the logic, trajectories, and dynamics of content …


Who's Afraid of ChatGPT?: Large Language Models and Conspiracies

… GPT-3 added Common Crawl to the training datasets, 570GB of text obtained after filtering from 45TB of plaintext. The filtering was done by comparing new documents to documents in WebText, with the latter acting as a proxy for high-quality …


Why 'open'AI systems are actually closed, and why this matters

DG Widder, M Whittaker, SM West - Nature, 2024

This paper examines ‘open’ artificial intelligence (AI). Claims about ‘open’ AI often lack precision, frequently eliding scrutiny of substantial industry concentration in large-scale AI development and deployment, and often incorrectly applying …


WikiDBs: A Large-Scale Corpus Of Relational Databases From Wikidata

L Vogel, JM Bodensohn, C Binnig - The Thirty-eight Conference on Neural Information …

Deep learning on tabular data, and particularly tabular representation learning, has recently gained growing interest. However, representation learning for relational databases with multiple tables is still an underexplored area, which may be …


Willkommens-Merkel, Chaos-Johnson, and Tore-Klose: Modeling the Evaluative Meaning of German Personal Name Compounds

We present a comprehensive computational study of the under-investigated phenomenon of personal name compounds (PNCs) in German such as Willkommens-Merkel (‘Welcome-Merkel’). Prevalent in news, social media, and …


WITH LLaMA-3?

… Although Datacomp is derived from Common Crawl and represents a snapshot of data in the public internet, it inevitably includes noisy and potentially unsafe content. The dataset’s curation involves rigorous filtering, including NSFW filters on images …


Worldwide Federated Training of Language Models

A Iacob, L Sani, B Marino, P Aleksandrov, ND Lane - arXiv preprint arXiv:2405.14446, 2024

… To simulate geographically distributed systems, we use a subset of Multilingual Colossal Common Crawl (mC4)[81], covering high and low-… For IID experiments, we use the standard Cleaned Colossal Common Crawl (C4) English dataset …


WRITTEN IN PERSIAN USING A TRANSFORMER-BASED MODEL

T Firoozi, MJ Gierl - The Routledge International Handbook of Automated …, 2024

… This model is created using contents from Common Crawl and Wikipedia. It is trained using CBOW with position weights, in dimension 300, with character n-grams of length 5, and a window of size 5 and 10. The model gives 300-dimensional vector …


X-Phishing-Writer: A Framework for Cross-Lingual Phishing Email Generation

SW Guo, YC Fan - ACM Transactions on Asian and Low-Resource …, 2024

… The mBART model employs various language corpora from the Common Crawl dataset and undergoes multilingual pretraining based on the BART architecture. The training data is a concatenation of data from � diferent languages, denoted as � = �1,�2...�� …


XLM for Autonomous Driving Systems: A Comprehensive Review

S Fourati, W Jaafar, N Baccar, S Alfattani - arXiv preprint arXiv:2409.10484, 2024

Large Language Models (LLMs) have showcased remarkable proficiency in various information-processing tasks. These tasks span from extracting data and summarizing literature to generating content, predictive modeling, decision-making …


XLMR4MD: New Vietnamese dataset and framework for detecting the consistency of description and permission in Android applications using large language models

QN Nguyen, NT Cam, K Van Nguyen - Computers & Security, 2024

… The most significant update that XLMRoBERTa provides compared to the original is that the amount of training data has increased significantly, the cleaned CommonCrawl data it was trained on takes up a considerable memory block of up to …


XLORE 3: A Large-scale Multilingual Knowledge Graph from Heterogeneous Wiki Knowledge Resources

K Zeng, H Jin, X Lv, F Zhu, L Hou, Y Zhang, F Pang… - ACM Transactions on …, 2024

In recent years, Knowledge Graph (KG) has attracted significant attention from academia and industry, resulting in the development of numerous technologies for KG construction, completion, and application. XLORE is one of the largest …


Y-MAP-Net: Real-time depth, normals, segmentation, multi-label captioning and 2D human pose in RGB images

A Qammaz, N Vasilikopoulos, I Oikonomidis… - arXiv preprint arXiv …, 2024

We present Y-MAP-Net, a Y-shaped neural network architecture designed for real-time multi-task learning on RGB images. Y-MAP-Net, simultaneously predicts depth, surface normals, human pose, semantic segmentation and generates multi-label …


Yankari: A Monolingual Yoruba Dataset

M Akpobi - arXiv preprint arXiv:2412.03334, 2024

This paper presents Yankari, a large-scale monolingual dataset for the Yoruba language, aimed at addressing the critical gap in Natural Language Processing (NLP) resources for this important West African language. Despite being spoken by over …


Yes, This Is A Puff Piece? A Comparative Analysis of the Vendor Defences of Puffery, Statements of Future Intent and Disclaimers–Part 2—How far does the …

J Beardwood - Computer Law Review International, 2024

… Plaintiffs allege that “thousands” of their copyrighted works of journalism were caught in this “scrape,” stripped of their author, title, and copyright information, and input into at least three of OpenAI’s training sets (WebText, WebText2, and Common …


Yi-Lightning Technical Report

A Wake, A Wang, B Chen, CX Lv, C Li, C Huang, C Cai… - arXiv preprint arXiv …, 2024

This technical report presents Yi-Lightning, our latest flagship large language model (LLM). It achieves exceptional performance, ranking 6th overall on Chatbot Arena, with particularly strong results (2nd to 4th place) in specialized categories including …


Yi: Open Foundation Models by 01. AI

A Young, B Chen, C Li, C Huang, G Zhang, G Zhang… - arXiv preprint arXiv …, 2024

… We start with web documents from Common Crawl, use the CCNet pipeline [79] for language identification and perplexity scoring. Then … Notably, the Chinese content extracted from Common Crawl present unique challenges, particularly with a …


Yuan 2.0-M32: Mixture of Experts with Attention Router

S Wu, J Luo, X Chen, L Li, X Zhao, T Yu, C Wang… - arXiv preprint arXiv …, 2024

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention Router, is proposed and adopted for a more efficient selection of experts …


YuLan: An Open-source Large Language Model

Y Zhu, K Zhou, K Mao, W Chen, Y Sun, Z Chen, Q Cao… - arXiv preprint arXiv …, 2024

… In addition to these sources, we process raw data from Common Crawl (CC) dumps, particularly focusing on events that occurred between January 2021 and February 2023. To manage the practical challenges of HTML content extraction and …


Zero and Few Short Learning Using Large Language Models for De-Identification of Medical Records

YS Yashwanth, R Shettar - IEEE Access, 2024

… Among the word embeddings, CommonCrawl, a general English word embedding, demonstrated superior performance compared to the other four. The highest F-measure for CommonCrawl was 96.46, with corresponding precision and recall scores of …


Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data

N Popp, JH Metzen, M Hein - arXiv preprint arXiv:2404.16637, 2024

… The first one involves relying on large-scale data such as common crawl datasets [7,38]. While this approach is feasible for large foundation models, it poses challenges for smaller models as these lack the same level of generalization capabilities due to …


Zero-shot learning for multilingual discourse relation classification

E Metheniti, P Muller, C Braud, M Hernández-Casas - 5th Workshop on …, 2024

Classifying discourse relations is a hard task: discourse-annotated data is scarce, especially for languages other than English, and there exist different theoretical frameworks that affect textual spans to be linked and the label set used. Thus, work …


Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models

A Scius-Bertrand, M Jungo, L Vögtlin, JM Spat… - International Conference on …, 2025

… LLMs typically rely on unsupervised pre-training of decoder-only transformer architectures on a large body of texts from the internet, such as the Common Crawl dataset [17], followed by reinforcement learning from human feedback, to achieve …


Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon

F Koto, T Beck, Z Talat, I Gurevych, T Baldwin - arXiv preprint arXiv:2402.02113, 2024

Improving multilingual language models capabilities in low-resource languages is generally difficult due to the scarcity of large-scale data in those languages. In this paper, we relax the reliance on texts in low-resource languages by using …


Zero-Shot vs. Few-Shot Multi-Speaker TTS Using Pre-trained Czech SpeechT5 Model

J Lehečka, Z Hanzlíček, J Matoušek, D Tihelka - arXiv preprint arXiv:2407.17167, 2024

… Text Data As a source of unlabeled text data, we used the Common Crawl project4 which is a huge public web archive consisting of petabytes of crawled web pages. We used the language information provided in the index files to select Czech …


Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training

E Cunegatti, LL Custode, G Iacca - arXiv preprint arXiv:2411.07066, 2024

Network pruning is a set of computational techniques that aim to reduce a given model's computational cost by removing a subset of its parameters while having minimal impact on performance. Throughout the last decade, the most widely used …


ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment

E Obbad, I Mlauzi, B Miranda, R Schaeffer, K Obbad… - arXiv preprint arXiv …, 2024

Data selection is crucial for optimizing language model (LM) performance on specific tasks, yet most existing methods fail to effectively consider the target task distribution. Current approaches either ignore task-specific requirements entirely or …


Zipfian Whitening

S Yokoi, H Bao, H Kurita, H Shimodaira - arXiv preprint arXiv:2411.00680, 2024

The word embedding space in neural models is skewed, and correcting this can improve task performance. We point out that most approaches for modeling, correcting, and measuring the symmetry of an embedding space implicitly assume …


ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using ChatGPT

A Kabir, S Wang, Y Tian, M Asaduzzaman, W Zhang - arXiv preprint arXiv …, 2024

Technical question and answering (Q&A) sites such as Stack Overflow have become an important source for software developers to seek knowledge. However, code snippets on Q&A sites are usually uncompilable and semantically incomplete for …


Zyda-2: a 5 Trillion Token High-Quality Dataset

Y Tokpanov, P Glorioso, Q Anthony, B Millidge - arXiv preprint arXiv:2411.06068, 2024

… Given that all open datasets ultimately originated from similar Common Crawl scrapes, we expect them to contain significant fractions of duplicated documents. Deduplication has been found to generally improve language modeling performance (Lee et al.…


Zyda: A 1.3 T Dataset for Open Language Modeling

Y Tokpanov, B Millidge, P Glorioso, J Pilault, A Ibrahim… - arXiv preprint arXiv …, 2024

… Additionally, unlike these works which directly filter from Common Crawl, we aim to instead collect and collate existing open-source datasets under one banner, and to uniformly filter and deduplicate them against one another so as to create a dataset …