SEMANTIC MATCHING FOR DATA CATALOGS

Modern organizations have a lot of data, and more is coming. This data is often spread across the entire organization, appearing in various forms, such as CSV files, databases, PPTX slides, emails, etc. To organize and manage the various forms of …


Semantic-Aware Methods for the Analysis of Bias and Underrepresentation in Language Resources

… Within this research I did not analyze Common Crawl5, even if it is mentioned in some LMs papers. This is because it is not a proper … CC-News is a snapshot of news texts derived from the CommonCrawl initiative9) that has been exploited for …


Semantic-guided spatio-temporal attention for few-shot action recognition

J Wang, B Liu - Applied Intelligence, 2024

… We utilize GloVe [49] to extract text embeddings for action labels, which is pre-trained on Common Crawl dataset of 840B tokens. For labels with multiple words, we average the embeddings of all their words. Finally, one action label is represented …


Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts

R Lamsal, MR Read, S Karunasekera - arXiv preprint arXiv:2403.16614, 2024

Tasks such as semantic search and clustering on crisis-related social media texts enhance our comprehension of crisis discourse, aiding decision-making and targeted interventions. Pre-trained language models have advanced performance in …


Semi-Automated Classification of Adverse Events in Alzheimer's Disease Clinical Trials

Investigators conducting clinical trials have an ethical, scientific, and regulatory obligation to protect the safety of trial participants. Traditionally, safety monitoring includes manual review and coding of adverse event data by expert clinicians. Our …


Semi-Automated Coding for Conflict Mediation Research: Database Development

… In plain terms, our database employs a combination of Common Crawl and real-time APIs from major news websites to gather comprehensive event data. Common Crawl (2008) is a vast archive that regularly scans the internet, collecting articles …


Semiotics of Machinic Co-Enunciation

1 In this paper, we propose a semiotic study on generative artificial intelligences, considering the work of Midjourney and DALL· E. 1 The aim is to frame their functioning from a semiotic point of view and to describe the operations that can be …


SEN-CTD: semantic enhancement network with content-title discrepancy for fake news detection

J Fang, K Ma, Y Qiu, K Ji, Z Chen, B Yang - International Journal of Web Information …, 2024

Purpose The discrepancy between the content of an article and its title is a key characteristic of fake news. Current methods for detecting fake news often ignore the significant difference in length between the content and its title. In addition, relying …


SeNSe: embedding alignment via semantic anchors selection

L Malandri, F Mercorio, M Mezzanzanica, F Pallucchini - International Journal of Data …, 2024

Word embeddings have proven extremely useful across many NLP applications in recent years. Several key linguistic tasks, such as machine translation and transfer learning, require comparing distributed representations of words belonging to …


SENTA: Sentence Simplification System for Slovene

A Žagar, M Klemen, M Robnik-Šikonja, I Kosem - Proceedings of the 2024 Joint …, 2024

… It uses sentence-level paraphrase data obtained by mining paraphrases from Common Crawl in combination with controllable generation mechanisms to adjust the length and lexical complexity of a simplified candidate. …


Sentence Embeddings for Massively Multilingual Speech and Text Processing

… CommonCrawl raw text data corresponds to partial snapshots of the internet totalling terabytes of text data extracted from web pages in many languages. The authors used the open-source FAISS library (Johnson et al. 2019) in order to …


Sentence Level Analysis Model for Phishing Detection Using KNN

Phishing emails have experienced a rapid surge in cyber threats globally, especially following the emergence of the COVID-19 pandemic. This form of attack has led to substantial financial losses for numerous organizations. Although various models …


Sentiment Analysis and Rating Prediction for App Reviews Using Transformer-based Models

In this study, we present the sentiment analysis of Spotify app reviews, the implementation of Natural Language Processing (NLP) methods, and the use of transformer-based models including BERT, DistilBERT, RoBERTa, and XLM-RoBERTa …


Sentiment Analysis Based on RoBERTa for Amazon Review: An Empirical Study on Decision Making

X Guo - arXiv preprint arXiv:2411.00796, 2024

In this study, we leverage state-of-the-art Natural Language Processing (NLP) techniques to perform sentiment analysis on Amazon product reviews. By employing transformer-based models, RoBERTa, we analyze a vast dataset to derive sentiment …


Sentiment Analysis of Low-Resource Yorùbá Tweets Using Fine-Tuned Bert Models

VT Odumuyiwa, OO Shoyemi, AE Fagoroye - University of Ibadan Journal of Science …, 2024

Sentiment analysis in low-resource languages poses a notable challenge because of the scarcity of labelled data and language-specific models. This study addresses this challenge of Yorùbá sentiment analysis using fine-tuned variants of Bidirectional …


Sentiment Analysis of Solar Energy in US Cities: A 10-Year Analysis Using Transformer-Based Deep Learning

This study examines US public sentiment toward solar energy from 2013 to 2022, analyzing 8 million social media posts using RoBERTa, a transformer-based deep learning algorithm. Sentiment has generally been positive but has declined since …


Sentiment analysis on the issue of COVID-19 vaccination using LSTM and fasttext

WN Dewani, Y Azhar, N Hayatin - AIP Conference Proceedings, 2024

… The fasttext model is the result of a word vector previously trained from Common Crawl and Indonesian language Wikipedia data from Facebook. This model is trained using the Continuous Bag of Words (CBOW) method with position weight [13] …


Sentiment Triplet Extraction with Multi-view Contrastive Learning

W Wu, D Wang, M Wang, S Feng, Y Zhang - IEEE Transactions on Affective …, 2024

… And Common Crawl refers to a dataset obtained from the Common Crawl project, which is a web archive that freely provides a large collection of web page data, here CC25, mC4 and C4 specify a specific version or subset of the Common Crawl data …


SENTIMENTAL REFLECTION OF GLOBAL CRISES: CZECH AND UKRAINIAN VIEWS ON POPULAR EVENTS THROUGH THE PRISM OF INTERNET …

Social media have become a part of our lives, and their use helps us learn about events and comment on them with certain emotions. The purpose of our study was to determine the most frequent tone (positive, negative, neutral) of comments on …


Sequence-Level Analysis of Leakage Risk of Training Data in Large Language Models

T Tiwari, GE Suh - arXiv preprint arXiv:2412.11302, 2024

… We perform this study on two pre-trained models, LLaMa and OPT, trained on the Common Crawl and Pile respectively. We discover that 1) Extraction rate, the predominant metric used in prior quantification work, underestimates the threat of …


Sequence-to-Sequence Architectures: Encoder-Decoders and Decoders

PM Nugues - … for Natural Language Processing: Programming with …, 2024

The transformer consisted originally of two parts: the encoder and the decoder. This chapter describes this complete structure and, to make it concrete, illustrates it with a machine-translation program in PyTorch. For sake of feasibility, we replace the word …


SETEM: Self-ensemble training with Pre-trained Language Models for Entity Matching

H Ding, C Dai, Y Wu, W Ma, H Zhou - Knowledge-Based Systems, 2024

… This benchmark is constructed by extracting schema.org data from the Common Crawl and using the following attributes: brand, title, currency and price. The WDC Products benchmark consists of overall nine training, nine validation and nine test …


SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments

SAG Shakhadri, K KR, R Aralimatti - arXiv preprint arXiv:2410.11331, 2024

We introduce Shakti, a 2.5 billion parameter language model specifically optimized for resource-constrained environments such as edge devices, including smartphones, wearables, and IoT systems. Shakti combines high-performance NLP …


Shapley Values-Powered Framework for Fair Reward Split in Content Produced by GenAI

A Glinsky, A Sokolsky - arXiv preprint arXiv:2403.09700, 2024

… DATACOMP introduces a new candidate pool of 12.8 billion image-text pairs sourced from Common Crawl, which is broader than the collection in LAION-5B. This extensive pool offers a more diverse base for dataset creation. Unlike LAION-5B …


ShieldGemma: Generative AI Content Moderation Based on Gemma

W Zeng, Y Liu, R Mullins, L Peran, J Fernandez… - arXiv preprint arXiv …, 2024

We present ShieldGemma, a comprehensive suite of LLM-based safety content moderation models built upon Gemma2. These models provide robust, state-of-the-art predictions of safety risks across key harm types (sexually explicit, dangerous …


Short-text topic modeling with dual reinforcement from internal and external semantics

J Wang, L Chen, Z Zhang, J He, X Zhou - International Journal of Machine Learning …, 2024

… We use a 300 dimensional Common Crawl version for all of the four datasets. For Oscars, we additionally use a 200 dimensional Twitter … Since we use two versions of GloVe word embeddings (ie, Common Crawl version and Twitter version) for the …


Should we stay silent on violence? An ensemble approach to detect violent incidents in Spanish social media texts

D Sharma, V Gupta, VK Singh, D Pinto - Natural Language Processing

… For training, more than two terabytes of filtered CommonCrawl data was used. The implementation of XLM-RoBERTa includes a preprocessing step, as described earlier. After preprocessing the data, the tweets are tokenized using the XLM-RoBERTa …


SIGN LANGUAGE RECOGNITION OF WORDS AND SENTENCE PREDICTION USING LSTM AND NLP

R GAIKWAD, L ADMUTHE - Journal of Theoretical and Applied Information …, 2024

… The dataset used for the training of the T5 model is the C4 dataset called as Colossal Clean Crawled Corpus which is collected from Common Crawl, a publicly available web archive. It consists of 750 GB clean English text scraped from the …


Silencing the Risk, Not the Whistle: A Semi-automated Text Sanitization Tool for Mitigating the Risk of Whistleblower Re-Identification

D Staufer, F Pallas, B Berendt - arXiv preprint arXiv:2405.01097, 2024

Whistleblowing is essential for ensuring transparency and accountability in both public and private sectors. However, (potential) whistleblowers often fear or face retaliation, even when reporting anonymously. The specific content of their …


SimLESS: A Secure Deduplication System over Similar Data in Cloud Media Sharing

M Song, Z Hua, Y Zheng, T Xiang, X Jia - IEEE Transactions on Information Forensics …, 2024

With the growing popularity of cloud computing, sharing media data through the cloud has become a common practice. Due to high information redundancy, media data take up a significant amount of storage space. Moreover, similar media data …


SimLLM: Calculating Semantic Similarity in Code Summaries using a Large Language Model-Based Approach

X Jin, Z Lin - Proceedings of the ACM on Software Engineering, 2024

Code summaries are pivotal in software engineering, serving to improve code readability, maintainability, and collaboration. While recent advancements in Large Language Models (LLMs) have opened new avenues for automatic code …


Simple and Scalable Strategies to Continually Pre-train Large Language Models

A Ibrahim, B Thérien, K Gupta, ML Richter, Q Anthony… - arXiv preprint arXiv …, 2024

… We then generate a fixed token-length response for each of the models trained German Common Crawl. As a baseline, we also evaluate … of German-language outputs from the models trained on German Common Crawl when compared to the …


Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulation

J Barnett, K Kieslich, N Diakopoulos - arXiv preprint arXiv:2405.09679, 2024

… They utilize proprietary datasets as well as opensource web data such as the continuously growing Common Crawl, which contains more than 250 billion pages of over a decade’s worth of textual content. While these models primarily learn how …


Simulating Subjects: The Promise and Peril of AI Stand-ins for Social Agents and Interactions

Large Language Models (LLMs), through their exposure to massive collections of online text, learn to reproduce the perspectives and linguistic styles of diverse social and cultural groups. This capability suggests a powerful social scientific application–the …


Single Cells Are Biological Tokens: Towards Cell Language Models

The rapid advancement of single-cell technologies allows for simultaneous measurement of multiple molecular features within individual cells, providing unprecedented multimodal data through single-cell multi-omics and spatial omics …


Single Layer Single Gradient Unlearning

Z Cai, Y Tan, MS Asif - arXiv preprint arXiv:2407.11867, 2024

… Large-scale web scrapers, such as Common Crawl, archive billions of webpages monthly, providing massive data sources to train such revolutionary AI models. While these large unsupervised datasets significantly enhance performance under …


Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model

H Hajimolahoseini, M Hassanpour, F Ataiefard, B Chen… - arXiv preprint arXiv …, 2024

This paper introduces a novel method of Progressive Low Rank Decomposition (PLRD) tailored for the compression of large language models. Our approach leverages a pre-trained model, which is then incrementally decompressed to smaller sizes using …


SiTSE: Sinhala Text Simplification Dataset and Evaluation

S Ranathunga, R Sirithunga, H Rathnayake, L De Silva… - arXiv preprint arXiv …, 2024

Text Simplification is a task that has been minimally explored for low-resource languages. Consequently, there are only a few manually curated datasets. In this paper, we present a human curated sentence-level text simplification dataset for the …


Sketch Grammar for Japanese

Sketch Grammar is a formalism used to extract frequent word collocations from text corpora. This thesis aims at analyzing the current sketch grammar for Japanese used in the Sketch Engine corpus manager and implementing a new version with …


SkillMatch: Evaluating Self-supervised Learning of Skill Relatedness

JJ Decorte, J Van Hautte, T Demeester, C Develder - arXiv preprint arXiv:2410.05006, 2024

… For the generic fastText vectors, we use cc.en.300.bin, a 300-dimensional vector model trained on Common Crawl and Wikipedia [4]. To train their domain-specific versions, we use a dataset of 755k job ads, also from 2024. Note that these are not included …


SlamaTrain–Representative Training Dataset for Slavonic Large Language Models

… However, most of the largest collections are based on Common Crawl [12] that, naturally reflecting the proportions of texts available on-line, are imbalanced in favor of English and … Patel, JM: Introduction to Common Crawl Datasets, pp. 277–324 …


Slight Corruption in Pre-training Data Makes Better Diffusion Models

H Chen, Y Han, D Misra, X Li, K Hu, D Zou… - arXiv preprint arXiv …, 2024

… For example, Stable Diffusion [26] was pre-trained on LAION-2B [17], which contains billion-scale image-text pairs collected from Common Crawl [27]. Despite the heavy filtering mechanisms used in collecting pre-training datasets [17, 28], they …


Slovak morphological tokenizer using the Byte-Pair Encoding algorithm

D Držík, F Forgac - PeerJ Computer Science, 2024

This study introduces a new approach to text tokenization, SlovaK Morphological Tokenizer (SKMT), which integrates the morphology of the Slovak language into the training process using the Byte-Pair Encoding (BPE) algorithm. Unlike conventional …


Small Language Models and Large Language Models in Oppositional thinking analysis: Capabilities and Biases and Challenges

Á Huertas-García, C Martí-González, J Muñoz… - Working Notes of CLEF, 2024

The proliferation of misinformation and conspiracy theories needs robust methods to differentiate legitimate critical discourse from harmful conspiratorial narratives. This study investigates discerning critical messages from conspiracy theories within …


Small Language Models for Emotion Recognition in Polish Stock Market Investor Opinions

B Koptyra, M Oleksy, E Dziecioł, J Kocon

In this paper, we explore the application of small language models for emotion recognition in Polish stock market investor opinions. Emotion recognition has been shown to enhance stock price prediction models by providing meaningful text …


Small Language Models: Survey, Measurements, and Insights

Z Lu, X Li, D Cai, R Yi, F Liu, X Zhang, ND Lane, M Xu - arXiv preprint arXiv …, 2024

… • RedPajama [21] (1.2T tokens): includes over 100B text documents coming from 84 CommonCrawl snapshots and processed using the … • DCLM-baseline [39] (1.35T tokens): a standardized corpus extracted from Common Crawl, effective pretraining …


Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

J Tan, Z Dou, Y Zhu, P Guo, K Fang, JR Wen - arXiv preprint arXiv:2402.12052, 2024

The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the …


Smaller Can Be Better: Efficient Data Selection for Pre-training Models

G Fang, S Wang, M Wang, Y Yang, H Huang - Asia-Pacific Web (APWeb) and Web …, 2024

To reduce the resource consumption for pre-training models on large-scale corpora, existing research has primarily focused on refining network architectures and training methods. Nevertheless, the concomitant cost tends to be expensive. This …


Smaller Language Models are Better Zero-shot Machine-Generated Text Detectors

N Mireshghallah, J Mattern, S Gao, R Shokri… - Proceedings of the 18th …, 2024

As large language models are becoming more embedded in different user-facing services, it is important to be able to distinguish between human-written and machine-generated text to verify the authenticity of news articles, product reviews, etc. Thus, in this paper …


Smart Automation Using LLM

P Ethape, R Kane, G Gadekar, S Chimane - International Research Journal of …, 2023

… Such large-scale models can ingest massive amounts of data, often from the internet, but also from sources such as the Common Crawl, which comprises more than 50 billion web pages, and Wikipedia, which has approximately 57 million pages. …


Smart Bilingual Focused Crawling of Parallel Documents

C García-Romero, M Esplà-Gomis, F Sánchez-Martínez - arXiv preprint arXiv …, 2024

Crawling parallel texts $\unicode{x2014}$texts that are mutual translations$\unicode{x2014}$ from the Internet is usually done following a brute-force approach: documents are massively downloaded in an unguided process, and only a fraction of them end up …


Smart Shields against Cyber Threats: Machine Learning-Driven Phishing URL Detection

Phishing attacks remain a prevalent cybersecurity threat, exploiting vulnerabilities in digital platforms to compromise sensitive user data. This paper introduces a novel machine learning-based framework for phishing URL detection, combining …


SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction

Y Zhou, H Shao, L Wang, SL Waslander, H Li, Y Liu - arXiv preprint arXiv:2410.08669, 2024

Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed environments. However, the scarcity of large-scale driving datasets has hindered the development …


SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization

K Park, S Lee - arXiv preprint arXiv:2412.08894, 2024

We propose SMMF (Square-Matricized Momentum Factorization), a memory-efficient optimizer that reduces the memory requirement of the widely used adaptive learning rate optimizers, such as Adam, by up to 96%. SMMF enables flexible and efficient …


Smoky Quartz: A Transparent Bilingual Large Language Model

Q Huang, C Zhang, M Tao, J Lin, S Huang, Y Feng

Recent advancements in large-scale models have been remarkable; however, a significant challenge persists in their lack of transparency, particularly concerning the training data utilized. This obscurity raises concerns about potential data leaks …


SMW Cloud: A Corpus of Domain-Specific Knowledge Graphs from Semantic MediaWikis

Semantic wikis have become an increasingly popular means of collaboratively managing Knowledge Graphs. They are powered by platforms such as Semantic MediaWiki and Wikibase, both of which enable MediaWiki to store and publish …


Social Evolution of Published Text and The Emergence of Artificial Intelligence Through Large Language Models and The Problem of Toxicity and Bias

A Khan, P Saravanan, SK Venkatesan - arXiv preprint arXiv:2402.07166, 2024

We provide a birds eye view of the rapid developments in AI and Deep Learning that has led to the path-breaking emergence of AI in Large Language Models. The aim of this study is to place all these developments in a pragmatic broader historical social …


Social Media Sentiment Analysis on Multilingual Dataset

This thesis presents using unlabeled data and soft targets from a LLM to train a pre-trained model for sentiment analysis. Inspired by the study “What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis” by Deng et …


SocialQuotes: Learning Contextual Roles of Social Media Quotes on the Web

J Palowitch, H Alvari, M Kazemi, T Amin, F Radlinski - arXiv preprint arXiv …, 2024

… We build and release SocialQuotes1, a data set computed from the Common Crawl with over 32M social media quotes, accompanied by role labels for 8.3k quotes obtained from human annotators. 3. We validate our main hypothesis with …


Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification

K Tatariya, H Lent, J Bjerva, M de Lhoneux - arXiv preprint arXiv:2402.03137, 2024

Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and …


SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training

N He, W Xiong, H Liu, Y Liao, L Ding, K Zhang, G Tang… - arXiv preprint arXiv …, 2024

… In summary, our experiments on the RedPajama CommonCrawl dataset substantiate that the soft deduplication method is capable of reducing perplexity and enhancing the accuracy of downstream tasks more efficiently compared to the …


SoK: SSO-MONITOR—The Current State and Future Research Directions in Single Sign-On Security Measurements

L Jannett, M Westers, T Wich, C Mainka, A Mayer…

Single Sign-On (SSO) with OAuth 2.0 and OpenID Connect 1.0 is essential for user authentication and authorization on the Internet. Billions of users rely on SSO services provided by Google, Facebook, and Apple. For large-scale measurements …


Some Considerations for the Preservation of Endangered Languages Using Low-Resource Machine Translation

A Kho, DS Pham, S Soon, KY Chan - Australasian Joint Conference on Artificial …, 2024

While widespread languages remain actively prevalent in digital mediums, endangered languages such as Indigenous Australian languages, are often scarce in textual resources and lack a substantial digital presence. This diminishes the …


Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers

E Liu, E Luo, S Shan, GM Voelker, BY Zhao, S Savage - arXiv preprint arXiv …, 2024

… We used the robots.txt files crawled by the Common Crawl from 2022 to 2024. We find that the coverage of the Common Crawl robots.txt data is approximately 83%. Ultimately, for sites that did not return a 200 or 404 HTTP status code to Common …


Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

M Walsh, A Preus, M Antoniak - arXiv preprint arXiv:2406.18906, 2024

… About 30% of our poems are found in the Common Crawl data included in Dolma, with the C4 dataset close behind. Wikipedia and Semantic Scholar contain the fewest detected poems. Overall, if at least one line from a poem is detected, it is …


Sourceformer: Tool Integration into Transformer Large Language Models Through Source Code

LLMs have been shown to use tools well. By allowing specific tools to increase the capabilities that LLMs struggle with, these models can become much more useful. Previous works use handcrafted examples of simple tool use during self-training …


Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales

Z Delbari, NS Moosavi, MT Pilehvar - Proceedings of the AAAI Conference on …, 2024

With the alarming rise of hate speech in online communities, the demand for effective NLP models to identify instances of offensive language has reached a critical point. However, the development of such models heavily relies on the …


Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

Y Luo, C Song, X Han, Y Chen, C Xiao, Z Liu, M Sun - arXiv preprint arXiv …, 2024

Activation sparsity denotes the existence of substantial weakly-contributed elements within activation outputs that can be eliminated, benefiting many important applications concerned with large language models (LLMs). Although promoting …


Spatial Language Understanding: Deep Learning, Reasoning, and Evaluation

… Pretrained language models (PLM) are large neural network models that have been trained on vast amounts of text data such as the entire text of Wikipedia or the Common Crawl web corpus in a un/supervised way. These models use transformers …


Speaker independent recognition of low-resourced multilingual Arabic spoken words through hybrid fusion

S Mehra, V Ranga, R Agarwal, S Susan - Multimedia Tools and Applications, 2024

This article introduces a supervised strategy designed to enhance spoken word recognition within the constraints of a resource-limited multilingual dataset, specifically focusing on the Arabic language. Notably, existing methodologies often …


Speaker Intimacy Estimation in Chat-Talks Based on Verbal and Non-Verbal Information

Conversations based on mutual intimacy are critical for maintaining positive relationships. A detailed understanding of speaker relationships in dialogues enhances various applications, such as information recommendation systems. Such …


Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction

J Pang, I Vulić - arXiv preprint arXiv:2405.10625, 2024

Transformer-based encoder-decoder models have demonstrated impressive results in chemical reaction prediction tasks. However, these models typically rely on pretraining using tens of millions of unlabelled molecules, which can be time-consuming …


Specialized Language Models with Cheap Inference from Limited Domain Data

D Grangier, A Katharopoulos, P Ablin, A Hannun - arXiv preprint arXiv:2402.01093, 2024

Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets. This work formalizes these constraints and distinguishes four important variables …


Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models

A Kaushal, T Pandey, T Vaidhya, A Bhagat, I Rish - arXiv preprint arXiv:2407.12327, 2024

Post-training quantization is the leading method for addressing memory-related bottlenecks in LLM inference, but unfortunately, it suffers from significant performance degradation below 4-bit precision. An alternative approach involves …


Speculative Enquiries: The third annual conference of the European Culture and Technology Laboratory.

… The collective work of our writing is published on the internet through Common Crawl, for example, which contributes substantially to its corpus. But interestingly, most of the generative Large Language Models like ChatGPT and Gemini demur …


SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

D Liu, R Zhang, L Qiu, S Huang, W Lin, S Zhao, S Geng… - Forty-first International Conference …

P Gao, R Zhang, C Liu, L Qiu, S Huang, W Lin, S Zhao… - arXiv preprint arXiv …, 2024

… Specifically, we first collect large-scale PDF datasets from Common Crawl 1 and arXiv websites. Then, we utilize PyMuPDF 2 to get the rendering results of each page in the PDF file and also save all the text annotations along with their bounding …


Spike No More: Stabilizing the Pre-training of Large Language Models

S Takase, S Kiyono, S Kobayashi, J Suzuki - arXiv preprint arXiv:2312.16903, 2023

… 2020) that consists of clean English texts extracted from Common Crawl7 as our LLM pre-training corpus. We also used the separated part of C4 as our validation data. We used GPT-2 vocabulary (Radford et al.… https://commoncrawl.org/ …


SPML: A DSL for Defending Language Models Against Prompt Attacks

RK Sharma, V Gupta, D Grossman - arXiv preprint arXiv:2402.11755, 2024

… The pre-training data included trillions of tokens sourced from publicly available datasets, such as Common Crawl, Wikipedia, and public domain books from Project Gutenberg. We use the versions with 7 billion and 13 billion parameters, available …


Sports, crisis, and social media: a Twitter-based exploration of the Tokyo Olympics in the COVID-19 era

V Mehra, P Singh, S Bharany, RS Sawhney - Social Network Analysis and Mining, 2024

The global landscape underwent a transformative shift due to the unexpected arrival of the COVID-19 pandemic, disrupting various aspects of life, including the staging of sports events. Notably, the Tokyo Olympics, a renowned international sports …


Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

A Hans, A Schwarzschild, V Cherepanova, H Kazemi… - arXiv preprint arXiv …, 2024

… When evaluating Binoculars on samples from languages that are not well represented in Common Crawl data (standard LLM pretraining data), we find that false-positives rates remain low, which is highly desirable from a harm reduction …


SRBerta—A Transformer Language Model for Serbian Cyrillic Legal Texts

M Bogdanović, J Kocić, L Stoimenov - Information, 2024

… CommonCrawl [5] is a collection of texts whose size is measured in petabytes. This collection was … , CommonCrawl contains texts of lower quality, so development teams often resorted to processing texts before training and extracting …


Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

W Du, T Luo, Z Qiu, Z Huang, Y Shen, R Cheng, Y Guo… - arXiv preprint arXiv …, 2024

LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth …


Stage-1 Report: Everything everywhere all at once? Exploring the richness of visual experience beyond words

L Qianchen, M Yamada, Y Aizawa, N Tsuchiya

In consciousness research, understanding the nature and mechanism of visual experience has been one of the most debated topics. For humans, vision is arguably the primary sensory modality to understand the environment (Cole & Balcetis, 2021) …


StarCoder 2 and The Stack v2: The Next Generation

A Lozhkov, R Li, LB Allal, F Cassano, J Lamy-Poirier… - arXiv preprint arXiv …, 2024

The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership with Software Heritage (SWH), we build The Stack v2 on …


Stars, Stripes, and Silicon: Unravelling the ChatGPT's All-American, Monochrome, Cis-centric Bias

F Torrielli - arXiv preprint arXiv:2410.13868, 2024

This paper investigates the challenges associated with bias, toxicity, unreliability, and lack of robustness in large language models (LLMs) such as ChatGPT. It emphasizes that these issues primarily stem from the quality and diversity of data on …


State of NLP in Kenya: A Survey

CJ Amol, EA Chimoto, RD Gesicho, AM Gitau, NA Etori… - arXiv preprint arXiv …, 2024

Kenya, known for its linguistic diversity, faces unique challenges and promising opportunities in advancing Natural Language Processing (NLP) technologies, particularly for its underrepresented indigenous languages. This survey provides a …


State of Title IX: A Knowledge Base for Title IX Documentation

Knowledge bases play a vital role in the modern world, offering a systematic and structured approach to integrate various entities, concepts, rules, and relationships associated with real-world information. In this study, we aggregated the official Title …


Statistical dataset evaluation: A case study on named entity recognition

C Wang, Q Dong, X Wang, Z Sui - Natural Language Processing

Datasets serve as crucial training resources and model performance trackers. However, existing datasets have exposed a plethora of problems, inducing biased models and unreliable evaluation results. In this paper, we propose a model-agnostic …


Steward: Natural Language Web Automation

B Tang, KG Shin - arXiv preprint arXiv:2409.15441, 2024

… They subsequently created WebAgent [17], another HTMLT5 type language model combined with a Flan-U-PaLM [8] program synthesis model trained on CommonCrawl [3] HTML and web interaction data. Their specially trained 540B parameter …


STFL: Utilizing a Semi-Supervised, Transfer-Learning, Federated-Learning Approach to Detect Phishing URL Attacks

I Sakazi, E Grolman, Y Elovici, A Shabtai - 2024 International Joint Conference on …, 2024

Phishing attacks are continually changing, so machine learning detection models must be continuously updated by collecting new URL data from users without compromising their privacy. Existing approaches for the detection of phishing URLs …


Stochastic Remembering and Distributed Mnemonic Agency: Recalling Twentieth Century Activists with ChatGPT

R Smit, T Smits, S Merrill - Memory Studies Review, 2024

This paper introduces the concept of stochastic remembering and uses two prompt engineering techniques to critically examine the text generated by ai chatbots. These techniques–step-by-step prompting and chain of thought reasoning–are then …


Strategies for Corpus Development for Low‐Resource Languages: Insights from Nepal

BK Bal, B Prasain, RR Ghimire, P Acharya - Automatic Speech Recognition and …, 2024

… 25 whereas the monolingual dataset were derived from Wikipedia and Common Crawl for languages under consideration. They also experimented with four variations of training settings: fully supervised, fully unsupervised, semisupervised …


Stratified Graph Indexing for E cient Search in Deep Descriptor Databases

Searching for unseen objects in extensive visual archives is challenging, demanding efficient indexing methods that can support meaningful similarity retrievals. This research paper presents the Stratified Graph (SG) approach for …


Strengthening the WiC: New Polysemy Dataset in Hindi and Lack of Cross Lingual Transfer

H Dubossarsky, F Dairkee - Proceedings of the 2024 Joint International Conference …, 2024

… TB of filtered CommonCrawl data as opposed to mBERT which was trained on Wikipedia data, and reported to outperform mBERT in a variety of cross-lingual benchmarks like XNLI, MLQA and NER. HindiBERT is a monolingual model trained …


Structured Packing in LLM Training Improves Long Context Utilization

K Staniszewski, S Tworkowski, S Jaszczur… - arXiv preprint arXiv …, 2023

Recent advances in long-context Large Language Models (LCLMs) have generated significant interest, especially in applications such as querying scientific research papers. However, their potential is often limited by inadequate context utilization. We …


Studies in conversational AI: multilingual capabilities, world knowledge, and evaluation strategies

This thesis studies the evolving landscape of conversational AI. The main research objective is to improve the conversational abilities of conversational agents, with a focus on integrating real-time knowledge and expanding multilingual capabil-ities …


Studying Peer Effects in Divergent Thinking: Theory and Method

In designing technology that supports user learning, an important first step is to understand how interactions among humans shape mutual learning. Much qualitative research in the realm of peer-assisted learning (PAL) has advanced the …


Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling

Y Chen, X Zhang, S Hu, X Han, Z Liu, M Sun - arXiv preprint arXiv:2410.07145, 2024

… Data We start from RedPajama-V2 (Computer, 2023), an open dataset with 30T tokens extracted from CommonCrawl4, and we perform deduplication to ensure data quality. During evaluation, we sample documents longer than 16K tokens and …


SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

Q Wang, Y Yuan, X Yang, R Zhang, K Zhao, W Liu… - arXiv preprint arXiv …, 2024

While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass …


Super donors and super recipients: Studying cross-lingual transfer between high-resource and low-resource languages

VPESE Voloshina, T Shavrina, A Panchenko

Despite the increasing popularity of multilingualism within the NLP community, numerous languages continue to be underrepresented due to the lack of available resources. Our work addresses this gap by introducing experiments on cross-lingual …


Supervised Learning

ER Vankov, K Gadhoumi - Statistical Methods in Epilepsy, 2024

… A corpus such as Common Crawl, which contains roughly 840 billion tokens of over two million unique words, provides such scale. There are other corpora used in these unsupervised processes, one such common candidate being a collection of all …


SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

J Kim, J Nam, S Mo, J Park, SW Lee, M Seo, JW Ha… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising …


Surface Aesthetics: The Secret Formula of Prompting

… OpenPageRank is a website ranking metrics system that draws on open source data provided by Common Crawl and Common Search. While there are many indicators to rank a website popularity, the page rank calculated by OpenPageRank …


Surfacing Text Changes in Archived Webpages

… Common Crawl datasets started in 2008, and are collected monthly at the present time. The most recent Common Crawl from July 2024 contains 2.5 billion webpages. Large datasets, like ClueWeb and CommonCrawl, crawl at different points in time …


Suri: Multi-constraint Instruction Following for Long-form Text Generation

CM Pham, S Sun, M Iyyer - arXiv preprint arXiv:2406.19371, 2024

Existing research on instruction following largely focuses on tasks with simple instructions and short responses. In this work, we explore multi-constraint instruction following for generating long-form text. We create Suri, a dataset with 20K human-written …


Survey of Cultural Awareness in Language Models: Text and Beyond

S Pawar, J Park, J Jin, A Arora, J Myung, S Yadav…

Large-scale deployment of large language models (LLMs) in various applications, such as chatbots and virtual assistants, requires LLMs to be culturally sensitive to the user to ensure inclusivity. Culture has been widely studied in psychology and …


Survey of different Large Language Model Ar-chitectures: Trends, Benchmarks, and Challenges

M SHAO, A BASIT, R KARRI, M SHAFIQUE - structure

Large Language Models (LLMs) represent a class of deep learning models adept at understanding natural language and generating coherent text in response to prompts or queries. These models significantly surpass conventional neural …


Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy

P Pattnayak, HL Patel, B Kumar, A Agarwal, I Banerjee… - arXiv preprint arXiv …, 2024

Multimodal learning, a rapidly evolving field in artificial intelligence, seeks to construct more versatile and robust systems by integrating and analyzing diverse types of data, including text, images, audio, and video. Inspired by the human ability …


Survey on Abstractive Text Summarization: Dataset, Models, and Metrics

GO Nnadi, F Bertini - arXiv preprint arXiv:2412.17165, 2024

… To extend the input data for large-scale news article generation, similar articles are sourced from the Common Crawl News dataset. The WCEP dataset is well-aligned with real-world industrial use cases, making it suitable for training and evaluating …


Survey On Enhancing Dialogue Agent Alignment via MiniLLM with Targeted Human Assessments

MS Mahajan, MC Vaidya, BL Narware, DR Yemde…

This paper presents the development of a com-pact and effective language model inspired by the LLaMA architecture. Our focus was on constructing this model based on the fundamental principles of LLaMA, which have influenced our architectural …


Survey on Semantic Interpretation of Tabular Data: Challenges and Directions

M Cremaschi, B Spahiu, M Palmonari, E Jimenez-Ruiz - arXiv preprint arXiv …, 2024

Tabular data plays a pivotal role in various fields, making it a popular format for data manipulation and exchange, particularly on the web. The interpretation, extraction, and processing of tabular information are invaluable for knowledge-intensive …


Surveying Model Collapse: Empirics, Universality, and Implications

… In the real world, generative outputs pollute the common crawl via accumulation, not replacement; while the real training data diminishes in proportion, as portrayed in Figure 5, it remains available for training at any generation, potentially allowing …


Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

A Havrilla, A Dai, L O'Mahony, K Oostermeijer, V Zisler… - arXiv preprint arXiv …, 2024

Synthetic data generation with Large Language Models is a promising paradigm for augmenting natural data over a nearly infinite range of tasks. Given this variety, direct comparisons among synthetic data generation algorithms are scarce, making …


SWEb: A Large Web Dataset for the Scandinavian Languages

T Norlund, T Isbister, AC Gyllensten, PD Santos… - arXiv preprint arXiv …, 2024

… SWEb is the result of running our proposed pipeline on 98 Common Crawl snapshots. SWEb contains … Common Crawl, we want to implement this step early to filter out all non-relevant content. We use CCNet to identify Scandinavian …


SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

H Li, C Tian, J Shao, X Zhu, Z Wang, J Zhu, W Dou… - arXiv preprint arXiv …, 2024

The remarkable success of Large Language Models (LLMs) has extended to the multimodal domain, achieving outstanding performance in image understanding and generation. Recent efforts to develop unified Multimodal Large Language …


Synthetic continued pretraining

Z Yang, N Band, S Li, E Candès, T Hashimoto - arXiv preprint arXiv:2409.07431, 2024

Pretraining on large-scale, unstructured internet text has enabled language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient -- to learn a given fact, models must be trained on …


Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation

S Soni, A Kunchukuttan, T Chakraborty, MS Akhtar - arXiv preprint arXiv …, 2024

The widespread online communication in a modern multilingual world has provided opportunities to blend more than one language (aka code-mixed language) in a single utterance. This has resulted a formidable challenge for the computational …


Synthetic data in biomedicine via generative artificial intelligence

B van Breugel, T Liu, D Oglic, M van der Schaar - Nature Reviews Bioengineering, 2024

The creation and application of data in biomedicine and healthcare often face privacy constraints, bias, distributional shifts, underrepresentation of certain groups and data scarcity. Some of these challenges may be addressed by synthetic data …


System for News Summarizing Using a Neural Network Algorithm

A Akhmedova, I Zhazhneva, G Galimov - 2024 International Russian Automation …, 2024

This paper presents implementation of a system for summarizing news using a neural network algorithm. Relevance of the development of this system is due to the need for users to receive news in a concise form, while maintaining objectivity and …


Systematic Evaluation of Linear Transformations for Cross-lingual Semantic Spaces

In the field of natural language understanding (NLU), a fundamental element is the representation of word meanings, a process that is integral to a wide range of applications. These applications span numerous domains including machine …


Systematic exploration and in-depth analysis of ChatGPT architectures progression

D Banik, N Pati, A Sharma - Artificial Intelligence Review, 2024

The fast evolution of artificial intelligence frameworks has resulted in the creation of increasingly sophisticated large language models (LLM), ChatGPT being the most famous one. This study paper dives into this LLM with a case study of ChatGPT’s …


T5G2P: Text-to-Text Transfer Transformer based Grapheme-to-Phoneme Conversion

M Řezáčková, D Tihelka, J Matoušek - IEEE/ACM Transactions on Audio, Speech …, 2024

… For the Czech language, we collected the CommonCrawl corpus at the end of August 2020. It contains 47.9GB of clean text, 20.5 million … Similarly, for both Russian and German we collected CommonCrawl corpus in the season from August …


Tübingen-CL at SemEval-2024 Task 1: Ensemble Learning for Semantic Relatedness Estimation

L Zhang, Ç Çöltekin - Proceedings of the 18th International Workshop on …, 2024

The paper introduces our system for SemEval-2024 Task 1, which aims to predict the relatedness of sentence pairs. Operating under the hypothesis that semantic relatedness is a broader concept that extends beyond mere similarity of sentences …


TabDPT: Scaling Tabular Foundation Models

J Ma, V Thomas, R Hosseinzadeh, H Kamkari… - arXiv preprint arXiv …, 2024

The challenges faced by neural networks on tabular data are well-documented and have hampered the progress of tabular foundation models. Techniques leveraging in-context learning (ICL) have shown promise here, allowing for dynamic adaptation to unseen …


Table Representation Learning

The increasing amount of data being collected, stored, and analyzed, induces a need for efficient, scalable, and robust methods to handle this data. Representation learning, ie, the practice of leveraging neural networks to obtain generic vector …


Tackling an Unbalanced Dataset for Classifying Indonesian E-Commerce Reviews Using Multi Word Embedding Model

R Adi, BR Irnawan, J Li - 2023 Eighth International Conference on Informatics …, 2023

… Our research used a pre-trained FastText that was trained on Indonesian Common Crawl and Wikipedia provided by [24]. 3) GloVe: GloVe is an unsupervised algorithm that can produce a vector representation of words created by [22]. Besides …


Tackling the Problem of Multilingualism in Voice Assistants

… from the C4 dataset - a vast collection of English text from Common Crawl, totaling around 750GB. The model's strengths lie in its unified … This multilingual model was pre-trained on a dataset derived from Common Crawl, encompassing an …


Take the Bull by the Horns: Hard Sample-Reweighted Continual Training Improves LLM Generalization

X Chen, Z Wang, D Sow, J Yang, T Chen, Y Liang… - arXiv preprint arXiv …, 2024

… subset of the Common Crawl’s web crawl corpus. More recently, LLaMA (Touvron et al.… datasets to encompass multiple domains such as Common Crawl, C4, and source codes from … contents are mostly unintelligible” in the Common Crawl. All …


Tamil Language Computing: the Present and the Future

K Sarveswaran - arXiv preprint arXiv:2407.08618, 2024

This paper delves into the text processing aspects of Language Computing, which enables computers to understand, interpret, and generate human language. Focusing on tasks such as speech recognition, machine translation, sentiment …


Task Oriented In-Domain Data Augmentation

X Liang, X Hu, S Zuo, Y Gong, Q Lou, Y Liu, SL Huang… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) have shown superior performance in various applications and fields. To achieve better performance on specialized domains such as law and advertisement, LLMs are often continue pre-trained on in-domain data …


Tasneef: A Fast and Effective Hybrid Representation Approach for Arabic Text Classification

M Louail, CKM Hamdi-Cherif, A Hamdi-Cherif - IEEE Access, 2024

The Arabic language role in actual global affairs entails sophisticated natural language processing techniques, especially in text classification. This paper presents Tasneef as a novel hybrid approach to tackle computational challenges by …


TEA+: A Novel Temporal Graph Random Walk Engine With Hybrid Storage Architecture

C Huan, Y Liu, H Zhang, S Song, S Pandey, S Chen… - ACM Transactions on Architecture …

Many real-world networks are characterized by being temporal and dynamic, wherein the temporal information signifies the changes in connections, such as the addition or removal of links between nodes. Employing random walks on these …


Team Aditya at EXIST 2024—Detecting Sexism In Multilingual Tweets Using Contrastive Learning Approach

A Shah, A Gokhale - Working Notes of CLEF, 2024

Due to the growing impact of social media, the necessity for automated mechanisms that can identify sexism and other forms of disrespectful and hateful conduct is rising, aiming to create a more inclusive and respectful digital space. However, it poses …


Team OpenWebSearch at CLEF 2024: LongEval

D Alexander, M Fröbe, G Hendriksen, F Schlatt… - 2024

We describe the OpenWebSearch group’s participation in the CLEF 2024 LongEval IR track. Our submitted runs explore how historical data from the past can be transferred into future retrieval systems. Therefore, we incorporate relevance …


Technical Report: Small Language Model for Japanese Clinical and Medicine

S Watanabe - arXiv preprint arXiv:2412.16423, 2024

This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. This 1B parameters model was trained using Japanese text classified to be of high-quality. Moreover, NCVC-slm-1 was …


TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu

G Kanumolu, L Madasu, N Surange, M Shrivastava - arXiv preprint arXiv:2404.11349, 2024

… 2019) is a multilingual version of the RoBERTa model, and it was pre-trained on a vast 2.5TB CommonCrawl dataset, which included text from 100 languages. For our experiments, we utilized the xlm-roberta-base variant, boasting 270 million parameters. …


TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain

C Barboule, VP Huynh, A Bufort, Y Chabot, G Damnati… - arXiv preprint arXiv …, 2024

… For documents collected in Common Crawl, PDFMiner is used to extract the texts. To further assess the relevance and quality of the texts extracted, the Zephyr-7b (Tunstall et al., 2023) model was prompted to classify whether a text is linked to the …


Tele-FLM Technical Report

X Li, Y Yao, X Jiang, X Fang, C Wang, X Liu, Z Wang… - arXiv preprint arXiv …, 2024

… CommonCrawl1 is often considered to be a repository containing diverse human experience … CommonCrawl are primarily concentrated in the English segment, with the Chinese content exhibiting relatively lower information density and quality …


Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications

A Maatouk, KC Ampudia, R Ying, L Tassiulas - arXiv preprint arXiv:2409.05314, 2024

… is the Common Crawl dataset, which comprises web archives from across the internet. To avoid issues with duplicates, non-English content, and potential profanity found in raw dumps, we utilized the refined web dataset [26]. This curated …


TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models

H Zou, Q Zhao, Y Tian, L Bariah, F Bader, T Lestable… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. However, current mainstream LLMs generally lack the specialized knowledge in telecom domain. In this paper, for the …


Tell me a story: a framework for critically investigating AI language models

L Munn, L Henrickson - Learning, Media and Technology, 2024

Large language models are rapidly being rolled out into high-stakes fields like healthcare, law, and education. However, understanding of their design considerations, operational logics, and implicit biases remains limited. How might …


TEMA: Token Embeddings Mapping for Enriching Low-Resource Language Models

R Zevallos, N Bel, M Farrús - Proceedings of the 2024 Conference on Empirical …, 2024

The objective of the research we present is to remedy the problem of the low quality of language models for low-resource languages. We introduce an algorithm, the Token Embedding Mapping Algorithm (TEMA), that maps the token embeddings of a …


Temporal Blind Spots in Large Language Models

J Wallat, A Jatowt, A Anand - arXiv preprint arXiv:2401.12078, 2024

… A mixture of training datasets has been used: a filtered version of CommonCrawl, an expanded version of WebText [44], two not further specified internet-based book corpora, and the English Wikipedia. The most recent information in text-davinci-003’s …


Temporally-Varied News Headline Generation with GPT-3


Terra: A Multimodal Spatio-Temporal Dataset Spanning the Earth

W Chen, X Hao, Y Liang - The Thirty-eight Conference on Neural Information …

Since the inception of our planet, the meteorological environment, as reflected through spatio-temporal data, has always been a fundamental factor influencing human life, socio-economic progress, and ecological conservation. A …


Testing and Adapting the Representational Abilities of Large Language Models on Folktales in Low-Resource Languages

JA Meaney, B Alex, W Lamb - Proceedings of the 4th International Conference on …, 2024

Folktales are a rich resource of knowledge about the society and culture of a civilisation. Digital folklore research aims to use automated techniques to better understand these folktales, and it relies on abstract representations of the textual …


Testing proportional sentence analogies on SATS: from vector offsets to conditional generation

… 23], SBERT9 [18], and summed embeddings from the English Common Crawl CBOW FastText embeddings [24]. We use the Base checkpoint for all models where applicable. To the second point, we examine three vector solvers: 3CosAdd, a …


TEXAS INTELLECTUAL PROPERTY LAW JOURNAL

J Nacht, A Neill, J Thomas, E Lee, KE Noonan… - 2024

… 4t Common Crawl contains information from over 250 billion web pages since 2007 using web crawlers … in Common Crawl was filtered before being incorporated into GPT-3 to maintain a higher level of quality for training purposes.Websites …


Text classification for private procurement: a survey and an analysis of future trends

F Bellomi, M Cristani - Information Technology and Management, 2024

… A widely used base of these techniques is the Common Crawl, an open source collection of textual (and also diverse) datasets coming from the web that can be used as a basis for similarity analysis [14]. Recently, the common crawl has also …


Text Emotion Recognition Using Fast Text Word Embedding in Bi-Directional Gated Recurrent Unit

C Akalya Devi, D Karthika Renuka, T Harisudhan…

Emotions are states of readiness in the mind that result from evaluations of one's own thinking or events. Although almost all of the important events in our lives are marked by emotions, the nature, causes, and effects of emotions are some of the …


Text Filtering Classifiers for Medium-Resource Languages

J Daðason, H Loftsson - Proceedings of the 2024 Joint International Conference …, 2024

… Online texts are often obtained from large datasets, such as those published by Common Crawl (CC).The GPT-3 model was pre-trained on 499B tokens, 410B of which were obtained from CC (Brown et al.… The Icelandic Common Crawl Corpus (IC3) …


Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy

A Plum, T Ranasinghe, C Purschke - arXiv preprint arXiv:2412.09415, 2024

This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. Despite its active development, Luxembourgish faces a digital data scarcity, exacerbated by Luxembourg's …


Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges

J Becker, JP Wahle, B Gipp, T Ruas - arXiv preprint arXiv:2405.15604, 2024

Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review …


TEXT SUMMARIZATION SYSTEM USING NATURAL LANGUAGE PROCESSING AND MACHINE LEARNING

Information overload, caused by the rapid growth of the Internet, has become a significant issue. The abundance of information available online necessitates the simplification of relevant content through summarization. Manual summarization of …


Text to SQL Transformation Using LLM: A Comparative Research of T5, Seq2Seq, and SQLNet Models

Z Shaikhiyeva, M Mansurova, G Amirkhanova - 2024 9th International Conference on …, 2024

The transformation of natural language text into SQL queries is a critical task in the domain of natural language processing and database management. This paper presents a comparative analysis of three prominent models: T5 (Text-to-Text …


Text-to-text generative approach for enhanced complex word identification

P Śliwiak, SAA Shah - Neurocomputing, 2024

This paper presents a novel approach for solving the Complex Word Identification (CWI) task using the text-to-text generative model. The CWI task involves identifying complex words in text, which is a challenging Natural Language Processing task. To …


TextGram: Towards a Better Domain-Adaptive Pretraining

O Dhekane, S Hiwarkhedkar, R Joshi, A Ladkat - … , Perundurai, Erode, India, December 6–8 …

… The RealNews Dataset is a large corpus of news articles scraped from Common Crawl [16]. It consists of 5000 news domains that are indexed by Google news. IMDb Movie Reviews is a binary sentiment analysis dataset made up of 50000 reviews …


TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens

YQ Yu, M Liao, J Zhang, J Wu - arXiv preprint arXiv:2410.05261, 2024

… We also use PDFPlumber to extract text lines from Common Crawl PDFs. To improving English handwriting recognition, we incorporate RenderedText (StabilityAI & LAION, 2023). Additional end-to-end OCR datasets, including ArT (Chng et al …


Textual Coverage of Eventive Entries in Lexical Semantic Resources

E Fucikova, CF Alcaina, J Hajic, Z Urešová - Proceedings of the 2024 Joint …, 2024

This short paper focuses on the coverage of eventive entries (verbs, predicates, etc.) of some well-known lexical semantic resources when applied to random running texts taken from the internet. While coverage gaps are often reported for manually …


Textual emotion detection–A systematic literature review

Textual emotion detection is a critical area of study with significant applications in business, education, and healthcare. Despite substantial theoretical advancements over the years, there is a notable gap in the practical implementation of these …


The advantages of context specific language models: the case of the Erasmian Language Model

J Gonçalves, N Jelicic, M Murgia, E Stamhuis - arXiv preprint arXiv:2408.06931, 2024

… The dominant trend in improving language models seems to be based on scaling up the number of parameters (eg GPT4 has approximately 1.7 trillion parameters) or the amount of training data fed into the model (eg the Common Crawl dataset …


The African Woman is Rhythmic and Soulful: An Investigation of Implicit Biases in LLM Open-ended Text Generation

This paper investigates the subtle and often concealed biases present in Large Language Models (LLMs), focusing on implicit biases that may remain despite passing explicit bias tests. Implicit biases are significant because they influence the …


The African Woman is Rhythmic and Soulful: Evaluation of Open-ended Generation for Implicit Biases

S Lim - arXiv preprint arXiv:2407.01270, 2024

This study investigates the subtle and often concealed biases present in Large Language Models (LLMs), which, despite passing explicit bias tests, can still exhibit implicit biases akin to those observed in humans who profess egalitarian beliefs yet …


The AI Companion in Education: Analyzing the Pedagogical Potential of ChatGPT in Computer Science and Engineering

Z He, T Nguyen, T Miari, M Aliasgari, S Rafatirad…

… ChatGPT3 employs a pre-trained large language model [29] trained over the Common Crawl dataset [30]. During its training process, it first preprocess data from the Common Crawl dataset, data quality is refined by eliminating irrelevant or noisy …


The AI Did My Job Well

Artificial intelligence (AI), an idea first proposed back in 1955 (McCarthy, Minsky, & Rochester) seemingly has burst into public consciousness nearly 70 years later. Like the chained prisoners emerging from Plato’s allegorical cave, many of us find …


The Aranea Corpora Family: Ten+ Years of Processing Web-Crawled Data

V Benko - International Conference on Text, Speech, and …, 2024

Aranea is a project to create a family of web-crawled corpora for languages taught at Slovak Universities. Since 2013, more than two dozen languages have been added to the project: they are often represented with Gigaword+ size corpora and …


The Bangla/Bengali Seed Dataset Submission to the WMT24 Open Language Data Initiative Shared Task

We contribute a seed dataset for the Bangla/Bengali language as part of the WMT24 Open Language Data Initiative shared task. We validate the quality of the dataset against a mined and automatically aligned dataset (NLLBv1) and two other existing …


The BEA 2024 Shared Task on the Multilingual Lexical Simplification Pipeline

M Shardlow, F Alva-Manchego, RT Batista-Navarro… - … on Innovative Use of NLP for …, 2024

We report the findings of the 2024 Multilingual Lexical Simplification Pipeline shared task. We released a new dataset comprising 5,927 instances of lexical complexity prediction and lexical simplification on common contexts across 10 languages, split …


The Blue Behemoth

Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient---to learn a fact, models must be trained on hundreds to …


The bounded intelligence of AI: Superficiality and deceivability

H Ma, M Su - Organizational Dynamics, 2024

… A 2024 study by the Pew Research Center found that, in their sample of about one million webpages from the Internet archive Common Crawl collected between 2013 and October 2023, a quarter were no longer accessible (showing 404 Not …


The Case For Data Centre Hyperloops

G López-Paradís, IM Hair, S Kannan, R Rabbat… - 2024 ACM/IEEE 51st …, 2024

G López-Paradıs, IM Hair, S Kannan, R Rabbat…

Data movement is a hot-button topic today, with workloads like machine learning (ML) training, graph processing, and data analytics consuming datasets as large as 30PB. Such a dataset would take almost a week to transfer at 400gbps while consuming …


The Case for Developing a Foundation Model for Planning-like Tasks from Scratch

B Srivastava, V Pallagani - arXiv preprint arXiv:2404.04540, 2024

Foundation Models (FMs) have revolutionized many areas of computing, including Automated Planning and Scheduling (APS). For example, a recent study found them useful for planning problems: plan generation, language translation, model …


The Case for Nurturing AI Literacy in Law Schools

S Migliorini, JI Moreira - Asian Journal of Legal Education, 2024

The debate surrounding the permissibility of generative artificial intelligence (AI) tools in legal education has garnered widespread attention. However, this discourse has largely oscillated between the advantages and disadvantages of generative AI …


The correlation between nativelike selection and prototypicality: a multilingual onomasiological case study using semantic embedding

H Zhang - arXiv preprint arXiv:2405.13529, 2024

In native speakers' lexical choices, a concept can be more readily expressed by one expression over another grammatical one, a phenomenon known as nativelike selection (NLS). In previous research, arbitrary chunks such as collocations have …


The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

A Birhane, S Dehdashtian, VU Prabhu, V Boddeti - arXiv preprint arXiv:2405.04623, 2024

… This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the …


The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination

D Barman, Z Guo, O Conlan - Machine Learning with Applications, 2024

Disinformation - the deliberate spread of false or misleading information poses a significant threat to our society by undermining trust, exacerbating polarization, and manipulating public opinion. With the rapid advancement of artificial intelligence and …


THE DETERMINANTS OF SOFTWARE DEVELOPERS'INTENTION TO ADOPT CHAT GPT AS PROGRAMMING INFORMATION MEDIA

ID WIRAWAN, T OKTAVIA - Journal of Theoretical and Applied Information …, 2024

In the digital era, technology is developing rapidly. One of them in the field of artificial intelligence is chatbots. A chatbot is a computer program designed to stimulate conversation or interactive communication with (human) users via text …


The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study

DM Levine, R Tuwani, B Kompa, A Varma… - The Lancet Digital Health, 2024

… GPT-3 is one of the largest AI models ever created, containing over 175 billion model parameters, and was trained on approximately 570 gigabytes of data from the common crawl dataset, which contains nearly all unstructured text available on the …


The effect of generative artificial intelligence on managerial behavior across functions

GAI is becoming readily available and easily accessible for those who are willing to take advantage of it. As a result, the world is witnessing a transformation regarding how managers perform their daily tasks. This study aims to understand the …


THE ENHANCEMENT OF OPEN DATA USABILITY BY DEFINING A CATEGORIZATION METHOD BASED ON THE OPEN DATA PORTAL METADATA

Due to numerous data transparency and open government initiatives, a large volume of data was published on open data portals. To make it more accessible and visible, these portals have introduced data filtering by category, tags, format …


The ethical situation of DALL-E 2

E Hogea, J Rocafortf - arXiv preprint arXiv:2405.19176, 2024

A hot topic of Artificial Intelligence right now is image generation from prompts. DALL-E 2 is one of the biggest names in this domain, as it allows people to create images from simple text inputs, to even more complicated ones. The company that made this …


The Ethics and Challenges of Legal Personhood for AI

Hon. Katherine B. Forrest (Fmr.) abstract. AI’s increasing cognitive abilities will raise challenges for judges.“Legal personhood” is a flexible and political concept that has evolved throughout American history. In determining whether to expand that concept …


The Evolution of Multimodal Model Architectures

SN Wadekar, A Chaurasia, A Chadha, E Culurciello - arXiv preprint arXiv:2405.17927, 2024

This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type facilitates monitoring of developments in …


The Fill-Mask Association Test (FMAT): Measuring Propositions in Natural Language

HWS Bao - Journal of Personality and Social Psychology, 2024

L Lin, B Wang, X Wang, ZX Wang, A Wiśniowski…

Recent advances in large language models are enabling the computational intelligent analysis of psychology in natural language. Here, the Fill-Mask Association Test (FMAT) is introduced as a novel and integrative method leveraging …


The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

G Penedo, H Kydlíček, A Lozhkov, M Mitchell, C Raffel… - arXiv preprint arXiv …, 2024

… In this work, we introduce FineWeb, a 15-trillion token dataset derived from 96 Common Crawl snapshots that produces better-performing LLMs than other open pretraining datasets. To advance the understanding of how best to curate high-quality …


The Future of Chatbots: Survey on Generative Pretrained Models, Ernie, and Gemini

A Raut, J Katti - 2024 International Conference on Innovations and …, 2024

The evolution of chatbots from rule-based, retrieval-based, contextual systems, to hybrid and generative AIpowered models. It aimsto solve complex natural language processing and task-specific challenges. The finding explores innovative self-attention …


The Future of Data Science in Materials Science

P Chundi, V Bommanapally, V Gadhamshetty - Machine Learning in 2D Materials …, 2023

… GPT-3 is an autoregressive model built with 96 layers [31], increased embedding vector dimension of 12888, trained with 175B parameters on five datasets including Common crawl, WebText2, Books1, Books2, and Wikipedia. Though GPT-3 has …


The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare

… Namely, datasets used for training LLMs such as Common Crawl contain petabytes of data [260]. Therefore, it is not possible to have complete human oversight over the data cleaning process. Hence, this is why heuristic methods are …


The future

F Geeraert, M Kurzmeier, LA Denis, V Schafer… - The Routledge Companion …, 2024

Friedel Geeraert: I hope that we will be able to continue the work that was started in the WARCnet network in the context of a new research framework in Europe, and beyond, thereby helping to promote the value of web archives for research. As an …


The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)

S Zeng, J Zhang, P He, Y Xing, Y Liu, H Xu, J Ren… - arXiv preprint arXiv …, 2024

… 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. … 2021), we randomly select chunks from the Common Crawl dataset to serve as the {information} component. Due to the random …


The High Stakes of AI Ethics: Evaluating OpenAI's Potential Shift to “NSFW” Content and Other Concerns

… These address concerns surrounding the dubious curation practices used to generate these datasets, the sordid quality of alt-text data available on the world wide web, the problematic content of the CommonCrawl dataset often used as a …


The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis

M Zhang, V Gautam, M Wang, JO Alabi, X Shen… - arXiv preprint arXiv …, 2024

… The order of languages follows their data ratio in the CommonCrawl corpus10 from high-resource to low-resource. We observe large variations in model performance across different languages. For instance, there exists a large …


The Impact of Element Ordering on LM Agent Performance

W Chi, A Talwalkar, C Donahue - arXiv preprint arXiv:2409.12089, 2024

… In our approach, we leverage common crawl [3] to train an object detection model [16] that detects interactable UI elements specifically for agents. To the best of our knowledge, the elements detected through these approaches are ordered arbitrarily (eg …


The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

M Munn, B Dherin, J Gonzalvo - arXiv preprint arXiv:2405.15706, 2024

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical …


The Impact of Infrastructure: Considerations of Generative AI in the Classroom

ES Kehlenbach - Journal of Political Science Education, 2024

How are we to grapple with the increasing intrusion of technology in the classroom? What considerations need to be addressed before we begin bringing advanced technological systems like ChatGPT into our classroom spaces? A close attention to …


The impact of language family on D2T generation in under-resourced languages

The paper delves deep into the challenges and methods of generating text from structured data(RDF triples) for under-resourced languages based on the WebNLG challenge. The main question of this paper is to assess how important language …


The Impact of Online Censorship on LLMs

M Ahmed, J Knockel - Free and Open Communications on the Internet, 2024

… the Common Crawl contains censored content, there may be downstream effects on the models. To investigate whether the Common Crawl … Of which, we found pages indexed from 325 of the censored websites in the Common Crawl dataset that …


The Impact of OpenAI on the Work Environment located in Finland

In the modern era, technology has undergone a rapid transformation that touches every aspect of our lives. At the forefront of this transformation stands artificial intelligence (AI), a field rapidly evolving through groundbreaking innovations, and …


The Influence of ChatGpt on Students' Opportunistic Behavior and Reduced Effort

This study challenges the assumption that students become less invested in tasks once they achieve an average grade using AI tools like ChatGPT, this leads to growing concerns about students overreliance on ChatGPT to complete their tasks …


The influence of noise traders on the Vietnamese stock market

NT Hoài - Tạp chí Khoa học Thăng Long: Khoa học Ứng dụng, 2024

The study examines the impact of noise trading on stock returns. The investor sentiment index is constructed as a representative of noise trading. This index is extracted from the social network Facebook and the mainstream online newspapers …


the landscape to quantify creative

IU Haq, M Pifarré - Education Reimagined: The Impact of Advanced …, 2024

… or unsupervised methods (no prior training) that are trained on a large corpus, eg, Wikipedia and common crawl dataset. Deep learning models such as Word2Vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) use knowledge from large datasets …


The Large Language Model GreekDeBERTa

ABSTRACT Language Models for Natural Language Processing (NLP) have constituted a significant area of research and development for over two decades, with a notable trend towards increasing model complexity and size. The general …


The life cycle of large language models in education: A framework for understanding sources of bias

J Lee, Y Hicke, R Yu, C Brooks, RF Kizilcec - British Journal of Educational …, 2024

Large language models (LLMs) are increasingly adopted in educational contexts to provide personalized support to students and teachers. The unprecedented capacity of LLM‐based applications to understand and generate natural language can …


The losT daTa: how aI sysTems censor lGBTQ+ conTenT In The name of safeTy

… The Google team made the C4 dataset using a snapshot of the Internet in April 2019, as collected by the non-profit Common Crawl, which uses bots to make a free and publicly available copy of a subset of the Internet. But raw Common Crawl data …


The Mirage of Artificial Intelligence Terms of Use Restrictions

P Henderson, MA Lemley - arXiv preprint arXiv:2412.07066, 2024

… For example, many model creators start with Common Crawl and then filter the data for quality.The selection of “everything available for scraping on the internet” is not sufficiently creative to qualify for copyright protection. If there is enough creativity …


The Mitigation of Excessive Retrieval Augmentation and Knowledge Conflicts in Large Language Models

M Galway, M DiRenzo, D Esposito, R Marchand… - 2024

There have been significant advances with large language models in generating coherent and contextually relevant responses, but their limitations in accessing and integrating real-time or specialized information have driven the development of …


The model student: GPT-4 performance on graduate biomedical science exams

D Stribling, Y Xia, MK Amer, KS Graim, CJ Mulligan… - Scientific Reports, 2024

… As these study materials may have been incorporated into the GPT-4 training data such as the Common Crawl 43 , standardized examinations may not be an accurate assessment of domain-specific model knowledgebase and capability …


The OECD-UNSD Multinational Enterprise Information Platform

… (3) Common Crawl is an open source initiative to provide a monthly snapshot of the internet through web scraping. As part of the project, they provide a quarterly network graph of hyperlinks between domains, which can be used to determine …


The Open Web Index

G Hendriksen, M Dinzinger, SM Farzana, NA Fathima…

… To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files.This approach allows us …


The Open Web Index: Crawling and Indexing the Web for Public Use

G Hendriksen, M Dinzinger, SM Farzana, NA Fathima… - European Conference on …, 2024

… To operationalize our crawling pipeline, we introduce a re-crawl mechanism specifically for Common Crawl dumps. Unlike Common Crawl, we crawl on a daily basis and save the results as WARC (Web ARChive) files. Footnote 11 This …


The performance of the LSTM-based code generated by Large Language Models (LLMs) in forecasting time series data

S Gopali, S Siami-Namini, F Abri, AS Namin - Natural Language Processing Journal, 2024

Generative AI, and in particular Large Language Models (LLMs), have gained substantial momentum due to their wide applications in various disciplines. While the use of these game changing technologies in generating textual information has …


The poison of dimensionality

LN Hoang - arXiv preprint arXiv:2409.17328, 2024

This paper advances the understanding of how the size of a machine learning model affects its vulnerability to poisoning, despite state-of-the-art defenses. Given isotropic random honest feature vectors and the geometric median (or clipped mean) …


The Power of Absence: Thinking with Archival Theory in Algorithmic Design

J Sherman, R Morrison, L Klein, DK Rosner - arXiv preprint arXiv:2405.05420, 2024

… , for example–but also of datasets like the Colossal Clean Common Crawl corpus, or C4, which explicitly documents its decision to include … [33] have shown how the quality filters imposed on the original Common Crawl dataset exhibit a preference …


The Practical Epistemologies of Design and Artificial Intelligence

W Billingsley - Science & Education, 2024

This article explores the epistemological trade-offs that practical and technology design fields make by exploring past philosophical discussions of design, practitioner research, and pragmatism. It argues that as technologists apply Artificial …


The principal components of meaning, revisited

C Westbury, M Yang, K Anderson - Psychonomic Bulletin & Review, 2024

… the Common Crawl corpus (https://commoncrawl.org/), a web-derived corpus. As its name suggests, the model has vectors of length 300. … ://nlp.stanford.edu/projects/glove/), a matrix with vectors of length 300 that was trained on 840 billion tokens from …


The Progenitors of the Information Age and the Birth of AI: Introducing the “Political Economy of the Fourth Industrial Revolution”

America’s economy is increasingly fueled by digital devices, machines, and appliances that communicate with each other over the internet, ever-improving algorithms for driverless cars and virtual reality, and generative Artificial Intelligence (AI) …


The research landscape on generative artificial intelligence: a bibliometric analysis of transformer-based models

Purpose The aim of this study is to offer valuable insights to businesses and facilitate better understanding on transformer-based models (TBMs), which are among the widely employed generative artificial intelligence (GAI) models, garnering …


The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources

S Longpre, S Biderman, A Albalak, H Schoelkopf… - arXiv preprint arXiv …, 2024

… These corpora often merge and deduplicate multiple years of common crawl scrapes, or supplement additional sources including biomedical text (PubMed), legal text (Freelaw, … 2023) provides a 3 trillion token, 2023 processed split of …


The Revealed Preferences of Pre-authorized Licenses and Their Ethical Implications for Generative Models

VM Suriyakumar, P Menell, D Hadfield-Menell…

This work examines the revealed preferences of creators as reflected in open and quasi-open licensing regimes based on the most commonly used licenses by copyright holders of images in the Creative Commons and copyright holders of code …


The Right Prompts for the Job: Repair Code-Review Defects with Large Language Model

Z Zhao, Z Xu, J Zhu, P Di, Y Yao, X Ma - arXiv preprint arXiv:2312.17485, 2023

Automatic program repair (APR) techniques have the potential to reduce manual efforts in uncovering and repairing program defects during the code review (CR) process. However, the limited accuracy and considerable time costs associated with …


The Right to Scrap Data on the Internet: From the US Case hiQLabs, Inc. v. LinkedIn Corp. to the ChatGPT Scraping Cases: Differences Between US and EU Law [pre …

JTS y López - Global Privacy Law Review, 2024

… a doctor when she was undergoing treatment for a rare genetic condition ended up in the Common Crawl’s archive and LAION’s dataset, and therefore in the GhatGPT datasets (because the Common Crawl is one of the ChatGPT training …


The Rise of Generative Artificial Intelligence: Impact on Societies, Economies and Enterprises

This timely book explores how generative artificial intelligence (GAI) is developing and diffusing, highlighting the diverse impacts this technology is likely to have on economies and societies. It also examines the effects on and the responses of …


The Role and Prospects of Using ChatGPT Artificial Intelligence-Based Chatbots in Healthcare Improvement

NA Mumtaz - … OF INTERNATIONAL SEMINAR AND WORKSHOP ON …

In the age of digital transformation, healthcare is undergoing a revolution through information technology and artificial intelligence (AI). This article explores the role of AI-based chatbots, like ChatGPT, in healthcare enhancement. Utilizing a literature …


The role of large language models in agriculture: harvesting the future with LLM intelligence

TA Shaikh, T Rasool, K Veningston, SM Yaseen - Progress in Artificial Intelligence, 2024

Significant accomplishments in many agricultural applications during the past decade attest to the fast progress and use of deep learning and machine learning methods in agricultural systems. However, these conventional models have a few …


The Science of Data Filtering: Data Curation cannot be Compute Agnostic

S Goyal, P Maini, ZC Lipton, A Raghunathan, JZ Kolter - ICLR 2024 Workshop on …

… In fact, we show that even a model trained on *unfiltered common crawl* obtains higher accuracy than that trained on the LAION dataset post 40 or more repetitions. While past research in neural scaling laws has considered web data to be …


The science of implicit race bias: Evidence from the Implicit Association Test

KN Morehouse, MR Banaji - Daedalus, 2024

Beginning in the mid-1980s, scientific psychology underwent a revolution–the implicit revolution–that led to the development of methods to capture implicit bias: attitudes, stereotypes, and identities that operate without full conscious awareness …


The Self in Action: Narrating Agentic Moments

S Zilberstein, E Ayala-Hurtado, M Sanchez, D Robey - RSF: The Russell Sage …, 2024

This article develops a cultural and contextual approach to studying agency that attends to variation in how people narrate their experiences. Drawing on the large-scale, nationally representative American Voices Project data, the article uses …


The Shape of Word Embeddings: Recognizing Language Phylogenies through Topological Data Analysis

O Draganov, S Skiena - arXiv preprint arXiv:2404.00500, 2024

Word embeddings represent language vocabularies as clouds of $d$-dimensional points. We investigate how information is conveyed by the general shape of these clouds, outside of representing the semantic meaning of each token. Specifically, we …


The Silicone Ceiling: Auditing GPT's Race and Gender Biases in Hiring

L Armstrong, A Liu, S MacNeil, D Metaxa - arXiv preprint arXiv:2405.04412, 2024

Large language models (LLMs) are increasingly being introduced in workplace settings, with the goals of improving efficiency and fairness. However, concerns have arisen regarding these models' potential to reflect or exacerbate social biases …


The SlayQA benchmark of social reasoning: testing gender-inclusive generalization with neopronouns

B Bunzeck, S Zarrieß - Proceedings of the 2nd GenBench Workshop on …, 2024

… C4 is based on CommonCrawl web dumps that were then cleaned, filtered and deduplicated to certain degrees. RedPajamaData-1T and Dolma also contain considerable portions of CommonCrawl enriched with data from diverse sources …


The Social Impact of Generative LLM-Based AI

Y Xie, S Avila - arXiv preprint arXiv:2410.21281, 2024

Liking it or not, ready or not, we are likely to enter a new phase of human history in which Artificial Intelligence (AI) will dominate economic production and social life -- the AI Revolution. Before the actual arrival of the AI Revolution, it is time for us to …


The Sociolinguistic Foundations of Language Modeling

J Grieve, S Bartl, M Fuoli, J Grafmiller, W Huang… - arXiv preprint arXiv …, 2024

In this paper, we introduce a sociolinguistic perspective on language modeling. We claim that large language models are inherently models of varieties of language, and we consider how this insight can inform the development and deployment of …


The Tokenization Problem: Understanding Generative AI's Computational Language Bias

M Asprovska, N Hunter - Ubiquity Proceedings, 2024

… This paper used the Common Crawl corpus as a stand-in due to its wide linguistic representation, but the actual distribution within the pre-training dataset may differ. The discrepancies between these datasets might significantly influence tokenization …


The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classification

MD Bui, K von der Wense - arXiv preprint arXiv:2405.02010, 2024

… 2019) is derived from 393,423 online biographies in English from the Common Crawl corpus, each including the subject’s occupation and gender. The dataset contains 28 occupations, assuming a binary gender classification. Gender identification …


The use of machine learning and deep learning models in detecting depression on social media: A systematic literature review

WA Gadzama, D Gabi, MS Argungu, HU Suru - Personalized Medicine in Psychiatry, 2024

Depression is regarded as one of the world's primary concerns. Recent researchers use artificial intelligence techniques like machine learning and deep learning to identify depressive symptoms automatically. This literature review focuses on using …


The Web Data Commons Schema. org Table Corpora

R Peeters, A Brinkmann, C Bizer - Companion Proceedings of the ACM on Web …, 2024

… This paper presented the WDC Schema.org Table Corpora that we extracted from the Common Crawl. To the best of our knowledge, the corpora are the largest public table corpora that contain tables from many different sources which all share a …


The Web Is Missing an Essential Part of Infrastructure: An Open Web Index A proposal for building an index of the Web that separates the infrastructure part of the …

The web as it currently exists would not be possible without search engines. They are an integral part of the Web and can also be seen as a part of the Web’s infrastructure. Google alone now serves over two trillion search queries per year. 11 …


The wompom

RH Baayen - Corpus Linguistics and Linguistic Theory, 2024

In this essay, I reflect on the past, present, and future of corpus linguistics. My own research has focused primarily on morphology, the mental lexicon, and lexical processing. A consequence of this narrow, myopic focus on just a tiny part of the rich …


The work of art in the age of artificial intelligibility

The emergence of complex deep-learning models capable of producing novel images on a practically innumerable number of subjects and in an equally wide variety of artistic styles is beginning to highlight serious inadequacies in the ethical …


The Zamba2 Suite: Technical Report

P Glorioso, Q Anthony, Y Tokpanov, A Golubeva… - arXiv preprint arXiv …, 2024

In this technical report, we present the Zamba2 series -- a suite of 1.2B, 2.7B, and 7.4B parameter hybrid Mamba2-transformer models that achieve state of the art performance against the leading open-weights models of their class, while …


Themes and sentiments in conversations about food waste on Twitter: Proposal of a framework using neural topic modeling

MW Barbosa, A Gomes - Food Quality and Preference, 2024

… In this database, Twitter data was extracted from CommonCrawl. This method resulted in a collection spanning 80,000 distinct users. Historical snapshots from CommonCrawl, which spanned two years, were also considered to augment the …


Theoretical and Methodological Framework for Studying Texts Produced by Large Language Models

J Milička - arXiv preprint arXiv:2408.16740, 2024

This paper addresses the conceptual, methodological and technical challenges in studying large language models (LLMs) and the texts they produce from a quantitative linguistics perspective. It builds on a theoretical framework that …


Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization

Z Li, Y Wang, B Hooi, Y Cai, N Cheung, N Peng… - arXiv preprint arXiv …, 2024

… We consider the data ratio from the CommonCrawl corpus1 and its intersection with two datasets, aligning with the languages used in the previous … 1http://commoncrawl.org 2https://github.com/csebuetnlp/xl-sum/tree/ master/multilingual_rouge_scoring 3https://github.com/Tiiiger/…


Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling

J Li, HT Ng - arXiv preprint arXiv:2412.14860, 2024

Despite their outstanding capabilities, large language models (LLMs) are prone to hallucination and producing factually incorrect information. This challenge has spurred efforts in attributed text generation, which prompts LLMs to generate content …


Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

SB Tete - arXiv preprint arXiv:2406.11007, 2024

The advent of Large Language Models (LLMs) has revolutionized various applications by providing advanced natural language processing capabilities. However, this innovation introduces new cybersecurity challenges. This paper …


THROUGH THE LENS OF WEB ARCHIVES

MS Weber - The Routledge Companion to Transnational Web …, 2024

Information ecosystems are a central component of modern society, and increasingly a critical component of our social fabric. In the past 50 years, however, these networks of media and other information sources have become incredibly …


TiC-LM: A Multi-Year Benchmark for Continual Pretraining of Language Models

J Li, M Armandpour, SI Mirzadeh, S Mehta, V Shankar… - NeurIPS 2024 Workshop on …

… Figure 1: (Left) Total number of tokens per month in our TiC-CommonCrawl. We draw training sets from a tenth of the data in each month … timesteps, we consider training on a generic web-crawl (ie, Common Crawl) 46 spanning 114 different …


Time for a Pause: Without Effective Public Oversight, AI in Schools Will Do More Harm Than Good.

B Williamson, A Molnar, F Boninger - 2024

Public education is a public and private good essential to democratic civic life. The public must, therefore, be able to provide meaningful direction over schools through transparent democratic governance structures. And yet artificial intelligence (AI) 1 …


Time Machine GPT

F Drinkall, E Rahimikia, JB Pierrehumbert, S Zohren - arXiv preprint arXiv:2404.18543, 2024

… To scale to even larger models, processing the annual Common Crawl datasets is a necessary step, though the dataset has proved problematic due to its scale and lack of … Aside from this, cleaning Common Crawl would also demand significant …


Timely Quality Problem Resolution in Peer-Production Systems: The Impact of Bots, Policy Citations, and Contributor Experience

V Mindel, A Aaltonen, A Rai, L Mathiassen, W Jabr - Information Systems Research, 2024

Although online peer-production systems have proven to be effective in producing high-quality content, their open call for participation makes them susceptible to ongoing quality problems. A key concern is that the problems should be addressed …


TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers

SV Kandala, P Medaranga, A Varshney - arXiv preprint arXiv:2412.15304, 2024

… Specifically, we utilized the Fineweb dataset [39], selecting 9 billion tokens from a collection of over 15 trillion tokens compiled from CommonCrawl dumps since 2013. Additionally, we incorporated the SHL dataset [18], which contains annotated data …


To Code, or Not To Code? Exploring Impact of Code in Pre-training

V Aryabumi, Y Su, R Ma, A Morisot, I Zhang, A Locatelli… - arXiv preprint arXiv …, 2024

Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role …


To store or not: Online cost optimization for running big data jobs on the cloud

X Fu, L Pan, S Liu - Future Generation Computer Systems, 2024

As businesses increasingly rely on cloud-based big data analytics services to drive insights, reducing the cost of storing and analyzing large volumes of data in the cloud has become a major concern. During the execution of big data analysis jobs …


TO WHOM DOES THE WORLD BELONG?

… Indeed, crawls contain enormous amounts of copyrighted information; the Common Crawl alone, a standard repository maintained by a nonprofit and used to train many LLMs, contains most of b-ok.org, a huge repository of pirated ebooks that …


TOFU: A Task of Fictitious Unlearning for LLMs

P Maini, Z Feng, A Schwarzschild, ZC Lipton, JZ Kolter - arXiv preprint arXiv …, 2024

… LLMs trained on Common Crawl data1 may be able to correctly answer factual questions about this person and they may wish to have their data removed from an LLM. In fact, regulations around the Right to be Forgotten that focus on this situation …


Tokenization and Morphology in Multilingual Language Models: A~ Comparative Analysis of mT5 and ByT5

TA Dang, L Raviv, L Galke - arXiv preprint arXiv:2410.11627, 2024

Morphology is a crucial factor for multilingual language modeling as it poses direct challenges for tokenization. Here, we seek to understand how tokenization influences the morphological knowledge encoded in multilingual language models …


TOMGPT: Reliable Text-Only Training Approach for Cost-Effective Multi-modal Large Language Model

Y Chen, Q Wang, S Wu, Y Gao, T Xu, Y Hu - ACM Transactions on Knowledge …, 2024

… For instance, c4 [37] dataset, a colossal and cleaned version of the web crawl corpus from Common Crawl, serves as a suitable option. Since the CLIP text encoder has limitations in processing long text, we subdivided the text in the c4 …


Tonirodriguez at CheckThat! 2024: Is it Possible to Use Zero-Shot Cross-Lingual Methods for Subjectivity Detection in Low-Resources Languages?

A Rodríguez, E Golobardes, J Suau - 2024

… on 2.5TB of filtered CommonCrawl data containing 100 languages. Following the work of XLM and RoBERTa, the XLM-RoBERTa or XLM-R model takes multilingual pretraining one step further by massively upscaling the training data [19]. Using the …


Tooling for big data extraction

R Beilich - HPI Future SOC Lab–Proceedings 2020, 2024

… , provided by Common Crawl, what may causes libraries to crumble and others to rise. … The Common Crawl Foundation created “an open repository of web crawl data that can be … [5] For the Common Crawl dataset this results in three types of …


Topic Modeling for Auditing Purposes in the Banking Sector

A Giaconia, V Chiariello, S Giannuzzi, M Passarotti - 2024

This study explores the application of topic modeling techniques for auditing purposes in the banking sector, focusing on the analysis of reviews of anti-money laundering alerts. We compare three topic modeling algorithms: Latent Dirichlet …


Topic Modelling Swahili Using LDA and Contextualized Embeddings

DW Kahenya, W Mwangi, R Rimiru - 2024 Intelligent Methods, Systems, and …, 2024

… The XLMRoBERTa model, developed by [19] in their study on Unsupervised Cross-lingual Representation Learning at Scale, was trained on 2.5TB of filtered CommonCrawl data containing 100 languages. Flair framework [20] was used to …