EvalQuiz: self-assessment generated through language transformer models

This thesis explores the constraints of self-assessment creation in higher education, focusing on the lack of tools, standardization, and time. We conduct didactic field expert interviews to investigate how teaching evolves towards learning management …


Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languages

JA Meaney, B Alex, W Lamb - arXiv preprint arXiv:2411.05593, 2024

Folktales are a rich resource of knowledge about the society and culture of a civilisation. Digital folklore research aims to use automated techniques to better understand these folktales, and it relies on abstract representations of the textual …


Evaluating and Aligning CodeLLMs on Human Preference

J Yang, J Yang, K Jin, Y Miao, L Zhang, L Yang, Z Cui… - arXiv preprint arXiv …, 2024

… Recall from Common Crawl. A trained fasttext is used to distinguish the code-related text and other common raw text, which is used to recall and clean potential code data and filter out low-quality content using weak model-based classifiers and …


Evaluating and improving lexical language understanding in neural machine translation

Lexical understanding is an inalienable component of the translation process. In order to correctly map the meaning of a linguistic unit to the appropriate target language expression, the meaning of its constituent words has first to be identified …


Evaluating and Improving the Reasoning Abilities of Language Models

This thesis focuses on evaluating and improving the reasoning abilities of Smaller Language Models (SLMs) and Large Language Models (LLMs). It explores SLMs’ performance on complex tasks and their limitations with simpler ones. This thesis …


Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

P Hager, F Jungmann, K Bhagat, I Hubrecht, M Knauer… - medRxiv, 2024

Clinical decision making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from AI solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent …


Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English

F Dawson, Z Mosunmola, S Pocker, RA Dandekar… - arXiv preprint arXiv …, 2024

… Table 1 shows the distribution of languages in Common Crawl, a dataset commonly used for training LLMs. … The Common Crawl data 1 shows that English dominates LLM training data (45%), while Yoruba and Malayalam account for only …


Evaluating Differential Privacy Approaches for Query Obfuscation in Information Retrieval

Protecting the privacy of a user while they interact with an Information Retrieval (IR) system is crucial. This becomes more challenging when the IR system is not cooperative in satisfying the user’s privacy needs. Recent advancements in Natural …


Evaluating embedded semantics for accessibility description of web crawl data

R Navarrete, D Martinez-Mosquera, L Recalde… - AHFE (2023) International …, 2023

… The dataset for analysis is provided by Web Data Commons (WDC), an organization that releases extracted data from Common Crawl (CC), the largest web corpus available to the public. The dataset was released in October 2021 …


Evaluating Embedding Models for Clustering Italian Political News: A Comparative Study of Text-Embedding-3-Large and UmBERTo

… This study compares two embedding models: OpenAI's text-embedding-3-large model and umberto-commoncrawl-cased-v1 (hereafter referred to as UmBERTo) (Parisi et al., 2020). UmBERTo, a BERT-based model (more specifically RoBERTa-based) …


Evaluating English-language morphological awareness assessments

CL Hudson Kam, E Sadlier-Brown, S Clark, C Jang… - First Language, 2024

CLH Kam, E Sadlier-Brown, S Clark, C Jang, CD Epp… - First Language, 2024

… These corpora were selected because they contain language that children are exposed to, and so, provide a more realistic picture of children’s potential knowledge than an adult-language based corpus would (eg the Enron email dataset …


Evaluating Gender, Racial, and Age Biases in Large Language Models: A Comparative Analysis of Occupational and Crime Scenarios

V Mirza, R Kulkarni, A Jadhav - arXiv preprint arXiv:2409.14583, 2024

Recent advancements in Large Language Models(LLMs) have been notable, yet widespread enterprise adoption remains limited due to various constraints. This paper examines bias in LLMs-a crucial issue affecting their usability, reliability, and …


Evaluating Generative Language Models in Japanese Using Question-Answering Templates: A Synthetic Data Generation Approach

In the last five years, transformer-based large language models have not only made tremendous advancements in fluent and accurate text generation, but also show unparalleled performance as zero-or few-shot shot learners of numerous …


Evaluating GPT and BERT models for Protein-Protein interaction identification in biomedical text

H Rehana, NB Çam, M Basmaci, J Zheng, C Jemiyo… - Bioinformatics Advances, 2024

Detecting protein-protein interactions is crucial for understanding genetic mechanisms, disease pathogenesis, and drug design. As biomedical literature continues to grow rapidly, there is an increasing need for automated and accurate …


Evaluating Language Model Vulnerability to Poisoning Attacks in Low-Resource Settings

R Plant, MV Giuffrida, N Pitropakis, D Gkatzia - IEEE/ACM Transactions on Audio …, 2024

Pre-trained language models are a highly effective source of knowledge transfer for natural language processing tasks, as their development represents an investment of resources beyond the reach of most researchers and end users. The widespread …


Evaluating Large Language Model Performance on Haskell

I introduce HaskellEval, a Haskell evaluation benchmark for Large Language Models. HaskellEval’s curation leverages a novel synthetic generation framework, streamlining the process of dataset curation by minimizing manual intervention. The …


Evaluating Large Language Models for Generalization and Robustness via Data Compression

Y Li, Y Guo, F Guerin, C Lin - arXiv preprint arXiv:2402.00861, 2024

Existing methods for evaluating large language models face challenges such as data contamination, sensitivity to prompts, and the high cost of benchmark creation. To address this, we propose a lossless data compression based evaluation …


Evaluating Large Language Models on Academic Literature Understanding and Review: An Empirical Study among Early-stage Scholars

J Wang, H Hu, Z Wang, Y Song, Y Sheng, D He - 2024

The rapid advancement of large language models (LLMs) such as ChatGPT makes LLM-based academic tools possible. However, little research has empirically evaluated how scholars perform different types of academic tasks with LLMs …


Evaluating LLMs on large contexts: a RAG approach on text comprehension

While the latest Large Language Models (LLMs) continue to expand in size and context window capacity, their knowledge base remains constrained by their training corpus. Retrieval Augmented Generation (RAG) offers a solution to this limitation by …


Evaluating Machine Learning and Deep Learning Approaches for Phishing URL Detection: A Systematic Review and Future Directions

PS PREETI - International Journal of Computer Science and …, 2024

Attacks involving phishing URLs have become a significant menace in digital realm, presenting substantial dangers to individuals. In response to this escalating danger, researcher and professionals have formulated a range of anti-phishing strategies …


Evaluating Multilingual Abstractive Dialogue Summarization in Indian Languages using mT5-small & IndicBART

M Sharma, G Goyal, A Gupta, R Rani, A Sharma, A Dev - 2024 IEEE 9th International …, 2024

… The mC4 dataset consists of natural text in 101 languages including Indian languages like Hindi, Mararti, Punjabi and others, that was collected from the public Common Crawl web scrape. Introduced as a multilingual, sequence-to-sequence pre-trained …


Evaluating Natural Monopoly Conditions in the AI Foundation Model Market

The authors of this report examined the economic and production attributes of pre-trained artificial intelligence (AI) foundation models to answer the following questions: Does the market for foundation models have the characteristics of a natural monopoly, and …


Evaluating Prediction-Based Theories of Bilingual Comprehension of Spanish/English Codeswitches.

This dissertation investigates how bilinguals use their two grammars to comprehend written intra-sentential codeswitches. I focus on adjective/noun constructions in Spanish and English where I manipulate the congruence of grammatical word order …


Evaluating Shortest Edit Script Methods for Contextual Lemmatization

O Toporkov, R Agerri - arXiv preprint arXiv:2403.16968, 2024

Modern contextual lemmatizers often rely on automatically induced Shortest Edit Scripts (SES), namely, the number of edit operations to transform a word form into its lemma. In fact, different methods of computing SES have been proposed as an …


Evaluating SQL Understanding in Large Language Models

A Rahaman, A Zheng, M Milani, F Chiang, R Pottinger - arXiv preprint arXiv …, 2024

… Released by OpenAI in late 2022, GPT3.5 consists of 175 billion parameters and is trained on a large corpus including Common Crawl, Wikipedia, and various books and academic texts. It is designed to handle diverse NLP tasks [3]. GPT4. …


Evaluating the Capabilities of GPT-4 in Full-Stack Web Development: A Practical Approach

With the rapid advancements in Artificial Intelligence, leveraging machine learning models for various domains has become increasingly prevalent. This thesis explores the capabilities of OpenAI’s Generative Pre-trained Transformer 4 (GPT-4) within full-stack …


Evaluating the Effectiveness of Machine Translation in Preserving Metaphorical Expressions from Source Texts to Target Languages

Л Матякубова - Лингвоспектр, 2024

Annotation. With the rapid proliferation of Neural Machine Translation (NMT) systems, questions persist about their ability to handle nuanced linguistic phenomena—particularly metaphorical expressions. Metaphors are culturally and contextually embedded …


Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health Support

Z Ma, Y Mei, Y Long, Z Su, KZ Gajos - arXiv preprint arXiv:2402.09260, 2024

LGBTQ+ individuals are increasingly turning to chatbots powered by large language models (LLMs) to meet their mental health needs. However, little research has explored whether these chatbots can adequately and safely provide tailored support …


Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

S Ramprasad, K Krishna, ZC Lipton, BC Wallace - arXiv preprint arXiv:2402.03509, 2024

Recent work has shown that large language models (LLMs) are capable of generating summaries zero-shot (ie, without explicit supervision) that, under human assessment, are often comparable or even preferred to manually composed …


Evaluating the Impact of Advanced LLM Techniques on AI-Lecture Tutors for a Robotics Course

S Kahl, F Löffler, M Maciol, F Ridder, M Schmitz… - arXiv preprint arXiv …, 2024

This study evaluates the performance of Large Language Models (LLMs) as an Artificial Intelligence-based tutor for a university course. In particular, different advanced techniques are utilized, such as prompt engineering, Retrieval-Augmented-Generation …


EVALUATING THE IMPACT OF ARTIFICIAL INTELLIGENCE ON BUSINESS THROUGH SENTIMENT ANALYSIS.

IC Marian - Young Economists Journal/Revista Tinerilor …, 2024

This article investigates the impact of artificial intelligence (AI) on business and economics using the technique of sentiment analysis. This technique uses natural language processing (NLP) to gauge audience attitudes and perceptions. By …


Evaluating the impact of design decisions on passive DNS-based domain rankings

V Le Pochat, S Fernandez, T Van Goethem…

… They recommend that researchers select a random sample of websites among Common Crawl hosts. Ruth et al. [6] compared top lists to Cloudflare traffic data, concluding that the Chrome User Experience Report [11] most accurately represents …


Evaluating the Performance of Topic Modeling Techniques with Human Validation to Support Qualitative Analysis

JD Romero, MA Feijoo-Garcia, G Nanda, B Newell… - Big Data and Cognitive …, 2024

Examining the effectiveness of machine learning techniques in analyzing engineering students’ decision-making processes through topic modeling during simulation-based design tasks is crucial for advancing educational methods and …


Evaluating the Quality of AI-Generated Items for a Certification Exam

AD Mead, C Zhou - Journal of Applied Testing Technology, 2024

OpenAI’s GPT-3 model can write multiple-choice exam items. This paper reviewed the literature on automatic item generation and then described the recent history of OpenAI GPT models and their operation, and then described a methodology for …


Evaluating Transformers and Linguistic Features integration for Author Profiling tasks in Spanish

JA García-Díaz, G Beydoun, R Valencia-García - Data & Knowledge Engineering, 2024

… This model is a training from scratch of RoBERTa, trained on the Spanish texts from mC4, compiled from the public Common Crawl web … This is a multilingual version of RoBERTa, pre-trained with about 2.5 TB of data from the 100 different …


Evaluating Turkish BERT-based Language Models for Effective Customer Feedback Interpretation in CRM

C İşcan, MF Özkara, AE Çelik, A Akbulut - 2024 9th International Conference on …, 2024

… It is a multilingual version of the RoBERTa model which is pre-trained on 2.5TB of filtered CommonCrawl data comprising 100 languages. In this study, we employed a selection of cutting-edge and highly efficient tools that we discovered during our …


Evaluating Word Embedding Methods for Sentiment Analysis

We evaluate the performance of various word embedding models for sentiment prediction. Specifically, we train Word2vec, GloVe, and FastText models on 3.6 million Amazon product reviews and assess their predictive performance compared …


Evaluation and Adaptation of Neural Language Models for Under-Resourced Languages

Abstract Language models are now commonly used by researchers, industry, and anyone interested. However, language models of all sizes and types are primarily developed for the English language while efforts on other languages lag behind …


Evaluation of AI-generated Responses by Different Artificial Intelligence Chatbots to the Clinical Decision-Making Case-Based Questions in Oral and Maxillofacial …

A Azadi, F Gorjinejad, H Mohammad-Rahimi, R Tabrizi… - Oral Surgery, Oral Medicine …, 2024

Objectives This study aims to evaluate the correctness of the generated answers by Google Bard, GPT-3.5, GPT-4, Claude-Instant, and Bing chatbots to decision-making clinical questions in the oral and maxillofacial surgery (OMFS) area. Study Design A …


Evaluation of Geographical Distortions in Language Models: A Crucial Step Towards Equitable Representations

R Decoupes, R Interdonato, M Roche, M Teisseire… - arXiv preprint arXiv …, 2024

Language models now constitute essential tools for improving efficiency for many professional tasks such as writing, coding, or learning. For this reason, it is imperative to identify inherent biases. In the field of Natural Language Processing …


Evaluation of word embedding models used for diachronic semantic change analysis

Y Maslennikova, V Bochkarev - Journal of Physics: Conference Series, 2024

… As a reference word2vec model, we used pre-trained word vectors that were trained on English versions of Common Crawl dataset using fastText library [14]. fastText is a library for efficient learning of word representations and sentence …


Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization

NZ Kayalı, Sİ Omurca - Sakarya University Journal of Computer and …, 2024

… A multilingual variation of the T5 model, which is trained on common crawl-based data and covers 101 different languages, is called mT5 [27]. Our studies used the mT5 model as the second language model after the BERTurk model. …


Evaluations Using Wikipedia without Data Contamination: From Trusting Articles to Trusting Edit Processes

… Even when Wikipedia is not explicitly including in training data, it is a substantial part of Common Crawl1, a massive web dataset frequently used for model training. … 1https://commoncrawl.github.io/cc-crawl-statistics/plots/domains …


Event extraction using a Large Language Model

I Kovac, T Madzarac - Text Analysis and Retrieval 2024 Course Project …

Event extraction is an important problem of Natural Language Processing, and its goal is to extract events that have happened in the text alongside with its attributes. There are a lot of models specified for this task, but in this paper, we decided to test …


Evidence for a digital divide? Measuring DNS dependencies in the context of the indigenous population of Australia

R Holz, N Nazemi, O Tavallaie, AY Zomaya

We recently presented a work-in-progress paper at the Workshop on Transparency, Accountability and User Control for a Responsible Internet (TAURIN 2023). Our paper investigates the relationship between the digital divide, Internet transparency …


EVLM: An Efficient Vision-Language Model for Visual Understanding

K Chen, D Shen, H Zhong, H Zhong, K Xia, D Xu… - arXiv preprint arXiv …, 2024

In the field of multi-modal language models, the majority of methods are built on an architecture similar to LLaVA. These models use a single-layer ViT feature as a visual prompt, directly feeding it into the language models alongside textual tokens …


Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models

Z Chen, L Xu, H Zheng, L Chen, A Tolba, L Zhao, K Yu… - Computers, Materials and …, 2024

Since the 1950s, when the Turing Test was introduced, there has been notable progress in machine language intelligence. Language modeling, crucial for AI development, has evolved from statistical to neural models over the last two decades …


Evolution of Generative AI

R Gogula, R Teja Swarup, PRK Reddy, PP Kumar - … the Effects of Generative AI on …, 2024

… For language models such as GPT-3, the Common Crawl dataset used to train the model has an original size of 45 terabytes before pre-processing. After applying the filtering method, the data is then reduced to a size of 570 gigabytes. Big data …


EXAMINING ACCURACY HETEROGENEITIES IN CLASSIFICATION OF MULTILINGUAL AI-GENERATED TEXT

Tools for detection of AI-generated texts are used globally, however, the nature of the apparent accuracy disparities between languages must be further observed. This paper aims to examine the nature of these differences through testing OpenAI’s …


Experience of Training a 1.7 B-Parameter LLaMa Model From Scratch

MQ Li, B Fung, SC Huang - arXiv preprint arXiv:2412.13335, 2024

Pretraining large language models is a complex endeavor influenced by multiple factors, including model architecture, data quality, training continuity, and hardware constraints. In this paper, we share insights gained from the experience of training …


Experimental Evaluation of Possible Feature Combinations for the Detection of Fraudulent Online Shops

A Janavičiūtė, A Liutkevičius, G Dabužinskas… - Applied Sciences, 2024

Online shopping has become a common and popular form of shopping, so online attackers try to extract money from customers by creating online shops whose purpose is to compel the buyer to disclose credit card details or to pay money for …


Expert Phishing Detection System

A Alsarhan, I Al-Aiash, D Al-Fraihat, M Aljaidi… - 2024 IEEE International …, 2024

… and Common Crawl datasets. The highest accuracy reported in this work was 83.00%, achieved with the RF classifier. In the study conducted by [16], the authors employed Machine Learning (ML) and Heuristic methods on three datasets—PhishTank …


Expert-Like Systems Engineering Artifacts? Insights from an Empirical

M Husain, P Wach, TG Topcu - The Proceedings of the 2024 Conference on Systems …

The old saga of complex system development continues as the vast majority of government and industry programs continue to result in cost and schedule overruns [1–7]. While numerous researchers and government reports attribute this trend to the ever-increasing …


Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda

J Schneider - arXiv preprint arXiv:2404.09554, 2024

Generative AI (GenAI) marked a shift from AI being able to recognize to AI being able to generate solutions for a wide variety of tasks. As the generated solutions and applications become increasingly more complex and multi-faceted, novel needs …


Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-resource Language Translation

S Ranathungaa, S Nayak, STC Huang, Y Mao, T Su…

Neural Machine Translation (NMT) systems built on multilingual sequenceto-sequence Language Models (msLMs) fail to deliver expected results when the amount of parallel data for a language, as well as the language’s representation in the model …


Exploration of Content-Based Cross-Domain Podcast Recommender Systems

Podcasts have become a popular medium in the last decade. The huge amount of data available motivates research on podcast recommender systems to make this data accessible to users. Since interaction-based datasets for podcasts are only …


Exploring AI-driven approaches for unstructured document analysis and future horizons

SV Mahadevkar, S Patil, K Kotecha, LW Soong… - Journal of Big Data, 2024

In the current industrial landscape, a significant number of sectors are grappling with the challenges posed by unstructured data, which incurs financial losses amounting to millions annually. If harnessed effectively, this data has the potential to …


Exploring Automatic Text Simplification for Lithuanian

J Mandravickaitė, E Rimkiene, D Kalinauskaitė… - Proceedings of the 20th …, 2024

… To train mT5, the authors introduced a multilingual variant of the C4 dataset called mC4, which comprises textual data in 101 languages drawn from the public Common Crawl web scrape. It makes mT5 model particularly suitable for languages …


Exploring Emergent Phenomena in AI: A Pantheistic Approach to the Underlying Source of Information

Emergent phenomena in artificial intelligence (AI) systems, where creative, novel, and unexpected outputs arise, have long intrigued researchers. While these behaviors are typically seen as products of complex computation, this paper …


Exploring Fine-tuned Generative Models for Keyphrase Selection: A Case Study for Russian

A Glazkova, D Morozov - arXiv preprint arXiv:2409.10640, 2024

Keyphrase selection plays a pivotal role within the domain of scholarly texts, facilitating efficient information retrieval, summarization, and indexing. In this work, we explored how to apply fine-tuned generative transformer-based models to the …


Exploring Intermediate Training for Chinese Offensive Language Detection

Offensive language detection (OLD) has become a crucial area of research in recent years. Despite growing interest, the majority of current studies have focused on English, leaving many other languages, like Chinese, relatively unexplored. In this …


Exploring Korean Medicine professions' perspectives on the applicability of ChatGPT in facial palsy practice: A web-based survey

JS Lee, SA Kim, T Kim, S Lee, TH Kim, JW Kang - European Journal of Integrative …, 2024

Introduction Since November 2022, when Chat generative pre-trained transformer (ChatGPT) was released, studies have been conducted to explore its potential use in various text-based areas. In the field of Korean Medicine (KM), studies evaluating the …


Exploring Language Models and Question Answering in Biomedical and Arabic Domains

Despite the success of the Transformer model and its variations (eg, BERT, ALBERT, ELECTRA, T5) in addressing NLP tasks, similar success is not achieved when these models are applied to specific domains (eg, biomedical) and limited-resources …


Exploring large language models for microstructure evolution in materials

P Satpute, S Tiwari, M Gupta, S Ghosh - Materials Today Communications, 2024

… These models are based on massive machine learning (ML) architectures (specifically transformer neural networks [3], [4]) pre-trained on a massive corpus of text data from diverse internet sources like Common Crawl [5] and Wikipedia [6], each …


Exploring LLMs' Capabilities for Error Detection in Dutch L1 and L2 Writing Products

J Kruijsbergen, S Van Geertruyen, V Hoste… - Computational Linguistics in …, 2024

This research examines the capabilities of Large Language Models for writing error detection, which can be seen as a first step towards automated writing support. Our work focuses on Dutch writing error detection, targeting two envisaged end-users …


Exploring Low-Resource Machine Translation: Case Study of Lao-Vietnamese Translation

QD Tran - 2024 International Conference on Multimedia Analysis …, 2024

Despite ongoing research efforts in recent years, machine translation in low-resource settings remains challenging. This paper explores recent methods to enhance low-resource machine translation (LoResMT), especially for both-side low-resource language …


Exploring Machine Learning for Malware Detection With Feature Selection, Explainable AI, and Generative Adversarial Networks

This research focuses on the detection of malware in sample datasets using machine learning algorithms. As malware becomes more advanced against detection in the cybersecurity sector, researchers have looked for other methods to …


Exploring Multilingual Embeddings for Italian Semantic Search: A Pretrained and Fine-tuned Approach

This thesis explores the application of multilingual embedding models to the semantic search for the Italian language, a critical step toward integrating these technologies into Retrieval-Augmented Generation (RAG) frameworks. The work …


Exploring Recent NLP Advances for Tamil: Word Vectors and Hybrid Deep Learning Architectures

A Aravinthan, C Eugene - International Journal on Advances in ICT for Emerging …, 2024

The advancements of deep learning methods and the availability of large corpora and data sets have led to an exponential increase in the performance of Natural Language Processing (NLP) methods resulting in successful NLP applications for …


Exploring Scaling Laws for Local SGD in Large Language Model Training

Q He, X Zhuang, Z Wu - arXiv preprint arXiv:2409.13198, 2024

This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices. Through extensive experiments, we show that local SGD achieves competitive results …


Exploring the Abilities of Large Language Models to Solve Proportional Analogies via Knowledge-Enhanced Prompting

T Wijesiriwardene, R Wickramarachchi, S Vennam… - arXiv preprint arXiv …, 2024

Making analogies is fundamental to cognition. Proportional analogies, which consist of four terms, are often used to assess linguistic and cognitive abilities. For instance, completing analogies like "Oxygen is to Gas as is to " requires identifying the …


Exploring the competence of ChatGPT for customer and patient service management

A Haleem, M Javaid, RP Singh - Intelligent Pharmacy, 2024

The modern language generation model ChatGPT, created by Open Artificial Intelligence (AI), is recognised for its capacity to comprehend context and produce pertinent content. This model is built on the transformer architecture, which enables …


Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models

AMK Mamaghan, S Papa, KH Johansson, S Bauer… - arXiv preprint arXiv …, 2024

Object-centric (OC) representations, which represent the state of a visual scene by modeling it as a composition of objects, have the potential to be used in various downstream tasks to achieve systematic compositional generalization and facilitate …


Exploring the Potential of Emerging Digitainability—GPT Reasoning in Energy Management of Kindergartens

N Jurišević, D Gordić, D Nikolić, A Nešović, R Kowalik - Buildings, 2024

One of the barriers to the rapid transition of societies toward a more sustainable future is a scarcity of field experts. Members of scientific and professional communities believe that this obstacle could be overcome by supplementing the …


Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning

Y Wang, W Chen, X Han, X Lin, H Zhao, Y Liu, B Zhai… - arXiv preprint arXiv …, 2024

Strong Artificial Intelligence (Strong AI) or Artificial General Intelligence (AGI) with abstract reasoning ability is the goal of next-generation AI. Recent advancements in Large Language Models (LLMs), along with the emerging field of Multimodal Large …


Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties

E Artemova, V Blaschke, B Plank - arXiv preprint arXiv:2402.02078, 2024

Mainstream cross-lingual task-oriented dialogue (ToD) systems leverage the transfer learning paradigm by training a joint model for intent recognition and slot-filling in English and applying it, zero-shot, to other languages. We address a gap in prior …


Exploring the Role of ChatGPT in Cardiology: A Systematic Review of the Current Literature

A Sharma, T Medapalli, M Alexandrou, E Brilakis… - Cureus, 2024

Chat Generative Pre-Trained Transformer (ChatGPT) is a chatbot based on a large language model that has gained public interest since its release in November 2022. This systematic review examines the current literature on the potential applications …


Exploring the role of ChatGPT in higher education: opportunities, challenges and ethical considerations

A Zeb, R Ullah, R Karim - The International Journal of Information and Learning …, 2024

Purpose This paper aims to examine the opportunities and challenges of using ChatGPT in higher education. Furthermore, it is also discuss the potential risks and plunders of these tools. Design/methodology/approach The paper discuss the use of …


Exploring transformer models in the sentiment analysis task for the under-resource Bengali language

MN Hoque, U Salma, MJ Uddin, MM Ahamad, S Aktar - Natural Language Processing …, 2024

In the sentiment analysis (SA) task, we can obtain a positive or negative-typed comment or feedback from an online user or a customer about any object, such as a movie, drama, food, and others. This user’s sentiment may positively impact various …


Exploring very low-resource translation with llms: The university of edinburgh's submission to americasnlp 2024 translation task

V Iyer, B Malik, W Zhu, P Stepachev, P Chen… - Proceedings of the 4th …, 2024

This paper describes the University of Edinburgh’s submission to the AmericasNLP 2024 shared task on the translation of Spanish into 11 indigenous American languages. We explore the ability of multilingual Large Language Models (LLMs) to …


EXPRESS: AI-Human Hybrids for Marketing Research: Leveraging LLMs as Collaborators

N Arora, I Chakraborty, Y Nishimura - Journal of Marketing, 2024

The authors’ central premise is that a human-LLM hybrid approach leads to efficiency and effectiveness gains in the marketing research process. In qualitative research, they show that LLMs can assist in both data generation and analysis; …


EXPRESS: Malay Lexicon Project 3: The Impact of Orthographic-Semantic Consistency on Lexical Decision Latencies

M Maziyah Mohamed, D Jared - Quarterly Journal of Experimental Psychology, 2024

Theories of word processing propose that readers are sensitive to statistical co-occurrences between spelling and meaning. Orthographic-Semantic Consistency (OSC) measures provide a continuous estimate of the statistical regularities between …


Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation

SM Sani, P Sadeghi, TT Vu, Y Yaghoobzadeh… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have made great progress in classification and text generation tasks. However, they are mainly trained on English data and often struggle with low-resource languages. In this study, we explore adding a new …


Extending the Comparative Argumentative Machine: Multilingualism and Stance Detection

I Nikishina, A Bondarenko, S Zaczek, OL Haag…

The comparative argumentative machine CAM can retrieve arguments that answer comparative questions—questions that ask which of several to-be-compared options should be favored in some scenario. In this paper, we describe how we equipped …


Extending Translate-Train for ColBERT-X to African Language CLIR

E Yang, DJ Lawrie, P McNamee, J Mayfield - arXiv preprint arXiv:2404.08134, 2024

… We used Common Crawl documents in Afriberta Corpus [6] for the four African languages and collected additional English Common Crawl documents to match the genre. We fine-tune the model for 200,000 update steps using a learning rate of 1×10-5 …


External Attack-Surface of Modern Organizations

N Gelernter, H Schulmann, M Waidner - Proceedings of the 19th ACM Asia …, 2024

Navigating the maze of contemporary organizational attack surfaces is paramount in fortifying our defenses against the relentless tide of cyber incidents. However, existing network reconnaissance and security measurements, which enumerate IP …


Extracting emotion from resource poor language through transfer learning

A Majeed, U Imtiaz, MA Nseem, M Aleem, W Shahzad… - Multimedia Tools and …, 2024

… Common Crawl materials in over 100 languages are used to train the algorithm. XLM-Roberta has seen a significant increase in training data since its first release. Several cross-language benchmarks show that XLM-R performs the best. …


Extracting intersectional stereotypes from embeddings: Developing and validating the Flexible Intersectional Stereotype Extraction procedure

TES Charlesworth, K Ghate, A Caliskan, MR Banaji - PNAS Nexus, 2024

… text corpora, ranging from static GloVe embeddings trained on 840 billion words from the Common Crawl to contextualized BERT embeddings trained on a combination of Wikipedia and Common Crawl text. Additionally, in supplemental …


Extracting knowledge from polymorphic textual data using graph-based machine learning and transformer-based deep learning techniques

The aim of this thesis is to develop novel approaches that extract meaningful knowledge from polymorphic (unstructured and structured) textual data. To achieve this, we rely on recent advancements from the areas of Machine Learning (ML) …


Extracting Large-Scale Multimodal Datasets From Web Archives

… In this research, Common Crawl's WARC files are utilized to create a largescale multimodal dataset of text-image pairs. Two Python scripts, formed the core pipeline for extracting data that could be used for training a model to recognize alt tags and …


Factuality of Large Language Models in the Year 2024

Y Wang, M Wang, MA Manzoor, G Georgiev, RJ Das… - arXiv preprint arXiv …, 2024

Large language models (LLMs), especially when instruction-tuned for chat, have become part of our daily lives, freeing people from the process of searching, extracting, and integrating information from multiple sources by offering a …


Factuality of Large Language Models: A Survey

Y Wang, M Wang, MA Manzoor, F Liu, G Georgiev… - Proceedings of the 2024 …, 2024


Fair Compensation for Copyrighted Data Used in AI training

… LAION sources its data from Common Crawl,106 a nonprofit organization that collects data from the web, creating extensive archives and datasets. Despite … Common Crawl is a nonprofit organization systematically exploring the web …


FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication

E Slyman, S Lee, S Cohen, K Kafle - arXiv preprint arXiv:2404.16123, 2024

Recent dataset deduplication techniques have demonstrated that content-aware dataset pruning can dramatically reduce the cost of training Vision-Language Pretrained (VLP) models without significant performance losses compared to …


Fairness and Bias in Multimodal AI: A Survey

T Adewumi, L Alkhaled, N Gurung, G van Boven… - arXiv preprint arXiv …, 2024

The importance of addressing fairness and bias in artificial intelligence (AI) systems cannot be over-emphasized. Mainstream media has been awashed with news of incidents around stereotypes and bias in many of these systems in recent years. In …


Faithfulness of Natural Language Generation

P Schmidtová - Proceedings of the 20th Workshop of Young …, 2024

… With works examining the presence of datasets in CommonCrawl (Li et al., 2024), we cannot even be entirely sure that open-weight models … • Data contamination: to what extent should we examine and worry about dialogue datasets being contained …


Fake news detection in low-resource languages: A novel hybrid summarization approach

J Alghamdi, Y Lin, S Luo - Knowledge-Based Systems, 2024

… Pre-training encompasses a vast dataset exceeding 2TB of CommonCrawl data. The overarching concept involves mapping any given language to a language-agnostic vector space, ensuring that inputs in various languages converge to corresponding …


Fake News Detection Revisited: An Extensive Review of Theoretical Frameworks, Dataset Assessments, Model Constraints, and Forward-Looking Research Agendas …

S Harris, HJ Hadi, N Ahmad, MA Alshara - 2024

The emergence and acceptance of digital technology have caused information pollution and an infodemic on Online Social Networks (OSNs), blogs, and online websites. The malicious broadcast of illegal, objectionable and misleading content …


Falcon Mamba: The First Competitive Attention-free 7B Language Model

J Zuo, M Velikanov, DE Rhaiem, I Chahed, Y Belkada… - arXiv preprint arXiv …, 2024

In this technical report, we present Falcon Mamba 7B, a new base large language model based on the novel Mamba architecture. Falcon Mamba 7B is trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure Mamba-based model …


Falcon-UI: Understanding GUI Before Following User Instructions

H Shen, C Liu, G Li, X Wang, Y Zhou, C Ma, X Ji - arXiv preprint arXiv:2412.09362, 2024

… After downloading the raw data from Common Crawl, we filter out duplicates, unreachable URLs, and non-English, NSFW websites based on the main content of each page. We apply the puppeteer library2 to simulate a browser environment. To …


Falcon2-11B Technical Report

Q Malartic, NR Chowdhury, R Cojocaru, M Farooq… - arXiv preprint arXiv …, 2024

We introduce Falcon2-11B, a foundation model trained on over five trillion tokens, and its multimodal counterpart, Falcon2-11B-vlm, which is a vision-to-text model. We report our findings during the training of the Falcon2-11B which follows a multi-stage …


False memories from nowhere: 2 humans falsely recognize words that are not attested in their vocabulary 3

D Gatti, M Petilli, M Marchetti, T Vecchi, G Mazzoni…

Views and opinions expressed are however those of the author (s) only and do not 37 necessarily reflect those of the European Union or the European Research Council 38 Executive Agency. Neither the European Union nor the granting authority …


Fast and Lightweight Distributed Suffix Array Construction--First Results

M Haag, F Kurpicz, P Sanders, M Schimek - arXiv preprint arXiv:2412.10160, 2024

… CommonCrawl (CC). This input consists of websites crawled by the Common Crawl Project. We use the WET files, which contain only the … Furthermore, we removed the meta information added by the Commoncrawl corpus. We used the …


FastDraft: How to Train Your Draft

O Zafrir, I Margulis, D Shteyman, G Boudoukh - arXiv preprint arXiv:2411.11055, 2024

Speculative Decoding has gained popularity as an effective technique for accelerating the auto-regressive inference process of Large Language Models (LLMs). However, Speculative Decoding entirely relies on the availability of efficient draft …


Feasibility of Big Data Analytics in Disaster Psychiatry: Impact of Seoul Itaewon Tragedy on Sentiment Distribution on Twitter

J Kim, M Ki, J Yang, C Na, J Kim, C Han - Social Science & Medicine, 2024

Numerous studies have highlighted the significant impact of disasters on mental health, often leading to psychiatric disorders among affected individuals. Timely identification of disaster-related mental health problems is crucial to prevent long-term …


Feature-augmented model for multilingual discourse relation classification

E Metheniti, C Braud, P Muller - Proceedings of the 5th Workshop on Computational …, 2024

Discourse relation classification within a multilingual, cross-framework setting is a challenging task, and the best-performing systems so far have relied on monolingual and mono-framework approaches. In this paper, we introduce transformer-based …


Federated learning for privacy-preserving depression detection with multilingual language models in social media posts

SS Khalil, NS Tawfik, M Spruit - Patterns, 2024

The incidences of mental health illnesses, such as suicidal ideation and depression, are increasing, which highlights the urgent need for early detection methods. There is a growing interest in using natural language processing (NLP) models to analyze …


Few shot clinical entity recognition in three languages: Masked language models outperform LLM prompting

M Naguib, X Tannier, A Névéol - arXiv preprint arXiv:2402.12801, 2024

Large Language Models are becoming the go-to solution for many natural language processing tasks, including in specialized domains where their few-shot capacities are expected to yield high performance in low-resource settings. Herein, we aim to …


FicSim: An Ethically Constructed Dataset for Long-Context Semantic Similarity Comparison within Fiction

… We describe our processes for selecting text not included in CommonCrawl scrapes, for obtaining and maintaining author consent for the use of their works, and for constructing pairwise similarity measurements corresponding to different facets of …


Final Thoughts: Digital Humanities Looking at Generative AI

M Aguiar, S Araújo - Digital Humanities Looking at the World: Exploring …, 2024

In this chapter, we examine generative artificial intelligence in the context of Digital Humanities. We commence by providing a concise overview of this technology and its prevalent models. Following that, we provide a brief survey of the existing …


Final Words

M McTear, M Ashurkina - Transforming Conversational AI: Exploring the Power …, 2024

Conversational AI is a dynamic and fast moving field, and a lot has happened in the six months or so since we began writing this book. We have tried to ensure that what we have covered in the preceding chapters provides a sufficiently general and …


Findings of the WMT24 general machine translation shared task: the LLM era is here but mt is not solved yet

T Kocmi, E Avramidis, R Bawden, O Bojar… - Proceedings of the Ninth …, 2024

This overview paper presents the results of the General Machine Translation Task organised as part of the 2024 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for …


Fine grain emotion analysis in Spanish using linguistic features and transformers

A Salmerón-Ríos, JA García-Díaz, R Pan… - PeerJ Computer Science, 2024

Mental health issues are a global concern, with a particular focus on the rise of depression. Depression affects millions of people worldwide and is a leading cause of suicide, particularly among young people. Recent surveys indicate an increase in …


Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations

K VM, H Warrier, Y Gupta - arXiv preprint arXiv:2404.10779, 2024

… The datasets are massive text corpus like Common crawl, The Pile and code repositories from Github. Further the model can be fine tuned for specific tasks using specialised datasets. One such series of datasets are called Instruction datasets like …


Fine Tuning LLMs for Low Resource Languages

S Joshi, MS Khan, A Dafe, K Singh, V Zope, T Jhamtani - 2024 5th International …, 2024

Large Language Models (LLMs) hold immense potential, but their data hunger can limit its performance in processing languages with limited resources. This research study explores the techniques for fine-tuning LLMs specifically for low-resource …


Fine tuning the large language pegasus model for dialogue summarization

Sarthak, V Rishiwal, P Yadav, M Yadav, S Gangwar… - International Journal of …, 2024

Dialogue summarization, a subset of task-oriented Natural Language Processing (NLP), faces challenges in providing concise and informative summaries of conversational data, which is crucial for various real-world applications. This study optimizes the …


Fine-Tuned Large Language Models for Symptom Recognition from Spanish Clinical Text

MA Shaaban, A Akkasi, A Khan, M Komeili, M Yaqub - arXiv preprint arXiv …, 2024

… XLM-RL (561M parameters) and XLM-RB (279M parameters) are multilingual masked language models, trained on a filtered CommonCrawl dataset of over two terabytes. We leverage BBS and BBES both pretrained on Spanish clinical data …


Fine-Tuned Understanding: Enhancing Social Bot Detection with Transformer-based Classification

A Sallah, S Agoujil, MA Wani, M Hammad… - IEEE Access, 2024

… fined common crawl data, XLM-RoBERTa is purposefully crafted as a multilingual version of RoBERTa. The acronym "XLM" denotes its capabilities in crosslingual language modeling; 3… It is trained on 2.5 TB of freshly curated, clean …


Fine-tuning an open source chatbot to translate code from Python to Java using Qlora: translating for more energy efficient code

In recent years large language models (LLM) based tools have become more commonplace with ChatGPT being the precursor to many competing products such as Google Gemini. While the training and finetuning of LLMs has been mostly …


FINE-TUNING BERT, DISTILBERT, XLM-ROBERTA AND UKR-ROBERTA MODELS FOR SENTIMENT ANALYSIS OF UKRAINIAN LANGUAGE REVIEWS

… It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. The architecture of the XLMRoBERTa base model consists of 12 layers, 12 attention heads with 768 dimensions, and a feed-forward network [19]. This model is trained …


Fine-tuning BERT-based Models for Negative Content Identification on Indonesian Tweets

AF Hidayatullah, K Kalinaki, MM Aslam, RY Zakari…

Social media platforms like Twitter have become substantial sources of user-generated content, enabling people to easily express their emotions and opinions. However, this freedom has increased the spread of harmful content, such as abusive language …


Fine-tuning foundational models to code diagnoses from veterinary health records

MR Boguslav, A Kiehl, D Kott, GJ Strecker, T Webb… - arXiv preprint arXiv …, 2024

Veterinary medical records represent a large data resource for application to veterinary and One Health clinical research efforts. Use of the data is limited by interoperability challenges including inconsistent data formats and data siloing …


Fine-tuning Large Language Models for Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection

F Xiong, T Markchom, Z Zheng, S Jung, V Ojha… - arXiv preprint arXiv …, 2024

SemEval-2024 Task 8 introduces the challenge of identifying machine-generated texts from diverse Large Language Models (LLMs) in various languages and domains. The task comprises three subtasks: binary classification in monolingual …


Fine-tuning large language models: from accuracy enhancement to bias mitigation

As the frontier of artificial intelligence (AI) continues to expand, Large Language Models (LLMs) have emerged as pivotal tools across a broad spectrum of applications. These sophisticated AI systems, powered by vast amounts of data and …


FINETUNING COMMERCIAL LARGE LANGUAGE MODELS WITH LORA FOR ENHANCED ITALIAN LANGUAGE UNDERSTANDING

J Hartsuiker, P Torroni, AE Ziri, DF Alise, F Ruggeri

The world of Artificial intelligence is booming, especially the fields of natural language processing (NLP) and Computer Vision (CV). Recently the chatGPT models from OpenAI have caused another shock-wave. This was a clear sign to the …


Fineweb-Edu-Ar: Machine-translated Corpus to Support Arabic Small Language Models

S Alrashed, D Khizbullin, DR Pugh - arXiv preprint arXiv:2411.06402, 2024

… In general, we see two ways to obtain the educational-grade Arabic language dataset: (1) translating the English FineWeb-Edu, and (2) adapting the FineWeb-Edu pipeline for Arabic with subsequent re-running it on CommonCrawl. One notable …


FineWeb-zhtw: Scalable Curation of Traditional Chinese Text Data from the Web

CW Lin, WH Hsieh, KX Guan, CJ Hsu, CC Kuo, CL Lai… - arXiv preprint arXiv …, 2024

… We have implemented a comprehensive pipeline to process Common Crawl data, utilizing various filtering techniques at each stage. The process begins with extracting documents from WARC files. Then, filters mentioned in the sections above …


Finite State Automata on Multi-Word Units for Efficient Text-Mining

Text mining is crucial for analyzing unstructured and semi-structured textual documents. This paper introduces a fast and precise text mining method based on a finite automaton to extract knowledge domains. Unlike simple words, multi-word …


FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

G Bhatia, EMB Nagoudi, H Cavusoglu… - arXiv preprint arXiv …, 2024

We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific …


Five questions and answers about artificial intelligence

A Prieto, B Prieto - arXiv preprint arXiv:2409.15903, 2024

Rapid advances in Artificial Intelligence (AI) are generating much controversy in society, often without scientific basis. As occurred the development of other emerging technologies, such as the introduction of electricity in the early 20th century …


For Perception Tasks: The Cost of LLM Pretraining by Next-Token Prediction Outweigh its Benefits

R Balestriero, H Huang - NeurIPS 2024 Workshop: Self-Supervised Learning …

We question the usefulness of next-token prediction pretraining onto Large Language Models (LLMs)' ability to solve perception tasks, eg, sentiment analysis, spam detection, or toxicity detection. In fact, while companies spend tremendous …


Foregrounding Artist Opinions: A Survey Study on Transparency, Ownership, and Fairness in AI Generative Art

J Lovato, J Zimmerman, I Smith, P Dodds, J Karson - arXiv preprint arXiv:2401.15497, 2024

Generative Artificial Intelligence (AI) tools are used to create art-like outputs and aid in the creative process. While these tools have potential benefits for artists, they also have the potential to harm the art workforce and infringe upon artistic and intellectual …


Foreign Influence by Authoritarian Governments: Introducing New Data and Evidence

J Springman, FS Adiguzel, MV Chaparro, ZR Lin… - 2024

The third wave of democratization was accompanied by a spectacular decline in the international influence of authoritarian governments. Culminating with the collapse of the Soviet Union, the ascendance of advanced democracies resulted in pressure …


Formal Mathematical Reasoning: A New Frontier in AI

K Yang, G Poesia, J He, W Li, K Lauter, S Chaudhuri… - arXiv preprint arXiv …, 2024

AI for Mathematics (AI4Math) is not only intriguing intellectually but also crucial for AI-driven discovery in science, engineering, and beyond. Extensive efforts on AI4Math have mirrored techniques in NLP, in particular, training large language models on …


Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Family

R Santos, J Rodrigues, L Gomes, J Silva, A Branco… - arXiv preprint arXiv …, 2024

… The OSCAR subset for Portuguese we use is based on November/December 2022 version of Common Crawl, which is an automatic crawl from the web. Despite being a crawl, the final dataset is of relatively good quality due the filtering performed …


Foundation Models for Autonomous Robots in Unstructured Environments

H Naderi, A Shojaei - arXiv preprint arXiv:2407.14296, 2024

Automating activities through robots in unstructured environments, such as construction sites, has been a long-standing desire. However, the high degree of unpredictable events in these settings has resulted in far less adoption compared to …


Foundational Large Language Models for Materials Research

V Mishra, S Singh, D Ahlawat, M Zaki, V Bihani… - arXiv preprint arXiv …, 2024

Materials discovery and development are critical for addressing global challenges. Yet, the exponential growth in materials science literature comprising vast amounts of textual data has created significant bottlenecks in knowledge extraction, synthesis …


Foundational Segmentation Models and Clinical Data Mining Enable Accurate Computer Vision for Lung Cancer

NC Swinburne, CB Jackson, AM Pagano, JN Stember… - Journal of Imaging …, 2024

This study aims to assess the effectiveness of integrating Segment Anything Model (SAM) and its variant MedSAM into the automated mining, object detection, and segmentation (MODS) methodology for developing robust lung cancer detection and …


FOX-1 TECHNICAL REPORT

Z Hu, J Zhang, R Pan, Z Xu, S Avestimehr, C He…

… We selected the Common Crawl (CC) split from all four datasets, which occupies almost 5T of storage. The SlimPajama-CC ranges from 2019 to … half of the Common Crawl collection data to complete the first stage of pre-training. By initially …


Frequently asked questions on erectile dysfunction: evaluating artificial intelligence answers with expert mentorship

M Baturu, M Solakhan, TG Kazaz, O Bayrak - International Journal of Impotence …, 2024

The present study assessed the accuracy of artificiaI intelligence-generated responses to frequently asked questions on erectile dysfunction. A cross-sectional analysis involved 56 erectile dysfunction-related questions searched on Google …


From Abstract Syntax to Natural Language Addressing Natural Language Generation Challenges in Arabic Using GFWordnet as Lexical Resources.

This thesis explores the development and evaluation of Arabic natural language generation using the Grammatical Framework (GF) within GFPedia. GFPedia is a framework that generates multilingual content using predefined abstract syntax trees …


From archived web corpus to readable data for history research

S Gebeil, J Thièvre - The Routledge Companion to Transnational Web …, 2024

… most referenced and most widely used in the community, notably by Common Crawl and ArchiveIt. The WAT format is specifically designed for … In alignment with the practices of Common Crawl and Archive-It (Lohndorf, 2023), our project adopts a …


From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models

W Messner, T Greene, J Matalone - arXiv preprint arXiv:2312.17256, 2023

Large language models (LLMs) are able to engage in natural-sounding conversations with humans, showcasing unprecedented capabilities for information retrieval and automated decision support. They have disrupted human-technology …


From Cosine Similarity to Likelihood Ratio: Coupling Representations From Machine Learning (and Other Sources) With Cognitive Models

Modern machine learning models yield vector representations that capture similarity relations between complex items like text and images. These representations can help explain and predict how individuals respond to those items in particular tasks …


From Data Creator to Data Reuser: Distance Matters

CL Borgman, PT Groth - arXiv preprint arXiv:2402.07926, 2024

… Common Crawl, launched in 2007, now spends roughly $200,000 per year to maintain petabytes of openly available web crawl data (Common Crawl, 2023; Roberts, 2023). The use of these data to train large language models has …


From Data Quality to Model Performance: Navigating the Landscape of Deep Learning Model Evaluation

M Akram, WH Moosa - Deep Learning for Multimedia Processing Applications, 2024

Deep learning (DL) technologies have revolutionized the analysis of multimedia data, with natural language processing, visual data analytics, and audio recognition being just a few examples of multimedia applications that have effectively utilized DL. This …


From Development to Dissemination: Social and Ethical Issues with Text-to-Image AI-Generated Art

… Model developed by the Computer Vision & Learning Group (CompVis) lab at Ludwig Maximilian University of Munich was trained with a subset of the LAION-5B dataset [6], comprising of about 2.3 billion CLIP-filtered image-text pairs by parsing …


From Discrete to Continuous Classes: A Situational Analysis of Multilingual Web Registers with LLM Annotations

E Henriksson, A Myntti, S Hellström… - Proceedings of the 4th …, 2024

In corpus linguistics, registers–language varieties suited to different contexts–have traditionally been defined by their situations of use, yet recent studies reveal significant situational variation within registers. Previous quantitative studies …


From Enclosure to Foreclosure and Beyond: Opening AI's Totalizing Logic

This paper reframes the issue of appropriation, extraction, and dispossession through AI—an assemblage of machine learning models trained on big data—in terms of enclosure and foreclosure. While enclosures are the product of a well-studied …


From Form (s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency

X Ohmer, E Bruni, D Hupkes - arXiv preprint arXiv:2404.12145, 2024

… We use the current common crawl statistics6 to compute an estimate of how low- or high-resource these languages are in web-based corpora. Of this corpus, English constitutes 46% of the data, German 5.8%, Italian 2.7%, Dutch 2.2%, and Swedish …


From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients

A Jaiswal, L Yin, Z Zhang, S Liu, J Zhao, Y Tian… - arXiv preprint arXiv …, 2024

… The C4 dataset is a massive collection of Common Crawl’s web crawl corpus, meticulously filtered and cleaned to ensure high-quality language modeling and training. For downstream task finetuning of compressed models, we consider a good …


From general LLM to translation: How we dramatically improve translation quality using human evaluation data for LLM finetuning

D Elshin, N Karpachev, B Gruzdev, I Golovanov… - Proceedings of the Ninth …, 2024

This paper describes Yandex submission to the WMT2024 General Translation Task. More specifically, we present a novel pipeline designed to build a strong paragraph-level translation engine with an emphasis on video subtitles domain. In particular, we …


From Graph Theory for Robust Deep Networks to Graph Learning for Multimodal Cancer Analysis

This dissertation explores the intersection of graph theory and deep learning, focusing on enhancing the robustness of deep neural networks (DNNs) and applying these advancements to complex problems like cancer diagnosis and …


From ML to LLM: Evaluating the Robustness of Phishing Webpage Detection Models against Adversarial Attacks

A Kulkarni, V Balachandran, DM Divakaran, T Das - arXiv preprint arXiv:2407.20361, 2024

Phishing attacks attempt to deceive users into stealing sensitive information, posing a significant cybersecurity threat. Advances in machine learning (ML) and deep learning (DL) have led to the development of numerous phishing webpage detection …


From News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarization

B Barta, D Lakatos, A Nagy, MK Nyist, J Ács - arXiv preprint arXiv:2404.03555, 2024

… We use the freely available Common Crawl dataset4 as a basis for constructing the corpus. It contains petabytes of crawled web pages from the past 25 years and it is available on Amazon S3 in WARC format. Retrieval and deduplication of the raw …


From Pixels to Prose: A Large Dataset of Dense Image Captions

V Singla, K Yue, S Paul, R Shirkavand… - arXiv preprint arXiv …, 2024

Training large vision-language models requires extensive, high-quality image-text pairs. Existing web-scraped datasets, however, are noisy and lack detailed image descriptions. To bridge this gap, we introduce PixelProse, a comprehensive dataset …


FROM PIXELS TO PROSE: ALarge DATASET OF DENSE IMAGE CAPTIONS


From Populism to Platforms: Antitrust Law and the AI Revolution

This chapter, from my book manuscript “The Political Economy of the Fourth Industrial Revolution,” examines how changes to US antitrust law catalyzed the rise of digital platforms and the Artificial Intelligence (AI) Revolution. It traces the …


From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy …

N Shahin, L Ismail - arXiv preprint arXiv:2408.14825, 2024

With the growing Deaf and Hard of Hearing population worldwide and the persistent shortage of certified sign language interpreters, there is a pressing need for an efficient, signs-driven, integrated end-to-end translation system, from sign to gloss to …


From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items

LLMs can now perform a variety of complex writing tasks. They also excel in answering questions pertaining to natural language inference and commonsense reasoning. Composing these questions is itself a skilled writing task, so in this paper …


From Text to Transformation: A Comprehensive Review of Large Language Models' Versatility

P Kaur, GS Kashyap, A Kumar, MT Nafis, S Kumar… - arXiv preprint arXiv …, 2024

This groundbreaking study explores the expanse of Large Language Models (LLMs), such as Generative Pre-Trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT) across varied domains ranging from …


From'Showgirls' to'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs

M Bartl, S Leavy - arXiv preprint arXiv:2407.04434, 2024

Gender bias is not only prevalent in Large Language Models (LLMs) and their training data, but also firmly ingrained into the structural aspects of language itself. Therefore, adapting linguistic structures within LLM training data to promote gender-inclusivity …


Frozen or Fine-tuned? Analyzing Deep Learning Models and Training Strategies for Optimizing Big Five Personality Traits Prediction from Text

M Soleimani, HB Kashani - 2024 10th International Conference on Artificial …, 2024

In today’s digital age, the comprehension and prediction of human personality traits have assumed paramount significance. This study embarks on the task of forecasting the Big Five personality traits through textual data, harnessing the …


FuLG: 150B Romanian Corpus for Language Model Pretraining

VA Bădoiu, MV Dumitru, AM Gherghescu, A Agache… - arXiv preprint arXiv …, 2024

… billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for … , we delve into filtering Romanian content from CommonCrawl, documenting the process and … Second, we document the process of filtering data …


Fully Authentic Visual Question Answering Dataset from Online Communities

Visual Question Answering (VQA) entails answering questions about images. We introduce the first VQA dataset in which all contents originate from an authentic use case. Sourced from online question answering community forums, we call it …


Fully Open Source Moxin-7B Technical Report

P Zhao, X Shen, Z Kong, Y Shen, SE Chang… - arXiv preprint arXiv …, 2024

… First, resiliparse is employed to extract text from CommonCrawl. Second, deduplication is performed using MinHash [58] within a suffix array pipeline [59, 36] and near-duplicate Bloom filtering, which enhances the exact document and …


Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

M Dallabetta, C Dobberstein, A Breiding, A Akbik - arXiv preprint arXiv:2403.15279, 2024

… Specifically, we support the CC-NEWS3 dataset provided by the COMMONCRAWL initiative. At the time of writing, this dataset comprises around 40 terabytes of WARC-formatted data, containing millions of news articles dating back …


Fusing Domain-Specific Content from Large Language Models into Knowledge Graphs for Enhanced Zero Shot Object State Classification

F Gouidis, K Papantoniou, KPT Patkos, A Argyros… - arXiv preprint arXiv …, 2024

Domain-specific knowledge can significantly contribute to addressing a wide variety of vision tasks. However, the generation of such knowledge entails considerable human labor and time costs. This study investigates the potential of Large Language …


Fusion of Deep Learning with Advanced and Traditional Embeddings in Sentiment Analysis

K Kiran, PD Shenoy - 2024 IEEE 9th International Conference for …, 2024

In todays changing landscape the abundance of unprocessed data plays a vital role. It is crucial to curate this dataset and conduct a thorough analysis of the sentiments it captures. The aim of this research study is to explore learning techniques such as Bi …


Fusion of Knowledge: Enhancing AI Reasoning Through Language Models and Knowledge Graphs

Large Language Models (LLMs) and Knowledge Graphs (KGs) have rapidly emerged as important areas in Artificial Intelligence (AI). LLMs leverage vast amounts of unstructured text to understand and generate natural language. KGs are …


Fusion of Visual and Textual Data for Enhanced Semantic Representations

Generic text embeddings have demonstrated considerable success across a multitude of applications. However, these embeddings are typically derived by modeling the co-occurrence patterns within text-only corpora, which can limit their …


FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data

H Sun, R Jin, S Xu, L Pan, M Cui, J Dui, Y Lei, L Yang… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have demonstrated prowess in a wide range of tasks. However, many LLMs exhibit significant performance discrepancies between highand low-resource languages. To mitigate this challenge, we present FuxiTranyu, an …


Galactica's dis-assemblage: Meta's beta and the omega of post-human science

N Chartier-Edwards, E Grenier, V Goujon - AI & SOCIETY, 2024

Released mid-November 2022, Galactica is a set of six large language models (LLMs) of different sizes (from 125 M to 120B parameters) designed by Meta AI to achieve the ultimate ambition of “a single neural network for powering scientific tasks” …


GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

J Zhao, Z Zhang, B Chen, Z Wang, A Anandkumar… - arXiv preprint arXiv …, 2024

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank adaptation (LoRA), add a trainable …


GAOKAO-Eval: Does high scores truly reflect strong capabilities in LLMs?

Z Lei, T Liang, H Hu, J Zhang, Y Zhou, Y Shao, L Li… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) are commonly evaluated using human-crafted benchmarks, under the premise that higher scores implicitly reflect stronger human-like performance. However, there is growing concern that LLMs may ``game" these …


Gauging Novelty in Crowdfunding Projects: A Theory-Driven Text Analysis Approach

The explosion of crowdfunding projects has called for new ways to screen for novel projects. Building on the multidimensional nature of novelty, we draw on the literature to propose two measures of project novelty that are theoretically sound and …


GEB-1.3 B: Open Lightweight Large Language Model

J Wu, Y Zhu, L Shen, X Lu - arXiv preprint arXiv:2406.09900, 2024

… Data Processing The Common Crawl dataset is crawled from the web and often suffers from quality issues, because of including garbled … Rule Cleaning First, we extract a Chinese corpus from the Common Crawl dataset, by applying specialized …


GemBode and PhiBode: Adapting Small Language Models to Brazilian Portuguese

GL Garcia, PH Paiola, E Garcia, JR Ribeiro Manesco… - Iberoamerican Congress on …, 2024

… The Canarim model was obtained by pretraining an LLaMa2-7B model using the Portuguese subset of Common Crawl solely on the language modeling task. Conversely, Bode employs low-rank adaptation (LoRA) to fine-tune and improve the …


GenAI Advertising: Risks of Personalizing Ads with LLMs

BJ Tang, K Sun, NT Curran, F Schaub, KG Shin - arXiv preprint arXiv:2409.15436, 2024

Recent advances in large language models have enabled the creation of highly effective chatbots, which may serve as a platform for targeted advertising. This paper investigates the risks of personalizing advertising in chatbots to their users. We …


GenDec: A robust generative Question-decomposition method for Multi-hop reasoning

J Wu, L Yang, Y Ji, W Huang, BF Karlsson, M Okumura - arXiv preprint arXiv …, 2024

Multi-hop QA (MHQA) involves step-by-step reasoning to answer complex questions and find multiple relevant supporting facts. However, Existing large language models'(LLMs) reasoning ability in multi-hop question answering remains exploration, which is …


Gender Ambiguity of Chinese Names in the United States

… Using the GloVe word embeddings algorithm trained on Common Crawl—a standard English corpus of online text including 840 billion words, I extract associations between Chinese names and five dimensions of gender meanings …


Gender Bias in Natural Gender Language and Grammatical Gender Language within Children's Literature

There has been much research on the connection between language and gender bias but there is little comparing natural gender language, grammatical gender language, and gender bias. This research is important because it can offer an …


Gender Bias in Natural Language Processing and Computer Vision: A Comparative Survey

M Bartl, A Mandal, S Leavy, S Little - ACM Computing Surveys, 2024

Taking an interdisciplinary approach to surveying issues around gender bias in textual and visual AI, we present literature on gender bias detection and mitigation in NLP, CV, as well as combined visual-linguistic models. We identify conceptual …


Gender Identity and Representation in the Context of Economic Development in India

… It is a large multi-lingual language model, trained on 2.5TB of filtered CommonCrawl data. It represents some of the most sophisticated pre-trained models prevalently being used in natural language processing for text classification …


Gendered Grammar or Ingrained Bias? Exploring Gender Bias in Icelandic Language Models

SR Friðriksdóttir, H Einarsson - Proceedings of the 2024 Joint International …, 2024

Large language models, trained on vast datasets, exhibit increased output quality in proportion to the amount of data that is used to train them. This data-driven learning process has brought forth a pressing issue where these models may not only reflect …


Gendered Language in Job Advertisements Relates to Gender Sorting in Public Labor Markets: A Multi-Source Analysis

Increasing gender diversity constitutes a desirable goal for policymakers and recruiters in public organizations. However, contemporary research lacks focus on gender sorting, referring to structural self-selection among job seekers in the public …


General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model

H Wei, C Liu, J Chen, J Wang, L Kong, Y Xu, Z Ge… - arXiv preprint arXiv …, 2024

… For the document-level data, we first collect open-source PDF-style files from the Common Crawl and employ the Fitz Python package to extract corresponding dense text content. In such a process, we gain 1.2M full-page PDF-style image-text pairs …


Generating Apparel Images by Using Stable Diffusion with Prompt Recommendation

XZ Liu, HM Hung, LH Chen - 人工知能学会全国大会論文集 第 38 回 (2024), 2024

Fashion is a profound expression of personal identity, style, and culture in daily life. Making oneself look attractive, which often equates to being fashionable or stylish, is always a priority for many people. In this paper, we proposed a clothing …


Generating Character Lines in Four-Panel Manga

M Inaba - Proceedings of the 37th Pacific Asia Conference on …, 2023

Automatic content generation based on natural language processing is an active research area, especially for story generation. Research on story generation has focused on generating consistent text pertaining to characters’ actions and events; …


Generating Interpretations of Policy Announcements

A Marfurt, A Thornton, D Sylvan, J Henderson - … of the 4th International Conference on …, 2024

Recent advances in language modeling have focused on (potentially multiple-choice) question answering, open-ended generation, or math and coding problems. We look at a more nuanced task: the interpretation of statements of political actors. To this end …


Generating Probabilistic Scenario Programs from Natural Language

K Elmaaroufi, D Shankar, A Cismaru… - arXiv preprint arXiv …, 2024

… Most of these model’s are fine-tuned variants of a foundational model that was trained on a wider body of text such as Common Crawl. For instance Codex is ”a GPT language model fine-tuned on publicly available code from GitHub” and it’s …


Generation of API Documentation using Large Language Models–Towards Self-explaining APIs

Application Programming Interfaces (APIs) play a crucial role in modern software development. They are key in not only in enabling software reuse and adaptation, which greatly increase productivity, but also in facilitating exploration of new …


Generative adversarial network-based phishing URL detection with variational autoencoder and transformer

JK Sasi, A Balakrishnan - Int J Artif Intell, 2024

Phishing attacks pose a constant threat to online security, necessitating the development of efficient tools for identifying malicious URLs. In this article, we propose a novel approach to detect phishing URLs employing a generative …


Generative AI and Business: A Review and Research Agenda

J Woolley - Oxford Research Encyclopedia of Business and …, 2024

Generative Artificial Intelligence (Generative AI or GenAI) is an emerging and fast-growing technology with important consequences for workers, businesses, and society. GenAI is a subset of Machine Learning, which is a branch of Artificial Intelligence (AI) …


Generative AI and future education: a review, theoretical validation, and authors' perspective on challenges and solutions

WK Monib, A Qazi, RA Apong, MT Azizan, L De Silva… - PeerJ Computer Science, 2024

Generative AI (Gen AI), exemplified by ChatGPT, has witnessed a remarkable surge in popularity recently. This cutting-edge technology demonstrates an exceptional ability to produce human-like responses and engage in natural language …


Generative AI and Large Language Models for Cyber Security: All Insights You Need

MA Ferrag, F Alwahedi, A Battah, B Cherif, A Mechri… - arXiv preprint arXiv …, 2024

This paper provides a comprehensive review of the future of cybersecurity through Generative AI and Large Language Models (LLMs). We explore LLM applications across various domains, including hardware design security, intrusion detection …


Generative AI and the Continuing Importance of Information Literacy

GAI models are embedded in a variety of applications. These models are trained using existing information, which is then formatted and delivered within the context of the GAI application. Bias, misand disinformation, and ethical use of information …


Generative AI Application for Building Industry

H Wan, J Zhang, Y Chen, W Xu, F Feng - arXiv preprint arXiv:2410.01098, 2024

This paper investigates the transformative potential of generative AI technologies, particularly large language models (LLMs), within the building industry. By leveraging these advanced AI tools, the study explores their application across key …


Generative AI as a New Platform for Applications Development

MA Cusumano, VF Farias, R Ramakrishnan - 2024

… Earlier generations of foundation models trained on text scraped from public databases like Wikipedia and Common Crawl for the first ‘pre-training’ phase and human-created instruction–answer pairs for the ‘instruction tuning’ phase.58 Based …


Generative AI for Math: Part I--MathPile: A Billion-Token-Scale Pretraining Corpus for Math

Z Wang, R Xia, P Liu - arXiv preprint arXiv:2312.17120, 2023

… expanse of Common Crawl snapshots, a venture we aim to pursue in our future work. … We provide a near-duplicate example found in Common Crawl, as shown in 2 (See Table 8 to … StackExchange that appeared in a document from Common …


Generative AI for Unmanned Vehicle Swarms: Challenges, Applications and Opportunities

G Liu, N Van Huynh, H Du, DT Hoang, D Niyato, K Zhu… - arXiv preprint arXiv …, 2024

With recent advances in artificial intelligence (AI) and robotics, unmanned vehicle swarms have received great attention from both academia and industry due to their potential to provide services that are difficult and dangerous to perform by humans …


Generative AI from Theory to Practice: A Case Study of Financial Advice

We identify some of the most pressing issues facing the adoption of large language models (LLMs) in practical settings and propose a research agenda to reach the next technological inflection point in generative AI. We focus on three challenges …


Generative AI in Education From the Perspective of Students, Educators, and Administrators

This research explores how advanced artificial intelligence (AI), like the technology that powers tools such as ChatGPT, is changing the way we teach and learn in schools and universities. Imagine AI helping to summarize thick legal documents …


Generative AI

After almost six years of study, earning two bachelor’s degrees and now completing this master’s, my student journey comes to an end with this thesis, for now. My time at TU Delft has been a journey of both intellectual growth and personal discovery …


Generative AI, Plagiarism, and Copyright Infringement in Legal Documents

AB Cyphert - Minnesota Journal of Law, Science & Technology, 2024

… OpenAI acknowledged in the GPT-3 research paper that GPT-3 was trained on the Common Crawl dataset, “which includes everything from traditional news sites like the New York Times to sites like Reddit.” Cyphert, supra note 1, at 407, citing Liz …


Generative AI-Augmented Decision-Making for Business Information Systems

E Kromidha, RM Davison - IFIP International Conference on Human Choice and …, 2024

… This is only influenced by the data it has been trained on, and its users which is one of its main limitations as ChatGPT was trained on datasets like Wikipedia, BookCorpus for books, and Common Crawl for web content which are mainly in …


Generative AI-Based Text Generation Methods Using Pre-Trained GPT-2 Model

R Pandey, H Waghela, S Rakshit, A Rangari, A Singh… - arXiv preprint arXiv …, 2024

This work delved into the realm of automatic text generation, exploring a variety of techniques ranging from traditional deterministic approaches to more modern stochastic methods. Through analysis of greedy search, beam search, top-k …


GENERATIVE AI-POWERED FRAMEWORK

K VAYADANDE - Deep Learning Model Optimization, Deployment and …, 2024

Artificial intelligence (AI) tools encompass a diverse array of applications and software designed to simulate intelligent behavior, facilitate decision making, and automate tasks across various domains. These tools leverage artificial intelligence …


Generative Artificial Intellegence (AI) in Pathology and Medicine: A Deeper Dive

HH Rashidi, J Pantanowitz, A Chamanzar, B Fennell… - Modern Pathology, 2024

This review article builds upon the introductory piece in our seven-part series, delving deeper into the transformative potential of generative artificial intelligence (Gen AI) in pathology and medicine. The article explores the applications of Gen AI …


Generative Artificial Intelligence (AI) for the Data Economy: Use Cases in Nursing

… The core recent AIGC advances stem from training more sophisticated generative models on larger datasets; for example, while the main framework maintains the same as GPT-2, GPT-3’s pre-training data size grows from WebTex (38GB) to …


GENERATIVE ARTIFICIAL INTELLIGENCE (AI) IN EDUCATION: A CROSS-NATIONAL SURVEY ON UNIVERSITY TEACHERS'PERCEPTIONS ON THE USE OF …

… A year and three months later, the GPT-3 emerged with the Common Crawl as the main source of its data. The current GPT-4 started in March 2023 with the “unpublished” data source (Wu et al., 2023, p. 1123). …


Generative Artificial Intelligence and Copyright in the Film and Media Industry

VA Oberting IV - Washington and Lee Law Review Online, 2024

The development of generative artificial intelligence (“GAI” or “generative AI”) introduces compelling benefits and capabilities to filmmakers and artists, but also brings complications regarding copyright of creative works. The American film and …


generative artificial intelligence

AFJ Costa, FF Cebrian, LC dos Anjos, MS Guedes…

D Carnahan - Digital Health, AI and Generative AI in Healthcare: A …, 2025

… An example of large databases available and used to train AI models24 is Common Crawl. This is a non-profit organization that regularly crawls and collects data on the web using crawlers, ie automated programs, with the aim of generating …


Generative Artificial Intelligence-Guided User Studies: An Application for Air Taxi Services

S Xiao, J Li, T Fushimi, Y Ochiai - arXiv preprint arXiv:2406.12296, 2024

User studies are crucial for meeting user needs. In user studies, real experimental scenarios and participants are constructed and recruited. However, emerging and unfamiliar studies face limitations, including safety concerns and iterative efficiency …


Generative Artificial Intelligence: Models, Benefits, Dangers and Detection of AI-Generated Text on Specialized Domains

… However, Radford et al. found that a large amount of document within the hashed pages of Common Crawl was unintelligible. A new approach could be to simply use a curated subset of Common Crawl, but they also created a web scrape from Reddit …


Generative Insights Unveiling AI's Evolution and Algorithms

D Elavarasi, MS Ramadevi, JK Jayabarathan… - … of Generative AI for …, 2025

… LLMs are trained on massive datasets such as Common Crawl and the Pile. The terabytes of data comprising these datasets were extracted from billions of publicly accessible websites. On the other hand, Small language models are trained on …


Generative Knowledge Management for Financial Inclusion Through Financial Literacy: A Systematic Review.

S Vadari, C Malladi - IUP Journal of Knowledge Management, 2024

… OpenAI utilized an openly available corpus of webpages called Common Crawl, which in turn derives from a bot that crawls the entire Internet. This Common Crawl dataset contains billions of webpages and is one of the colossal textual datasets …


Generic Embedding-Based Lexicons for Transparent and Reproducible Text Scoring

C Moez - arXiv preprint arXiv:2411.00964, 2024

… I use is trained on Common Crawl, a large collection of web-scraped text. (Please note that GloVe is also available in versions estimated on Common Crawl text, and on Twitter data). The two vocabularies are initially 400,000 English words in size (GloVe) …


Generics are puzzling. Can language models find the missing piece?

G Cilleruelo, E Allaway, B Haddow, A Birch - … of the 31st International Conference on …, 2025

GC Calderón, E Allaway, B Haddow, A Birch - arXiv preprint arXiv:2412.11318, 2024

… DOLMA is a cleaner version of Common Crawl and may have been used in the training data for popular language models (eg, MISTRAL). Therefore, we include recent Reddit comments to validate our findings on data the models have not been …


GeniL: A Multilingual Dataset on Generalizing Language

AM Davani, S Gubbi, S Dev, S Dave, V Prabhakaran - arXiv preprint arXiv …, 2024

… For each language, we queried the Multilingual Common Crawl (mC4) language corpus to collect naturally occurring sentences that contain both the terms in the tuples present in SGM. To ensure a diverse representation of different associations …


Geohumanities 2.0

M Böhlen - On the Logics of Planetary Computing

The last chapter discusses why smaller AI systems are becoming increasingly important in AI. I look at existing foundation models in AI and discuss the unease many researchers express in the homogenization of AI due to monopolies on ever …


German Text Embedding Clustering Benchmark

S Wehrli, B Arnrich, C Irrgang - arXiv preprint arXiv:2401.02709, 2024

This work introduces a benchmark assessing the performance of clustering German text embeddings in different domains. This benchmark is driven by the increasing use of clustering neural text embeddings in tasks that require the grouping of texts (such …


GFWeb: Measuring the Great Firewall's Web Censorship at Scale

NP Hoang, J Dalek, M Crete-Nishihata, N Christin…

… Aggregating the ranking information provided by the Tranco list [62] and the Common Crawl dataset [1], we analyze the popularity of the base censored domains discovered by GFWeb for HTTP(S) filters in comparison to the GFWatch (DNS filter) …


Ghost Sentence: A Tool for Everyday Users to Copyright Data from Large Language Models

S Zhao, L Zhu, R Quan, Y Yang - arXiv preprint arXiv:2403.15740, 2024

Web user data plays a central role in the ecosystem of pre-trained large language models (LLMs) and their fine-tuned variants. Billions of data are crawled from the web and fed to LLMs. How can \textit{\textbf{everyday web users}} confirm if LLMs …


Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

S Singh, A Romanou, C Fourrier, DI Adelani, JG Ngui… - arXiv preprint arXiv …, 2024

Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. These biases stem not only from language but also from the cultural knowledge required to interpret questions, reducing the …


GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages

AH Kargaran, F Yvon, H Schütze - arXiv preprint arXiv:2410.23825, 2024

… In this paper, we adopt the Ungoliant pipeline [3] for extracting text from CommonCrawl. To address the limitations of current LID models (hash collisions and limited language coverage), we develop a new LID model, GlotLID v3.0, an …


Goldfish: Monolingual Language Models for 350 Languages

TA Chang, C Arnett, Z Tu, BK Bergen - arXiv preprint arXiv:2408.10441, 2024

For many low-resource languages, the only available language models are large multilingual models trained on many languages simultaneously. However, using FLORES perplexity as a metric, we find that these models perform worse than …


GPT‐3‐and DALL‐E‐Powered Applications: A Complete Survey

K Vayadande, CB Pednekar, PA Khune… - How Machine Learning is …, 2024

In the new millennium, the advancement of language and image processing technologies has witnessed remarkable progress, and various applications have demonstrated promising results by utilizing sophisticated models such as GPT‐3 …


GPThingSim: A IoT Simulator Based GPT Models Over an Edge-Cloud Environments

MF Khalfi, MN Tabbiche - International Journal of Networked and Distributed …, 2025

… It was trained on an even larger dataset, benefiting from sources such as Common Crawl and 3 billion Wikipedia tokens [4]. While GPT\(-\)3.5 (GPT Turbo) only accepted inputs in the form of text, GPT-4 can also receive and even interpret …


GPTs: Concerns, Limitations and (Some) Responses

… The ones of these that are large and containing substantial source material from the Internet (eg from Common Crawl) will have some harms of representation bias. Then, if the training data has bias then so too will LLMs. To address this, there seem …


Grammatical Redundancy in Scales: Using the “ConGRe” Process to Create Better Measures

L Alley, I Kadolkar, A Gupta, JM Cortina, KP Winsler - Journal of Management, 2024

As theoretical models become more complex, there is more pressure to use less time-consuming methods generally, and shorter scales specifically. Although reliability is related to scale length, reliability cutoffs are easily met, even in very short scales, by writing or …


Grammatical versus Spelling Error Correction: An Investigation into the Responsiveness of Transformer-Based Language Models Using BART and MarianMT

R Raju, PB Pati, SA Gandheesh, GS Sannala… - Journal of Information & …, 2024

Text continues to remain a relevant form of representation for information. Text documents are created either in digital native platforms or through the conversion of other media files such as images and speech. While the digital native text is …


Grammatical vs Spelling Error Correction: An Investigation into the Responsiveness of Transformer-based Language Models using BART and MarianMT

R Raju, PB Pati, SA Gandheesh, GS Sannala, S KS - arXiv preprint arXiv:2403.16655, 2024

… C4 Dataset is an Open-Source dataset obtained with Common Crawl web scrape [30]. This dataset contains millions of collected sentences along with their target sentences. The collected statements, referred to as input sentences in this work …


GRANITE 3.0 LANGUAGE MODELS

This report presents Granite 3.0, a new set of lightweight, state-of-the-art, open foundation models ranging in scale from 400 million to 8 billion active parameters. Equipped with native support of multilingual, coding, function calling, and strong …


Granite Code Models: A Family of Open Foundation Models for Code Intelligence

M Mishra, M Stallone, G Zhang, Y Shen, A Prasad… - arXiv preprint arXiv …, 2024

Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environments to improve the productivity of human programmers, and …


Granularity is crucial when applying differential privacy to text: An investigation for neural machine translation

DNL Vu, T Igamberdiev, I Habernal - arXiv preprint arXiv:2407.18789, 2024

… The computation of b is carried out using Common Crawl11 monolingual datasets. For each language and contextual embedding model, one million candidate-reference pairs are created by grouping two random sentences. Due to the random pairing …


Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions

YG Hsieh, CY Hsieh, SY Yeh, L Béthune, HP Ansari… - arXiv preprint arXiv …, 2024

… In a similar vein, Meta-CLIP [68] reproduces the processing of the seminal CLIP paper [50] on a subset of the Common Crawl dataset, … to filter data and remove duplicates, while DFN [17] uses filtering networks trained on high quality data to …


Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradients

A Muhamed, O Li, D Woodruff, M Diab, V Smith - arXiv preprint arXiv:2406.17660, 2024

Large language model (LLM) training and finetuning are often bottlenecked by limited GPU memory. While existing projection-based optimization methods address this by projecting gradients into a lower-dimensional subspace to reduce optimizer …


GreekT5: Sequence-to-Sequence Models for Greek News Summarization

N Giarelis, C Mastrokostas, N Karacapilidis - IFIP International Conference on …, 2024

… mBART was pre-trained for multiple languages using the CC25 dataset [15], which contains documents from 25 different languages from Common Crawl. The authors have also published an updated mBART model, which supports 50 natural …


Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation

S Xia, X Wang, J Liang, Y Zhang, W Zhou, J Deng, F Yu… - arXiv preprint arXiv …, 2024

Retrieval-Augmented Generation (RAG) has been widely adopted to enhance Large Language Models (LLMs) in knowledge-intensive tasks. Recently, Attributed Text Generation (ATG) has attracted growing attention, which provides citations to …


Grounded Content Automation: Generation and Verification of Wikipedia in Low-Resouce languages.

In this thesis, we work towards improving the representation of low-resource languages in the digital world by easing the access and participation of these communities to reliable information hubs like Wikipedia. Although the internet has …


Grounding Language in Images and Videos

While machine learning research has traditionally explored image, video and text understanding as separate fields, the surge in multi-modal content in today’s digital landscape underscores the importance of computation models that adeptly navigate …


Growing Smaller Language Models Using Knowledge Distillation from Larger Models

M Featherstone, E Cuthbertson, D Appleyard, S Gittins

… Publicly available datasets, such as The Pile and Common Crawl, were employed to ensure a comprehensive and representative sample of language usage across various domains. Data preprocessing involved several crucial steps …


Hacker News new| past| comments| ask| show| jobs| submit login

I don't think these dichotomies in quality are going to go away. It's the exact same thing in every neural net domain. Self driving is probably the most sophisticated domain besides chat, and there too it's the exact same problem. They can drive a …


Hallucinating (or poorly fed) LLMs? The problem of data accuracy

… This is the case of “Common Crawl”, which consists in a massive collection of web pages and … To give a picture, the Common Crawl dataset constitutes nearly a trillion words. … were publicly available online and memorised in the Common …


Handling imbalanced textual data: an attention-based data augmentation approach

AK Sah, M Abulaish - International Journal of Data Science and Analytics, 2024

In this paper, we present an attention-based data augmentation (ADA) approach to address the issue of poor performance of classification algorithms on imbalanced text datasets. The proposed approach begins by ranking the vocabulary of the …


Handwritten Text Recognition for Historical Documents using Visual Language Models and GANs

In this study, we focus on Handwriting Text Recognition (HTR) on Medieval and Early Modern documentary manuscripts (10th-16th centuries) using Vision Language models (VLM). We leverage the TrOCR architecture and integrate domain-specific …


Harder Tasks Need More Experts: Dynamic Routing in MoE Models

Q Huang, Z An, N Zhuang, M Tao, C Zhang, Y Jin, K Xu… - arXiv preprint arXiv …, 2024

In this paper, we introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models, aiming to enhance computational efficiency and model performance by adjusting the number of activated experts based on input difficulty …


Hardware Phi-1.5 B: A Large Language Model Encodes Hardware Domain Specific Knowledge

W Fu, S Li, Y Zhao, H Ma, R Dutta, X Zhang, K Yang…

… The first segment includes data from specialized platforms such as Arxiv , Books, Wikipedia, and StackExchange; the second segment is derived from broader internet content via CommonCrawl and C4. Table II provides a comprehensive …


Harmony in the Australian Domain Space

X Gong, PX McCarthy, MA Rizoiu, P Boldi - arXiv preprint arXiv:2404.10006, 2024

… Though not focused on localized networks, the extensive and all-encompassing web graph from Common Crawl utilized in our study enables the extraction of implicit information on a broader scale. … We utilize the datasets mainly from …


Harnessing AI for Writing Development: Exploring Emerging Multilinguals' ChatGPT Usage Patterns

Since the release of advanced Large Language Models (LLMs) like ChatGPT in the winter of 2022, educators have been challenged to integrate AI in educational contexts without adequate preparation. This multimethod study aims to provide …


Harnessing Artificial Intelligence in Bariatric Surgery: Comparative Analysis of ChatGPT-4, Bing, and Bard in Generating Clinician-Level Bariatric Surgery …

Y Lee, T Shin, L Tessier, A Javidan, J Jung, D Hong… - Surgery for Obesity and Related …

Background The formulation of clinical recommendations pertaining to bariatric surgery is essential in guiding healthcare professionals. However, the extensive and continuously evolving body of literature in bariatric surgery presents considerable …


Harnessing generative AI for overcoming labeled data challenges in social media NLP

With the introduction of Transformers and Large Language Models, the field of NLP has significantly evolved. Generative AI, a prominent transformer-based technology for crafting human-like content, has proven powerful skills across numerous NLP …


Harnessing Small AI Model Collaboration and Debate Mechanisms in 6G Networks: Distributed Architectures and Layered Privacy Protection

C Han, T Yang, X Sun, Z Cui - ICC 2024-IEEE International Conference on …, 2024

… was used for classification tasks, while OpenSubtitles and CommonCrawl were used for text classification tasks. Our primary evaluation metrics revolve around model accuracy, convergence speed, and computational and communication …


Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

L Gao - arXiv preprint arXiv:2408.15650, 2024

… For features that require embedding words, we use the 300-dimensional GloVe word embeddings [100] pretrained on the 42 billion token Common Crawl corpus. The GloVe embeddings are provided in decreasing order by frequency, and some …


Harnessing the Power of Language Models for Intelligent Digital Health Services

S Garima, M Swapnil, S Shashank - 2024 ITU Kaleidoscope: Innovation and Digital …, 2024

This research proposes a novel framework that integrates state-of-the-art large language models (LLMs) with curated medical knowledge bases to enable personalized, reliable, and user-centric digital health services. The architecture …


Harnessing the Power of Metadata for Enhanced Question Retrieval in Community Question Answering

S Ghasemi, A Shakery - IEEE Access, 2024

… GPT-3 was trained on an open source dataset called ‘CommonCrawl’ and other texts from OpenAI such as Wikipedia entries. GPT-4 is the newest version of OpenAI’s language model systems. In this article, we fine-tune GPT-2 and GPT-3 pretrained …


Harnessing Transfer Learning from Swahili: Advancing Solutions for Comorian Dialects

NA Mohamed, Z Erraji, A Bahafid, I Benelallam - arXiv preprint arXiv:2412.12143, 2024

… This model is trained on mC46, a multilingual dataset comprising texts in 101 languages sourced from CommonCrawl. mT5 excels across various NLP tasks, demonstrating state-of-the-art performance on multiple benchmark datasets. A …