Cross-cultural Inspiration Detection and Analysis in Real and LLM-generated Social Media Data

O Ignat, GG Lakshmy, R Mihalcea - arXiv preprint arXiv:2404.12933, 2024

Inspiration is linked to various positive outcomes, such as increased creativity, productivity, and happiness. Although inspiration has great potential, there has been limited effort toward identifying content that is inspiring, as opposed to just engaging …


Cross-Language Harmonization of Linguistic Resources

The presented work consists of two parts. In the first part I summarize the main directions of my research since the defense of my PhD thesis in 2005. I start with cross-language transfer of parsing models to languages that have little or no …


Cross-language Retrieval

P Galuščáková, DW Oard, S Nair - Information Retrieval: Advanced Topics and …, 2024

8.1 Two key assumptions shape the usual view of ranked retrieval:(1) that the searcher can choose words for their query that might appear in the documents that they wish to see, and (2) that ranking retrieved documents will suffice because the …


Cross-Lingual Cross-Modal Retrieval with Noise-Robust Fine-Tuning

R Cai, J Dong, T Liang, Y Liang, Y Wang, X Yang… - IEEE Transactions on …, 2024

… from snapshots of the CommonCrawl public dataset2. Experiments in this paper concern parallel data between English and other four languages, namely Chinese (ZH), French (FR), German (DE) and Czech (CZ). Evaluation metrics. … https://commoncrawl.org/ …


Cross-lingual Editing in Multilingual Language Models

H Beniwal, M Singh - arXiv preprint arXiv:2401.10521, 2024

The training of large language models (LLMs) necessitates substantial data and computational resources, and updating outdated LLMs entails significant efforts and resources. While numerous model editing techniques (METs) have emerged to …


Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization

K Uhlig, J Wuebker, R Reinauer, J DeNero - arXiv preprint arXiv:2409.17673, 2024

Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying …


Cross-Lingual Named Entity Recognition for Low-Resource Languages: A Hindi-Nepali Case Study Using Multilingual BERT Models

D Yadav, S Suravee, T Strauß, K Yordanova - Proceedings of the Fourth Workshop on …, 2024

… RemBERT (Rebalanced multilingual BERT) is a transformer model trained on large unlabeled Wikipedia and Common Crawl 2 data in … ) MuRIL (Multilingual Representations for Indian Languages) is a transformer-based model trained on the …


Cross-lingual Transfer and Multilingual Learning for Detecting Harmful Behaviour in African Under-Resourced Language Dialogue

TO Ajayi, M Arcan, P Buitelaar - Proceedings of the 25th Annual Meeting of the …, 2024

Most harmful dialogue detection models are developed for high-resourced languages. Consequently, users who speak under-resourced languages cannot fully benefit from these models in terms of usage, development, detection and …


Cross-lingual transfer for the annotation of the SynSemClass ontology

This work compares two approaches to automatic preannotation of semantic class to verbs in a sentence for the purpose of adding a new language to the SynSemClass ontology. Both approaches rely on a multilingual deep learning classification model …


Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation

V Neplenbroek, A Bisazza, R Fernández - arXiv preprint arXiv:2412.14050, 2024

… We select the Common Crawl corpus to investigate percentages of language data as we assume models have likely been trained on it, or a similar collection of web crawl data. We select the Common Crawl version from week 30 of 2024, as this is …


Cross-lingual transfer of multilingual models on low resource African Languages

H Thangaraj, A Chenat, JS Walia, V Marivate - arXiv preprint arXiv:2409.10965, 2024

… 2019) introduce XLM-R, a large-scale multilingual language model trained on 100 languages using two terabytes of CommonCrawl data. XLM-R offers better performance than models such as mBERT, particularly in low-resource languages such as …


Cultural Fidelity in Large-Language Models: An Evaluation of Online Language Resources as a Driver of Model Performance in Value Representation

S Kazemi, G Gerhardt, J Katz, CI Kuria, E Pan… - arXiv preprint arXiv …, 2024

The training data for LLMs embeds societal values, increasing their familiarity with the language's culture. Our analysis found that 44% of the variance in the ability of GPT-4o to reflect the societal values of a country, as measured by the World Values …


Current strategies to address data scarcity in artificial intelligence-based drug discovery: A comprehensive review

A Gangwal, A Ansari, I Ahmad, AK Azad… - Computers in Biology and …, 2024

Artificial intelligence (AI) has played a vital role in computer-aided drug design (CADD). This development has been further accelerated with the increasing use of machine learning (ML), mainly deep learning (DL), and computing hardware and software …


Cyber Risks of Machine Translation Critical Errors: Arabic Mental Health Tweets as a Case Study

H Saadany, A Tantawy, C Orasan - arXiv preprint arXiv:2405.11668, 2024

With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like …


Cyberbullying Detection Using PCA Extracted GLOVE Features and RoBERTaNet Transformer Learning Model

M Umer, EA Alabdulqader, AA Alarfaj, L Cascone… - IEEE Transactions on …, 2024

Online platforms are nurturing social interactions, yet regrettably, they have also led to the proliferation of antisocial behaviors such as cyberbullying, trolling, and hate speech on a global scale. The identification of hate speech and aggression has …


D3. 2: Report on Citizen discourses and attitudes towards controversies

P Troullinou, I Bonavita, FU TRI, FT ERI, S Stoyanov…

… Social media listening analysed data from the open web repository, CommonCrawl, for the period 2013-2021. The analysis indicated which … Social listening The findings of the analysis conducted on the CommonCrawl database will …


DANSK: Domain Generalization of Danish Named Entity Recognition

K Enevoldsen, ET Jessen, R Baglini - Northern European Journal of Language …, 2024

Named entity recognition is an important application within Danish NLP, essential within both industry and research. However, Danish NER is inhibited by a lack coverage across domains and entity types. As a consequence, no current models …


Data Analysis of Twitter's Nasdaq100 Sentiments and Topics as Indicators for News Articles Retrieval: Fine-Tuning RoBERTa and RAG

This study explores the combination of sentiment analysis with vader-lexicon and semantic analysis with latent dirichlet allocation to identify real-life events, particularly in the context of Twitter datasets. While sentiment analysis alone may not …


Data Analyst Competencies: A Theory-Driven Investigation of Industry Requirements in the Field of Data Analytics

CA Collier, AL Powell - Journal of Information Systems Education, 2024

… First, the job postings utilized in Dong and Triche’s study were not comprehensive but rather only those captured by the Common Crawl tool to facilitate the longitudinal nature of their study. Second, the wildcard searches used by Dong and Triche …


Data Augmentation and Large Language Model for Legal Case Retrieval and Entailment

MQ Bui, DT Do, NK Le, DH Nguyen, KVH Nguyen… - The Review of Socionetwork …, 2024

The Competition on Legal Information Extraction and Entailment (COLIEE) is a well-known international competition organized each year with the goal of applying machine learning algorithms and techniques in the analysis and understanding of legal …


Data augmentation for language generation inspired by machine translation

The field of natural language processing has witnessed a surge in the adoption of deep learning, which faces notable hurdles when the training data is scarce. This thesis aims to study automatic data augmentation for language generation tasks …


Data Augmentation with Semantic Enrichment for Deep Learning Invoice Text Classification

WW Chi, TY Tang, NM Salleh, M Mukred, H AlSalman… - IEEE Access, 2024

Natural language processing (NLP) is a research field that provides huge potential to automate accounting tasks dealing with text data. This research studies the application of NLP in automatically categorizing invoices based on the invoice text …


Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

S Longpre, R Mahari, N Obeng-Marnu, W Brannon… - arXiv preprint arXiv …, 2024

… Common Crawl is accurate with wide coverage, but not detailed. Hugging Face can be inaccurate with varying levels of detail, but it has extensive coverage. The Data Provenance Initiative is highly accurate and detailed but is currently limited in scope. …


Data Collection Pipeline for Low-Resource Languages: A Case Study on Constructing a Tetun Text Corpus

G de Jesus, SS Nunes - Proceedings of the 2024 Joint International Conference …, 2024

This paper proposes Labadain Crawler, a data collection pipeline tailored to automate and optimize the process of constructing textual corpora from the web, with a specific target to low-resource languages. The system is built on top of Nutch, an …


Data Contamination Report from the 2024 CONDA Shared Task

O Sainz, I García-Ferrero, A Jacovi, JA Campos… - arXiv preprint arXiv …, 2024

… The reported corpora are mainly based on CommonCrawl snapshots, GitHub, or a mix of sources. For CommonCrawlbased corpora, there are 35 events reported for C4 (Raffel et al.… Ortiz Su’arez et al., 2019) and 6 for CommonCrawl (Rana…


Data Engineering for Scaling Language Models to 128K Context

Y Fu, R Panda, X Niu, X Yue, H Hajishirzi, Y Kim… - arXiv preprint arXiv …, 2024

We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize information at arbitrary input …


Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining

C Ge, Z Ma, D Chen, Y Li, B Ding - arXiv preprint arXiv:2405.14908, 2024

Large language models exhibit exceptional generalization capabilities, primarily attributed to the utilization of diversely sourced data. However, conventional practices in integrating this diverse data heavily rely on heuristic schemes, lacking …


Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions

J Hayase, A Liu, Y Choi, S Oh, NA Smith - ICML 2024 Workshop on Foundation Models in …

J Hayase, A Liu, Y Choi, S Oh, NA Smith - The Thirty-eighth Annual Conference on Neural …

… Natural Language Mixtures We use the Oscar v23.01 corpus [1], which is based on the Nov/Dec 2022 dump from Common Crawl. We … -23.01, which is the January 2023 version of the OSCAR Corpus based on the November/December …


Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

J Hayase, A Liu, Y Choi, S Oh, NA Smith - arXiv preprint arXiv:2407.16607, 2024

… on the Nov/Dec 2022 dump from Common Crawl. We consider the 112 languages with at least 1 MB of data. … The training data is reportedly sourced from Common Crawl … November/December 2022 dump of Common Crawl. We only …


Data Processing for the OpenGPT-X Model Family

N Brandizzi, H Abdelwahab, A Bhowmick, L Helmer… - arXiv preprint arXiv …, 2024

… RefinedWeb [40] is a 2.8TB English dataset created from filtered CommonCrawl data, employing the MacroData refinement pipeline. … CommonCrawl derived data combined with code, social media, scientific papers, and fiction from the specialized …


Data Proportion Detection for Optimized Data Management for Large Language Models

H Liang, K Zhao, Y Yang, B Cui, G Dong, Z Zhou… - arXiv preprint arXiv …, 2024

… initial data proportion detection process accurately identifies the proportions of common-crawl and code, as they constitute a significant portion of the dataset. However, for other categories, the preliminary detection method fails to accurately …


Data Selection for Generalization in Unimodal & Multimodal Models

In this thesis, I present research on improving datasets for deep learning models with automateddata transformation methods. The immediate goal of this work is to maximize the in-domainperformance, out-of-domain generalization, and robustness …


Data Selection for Task-Specific Model Finetuning

Z Liu, A Karbasi, T Rekatsinas - arXiv preprint arXiv:2410.11303, 2024

… For example, a study [13] on several snapshots of ClueWeb2 and Common Crawl shows that 14% to 52% of the documents are near-duplicates. Previous … The high volume of candidate data (eg, 250 billion pages in Common Crawl) poses a great …


Data Selection via Optimal Control for Language Models

Y Gu, L Dong, H Wang, Y Hao, Q Dong, F Wei… - arXiv preprint arXiv …, 2024

… In our experiments, we select data from the CommonCrawl2 with PDS and pre-train LMs with 160M, 470M, 1B, and 1.7B parameters from scratch. We observe around 2 times speed-up in pre-training on the 1.7B LM and constant improvement in …


Data, Data Everywhere: A Guide for Pretraining Dataset Construction

J Parmar, S Prabhumoye, J Jennings, B Liu… - arXiv preprint arXiv …, 2024

… We close this gap by conducting a large-scale analysis on over 90 Common Crawl web snapshots for the attributes of domain, quality, toxicity, and type of speech. We then show how such data attributes can aid in pretraining set construction to …


Data-centric Advanced Knowledge Interface for Legal Archives

M Roque, S Chu, M Cabatuan, N Abejuela, M Ng…

This study presents an innovative and novel approach to section retrieval from the National Internal Revenue Code (NIRC) of the Philippines, leveraging advanced Natural Language Processing (NLP) techniques. The study shares the first …


Data-Centric AI Governance: Addressing the Limitations of Model-Focused Policies

R Gupta, L Walker, R Corona, S Fu, S Petryk… - arXiv preprint arXiv …, 2024

… 2009) and Common Crawl,9 brought modern machine learning capabilities to bear. Today, AI datasets are often orders of magnitude larger, created by scraping content across the internet. … 9https://commoncrawl.org/the-data/ …


Data-Centric and Heterogeneity-Adaptive Sequence Parallelism for Efficient LLM Training

Y Wang, S Wang, S Zhu, F Fu, X Liu, X Xiao, H Li, J Li… - arXiv preprint arXiv …, 2024

… mechanism in sequence blaster (§4.2), we compare the performance of complete version of FlexSP against various ablated versions on CommonCrawl, as shown in Fig. 7. Sequence sorting in sequence blaster helps reduce sequence length …


Data-Centric Methods for Decentralizing Large Language Models

… from large web crawl services like Common Crawl. Web crawl is perceived to be wholly representative of activity on the Internet [Brügger… Indeed, web dumps like Common Crawl offer the promise of more diverse text than what is available in …


Data-Prep-Kit: getting your data ready for LLM application development

D Wood, B Lublinsky, A Roytman, S Singh, A Adebayo… - arXiv preprint arXiv …, 2024

Data preparation is the first and a very important step towards any Large Language Model (LLM) development. This paper introduces an easy-to-use, extensible, and scale-flexible open-source data preparation toolkit called Data Prep Kit (DPK). DPK …


Data-to-Text Generation with Neural Language Models

Data-to-text generation systems need to produce texts with high levels of semantic accuracy. Rule-based systems can guarantee this aspect, but their fluency and adaptability to new domains remain limited. Meanwhile, neural language models …


DataComp Challenge

… billion sample image-text dataset filtered from a comparable pool of samples to Common Crawl. For the creation of the LAION-2B dataset, first language filtering is applied to the original Common Crawl pool so that only samples with English texts …


DataComp-LM: In search of the next generation of training sets for language models

J Li, A Fang, G Smyrnis, M Ivgi, M Jordan, S Gadre… - arXiv preprint arXiv …, 2024

… As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark …


DataDecon: Data Cleansing Tools for Large Language Model with Efficient Decontamination Techniques

S Yuenyong, N Buppodom, K Sangkaew… - 2024 19th International Joint …, 2024

… coRpus) Dataset is a multilingual corpus extracted from Common Crawl by crawling the web and filtering for high quality … from Common Crawl and cleaning the extracted documents. 3) mC4: The mC4 Dataset contains English documents …


DataSculpt: Crafting Data Landscapes for LLM Post-Training through Multi-objective Partitioning

K Lu, Z Liang, X Nie, D Pan, S Zhang, K Zhao, W Chen… - arXiv preprint arXiv …, 2024

… For instance, less than 5% of documents within CommonCrawl exceed 2k tokens in length [8], which hinders LLMs’ exposure to complex narratives. Despite these challenges, significant strides have been made toward enhancing LLMs’ capabilities …


Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

H Pouransari, CL Li, JHR Chang, PKA Vasu, C Koc… - arXiv preprint arXiv …, 2024

… Although the benefits of ICLM with large-scale Common Crawl data (used in our experiments) are marginal in regular evaluation, it significantly boosts long-context evaluation metrics from 27.5 to 28.7 by constructing long data sequences through …


Dataset Growth

Z Qin, Z Xu, Y Zhou, Z Zheng, Z Cheng, H Tang… - arXiv preprint arXiv …, 2024

Deep learning benefits from the growing abundance of available data. Meanwhile, efficiently dealing with the growing data scale has become a challenge. Data publicly available are from different sources with various qualities, and it is …


Datasets for Large Language Models: A Comprehensive Survey

Y Liu, J Cao, C Liu, K Ding, L Jin - arXiv preprint arXiv:2402.18041, 2024

… The first method involves building upon Common Crawl1. Common Crawl is a massive, unstructured, multilingual web corpus that provides public access to web archives by regularly crawling and storing webpage data from the Internet. However …


Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

H Park, S Lee, G Gim, Y Kim, D Kim, C Park - arXiv preprint arXiv:2403.19340, 2024

To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of …


Dated Data: Tracing Knowledge Cutoffs in Large Language Models

J Cheng, M Marone, O Weller, D Lawrie, D Khashabi… - arXiv preprint arXiv …, 2024

… Our analysis reveals two reasons for these inconsistencies: (1) temporal biases of CommonCrawl data due to non-trivial amounts of old data in new dumps and (2) complications in LLM deduplication schemes involving semantic duplicates and …


DDK: Distilling Domain Knowledge for Efficient Large Language Models

J Liu, C Zhang, J Guo, Y Zhang, H Que, K Deng, Z Bai… - arXiv preprint arXiv …, 2024

Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the …


DE-COP: Detecting Copyrighted Content in Language Models Training Data

AV Duarte, X Zhao, AL Oliveira, L Li - arXiv preprint arXiv:2402.09910, 2024

How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim …


Dear GPT-3: Collaborative Writing with Neural Networks

J Becker - Artificial Intelligence-Intelligent Art?: Human-Machine …, 2024

GPT-3 and I engaged in this dialogue about four months ago. I wanted to develop a collaborative writing project, not only to collectively realise a novel, but also to negotiate poetological questions which might arise during the process. These …


DeBERTa-BiLSTM: A multi-label classification model of arabic medical questions using pre-trained models and deep learning

BS Al-Smadi - Computers in Biology and Medicine, 2024

… Using data from CommonCrawl in 100 different languages, the authors in [38] introduce XLM-RoBERTa, an advanced masked multilingual language model. XLM-RoBERTa outperformed previous multilingual models such as XLM and mBERT. [39] …


Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Models

R Abbasi, MH Rohban, MS Baghshah - arXiv preprint arXiv:2407.05897, 2024

… In our experiments, we evaluate CLIP models trained on a diverse selection of datasets, including OpenAI’s private dataset, LAION, YFCC15m, CC12m, DataComp, DFN-5B, WebLI, and CommonCrawl. These models leverage a variety of backbone …


Decluttering the data mess in LLM training

… When training an LLM, developers first collect training data from various sources, such as Wikipedia, Common Crawl, or ArXiv. They then clean the data, which typically involves deduplicating, filtering, and applying classifiers to the data …


Decoding the Diversity: A Review of the Indic AI Research Landscape

S KJ, V Jain, S Bhaduri, T Roy, A Chadha - arXiv preprint arXiv:2406.09559, 2024

This review paper provides a comprehensive overview of large language model (LLM) research directions within Indic languages. Indic languages are those spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri Lanka, Nepal, and …


Deconstructing AI literacy for librarians

In this presentation, Samantha shares ways we can think about AI literacy-how it is similar and dissimilar from traditional information literacy, what we need to know about the mechanics of AI tools, and how SMU Libraries support the different users …


Deconstructing In-Context Learning: Understanding Prompts via Corruption

N Shivagunde, V Lialin, S Muckatira, A Rumshisky - arXiv preprint arXiv:2404.02054, 2024

The ability of large language models (LLMs) to "learn in context" based on the provided prompt has led to an explosive growth in their use, culminating in the proliferation of AI assistants such as ChatGPT, Claude, and Bard. These AI …


DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models

R Zhao, ZL Thai, Y Zhang, S Hu, Y Ba, J Zhou, J Cai… - arXiv preprint arXiv …, 2024

… Specifically, we select five large pre-training datasets including Common Crawl Chn (CC-CN), Dolma, C4, The Pile, and Baidu Wiki (BD-Wiki). Due to limited resources, we only sample a volume of 100 billion tokens from these datasets. …


Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages

Multilingual neural machine translation systems learn to map sentences of different languages into a common representation space. Intuitively, with a growing number of seen languages the encoder sentence representation grows more flexible and …


Deep learning aided clinical decision support

Medical professionals create vast amounts of clinical texts during patient care. Often, these documents describe medical cases from anamnesis to the final clinical outcome. Automated understanding and selection of relevant medical records pose …


Deep Learning Based Multi-document Summarization

In this era of rapidly advancing technology, the exponential increase of data availability makes analyzing and understanding text files a tedious, labor-intensive, and time-consuming task. Multi-document summarization (MDS) is an effective tool …


Deep Learning based Visually Rich Document Content Understanding: A Survey

Y Ding, J Lee, SC Han - arXiv preprint arXiv:2408.01287, 2024

Visually Rich Documents (VRDs) are essential in academia, finance, medical fields, and marketing due to their multimodal information content. Traditional methods for extracting information from VRDs depend on expert knowledge and manual labor …


Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlook

X Zou, Y Yan, X Hao, Y Hu, H Wen, E Liu, J Zhang, Y Li… - arXiv preprint arXiv …, 2024

As cities continue to burgeon, Urban Computing emerges as a pivotal discipline for sustainable development by harnessing the power of cross-domain data fusion from diverse sources (eg, geographical, traffic, social media, and environmental data) …


Deep Learning for Economists

M Dell - arXiv preprint arXiv:2407.15339, 2024

Deep learning provides powerful methods to impute structured information from large-scale, unstructured text and image datasets. For example, economists might wish to detect the presence of economic activity in satellite images, or to measure the topics or …


Deep Learning Impacts in the Field of Artificial Intelligence

R Gulwani, M Aggarwal - Deep Learning Concepts in Operations Research, 2024

This book chapter explores the profound impacts of deep learning on the field of artificial intelligence (AI). Deep learning has revolutionized various domains within AI, enabling significant advancements in computer vision, natural language processing …


Deep Learning Model for Tamil Part-of-Speech Tagging

H Visuwalingam, R Sakuntharaj, J Alawatugoda… - The Computer Journal, 2024

… These pre-trained word embeddings are trained on Common Crawl and Wikipedia. This model was trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window of size 5 and 10 negatives. …


DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

X Bi, D Chen, G Chen, S Chen, D Dai, C Deng, H Ding… - arXiv preprint arXiv …, 2024

… Our analysis revealed that deduplicating the entire Common Crawl corpus results in higher removal of duplicate instances compared to deduplicating within a single dump. Table 1 illustrates that deduplicating across 91 dumps eliminates four times …


DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Q Zhu, D Guo, Z Shao, D Yang, P Wang, R Xu, Y Wu… - arXiv preprint arXiv …, 2024

… For the math corpus, we collect 221B math-related tokens sourced from CommonCrawl using the same pipeline, which approximately doubles the size of the 120B DeepSeekMath corpus (Shao et al., 2024), while for the natural language …


DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

Z Wu, X Chen, Z Pan, X Liu, W Liu, D Dai, H Gao, Y Ma… - arXiv preprint arXiv …, 2024

We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, through two key major upgrades. For the vision component, we …


DeepSeek-VL: Towards Real-World Vision-Language Understanding

H Lu, W Liu, B Zhang, B Wang, K Dong, B Liu, J Sun… - arXiv preprint arXiv …, 2024

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed for real-world vision and language understanding applications. Our approach is structured around three key dimensions: We strive to ensure our data is diverse …


DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Z Shao, P Wang, Q Zhu, R Xu, J Song, M Zhang, YK Li… - arXiv preprint arXiv …, 2024

… the DeepSeekMath Corpus from Common Crawl. As depicted in Figure 2, … Common Crawl, we employ URL-based deduplication and near-deduplication techniques, resulting in 40B HTML web pages. We then recall mathematical web …


Defining Boundaries: The Impact of Domain Specification on Cross-Language and Cross-Domain Transfer in Machine Translation

L Shahnazaryan, M Beloucif - arXiv preprint arXiv:2408.11926, 2024

… 2021) and mined from the Common Crawl corpus.While all languages used in our experiments, except for Italian, are considered bridge languages in the M2M-100 model, the lack of information on the exact data sizes mined for each language limits …


Deliverable 3.1: WP3 1st Interim technical report

1.1. Background This document is the first Interim Technical Report (deliverable 3.1) prepared within the WP3 New usecases for the ESSnet Trusted Smart Statistics–Web Intelligence Network project (ESSnet TSS-WIN). The report covers the period from …


Delta: A Cloud-assisted Data Enrichment Framework for On-Device Continual Learning

C Gong, Z Zheng, F Wu, X Jia, G Chen - arXiv preprint arXiv:2410.18378, 2024

In modern mobile applications, users frequently encounter various new contexts, necessitating on-device continual learning (CL) to ensure consistent model performance. While existing research predominantly focused on developing …


Demonstration of Efficacy of Exploiting ChatGPT Data to the Transformers-Based Models by Performing Bangla Intent Analysis

K Shimada - International Journal of Integrated Engineering, 2024

With the expanding mode of online opinion sharing, an automatic approach to intent analysis is necessary and useful in the practical scenario. Intent analysis inspects persons' and entities’ viewpoints from online user-created texts. Conventional …


Demystifying Workload Imbalances in Large Transformer Model Training over Variable-length Sequences

H Li, F Fu, S Lin, H Ge, X Wang, J Niu, J Jiang, B Cui - arXiv preprint arXiv …, 2024

… Using the LLaMA 7B model and the CommonCrawl dataset, we start with a default setup of 16 GPUs, a maximum sequence length of 32K, and 200K tokens per iteration. By systematically altering one factor while keeping the others constant, we …


Deriving the Meaning of Out-of-Vocabulary Words

… It consists of the articles that were collected using CommonCrawl[34]. The articles originated from five Czech news portals: … It is trained on more than 2TB of data from the filtered CommonCrawl[34] containing 100 languages. They report that the …


Design and Analysis of Machine Learning Algorithms for Predicting Pattern and Trends of Crime in India

Crime is very a old concept which transmitted in our society from generation to generation. No one is safe today. Crime is a social evil. Our society suffers a lot because of crimes committed by its members. The world is full of criminals and …


Design of Artificial Intelligence Companion Chatbot

With the development of cities and the prevalence of networks, interpersonal relationships have become increasingly distant. When people crave communication, they hope to find someone to confide in. With the rapid advancement of deep …


Design, Implementation and Evaluation of a Chatbot for Accounting Firm: A Fine-Tuning Approach With Two Novel Dataset

Artificial intelligence, particularly in the field of Chatbots, is fundamentally reshaping learning, communication, and work paradigms. This phenomenon has sparked growing interest among businesses, viewing Chatbots as a means to streamline …


Designing a course for pre-service science teachers using ChatGPT: what ChatGPT brings to the table

HZ Okulu, N Muslu - Interactive Learning Environments, 2024

ChatGPT holds significant potential for enhancing learning through integration into education as an advanced chatbot. With the goal of harnessing this potential, our research focused on exploring the utilization of ChatGPT in designing a course plan …


Designing a Robust Radiology Report Generation System

S Singh - arXiv preprint arXiv:2411.01153, 2024

Recent advances in deep learning have enabled researchers to explore tasks at the intersection of computer vision and natural language processing, such as image captioning, visual question answering, visual dialogue, and visual language …


Designing a Tactile Document UI for 2D Refreshable Tactile Displays: Towards Accessible Document Layouts for Blind People

S Alzalabny, O Moured, K Müller, T Schwarz, B Rapp… - Multimodal Technologies …, 2024

… The dataset was initially curated by crawling 1017 documents from Commoncrawl using a keyword-based filtering approach to capture eight unique artistic categories (refer to Table 1). We then manually selected 37 documents from each category, ensuring …


Designing an Intelligent System to Map Global Connections

E Bellamy, K Farrell, A Hopping, J Pinter, M Saju… - 2024

… Extended Abstract: This study develops a knowledge graph from the Common Crawl News Dataset to provide situational awareness and … We developed a data pipeline to extract semantic content from the Common Crawl News feed and filter it …


Designing and developing a dedicated Natural Language Processing Framework for Healthcare Information Technology Management and Assessment

The escalating complexity of the hospital environment, propelled by technological advancements, necessitates a comprehensive exploration of the integration and management of diverse tools and technologies in healthcare settings. In this context …


Designing and engineering a Q&A LLM for network packet representation

As internet traffic continues its ever-growing rapid expansion, the development of solutions for automatic analysis becomes increasingly important for ensuring network performance, security, and reliability. Tools like Wireshark and traffic …


Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

F Xing - arXiv preprint arXiv:2401.05799, 2024

… GPT-3.5 was trained mainly on the Common Crawl corpus [2], which archives the web. BLOOMZ was trained on an even larger Open-science Open-collaboration Text Sources corpus [19], which is mainly crowd-sourced scientific datasets. The five …


Detect Secrets using Machine Learning

Leaked secrets detected by malicious actors can lead to a system compromise and have far reaching consequences for entire IT systems. Secrets such as API tokens can be designed to be easily detectable. However, cryptographic keys and …


Detecting Dementia from Transcribed Speech in Slovak using Pre-trained BERT Models

J Staš, D Hládek, A Kopnický - 2024 34th International Conference Radioelektronika …, 2024

… TB of texts written in 100 languages from the Common Crawl dataset [14]; • RemBERT model6: a Rebalanced multilingual BERT model pre-trained on a large unlabeled text created by the combination of mC4 Common Crawl and Wikipedia …


Detecting Depression and Anxiety on Social Media Using Selective Masking

Mental health problems are one of the major problems in the world. It is estimated that once in their life, at least one mental health condition will affect one in four people. Depression is the most common condition, with 5\% of adults suffering from …


Detecting Hidden Meaning in Stock Images

OBJECTIVE − Automated extraction of hidden meaning in stock images− Is it possible to distinguish what is shown from what is meant?− Examine the divergence of text and image− Analyze the usage of stock images and textual descriptions on the web …


Detecting Machine-Generated News Using Fine-Tuned Transformers

… The dataset for this research was a collection of pre-generated GPT-2 news articles, along with real news articles gathered from the internet using Common Crawl. This data combined with the latest popular LLMs such as Llama, Gemma …


Detecting Textual Deception in Courtroom Testimonies for Romanian Language Using Neural Network Methods

M Crudu, M Lupea - 2024 IEEE 20th International Conference on Intelligent …, 2024

Detecting deception through automated tools involves computational techniques to identify falsehoods in text, which is crucial for evaluating the truthfulness of testimonies in the legal field. The present research explores deception detection in …


Detection of adversarial phishing attack using machine learning techniques

KM Sudar, M Rohan, K Vignesh - Sādhanā, 2024

The frequency of cyberattacks, particularly phishing attacks, is increasing exponentially every day. Many users fall victim to clicking on malicious URLs, leading to the exploitation of their information. The traditional methodologies …


Detection of bullying with MachineLearning: Using Supervised Machine Learning and LLMs to classify bullying in text

In recent years, there has been a increase in the issue of bullying, particularly in academic settings [1]. This degree project examines the use of supervised machine learning techniques to identify bullying in text data from school surveys provided by …


Detection of Hate Speech and Offensive Language CodeMix Text in Dravidian Languages using Cost-Sensitive Learning Approach

K Sreelakshmi, B Premjith, BR Chakravarthi… - IEEE Access, 2024

… It was trained using more than two terabytes of filtered CommonCrawl data. It has significantly improved performance on various cross-lingual transfer tasks and outperformed the mBERT model. XLM-R has 24 layers and 16 attention heads …


DETECTION OF KEY INFORMATION IN EMERGENCY CALLS

… The model used in the experiments was trained on 2.5TB of filtered CommonCrawl data containing 100 languages and has a vocabulary of 250k tokens. Conneau et al. [8] show that such multilingual models can outperform their …


Detection of phishing addresses and pages with a data set balancing approach by generative adversarial network (GAN) and convolutional neural network (CNN) …

S Jafari, N Aghaee‐Maybodi - Concurrency and Computation: Practice and …

Phishing attacks have a remarkable ability to steal user information by using simple techniques. Phishing attacks steal valuable information, such as user names and passwords. The loss caused by phishing attacks is significant, and every year …


Detection of Sarcasm in Urdu Tweets using Deep Learning and Transformer based Hybrid Approaches

ME Hassan, M Hussain, I Maab, U Habib, MA Khan… - IEEE Access, 2024

Sarcasm has a significant role in human communication especially on social media platforms where users express their sentiments through humor, satire, and criticism. The identification of sarcasm is crucial in comprehending the sentiment and the …


Detection of Twitter Spam Using GLoVe Vocabulary Features, Bidirectional LSTM and Convolution Neural Network

P Manasa, A Malik, I Batra - SN Computer Science, 2024

… The training of the GloVe model involves a substantial text corpus, like Wikipedia or Common Crawl, which facilitates the learning of connections between words and phrases. For every word in the corpus, the model creates a vector, wherein each …


Developing a Question Answering (QA) Subsystem for Dialogue Systems. Case Study: Theano, a Greek-speaking Conversational Agent for COVID-19

Basic retrieval-based methods in dialogue systems are restricted by a finite set of predefined responses, often leaving user questions unanswered. In this work, we develop a single-turn Question Answering (QA) subsystem to enhance a closed-domain …


Developing a Thai Grammatical Error Correction Tool for Deaf Students

S Traitruengsakul, E Chuangsuwanich - IEEE Access, 2024

Deaf students face challenges in written communication due to errors such as insertion, deletion, word order issues, misusage, and misspellings. Grammatical error correction (GEC) technology can help mitigate these issues. However, existing …


Developing and Backtesting a Trading Strategy Using Large Language Models, Macroeconomic and Technical Indicators

This thesis explores the development and backtesting of a trading strategy that integrates Large Language Models (LLMs) with macroeconomic and technical indicators. The primary objective is to enhance stock return predictions by …


Developing and Deploying AI Applications on NVIDIA Jetson Orin NX: A Comprehensive Guide

The NVIDIA Jetson Orin NX is a cutting-edge platform that brings powerful AI capabilities to the edge, enabling a wide range of applications from real-time computer vision to advanced robotics. This comprehensive guide provides detailed …


Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering

Q Li, L Li, Y Li - arXiv preprint arXiv:2401.07510, 2024

ChatGPT explores a strategic blueprint of question answering (QA) in delivering medical diagnosis, treatment recommendations, and other healthcare support. This is achieved through the increasing incorporation of medical domain data via natural …


Development and Evaluation of a German Language Model for the Financial Domain

N Kozaeva, S Hamotskyi, C Hänig - Proceedings of the Joint Workshop of the 7th …, 2024

… The German colossal, cleaned Common Crawl corpus15 was employed, comprising texts of varying lengths. Further the results of LM performance for mixed datasets (financial corpus mixed with common language sentences and financial …


Development and Evaluation of Pre-trained Language Models for Historical Danish and Norwegian Literary Texts

A Al-Laith, A Conroy, J Bjerring-Hansen… - Proceedings of the 2024 …, 2024

We develop and evaluate the first pre-trained language models specifically tailored for historical Danish and Norwegian texts. Three models are trained on a corpus of 19th-century Danish and Norwegian literature: two directly on the corpus with no …


Development of a Geographical Question-Answering System in the Kazakh Language

A Mukanova, A Barlybayev, A Nazyrova, L Kussepova… - IEEE Access, 2024

… Among the pre-trained models, the KazBERT model, trained on cleansed CommonCrawl data in the Kazakh language, achieved the most favorable outcomes. For the test dataset, the KazBERT model demonstrated an Exact Match (EM) …


Development of an Adaptive Multi-Domain Artificial Intelligence System Built using Machine Learning and Expert Systems Technologies

J Straub - arXiv preprint arXiv:2406.11272, 2024

Producing an artificial general intelligence (AGI) has been an elusive goal in artificial intelligence (AI) research for some time. An AGI would have the capability, like a human, to be exposed to a new problem domain, learn about it and then use …


Development of an Interface for Multi-Dimensional Clinical Language Analysis of Dementia and Assessment of its Market Viability

… This model, from Microsoft, is initialized from the transformer model XLM-RoBERTa-Large, which is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages [24]. CommonCrawl data consists of a comprehensive corpus of web …


Development of Natural Language Processing Tools and Resources for Assamese Text

It is my immense pleasure to convey my heartfelt gratitude to my supervisors, Prof. Sukumar Nandi and Prof. Priyankoo Sarmah, for their unwavering support and guidance during my research work. I am grateful to both of my supervisors for …


Diagonalization & Forcing FLEX: From Cantor to Cohen and Beyond

Background: Maybe very high energy cosmic radiation started a synaptic butterfly cascade 2... who knows? For one reason or another, I decided to embark on a journey of exploring and discussing the possible role of Artificial Intelligence (AI) on …


Dialectal and Low Resource Machine Translation for Aromanian

AI Jerpelea, AŞ Rădoi, S Nisioi - arXiv preprint arXiv:2410.17728, 2024

We present a neural machine translation system that can translate between Romanian, English, and Aromanian (an endangered Eastern Romance language); the first of its kind. BLEU scores range from 17 to 32 depending on the direction and …


Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning

XY Liu, R Zhu, D Zha, J Gao, S Zhong, M Qiu - arXiv preprint arXiv:2312.17493, 2023

The surge in interest and application of large language models (LLMs) has sparked a drive to fine-tune these models to suit specific applications, such as finance and medical science. However, concerns regarding data privacy have emerged …


Diffused Seeing: The Epistemological Challenge

This article examines the transformation of the relationship between seeing and understanding in humans and machines by the technologies of machine learning known as ‘generative AI’. Taking Stable Diffusion as the main case study, while also …


Digilog: Enhancing Website Embedding on Local Governments-A Comparative Analysis

J Gerber, B Kreiner, J Saxer, A Weiler - International Symposium on Methodologies …, 2024

The ability to understand and process websites, known as website embedding, is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local …


Digital Protection and Innovative Development Path of Red Culture Resources Based on Distributed Machine Learning Supported by Intelligent Information

M Huang, X Zeng - Journal of Combinatorial Mathematics and …

As a product of the revolutionary war years, red culture, with its strong vitality, strong cohesion and extraordinary charm, with its incomparable positive energy, resists vulgar and flattering culture, promotes people to rebuild their faith, purify their minds …


Digital Threats

S Samtani, E Raff, H Anderson, E Domschot… - 2024

… First, we used the common crawl GloVe embedding, created by Stanford [28]. This embedding was trained over 42 billion tokens, with a 300-dimensional vector space, and was used as the embedding layer of the BiLSTM model. Results are …


DINOv2 Meets Text: A Unified Framework for Image-and Pixel-Level Vision-Language Alignment

C Jose, T Moutakanni, D Kang, F Baldassarre, T Darcet…

… We apply the text and image curation process described in Section 3.2 to an initial data pool derived from CommonCrawl [1], consisting of 2.3 billion image-text pairs. We sample 650 million pairs per-epoch using our curation strategy. For the text-based …


DiPaCo: Distributed Path Composition

A Douillard, Q Feng, AA Rusu, A Kuncoro, Y Donchev… - arXiv preprint arXiv …, 2024

Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodating ML approaches that require high bandwidth …


Dipartimento di Informatica

In recent years, the world of Natural Language Processing (NLP) underwent a revolution that shook the foundation of the field. This brought results that were never achieved before and lead to the creation of products that became commercial hits …


Directionality and representativeness are differentiable components of stereotypes in large language models

G Nicolas, A Caliskan - PNAS Nexus, 2024

… The ChatGPT model was trained on vast 19 amounts of data, including the Common Crawl (a large scraping of internet webpages), books, 20 Reddit, and Wikipedia (45), as well as human feedback in reinforcement learning (46), and 21 …


Disambiguating Homographs and Homophones Simultaneously: A Regrouping Method for Japanese

Y Sato - Proceedings of the 2024 Joint International Conference …, 2024

We present a method that re-groups surface forms into clusters representing synonyms, and help disambiguate homographs as well as homophone. The method is applied post-hoc to trained contextual word embeddings. It is beneficial to …


Disambiguating natural language via aligning meaningful descriptions

Artificial Intelligence (AI) technologies are increasingly pervading aspects of our lives. Because people use natural language to communicate with each other, computers should also use natural language to communicate with us. One of the principal …


DISENTANGLING AND INTEGRATING RELATIONAL AND SENSORY INFORMATION IN TRANSFORMER ARCHITECTURES

… We train on 10 billion GPT2 tokens of the FineWeb-Edu dataset [37], which is a curated dataset of high-quality educational text data from CommonCrawl. We train models at multiple parameter scales to study the scaling properties of DAT on …


Disfluency annotated corpora for Indian English in technical domains

V Mujadia, P Mishra, DM Sharma - Language Resources and Evaluation, 2024

Disfluencies are common in spontaneous speech and can significantly affect the accuracy of automated systems that process spoken input. In this work, we tackled this issue for Indian English by developing a human-annotated disfluency corpus (DASIE …


Display options

Chemical patents are a commonly used channel for disclosing novel compounds and reactions, and hence represent important resources for chemical and pharmaceutical research. Key chemical data in patents is often presented in tables …


Dissecting Whiteness: consistencies and differences in the stereotypes of lower-and upper-class White US Americans

T Morgenroth, CT Begeny, TA Kirby, B Paaßen, Y Zeng - Self and Identity, 2024

Economic inequality is increasing in the United States, making categorization and stereotyping based on social class more likely. Yet, social class stereotypes have received relatively little attention. Focusing on spontaneously generated stereotypes …


Distance Comparison Operators for Approximate Nearest Neighbor Search: Exploration and Benchmark

Z Wang, H Xiong, Z He, P Wang - arXiv preprint arXiv:2403.13491, 2024

Approximate nearest neighbor search (ANNS) on high-dimensional vectors has become a fundamental and essential component in various machine learning tasks. Prior research has shown that the distance comparison operation is the bottleneck of …


Distortions in Judged Spatial Relations in Large Language Models

N Fulman, A Memduhoğlu, A Zipf - The Professional Geographer, 2024

We present a benchmark for assessing the capability of large language models (LLMs) to discern intercardinal directions between geographic locations and apply it to three prominent LLMs: GPT-3.5, GPT-4, and Llama-2. This benchmark specifically …


Diversifying Multi-Head Attention in the Transformer Model

N Ampazis, F Sakketou - Machine Learning and Knowledge Extraction, 2024

Recent studies have shown that, due to redundancy, some heads of the Transformer model can be pruned without diminishing the efficiency of the model. In this paper, we propose a constrained optimization algorithm based on Hebbian learning, which …


Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

R van Noord, T Kuzman, P Rupnik, N Ljubešić… - arXiv preprint arXiv …, 2024

… rest of corpora compared in this paper, MaCoCu corpora are not obtained by processing Common Crawl data. Instead, a strategy consisting of crawling relevant internet top-level domains directly for the targeted languages is followed (eg, .al for …


Do llms plan like human writers? comparing journalist coverage of press releases with llms

A Spangher, N Peng, S Gehrmann, M Dredze - … of the 2024 Conference on Empirical …, 2024

… We query Common Crawl for all URLs from 9 major financial newspapersin all scrapes since 2021, resulting in 114 million URLs. From these URLs, we discover 940,000 URLs of news articles, specifically, using a supervised model by Welsh (2022) …


Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarks

The concept of localization in LLMs is often mentioned in prior work; however, methods for localization have never been systematically and directly evaluated. We propose two complementary benchmarks that evaluate the ability of localization …


Do Multilingual Large Language Models Mitigate Stereotype Bias?

S Nie, M Fromm, C Welch, R Görge, A Karimi, J Plepi… - arXiv preprint arXiv …, 2024

While preliminary findings indicate that multilingual LLMs exhibit reduced bias compared to monolingual ones, a comprehensive understanding of the effect of multilingual training on bias mitigation, is lacking. This study addresses this gap by …


Do not shut up and do dribble: social media and TV consumption

M Pazzona, N Spagnolo - Journal of Population Economics, 2024

… Footnote 11 The NLP model utilized in this study, XLM-RoBERTa, undergoes pre-training on 2.5TB of filtered CommonCrawl data encompassing 100 languages. For tweet classification, we employed the general-purpose Python library TweetNLP (Camacho-Collados …


Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction

Q Zhang, VSJ Huang, B Wang, J Zhang, Z Wang… - arXiv preprint arXiv …, 2024

Document parsing is essential for converting unstructured and semi-structured documents-such as contracts, academic papers, and invoices-into structured, machine-readable data. Document parsing extract reliable structured data from …


Document Type Classification using File Names

Z Li, S Larson, K Leach - arXiv preprint arXiv:2410.01166, 2024

… We train file name classifiers on the Web Search dataset described in Section 5.1 and evaluate our models against the Common Crawl dataset described in Section 5.2. Recall that 22.2% of the file names in our Common Crawl dataset belong to a …


Document-Level Machine Translation with Large-Scale Public Parallel Corpora

Despite the fact that document-level machine translation has inherent advantages over sentence-level machine translation due to additional information available to a model from document context, most translation systems continue to operate at a …


Document-Level Similarity Analysis on Pretraining Language Models

… CommonCrawl is a project that periodically crawls the entire web, storing snapshots of web content, including textual information, which is made publicly available. C4 (14), or the Colossal Clean Crawled Corpus, is a dataset that refines …


Document-level Translation with LLM Reranking: Team-J at WMT 2024 General Translation Task

K Kudo, H Deguchi, M Morishita, R Fujii, T Ito, S Ozaki… - Proceedings of the Ninth …, 2024

… models, we only used the Common Crawl and Extended Common Crawl due to the limited … We qualitatively examined the Common Crawl and Extended Common Crawl datasets. Our … Qualitative analysis of the Common Crawl data revealed a …


Documenting Geographically and Contextually Diverse Language Data Sources

A McMillan-Major, F De Toni, Z Alyafeai, S Biderman… - Northern European Journal …, 2024

… Typically, this data is collected from online sources, ranging from highly edited and structured text such as Wikipedia to the myriad text and audiovisual components of web pages, eg, collected by the Common Crawl Founda… Data from Common …


Does Context Help Mitigate Gender Bias in Neural Machine Translation?

H Gete, T Etchegoyhen - arXiv preprint arXiv:2406.12364, 2024

Neural Machine Translation models tend to perpetuate gender bias present in their training data distribution. Context-aware models have been previously suggested as a means to mitigate this type of bias. In this work, we examine this claim by …


Does Lack of Knowledge and Hardship of Information Access Signify Powerful AI? A Large Language Model Perspective

IA Zahid, SS Joudar - Applied Data Science and Analysis, 2023

Large Language Models (LLMs) are evolving and expanding enormously. With the consistent improvement of LLMs, more complex and sophisticated tasks will be tackled. Handling various tasks and fulfilling different queries will be more precise …


Does sentiment help in asset pricing? A novel approach using large language models and market-based labels

F Audrino, J Schüttler, F Sigrist - 2024

We present a novel approach to sentiment analysis in financial markets by using a state-ofthe-art large language model, a market data-driven labeling approach, and a large dataset consisting of diverse financial text sources including earnings call …


Does the Language Matter? Curriculum Learning over Neo-Latin Languages

G Pucci, L Ranaldi - Proceedings of the 2024 Joint International Conference …, 2024

Curriculum Learning (CL) is emerging as a relevant technique to reduce the cost of pre-training Large Language Models. The idea, tested for the English language, is to train LLMs by organizing training examples from the simplest to the most complex …


Does your data spark joy? Performance gains from domain upsampling at the end of training

C Blakeney, M Paul, BW Larsen, S Owen, J Frankle - arXiv e-prints, 2024

Pretraining datasets for large language models (LLMs) have grown to trillions of tokens composed of large amounts of CommonCrawl (CC) web scrape along with smaller, domain-specific datasets. It is expensive to understand the impact of these …


Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

L Soldaini, R Kinney, A Bhagia, D Schwenk, D Atkinson… - arXiv preprint arXiv …, 2024

… In total, we find 0.02% of documents in the 25 Common Crawl snapshots match this filter. … Head, Middle, and Tail parts of our Common Crawl data. The correlation is computed for 24M, … The correlation among the documents flagged for removal …


Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field

T Kerner - arXiv preprint arXiv:2407.14076, 2024

… CommonCrawl, or collecting multiple datasets that are at least slightly related to our domain. Large-scale webscraping is a timeand resource-intensive task, so it is recommended to consider using data from CommonCrawl … [3] Assuming we chose …


DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and Classification

AE Mahdaouy, S Lamsiyah, MJ Idrissi, H Alami… - arXiv preprint arXiv …, 2024

… This is a cleaned version of the Common Crawl’s web corpus, curated by the Allen Institute for Artificial Intelligence [38], containing … This dataset is compiled from CommonCrawl, using strict filtering and extensive deduplication [39], and …


DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining

V Arannil, SS Bhabesh, N Narwal, SN Thirandas… - arXiv preprint arXiv …, 2024

… From a large corpus D like the Common Crawl [16], we can now mine and collate real-world domain-representative documents for all industries listed in table 1. It is possible that a mined document may belong to multiple industry domains. We …


Downstream bias mitigation is all you need

A Baksi, R Singh, T Joshi - arXiv preprint arXiv:2408.00612, 2024

The advent of transformer-based architectures and large language models (LLMs) have significantly advanced the performance of natural language processing (NLP) models. Since these LLMs are trained on huge corpuses of data from the web and …


DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization

R Chen, J Tan, Z Yang, X Yang, Q Dai, Y Cheng, L Lin - IEEE Transactions on …, 2024

Natural Language Video Localization (NLVL) has recently attracted much attention because of its practical significance. However, the existing methods still face the following challenges: 1) When the models learn intra-modal semantic association …


Dr. Nahid Ebrahimi Majd

The issue of web phishing attacks has been more prevalent in recent years, and phishing is one of the riskiest online crimes that can have disastrous consequences. The aim of phishing is to collect confidential information by tricking a target and …


DrawL: Understanding the Effects of Non-Mainstream Dialects in Prompted Image Generation

JN Williams, M FitzMorris, O Aka, S Laszlo - arXiv preprint arXiv:2405.05382, 2024

… Stable Diffusion is trained in part using the LAION-5B dataset [50], which consists of text-image pairs from Common CrawlCommon Crawl includes archived data from a large variety of sources, including sites such as Reddit and a variety of blogs …


DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain

Y Labrak, A Bazoge, OE Khettari, M Rouvier, N Grabar… - arXiv preprint arXiv …, 2024

The biomedical domain has sparked a significant interest in the field of Natural Language Processing (NLP), which has seen substantial advancements with pre-trained language models (PLMs). However, comparing these models has proven …


Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data

B AlKhamissi, Y Tang, A Gökce, J Mehrer, M Schrimpf - arXiv preprint arXiv …, 2024

While today's large language models exhibit impressive abilities in generating human-like text, they require massive amounts of data during training. We here take inspiration from human cognitive development to train models in limited data …


Dredge Word, Social Media, and Webgraph Networks for Unreliable Website Classification and Identification

EM Williams, P Carragher, KM Carley - arXiv preprint arXiv:2406.11423, 2024

In an attempt to mimic the complex paths through which unreliable content spreads between search engines and social media, we explore the impact of incorporating both webgraph and large-scale social media contexts into website credibility …


DsDm: Model-Aware Dataset Selection with Datamodels

L Engstrom, A Feldmann, A Madry - arXiv preprint arXiv:2401.12926, 2024

When selecting data for training large-scale models, standard practice is to filter for examples that match human notions of data quality. Such filtering yields qualitatively clean datapoints that intuitively should improve model behavior. However, in …


Dual Modalities of Text: Visual and Textual Generative Pre-training

Y Chai, Q Liu, J Xiao, S Wang, Y Sun, H Wu - arXiv preprint arXiv:2404.10710, 2024

… Meanwhile, The C4 dataset represents a substantial refinement of the Common Crawl corpus. This dataset, derived from the extensive Common Crawl web scrape, undergoes rigorous cleaning and preprocessing to ensure the quality and relevance …


Duth at semeval 2024 task 8: Comparing classic machine learning algorithms and llm based methods for multigenerator, multidomain and multilingual machine …

T Kyriakou, I Maslaris, A Arampatzis - Proceedings of the 18th International Workshop …, 2024

Text-generative models evolve rapidly nowadays. Although, they are very useful tools for a lot of people, they have also raised concerns for different reasons. This paper presents our work for SemEval2024 Task-8 on 2 out of the 3 subtasks. This …


Dynamic Context-Aware Representation for Semantic Alignment in Large Language Models

J Baronova, C Stevens, L Tennant, A MacPhee

… The model was fine-tuned on the Common Crawl dataset, which provided a large and diverse corpus of text to ensure that the LLM could generalize well across different topics and domains. The experiment utilized an NVIDIA A100 GPU to …


Dynamic decoding and dual synthetic data for automatic correction of grammar in low-resource scenario

A Musyafa, Y Gao, A Solyman, S Khan, W Cai… - PeerJ Computer Science, 2024

… The data was collected from January–December 2018 using Commoncrawl snapshots in the CC-Net repository and organized into a single text document with a total data size of 36 GB. We selected the CC100 Indonesian corpus due to its open …


Dynamic Ensemble Reasoning for LLM Experts

J Hu, Y Wang, S Zhang, K Zhou, G Chen, Y Hu, B Xiao… - arXiv preprint arXiv …, 2024

Ensemble reasoning for the strengths of different LLM experts is critical to achieving consistent and satisfactory performance on diverse inputs across a wide range of tasks. However, existing LLM ensemble methods are either computationally …


Dynamic Few-shot Learning for Computational Social Science

R Malla, TG Coan, V Srinivasan, C Boussalis

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) and have recently demonstrated remarkable potential in social science research through their capacity to efficiently perform a wide range of text-as-data tasks …


Dynamic Neural Embedding Framework for Accurate Knowledge Representation

D Lemal, B Lindholm, A Falkenberg, I Fairbairn… - 2024

… The datasets utilized for training and testing encompassed a diverse range of text corpora, including the Common Crawl dataset for … model with the integrated dynamic embedding layer was pretrained on the Common Crawl dataset, allowing …


Dynamic Tensor Reconfiguration in Large Language Models Using Adaptive Contextual Folding

… The experimental data comprised a diverse corpus of text datasets, including the Common Crawl, Wikipedia, and BookCorpus. Preprocessing steps involved tokenization, normalization, and the removal of duplicates to ensure data quality and …


Easy Problems That LLMs Get Wrong

J Huckle, S Williams - … in Information and Communication: Proceedings of the …

S Williams, J Huckle - arXiv preprint arXiv:2405.19616, 2024

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others. Through a series of …


Echoes of culture: Relationships of implicit and explicit attitudes with contemporary English, historical English, and 53 non-English languages

TES Charlesworth, K Morehouse, V Rouduri… - Social Psychological and …, 2024

Attitudes are intertwined with culture and language. But to what extent? Emerging perspectives in attitude research suggest that cultural representations in language are more related to implicitly measured (vs. explicitly measured) attitudes, and that …


Ecosystem Graphs: Documenting the Foundation Model Supply Chain

R Bommasani, D Soylu, TI Liao, KA Creel, P Liang - … of the AAAI/ACM Conference on …, 2024

Foundation models (eg GPT-4, Gemini, Llama 3) pervasively influence society, warranting greater understanding. While the models garner much attention, accurately characterizing their impact requires considering the broader …


Edge-LLMs: Edge-Device Large Language Model Competition

S Liu, K Han, A Fernandez-Lopez, A Jaiswal…

… The C4 dataset is a colossal, cleaned version of Common Crawl’s web crawl corpus, which is mainly intended to pre-train language models and word representations. Language models like MPT-7B and T5 are pre-trained with the C4 …


EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

X Chen, H Li, J Liang, S Jiang, D Yang - arXiv preprint arXiv:2410.19461, 2024

… 24], we collect general webpages from Common Crawl1, a large-scale web crawl data repository, and drive Playwright2 to automatically annotate them. Then we format these annotations to synthesize question-answering (QA) data using various …


EEG Emotion Copilot: Pruning LLMs for Emotional EEG Interpretation with Assisted Medical Record Generation

H Chen, W Zeng, C Chen, L Cai, F Wang, L Wang… - arXiv preprint arXiv …, 2024

… General LLMs are typically trained on broad datasets like Wikitext [20] and Common Crawl [21] for general knowledge and logical reasoning. For domainspecific applications, they are trained on specialized datasets, such as BioBERT [58] for …


Effects of Language-and Culture-Specific Prompting on ChatGPT

… Some of these tokens come from wellknown datasets such as CommonCrawl and RefinedWeb, while others come from undisclosed sources [28], [29]. GPT-4 was first fine-tuned with data sourced from ScaleAI plus text data from OpenAI. Subsequently …


Effektivisering av fakturaklassificering enligt UNSPSC-standarden: en maskininlärningslösning

Procurement requirements and procurement analysis are approaches used by the Swedish National Agency for Public Procurement to ensure and maintain a sustainable societal development. The aim is to safeguard tax funds and ensure that …


Efficiency Comparison of Dataset Generated by LLMs using Machine Learning Algorithms

P Pawade, M Kulkarni, S Naik, A Raut, KS Wagh - 2024 International Conference on …, 2024

… • Bing Chat was trained on web text data, including Common Crawl, Wikipedia, news articles, social media posts, and other publicly available text, via a technique called denoising autoencoder (DAE). In DAE, the model is given a noisy version of a …


Efficient learning in spiking neural networks

A Rast, MA Aoun, EG Elia, N Crook - Neurocomputing, 2024

… Training data required is similarly massive: the CommonCrawl dataset is 242 TB in size, and hyperparameter tuning remains an expensive, search-based process that can require ∼ 1000 − 10 , 000 + sweeps. After evaluating a number of possible …


Efficient Model-Relational Data Management: Challenges and Opportunities

V Sanca, A Ailamaki - IEEE Transactions on Knowledge and Data …, 2024

As modern data pipelines continue to collect, produce, and store various data formats, extracting and combining value from traditional and context-rich sources becomes unsuitable for RDBMS. To tap into the dark data, domain experts analyze …


Efficient multilingual and domain adaptation of language models under resource constraints

… It has been trained on 25 languages using monolingual Common Crawl data using a subword vocabulary of 250K tokens. It has 680M … Second, in EnDe, we use high-quality corpora for both languages (NewsCrawl), whereas Mk and Sq are …


Efficient query clustering and information retrieval using Sequenced User Search Pattern Query Optimization

S Surya, P Sumitra - Multimedia Tools and Applications, 2024

The rapid growth of the internet has ushered in an era of unprecedented information availability. However, this expansion has also given rise to the formidable challenge of efficiently extracting relevant data from the vast expanse of web pages …


Efficient Training and Inference: Techniques for Large Language Models Using Llama

… The training dataset consisted of a mixture of the Common Crawl, Wikipedia, and BooksCorpus datasets, encompassing a diverse range of linguistic structures and vocabularies. For evaluation, the GLUE benchmark suite was utilized, providing a …


Efficient URL and URI Compression

F Savins, K Saric, GS Ramachandran, R Jurdak - 2024 33rd International Conference …, 2024

… The dataset explored was taken from the Common Crawl project and consists of 5 million URIs in 5 sets. The URIs were found to be 99.4% URLs with the rest being other URI formats such as “Javascript:” and “Tel:”. These URIs were selected …


Efficiently Identifying Watermarked Segments in Mixed-Source Texts

X Zhao, C Liao, YX Wang, L Li - arXiv preprint arXiv:2410.03600, 2024

… The “Colossal Clean Crawled Corpus” (C4) dataset is a collection of English-language text sourced from the public Common Crawl web scrape, a rich source for unwatermarked human-written text. We use random samples from the news-like …


Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

J Hübotter, S Bongni, I Hakimi, A Krause - arXiv preprint arXiv:2410.08020, 2024

Recent efforts in fine-tuning language models often rely on automatic data selection, commonly using Nearest Neighbors retrieval from large datasets. However, we theoretically show that this approach tends to select redundant data, limiting its …


Eir: Thai Medical Large Language Models

Y Thiprak, R Ngodngamthaweesuk… - arXiv preprint arXiv …, 2024

We present Eir Thai Medical LLM, a large language model with 8 billion parameters, specifically designed to enhance the accuracy of handling medical tasks in the Thai language. This model focuses on providing clear and easy-to-understand answers …


Eliciting Big Five Personality Traits in Large Language Models: A Textual Analysis with Classifier-Driven Approach

A Hilliard, C Munoz, Z Wu, AS Koshiyama - arXiv preprint arXiv:2402.08341, 2024

… GPT-3 followed, with 175 billion parameters and training on datasets like Common Crawl [15]. GPT-3.5 Turbo was introduced to enhance real-time performance, and the latest, GPT-4, boasts 1.76 trillion parameters and capabilities …


EMBA: Entity Matching using Multi-Task Learning of BERT with Attention-over-Attention

… The WDC Product Data Corpus for Large-scale Product Matching [34], was built by extracting product offers from Common Crawl. The WDC datasets serve as a popular benchmark and have been used for evaluation in DITTO, JointBERT, and …


Embedding Geometries of Contrastive Language-Image Pre-Training

JCC Chou, N Alam - arXiv preprint arXiv:2409.13079, 2024

… Variance of the dataset: That is, if we construct a new version of DataComp datasets with newer Common Crawl or new version of RedCaps with newer reddit images, the results may change again. 3. Difference in numerical smoothing …


Embedding-based Query Spelling Correction

For many retrieval systems, correcting spelling errors in the queries that searchers submit is an essential step of query understanding. Inspired by a blog post on spelling correction from 2018, we implement and analyze a simple embedding-based …


Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers

S Han, J Song, J Gore, P Agrawal - arXiv preprint arXiv:2412.12276, 2024

Humans distill complex experiences into fundamental abstractions that enable rapid learning and adaptation. Similarly, autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. In this …


Emergent Abilities in Reduced-Scale Generative Language Models

S Muckatira, V Deshpande, V Lialin, A Rumshisky - arXiv preprint arXiv:2404.02204, 2024

Large language models can solve new tasks without task-specific fine-tuning. This ability, also known as in-context learning (ICL), is considered an emergent ability and is primarily seen in large language models with billions of parameters. This …


Emergent Lexical Synthesis Through Contextual Feedback Mechanisms in Large Language Models

Adaptive mechanisms capable of refining intermediate latent states during text generation are essential for addressing limitations in current model architectures, particularly with respect to contextual drift, linguistic coherence, and output diversity …


Emerging AI Tools for Education and Research: Perspective and Policies for IISc

The Director, IISc, constituted a committee on August 21, 2023 with the following mandate:(a) Explore the challenges and benefits of emerging AI tools in the context of academic teaching and learning,(b) provide guidance and recommendations to …


Emerging Roots: Investigating Early Access to Meaning in Maltese Auditory Word Recognition

J Nieder, R van de Vijver, A Ussishkin - Cognitive Science, 2024

In Semitic languages, the consonantal root is central to morphology, linking form and meaning. While psycholinguistic studies highlight its importance in language processing, the role of meaning in early lexical access and its representation remain …


EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

S Ji, Z Li, I Paul, J Paavola, P Lin, P Chen, D O'Brien… - arXiv preprint arXiv …, 2024

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on texts across 546 languages designed for enhanced multilingual performance, focusing on improving language coverage for low-resource languages …


EMMeTT: Efficient Multimodal Machine Translation Training

P Żelasko, Z Chen, M Wang, D Galvez, O Hrinchuk… - arXiv preprint arXiv …, 2024

A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal …


Emotion Analysis in Narrative Economics

The purpose of this quantitative cross-sectional design using a structural equation model was to determine if emotions mediate the viral spread of narratives with potentially far-reaching economic and political consequences. This dissertation …


Emotional significance in Cross-Cultural Semantic Crossmodal Correspondences

J Alvarado - Proceedings of the Annual Meeting of the Cognitive …, 2024

Crossmodal correspondences are associations between perceptual features from different senses that aid in crossmodal binding. The semantic coding of these correspondences is expected to capture and mediate the emergence of perceptual …


Employing Siamese MaLSTM Model and ELMO Word Embedding for Quora Duplicate Questions Detection

A Altamimi, M Umer, D Hanif, S Alsubai, TH Kim… - IEEE Access, 2024

… 3) FastText Subword FastText Subword comprises a collection of 2 million word vectors trained on the Common Crawl dataset, which consists of a massive 600 billion tokens. In contrast to traditional word embeddings, sub-word embeddings …


Empowering geoportals HCI with task-oriented chatbots through NLP and deep transfer learning

In the past ten years, chatbot development has matured to become one of the most well-distinguished outcomes of artificial intelligence. Despite some criticism, Bing AI, ChatGPT and other natural language processing (NLP) products of similar nature …


Empowering hate speech detection: leveraging linguistic richness and deep learning

IGBJ Abasan, EB Setiawan - Bulletin of Electrical Engineering and Informatics, 2024

… What they don’t realize is the use of pre-trained models, namely CommonCrawl and Wiki, in extraction using FastText or GloVe. The pre-trained model used is not specific for hate speech detection and, of course, consists of many languages, and …


Empowering Humans with Machines: From Explanations to Teaching

… This dataset contains hundreds of thousands of online biographies from the Common Crawl corpus. The task is to predict a person’s profession given a biography. The original dataset consists of 29 professions, and we narrow it down to …


Empowering knowledge through AI: open scholarship proactively supporting well trained generative AI

… The bulk of the content consists of the common crawl dataset, a free-to-use dataset consisting of over 250 billion webpages from the last 17 … Both the common crawl dataset and the WebText2 dataset rely heavily on well-linked and well-described …


Empowering Multi-step Reasoning across Languages via Program-Aided Language Models

L Ranaldi, G Pucci, B Haddow, A Birch - Proceedings of the 2024 Conference on …, 2024

In-context learning methods are popular inference strategies where Large Language Models (LLMs) are elicited to solve a task using provided demonstrations without parameter updates. Among these approaches are the reasoning methods …


Enabling action crossmodality for a pretrained large language model

A Caesar, O Özdemir, C Weber, S Wermter - Natural Language Processing Journal, 2024

Natural language processing and vision tasks have seen large improvements recently through the rise of Transformer architectures. The high performing large language models (LLMs) benefit from large textual datasets that are numerously …


Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment

A Agarwalla, A Gupta, A Marques, S Pandit, M Goin… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs that achieve full …


Enabling Parallelism Hot Switching for Efficient Training of Large Language Models

H Ge, F Fu, H Li, X Wang, S Lin, Y Wang, X Nie… - Proceedings of the ACM …, 2024

… We consider two datasets in our experiments, ie, CommonCrawl and GitHub, and the distributions of sequence lengths are shown in Figure 3. Protocols. For each pair of model and dataset, we vary the context length (s) from 4K to 32K to assess the …


Enabling self-identification in intelligent agent: insights from computational psychoanalysis

L Li, C Li - arXiv preprint arXiv:2403.07664, 2024

Building upon prior framework of computational Lacanian psychoanalysis with the theory of active inference, this paper aims to further explore the concept of self-identification and its potential applications. Beginning with two classic paradigms in psychology …


End to End Urdu Abstractive Text Summarization with Dataset and Improvement in Evaluation Metric

Urdu, being a common language in South Asia, has not received significant attention in terms of language processing compared to more advanced languages. In the field of Natural Language Processing (NLP), the task of text summarization …


Energy and AI

J Göpfert, JM Weinand, P Kuckertz, D Stolten

… Because models can be trained on this objective in a self-supervised setting, large unlabeled corpora (such as Wikipedia, book corpora, and Common Crawl data) can be used for training. The computational advantages of the transformer …


English-language abstract text summarization using the T5 model

RD Darshan, I Surya, G Malarselvi - AIP Conference Proceedings, 2024

… The Common Crawl public web scrape is the source for the 750GB of English-language text that makes up the C4 dataset. The news summary dataset that we've used in this instance has 4515 samples and includes the following information: Headlines, Short …


Enhanced Blood Cell Classification Performance and Conditional Image Generation With Transformer Based Models

Blood cancer accounted for over 1.24 million cases globally in 2020 (6.6% of all cancers) and has imposed significant financial and health burdens. The World Health Organization (WHO) emphasizes the need for improved detection and …


EnhancedBERT: A Feature-rich Ensemble Model for Arabic Word Sense Disambiguation with Statistical Analysis and Optimized Data Collection

S Kaddoura, R Nassar - Journal of King Saud University-Computer and …, 2024

Accurate assignment of meaning to a word based on its context, known as Word Sense Disambiguation (WSD), remains challenging across languages. Extensive research aims to develop automated methods for determining word senses in …


Enhancing 3D Asset Retrieval with Semantic Search

Semantic search goes beyond exact keyword matching and instead focuses on the comprehension of search queries' purpose and the surrounding context to deliver relevant search results. In this work, we focus on integrating hybrid search, which …


Enhancing AI Tutoring in Robotics Education: Evaluating the Effect of Retrieval-Augmented Generation and Fine-Tuning on Large Language Models

S Kahl, F Löffler, M Maciol, F Ridder, M Schmitz… - 2024

… The majority of data is gained from CommonCrawl with about 850 billion tokens. CommonCrawl is a free and open of data gathered by web crawls.In addition, the corpus is based on data from C4 (about 190 billion tokens), Github (about 100 billion …


Enhancing Assamese NLP Capabilities: Introducing a Centralized Dataset Repository

S Tamang, DJ Bora - arXiv preprint arXiv:2410.11291, 2024

This paper introduces a centralized, open-source dataset repository designed to advance NLP and NMT for Assamese, a low-resource language. The repository supports various tasks like sentiment analysis, named entity recognition, and …


ENHANCING AUTOMATIC INVOICE CODING PERFORMANCE WITH UNSTRUCTURED DATA

… The published word vectors are 300 dimensional and trained on the internet data from Common Crawl3. The pretrained word vectors can be further fine-tuned using the provided code and datasets more similar to the final usage scenario to increase …


Enhancing Automatic Keyphrase Labelling with Text-to-Text Transfer Transformer (T5) Architecture: A Framework for Keyphrase Generation and Filtering

J Gabín, ME Ares, J Parapar - arXiv preprint arXiv:2409.16760, 2024

Automatic keyphrase labelling stands for the ability of models to retrieve words or short phrases that adequately describe documents' content. Previous work has put much effort into exploring extractive techniques to address this task; however, these …


Enhancing Bias Assessment for Complex Term Groups in Language Embedding Models: Quantitative Comparison of Methods

M Gray, M Milanova, L Wu - JMIR Medical Informatics, 2024

… 300d” from the Stanford NLP Group [15], which was trained on 840 billion tokens from Common Crawl data, has a vocab size of 2.2 million, and generates 300-dimension vectors. Moreover, other input embedding methods, including those of BERT [12], Sci-BERT …


Enhancing Computer Programming Education with LLMs: A Study on Effective Prompt Engineering for Python Code Generation

T Wang, N Zhou, Z Chen - arXiv preprint arXiv:2407.05437, 2024

Large language models (LLMs) and prompt engineering hold significant potential for advancing computer programming education through personalized instruction. This paper explores this potential by investigating three critical research questions: the …


Enhancing Cyber Security Enhancement Through Generative AI

… GPT-1: Released in 2018, GPT-1 was trained on the Common Crawl and BookCorpus datasets. It demonstrated proficient language comprehension but limited conversational coherence and retention of context. GPT-2: Trained on …


Enhancing Cybersecurity: A Review and Comparative Analysis of Convolutional Neural Network Approaches for Detecting URL-Based Phishing Attacks

M Nanda, M Saraswat, PK Sharma - e-Prime-Advances in Electrical Engineering …, 2024

Phishing attempts to mimic the official websites of businesses, including banks, e-commerce, government offices, and financial institutions. Phishing websites aim to collect and retrieve sensitive data from users, including passwords, credit card numbers, email …


Enhancing Ecological Knowledge Discovery Using Large Language Models

Earth is home to an astonishing diversity of life forms. Among these, vascular plants stand as one of the most vital and ubiquitous groups, accounting for approximately 80% of global biomass [3]. The census of vascular plants, which presently exceeds …


ENHANCING EDUCATIONAL CHATBOTS WITH RETRIEVAL-AUGMENTED GENERATION SYSTEMS: A STUDY ON PHYSICS AND MATHEMATICS COURSES

H Monteiro, H Mokayed - ICERI2024 Proceedings, 2024

The integration of Large Language Models (LLMs) with educational tools offers significant potential to enhance student learning by providing tailored, contextually accurate responses through chatbots. This thesis investigates the implementation of …


Enhancing English Translation Quality Assessment through Knowledge Transfer in Artificial Intelligence Context

Abstract Machine translation technology, which employs computers to autonomously convert text between source and target languages, represents a pivotal realm within artificial intelligence and natural language processing research. This paper …


ENHANCING JAPANESE LEXICAL NETWORKS USING LARGE LANGUAGE MODELS

This study presents an innovative approach to crafting and enhancing Japanese lexical networks by incorporating large language models (LLMs), especially GPT-4o, utilizing data from Matsushita’s (2011) Vocabulary Database for Reading Japanese …


Enhancing knowledge graphs with microdata and LLMs: the case of Schema. org and Wikidata in touristic information

L Gonzalez-Garcia, G González-Carreño… - The Electronic Library, 2024

… We have used the Common Crawl (CC) corpus of crawled Web pages to extract metadata published in the wild, and then used a mapping of Schema.org annotations to Wikidata to get estimates of the added value (in terms of additional …


Enhancing Large Language Models for Structured Data Extraction

This thesis examines the application of large language models for structured data extraction using microdata-annotated webpages from the Web Data Commons (WDC). It compares the performance of models trained on WDC data with those trained on …


Enhancing Large Language Models with Stochastic Multi-Level Embedding Fusion: An Experimental Approach on Open-Source LLM

H Raines, E Ferreira, L Fitzwilliam, B Everson… - Authorea Preprints, 2024

… The experiment utilized the Common Crawl dataset for training, given its extensive coverage of diverse web-based text, which offered a wide variety of linguistic structures and semantic contexts. The dataset was particularly valuable for …


Enhancing misogyny detection in bilingual texts using explainable AI and multilingual fine-tuned transformers

E Hashmi, SY Yayilgan, MM Yamin, M Ullah - Complex & Intelligent Systems, 2025

… It boasts a vast dictionary containing 2 million words from the Common Crawl database, with each word represented in a 300-dimensional vector space, totaling an impressive 600 billion word vectors. In our research, we have employed both the …


Enhancing Misogyny Detection in Bilingual Texts Using FastText and Explainable AI

E Hashmi, MM Yamin, S Imran, SY Yayilgan, M Ullah - 2024 International Conference …, 2024

… It features an extensive dictionary with 2 million words drawn from the Common Crawl database. Every word is depicted in a 300-dimensional vector space, amassing an enormous total of 600 billion word vectors. 1) Unsupervised FastText …


Enhancing Multilingual Hate Speech Detection: From Language-Specific Insights to Cross-Linguistic Integration

E Hashmi, SY Yayilgan, IA Hameed, MM Yamin… - IEEE Access, 2024

… It features a vast vocabulary of 2 million words from the Common Crawl dataset, mapping each to a 300-dimensional vector space. What … This model was built using Common Crawl and Wikipedia using FastText’s unsupervised learning …


Enhancing News Articles: Automatic SEO Linked Data Injection for Semantic Web Integration

H Salem, H Salloum, K Sabbagh, M Mazzara - 2024

This paper presents a novel solution aimed at enhancing news web pages for seamless integration into the Semantic Web. By utilizing advanced pattern mining techniques alongside OpenAI’s GPT-3, we rewrite news articles to improve their …


Enhancing Parameter Efficiency in Model Inference Using an Ultralight Inter-Transformer Linear Structure

… It was constructed from the Chuweb21 generated based on the April 2021 block of the Common Crawl dataset. The WWW-4 dataset comprises 50 queries. Robust04 dataset is also an English web corpus generated from TREC disks 4 and 5, and …


Enhancing Phishing Detection, Leveraging Deep Learning Techniques

A Ullah, RA Shah, SA Nawaz, N Ahmad, MH Malik - Journal of Computing & …, 2024

… To achieve this, we utilized the Common Crawl Corpus, which aggregates extensive data collected over seven years. This corpus comprises raw web page data, metadata, and textual content. Legitimate domains with substantial backlink …


Enhancing pre-trained models for text summarization: a multi-objective genetic algorithm optimization approach

GB Mohan, RP Kumar, R Elakkiya - Multimedia Tools and Applications, 2024

… A significant aspect of their work was the introduction of C4, a substantial text corpus derived from Common Crawl, utilized in several of their model variations. During its pre-training phase, T5 employed an intriguing technique of masking …


Enhancing Product Design through AI-Driven Sentiment Analysis of Amazon Reviews Using BERT

MK Shaik Vadla, MA Suresh, VK Viswanathan - Algorithms, 2024

Understanding customer emotions and preferences is paramount for success in the dynamic product design landscape. This paper presents a study to develop a prediction pipeline to detect the aspect and perform sentiment analysis on review …


Enhancing Sentiment Analysis Accuracy in Borobudur Temple Visitor Reviews through Semi-Supervised Learning and SMOTE Upsampling

C Agustina, P Purwanto, F Farikhin - Journal of Advances in Information Technology, 2024

The level of visitor satisfaction with tourist destinations can be known from reviews on social media. One method used is to carry out sentiment analysis on comments given by visitors on social media or related websites. This study was envisioned as a …


Enhancing Sentiment Analysis on Social Media Data with Advanced Deep Learning Techniques.

HH Nguyen - International Journal of Advanced Computer Science & …, 2024

This paper introduces a comprehensive methodology for conducting sentiment analysis on social media using advanced deep learning techniques to address the unique challenges of this domain. As digital platforms play an increasingly pivotal …


Enhancing Text Summarization in a Limited Data and Long Sample Setting

This thesis addresses the challenges of automatic text summarization in a limited amount of data and a significantly large sample size setting. The study introduces a novel dataset of Ukrainian legislative documents professionally translated into …


Enhancing Text Summarization in a Limited Data and Long Sample

In our rapidly digitizing world, the volume of online information is growing at an unprecedented rate. Legal systems worldwide do not lag behind and generate massive amounts of documents daily. Lawyers are tasked with navigating an ever-growing …


Enhancing Vision-Language Pre-training with Rich Supervisions

Y Gao, K Shi, P Zhu, E Belval, O Nuriel, S Appalaraju… - arXiv preprint arXiv …, 2024

… This is expected as our data collected from Common Crawl has different distribution against the original Pix2struct data due to the data size (2M vs. 80M) and potentially different website filtering strategies. Specifically, we filter out website …


Enriching Interactive Explanations with Fuzzy Temporal Constraint Networks

M Canabal-Juanatey, JM Alonso-Moral, A Catala… - International Journal of …, 2024

… such as BERT [2] (Bidirectional Encoder Representations from Transformers), GPT [26] (Generative Pre-trained Transformer) and their successors (ALBERT [4], RoBERTa [3], BART [7], GPT-3 [5] or GPT-4 [27]), which were trained with huge …


Enriching satellite image annotations of forests with keyphrases from a specialized corpus

N Neptune, J Mothe - Multimedia Tools and Applications, 2024

… We test fastText embeddings trained on Common Crawl Footnote 4 and Wikipedia Footnote 5 with dimension 300 and a context size of 5 … These fastText models were pre-trained on Common Crawl and Wikipedia. None of the models was …


Ensemble based high performance deep learning models for fake news detection

M E. Almandouh, MF Alrahmawy, M Eisa, M Elhoseny… - Scientific Reports, 2024

… It boasts an extensive lexicon of 2 million words obtained from Common Crawl. Each word is embedded in a 300-dimensional vector space, resulting in a comprehensive library of 600-billion-word vectors. FastText architecture, as seen in Fig. …


Entrant: A large financial dataset for table understanding

E Zavitsanos, D Mavroeidis, E Spyropoulou… - Scientific Data, 2024

Tabular data is a way to structure, organize, and present information conveniently and effectively. Real-world tables present data in two dimensions by arranging cells in matrices that summarize information and facilitate side-by-side comparisons …


Entropy and type-token ratio in gigaword corpora

P Rosillo-Rodes, MS Miguel, D Sanchez - arXiv preprint arXiv:2411.10227, 2024

Lexical diversity measures the vocabulary variation in texts. While its utility is evident for analyses in language change and applied linguistics, it is not yet clear how to operationalize this concept in a unique way. We here investigate entropy and text-token …


EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge

X Zhao, Q Zhao, T Tanaka - arXiv preprint arXiv:2401.05908, 2024

With large training datasets and massive amounts of computing sources, large language models (LLMs) achieve remarkable performance in comprehensive and generative ability. Based on those powerful LLMs, the model fine-tuned with domain-specific …


Error Patterns in Vision Transformers: A Focused Study on InternVL's Real-World Performance

This study focuses on the errors in Vision Transformers, particularly examining the realworld performance of InternVL, an opensource multimodal model designed to rival commercial systems. The primary tasks explored include image captioning and …


Establishing Global AI Accountability: Training Data Transparency, Copyright, and Misinformation

As artificial intelligence (AI) technologies continue advancing at a rapid pace, the systems' growing capabilities as well as their expanding integration into vital social functions are raising complex questions around trust and accountability. AI models …


Establishing the importance of co-creation and self-efficacy in creative collaboration with artificial intelligence

J McGuire, D De Cremer, T Van de Cruys - Scientific Reports, 2024

The emergence of generative AI technologies has led to an increasing number of people collaborating with AI to produce creative works. Across two experimental studies, in which we carefully designed and programmed state-of-the-art human–AI …


Ethical aspects of ChatGPT: An approach to discuss and evaluate key requirements from different ethical perspectives

M Steen, J Greeff, M Boer, C Veenman - AI and Ethics, 2024

… Many LLMs have been trained with data from Common Crawl, Footnote 9 which contains inaccuracies, errors, and bias. Another concern is whether people can access ChatGPT and, for example, change parameters or delete data, so that the …


Ethical Considerations for Generative AI in Social Science Research

BA Brown, KL Heitner - Generative AI and Implications for Ethics, Security, and …, 2024

Social science research embodies the inquiry into people as individuals and their interpersonal interactions with each other in communities and varied societies, with due consideration for their natural, technological, and constructed environments …


Ethical recommendations for Artificial Intelligence technology in the Geological Sciences-with a focus on Language Models

PH Cleverley - Journal of Geoethics and Social Geosciences, 2024

… Common Crawl, an open and up to date dataset of large swaths of the Internet has been freely available for several years and is used by most foundation LLMs and also some domain specific ones3. Over recent years, the amount of freely …


Ethics of AI in the Teaching of English

New technologies have always impacted literacy, and artificial intelligence (AI) is no exception (Tyner, 1998). English teachers find themselves having to consider how AI may impact how they teach. NCTE’s Beliefs for Integrating Technology Into the …


EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation

AL Tonja, IA Azime, TD Belay, MG Yigezu… - arXiv preprint arXiv …, 2024

Large language models (LLMs) have gained popularity recently due to their outstanding performance in various downstream Natural Language Processing (NLP) tasks. However, low-resource languages are still lagging behind current state-of-the-art …


Euphemism in AI Discourse as Colonial Harm & Effect: Learnings from Subaltern Puerto Rico

Through three integrated studies, this dissertation explores complex human-AI interactions by comparing the perceptions of two groups of US citizens: subaltern Puerto Ricans and stateside Americans’ interactions with ChatGPT. This corpus-assisted …


Evading VBA Malware Classification using Model Extraction Attacks and Stochastic Search Methods

B Fehrman, F Akowuah, R Hoover - 2024 Cyber Awareness and Research …, 2024

Antivirus (AV) software that relies on learning-based methods is potentially vulnerable to adversarial attacks from threat actors. Threat actors can utilize model-extraction attacks against AV software to create a surrogate model. Malware samples can be …