Lingualyzer: A computational linguistic tool for multilingual and multidimensional text analysis

GM Linders, MM Louwerse - Behavior Research Methods, 2023

… resources utilize large amounts of publicly available data in many languages, such as data coming from Wikipedia and Common Crawl. … We used the 300 dimensional fastText word vectors, which were created from Wikipedia and Common …


Linking Representations with Multimodal Contrastive Learning

A Arora, X Yang, SY Jheng, M Dell - arXiv preprint arXiv:2304.03464, 2023

… [52] find that GPT-3 performs substantially better on samples from ReCoRD that have near duplicates in RealNews, a subset of Common Crawl included in the GPT-3 training data, compared to those that did not. While most deduplication of noisy text …


Literature Review: Challenges In Creating A Generic Model For Text Classification In Multiple Languages

R Sinha, R Srivastava - resmilitaris, 2023

… Authors [60] used a dataset that included Wikipedia and Common Crawl to train the Skip-gram and CBOW word vector models. They also developed a quick language identifier that recognizes 176 different languages. While researchers [61]evaluated …


LLaMA 2: The New Open Source Language Model

Language models have revolutionized the field of artificial intelligence, by enabling machines to understand and generate human-like text. Among the plethora of language models, LLaMA (Large Language Models AI) stands out as the latest …


LLaMA: Open and Efficient Foundation Language Models

H Touvron, T Lavril, G Izacard, X Martinet, MA Lachaux… - arXiv preprint arXiv …, 2023

… We preprocess five CommonCrawl dumps, ranging from 2017 to 2020, with the CCNet pipeline (… We expect these biases to come from CommonCrawl despite multiple filtering steps. … This allowed to train a 5-gram model on 975 billions tokens …


LLEMMA: AN OPEN LANGUAGE MODEL FOR MATHEMATICS

Z Azerbayev, H Schoelkopf, K Paster, M Dos Santos… - Minerva

We present LLEMMA, a large language model for mathematics. We continue pretraining Code Llama on Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding LLEMMA. On the MATH …


LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked

A Helbling, M Phute, M Hull, DH Chau - arXiv preprint arXiv:2308.07308, 2023

Large language models (LLMs) have skyrocketed in popularity in recent years due to their ability to generate high-quality text in response to human prompting. However, these models have been shown to have the potential to generate harmful content in …


LLM's for Spanish Song Text Analysis and Classification Using Language Variants

O García-Vázquez, T Alcántara, H Calvo, G Sidorov - … International Conference on …, 2023

… The results show that tweets are the best for song lyrics analysis, while Google News and Common Crawl are the best for movie analysis, since the vocabulary that used in these portals is very similar in both cases. Models trained with GLoVe …


LLM-augmented Preference Learning from Natural Language

I Kang, S Ruan, T Ho, JC Lin, F Mohsin, O Seneviratne… - arXiv preprint arXiv …, 2023

… The authors query for sentences that contain mentions of both alternatives from the Common Crawl dataset and present a final dataset of 7,199 sentences with 217 unique pairs of alternatives and 3 classes – 𝐴 ≻ 𝐵, 𝐴 ≺ 𝐵, N/A. Because of the …


LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

Z Liu, B Oguz, C Zhao, E Chang, P Stock, Y Mehdad… - arXiv preprint arXiv …, 2023

Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investigate quantization …


LLM-RM at SemEval-2023 Task 2: Multilingual Complex NER using XLM-RoBERTa

R Mehta, V Varma - arXiv preprint arXiv:2305.03300, 2023

Named Entity Recognition(NER) is a task of recognizing entities at a token level in a sentence. This paper focuses on solving NER tasks in a multilingual setting for complex named entities. Our team, LLM-RM participated in the recently organized …


LLM360: Towards Fully Transparent Open-Source LLMs

Z Liu, A Qiao, W Neiswanger, H Wang, B Tan, T Tao… - arXiv preprint arXiv …, 2023

The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model …


LLMs: A Promising New Tool for Improving Healthcare in Low-Resource Nations

A Gangavarapu - 2023 IEEE Global Humanitarian Technology …, 2023

This paper explores the potential of large language models (LLMs) in addressing healthcare inequalities, particularly in underserved nations with provider shortages, limited resources, and funding constraints. The UN’s Sustainable Development Goal …


LMU at HaSpeeDe3: Multi-Dataset Training for Cross-Domain Hate Speech Detection

We describe LMU Munich’s hate speech detection system for participating in the cross-domain track of the HaSpeeDe3 shared task at EVALITA 2023. The task focuses on the politics and religion domains, having no in-domain training data for …


Logical definition-based identification of potential missing concepts in SNOMED CT

X Hao, R Abeysinghe, K Roberts, L Cui - BMC Medical Informatics and Decision …, 2023

Biomedical ontologies are representations of biomedical knowledge that provide terms with precisely defined meanings. They play a vital role in facilitating biomedical research in a cross-disciplinary manner. Quality issues of biomedical …


Long Form Question Answering Dataset Creation for Business Use Cases using Noise-Added Siamese-BERT

T Cekiç, Y Manav, B Helvacıoglu, EB Dündar, O Deniz… - 2022

… To generate related documents, they use a TFIDF retriever on the CommonCrawl web dump, to retrieve related documents and combine all documents related parts to generate one supporting document.In the KILT (Petroni et al., 2021) benchmark …


LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery

T Chen, T Ding, B Yadav, I Zharkov, L Liang

Large Language Models (LLMs) have transformed the landscape of artificial intelligence, while their enormous size presents significant challenges in terms of computational costs. We introduce LoRAShear, a novel efficient approach to …


Lost in Translation: Large Language Models in Non-English Content Analysis

G Nicholas, A Bhatia - arXiv preprint arXiv:2306.07377, 2023

In recent years, large language models (eg, Open AI's GPT-4, Meta's LLaMa, Google's PaLM) have become the dominant approach for building AI systems to analyze and generate language online. However, the automated systems that …


Low Resource Summarization using Pre-trained Language Models

M Munaf, H Afzal, N Iltaf, K Mahmood - arXiv preprint arXiv:2310.02790, 2023

With the advent of Deep Learning based Artificial Neural Networks models, Natural Language Processing (NLP) has witnessed significant improvements in textual data processing in terms of its efficiency and accuracy. However, the research is mostly …


Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin

PJ Lin, M Saeed, E Chang, M Scholman - arXiv preprint arXiv:2307.00382, 2023

Developing effective spoken language processing systems for low-resource languages poses several challenges due to the lack of parallel data and limited resources for fine-tuning models. In this work, we target on improving upon both text …


Low-resource image captioning

… The language model is pretrained on the colossal clean crawled corpus (C4), which is a cleaned version of the common crawl dataset [70] and consists of 700 GB of unlabelled text. The cleaning applications include removing non-English text …


LSTM-Based Encoder–Decoder Attention Model for Text Translation and Simplification on the Constitution of India

M Navlakha, R Lobo, R Bhargava, RB Karani - … in Computing: ICRIC 2022, Volume 2, 2023

Natural language processing techniques can be used on judicial and legislative documents like the Constitution for making it more accessible to the general audience. Various approaches such as Natural Machine Translation (NMT), text …


LT at SemEval-2023 Task 1: Effective Zero-Shot Visual Word Sense Disambiguation Approaches using External Knowledge Sources

… Further, we assessed the suitability of three external knowledge sources: Wikipedia, a large-scale English Common Crawl corpus, and the multi-modal knowledge graph VisualSem. Our best-performing approach involved the Common …


M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

W Zhang, SM Aljunied, C Gao, YK Chia, L Bing - arXiv preprint arXiv:2306.05179, 2023

… We rank languages by their ratio in the CommonCrawl corpus, which is a widely-used data source for training LLMs. It can be observed that our selected languages span a wide range, from high-resource languages like English and Chinese, to extremely …


MA-INF 4328

… • GeoVectors adopts the 300-dimensional English word vectors trained on the Common Crawl, and Wikipedia …


MACEDONIZER-The Macedonian Transformer Language Model

J Dobreva, T Pavlov, K Mishev, M Simjanoska… - International Conference on …, 2022

Contextualized language models are becoming omnipresent in the field of Natural Language Processing (NLP). Their learning representation capabilities show dominant results in almost all downstream NLP tasks. The main challenge that low-resource …


Machine learning and Bayesian inference in nuclear fusion research: an overview

A Pavone, A Merlo, S Kwak, J Svensson - Plasma Physics and Controlled Fusion, 2023

This article reviews applications of Bayesian inference and machine learning in nuclear fusion research. Current and next-generation nuclear fusion experiments require analysis and modelling efforts that integrate different models consistently …


Machine Learning and Collective Unintelligence

E Fournier-Tombs - Gender Reboot: Reprogramming Gender Rights in the …, 2023

… The Common Crawl data, which is a sizable collection of articles pulled from the Internet (as its name suggests), is well known for having significant gender bias. WebText is a very comparable data source that was also created by OpenAI by web …


Machine Learning and Knowledge Discovery in Databases: Research Track: European Conference, ECML PKDD 2023, Turin, Italy, September 18–22, 2023 …

D Koutra, C Plant, MG Rodriguez, E Baralis, F Bonchi - 2023

The multi-volume set LNAI 14169 until 14175 constitutes the refereed proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2023, which took place in Turin, Italy, in September 2023 …


Machine Learning for Malicious URL Classification: A Temporal Analysis

When Machine Learning (ML) is applied to labelled datasets, it is standard practice that 70% of the dataset is used for training models, with 30% used for testing. That approach assumes the dataset does not change over time. This praxis makes the …


MACHINE LEARNING IN CYBERSECURITY: COMPREHENSIVE ANALYSIS AND DETECTION OF URL PHISHING ATTACKS

B Patel - Journal of Research Administration, 2023

In the face of escalating cyber threats, particularly phishing attacks, this research provides a comprehensive analysis of machine learning techniques for effective phishing URL detection. Leveraging a meticulously curated dataset comprising …


Machine Learning Insights into Multifaceted Social Impacts on Global Urban Slum Population

The growth of the urban slum population proportion highlights the extent of urban inequality. This article uses machine learning to surpass the limitations of traditional static linear regression models. By handling large data volumes and uncovering …


Machine Learning Model for Paraphrases Detection Based on Text Content Pair Binary Classification

N Kholodna, V Vysotska, O Markiv, S Chyrun

This article dwells process of ML-model development for detecting paraphrasing by binary classification of texts pair. For this study, the following semantic similarity metrics or indicators have been selected as features: Jacquard coefficient for shared …


Machine Learning on Knowledge Graphs for Health applications.

In the past decade, many impressive results have been reported for machine learning (ML) tools that are developed to assist in clinical decision making. However, translating these largely theoretical achievements to realistic settings remains a …


Machine Learning-Based Phishing Detection Using URL Features: A Comprehensive Review

AUZ Asif, H Shirazi, I Ray - International Symposium on Stabilizing, Safety, and …, 2023

… Common Crawl is a large-scale web crawl that is made up of petabytes of data that have been collected since 2008. It includes raw web page data, extracted metadata, and text extractions. This repository’s material is maintained in Web …


Machine translation and its evaluation: a study

SK Mondal, H Zhang, HMD Kabir, K Ni, HN Dai - Artificial Intelligence Review, 2023

Abstract Machine translation (namely MT) has been one of the most popular fields in computational linguistics and Artificial Intelligence (AI). As one of the most promising approaches, MT can potentially break the language barrier of people from all over …


Machine visions: Mapping depictions of machine vision through critical image synthesis

R Carter, RA Carter - Open Library of Humanities, 2023

This paper conducts a speculative examination of how AI image synthesisers, which generate novel imagery in response to inputted textual prompts – such as DALL-E, Midjourney, and Stable Diffusion – can be employed reflexively to investigate …


Machine-Assisted Social Psychology Hypothesis Generation

S Banker, P Chatterjee, H Mishra, A Mishra - 2023

… GPT-3 was developed by OpenAI and has been trained on several large text corpora such as Common Crawl, Wikipedia, digitized books, WebText2 (which is based on Reddit posts), etc., amounting to about 45 terabytes of training data. GPT-3 …


MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

S Kudugunta, I Caswell, B Zhang, X Garcia… - arXiv preprint arXiv …, 2023

… We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in …


MahaEmoSen: Towards Emotion-aware Multimodal Marathi Sentiment Analysis

P Chaudhari, P Nandeshwar, S Bansal, N Kumar - ACM Transactions on Asian and …, 2023

… It is a BERT-based model trained on PMINDIA [13], commoncrawl [14], Wikipedia3, Dakshina corpus [15] for 17 diferent Indic languages like Hindi, Marathi, Tamil etc. It is also trained on … Total of 100 languages from CommonCrawl data were used as …


Make PDF safe again

Malware is an ever-present problem, and while most people are aware that executable files can be dangerous, other file formats are deemed as trustworthy and safe. One of these file formats is PDF. Billions of PDF files are created each day and …


Making Changes in Webpages Discoverable: A Change-Text Search Interface for Web Archives

L Frew, ML Nelson, MC Weigle - arXiv preprint arXiv:2305.00546, 2023

Webpages change over time, and web archives hold copies of historical versions of webpages. Users of web archives, such as journalists, want to find and view changes on webpages over time. However, the current search interfaces for web …


Man vs the machine: The Struggle for Effective Text Anonymisation in the Age of Large Language Models

C Patsakis, N Lykousas - arXiv preprint arXiv:2303.12429, 2023

… According to OpenAI, the training data for GPT-3 includes various sources such as books, articles, and websites, with a primary source being the Common Crawl2, a repository of web pages and documents that is regularly updated and maintained …


Man vs. Machine: An applied study comparing a man-made lexicon, a machine learned lexicon, and OpenAI's GPT for sentiment analysis.

Sentiment analysis, at scale, has become an essential tool in the methodological toolbox of finance. In this thesis, we construct a sentiment lexicon using a supervised machine learning model by Taddy (2013) and compare it to the traditional finance …


Mapping Language Literacy At Scale: A Case Study on Facebook

YR Lin, S Wu, W Mason - arXiv preprint arXiv:2303.12179, 2023

Literacy is one of the most fundamental skills for people to access and navigate today's digital environment. This work systematically studies the language literacy skills of online populations for more than 160 countries and regions across the world …


MARGEN: Marathi Question Answering Generative Conversation Model

SV Bhalshankar, RR Deshmukh - … Conference on Applications of Machine Intelligence …, 2023

The conversational system aka chatbot market capture was worth USD 526 million in 2021 around the world. The innovations created such as Machine learning, Deep learning, Natural Language Processing (NLP), and Big data analytics have given a …


Massively Multi-Lingual Event Understanding: Extraction, Visualization, and Search

C Jenkins, S Agarwal, J Barry, S Fincke, E Boschee - arXiv preprint arXiv:2305.10561, 2023

In this paper, we present ISI-Clear, a state-of-the-art, cross-lingual, zero-shot event extraction system and accompanying user interface for event visualization & search. Using only English training data, ISI-Clear makes global events available on-demand …


MatSciRE: Leveraging pointer networks to automate entity and relation extraction for material science knowledge-base construction

A Mullick, A Ghosh, GS Chaitanya, S Ghui, T Nayak… - Computational Materials …, 2024

Material science literature is a rich source of factual information about various categories of entities (like materials and compositions) and various relations between these entities, such as conductivity, voltage, etc. Automatically extracting …


Mazen ALSAREM

In our knowledge-driven society, the acquisition and the transfer of knowledge play a principal role. Web search engines are somehow tools for knowledge acquisition and transfer from the web to the user. The search engine results page (SERP) …


MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods

M Finkelstein, M Freitag - arXiv preprint arXiv:2309.10966, 2023

… For that, we take a random sample of 200,000 English sentences from Common Crawl1. … Phase 2 MBR and QE finetuning from the base checkpoint on the larger Common Crawl … on either self-QE-Common-Crawl or self-MBRCommon-Crawl …


MCPG: A Flexible Multi-Level Controllable Framework for Unsupervised Paraphrase Generation

Y Chen, H Jiang, L Liu, R Wang, S Shi, R Xu - Findings of the Association for …, 2022

We present MCPG: a simple and effectiveapproach for controllable unsupervised paraphrase generation, which is also flexible toadapt to specific domains without extra training. MCPG is controllable in different levels: local lexicons, global …


MCTS: A Multi-Reference Chinese Text Simplification Dataset

R Chong, L Lu, L Yang, J Nie, S Zhou, Y Li, E Yang - arXiv preprint arXiv:2306.02796, 2023

Text simplification aims to make the text easier to understand by applying rewriting transformations. There has been very little research on Chinese text simplification for a long time. The lack of generic evaluation data is an essential reason for this …


Measuring and Evading Turkmenistan's Internet Censorship

SNVTX Jiang, K Bock, N Feamster, NP Hoang, D Levin - 2023

… As shown in Figure 2, the payload of our probes contains domains curated from the Citizen Lab lists [5], the full Tranco list [42], and Common Crawl Project [8]. Due to limited resources of our VPS, we opt to probe the frst 10M FQDNs ranked by the …


Measuring and Evading Turkmenistan's Internet Censorship: A Case Study in Large-Scale Measurements of a Low-Penetration Country

S Nourin, V Tran, X Jiang, K Bock, N Feamster… - arXiv preprint arXiv …, 2023

… [5], the full Tranco list [42] of most popular websites, and Common Crawl Project [8]. Due to limited resources of our VPS, we opt to probe the first 10M FQDNs ranked by the Common Crawl Project instead of the full list of almost 400M FQDNs. The …


Measuring Gender Bias in West Slavic Language Models

S Martinková, KSI Augenstein - arXiv preprint arXiv:2304.05783, 2023

Pre-trained language models have been known to perpetuate biases from the underlying datasets to downstream tasks. However, these findings are predominantly based on monolingual language models for English, whereas there …


Measuring Normative and Descriptive Biases in Language Models Using Census Data

S Touileb, L Øvrelid, E Velldal - arXiv preprint arXiv:2304.05764, 2023

We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment …


Medical knowledge-enhanced prompt learning for diagnosis classification from clinical text

Y Lu, X Zhao, J Wang - Proceedings of the 5th Clinical Natural Language …, 2023

Artificial intelligence based diagnosis systems have emerged as powerful tools to reform traditional medical care. Each clinician now wants to have his own intelligent diagnostic partner to expand the range of services he can provide. When reading a …


MedMine: Examining Pre-trained Language Models on Medication Mining

H Alrdahi, L Han, H Šuvalov, G Nenadic - arXiv preprint arXiv:2308.03629, 2023

Automatic medication mining from clinical and biomedical text has become a popular topic due to its real impact on healthcare applications and the recent development of powerful language models (LMs). However, fully-automatic …


MEE4 and XLsim: IIIT HYD's Submissions' for WMT23 Metrics Shared Task

A Mukherjee, M Shrivastava - Proceedings of the Eighth Conference on Machine …, 2023

This paper presents our contributions to the WMT2023 shared metrics task, consisting of two distinct evaluation approaches: a) Unsupervised Metric (MEE4) and b) Supervised Metric (XLSim). MEE4 represents an unsupervised, reference-based …


Meet in the Middle: A New Pre-training Paradigm

A Nguyen, N Karampatziakis, W Chen - arXiv preprint arXiv:2303.07295, 2023

… • CC-Stories contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB). • CC-100 is a dataset extracted from CommonCrawl snapshots between January 2018 and December 2018, filtered to …


MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

L Yu, D Simig, C Flaherty, A Aghajanyan, L Zettlemoyer… - arXiv preprint arXiv …, 2023

Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Megabyte, a multi-scale decoder architecture that enables end-to-end …


MELA: Multilingual Evaluation of Linguistic Acceptability

Z Zhang, Y Liu, W Huang, J Mao, R Wang, H Hu - arXiv preprint arXiv:2311.09033, 2023

Recent benchmarks for Large Language Models (LLMs) have mostly focused on application-driven tasks such as complex reasoning and code generation, and this has led to a scarcity in purely linguistic evaluation of LLMs. Against this background …


Memorization of Named Entities in Fine-Tuned BERT Models

A Diera, N Lell, A Garifullina, A Scherp - International Cross-Domain Conference for …, 2023

… During prompt creation, we sampled a string with a character length of 100 either from the Common Crawl dataset (naive prompting) or from the test set (informed prompting). We set the sequence length to 256 tokens. We removed the tokens of …


Method for Sentiment Analysis of Ukrainian-Language Reviews in E-Commerce Using RoBERTa Neural Network

O Zalutska, M Molchanova, O Sobko, O Mazurets… - 2023

… This layer is based on the pre-trained model "xlm_roberta_multi_cased_L-12_H-768_A-12" [41], which is the result of unsupervised crosslanguage representative training at scale (XLM-RoBERTa) [41] and is pre-trained on 2.5 TB of filtered CommonCrawl …


Methods for Leveraging Auxiliary Signals for Low-Resource NLP

… The standard pre-trained word vectors used for English are the GloVe [69] ones trained on 840 billion tokens of Common Crawl data1, while for other languages, we rely on the Facebook fastText Wikipedia embeddings [76] as input representations …


Methods for Text Generation in NLP

S Matwin, A Milios, P Prałat, A Soares, F Théberge - Generative Methods for Social …, 2023

… RoBERTa incorporates far more training data than the original BERT, using the Book Corpus and English Wikipedia datasets from the original paper, but adds several additional datasets for further training (CommonCrawl News, CommonCrawl Stories 6…


Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection

T Ogunremi, D Jurafsky, CD Manning - Findings of the Association for Computational …, 2023

With the prominence of large pretrained language models, low-resource languages are rarely modelled monolingually and become victims of the “curse of multilinguality” in massively multilingual models. Recently, AfriBERTa showed that training …


Mining Electronic Health Records: Turning Unstructured Text into Research Data Using Natural Language Processing

The electronic health record (EHR) contains a detailed account of the patients’ medical history and is an important source of clinical research data. It is estimated that 80% of the information in the EHR is unstructured and this complicates locating …


Mining Parallel Sentences from Internet with Multi-view Knowledge Distillation for Low-resource Language Pairs

… For the future, we will increase the number of mining data by crawling common crawl platform and focus on more low-resource language pairs. The large amount of parallel data also attracts interesting attention how to use it best. Therefore, we will …


Misinformation Detection in the Wild: News Source Classification as a Proxy for Non-article Texts

M Bohacek - Proceedings of the Second Workshop on NLP for …, 2022

… To first obtain URLs of sites with individual articles from those domains, we used the Commoncrawl API 2. We specified for the API to include only articles discovered between January 2019 and September 2021. Once these were obtained, we …


ML and DL-based Phishing Website Detection: The Effects of Varied Size Datasets and Informative Feature Selection Techniques

K Adane, B Beyene, M Abebe - Journal of Artificial Intelligence and Technology, 2023

One must interact with a specific webpage or website in order to use the Internet for communication, teamwork, and other productive activities. However, because phishing websites look benign and not all website visitors have the same knowledge …


Modeling Brain Representations of Words' Concreteness in Context Using GPT‐2 and Human Ratings

A Bruera, Y Tao, A Anderson, D Çokal, J Haber… - Cognitive Science, 2023

The meaning of most words in language depends on their context. Understanding how the human brain extracts contextualized meaning, and identifying where in the brain this takes place, remain important scientific challenges. But technological and …


Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval

S Zhang, Y Liang, M Gong, D Jiang, N Duan - arXiv preprint arXiv:2302.01626, 2023

… (2019), we collect a clean version of Common Crawl (CC) including a 2,500GB multi-lingual corpus covering 108 languages, which adopt the same pre-processing method as XLMR (Conneau et al., 2019). Note that we only train on CC without any …


Modelling an Efficient URL Phishing Detection Approach Based on a Dense Network Model.

AA Tenis, R Santhosh - Computer Systems Science & Engineering, 2023

… The PhishTank provides the URLs for phishing to be gathered, and the Common Crawl provides the legal URLs. The LSTM technique … The repositories of Common Crawl and PhishTank provide the dataset to be gathered. A deep belief network (DBN) …


Modern Applications With a Focus on Training ChatGPT and GPT Models: Exploring Generative AI and NLP

I Kondurkar, A Raj, D Lakshmi - Advanced Applications of Generative AI and Natural …, 2024

Generative AI (GAI) and natural language processing (NLP) have emerged as the most exciting and rapidly growing fields in artificial intelligence (AI). This book chapter provides a comprehensive exploration of the advanced applications of GAI …


Modular DiffPruning for Bias Mitigation

… They can leverage unlabelled text data which is available in large amounts on the internet and is collected by projects such as common crawl.Unsurprisingly raw text data crawled from the internet often contains bias towards different groups [3, 4] …


ModuleFormer: Learning Modular Large Language Models From Uncurated Data

Y Shen, Z Zhang, T Cao, S Tan, Z Chen, C Gan - arXiv preprint arXiv:2306.04640, 2023

Large Language Models (LLMs) have achieved remarkable results. But existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond pre-training data without forgetting previous knowledge. This …


MONITORING FILE FORMAT OBSOLESCENCE IN REPOSITORIES: An applied method

… Common Crawl and data from different institutional repositories (Netherlands Institute for Sound and Vision and Data Archiving and Networked Services (DANS)). The Common Crawl … and this prevents the need to process all the Common Crawl …


Mono-and Multilingual GPT-3 Models for Hungarian

ZG Yang, LJ Laki, T Váradi, G Prószéky - … Conference on Text, Speech, and Dialogue, 2023

… Common Crawl (CC): Most of our Hungarian text was collected from the Common Crawl database. Since the Webcorpus 2.0 contains text only until April of 2019, we collected the data that was created afterwards. For downloading and boilerplate-cleaning …


Monolingual and Cross-Lingual Survey Response Annotation

… This model is pre-trained on the CommonCrawl data of 100 languages using multilingual MLM, with a vocabulary size 250k. It outperforms state-of-the-art cross-lingual benchmark models, including mBERT, in cross-language classification, sequence …


Monolingual, multilingual and cross-lingual code comment classification

M Kostić, V Batanović, B Nikolić - Engineering Applications of Artificial Intelligence, 2023

Code comments are one of the most useful forms of documentation and metadata for understanding software implementation. Previous research on code comment classification has focused only on comments in English, typically extracted from a …


MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

J Portes, AR Trott, S Havens, D King, A Venigalla… - Thirty-seventh Conference …, 2023

Although BERT-style encoder models are heavily used in NLP research, many researchers do not pretrain their own BERTs from scratch due to the high cost of training. In the past half-decade since BERT first rose to prominence, many …


MosaicBERT: How to Train BERT with a Lunch Money Budget

J Portes, AR Trott, S Havens, D King, A Venigalla… - Workshop on Efficient …, 2023


mPLM-Sim: Unveiling Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

P Lin, C Hu, Z Zhang, AFT Martins, H Schütze - arXiv preprint arXiv:2305.13684, 2023

Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to …


mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

H Xu, Q Ye, M Yan, Y Shi, J Ye, Y Xu, C Li, B Bi, Q Qian… - arXiv preprint arXiv …, 2023

Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality …


muGen: Generative AI as Machinic Exploration of Cultural Archives

In recent years, generative AI has quickly become a new creative and artistic tool that could challenge our understanding of the creative process and the role of the machine. Despite having exhibited visually promising results, images generated by …


Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams

M Tavakoli, R Chandra, F Tian, C Bravo - arXiv preprint arXiv:2304.10740, 2023

Knowing which factors are significant in credit rating assignment leads to better decision-making. However, the focus of the literature thus far has been mostly on structured data, and fewer studies have addressed unstructured or multi-modal …


MultiFin: A Dataset for Multilingual Financial NLP

R Jørgensen, O Brandt, M Hartmann, X Dai, C Igel… - Findings of the Association …, 2023

… Embedding alignment We map monolingual fasttext embeddings trained on Wikipedia and Commoncrawl into a shared space using RCSLS, by computing pairwise mappings between source languages and English as a target language. As …


Multilanguage Word Embeddings for Social Scientists: Estimation, Inference and Validation Resources for 157 Languages

PL Rodriguez, A Spirling, BM Stewart, EM Wirsching

… from Common Crawl and Wikipedia. A strength of the fastText model is that it uses subword … Common Crawl includes a large number of typos and very rare terms (plus many English loan … Beyond this potential for noise, note also that Common Crawl …


MultiLegalPile: A 689GB Multilingual Legal Corpus

J Niklaus, V Matoshi, M Stürmer, I Chalkidis, DE Ho - arXiv preprint arXiv:2306.02069, 2023

… The Common Crawl corpus is a publicly available multilingual dataset of scraped web pages, … As a result, utilizing the Common Crawl dataset necessitates additional post-filtering and … 2020a) performed several cleaning steps on the April …


Multilingual and Multi-domain Opinion Mining

Everyday, multinational companies receive large amounts of customer feedback. Valuable insights can be extracted from it to address customers’ needs. This textual feedback can span various languages and domains. Due to this variability, generic …


Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspaper

C España-Bonet - arXiv preprint arXiv:2310.16269, 2023

Neutrality is difficult to achieve and, in politics, subjective. Traditional media typically adopt an editorial line that can be used by their potential readers as an indicator of the media bias. Several platforms currently rate news outlets according to their …


Multilingual Jailbreak Challenges in Large Language Models

Y Deng, W Zhang, SJ Pan, L Bing - arXiv preprint arXiv:2310.06474, 2023

… (2023), we determine the resource levels for each language by utilizing the data ratio from the CommonCrawl corpus1, which serves as the primary dataset for most LLMs’ pre-training. Specifically, a language is categorized as high-resource if its …


Multilingual Language Models are not Multicultural: A Case Study in Emotion

S Havaldar, S Rai, B Singhal, LLSC Guntuku, L Ungar - arXiv preprint arXiv …, 2023

Emotions are experienced and expressed differently across the world. In order to use Large Language Models (LMs) for multilingual tasks that require emotional sensitivity, LMs must reflect this cultural variation in emotion. In this study, we …


Multilingual Natural Language Processing Model for Radiology Reports The Summary is all you need!

M Lindo, AS Santos, A Ferreira, J Li, G Luijten… - arXiv preprint arXiv …, 2023

… guages gathered from the publicly available Common Crawl Web Scrape. The mT5 model offers five model sizes, one of which is the base size with 580M parameters, and was used in this project. One difference between T5 and mT5 is that mT5 was …


Multilingual semantic distance: Automatic verbal creativity assessment in many languages.

JD Patterson, HM Merseal, DR Johnson, S Agnoli… - Psychology of Aesthetics …, 2023

… MBERT is trained on the 104 languages with the largest Wikipedia databases, while XLMR is trained on cleaned CommonCrawl data that covers 100 languages. CommonCrawl is an archive of steadily increasing size that is produced by an …


Multilingual Simplification of Medical Texts

S Joseph, K Kazanas, K Reina, VJ Ramanathan, W Xu… - arXiv preprint arXiv …, 2023

… 2022) used Common Crawl to mine millions of sequence pairs in English, Spanish, and French. The significant difference between MUSS and our dataset is MUSS’s lack of cross-lingual alignment, ie, alignment of sequence pairs from one language …


Multilingual Summarization with Factual Consistency Evaluation

R Aharoni, S Narayan, J Maynez, J Herzig, E Clark… - Findings of the Association …, 2023

Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent …


Multilingual text categorization and sentiment analysis: a comparative analysis of the utilization of multilingual approaches for classifying twitter data

G Manias, A Mavrogiorgou, A Kiourtis, C Symvoulidis… - Neural Computing and …, 2023

… XLM-RoBERTa (XLM-R) XLM-RoBERTa (XLM-R) is a multilingual masked language model introduced by authors in [21] that trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages. It is an update to the original XLM-100 …


Multilingual Text Representation

F Faisal - arXiv preprint arXiv:2309.00949, 2023

Modern NLP breakthrough includes large multilingual models capable of performing tasks across more than 100 languages. State-of-the-art language models came a long way, starting from the simple one-hot representation of words capable of …


Multilingual, monolingual and mono-dialectal transfer learning for Moroccan Arabic sentiment classification

N Boudad, R Faizi, R Oulad Haj Thami - Social Network Analysis and Mining, 2024

Transfer learning has recently proven to be very powerful in diverse natural language processing (NLP) tasks such as Machine translation, Sentiment Analysis, and Question/Answering. In this work, we investigate the use of transfer learning (TL) …


Multilinguality in Language Models

• Increasing batch size and training time enhances model performance• Tokenization tools used by mBERT and XLM-100 make these models more diﬀicult to use on raw text• No performance loss for models trained with SPM, compared to …


Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text

W Zhu, J Hessel, A Awadalla, SY Gadre, J Dodge… - arXiv preprint arXiv …, 2023

… We first retrieve the original webpages for each document in the c4-en dataset from the Common Crawl version 2019-18, which is the default version for c4. Next, we extract the URLs for downloadable images from the raw WAT files. We restrict the …


Multimodal Coreference Resolution

In a situated dialog system, multimodal coreference resolution is the task of identifying which entity the user is referring to within a certain context defined by both natural language and visual modalities. This is a crucial task to tackle in order …


Multimodal text summarization with evaluation approaches

AFUR Khilji, U Sinha, P Singh, A Ali, SR Laskar… - Sādhanā, 2023

Multimodal text summarization is a complex and challenging task in the field of natural language processing. Its objective is to use a combination of features from various modalities to create a concise yet informative summary from a given set of …


MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark

D Macko, R Moro, A Uchendu, JS Lucas, M Yamashita… - arXiv preprint arXiv …, 2023

… We have used on-request author-provided processed data as well as CommonCrawl based links published in the GitHub repository. Both sources result in per-language … XLM-RoBERTa16 is a pre-trained on 2.5TB of filtered …


MultiVENT: Multilingual Videos of Events with Aligned Natural Text

K Sanders, D Etter, R Kriz, B Van Durme - arXiv preprint arXiv:2307.03153, 2023

… To incorporate multilinguality into the model’s frame-level features, we use a ViT architecture trained with a contrastive objective over multilingual image-caption pairs from the LAION-5B dataset [42], which is constructed from the Common Crawl …


Murreviikko-A Dialectologically Annotated and Normalized Dataset of Finnish Tweets

O Kuparinen - Tenth Workshop on NLP for Similar Languages …, 2023

This paper presents Murreviikko, a dataset of dialectal Finnish tweets which have been dialectologically annotated and manually normalized to a standard form. The dataset can be used as a test set for dialect identification and dialect-to-standard …


Mutation-Based Adversarial Attacks on Neural Text Detectors

G Liang, J Guerrero, I Alsmadi - arXiv preprint arXiv:2302.05794, 2023

Neural text detectors aim to decide the characteristics that distinguish neural (machine-generated) from human texts. To challenge such detectors, adversarial attacks can alter the statistical characteristics of the generated text, making the detection task more and …


NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders

LB Soares, D Gillick, JR Cole, T Kwiatkowski - arXiv preprint arXiv:2305.14499, 2023

Neural document rerankers are extremely effective in terms of accuracy. However, the best models require dedicated hardware for serving, which is costly and often not feasible. To avoid this serving-time requirement, we present a method of …


Named Entity Recognition for De-identifying Real-World Health Records in Spanish

G López-García, FJ Moreno-Barea, H Mesa, JM Jerez… - International Conference on …, 2023

… XLM-R: this multilingual version of the RoBERTa architecture [18] was pretrained on a massive 2.4TB CommonCrawl Corpus in 100 languages [2], using a large multilingual vocabulary of \(\sim \)250K subwords. We experimented with both the …


Named entity recognition in resumes

E Kesim, A Deliahmetoglu - arXiv preprint arXiv:2306.13062, 2023

Named entity recognition (NER) is used to extract information from various documents and texts such as names and dates. It is important to extract education and work experience information from resumes in order to filter them. Considering …


Named Entity-Oriented Sentiment Analysis with text2text Generation Approach

I Moloshnikov, M Skorokhodov, A Naumov, R Rybka… - Proceedings of the …, 2023

… The CommonCrawl data contains text in 100 languages, of which the Russian language is one of the most representative. … Wikipedia, books, news, and CommonCrawl texts were used to train them. The model dictionary size is 32101 …


Natural Language Inference for Arabic using Recurrent Neural Network and Word Embedding

M Bensghaier, W Bakari, M Neji - 2022 IEEE/ACS 19th International Conference on …, 2022

The Natural Language Inference (NLI) task is a field of Natural Language Processing (NLP) in which researchers try to find ways to decide about the inference relation between two sentences. In this paper, we present an early version of an …


Natural language processing deep learning models for the differential between high-grade gliomas and metastasis: what if the key is how we report them?

T Martín-Noguerol, P López-Úbeda, A Pons-Escoda… - European Radiology, 2023

Objectives The differential between high-grade glioma (HGG) and metastasis remains challenging in common radiological practice. We compare different natural language processing (NLP)–based deep learning models to assist radiologists …


Natural Language Processing For Automatic text summarization [Datasets]-Survey

AA AL-Banna, AK AL-Mashhadany - Wasit Journal of Computer and Mathematics …, 2022

… These stories are based on sources mentioned by WCEP editors and automatically collected from Common Crawl News and supplemented with items acquired automatically from t (WCEP) [14]. The best result in the dataset shown in [15] …


Natural Language Processing with Generative AI Models: A Methodological Approach for Their Application

… OpenAI trained eight different models of varying sizes, ranging from 125 million to 175 billion parameters, using a dataset refined from the CommonCrawl, which contained nearly a trillion words of … For GPT-3, the size refers to the filtered …


Natural Language Processing: Practical Approach

… Using External Data: There are several online resources that provide large text corpora such as Project Gutenberg, Common Crawl and Wikipedia. These resources can be accessed using the appropriate API and downloaded locally for further analysis. …


Natural Language-Assisted Sign Language Recognition

R Zuo, F Wei, B Mak - arXiv preprint arXiv:2303.12080, 2023

… Concretely, given a sign vocabulary containing N glosses, we leverage fastText [38] pretrained on Common Crawl to extract a 300-d feature for each gloss. We use E ∈ RN×300 to denote the N gloss features. Language-Aware Label Smoothing and …


Navigating Alignment for Non-identical Client Class Sets: A Label Name-Anchored Federated Learning Framework

J Zhang, X Zhang, X Zhang, D Hong, RK Gupta… - 2023

Traditional federated classification methods, even those designed for non-IID clients, assume that each client annotates its local data with respect to the same universal class set. In this paper, we focus on a more general yet practical setting, non-identical …


Navigating the Ethical Landmines of ChatGPT: Implications of Intelligent Chatbots in Plastic Surgery Clinical Practice

NC Oleck, HI Naga, DS Nichols, MX Morris, B Dhingra… - Plastic and Reconstructive …, 2023

… The large dataset used to train ChatGPT is derived from “internet dumps” obtained via web scraping from open source datasets such as Common Crawl and OpenWebText. These datasets are composed of data sourced from over 20 million …


Navigating the generative AI era: Introducing the AI assessment scale for ethical GenAI assessment

M Perkins, L Furze, J Roe, J MacVaugh - arXiv preprint arXiv:2312.07086, 2023

Recent developments in Generative Artificial Intelligence (GenAI) have created a paradigm shift in multiple areas of society, and the use of these technologies is likely to become a defining feature of education in coming decades. GenAI offers …


Negation and speculation processing: A study on cue-scope labelling and assertion classification in Spanish clinical text

N Perez, M Cuadros, G Rigau - Artificial Intelligence in Medicine, 2023

Natural Language Processing (NLP) based on new deep learning technology is contributing to the emergence of powerful solutions that help healthcare providers and researchers discover valuable patterns within insurmountable volumes of health …


Negativity spreads faster: A large-scale multilingual twitter analysis on the role of sentiment in political communication

D Antypas, A Preece, J Camacho-Collados - Online Social Networks and Media, 2023

Social media has become extremely influential when it comes to policy making in modern societies, especially in the western world, where platforms such as Twitter allow users to follow politicians, thus making citizens more involved in political …


Nested Named-Entity Recognition in Multilingual Code-Switched NLP Check for updates

A Patil, U Kolhe - Big Data, Machine Learning, and Applications …, 2024

… OSCAR Open Super-large Crawled Aggregated coRpus (OSCAR) is a huge multilingual corpus obtained by language classification and filtering of Common Crawl corpus. The dataset contains 166 languages in total. We are using this dataset for adaptive …


Netizens, Academicians, and Information Professionals' Opinions About AI With Special Reference To ChatGPT

A Subaveerapandiyan, A Vinoth, N Tiwary - Library Philosophy and Practice (e …, 2023

N Tiwary - arXiv preprint arXiv:2302.07136, 2023

This study aims to understand the perceptions and opinions of academicians towards ChatGPT-3 by collecting and analyzing social media comments, and a survey was conducted with library and information science professionals. The …


Neural correlates of object-extracted relative clause processing across English and Chinese

D Dunagan, M Stanojević, M Coavoux, S Zhang… - Neurobiology of Language

Are the brain bases of language comprehension the same across all human languages, or do these bases vary in a way that corresponds to differences in linguistic typology? English and Mandarin Chinese attest such a typological …


Neural Data Search for Table Augmentation

… WDC table corpus2 consists of 4.2 million relational tables generated by extracting schema.org3 annotations from the Common Crawl and grouping the annotations by class and host. All tables in this corpus share a common schema. By …


Neural Embeddings for Web Testing

A Stocco, A Willi, LLL Starace, M Biagiola, P Tonella - arXiv preprint arXiv …, 2023

Web test automation techniques employ web crawlers to automatically produce a web app model that is used for test generation. Existing crawlers rely on app-specific, threshold-based, algorithms to assess state equivalence. Such algorithms are hard …


Neural Language Generation for Content Adaptation: Explainable, Efficient Low-Resource Text Simplification and Evaluation

There are rich opportunities to reduce the language complexity of professional content (either human-written or computer-generated) and make it accessible to a broad audience. As a sub-task of natural language generation (NLG), text …


Neural machine translation for in‐text citation classification

I Safder, M Ali, NR Aljohani, R Nawaz, SU Hassan - Journal of the Association for …

The quality of scientific publications can be measured by quantitative indices such as the h‐index, Source Normalized Impact per Paper, or g‐index. However, these measures lack to explain the function or reasons for citations and the context of …


Neural Machine Translation For Low Resource Languages

V Goyle, K Goyle, P Krishnaswamy, KG Ravikumar… - arXiv preprint arXiv …, 2023

… It is trained on the monolingual common crawl corpus for 25 languages (CC25). The text corpus is further tokenized using the sentence-piece model (SPM) [6] which provides both subword tokenization and unigram language modelling additionally …


Neural machine translation to local languages

Neural Machine Translation (NMT) has demonstrated significant potential in facilitating communication across language barriers. However, adapting NMT systems to low-resource local languages poses significant challenges due to the …


Neural Machine Translation, Large Language Models and Literary Translation: The Story So Far

We understand ‘literary machine translation’as an emerging interdisciplinary field that embraces a range of phenomena related to the application of MT to the translation of literary texts. It touches upon or intersects with, among other areas: ● …


Neural semantic tagging for natural language-based search in building information models: Implications for practice

M Shahinmoghadam, SE Kahou, A Motamedi - Computers in Industry, 2024

While the adoption of open Building Information Modeling (open BIM) standards continues to grow, the inherent complexity and multifaceted nature of the built asset lifecycle data present a critical bottleneck for effective information retrieval. To …


NeuralMind-UNICAMP at 2022 TREC NeuCLIR: Large Boring Rerankers for Cross-lingual Retrieval

V Jeronymo, R Lotufo, R Nogueira - arXiv preprint arXiv:2303.16145, 2023

This paper reports on a study of cross-lingual information retrieval (CLIR) using the mT5-XXL reranker on the NeuCLIR track of TREC 2022. Perhaps the biggest contribution of this study is the finding that despite the mT5 model being fine-tuned …


Neuroinformatics, Neural Networks and Neurocomputers for Computational Intelligence

1. Learning from (big) data-> neural networks and deep NN 2. Explainability (extracting rules, associations)(explainable AI)→ fuzzy logic/neuro-fuzzy systems 3. Evolvability→ evolving connectionist systems (ECOS) and brain-inspired SNN (NeuCube). 4 …


New Challenges in Official Statistics: Big Data Analytics and Multi-level Product Classification of Web Scraped Data

The surge in internet usage has significantly amplified data generation, paving the way for researchers and policymakers to gain in-depth and granular insights about society. This shift has transformed data collection methodologies, fostering the …


New Evaluation Metrics Capture Quality Degradation due to LLM Watermarking

K Singh, J Zou - arXiv preprint arXiv:2312.02382, 2023

With the increasing use of large-language models (LLMs) like ChatGPT, watermarking has emerged as a promising approach for tracing machine-generated content. However, research on LLM watermarking often relies on simple perplexity …


New models developed for detection of misconceptions in physics with artificial intelligence

MU Demirezen, O Yilmaz, E Ince - Neural Computing and Applications, 2023

Students’ misconceptions of various topics in physics have been investigated by many researchers. The detection of misconceptions is very difficult and takes a long time as a human being. Our aim in the study carried out is to determine the …


nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

M Cosler, C Hahn, D Mendoza, F Schmitt, C Trippel - arXiv preprint arXiv:2303.04864, 2023

A rigorous formalization of desired system requirements is indispensable when performing any verification task. This often limits the application of verification techniques, as writing formal specifications is an error-prone and time-consuming …


NLP Meets Agronomy: Document Classification for Plant Health Surveillance

In the field of Plant Health Epidemiological Surveillance, accurately analyzing written reports of events affecting agriculture is crucial. This master’s thesis in Natural Language Processing leverages the power of Automatic Text Classification …


NLP-Based Automated Conspiracy Detection for Massive Twitter Datasets

The COVID-19 pandemic, which affected societies worldwide, and many were compelled to shut down, also affected social media platforms like Twitter. The COVID-19-related misinformation on Twitter covered various topics and contained many competing …


NosWalker: A Decoupled Architecture for Out-of-Core Random Walk Processing

S Wang, M Zhang, K Yang, K Chen, S Ma, J Jiang… - Proceedings of the 28th …, 2023

Out-of-core random walk system has recently attracted a lot of attention as an economical way to run billions of walkers over large graphs. However, existing out-of-core random walk systems are all built upon general out-of-core graph processing …


Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting

H Huang, T Tang, D Zhang, WX Zhao, T Song, Y Xia… - arXiv preprint arXiv …, 2023

Large language models (LLMs) demonstrate impressive multilingual capability, but their performance varies substantially across different languages. In this work, we introduce a simple yet effective method, called cross-lingual-thought prompting (XLT) …


Nougat: Neural Optical Understanding for Academic Documents

L Blecher, G Cucurull, T Scialom, R Stojnic - arXiv preprint arXiv:2308.13418, 2023

Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical …


NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation

P West, RL Bras, T Sorensen, BY Lin, L Jiang, X Lu… - arXiv preprint arXiv …, 2023

We present NovaCOMET, an open commonsense knowledge model, that combines the best aspects of knowledge and general task models. Compared to previous knowledge models, NovaCOMET allows open-format relations enabling direct …


Novel embeddings improve the prediction of risk perception

… fastText was trained on the Common Crawl, a corpus of web pages that is more diverse and with 600B tokens considerably larger than the … GloVe was trained on a slightly larger version of the Common Crawl than fastText comprising 804B. We …


Ntcir-17 mednlp-sc social media adverse drug event detection: Subtask overview

S Wakamiya, LK Pereira, L Raithel, HS Yeh, P Han… - Proceedings of the 17th …, 2023

… It is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. XLM-R has been shown to perform particularly well in low-resource languages, such as Swahili and Urdu. We use the XLM-R base model released by the authors. …


Oasis: Data Curation and Assessment System for Pretraining of Large Language Models

T Zhou, Y Chen, P Cao, K Liu, J Zhao, S Liu - arXiv preprint arXiv:2311.12537, 2023

… Aside from introducing Oasis, we demonstrate a complete case that utilizes this platform to build a high-quality and high-diversity Common Crawl corpus. Meanwhile, we holistic assess the corpus in the different development stages. The assessments …


OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents

H Laurençon, L Saulnier, L Tronchon, S Bekman… - arXiv preprint arXiv …, 2023

… We introduce the OBELISC dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We …


Objaverse-XL: A Universe of 10M+ 3D Objects

M Deitke, R Liu, M Wallingford, H Ngo, O Michel… - arXiv preprint arXiv …, 2023

… In language understanding, datasets like Common Crawl [1] have culminated in unprecedented capabilities of large language models such as GPT-4 [43], which in turn power mainstream applications like ChatGPT. The impact of large datasets is …


OHYEAH AT VLSP2022-EVJVQA CHALLENGE: A JOINTLY LANGUAGE-IMAGE MODEL FOR MULTILINGUAL VISUAL QUESTION ANSWERING

LN Dinh, H Le Ngoc, LQ Phan - Journal of Computer Science and Cybernetics, 2023

… mT5 [5] is a massively multilingual pre-trained text-to-text transformer, which leverages T5 [11] to train on a new Common Crawl-based dataset with up to 101 languages. It has been proven to outperform many multilingual models in a variety of …


Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

VD Lai, C Van Nguyen, NT Ngo, T Nguyen… - arXiv preprint arXiv …, 2023

… Table 1: List of 26 non-English languages in our Okapi framework along with language codes, numbers of first and second speakers (the “Pop.” column), data ratios in the CommonCrawl corpus, and language categories. The languages are …


On Bilingual Lexicon Induction with Large Language Models

Y Li, A Korhonen, I Vulić - arXiv preprint arXiv:2310.13995, 2023

Bilingual Lexicon Induction (BLI) is a core task in multilingual NLP that still, to a large extent, relies on calculating cross-lingual word representations. Inspired by the global paradigm shift in NLP towards Large Language Models (LLMs), we examine …


On Corpora and Writing

M Chitez, A Dinca - Digital Writing Technologies in Higher Education …, 2023

… For example, the filtered version of the Common Crawl Footnote 1 used for the pre-training dataset for GPT-3 (ie Generative Pre-trained Transformer 3 Footnote 2 ) consists of 410 billion tokens. This large quantity of data enables powerful quantitative analyses …


On generalization of the sense retrofitting model

YY Lee, TY Yen, HH Huang, YT Shiue, HH Chen - Natural Language Engineering, 2023

With the aid of recently proposed word embedding algorithms, the study of semantic relatedness has progressed rapidly. However, word-level representations are still lacking for many natural language processing tasks. Various sense-level …


On Hate Scaling Laws For Data-Swamps

A Birhane, V Prabhu, S Han, VN Boddeti - arXiv preprint arXiv:2306.13141, 2023

… Given that both LAION-400M and LAION-2B-en are extracted from the CommonCrawl dataset, we hypothesize that during the race to expand the dataset to 2 billion samples, the dataset scraping module might have sampled from the low-quality …


On Methods of Data Standardization of German Social Media Comments

L Melnyk, L Feld - Journal of Computer-Assisted Linguistic Research, 2023

This article is part of a larger project aiming at identifying discursive strategies in social media discourses revolving around the topic of gender diversity, for which roughly 350,000 comments were scraped from the comments sections below …


On reading and interpreting black box deep neural networks

JE Dobson - International Journal of Digital Humanities, 2023

The deep neural networks used in computer vision and in recent large language models are widely recognized as black boxes, a term that describes their complicated architectures and opaque decision-making mechanisms. This essay …


On the Benefits of Learning to Route in Mixture-of-Experts Models

N Dikkala, N Ghosh, R Meka, R Panigrahy, N Vyas… - Proceedings of the 2023 …, 2023

… We study language modeling objective on the Common Crawl (mC4) dataset with T5 transformers. A first approach one could consider would be to compare neural networks where the router is learned versus networks where the router is frozen …


On the Effectiveness of Log Representation for Log-based Anomaly Detection

Logs are an essential source of information for people to understand the running status of a software system. Due to the evolving modern software architecture and maintenance methods, more research efforts have been devoted to automated log …


On the Evolution of Knowledge Graphs: A Survey and Perspective

X Jiang, C Xu, Y Shen, X Sun, L Tang, S Wang, Z Chen… - arXiv preprint arXiv …, 2023

Knowledge graphs (KGs) are structured representations of diversified knowledge. They are widely used in various intelligent applications. In this article, we provide a comprehensive survey on the evolution of various types of knowledge graphs (ie …


On the fractal patterns of language structures

LC Ribeiro, AT Bernardes, H Mello - Plos one, 2023

… -profit CommonCrawl Foundation [39–41]. The data is publicly available in the Amazon cloud and were collected from many independent crawls from 2008 to 2017. As a reference of the magnitude of the raw data used by December 2017, the …


On the Generalization Ability of Retrieval-Enhanced Transformers

T Norlund, E Doostmohammadi, R Johansson… - arXiv preprint arXiv …, 2023

Recent work on the Retrieval-Enhanced Transformer (RETRO) model has shown that off-loading memory from trainable weights to a retrieval database can significantly improve language modeling and match the performance of non-retrieval …


On the Importance of Data Representation for the Success of Text Classification

CY Cuello, V Jofre Caradonna, MJ Garciarena Ucelay… - … (CACIC)(La Rioja, 3 al 6 de …, 2023

… We also used the GloVe embeddings trained with Common Crawl data which have 300 components. For both methods, the document … metrics were reached with vectors of 300 dimensions and, in the case of GloVe, using the pre-trained …


On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments

T Sakai, S Tao, N Chen, Y Li, M Maistro, Z Chu, N Ferro - ACM Transactions on …, 2023

The present study leverages a recent opportunity we had to create a new English web search test collection for the NTCIR-16 We Want Web (WWW-4) task, which concluded in June 2022. More specifically, through the test collection construction …


On the Question of Authorship in Large Language Models (LLMs)

… GPT-3’s gigantic scale came along with, of course, a gigantic training set, which includes English Wikipedia and BookCorpus along with the CommonCrawl dataset, which is a web crawl dataset consisting of the text from billions of web pages. Like …


On the Reliability of Watermarks for Large Language Models

J Kirchenbauer, J Geiping, Y Wen, M Shu, K Saifullah… - arXiv preprint arXiv …, 2023

… [2023], we use the Colossal Common Crawl Cleaned corpus (C4) dataset as a source of prompts for open-ended generation. For the human study, we adopt the “Long-Form Question Answering” (LFQA) dataset curated by Krishna et al. [2023] based on a …


On the Role of Morphological Information for Contextual Lemmatization

O Toporkov, R Agerri - arXiv preprint arXiv:2302.00407, 2023

Lemmatization is a Natural Language Processing (NLP) task which consists of producing, from a given inflected word, its canonical form or lemma. Lemmatization is one of the basic tasks that facilitate downstream NLP applications, and is of …


On the State of German (Abstractive) Text Summarization

D Aumiller, J Fan, M Gertz - arXiv preprint arXiv:2301.07095, 2023

With recent advancements in the area of Natural Language Processing, the focus is slowly shifting from a purely English-centric view towards more language-specific solutions, including German. Especially practical for businesses to analyze their …


On the use of sentiment analysis for linguistics research. Observations on sentiment polarity and the use of the progressive in Italian

Today computational methods are used more and more in domains outside of their original conception. For example, sentiment analysis (SA)–a technique originally designed to infer general opinions from product reviews–is applied to make stock …


On Web-based Visual Corpus Construction for Visual Document Understanding

D Kim, T Hong, M Yim, Y Kim, G Kim - International Conference on Document Analysis …, 2023

… 5.1 Scale-up Using CommonCrawl Our initial goal was to handle tons of HTML data, such as CommonCrawl dataset Footnote 13 which consists of petabytes of data. … https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlsize. …


On" Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research

MN Nityasya, HA Wibowo, AF Aji, GI Winata… - arXiv preprint arXiv …, 2023

This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research …


One Textbook Is All You Need

A Sawyer, I Moradi, L Welch, J Zhao, S Huang… - 2023

Large Language Models don’t have a strong understanding of under-represented languages due to the lack of requisite training data [12]. Previous efforts increased model performance on low-resource languages by using news [15], synthetic data [2] …


One True Pairing: Evaluating Effective Language Pairings for Fake News Detection Employing Zero-Shot Cross-Lingual Transfer

S Kasim - Soft Computing and Its Engineering Applications: 4th …, 2023

… [5] conducted unsupervised training of cross-lingual embeddings at very large scale using the CommonCrawl Corpus. To develop XLM-R, which is a transformer-based multilingual masked language model, they applied a SentencePiece model directly …


Open Government Data Corpus for Table Search

M Glass, S Bagchi, O Hassanzadeh, G Rossiello… - arXiv preprint arXiv …, 2023

… Web Table Retrieval [1] uses HTML tables from Common Crawl and uses crowd sourcing and query logs to gather 60 queries. These queries are then attempted over several baseline methods and the resulting tables are pooled for relevance …


Open Problems in Applied Deep Learning

M Raissi - arXiv preprint arXiv:2301.11316, 2023

This work formulates the machine learning mechanism as a bi-level optimization problem. The inner level optimization loop entails minimizing a properly chosen loss function evaluated on the training data. This is nothing but the well-studied training …


OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model Pre-trained from Scratch

J Li, Z Tang, Y Ding, P Wang, P Guo, W You, D Qiao… - arXiv preprint arXiv …, 2023

Large language models (LLMs) with billions of parameters have demonstrated outstanding performance on various natural language processing tasks. This report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model, to …


OpenFact at CheckThat! 2023: Head-to-Head GPT vs. BERT-A Comparative Study of Transformers Language Models for the Detection of Check-worthy

M Sawiński, K Węcel, E Księżniak, M Stróżyna… - 2023

This paper presents the research findings resulting from experiments conducted as part of the Check-That! Lab Task 1B-English submission at CLEF 2023. The aim of the research was to evaluate the check-worthiness of short texts in English. Various …


OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

K Paster, MD Santos, Z Azerbayev, J Ba - arXiv preprint arXiv:2310.06786, 2023

… Due to the high variance in the quality of documents from Common Crawl, we additionally use several methods for filtering and boilerplate reduction. Throughout the creation of OpenWebMath, we iteratively refined these methods to ensure that …


Opportunities for Large Language Models and Discourse in Engineering Design

J Göpfert, JM Weinand, P Kuckertz, D Stolten - arXiv preprint arXiv:2306.09169, 2023

In recent years, large language models have achieved breakthroughs on a wide range of benchmarks in natural language processing and continue to increase in performance. Recently, the advances of large language models have raised interest …


OPTIMIZATION METHODS FOR MODELING DIVERSITY IN LANGUAGE TECHNOLOGIES

Abstract Language use varies across individuals, communities, and populations giving rise to different variations with diverging vocabularies, syntax, semantics, and pragmatics. Despite rapid improvements in natural language processing systems on …


Optimized Tokenization for Transcribed Error Correction

T Wullach, SE Chazan - arXiv preprint arXiv:2310.10704, 2023

… 2020) data set, a variant of the Common Crawl dataset that consists of 101 languages. For each language, we randomly select a set of 20M sentences that ranges from 3 to 15 words long. We use lowercase synthetic texts, remove …


Optimizing Machine Translation for Virtual Assistants: Multi-Variant Generation with VerbNet and Conditional Beam Search

In this paper, we introduce a domain-adapted machine translation (MT) model for intelligent virtual assistants (IVA) designed to translate natural language understanding (NLU) training data sets. This work uses a constrained beam search …


Optimizing Match-LSTM for SQuAD v2. 0

… The word vectors utilized in our model are trained on the 840B word corpora from Common Crawl data. The dimensionality of the embedding at 300 is significantly higher that the other choices of 50, 100 and 200 it provides more contextual …


Options Speak Louder Than Words: Strategic Negativity in Earnings Calls Prior to Option Grants

We investigate whether executive compensation affects disclosure during earnings conference calls. In particular, we hypothesize that executives who have an upcoming option grant will use overly negative language in earnings calls, intending …


ORGANIZATIONS AS ALGORITHMS: A NEW METAPHOR FOR ADVANCING MANAGEMENT THEORY

According to the ‘Point’essay, management research’s reliance on corporate data threatens to replace objective theory with profit-biased ‘corporate empiricism,’undermining the scientific and ethical integrity of the field. In this ‘Counterpoint’essay, we offer a …


Other Applications of Comparable Corpora

S Sharoff, R Rapp, P Zweigenbaum - Building and Using Comparable Corpora for …, 2023

This section concerns applications of comparable corpora beyond pure machine translation. It has been argued [ 1 , 2 ] that downstream applications such as cross-lingual document classification, information retrieval or natural language inference, apart …


Overview of the CLEF-2023 CheckThat! lab task 2 on subjectivity in news articles

A Galassi, F Ruggeri, A Barrón-Cedeño, F Alam… - M. Aliannejadi, G. Faggioli …, 2023

We describe the outcome of the 2023 edition of the CheckThat! Lab at CLEF. We focus on subjectivity (Task 2), which has been proposed for the first time. It aims at fostering the technology for the identification of subjective text fragments in news …


Overview of the NTCIR-17 FairWeb-1 Task

S Tao, N Chen, T Sakai, Z Chu, H Arai, I Soboroff… - Proceedings of the 17th …, 2023

This paper provides an overview of the NTCIR-17 FairWeb-1 Task. FairWeb-1 is an English web search task which seeks more than an ad-hoc web search task. Our task considers not only document relevance but also group fairness. We designed …


Overview of the TREC 2022 NeuCLIR Track

D Lawrie, S MacAvaney, J Mayfield, P McNamee…

… , and Russian, drawn from the Common Crawl news collection.The documents were obtained by the Common Crawl service between August 1, … , an alternative source of the document collection can be obtained directly from Common Crawl …


Overview of Touché 2023: Argument and Causal Retrieval

A Bondarenko, M Fröbe, J Kiesel, F Schlatt, V Barriere…

The goal of Touché is to foster and support the development of technologies for argument and causal retrieval and analysis. For the fourth time, we organize the Touché lab featuring four shared tasks:(a) argument retrieval for controversial topics …


OWLER: PRELIMINARY RESULTS FOR BUILDING A COLLABORATIVE OPEN WEB CRAWLER

M Dinzinger, S Zerhoudi, M Al-Maamari, M Istaiti…

… The used seed URLs are a random sample from a recent CommonCrawl (CC) dump. Note that a CC dump gives no guarantee on a fair or even distribution of URLs, so the set of seed URLs could possibly be biased in any direction. …


PaLM 2 Technical Report

R Anil, AM Dai, O Firat, M Johnson, D Lepikhin… - arXiv preprint arXiv …, 2023

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of …


Palmyra-Med: Instruction-Based Fine-Tuning of LLMs Enhancing Medical Domain Performance

The development of Large Language Models (LLMs) has greatly impacted natural language processing tasks by demonstrating exceptional performance across various applications. However, a significant challenge faced by these models is their …


Paloma: A Benchmark for Evaluating Language Model Fit

I Magnusson, A Bhagia, V Hofmann, L Soldaini, AH Jha… - arXiv preprint arXiv …, 2023

… pretraining without data beyond Common Crawl leads to inconsistent fit to many domains. … are trained only on data derived from Common Crawl (C4, FALCON REFINEDWEB, and MC4… majority of our domains are not sourced from Common …


PALS: Personalized Active Learning for Subjective Tasks in NLP

K Kanclerz, K Karanowski, J Bielaniewicz, M Gruza… - Proceedings of the 2023 …, 2023

For subjective NLP problems, such as classification of hate speech, aggression, or emotions, personalized solutions can be exploited. Then, the learned models infer about the perception of the content independently for each reader. To acquire …


Panda LLM: Training Data and Evaluation for Open-Sourced Chinese Instruction-Following Large Language Models

F Jiao, B Ding, T Luo, Z Mo - arXiv preprint arXiv:2305.03025, 2023

This project focuses on enhancing open-source large language models through instruction-tuning and providing comprehensive evaluations of their performance. We explore how various training data factors, such as quantity, quality, and linguistic …


PandeMedia: an annotated corpus of digital media for issue salience

… Though available at the start of this project, these large datasets from Common Crawl would not significantly reduce the amount of work necessary to build the pipeline, because metadata curation and data parsing are still necessary to locate …


PanGu-{\Sigma}: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

X Ren, P Zhou, X Meng, X Huang, Y Wang, W Wang… - arXiv preprint arXiv …, 2023

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI …


PaniniQA: Enhancing Patient Education Through Interactive Question Answering

P Cai, Z Yao, F Liu, D Wang, M Reilly, H Zhou, L Li… - arXiv preprint arXiv …, 2023

Patient portal allows discharged patients to access their personalized discharge instructions in electronic health records (EHRs). However, many patients have difficulty understanding or memorizing their discharge instructions. In this paper, we …


PAPERS IN ENGLISH

F Adilova, R Davronov, R Safarov - … редактор: Ахметов Сайранбек Махсутович, д-р …, 2023

… Distributed word representations for 157 distinct languages were created using the Wikipedia and Common Crawl databases [8]. The author's Uzbek Wikipedia model has 110K words, but the Common Crawl model only has 830K words. They …


Parameter-Efficient Fine-Tuning without Introducing New Latency

B Liao, Y Meng, C Monz - arXiv preprint arXiv:2305.16742, 2023

Parameter-efficient fine-tuning (PEFT) of pre-trained language models has recently demonstrated remarkable achievements, effectively matching the performance of full fine-tuning while utilizing significantly fewer trainable parameters, and consequently …


Pashto offensive language detection: a benchmark dataset and monolingual Pashto BERT

I Haq, W Qiu, J Guo, P Tang - PeerJ Computer Science, 2023

… XLM-R is trained on 2.5TB of CommonCrawl data in 100 languages simultaneously. This model outperforms other multilingual models on many NLP tasks, demonstrating its effectiveness at learning cross-lingual representations. We …


Pay Attention to the Robustness of Chinese Minority Language Models! Syllable-level Textual Adversarial Attack on Tibetan Script

The textual adversarial attack refers to an attack method in which the attacker adds imperceptible perturbations to the original texts by elaborate design so that the NLP (natural language processing) model produces false judgments. This method is also used to …


PDF investigation with parser differentials and ontology

This paper describes the Verifiable Automatic Language Analysis and Recognition for Inputs (VALARIN) system to process, evaluate, and flag unsafe PDFs. The system extracts error features from a collection of PDF parsers, and organizes the different …


PDFTriage: Question Answering over Long, Structured Documents

J Saad-Falcon, J Barrow, A Siu, A Nenkova, RA Rossi… - arXiv preprint arXiv …, 2023

… For our documents, we sampled 1000 documents from the common crawl to get visually-rich, professional documents from various domains, then subsampled 100 documents based on their reading level (Flesch… 3https://mturk.com 4https://commoncrawl.org/ …


PDHF: Effective Phishing Detection Model Combining Optimal Artificial and Automatic Deep Features

E Zhu, K Cheng, Z Zhang, H Wang - Computers & Security, 2023

Currently, the increasing number of high-volume phishing attacks is among the largest threats to networking environments on a daily basis. During such a severe attack, researchers prefer to extract numerous features to improve the accuracy of …


Pearls and pitfalls of ChatGPT in medical oncology

J Blum, AK Menta, X Zhao, VB Yang, MA Gouda… - Trends in Cancer, 2023

… Common Crawl is an open repository of web data from billions of web pages, consisting of petabytes of information on web pages, images… medical literature, it is diluted by unverified information contained in the Common Crawl, WebText2, and …


Performance of Generative Large Language Models on Ophthalmology Board Style Questions

LZ Cai, A Shaheen, A Jin, R Fukui, SY Jonathan… - American Journal of …, 2023

… The specialized medical information used to train these LLMs is limited, as they are currently trained on publicly available datasets designed for natural language processing such as Common Crawl, WebText, Wikipedia, and available Web Books …


PERFORMANCE STUDY OF THE TEXT ANALYSIS MODULE IN THE PROPOSED MODEL OF AUTOMATIC SPEAKER'S SPEECH ANNOTATION

О БАРКОВСЬКА - Computer systems and information technologies, 2022

… This solution was trained in 101 languages on a corpus of Common Crawl web pages, and supplemented with the XL-SUM dataset (covering 45 languages, highly abstract, concise and high-end as evidenced by human and internal evaluation) …


Perplexity-Driven Case Encoding Needs Augmentation for CAPITALIZATION Robustness

Subword segmentation methods are the predominant solution to vocab sparsity in NMT. However, they cannot currently handle capitalization well. We re-encode case to allow the perplexity-driven SPM unigram language model algorithm to learn how …


Persian Text Sentiment Analysis Based on BERT and Neural Networks

SR Zardak, AH Rasekh, MS Bashkari - Iranian Journal of Science and Technology …, 2023

Sentiment analysis and opinion mining are studies that analyze people’s opinions, feelings, experiences, and emotions through written language. This is one of the most active research fields in natural language processing and data mining, web …


Personality Traits in Large Language Models

M Safdari, G Serapio-García, C Crepy, S Fitz… - arXiv preprint arXiv …, 2023

The advent of large language models (LLMs) has revolutionized natural language processing, enabling the generation of coherent and contextually relevant text. As LLMs increasingly power conversational agents, the synthesized personality …


Perspective: Large Language Models in Applied Mechanics

N Brodnik, S Carton, C Muir, S Ghosh, D Downey… - Journal of Applied …, 2023

Large language models (LLMs), such as ChatGPT and PaLM, are able to perform sophisticated text comprehension and generation tasks with little or no training. Alongside their broader societal impacts, these capabilities carry great promise for …


Phish-armour: phishing detection using deep recurrent neural networks

P Dhanavanthini, SS Chakkravarthy - Soft Computing, 2023

Phishing is an illegal cybercrime, wherein a target gets victimized for sacrificing their personal and corporate information. It is one of the most straightforward forms of cyber-attack for hackers, as well as one of the simplest for victims to fall for. It can …


PHISHING WEBSITE DETECTION TECHNIQUES: A LITERATURE SURVEY

K Subashini, V Narmatha - Journal of Data Acquisition and Processing, 2023

… The names of legal sites that are likely to be used for phishing are included in Alexa and Common Crawl [26][27]. End-users submit suspicious URLs to Phish-tank and Open-Fish to assess if these websites are phishing scams [28][29]. The dataset …


Phishing website detection using genetic algorithm-based feature selection and parameter hypertuning

False webpages are created by cyber attackers who seek to mislead users into revealing sensitive and personal information, from credit card details to passwords. Phishing is a class of cyber attacks that mislead users into clicking on false websites …


Phonetically-Grounded Language Generation: The Case of Tongue Twisters

T Loakman, C Tang, C Lin - arXiv preprint arXiv:2306.03457, 2023

Previous work in phonetically-grounded language generation has mainly focused on domains such as lyrics and poetry. In this paper, we present work on the generation of tongue twisters - a form of language that is required to be phonetically …


Physics language and language use in physics--What do we know and how AI might enhance language-related research and instruction

P Wulff - European Journal of Physics, 2023

Abstract Language is an important resource for physicists and learners of physics to construe physical phenomena and processes, and communicate ideas. Moreover, any physics-related instructional setting is inherently language-bound, and physics …


PLACES: Prompting Language Models for Social Conversation Synthesis

M Chen, A Papangelis, C Tao, S Kim, A Rosenbaum… - arXiv preprint arXiv …, 2023

… 2020) or Common Crawl. This allows language models to have exposure to a large amount of linguistic diversity, but this also results in exposure to a lot of hateful, … a preliminary analysis of undesirable content in the common crawl corpus. arXiv …


Poisoning Web-Scale Training Datasets is Practical

N Carlini, M Jagielski, CA Choquette-Choo, D Paleka… - arXiv preprint arXiv …, 2023

… For example, both Wikipedia and Common Crawl regularly produce snapshots of their entire database. This simplifies access for people … We will investigate this attack on Wikipedia-derived datasets, but also discuss how similar vulnerabilities …


Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Models

T Schott, D Furman, S Bhat - Proceedings of the 2023 Conference on Empirical …, 2023

In this work, we assess the ability of foundation models to recall encyclopedic knowledge across a wide range of linguistic contexts. To support this, we: 1) produce a 20-language dataset that contains 303k factual associations paired with …


PolyLM: An Open Source Polyglot Large Language Model

X Wei, H Wei, H Lin, T Li, P Zhang, X Ren, M Li, Y Wan… - arXiv preprint arXiv …, 2023

… For instance, Thai and Indonesian have over 300 million (M) speakers, yet the size of these two languages in common crawl-based dataset … and CommonCrawl web documents as the negative samples. To sum up, about 28.3% data are filtered …


PooRaa‐Agri KG: An agricultural knowledge graph‐based simplified multilingual query system

N Sivakumar, P Srinivasan, M Kannan, VP - Expert Systems

The current work proposes PooRaa‐Agri KG, an agricultural knowledge graph‐based simplified multilingual query system that works in real time to provide concise answers for agriculture‐based queries. The proposed approach accommodates real‐time …


Popular LLMs

T Amaratunga - … Large Language Models: Learning Their Underlying …, 2023

… For the training of GPT-2, the CommonCrawl corpus was initially considered because of its large size. CommonCrawl is a large text corpus created … Unlike CommonCrawl, WebText was generated by scraping only pages linked to Reddit …


Posthuman Bodies: Why They (Still) Matter

Emphasizing the common capacity for cognition, this chapter presents an integrated framework that includes humans, nonhuman biological lifeforms, and computational media. It clearly distinguishes between the agential powers of material processes …


Practical Named Entity Recognition: The Role of Entity and Its Context

Neural supervised models for named entity recognition perform well within the same domain but fail to recognize entities not seen in the (pre-) training data with high accuracy. For better generalization, it is essential that models are able to recognize …


Pre-processing and Resource Modelling for English-Assamese NMT System

MA Ahmed, K Kashyap, SK Sarma - 2023 4th International Conference on Computing …, 2023

Neural Machine Translation (NMT) modelling for low-resourced languages is usually under-explored lacking proper analysis in the pre-processing stages which could potentially influence the performance of the model. Assamese, an indigenous …


Pre-Training for Manipulation: The Case for Shape Biased Vision Transformers

Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to learn visuallyrobust policies from pixels. To that end, past work has favored large object interaction data, such as first …


PrecogIIITH@ WASSA2023: Emotion Detection for Urdu-English Code-mixed Text

BH Vedula, P Kodali, M Shrivastava, P Kumaraguru

Code-mixing refers to the phenomenon of using two or more languages interchangeably within a speech or discourse context. This practice is particularly prevalent on social media platforms, and determining the embedded affects in a …


Predicting COVID-19 Related Tweets Using Ensemble of Transformers Models

Q Ismail, M Massadeh, QB Baker - 2022 IEEE/ACS 19th International Conference on …, 2022

… We also utilized RoBERTa in addition to BERT; RoBERTa is a powerful pre-trained linguistic model used for English; because RoBERTa was developed using 160GB of text from CommonCrawl, Wikipedia, news sources, novels, and other online …


PreSTU: Pre-Training for Scene-Text Understanding (Supplementary Material)

J Kil, S Changpinyo, X Chen, H Hu, S Goodman…

… Our language module is initialized from mT5-Base [33], a multilingual variant of T5 [23], pre-trained on a new Common Crawlbased dataset with 101 different languages. During training, all parameters in vision and language blocks are …


Preventing Generation of Verbatim Memorization in Language Models Gives a False Sense of Privacy

D Ippolito, F Tramèr, M Nasr, C Zhang, M Jagielski…

Studying data memorization in neural language models helps us understand the risks (eg, to privacy or copyright) associated with models regurgitating training data and aids in the development of countermeasures. Many prior works—and some …


Prevention or Promotion? Predicting Author's Regulatory Focus

A Velutharambath, K Sassenberg, R Klinger - Northern European Journal of …, 2023

People differ fundamentally in what motivates them to pursue a goal and how they approach it. For instance, some people seek growth and show eagerness, whereas others prefer security and are vigilant. The concept of regulatory focus is employed …


Priorities for Generative AI Regulation in the UK: CREATe response to the Digital Regulation Cooperation Forum (DRCF)

M Eben, K Erickson, M Kretschmer, G Cifrodelli, Z Li… - 2023

In July 2023 the UK Digital Regulation Cooperation Forum (DRCF) issued a request for comment on the status of generative AI and priority regulatory concerns. The DRCF is comprised of four of the UK’s major regulators: the Competition and …


Privacy Implications of Retrieval-Based Language Models

Y Huang, S Gupta, Z Zhong, K Li, D Chen - arXiv preprint arXiv:2305.14888, 2023

Retrieval-based language models (LMs) have demonstrated improved interpretability, factuality, and adaptability compared to their parametric counterparts, by incorporating retrieved text from external datastores. While it is well known that …


Privacy-Preserving Recommender Systems with Synthetic Query Generation using Differentially Private Large Language Models

AG Carranza, R Farahani, N Ponomareva, A Kurakin… - arXiv preprint arXiv …, 2023

We propose a novel approach for developing privacy-preserving large-scale recommender systems using differentially private (DP) large language models (LLMs) which overcomes certain challenges and limitations in DP training these complex …


PrivacyGLUE: A Benchmark Dataset for General Language Understanding in Privacy Policies

A Shankar, A Waldis, C Bless, MA Rodriguez… - 2023

Benchmarks for general language understanding have been rapidly developing in recent years of NLP research, particularly because of their utility in choosing strong-performing models for practical downstream applications. While benchmarks have been …


Private Federated Learning in Gboard

Y Zhang, D Ramage, Z Xu, Y Zhang, S Zhai, P Kairouz - arXiv preprint arXiv …, 2023

… We have employed model pre-training on public C4 corpus based on Common Crawl dataset [15] combined with training configuration tuning, and achieved almost neutral utility change measured by prediction picked ratio: the ratio of picked …


Private Web Search with Tiptoe

A Henzinger, E Dauterman, H Corrigan-Gibbs… - Cryptology ePrint Archive, 2023

… For text search, we search over the C4 data set, a cleaned version of the Common Crawl’s English web crawl corpus with 364M web pages [109, 110]. The Common Crawl data set is not as comprehensive as the crawls used by commercial …


Probing Pre-trained Large Language Models for Narrative Coherence

The extend to which PTLLMs capture narrative coherence, given (coherent) sequences of text and a set of possible ending sequences, in a zero-shot, multilingual setting has not been explored yet. This research presents an extensive …


Probing Reasoning of Language Models with Inductive In-Context Learning

I Fostiropoulos, L Itti - International Joint Conference on Artificial Intelligence …, 2023

… We attempt to use expressions directly to large corpus on 367 dataset like Common Crawl1 and Github repositories. Based 368 on sub-sample of our results it is computationally expensive 369 to perform exhaustive search and did not lead to …


Probing Taxonomic and Thematic Embeddings for Taxonomic Information

F Klubička, JD Kelleher - arXiv preprint arXiv:2301.10656, 2023

Modelling taxonomic and thematic relatedness is important for building AI with comprehensive natural language understanding. The goal of this paper is to learn more about how taxonomic information is structurally encoded in embeddings. To do …


Profiting from Data Commons: Theory, Evidence, and Strategy Implications

J Potts, A Torrance, D Harhoff, E von Hippel - Strategy Science, 2023

We define data commons as repositories of freely-accessible, “open source” innovation-related data, information and knowledge. Data commons are and can be a significant resource for both innovating and innovation-adopting firms and …


PROMPTING TECHNIQUES FOR NATURAL LANGUAGE GENERATION IN THE MEDICAL DOMAIN

M Rossini, P Torroni, E Cabrio, S Villata

… The model is pre-trained on a modified version of the data scraped by Common Crawl1, an open repository of data crawled from the web … Unfortunately, while the Common Crawl corpus has been used for various NLP applications throughout the …


Prosody Analysis of Audiobooks

C Pethe, Y Yin, S Skiena - arXiv preprint arXiv:2310.06930, 2023

Recent advances in text-to-speech have made it possible to generate natural-sounding audio from text. However, audiobook narrations involve dramatic vocalizations and intonations by the reader, with greater reliance on emotions, dialogues, and …


Protectbot: A Chatbot to Protect Children on Gaming Platforms

Online gaming no longer has limited access, as it has become available to a high percentage of children in recent years. Consequently, children are exposed to multifaceted threats, such as cyberbullying, grooming, and sexting. The online …


ProteinChat: Towards Achieving ChatGPT-Like Functionalities on Protein 3D Structures

The study of proteins is critical in various scientific disciplines, but understanding their complex structure-function relationships remains challenging. Recent advancements in large language models (LLMs) have demonstrated their ability to …


Provably Convergent Subgraph-wise Sampling for Fast GNN Training

J Wang, Z Shi, X Liang, S Ji, B Li, F Wu - arXiv preprint arXiv:2303.11081, 2023

Subgraph-wise sampling -- a promising class of mini-batch training techniques for graph neural networks (GNNs -- is critical for real-world applications. During the message passing (MP) in GNNs, subgraph-wise sampling methods discard …


QASA: Advanced Question Answering on Scientific Articles

Y Lee, K Lee, S Park, D Hwang, J Kim, H Lee, M Lee - 2023

Reasoning is the crux of intellectual thinking. While question answering (QA) tasks are prolific with various computational models and benchmark datasets, they mostly tackle factoid or shallow QA without asking deeper understanding. Dual process …


Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model

Q Ye, J Liu, D Chong, P Zhou, Y Hua, A Liu - arXiv preprint arXiv:2310.09089, 2023

Integrating large language models (LLMs) into healthcare presents potential but faces challenges. Directly pre-training LLMs for domains like medicine is resource-heavy and sometimes unfeasible. Sole reliance on Supervised Fine-tuning (SFT) can result …


QLoRA: Efficient Finetuning of Quantized LLMs

T Dettmers, A Pagnoni, A Holtzman, L Zettlemoyer - arXiv preprint arXiv:2305.14314, 2023

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a …


QRDP: A System that Facilitates the Selection of English Materials for Translator Education

T Abekawa, R Miyata, K Kageura - International Conference on Asian Digital Libraries, 2023

We are currently developing a system that facilitates teachers of translation to evaluate and select suitable document segments as materials for translator training. The system needs to satisfy several requirements. Firstly, because translation is an …


QUADRo: Dataset and Models for QUestion-Answer Database Retrieval

S Campese, I Lauriola, A Moschitti - arXiv preprint arXiv:2304.01003, 2023

… (2021): given an input question, we queried a 2020 CommonCrawl snapshot1 using BM25 and we selected the 200 most relevant documents. Then, documents are split into sentences and a state-of-the-art sentence selector (Lauriola … 1https://commoncrawl.org/2020/?utm_sou …


Qualitative Research Methods for Large Language Models: Conducting Semi-Structured Interviews with ChatGPT and BARD on Computer Science Education

A Dengel, R Gehrlein, D Fernes, S Görlich, J Maurer… - Informatics, 2023

… In 2020, they published GPT 3 [9], which was trained on around 570 GB of text from a filtered version of Common Crawl, a openly available crawl of the internet. For newer models such as GPT 3.5 and GPT 4 [1], which were released in early 2023 …


Quality Evaluation of Summarization Models for Patent Documents

J Ding, H Chen, S Kolapudi, L Pobbathi, H Nguyen - 2023 IEEE 23rd International …, 2023

Several recently developed neural network models have shown their potential for automated text summarization. However, the evaluation results of these models on summarization of long text are fairly close in almost every major evaluation …


Quality Optimization Methods in Neural Machine Translation Systems

Chapter 1 introduces the research problem, motivation, structure and scope of the thesis. It provides an overview of the included papers, together with details on authors, venues, presentation type, and the contribution of the thesis author. The …


Quality> Quantity: Synthetic Corpora from Foundation Models for Closed-Domain Extractive Question Answering

S Sengupta, C Heaton, S Ghosh, P Nakov, P Mitra - arXiv preprint arXiv:2310.16995, 2023

Domain adaptation, the process of training a model in one domain and applying it to another, has been extensively explored in machine learning. While training a domain-specific foundation model (FM) from scratch is an option, recent methods have focused on …


Quantum Natural Language Processing: A New and Promising Way to Solve NLP Problems

Y Bouakba, H Belhadef - Artificial Intelligence: Theories and Applications: First …, 2023

… It is well known that more dataset and more parameters are essential for training deep transformers from scratch such as, GPT-3 which has used 175 billion parameters, 96 attention layers, and a Common Crawl data-set that is 45 TB in size [7] …


Querying Large Language Models with SQL

M Saeed, N De Cao, P Papotti - arXiv preprint arXiv:2304.00472, 2023

… stored effectively in the parameters of the model: the CommonCrawl+ text corpus takes 45TB, while GPT-3 only 350GB. However, LLMs have their shortcomings, as we discuss next, including poor data manipulation skills, eg, they fail with numerical …