text:
Splunk Match Regex

 Regular expression calculator supports flags like global match (perform regex action on the whole text instead of the first occurrence), ignore case (case of the provided text will be ignored. Users can add, edit, rate, and test regular expressions. Few of them like m and s are important in Splunk based on use case. the bit before the first "|" pipe). 18% Solve Challenge. make permalink clear fields. - Experience with Extraction, Transformation, and Loading of data including skills in SPL and Regex - Experience with the Linux command line interface - Experience with a cloud-based Splunk deployment Pluses: - Bachelors Degree in Information Technology, Cyber Security, Computer Science, or related is preferred. Regular expressions are extremely useful in extracting information from text such as code, log files, spreadsheets, or even documents. The Match-zero-or-more Operator (*). Regular Expression Test Page for Perl. This name pattern matches FireFox 31 ono Windows 7 x86. The current regex pattern looks like it will only get a match when 's' is at the beginning of the string or line and 'a' is at the end of the string or line. In computer science, approximate string matching (often colloquially referred to as fuzzy string searching) is the technique of finding strings that match a pattern approximately (rather than exactly). However, as soon as the matching engine sees that there's no whitespace following the "Foo" that it had saved in $1 , it realizes its mistake and starts over again one character after. "find and replace"-like operations. Splunk Fundamentals I is a ~8 hour free online course that gives you a basic understanding of how Splunk works, I recommend signing up for that and using that as your springboard. The Match-zero-or-more Operator (*). Match type 1: a lookahead checks if we have an empty element - if it finds a comma or a newline. Kleene, Mr. Regex IDS allows you to take a list of regular expression rules and match them against your data, for example using the PHPIDS rules to identify known intrusion attempts in Apache access logs. Users can add, edit, rate, and test regular expressions. • Syslog-ng packages and configs pushed to clients via Puppet • Noisy logs (DHCP, Iptables, etc. Start studying Splunk Fundamentals - Part 2. Automatic lookups, which are set up using Splunk Manager, match values implicitly. Get fast answers and downloadable apps for Splunk, the IT Search solution for Log Management, Operations, Security, and Compliance. EasyMax Score: 5Success Rate: 96. Experience with perl regular expressions are preferred, but use of regex objects in any scripting language (bash, python, c++, ruby, etc. * Tell Splunk to monitor the /storage/datacenter directory * Set a whitelist for this input * Edit the REGEX to match all files that contain “host” in the underlying path * Edit the REGEX to match all files that contain “webserver” in the underlying path * Edit the REGEX to match all files that start with “200906”. Use CASE() and TERM() to match phrases - Splunk Documentation. Here are the regex in transforms. [aeiou] Matches any single character included in the specified set of characters. Capturing group. Welcome Welcome to Splunk Answers, a Q&A forum for users to find answers to questions about deploying, managing, and using Splunk products. Splunk regex cheat sheet: These regular expressions are to be used on characters alone, and the possible usage has been explained in the example section on the tabular form below. Regular Expression Engine Comparison Chart. Through the 'script' search command a user can call commands defined in their custom application which includes arbitrary perl or python code. When the match runs, the first part of the regular expression (\ b (foo)) finds a possible match right at the beginning of the string, and loads up$1 with "Foo". check_value: regex for matching on the value pulled out of the cdb when using types: address_match_key_value, match_key_value. $1) are allowed. Search & Replace has extensive support for Regular Expressions for advanced search &/or replace, where the search string follows a rule and is not always exactly the same. Note: Splunk uses Perl compatible regular expressions. Splunk eval command. Simple date dd/mm/yyyy Find Substring within a string that begins and ends with paranthesis 10-digit phone number with hyphens Match anything after the specified all except word Match if doesn't start with string RegEx for Json Find any word in a list of words Java Variable. startIndex = regexpi(str,expression) returns the starting index of each substring of str that matches the character patterns specified by the regular expression, without regard to letter case. Regular expressions match patterns of characters in text and are used for extracting default fields, recognizing binary file types, and automatic assignation of. That might include multiple delimiters. 3 The possessive quantifier can drastically change the way your regex matches its input so some care must be taken to ensure the regex is still matching what it should be matching. This primer helps you create valid regular expressions. Sorting results is the province of the sort command. This regular expression matches both 'abd' and 'acd'. rex [field=] ( [max_match=] [offset_field=]) | (mode=sed ). Press Ctrl+R to open the search and replace pane. Just paste your text in the form below, enter regex, press Extract Matches button, and you get all the data that matches your regular expression. The easy fix would be to move my stanza to the end, but that is too easy. This pattern would match the string A55_END as well as the string 123. Usage of Splunk commands : REGEX is as follows Regex command removes those results which don’t match with the specified regular expression. Click find next. 99 videos Play all Splunk Development Splunk & Machine Learning Visual Basic. Dollar ($) matches the position right after the last character in the string. Regex cheatsheet. conf: ——————————————————————-. By default splunk automatically extracts interesting fields and display them at left column is search result -only condition is log must contain key value pairs which means logs should contains field name and its value - like for username it should appear in log like usename=x or user:x. I need the regex to force "OR" condition to be enclosed with brackets. Bucket names in Splunk indexes are used to: determine who has access to the events. the bit before the first "|" pipe). The lookup definition tells Splunk software where the lookup table is and how to connect to it. Learn vocabulary, terms, and more with flashcards, games, and other study tools. Duplicate group numbers. In addition, logs from the nozzle itself are of sourcetype cf:splunknozzle. before, after, or between characters. It is reported from past two years (2015 and 2016) as one of the must have skill In IT industry and zero unemployment of people. rex [field=] ( [max_match=] [offset_field=]) | (mode=sed ). Matching to the end of line is important, otherwise you might drop your last token. To use the splunk driver as the default logging driver, set the keys log-driver and log-opts to appropriate values in the. I did have an O’Reilly book on Regex, and I have spent a great deal of time on the web looking up how to do regex. Please, refer to match_results for further information. The Matches method uses lazy evaluation to populate the returned. We'll start with the Split method. The problem of approximate string matching is typically divided into two sub-problems: finding approximate substring matches inside a given string and finding dictionary strings that match the. Here are the regex in transforms. com, it will match the asus and automatically replace syslog with asus as the source type in Splunk. conf: ——————————————————————– [. Kaspersky CyberTrace App takes input on port 3000 and forwards it to Feed Service on port 9999. Indexer: Processes machine data and create files organized into directories by age. Online regex tester, debugger with highlighting for PHP, PCRE, Python, Golang and JavaScript. mp4" in a directory and subdirectories? You need to use the find command on a Linux or Unix-like system to search through directories for files. It anonymizes data at search-time based on configuration that is shipped with Splunk. In a previous post I showed how the Anomali ThreatStream Splunk app can integrate with Splunk's own Enterprise Security app to provide analysts with familiar and powerful investigation workflows. Replace performs substitutions, letting you find matches and replace them with alternative text. The key features include: • Streamlining authentication and access from FortiGate such as. Certain regular expression engines will even allow you to specify a range for this repetition such that a{1,3} will match the a character no more than 3 times, but no less than once for example. It matches any single character in that list; if the first character of the list is the caret, "^", then it matches any character NOT in the list. A wealthy user may drop by and want to donate $25,000, while a normal user may want to donate$25. All events from Extract "from" and "to" ﬁelds using regular expressions. We then want to find out match ending with a delimiter, or the end of the line. conf of fortigate add-on Splunk_TA_fortinet_fortigate. World's simplest regexp string extractor. In the beginning. Search CheatSheet Here are some examples illustrating some useful things you can do with the search language. Welcome Welcome to Splunk Answers, a Q&A forum for users to find answers to questions about deploying, managing, and using Splunk products. This article lists common regular expressions and their uses in JavaScript and C#. Regular expression you are using to test if given string is a BASE64 encoded doesn't account for that. determine if the bucket should be searched based on the time range of the search. the rex or regex is the best for that. The regular expression pattern [^\w\. One way to express such a pattern. For example use the following SPL to extract IP Address from the data we used in our previous example:. Learn vocabulary, terms, and more with flashcards, games, and other study tools. By default splunk automatically extracts interesting fields and display them at left column is search result -only condition is log must contain key value pairs which means logs should contains field name and its value - like for username it should appear in log like usename=x or user:x. Most characters are ordinary: they stand for themselves in a pattern, and match the corresponding characters in the. You can remove values that aren't a match for the field you want to define. I did have an O'Reilly book on Regex, and I have spent a great deal of time on the web looking up how to do regex. Start studying Splunk Fundamentals - Part 2. This name pattern matches FireFox 31 ono Windows 7 x86. I cant seem to figure out how to get to write this in Jenkins pipeline? sed -i "txt" "s/id 'com. DevOps Automation. By default, this module uses the credential of. Demonstrate Splunk products and services to customers on site, remotely via webinar sessions, and at field events such as conferences and trade shows. Monitoring input files with a white list Here is a real-world working example of how to use a * Edit the REGEX to match all files that contain вЂњhostвЂќ in, To feed a new set of data to Splunk Enterprise, provide regex definitions You can find other interesting examples in the Splunk Blog's Tips & Tricks. At least one valid "match" must be supplied in order to get any data from a Prometheus federation endpoint. firehose and cloud_controller. admin_read_only (the latter is only required if ADD_APP_INFO is true) and grant-types client_credentials and refresh_token. If cloud_controller. Find below the skeleton of the usage of the command "regex" in SPLUNK :. splunk uses regex for identifying interesting fields in logs like username,credit card number,ip address etc. Dollar ($) matches the position right after the last character in the string. Select your answer. Users can add, edit, rate, and test regular expressions. iBasskung Recommended. Some helpful tools for writing regular expressions. Here is a basic. The input argument op specifies the type of translation that regexptranslate performs. github' ver '1. It seems you have two choices for topology: you can either install the universal forwarder on every host and configure it using local system, or install it on one host and use that one host to gather data from other hosts (using a domain account), which then gets shipped to the data head. Learn vocabulary, terms, and more with flashcards, games, and other study tools. A regular expression (sometimes called a rational expression) is a sequence of characters that define a search pattern, mainly for use in pattern matching with strings, or string matching, i. However, they tend to come with their own different flavor. For a discussion of regular expression syntax and usage, see an online resource such as www. @-] matches any character that is not a word character, a period, an @ symbol, or a hyphen. Splunk regular expressions are PCRE (Perl Compatible Regular Expressions) and use the PCRE C library. Regular expressions (regex) match and parse text. This is probably because of the way that Splunk searches for "tokens" in the index using string (or substring in the case of non-regex wildcard use) matching. Splunk SPL uses perl-compatible regular expressions (PCRE). It covers the most basic Splunk command in the SPL search. Many programs use regular expression to find & replace text. To do so, we might use a pattern like this: ((([^=]+=[^&]*)&?)+) Here we are matching three groups. Let me explain; assume we wanted to match a query string - not just a name-value pair, but the whole string of name-value pairs. [^aeiou] Matches any single character not in the specified set of characters. It began with me implementing regex (regular expressions) matching mechanism for our Sametime Bot to make a more flexible pattern-matching solution than current wildcard matching. filters results based on matching or not matching a regular expression. It matches any single character in that list; if the first character of the list is the caret, "^", then it matches any character NOT in the list. And likewise, [^n-p] will only match any single character. 76% Solve Challenge. Traditionally, approximate string matching algorithms are classified into two categories: on-line and off-line. The Matches method uses lazy evaluation to populate the returned. It took me several hours to make it work right for 3 sets with 3 alternative words in each set. g Global match i * Case-i nse nsitive m * Multiple lines s * Treat string as single line x * Allow comments and whitespace in pattern e * Evaluate replac ement U * Ungreedy pattern * PCRE modifier Regular Expressions Cheat Sheet by DaveChild - Cheatography. ; Select the Pub/Sub Subscription to BigQuery template from the Dataflow template drop-down menu. Today, I found myself looking for a regular expression that matches only the last occurrence of a given expression. ClusterMasterLevel - excess buckets on master Updated alerts: ForwarderLevel - Splunk HEC issues - corrected criteria for newer Splunk versions and added more matching in SearchHeadLevel - SHC Captain unable to establish common bundle - to remove special character from comment Renamed alert: IndexerLevel - Buckets are been frozen due to index sizing to IndexerLevel - Buckets have being frozen. The regex command is used to remove results that do not match the specified regular expressions in Splunk. In the transforms. Terms in this set (15) What are 3 components of Splunk. Start studying SPLUNK Fundamentals 1. For example, the pattern [0-6] will only match any single digit character from zero to six, and nothing else. Here is a basic example that will drop everything except events that contain the string login In props. With the help of regex command we can perfectly match the search string How to Make Search String Case Sensitive in Splunk. The non-greedy match ends the match at the 1st possible place a match can end, whereas a default match will take the longest match possible. I can match the first line up to and including the cr/lf at the end of the line, but I can' get the match to work when I try to include any additional characters beyond the end of the first line. Splunk Storm recipes for Heroku logs. Find below the skeleton of the usage of the command "regex" in SPLUNK :. Over the various lessons, you will be introduced to a number of special metacharacters used in regular expressions that can be used to match a specific type of character. conf, Splunk has the concept of ‘Priority’, which allows you to prioritize the data matches. You can configure Docker logging to use the splunk driver by default or on a per-container basis. Regular expressions are used to replace text within a string, validating forms, extracting a substring from a string based on a pattern match, and so much more. Difference between NOT and != Event segmentation and searching. Line Anchors. The source to apply the regular expression to. 55) What is input lookup command? This Splunk command returns lookup table in the search result. Usage of Splunk commands : REGEX is as follows. We will try to be as explanatory as possible to make you understand the usage and also the points that need to be noted with the usage. Splunk Sum By Field Value. Retrieve Events. EasyMax Score: 5Success Rate: 98. Traditionally, approximate string matching algorithms are classified into two categories: on-line and off-line. A lookahead doesn't consume characters in the string, but only asserts whether a match is possible or not. Regular Expression Engine Comparison Chart. Starting With Regular Expressions in Splunk In this post, you will to learn how to write RegEX for Splunk, one of the most widely used platforms for data monitoring and analysis. If a raw event contains "From: Susan To: Bob", then from=Susan and to=Bob. This is a Splunk extracted field. See Command types. Splunk offers two commands (rex and regex) in SPL that allow Splunk analysts to utilize regular expressions in order to assign values to new fields or narrow results on the fly as part of their search. A wealthy user may drop by and want to donate$25,000, while a normal user may want to donate $25. The problem of approximate string matching is typically divided into two sub-problems: finding approximate substring matches inside a given string and finding dictionary strings that match the. A string s matches the regex /r1*/ iff either s contains the empty string (always true) or s matches /r1(r1*)/ (or of. To use the splunk driver as the default logging driver, set the keys log-driver and log-opts to appropriate values in the. conf: sendCookedData = [ true|false ] * Set to false if you are sending to a third-party system. Hopefully, this helps with building better searches. You can configure Docker logging to use the splunk driver by default or on a per-container basis. The regular expression is a basic regular expression, and in addition you need to quote the delimiter for the s command. Match type 1: a lookahead checks if we have an empty element - if it finds a comma or a newline. 3 1 Answer. Net : Search in Access Database - DataGridView BindingSource Filter Part 1/2 - Duration: 24:59. Since the name of our example router is asus. Experience and hands-on skills with scripting languages such as Python and Perl, as well as with regular expression (RegEx) pattern matching would be an advantage; Familiar with other open. It has taken quite a bit of time for me to learn the ins and outs of how they work, but once you get the hang of them there is almost always some variation that can be used in the different programming. The corresponding field/value combinations are applied to your dataset as lookup fields. And believe it or not, there are tons of ways you can Splunk that data. Flashcards. WARNING: This is a very early release and has undergone only limited testing. The ABCs of RegEx Regular Expressions' syntax and variety of symbols can be daunting, but if you break them up into their components, and learn those components' rules, there's hope. Think about an email address, with a ruby regex you can define what a valid email address looks like. Each section in this quick reference lists a particular category of characters, operators, and constructs. Usage of Splunk commands : REGEX is as follows Regex command removes those results which don’t match with the specified regular expression. How to use Regex in Splunk searches Regex to extract fields # | rex field=_raw "port (?. There's a free download of Splunk Enterprise that enables all features for 30+ days, you can switch it to a free version after that which costs nothing. Experience with regular expressions and using regular expressions for data retrieval. Match type 1: a lookahead checks if we have an empty element - if it finds a comma or a newline. So in case somebody else might need it: Greedy matching % echo "foobar" | sed 's/<. The result is a regular expression that will match a string if a matches its first part and b matches the rest. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results. They can help you in pattern matching, parsing, filtering of results, and so on. Hello fellow Splunkers, We're new to Splunk, and looking to set it up to monitor our member servers. Note there is no perfect email regex, hence the 99. this results in an empty match. PCRE Cheatsheet lin. The regular expression (regex) tester for NGINX and NGINX Plus takes the guesswork out of regexes, telling you whether a regex for a location or map block matches values as you intend. A regular expression is a pattern that the regular expression engine attempts to match in input text. s2 and s1 matches /r1/ and s2 matches /r2/. conf, Splunk will look at the host where the data is coming from and then try to match based on the REGEX entry. The example "match" string given here matches all metrics. Windows Command Line Regex Replace. In a previous post I showed how the Anomali ThreatStream Splunk app can integrate with Splunk's own Enterprise Security app to provide analysts with familiar and powerful investigation workflows. We will rely on the regex101 website to assist in crafting, verifying and explaining the process. EasyMax Score: 5Success Rate: 98. All events from Extract "from" and "to" ﬁelds using regular expressions. tgz) a4349d006b4aff3edee15a3e9796321a82c9ff528c76f31d32f7f54186654978 SHA256 checksum (fuzzy-search-for-splunk_206. Regex command removes results which do not match with desired regular expression. The regex pattern generated for s*a is "^s. Online testing with the Perl engine is still in beta. After years of recovery attempts this is the only one that helped me through each stage of my recovery - it is so different for everyone and the forum allowed each individual to be honest about what was going on and to get support from a lot of wonderful people. This example shows how you can use the “GUID” pattern to validate some text as being a proper GUID formatted with braces and hyphens. * Tell Splunk to monitor the /storage/datacenter directory * Set a whitelist for this input * Edit the REGEX to match all files that contain “host” in the underlying path * Edit the REGEX to match all files that contain “webserver” in the underlying path * Edit the REGEX to match all files that start with “200906”. If you want the regular expression to match multiple words, you'll also need to include the space in between the words as an allowed character: [A-Z ]+ If you want to allow any whitespace character (e. Regular Expression Methods include re. The non-greedy match ends the match at the 1st possible place a match can end, whereas a default match will take the longest match possible. About Splunk regular expressions. EasyMax Score: 5Success Rate: 98. Replace regexp - replaces matching parts of the text with given string. Other than the Splunk 7. check_value: regex for matching on the value pulled out of the cdb when using types: address_match_key_value, match_key_value. And likewise, [^n-p] will only match any single character. For example, the regular expression "[0123456789]" matches any single digit. Still, I like the. Matches allows you to find the parts of an input string that match a pattern and return them. Match any character \w: Match a word chracter \W: Match a non-word character \d: Match a digit \D: Match any non-digit character \s: Match a whitespace character \S: Match a non-whitespace character \b: Match character at the beginning or end of a word \B: Match a character not at beginning or end of a word \0: Match a NUL character \t: Match a. conf: ——————————————————————-. Splunk is an extremely powerful tool for extracting information from machine data, but machine data is often structured in a way that makes sense to a particular application or process while appearing as a garbled mess to the rest of us. Regular expression you are using to test if given string is a BASE64 encoded doesn't account for that. Like this functionality. Looking at the raw Nessus Output: It was possible to log into the remote host using the supplied asymmetric keys. While I don’t recommend relying fully on erex, it can be a great way to learn regex. findall() Python Flags Many Python Regex Methods and Regex functions take an optional argument called Flags; This flags can modify the meaning of the given Regex pattern; Various Python flags used in Regex Methods are re. RegExr (splunk field team likes this!) Regex101 (PS likes this too!) RETester; Rubular (Ruby expression tester) Applications. Welcome Welcome to Splunk Answers, a Q&A forum for users to find answers to questions about deploying, managing, and using Splunk products. Use CASE() and TERM() to match phrases - Splunk Documentation. It also matches x64. github' ver '1. NET regular expression tester with real-time highlighting and detailed results output. In regular expressions, the asterisk is a metacharacter for zero or more instances of the preceding character. Centralizing Windows Logs. Quick tips for optimization. newStr = regexptranslate(op,str) translates str into a regular expression and returns the result in newStr. For using a regular expression in C# server side, we need to use the following namespace. Match type 1: a lookahead checks if we have an empty element - if it finds a comma or a newline. The current check has a check_type of CHECK_REGEX and fetches the banner setting from the registry. For them (and for Rex), regex is an abbreviation of regular expression. [aeiou] Matches any single character included in the specified set of characters. This pattern would match the string A55_END as well as the string 123. Filtering Metrics using the match_regex plugin. The conditional (? (UC)_END) checks whether the group named UC has been set. For example, imagine that you wrote a form that has a donation field that takes a numerical value in dollars. [startIndex,endIndex] = regexpi (str,expression) returns the starting and ending indices of all. Splunk regular expressions are PCRE (Perl Compatible Regular Expressions) and use the PCRE C library. Here are the regex in transforms. Removing one of the boundaries, \bcat would match cat in catfish , and cat\b would match cat in tomcat , but not vice-versa. The syntax (language format) described on this page is compliant with extended regular expressions (EREs) defined in IEEE POSIX 1003. search()& re. Regex quick reference. How do I reference other fields in a Splunk regex search? 2 Answers. Regular Expression Library provides a searchable database of regular expressions. I need the regex to force "OR" condition to be enclosed with brackets. When you get to the matching part of the regex, this allows you to match complex expressions with compact and readable syntax—and to match the same kind of expressions in multiple places without needing to repeat your regex code. A regular expression is a pattern that could be matched against an input text. You can configure Docker logging to use the splunk driver by default or on a per-container basis. You can use newStr as a regular expression in the regexp, regexpi, and regexprep functions. Literals--the actual characters to search for. [0-9a-fA-F]. Consult the regular expression documentation or the regular expression solutions to common problems section of this page for examples. You can pick a different delimiter to avoid having to quote /. The members of this class are those described for match_results, but using string::const_iterator as its BidirectionalIterator template parameter. When the match runs, the first part of the regular expression (\ b (foo)) finds a possible match right at the beginning of the string, and loads up$1 with "Foo". Splunk will be used to showcase practical applications with big data. For example, xy' (two match-self operators) matches xy'. So you could also just do a text match on that and forget the regex. GitHub Gist: instantly share code, notes, and snippets. Basically I need to match on "-" in the given example but any non numeric character needs to match. In my previous blog, I explained Splunk Events, Event types and Tags that help in simplifying your searches. Difference between NOT and != Event segmentation and searching. To clean up the panel I would like to use the rex command to extract the username from the Security_ID field. data onboarding cheatsheet. Splunkで正規表現を使って検索する方法をご紹介します。 大体以下のコマンドを使うことになると思います。 1. Any single character. Stating a regex in terms of what you don't want to match is a bit harder. Since Splunk is the ultimate swiss army knife for IT, or rather the “belt” in “blackbelt”, I wanted to share with you how I learned about Regex and some powerful ways to use it in your Splunk server. SHA256 checksum (fuzzy-search-for-splunk_207. conf rules, tags, and descriptive lookup file globally. 18% Solve Challenge. We then want to find out match ending with a delimiter, or the end of the line. Regex%vs%Restof%the%(paern)%World% %%%%% % %paern vsvalueExamples bash dos splunksearch'. For instance, with A*, the engine starts out matching zero characters, since * allows the engine to match "zero or more". I am completely new to regex and I would greatly appreciate any help. Here is a basic example that will drop everything except events that contain the string login In props. Regular Expression Library provides a searchable database of regular expressions. I will discuss why lookups are important and how you can associate data from an external source by matching the unique key value. Capturing group named Y. Here are the regex in transforms. Ruby regular expressions ( ruby regex for short) help you find specific patterns inside strings, with the intent of extracting data for further processing. A string s matches the regex /r1r2/ iff there exist substrings s1 and s2 such that s = s1. Hi! I'm trying to create a regular expression that matches formulas like: 1 AND (2 OR 3). tgz) Sometimes, the logs received by syslogd can be transformed and would not match the regex in the app, so no data will be shown on dashboards. Splunk is an extremely powerful tool for extracting information from machine data, but machine data is often structured in a way that makes sense to a particular application or process while appearing as a garbled mess to the rest of us. First, there are two ways to access the events logged in Windows - through the Event Viewer and using the Get-EventLog / Get-WinEvent cmdlets. rexコマンド マッチした値をフィールド値として保持したい場合 1. Ruby regular expressions ( ruby regex for short) help you find specific patterns inside strings, with the intent of extracting data for further processing. Looking through the documentation for props. When you want to search and replace specific patterns of text, use regular expressions. To clean up the panel I would like to use the rex command to extract the username from the Security_ID field. Start studying Splunk Fundamentals 2 Final Quiz. If there are no matches, startIndex is an empty array. The revised regular expression is: Regular expression: \d{3}\-\d{2}\-\d{4} Matches: All social security numbers of the form 123-12-1234. The predefined facilities group together related messages, but you can also match messages against strings and regular expressions to refine which messages from a facility are logged to a file, a user terminal, or a remote destination. or Match type 2: if 1 fails, check for a quoted string or Match type 3: then for a non-quoted string. The current check has a check_type of CHECK_REGEX and fetches the banner setting from the registry. Difference between NOT and != Event segmentation and searching. Thanks Date: 9/23/2019 2:13:42 PM. Pattern: GUID “GUID” is one of the patterns that you can select on the Match panel. Prometheus Metrics for Splunk. Anything here will not be captured and stored into the variable. WARNING: This is a very early release and has undergone only limited testing. M Series,MX Series,T Series,EX Series,SRX Series,QFX Series. Optimizing Searches. Select URLs are defined as Regular Expression only if the application or site URL is entered as a regular expression using the correct syntax. This article lists common regular expressions and their uses in JavaScript and C#. You can pick a different delimiter to avoid having to quote /. Use CASE () and TERM () to match phrases. Since Splunk is the ultimate swiss army knife for IT, or rather the “belt” in “blackbelt”, I wanted to share with you how I learned about Regex and some powerful ways to use it in your Splunk server. DevOps Linux. Travis_Nelson6. It also matches x64. I've recently passed the Splunk User cert exam and I am currently preparing for the Power User exam. info or a manual on the subject. splunk uses regex for identifying interesting fields in logs like username,credit card number,ip address etc. Regular Expression Library provides a searchable database of regular expressions. However, field 2 doesn't work as I am getting the results that do match the regex of field2 and not discarding them. Extract Text From Regular Expressions web developer and programmer tools. SHA256 checksum (fuzzy-search-for-splunk_207. txt) or read online for free. For using a regular expression in C# server side, we need to use the following namespace. Default is greedy. Regex command removes results which do not match with desired regular expression. It is reported from past two years (2015 and 2016) as one of the must have skill In IT industry and zero unemployment of people. At least one valid "match" must be supplied in order to get any data from a Prometheus federation endpoint. When extracting fields, we may choose to use our own regular expression. The ABCs of RegEx Regular Expressions' syntax and variety of symbols can be daunting, but if you break them up into their components, and learn those components' rules, there's hope. Optimizing Searches. This is probably because of the way that Splunk searches for "tokens" in the index using string (or substring in the case of non-regex wildcard use) matching. Prometheus prometheus. Filter and re-arrange how Splunk displays fields within search results. ) Anyone have an idea what I have to do to make Splunk stop reporting the stuff that I didn't capture to begin with? This behavior is really counter-intuitive. If we don’t specify any field with the regex command then by default the regular expression applied on the _raw field. Learn vocabulary, terms, and more with flashcards, games, and other study tools. We will rely on the regex101 website to assist in crafting, verifying and explaining the process. After searching on this sub and answers. Regex command removes those results which don't match with the specified regular expression. The regex pattern generated for s*a is "^s. Dollar (\$) matches the position right after the last character in the string. Get fast answers and downloadable apps for Splunk, the IT Search solution for Log Management, Operations, Security, and Compliance. Regex Multiple Occurrences Of Same Character. -Design & Architect Solutions Enterprises for SIEM Implementation(Splunk). If the destination field matches to an already existing field name, then it overwrites the value of the matched field with the eval expression's result. M Series,MX Series,T Series,EX Series,SRX Series,QFX Series. Rather they match a position i. "\0011" is the equivalent of "\001" & "1". About Splunk regular expressions. So you could also just do a text match on that and forget the regex. The regular expression is a basic regular expression, and in addition you need to quote the delimiter for the s command. Using regex for matching multiple words anywhere in the sentence This blog post is to save time for those developers who want to use Regular Expressions to determine whether ALL of the multiple words are located anywhere in the input string. conf, Splunk has the concept of ‘Priority’, which allows you to prioritize the data matches. Example: I want to search word 123456 in a paragraph. Each section in this quick reference lists a particular category of characters, operators, and constructs. 350, and similar, so I decided to write a more sophisticated regex that actually validates that the number is between 0 and 255. tgz) Sometimes, the logs received by syslogd can be transformed and would not match the regex in the app, so no data will be shown on dashboards. When you get to the matching part of the regex, this allows you to match complex expressions with compact and readable syntax—and to match the same kind of expressions in multiple places without needing to repeat your regex code. Regular expression you are using to test if given string is a BASE64 encoded doesn't account for that. A string s matches the regex /r1|r2/ iff either s matches /r1/ or s matches /r2/ (or both). This configuration is to gather all metrics from a Prometheus server. Just paste your text in the form below, enter regex, press Extract Matches button, and you get all the data that matches your regular expression. It's easier for me to work with and test that way, and it meant I didn't need to learn how to define and reference the primitives. Lesson 7: Mr. To use the splunk driver as the default logging driver, set the keys log-driver and log-opts to appropriate values in the. It seems you have two choices for topology: you can either install the universal forwarder on every host and configure it using local system, or install it on one host and use that one host to gather data from other hosts (using a domain account), which then gets shipped to the data head. Stating a regex in terms of what you don't want to match is a bit harder. When extracting fields, we may choose to use our own regular expression. In Azure Monitor, it's a relational operator. After searching on this sub and answers. Use the regex command to remove results that do not match the specified regular expression. mp4" in a directory and subdirectories? You need to use the find command on a Linux or Unix-like system to search through directories for files. Matches n, where n is an octal escape value. A regular expression is a pattern that is matched against a subject string from left to right. Match the Y'th captured group. regex matching in splunk. NET regular expression tester with real-time highlighting and detailed results output. newStr = regexptranslate(op,str) translates str into a regular expression and returns the result in newStr. I refused to search the web, since I wanted to use my brain, and it took me about 10 minutes to come up with this to match a digit between 0-255:. In my previous blog, I explained Splunk Events, Event types and Tags that help in simplifying your searches. Difference between NOT and != Event segmentation and searching. Recurse into entire pattern. check_value: regex for matching on the value pulled out of the cdb when using types: address_match_key_value, match_key_value. Rather they match a position i. Looking at the raw Nessus Output: It was possible to log into the remote host using the supplied asymmetric keys. Although Splunk allows for the definition of regex primitives that are combined to make more complex constructs, I chose to build one big regex. At last we are getting only those events where given search string( ABHAY ) is in Upper-Case. A regular expression (or RE) specifies a set of strings that matches it; the functions in this module let you check if a particular string matches a given regular expression. ) Anyone have an idea what I have to do to make Splunk stop reporting the stuff that I didn't capture to begin with? This behavior is really counter-intuitive. It covers the most basic Splunk command in the SPL search. This is a work in progress - Questions, comments, criticism, or requests can be directed Here. In the below post, I'll cover some of the Do's and Don'ts to Splunking your syslog, and for what it's worth, every environment is different. By default Base64OutputStream is using CHUNK_SIZE = 76, and CHUNK_SEPARATOR = {'\r', '\n'}. When Splunk software processes events at index-time and search-time, the software extracts fields based on configuration file definitions and user- defined patterns. Regex command removes those results which don't match with the specified regular expression. conf, then add rewrite to transforms. io, a Cloud Native Computing Foundation project, is a systems and service monitoring system. Default is greedy. and it should match as whole and event if these exists as. Get fast answers and downloadable apps for Splunk, the IT Search solution for Log Management, Operations, Security, and Compliance. I know, a no-brainer, but I wasted precious minutes on it and shell scripts should be, after all, quick and easy. Use CASE () and TERM () to match phrases. How to use Regex in Splunk searches Regex to extract fields # | rex field=_raw "port (?. Recurse into numbered group Y. For general information about regular expressions, see About Splunk regular expressions in the Knowledge Manager Manual. Examples of regular expression syntax are given later in this chapter. 96% Solve Challenge. Regular expression to match valid UK postcodes. About Splunk regular expressions. Regular expressions (regex) match and parse text. Simple date dd/mm/yyyy Find Substring within a string that begins and ends with paranthesis 10-digit phone number with hyphens Match anything after the specified all except word Match if doesn't start with string RegEx for Json Find any word in a list of words Java Variable. Start of string. The regular expression pattern for which the Matches (String) method searches is defined by the call to one of the Regex class constructors. Caret (^) matches the position before the first character in the string. Hi! I'm trying to create a regular expression that matches formulas like: 1 AND (2 OR 3). That might include multiple delimiters. Split Earlier in the regular expressions tutorial we saw two key methods. Matches allows you to find the parts of an input string that match a pattern and return them. If there are no matches, startIndex is an empty array. regular-expressions. Search CheatSheet Here are some examples illustrating some useful things you can do with the search language. You can still take a look, but it might be a bit quirky. For example use the following SPL to extract IP Address from the data we used in our previous example:. For example, the regular expression "[0123456789]" matches any single digit. Search commands that use regular expressions include rex and regex and evaluation functions such as match and replace. conf: ——————————————————————-. EasyMax Score: 5Success Rate: 96. Net : Search in Access Database - DataGridView BindingSource Filter Part 1/2 - Duration: 24:59. In case you ever wonder, below is the regex used in type=”email” from W3C:. Earlier in the regular expressions tutorial we saw two key methods. For a brief introduction, see. Regex IDS allows you to take a list of regular expression rules and match them against your data, for example using the PHPIDS rules to identify known intrusion attempts in Apache access logs. conf: sendCookedData = [ true|false ] * Set to false if you are sending to a third-party system. Use it to make a field match a globally unique identifier such as {7E9081B5-9A6D-4CC1-A8C3-47F69FB4198D}. RegExr (splunk field team likes this!) Regex101 (PS likes this too!) RETester; Rubular (Ruby expression tester) Applications. Using the lookup command matches values in external tables explicitly. Parsing the browscap. The power of a regular expression is that you can specify classes of character data such as all letters or all digits. Users can add, edit, rate, and test regular expressions. About retrieving events. This article lists common regular expressions and their uses in JavaScript and C#. One common question we’re hearing you ask, how can key-value pairs be extracted from fields within the JSON?. Default is greedy. It prints each. What does (?i) and [email protected] in this regex mean [duplicate] Ask Question Asked 5 years, Browse other questions tagged regex splunk or ask your own question. With on-line algorithms the pattern can be processed before searching but the text cannot. Matching Whitespace & Non-Whitespace Character. Matching Anything But a Newline. You can validate where the data originated from You can remove values that aren't a match for the field you want to define You cannot. Regular expression to match valid UK postcodes. Write better searches. Use the regex command to remove results that do not match the specified regular expression. A regular expression is a pattern that the regular expression engine attempts to match in input text. Part of the challenge here is that there is a line break in the plugin output that is not immediately apparent to the SC GUI. indicate where the bucket should be stored when it transfers from hot to cold. Line Anchors. Although Splunk allows for the definition of regex primitives that are combined to make more complex constructs, I chose to build one big regex. The meaning of the asterisk ( * ) depends on its use. If we don't specify any field with the regex command then by default the regular expression applied on the _raw field. This regex takes into account those rules. Regular Expressions (REGEXES) Regular Expressions are useful in multiple areas: search commands regex and rex; eval functions match() and replace(); and in field extraction. Users can add, edit, rate, and test regular expressions. In the single-instance integration scheme, Splunk Apps and Feed Service by default are configured to work on the same computer (IP address is 127. Sometimes it is easy to forget that these commands are using regex becuase it is so tightly integrated. The Nozzle requires a client with the authorities doppler. 3 Overview of Regular Expression Syntax. Regular expression to match valid UK postcodes. I updated my stanza to contain priority=100 for my data. The regular expression (regex) tester for NGINX and NGINX Plus takes the guesswork out of regexes, telling you whether a regex for a location or map block matches values as you intend. a specific sequence of. Simple date dd/mm/yyyy Find Substring within a string that begins and ends with paranthesis 10-digit phone number with hyphens Match anything after the specified all except word Match if doesn't start with string RegEx for Json Find any word in a list of words Java Variable. Filter match criteria in syslog-ng is flexible and can even match with regex. Splunk is totally domain independent and least unemployment skill. The regex \bcat\b would therefore match cat in a black cat, but it wouldn't match it in catatonic, tomcat or certificate. Install the RegEx Match & Replace Task in to your Visual Studio Team Services account and search for the task in the available tasks. filter = This will filter out and index only the content that matches the regex. The key features include: • Streamlining authentication and access from FortiGate such as. Pattern recognition connects regex code to solve the problem. Stating a regex in terms of what you don't want to match is a bit harder. And likewise, [^n-p] will only match any single character. For example, the pattern [0-6] will only match any single digit character from zero to six, and nothing else. egrep or sed. Any single character except: a, b, or c. It will find "she sells sea" as the first match. RegEx for pattern matching and extraction splunk splunk-enterprise field-extraction rex transforms. i am using the pcregextest binary in /opt/splunk/bin to test my regex and cannot get the access. 76% Solve Challenge. Description; Requirements and Dependencies. field extractions and the field extractor (Splunk) Splunk offers a visual aide called the field extractor which can offer two field extraction methods: regular expression and delimiters. Use CASE () and TERM () to match phrases. Regex command removes results which do not match with desired regular expression. if it is set, only send console log to splunk for the job whose build url matches the pattern splunkins. Regular expressions match patterns of characters in text and are used for extracting default fields, recognizing binary file types, and automatic assignation of. If they do, only the first two digits comprise the expression. And likewise, [^n-p] will only match any single character. Splunk Escape Backslash In Search i need a string like: a - '\hline' Error: '\h' is an unrecognized escape in character string starting "\h" to include some latex code in a xtable object, but I only mange to do: a. To know how to use sed, people should understand regular expressions (regexp for short). Pattern: GUID "GUID" is one of the patterns that you can select on the Match panel. A word character is any letter, decimal digit, or punctuation connector such as an underscore. tgz) a4349d006b4aff3edee15a3e9796321a82c9ff528c76f31d32f7f54186654978 SHA256 checksum (fuzzy-search-for-splunk_206. Earlier in the regular expressions tutorial we saw two key methods. Hello fellow Splunkers, We're new to Splunk, and looking to set it up to monitor our member servers. @-] matches any character that is not a word character, a period, an @ symbol, or a hyphen. Welcome Welcome to Splunk Answers, a Q&A forum for users to find answers to questions about deploying, managing, and using Splunk products. This is probably because of the way that Splunk searches for "tokens" in the index using string (or substring in the case of non-regex wildcard use) matching. It consists of four 1-byte segments (octets), each segment has a value between. DevOps Linux. [0-9a-fA-F]. Experience with Extraction, Transformation, and Loading of data including skills in SPL and Regex Experience with the Linux command line interface Experience with a cloud-based Splunk deployment. com Created Date:. anubhava Regular expression to match chunked BASE64 (with given chunk size 64 and separator \r ) string. Get fast answers and downloadable apps for Splunk, the IT Search solution for Log Management, Operations, Security, and Compliance. Why is the regular expression to match URI patterns in my Splunk search not working as expected? 1 Answer. Difference between NOT and != Event segmentation and searching. Regular expressions. g Global match i * Case-i nse nsitive m * Multiple lines s * Treat string as single line x * Allow comments and whitespace in pattern e * Evaluate replac ement U * Ungreedy pattern * PCRE modifier Regular Expressions Cheat Sheet by DaveChild - Cheatography. World's simplest regexp string extractor. You can remove values that aren't a match for the field you want to define. conf, then add rewrite to transforms. The "scrub" command is an interesting one. I refused to search the web, since I wanted to use my brain, and it took me about 10 minutes to come up with this to match a digit between 0-255:. Get fast answers and downloadable apps for Splunk, the IT Search solution for Log Management, Operations, Security, and Compliance. One easy way to exclude text from a match is negative lookbehind : But not all regex flavors support negative lookbehind. If the destination field matches to an already existing field name, then it overwrites the value of the matched field with the eval expression's result. Learn vocabulary, terms, and more with flashcards, games, and other study tools. You can configure Docker logging to use the splunk driver by default or on a per-container basis. EasyMax Score: 5Success Rate: 82. Parsing the browscap. Start studying Splunk Fundamentals - Part 2. This example shows how you can use the "GUID" pattern to validate some text as being a proper GUID formatted with braces and hyphens. firehose and cloud_controller. It prints each. Welcome Welcome to Splunk Answers, a Q&A forum for users to find answers to questions about deploying, managing, and using Splunk products. Hello fellow Splunkers, We're new to Splunk, and looking to set it up to monitor our member servers. try this to extract for example properties values and put them in one field:| rex max_match=0 field=_raw " HERE YOU PUT YOUR REGEX" If you cannot easily write regex like me, use IFX,do as if you want to extract the values, the IFX will provide the regular expression that can use there. Quick tips for optimization. The key features include: • Streamlining authentication and access from FortiGate such as. Expression to test. Write better searches. qlez60odegjymgb qr9jibgf7r qm36zpb7l2 kuwaamu04uilu wfkxglt75yvrr q7awtlbv6l3 6t5jdt1rixn 580vu00syc0 egz850xy0y6 8wytkjfghf9fl 0wm0yvg9wboeiw k60sdauk8bd umebz5ac6cg msujct6esr70n k2mfgbcrqvlrl tres8itxl1x wv25x9u8vy1 htn4xni33l 4oz3f342emk315h q4ikr9gmor xv9lcp6qw4 0y4cvr3mb9c 2ubmlikqmxeuo kuio0zw5vc2995y x941fjagd6yege z928hdi5686 ofjvnjxtk4bzzv2


text:
# Bootstrapped Neural Networks, RFs, and the Mushroom bandit

Thompson Sampling is a very simple yet effective method to addressing the exploration-exploitation dilemma in reinforcement/online learning. In this series of posts, I’ll introduce some applications of Thompson Sampling in simple examples, trying to show some cool visuals along the way. All the code can be found on my GitHub page here.

In this post, we explore more advanced algorithms, starting with a simple trial to get uncertainties out of Random Forests and then moving to Bootstrapped Neural Networks. We use these methods to solve the “Mushroom” bandit, a contextual bandit game where the agent must decide, from a bucket of edible and poisonous mushrooms, which ones to eat.

## Solving the Mushroom Bandit

In this notebook, we’ll run an experiment inspired by the paper Weight Uncertainty in Neural Networks (Blundell et. al, 2015), the Mushroom bandit. In this setting, we use the Mushroom dataset from UCI to simulate a bandit-like game, in order to test some Thompson Sampling-inspired algorithms that try to address the exploration-exploitation tradeoff. Particularly, we will test two interesting ideas:

1. Bootstrapped Neural Networks: based on Bootstrapped DQN, a proven method for deep exploration in reinforcement learning, presented by Osband et. al. (2016). Here, we do not have a full RL setting, so we call it a Bootstrapped Neural Network, rather than a Bootstrapped DQN.

2. Sampling from a Random Forest: let us try to use the individual tree predictions of a Random Forest to build uncertainty estimates for our game.

We will also test the greedy counterparts of these ideas, a regular neural network and a regular Random Forest.

## Test with toy data

Let us first run our models with simple toy data to check uncertainty estimates.

### Generating the data

We will use the formula given by the paper Weight Uncertainty in Neural Networks to generate the data:

$y = x + 0.3 \cdot{} sin(2π(x + \epsilon)) + 0.3 \cdot{} sin(4 \cdot{} \pi \cdot{}(x + \epsilon)) + \epsilon$

where $\epsilon \sim \mathcal{N}(0, 0.02)$. We take a hundred $x$ samples from an uniform distribution $\mathcal{U}(0, 0.5)$. This results in the following plot:

### Random Forest

Let us start with the simpler model, a Random Forest. What we will try to do is randomize the forest’s output, by sampling sets of decision trees and averaging their predictions. The following code accomplishes that, drawing 20 samples of our approximate posterior distribution.

# instance of RF
et = ExtraTreesRegressor(n_estimators=100, min_samples_leaf=2)

# let us fit it to the data
et.fit(x.reshape(-1,1),y)

# let us generate data to check the fit
x_grid = np.linspace(-0.5, 1.0, 200).reshape(-1,1)

# number of samples from the forest
N_SAMPLES = 20

# let us get the predictions of four trees at a time
tree_samples = [np.random.choice(et.estimators_, 4) for i in range(N_SAMPLES)]

# let us predict with these guys
y_grid = np.array([np.array([e.predict(x_grid) for e in np.random.choice(et.estimators_, 2)]).mean(axis=0) for tree_sample in tree_samples])

The sampling procedure results in the following plot:

The uncertainty estimates concentrate where there is more data, which is a good sign. However, uncertainties do not increase as we get distant from the samples. This may be due to the rule-based learning procedure of RF’s, which prevents them from extrapolating.

### Bootstrapped Neural Network

Let us check the uncertainty estimates produced by our Bootstrapped NN. As introduced by Osband et. al. (2016), we will train a network with a shared layer and many bootstrapped heads, which will diverge a little bit on their predictions. This divergence will drive efficient exploration and will give us reasonable uncertainty estimates.

I made a quick and dirty implementation of this model on Keras, but it’s far from ideal. Here, I generate the bootstrap samples at model compile time, and return $N$ bootstrapped samples of the training dataset, consuming a lot of memory. The model trains on 10 different inputs and outputs.

# defining the model #

# shared network
shared_net = Sequential([Dense(1, activation='linear', input_dim=1),
Dense(16, activation='relu'),
Dense(16, activation='relu'),
Dense(16, activation='relu')])

# bootstrap rate and number of heads
BOOTSTRAP_RATE = 1.00

# initializing lists
inputs = []
x_btrap = []
y_btrap = []

# loop for creating heads and sampled data points

# deciding sample of data points
sample_idx = np.random.choice(range(x.shape[0]), int(x.shape[0]*BOOTSTRAP_RATE), True)

# defining the heads in the graph
input_temp = Input(shape=(1,))
shared_temp = shared_net(input_temp)
inputs.append(input_temp)

# defining data
x_btrap.append(x[sample_idx])
y_btrap.append(y[sample_idx])

# defining the model and plotting it
plot_model(model)

# compiling

# fitting the model
model.fit(x_btrap, y_btrap, epochs=500, batch_size=64, verbose=0);

Even with my dirty code, the results are beautiful. We can see a reasonable uncertainty estimate given by the individual predictions of the bootstrapped heads:

We are now ready to play the Mushroom bandit game!

### The Mushroom bandit

The game is set up as follows:

1. The agent is initialized with 50 random mushrooms and if they’re edible or poisonous
2. At each round, the agent is presented with all the mushrooms which were not already eaten.
3. The agent ranks the mushrooms, with the most edible mushroom at the top.
4. The agent eats the top $k$ mushrooms and receives feedback. The agent reward is the actual observation.

Here’s my code to implement the game:

# coding a class for this
class MushroomBandit:

# initializing
def __init__(self, X, y):

# storing the initialization
self.X = X; self.y = y

# let us keep eaten and not eaten mushrooms as well
self.X_not_eaten = X.copy()
self.y_not_eaten = y.copy()
self.X_eaten = pd.DataFrame()
self.y_eaten = pd.Series()

# function to show not eaten mushrooms
def get_not_eaten(self):
return self.X_not_eaten

# function to show eaten mushrrooms
def get_eaten(self):
return self.X_eaten, self.y_eaten

# function to get which mushrooms will be eaten
def eat_mushrooms(self, mushroom_idx):

# get feedback
feedback = self.y_not_eaten.loc[mushroom_idx].copy()

# remove eaten mushrooms from not eaten dataset
self.X_not_eaten = self.X_not_eaten.drop(mushroom_idx)
self.y_not_eaten = self.y_not_eaten.drop(mushroom_idx)

# add eaten mushrooms from eaten dataset
self.X_eaten = pd.concat([self.X_eaten, self.X.loc[mushroom_idx]])
self.y_eaten = pd.concat([self.y_eaten, self.y.loc[mushroom_idx]])

# return feedback
return feedback

We first play the game with a random policy. The code for running an episode of the game is very simple:

# number of mushrooms eaten per round
K = 100

# instance of the game
mb = MushroomBandit(X, y)

# we initialize the agent by eating the first 50 mushrooms of the base

# and we get the number of rounds
n_rounds = int(mb.get_not_eaten().shape[0]/K) + 1

# list and to accumulate rewards
cum_rewards = 0
cum_rewards_list_random = []

# then, we play until no mushrooms are left
for round_id in tqdm(range(n_rounds)):

# choosing random mushrooms
to_eat = np.random.choice(mb.get_not_eaten().index, K)

# eating
feedback = mb.eat_mushrooms(to_eat)

# saving cumulative rewards
cum_rewards += feedback.sum()
cum_rewards_list_random.append(cum_rewards)

At the end of game we inspect the number of edible mushrooms eaten during the course of the episode. The random policy fares as we would expect: a straight line, as it eats edible mushrooms at the same rate as poisonous ones (considering class imbalance).

We then proceed to replacing the random policy with a regular neural network and our bootstrapped neural network. The regular net is the same as a greedy method, as it only uses the expectation learned by the net to make decisions. The bootstrapped net emulates Thompson Sampling, as each head acts like a sample of a posterior distribution. Thus, we expect that the BNN explores efficiently and makes better decisions than the regular net. The plot below shows the results achieved:

As we would expect, the boostrapped net finds all the edible mushrooms faster than the regular net, as it explores more efficiently. This technique provides a good bayesian approximation to neural networks, and can be used in a number of other settings. There’s great discussion about bayesian deep learning right now, particularly concerning the advantages of this approach compared to Monte Carlo dropout. I look forward to seeing the evolution of the field in the next years.

Finally, we run a regular (greedy) RF against our sampled RF. Interestingly, both RF models beat neural networks, with the sampled RF overcoming the greedy RF only at the very end of game. This is a very curious result, which teases us to test the models in other datasets. If you want to do that, remember that the code for this experiment is freely available at my GitHub page.

## Conclusion

Even that the mushroom bandit was easily solved by the models, we can check that randomizing decisions offer improvements over greedy decisions. On the final plot the randomized models outperformed the greedy models, especially on the hardest mushrooms (the last ones to be eaten). In this particular case, Random Forest compared favourably against neural networks, and are trained and sampled in a fraction of the time.

In the future, we should try out other methods like UCB and $\epsilon$-greedy, and check if they will be outperformed by Thompson Sampling, as recent research suggests.

format: plain


text:
My Math Forum Saw these at another forum

 Number Theory Number Theory Math Forum

 October 7th, 2018, 02:19 PM #1 Senior Member     Joined: Sep 2015 From: USA Posts: 2,195 Thanks: 1152 Saw these at another forum i) Without calculating $S$ show $S = \sum \limits_{r=1}^{1000}~(6r^2 + 32r) \text{ is a multiple of 2017}$ ii) Prove that $3\sum \limits_{r=1}^n~(r-2)(r-3)= (n-1)(n-2)(n-3)+6$ The second one could probably be proven by brute force but I'm sure you number theory folks have shortcuts for these. Thanks from Joppy
 October 7th, 2018, 06:18 PM #2 Global Moderator   Joined: Dec 2006 Posts: 19,950 Thanks: 1842 (i) Obtain a formula for the first n terms of the sum, then show that one factor of it evaluates to 2017 for n = 1000. (ii) Use mathematical induction.
 October 8th, 2018, 10:22 AM #3 Senior Member   Joined: Dec 2015 From: Earth Posts: 270 Thanks: 31 Can anyone post the solution of the first problem? I never saw such a problem Last edited by skipjack; October 9th, 2018 at 01:15 AM.
 October 8th, 2018, 11:33 AM #4 Math Team   Joined: Dec 2013 From: Colombia Posts: 7,508 Thanks: 2513 Math Focus: Mainly analysis and algebra \begin{align}\sum_{r=1}^{n}r^2 &= \frac16 n(2n+1)(n+1) \\ \sum_{r=1}^{n}r &= \frac12 n(n+1) \\[12pt] \sum_{r=1}^{n}(6r^2 + 32r) &= n(2n+1)(n+1) + 16n(n+1) \\ &= (2n+17)n(n+1) \\ \sum_{r=1}^{1000}(6r^2 + 32r) &= 2017 \times 1000 \times 1001\end{align} Thanks from greg1313 Last edited by v8archie; October 8th, 2018 at 11:37 AM.
 October 8th, 2018, 08:06 PM #5 Math Team   Joined: Dec 2013 From: Colombia Posts: 7,508 Thanks: 2513 Math Focus: Mainly analysis and algebra \begin{align}3\sum \limits_{r=1}^n~(r-2)(r-3) &= 3\sum_{r=1}^{n}(r^2 - 5r +6) \\ &= \tfrac12 n(2n+1)(n+1) - \tfrac{15}2 n(n+1) + 18n \\ &= \tfrac12(2n^3 + 3n^2 + n) - \frac{15}2(n^2+n) + \frac12 (36n) \\ &= \tfrac12(2n^3+3n^2+n - 15n^2-15n + 36n) \\ &= \tfrac12(2n^3 - 12n^2 + 22n) \\ &= n^3 - 6n^2 + 11n - 6 + 6 \\ &= (n-1)(n^2 - 5n + 6) + 6 \\ &= (n-1)(n-2)(n-3) + 6 \end{align} \begin{align}3\sum \limits_{r=1}^n~(r-2)(r-3) &= 3\sum_{r=-2}^{n-3}(r+1)r \\ &= 6 + 0 + 0 + 3\sum_{r=0}^{n-3}(r+1)r \\ &= 6 + 3\sum_{r=0}^{n-3}(r^2+r) \\ &= 6 + \tfrac36(n-3)\big(2(n-3)+1\big)\big((n-3)+1\big) + \tfrac32(n-3)\big((n-3)+1\big) \\ &= 6 + \tfrac12(n-3)(2n-5)(n-2) + \tfrac32(n-3)(n-2) \\ &= 6 + (n-3)(n-2)\left(\tfrac12(2n-5) + \tfrac32\right) \\ &= 6 + (n-3)(n-2)(n-1) \\ \end{align} Last edited by v8archie; October 8th, 2018 at 08:22 PM.
 October 9th, 2018, 01:27 AM #6 Global Moderator   Joined: Dec 2006 Posts: 19,950 Thanks: 1842 (ii) Let $f(n)$ denote the given function of n, and note that $f(1) = 6 = 3(1 - 2)(1 - 3)$, as required. \displaystyle \begin{align*}f(n) - f(n - 1) &= (n - 1)(n - 2)(n - 3) + 6 - ((n - 2)(n - 3)(n - 4) + 6) \\ &= (n - 2)(n - 3)(n - 1 - (n - 4)) \\ &= 3(n - 2)(n - 3) \end{align*} The above suffices to prove the given equation by mathematical induction. Thanks from v8archie

 Tags forum

 Thread Tools Display Modes Linear Mode

 Similar Threads Thread Thread Starter Forum Replies Last Post Rocket88 New Users 5 May 13th, 2016 12:20 PM spo0ofer Probability and Statistics 1 April 18th, 2016 04:02 PM Tores1 Pre-Calculus 4 November 2nd, 2015 04:42 AM Mark W. New Users 6 September 16th, 2008 09:15 AM

 Contact - Home - Forums - Cryptocurrency Forum - Top


text:
# An ambient air quality evaluation model based on improved evidence theory

### Data

To validate the performance of the proposed DCreWeight model, select hourly air pollution data in Xi’an from June 1, 2014, to May 1, 2016. The years are randomly selected. In this paper, the null values are processed using the linear interpolation method. According to the proposed DCreWeight model, the comprehensive air quality evaluation results on a day are as follows (see Fig. 2).

### Evaluation indicators

1. (1)

Evaluation indicators based on AQI

The national AQI standard (HJ633-2012[Z]) describes the air quality level. AQI standard denotes that the highest pollutant concentration determines the air quality level. It highlights the contribution of one pollutant. Equation (20) shows the calculation of AQI. It defines the concentration limits [BPLo, BPHi] and IAQI limits [IAQIHi, IAQILo].

$${ text {AQI}} = { text {max}} left ({ left ({{ text {IAQI}} _ {{{ text {Hi}}}} – { text {IAQI }} _ {{{ text {Lo}}}}}} right) * left ({{ text {C}} _ {{ text {P}}} – { text {BP}} _ { {{ text {Lo}}}}} right) / left ({{ text {BP}} _ {{{ text {Hi}}}} – { text {BP}} _ {{{ text {Lo}}}}} right) + { text {IAQI}} _ {{{ text {Lo}}}}} right)$$

(20)

where CP is the concentration of pollutant P.

Taking the national AQI as the pollution standard, the indicator MAE, RMSE and an index of agreement can be calculated to analyze the performance of evaluation models. Count the number of days when AQI is equal to the evaluation level of models, and define it as right_num.

Defined AQI_MAE, AQI_RMSE and AQI_an index of agreement as evaluation indicators. The above evaluation indicators based on AQI can be calculated as follows:

$${ text {AQI}} _ { text {MAE}} = frac {1} {n} mathop sum limits_ {i = 1} ^ {n} left ({h_ {i} – y_ {i}} right)$$

(21)

$${ text {AQI}} _ { text {RMSE}} = sqrt { frac {1} {n} mathop sum limits_ {i = 1} ^ {n} left ({h_ {i} – y_ {i}} right) ^ {2}}$$

(22)

$${ text {AQI}} _ { text {an}} ; { text {index}} ; { text {of}} ; { text {agreement}} = frac {{ { text {right}} _ { text {num}}}} {n}$$

(23)

where n is the number of samples, (y_ {i} ) is the actual AQI value of the i-th day, (h_ {i} ) is the evaluation result of a model.

1. (2)

Evaluation indicators based on AQCI

The national AQCI considers the comprehensive impacts of multiple pollutants on air quality. It highlights the contribution of six pollutants. AQCI is shown in Eq. (24).

$${ text {AQCI}} = { text {sum}} left ({{ text {C}} _ {{ text {P}}} / { text {S}} _ {{ text {P}}}} right)$$

(24)

where SP is the second concentration limit of pollutant P in the Ambient Air Quality Standards (GB 3095-2012).

Taking the national AQCI as the pollution standard, the indicator AQCI_MAE and AQCI_RMSE can be calculated by Eqs. (21) and (22) in the same way.

### Analysis and comparison of evaluation methods

Take national AQI and AQCI as pollution standards. The comparisons of the DCreWeight model with the D – S, KCre-Sun, Hybrid-Rule, and FSE models are in Fig. 4. For the clarity of the image, select four months from June 1, 2014 to March 31, 2015, which can roughly represent four seasons. Spring is represented by March. Summer represented by June. Autumn is represented by September. Winter is represented by December.

According to Figs. 3 and 4, the air pollution situations were Winter> Spring> Summer> Autumn. PM2.5 and PM10 were primary pollutants in the four months. In Winter, the weight of SO2 was greater than that of O3. But in the other three months, it was smaller than that of O3. It is because that the weak light made O3 concentration decreased, and coal burning for heating made an increase of SO2 in Winter. It is because the weak light reduces the O3 concentration while coal burning for heating increases SO2 concentration in Winter. Take national AQI and AQCI as pollution standards, the evaluated air quality levels of D – S, KCre-Sun, Hybrid-Rule, and FSE methods are mostly lower than AQI. The evaluated results of the above models deviate greatly from the AQI and AQCI, while the evaluated results of the DcreWeight model are closest to the national AQI and AQCI.

To validate the superiority of the models, take AQI_MAE, AQI_RMSE, AQI_an index of agreement, AQCI_MAE, and AQCI_RMSE as evaluation indicators. The performance comparison results of the evaluation methods under the AQI and AQCI standards are shown in Fig. 5.

According to Fig. 5, the DCreWeight model has the minimum MAE, RMSE under the AQI and AQCI standards and its index of agreement is the highest, which is superior to the D – S, KCre-Sun, Hybrid-Rule, and FSE methods .

### The application in Shanghai and Beijing

The superiority of the model has been validated according to air pollutants data in Xi’an in “Analysis and comparison of evaluation methods” section. In order to better check whether the model is suitable for other urban air quality assessments, we also selected hourly air pollution data from 2014 from June 1, 2014, to May 31, 2015, in Shanghai and Beijing. Firstly, the null data were processed using the linear interpolation method. Then, we applied the DCreWeight model to the two cities and compared the air quality between Shanghai and Beijing under the national AQI and AQCI standards.

Figure 6 shows the evaluation results of the DCreWeight model in Summer, from June 1, 2014, to June 31, 2014. The left vertical axis represents the air quality evaluation level, and the right vertical axis represents the AQCI value. National AQCI represents the comprehensive pollution degree. To clearly check the accuracy of the DCreWeight model, sort the days according to AQCI.

According to Fig. 6, the AQI level fluctuates as the AQCI value decreases. This is because the AQI level depends on individual pollutants. However, with the decrease of AQCI, the evaluation results of the DCreWeight model basically decline in steps. It indicates that the evaluation of the model is in line with the actual comprehensive pollution. Compared with AQCI, the proposed model describes air quality levels more intuitively.

Next, compare the air quality between Shanghai and Beijing, as shown in Fig. 7.

According to Fig. 7, the following conclusions can be drawn. The air quality in Beijing in Summer was worse than that in Shanghai. The comprehensive air quality is good or regular basically in Shanghai in Summer. However, many days are regular, lightly polluted, and moderately polluted in Beijing in Summer.

Finally, given that pollution control is a long-term process, we finally analyze the pollution characteristics in Beijing according to the weights of pollutants, as is shown in Fig. 8. We also analyze the possible reasons for pollution to help relevant departments make strong pollution strategies based on pollution characteristics and the current air quality level.

We can conclude that PM2.5O3and PM10 are the main pollutants in Beijing according to Fig. 8. There are many causes of PM2.5 in Beijing, including vehicle exhaust, industrial emissions, and dust from construction sites and road traffic, all of which increase the concentration of PM2.5. In summer, under the strong ultraviolet light, nitrogen oxides are more easily converted into O3 by photochemical reaction, so the concentration of O3 will increase. The exposed arable land around Beijing and the surrounding sandy areas, as well as the monsoon climate in Beijing, will cause PM10 concentration to increase under the wind effect. In addition, vehicle exhaust and industrial waste gases also cause a higher concentration of NO2. Therefore, although NO2 concentration is not as high as PM2.5O3and PM10 pollution, we should pay attention to NO2 concentration control in summer to avoid a high O3 concentration. The relevant departments should control air pollution from the sources of traffic, industry park, and construction sites, and protect the surrounding environment by planting green plants.


text:
# Congratulations to Brian M. Scott!! (200K edition) [duplicate]

It's been a little more than two weeks since our last celebration, but today another math.SE user has passed the 200K barrier!

Prof. Scott's habit of providing fantastic answers to questions of all sorts is, in my opinion, remarkable. But this is surpassed by his patience in providing further explanations as users need them. His dedication to math.SE is truly something to behold!

# A hearty and well-earned congratulations!!

(On a personal note, Brian has inspired many of my own personal answering habits on the main site, though I obviously have not caught on to his prolificity. As my mathematical interests are pretty much a proper subset of his, anytime an answer of mine is accepted over one of his I take it as a small — and exceedingly rare — badge of honour. While I don't really ask questions on the main site, I have learned a lot from Brian, mathematically and otherwise. Thanks!)

• Congrats @BrianMScott! Thank you for dedicating time and effort into making Math.SE a great site! – abiessu Oct 26 '13 at 14:54
• Definitely a user to keep hold of. – Git Gud Oct 26 '13 at 16:14
• thanks sir for help us tremendously – poton Oct 27 '13 at 3:38
• @Brian you rock! :) thanks for helping us! – HipsterMathematician Oct 27 '13 at 4:14
• @Thanks a lot Brian! Your answers are very student-friendly! Congrats! – user85362 Oct 27 '13 at 4:33
• Congratulations Brian! So when are you writing a book on general topology? :) And also, thanks for making us feel better. :) – Prism Oct 27 '13 at 19:33
• The "I beat Brian's answer once" badge is a good idea! and congratulations professor @BrianM.Scott . – Matemáticos Chibchas Oct 28 '13 at 0:18
• Thank you all very much. @Prism: I wrote perhaps two-thirds of an undergraduate-level topology text back in the $1970$s but stopped when I realized that it was going to end up too long to be publishable. I still have the manuscript, and I’ve occasionally thought about retyping it in electronic form, perhaps with revisions, making it available on-line, and perhaps even completing it. But it would be a real chore, and I keep putting it off in favor of other things — like answering questions here or helping people with Old Norse questions. – Brian M. Scott Oct 28 '13 at 11:47
• @Brian: If you think it would help you finish your text, I could suspend you for a year or so....$\;$;-)$\;\;$(Can't do anything about the Old Norse, however.) – anon Oct 28 '13 at 15:44
• @Brian Congrats! Your answers tend to be canonical in away that transcends the usual time horizon of internet life. – Michael Greinecker Nov 4 '13 at 0:56
• 2 Downvotes!!!! – Vishal Gupta Nov 4 '13 at 7:09
• What happenned to sir @BrianMScott ? His last seen is stuck at 1 year back 😥 – anir Jun 24 '18 at 18:23

Simply Pleased and Extremely Delighted, It is my great pleasure to congratulate Prof. Scott. $+\infty$ amount of Thank you Sir!


text:
## anonymous 3 years ago show that if vector v is orthogonal with vector w and u, then v orthogonal with (kw + mu) for all k and m. Please help

1. TuringTest

If two vectors are orthogonal, then their dot product is zero$\vec v\perp\vec w\implies\vec v\cdot\vec w=0$now remember that constants can be factored our of the dot product$\vec v\cdot(k\vec w)=k(\vec v\cdot\vec w)$and that the dot product is distributive$\vec v\cdot(\vec u+\vec w)=\vec v\cdot\vec w+\vec v\cdot\vec u$

2. anonymous

yes. then? I got something not what i want. please, give me more

3. TuringTest

I'm not going to do the whole problem, I am just here to guide you. Take the dot product following what I have written above to evaluate$\vec v\cdot(k\vec w+m\vec u)$What do you get?

4. anonymous

v.mu +v.kw

5. TuringTest

right, now how do the constants affect the dot product?

6. anonymous

m+k

7. TuringTest

no, the scalars can be factored out of dot product multiplication$\vec v\cdot(k\vec w)=k(\vec v\cdot\vec w)=?$

8. anonymous

kv.w

9. TuringTest

and what is $$\vec v\cdot\vec w$$ ?

10. anonymous

0

11. TuringTest

yes, so then what is$\vec v\cdot(k\vec w)$?

12. anonymous

0

13. TuringTest

yes, so then what is$\vec v\cdot(k\vec w+m\vec u)$?

14. anonymous

15. TuringTest

you're welcome!

16. anonymous

Let me put everything in neat, would you please check it for me?

17. anonymous

how to put orthogonal symbol?

18. TuringTest

\perp

19. anonymous

don't understand

20. ParthKohli

$$\backslash ( \text{\perp}\backslash )$$

21. TuringTest

are you trying to use the equation editor or are you writing in LaTeX ? ...yeah best is do what @ParthKohli wrote

22. ParthKohli

 $$v \perp u$$

23. anonymous

$$\perp$$ Lol. got it. thanks

24. anonymous

i need time to practice how to use the symbol. do many time but delete all at the end, sorry , waste your time, I try and send you mine in perfect . thanks thanks a lot for help in material

25. anonymous

by the way how to put vector symbol

26. TuringTest

$\backslash(\text{\vec u}\backslash)$, and if you want to get good at writing equations try learning the language LaTeX

27. anonymous

$$v\perp u$$ ---> $$v\perp ku$$ is it right?

28. anonymous

what LateX means

29. anonymous

$$v perp w$$ ---> $$v perp mw$$

30. TuringTest

LaTeX is the language used to write the math symbols. enclose things in brackets like so: $code \.] or $$code \.) (do not put the period, I just did that to illustrate how to put the backslashes and brackets http://omega.albany.edu:8008/Symbols.html yes 31. anonymous I will. thanks for all. 32. TuringTest for what you are doing you should write\[\backslash [\text{\vec v \perp \vec w}\backslash ]$backslashes and spaces are important 33. anonymous $\vec v\perp\vec w$. bingo 34. TuringTest nice :) keep practicing LaTeX, it's handy! 35. anonymous I will. thank you very much. 36. anonymous $\vec v\perp\vec w$---> $\vec v\perp\k vec w$ 37. anonymous ah .... mistake 38. TuringTest You are welcome! another tip: if you want to know how somebody wrote something, hover your cursor over the latex you want to know the code for and press right click -> show math as -> Tex commands and you can see the source code and copy and paste it into$\backslash[\dots\backslash]\text{ or }\backslash(\dots\backslash)$ It takes some practice to avoid syntax errors. The arrow can be done with \implies, though i? I think a double arrow is better here \iff (if and only if) 39. anonymous k\(\vec v$$. bingo

40. TuringTest

very good :D$\backslash[\text{\vec v\perp \vec w\iff \vec v\cdot\vec w=0}\backslash]$gives$\vec v\perp \vec w\iff \vec v\cdot\vec w=0$

41. ParthKohli

I never answered this question. A medal is definitely not needed here. Just doing my amby thing :-)

42. anonymous

i am using other's computer, and i don't know how to copy the conversation to read later or follow when practice. so i have to copy by hand what you guide. it takes time.

43. TuringTest

@parth But I never thought to show latex by writing \backslash the way you did, and that proved very useful :) @hoa typing in latex does take some time, but of course it becomes less when you get used to it.

44. ParthKohli

@hoa It took me a lot of time too!

45. anonymous

@ParthKohli . I don't know what you mean, but to me, being helpful person deserves all. I appreciate what others guide me.

46. anonymous

@ParthKohli . the net automatically put the medal , i didn't do. or other put it there

47. ParthKohli

Turing did. :-)

48. anonymous

yes. my question is: what does the medal mean?

49. TuringTest

Giving medals for best answer raises the recipients SmartScore. Parth and I have already gotten our fair share, that's why we're level 99.

50. anonymous

why ? to me, it's just for showing the appreciateness

51. ParthKohli

Mine wasn't too fair. :-P

52. TuringTest

I don't think most high-level users care much about scores and medals. It's just a sort of acknowledgement, but the teaching is what really matters.

53. anonymous

ok. I got it. I make question "iff" need explanation, so whenever there is someone helps me, i give medal to show my appreciation no matter what it 's the best answer or not . now, i know what it means, thanks

54. TuringTest

No, you should really only give a medal if you think the user really *deserves* it. Feel free not to give one for those who only try to give answers without teaching, of which there are many.

55. anonymous

they know who they are no matter how many medals they have, right? we are self- respect, they are the same. However, if you say so, I will be careful in giving medal. And if you say so, I feel my medal does not deserve at all. All my answer is for very very easy problems. shame on me

56. TuringTest

Don't feel bad, just help people in a way you can be proud of. Try to teach others, regardless of the level of the question and you will do great here!


text:
In OpenCog, the ContextLink is used to alter the interpretation of an Atom, giving it an alternate TruthValue within a sepcific context.

A truth value gives each Atom a valuation or interpretation, thus all Atoms always carry a default interpretation along with them. This prevents atoms in a single AtomSpace from having more than one default interpretation. The ContextLink can be used to assign additional interpretations to an Atom. That is, the ContextLink represents the contextualised truth value between an atom and another context node (which may be anything, although is usually a ConceptNode).

For example, we usually think of the sky as being blue, however, when the context is the Moon, this isn't the case:

ContextLink <0.0, 0.2>
ConceptNode "Moon"
PredicateNode isBlue
ConceptNode "sky"

Note that the EvaluationLink for isBlue has a higher strength than the ContextLink. The TV for the EvaluationLink is the default for Earth, where the sky is usually blue when its not cloudy. The TV on the ContextLink is appropriate for the Moon, where the sky is not blue.

Contexts provide a mechanism for OpenCog to store distinct multiple TVs for an atom (in the above example, for an EvaluationLink).

See the page claims and contexts to see an example of how a ContextLink is used to express claims such as He said that..., I believe that..., etc.

From the point of view of probability theory or of measure theory, a context is an element of a sigma algebra. That is, a context is a subset of the 'universe'; and one may take the intersection of two contexts (using the AndLink), or their union (with the OrLink). The rules for combining truth values on a context link are analogous to the requirement that a probability be sigma-additive. Similarly, just as the axioms of probability theory tell one how to work with probabilities, so contexts inform how to work with truth values, when considering a subset of the total universe of possibilities.

# Definition

ContextLink <TV>
C
R A B

is equivalent to

R <TV>
(A AND C)
(B AND C)

Here, C is an atom defining the context, and R is some relationship between A and B. The case where R is a node can be handled as follows. Since

A <TV>

is equivalent to

ExtensionalInheritanceLink <TV>
Universe
A

Therefore

ContextLink <TV>
C
A

is equivalent to

ExtensionalInheritanceLink <TV>
C
A AND C

When R is an EvaluationLink, then it appears that the PredicateNode does not need to be "Andified". Under some natural assumptions, only the arguments would be andified, such as follows:

ContextLink <0.0, 0.2>
Moon
PredicateNode isBlue
sky

The above would be equivalent to

EvaluationLink <0.0, 0.2>
PredicateNode isBlue
(sky AND Moon)

However, for the general case of a relation R (that is not an EvaluationLink), there is no apparent reason for skipping andification like this.

# Properties

This section discusses and reviews some basic properties of the ContextLink. If you are already familiar with the measure-theoretic formulation of probability theory, then contexts can be understood in a straight-forward way: they are essentially the members of a sigma algebra. Thus, the intersection (AndLink) and union (OrLink) of two contexts govern how truth values combine. This is expanded upon in the next section.

Note also that this conception of a context as a set-like thing implies that it is extensional, and not intentional: contexts are not defined in terms of their properties, but in terms of their size and membership. This is the same reason that SubsetLink uses extensional inheritance in it's definition. This is underscored below by making reference to the ExtensionalAndLink, rather than the plain AndLink.

ContextLink <TV1>
C
D
R<TV3> A B

is equivalent to

ContextLink <TV2>
D
R<TV3> A B

ContextLink <TV1>
(D ANDExt C)
R<TV3> A B

which is equivalent to

R<TV3> A B

R<TV2>
(D ANDExt A)
(D ANDExt B)

R<TV1>
(C ANDExt D ANDExt A)
(C ANDExt D ANDExt B)

These semantics, however, only hold when R is a form of inheritance/implication or similarity/equivalence; they don't hold for relationships R generally (see [1]).

However

ContextLink <TV1>
C
D
R<TV3> A B

is not equivalent to

ContextLink <TV2>
D
C
R<TV3> A B

That is because in the first case TV1 is the contextualized TV with context (C AND D ) while in the second case TV1 is the contextualized TV with context C only.

On the other hand

ContextLink <TV1>
C
D
R<TV3> A B

is equivalent to

ContextLink <TV1>
D
C
R<TV3> A B

where TV2' is a new contextualized TV with context C alone.

# Probabilistic interpretation

This section briefly reviews the relationship between contexts sigma algebras. Recall that a sigma algebra is a pair ${\displaystyle (\Omega ,\Sigma )}$ where ${\displaystyle \Omega }$ is a universe, and ${\displaystyle \Sigma }$ is a set of subsets of ${\displaystyle \Omega }$. The sigma algebra is closed under intersection and union: the intersection and union of any two elements in ${\displaystyle \Sigma }$ is also in ${\displaystyle \Sigma }$. A context, in opencog, can be imagined to be a member of a sigma-algebra, with AndLink defining the intersection, and OrLink defining the union.

More precisely, this is why contexts can only be combined with extensional and-links, and not intentional ones; this is the same reason that the SubsetLink is extensional, not intentional. Roughly speaking, it is because we are interested in the size of a set, and its members, rather than it's properties. When taking the intersection or union of two sets, we intersect or union based on membership; we do not intersect or union the properties of the sets.

The measure-theoretic formulation of probability theory is a triple ${\displaystyle (\Omega ,\Sigma ,\mu )}$; the third element ${\displaystyle \mu }$ being a (real-valued) function that assigns a probability to each member of ${\displaystyle \Sigma }$. That is, the probability P(A) of an event A in ${\displaystyle \Sigma }$ is just ${\displaystyle \mu (A)}$; that is, ${\displaystyle \mu (A)=P(A)}$ by definition; its just a different notation for the same thing. See probability space for details. Probabilities must obey certain rules when different events are combined: these are axioms that require that the function ${\displaystyle \mu }$ be sigma-additive. Analogously, truth values must be combined in such a way as to respect sigma-additivity.

# Quantification

Contexts are used for quantification. Thus,

Implication
Inheritance (Ben AND “possible universe C”) ugly
Inheritance (Ben AND “possible universe C”) beef_eater

is equivalent by definition to

ContextLink C
Implication
Inheritance Ben ugly
Inheritance Ben beef_eater

From this we conclude that

Implication <TV>
Inheritance Ben ugly
Inheritance Ben beef_eater

is implicitly taken as

AverageQuantifier $C <TV> AND Inheritance$C possible_world
Implication
Inheritance (Ben AND $C) ugly Inheritance (Ben AND$C) beef_eater

That is, in the probabilistic interpretation, the truth of an atom in the universe is the probability-weighted average of its truth in all possible contexts (denoted with the AverageLink).

# Context Rules

There are 3 context rules

# Implementation

At this time, there is no optimization for the handling of ConextLinks in the atomspace. However, due to thier fundamental importance, perhaps there should be one.

From the memory-management point of view, it might be best to insert all ContextLinks of a given kind into a sub-Atomspace; see multiple atom spaces for details.

## Implementation conception

To guide the implementation, think of the problem as follows. In the abstract: Atoms, by them selves, do not have a TV; it is impossible to assign a TV to any atom at all, when that atom is outside of a context. That is, atoms never have TV's; only context links do.

However, if we allow a default context, then one possible implementation would be exactly what we have today: every atom points to its TV for the default context. Viz, what we have today is conceptually identical to the case where only ContextLinks have TV's, and there is a default context.

Next, every PLN rule should be specified relative to a context. I suspect the current PLN design does not do this, and does not even anticipate this. I suspect that the current PLN design assumes only a default context for everything. I suspect that the current PLN does not even handle ContextLinks....

Next, if all computation is relative to a context, then we need some mechanism to rapidly obtain the TV associated with the (atom, context) pair. The old version-handle code did this (although the implementation caused severe technical issues). The time is coming that we once-again need a rapid lookup of the TV associated with a (atom, context) pair.

As Ben points out: actually having and using ContextLinks is kind-of klunky, so perhaps we should never, ever allow actual ContextLinks, and instead force PLN to just use TV's obtained from (atom, context) pairs, and have some super-easy-to-use default context (e.g. what we defacto have today).

So, in this way, we are lead full circle back to versioned TV's. It may be better if the AtomSpace API was written to always use (atom, context) pairs. That is, instead of fetching the TV by making the C++ call atom->getTV(); it should instead make a wrapper like this:

TVPtr getTV (AtomPtr a, ContextPtr c) {
if (c==NULL) return a->getTV();
else return do_something_else_to_get_tv(a,c);
}