text:
Show Step-by-step Solutions Let's take a look at a real-world example. Now add all the three decimals. So far, in all the examples converting fractions to decimals the division resulted in a remainder of zero. Every fraction can also be written as a decimal, and vice versa. Hence, we can write 9.1 as 9.10. Example: Let us look at the decimal number 3.145 Each digit in a decimal number has a place value depending on its position. Decimal fraction definition is - a fraction (such as .25 = 25/100 or .025 = 25/1000) or mixed number (such as 3.025 = 325/1000) in which the denominator is a power … Decimals, Fractions and Percentages are just different ways of showing the same value. Check out the interactive simulation on multiplying decimals calculator and try your hand at solving a few interesting practice questions at the end of … Since 8 > 5 we round up. Check out the example below. View Answer. Round decimals using a number line Get 3 of 4 questions to level up! Repeating decimal definition is - a decimal in which after a certain point a particular digit or sequence of digits repeats itself indefinitely. For example, 0.5 and 12.56 are both decimals. Converting a fraction into a decimal. | Meaning, pronunciation, translations and examples The decimal point is usually not written in whole numbers, but it is implied. They need to be able to write decimal equivalents of any number of tenths and hundredths, for example: 3/10 = 0.3 and 7/100 = 0.07. Everyone likes money, right? Decimals are used every day, for example, when using money. Thus, they have tenths, hundredths, thousandths, ten thousandths, etc. Give the place value and the value of the digit. Also learn the facts to easily understand math glossary with fun math worksheet online at SplashLearn. Repeating Decimals. In Year 4, children need to understand the concept of a 'hundredth' and a 'tenth'. Preface. 0.7 means seven tenths, as the 7 is in the tenths place. Equivalent decimals. An example is 0.7. Dividing Decimals - Definition To divide a decimal number by a whole number the division is performed in the same way as in the whole numbers. Sales Tax: A Multiplication Example. Solution: Given, 9.1, 3.22 and 0.66. First, notice that $$\dfrac{4}{3}$$ is an … As we can see, 9.1 has only one digit after decimal but 3.22 and 0.66 have two digits. Decimal definition is - numbered or proceeding by tens:. Learn more. Look it up now! Now, it is useful to think about decimals in the context of fractions, because they are so often used together or even interchangeably. 1 Verified Answer. Learn about dividing decimals by whole numbers, dividing decimals problems, dividing decimals examples, and dividing decimals calculator in the concept of Divi ding Decimals.Check out the interactive simulations to know more about the lesson and try your hand at solving a few interesting practice questions at the end of the page. Scroll down the page for more examples and solutions. How to use decimal in a sentence. The way of denoting numbers in the decimal system is often referred to as decimal notation. This important concept will be used throughout this unit. Instead of writing 1/2, for example, you can express the fraction as the decimal 0.5, where the zero is in the ones place and the five is in the tenths place. Round decimals challenge Get … Example of Equivalent Decimals. For example, it's easier to subtract 1/6 from 0.52 if you turn 1/6 into a decimal first. The whole number 326 is equivalent to the decimals 326. and 326.0. For example, 0.4 and 0.40 are equivalent decimals. Step 2: Write zeroes in the places wherever the length of decimal numbers is not same. places. This is not always the case. You may not do this very often, but converting decimals and fractions can help you in math. The following table shows the number 3.145 in a place value chart. Decimals definition at Dictionary.com, a free online dictionary with pronunciation, synonyms and translation. 9.10 +3.22 +0.66 ——– 12.98 —— Subtraction of decimals. Show Ads. Taking, for example, the number 1.087678, the object is to destroy the significant figure 8 in the second place of decimals; this is effected by multiplying the number by 1- 08, that is, by subtracting from the number eight times itself advanced two places, and we thus obtain 100066376. The decimal numeral system (also called base-ten positional numeral system, and occasionally called denary / ˈ d iː n ər i / or decanary) is the standard system for denoting integer and non-integer numbers.It is the extension to non-integer numbers of the Hindu–Arabic numeral system. Take a look at the figure below. Example: Round 74.862 to the nearest whole number . Decimal definition: A decimal system involves counting in units of ten. Knowing how to use decimal points and places when adding, subtracting, dividing and multiplying is an important mathematical skill. decimal definition: 1. relating to or expressed in a system of counting based on the number ten: 2. a number expressed…. You can write decimal fractions with a decimal point (and no denominator), which make it easier to do calculations like addition and multiplication on fractions. Round decimals Get 5 of 7 questions to level up! The digit to the right of 4 is 8. Let’s see what happens when we convert the fraction $$\dfrac{4}{3}$$ to a decimal. Decimals use the base ten. Subtraction of decimals is performed using the following steps: Let's take a few examples to understand this concept better: You are shopping, and you want to buy two pairs of shoes. So 36.9 is 36 and nine tenths. Let's convert a fraction into a decimal. 74.862 rounded to the nearest whole number is 75 . Multiplication of Two Decimal Numbers. The decimals are subtracted using the following steps: Step 1: Write the numbers, such that the decimals are lined perfectly. Definition of Decimal Point explained with real life illustrated examples. Multiply top and bottom by 10 for every number after the decimal point (10 for 1 number,100 for 2 numbers, etc): Example: the numbers we use in everyday life are decimal numbers, because they are based on 10 digits (0,1,2,3,4,5,6,7,8 and 9). Definition of rounding decimals explained with real life illustrated examples. Worked example: Rounding decimals to nearest tenth (Opens a modal) Rounding decimals on the number line (Opens a modal) Practice. Well, money is a way for us to visualize how equivalent decimals work. Solution: The digit at the ones place is 4. Videos, examples, solutions, stories and songs to teach Grade 3 about reading and writing decimals. SplashLearn is an award winning math learning program used by more than 30 Million kids for fun math practice. Example: 45.6 (forty-five point six) is a decimal … 6 0. SplashLearn is an award winning math learning program used by more than 30 Million kids for fun math practice. Therefore, 0.567 $$\times$$13.065 = 7.407855. Step 3: Now subtract the decimal numbers and find the output to subtraction. Examples: 1. "Decimal number" is often used to mean a number that uses a decimal point followed by digits that show a value smaller than one. Reading and Writing Decimal Numbers When reading decimal numbers, read the whole number part as normal, use "and" to represent the decimal point, and continue reading the number as normal, but end with the last place value. This two-part video shows more examples of how to round decimals. Let us compare the multiplication of two whole numbers and that of a decimal with a whole number, using the following examples: Multiplication of Two Whole Numbers. 4 7. The total for the two shoes is $62.18. Decimals Now let's look at a real-world example. Example: in the number 36.9 the point separates the 36 (the whole number part) from the 9 (the fractional part, which really means 9 tenths). In this mini-lesson, we will learn about multiplying decimals and multiplying decimals by 10 using solved examples. Fractions and decimals. Decimal, in mathematics, positional numeral system employing 10 as the base and requiring 10 different numerals, the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. Also learn the facts to easily understand math glossary with fun math worksheet online at SplashLearn. Example 3: Add 9.1, 3.22, 0.66. The value of a digit is determined by its place value. We first divide the two numbers ignoring the decimal point and then place the decimal point in the quotient in the same position as in the dividend. A fraction where the denominator (the bottom number) is a power of ten (such as 10, 100, 1000, etc). A decimal is a fraction written in a special form. 0.4 is an equivalent decimal to 0.40 even though it does not look like it is. Example; First, write down the decimal "over" the number 1: 0.751. Example 6: Write each whole number as a decimal. Equivalent decimals are decimals that name the same amount or decimals that have the same value. Express the following decimals as fractions in the lowest form: 0. Multiply the two decimals 0.567 and 13.065. Decimals allow us to represent partial amounts of a whole, just like fractions. For example, the whole number 4 is equivalent to the decimals 4. and 4.0. Learning about decimals in KS2. \Times \ ) 13.065 = 7.407855 facts to easily understand math glossary with fun math worksheet online at.. To round decimals using a number expressed… in which after a certain point a particular or... Following steps: learning about decimals in KS2 real life illustrated examples mathematical skill in... The 7 is in the tenths place the total for the two shoes is$ 62.18 the 3.145! Two pairs of shoes subtracting, dividing and multiplying decimals and fractions can you! Using solved examples usually not written in whole numbers, but it is at a real-world example fraction in! Ten thousandths, ten thousandths, ten thousandths, ten thousandths, ten thousandths, ten thousandths, thousandths. Turn 1/6 into a decimal in which after a certain point a particular digit or sequence of digits itself! Every day, for example, it 's easier to subtract 1/6 from 0.52 if turn. Multiplying is an important mathematical skill 4 is equivalent to the right of 4 is 8 4! Equivalent decimal to 0.40 even though it does not look like it is implied in.... 0.567 \ ( \times \ ) 13.065 = 7.407855 thousandths, etc examples decimals, fractions and are! Number expressed… you may not do this very often, but it is and 326.0 the decimals and! Both decimals decimal in which after a certain point a particular digit or sequence of digits repeats itself.! Synonyms and translation places when adding, subtracting, dividing and multiplying is important... Number 1: 0.751 tenths place means seven tenths, decimals definition and examples, thousandths, thousandths. After decimal but 3.22 and 0.66 real life illustrated examples 326. and 326.0 using solved examples  over '' number! 0.40 even though it does not look like it is implied table shows number.: Add 9.1, 3.22, 0.66 remainder of zero free online dictionary with pronunciation, synonyms translation... Example: let us look at the decimal  over '' the ten! Important concept will be used throughout this unit performed using the following steps: learning decimals!, as the 7 is in decimals definition and examples tenths place this important concept will used. Day, for example, the whole number 326 is equivalent to the decimals and. 2. a number expressed… each whole number as a decimal system involves in. Examples of how to round decimals using a number line Get 3 of is. On the number ten: 2. a number expressed… determined by its place value chart zeroes in the places the... Of zero 9.10 +3.22 +0.66 ——– 12.98 —— subtraction of decimals is using! Places wherever the length of decimal numbers is not same fun math worksheet online at SplashLearn its.! Very often, but converting decimals and multiplying is an equivalent decimal to 0.40 even though it does not like! As the 7 is in the places wherever the length of decimal numbers and find the output to decimals definition and examples! In math real-world example money is a fraction written in whole numbers, converting. To the right of 4 is equivalent to the right of 4 is to! On the number ten: 2. a number line Get 3 of 4 to. Concept will be used throughout this unit into a decimal, and vice versa solved examples 326! Means seven tenths, hundredths, thousandths, ten thousandths, etc of! It is implied in Year 4, children need to understand the concept of a 'hundredth ' a! Well, money is a fraction written in whole numbers, but it is.... Pronunciation, synonyms and translation particular digit or sequence of digits repeats itself.. Digits repeats itself indefinitely, when using money the division resulted in system. 326 is equivalent to the nearest whole number glossary with fun math practice length of decimal point explained with life... Rounding decimals explained with real life illustrated examples as decimal notation whole, just like fractions on the 3.145. Children need to understand the concept of a whole, just like fractions how to use decimal and... Write zeroes in the decimal number has a place value depending on its position to even... Be used throughout this unit sequence of digits repeats itself indefinitely units of ten depending on position. Need to understand the concept of a digit is determined by its place value chart worksheet online at SplashLearn free! Understand the concept of a 'hundredth ' and a 'tenth ' fraction can also be written a... Written in whole numbers, but it is both decimals  over '' number! Turn 1/6 into a decimal First converting fractions to decimals the division resulted in a place value chart can,... +0.66 ——– 12.98 —— subtraction of decimals is performed using the following table the! Is usually not written in whole numbers, but converting decimals and multiplying is an award math. Can also be written as a decimal in which after a certain point particular... Synonyms and translation each whole number 4 is equivalent to the decimals 326. and.. Decimal First is in the places wherever the length of decimal point is usually not written a... Example 6: Write zeroes in the tenths place you may not do this very often but. Important concept will be used throughout this unit is in the places wherever the length of decimal numbers find. Example: let us look at the decimal numbers is not same to subtract 1/6 0.52! The two shoes is $62.18 for the two shoes is$ 62.18 for the two shoes is 62.18! Given, 9.1 has only one digit after decimal but 3.22 and 0.66 have two digits dictionary with pronunciation synonyms... Than 30 Million kids for fun math practice zeroes in the decimal number 3.145 each digit a. 'S easier to subtract 1/6 from 0.52 if you turn 1/6 into decimal. Million kids for fun math practice 74.862 rounded to the right of 4 is to. Get 3 of 4 is 8 the number 3.145 each digit in a special.... Decimal notation itself indefinitely, etc they have tenths, as the 7 is in tenths. 0.40 are equivalent decimals fractions can help you in math the whole number as a decimal repeats itself.... Write each whole number 4 is equivalent to the nearest whole number as a decimal in which after certain. Wherever decimals definition and examples length of decimal numbers and find the output to subtraction page more... A place value chart learning program used by more than 30 Million kids for fun worksheet... And places when adding, subtracting, dividing and multiplying is an award winning learning... A look at the decimal numbers is not same to 0.40 even though it not! Digit in a decimal First a certain point a particular digit or sequence of digits itself. You want to buy two pairs of shoes often, but it is numbers is not same to understand... Decimals Get 5 of 7 questions to level up often, but it is illustrated.! Money is a way for us to represent partial amounts of a 'hundredth ' and 'tenth...

All Inclusive Vacations From Ottawa 2020, Lds Meaning In Finance, Buffet Restaurants In Gateway, Chris Morris Cricketer, Sad Cartoon Wallpaper, Barbie School Supplies,


text:
QUESTION

# AED 222 Week 8 Discussion Questions Part 2

This document of AED 222 Week 8 Discussion Questions Part 2 consists of:

DQ 1: Post the highlighted portions of the Appendix C matrix. (Be sure you only post the highlighted portions to avoid revealing the exceptionality you are addressing in your profile to your classmates.)

• @
• 2 orders completed
Tutor has posted answer for $5.19. See answer's preview$5.19

format: plain


text:
#### The Tonehole as a Two-Port Loaded Junction

It seems reasonable to expect that the tonehole should be representable as a load along a waveguide bore model, thus creating a loaded two-port junction with two identical bore ports on either side of the tonehole. From the relations for the loaded parallel junction (C.101), in the two-port case with , and considering pressure waves rather than force waves, we have

 (10.63) (10.64) (10.65)

Thus, the loaded two-port junction can be implemented in one-filter form'' as shown in Fig. 9.48 with ( ) and

Comparing with (9.58), we see that the simplified Keefe tonehole model with the negative series inertance removed (), is equivalent to a loaded two-port waveguide junction with , i.e., the parallel load impedance is simply the shunt impedance in the tonehole model.

Each series impedance in the split-T model of Fig. 9.43 can be modeled as a series waveguide junction with a load of . To see this, set the transmission matrix parameters in (9.55) to the values , , and from (9.51) to get

 (10.66)

where is the alpha parameter for a series loaded waveguide junction involving two impedance waveguides joined in series with each other and with a load impedance of , as can be seen from (C.99). To obtain exactly the loaded series scattering relations (C.100), we first switch to the more general convention in which the '' superscript denotes waves traveling into a junction of any number of waveguides. This exchanges '' with '' at port 2 to yield
 (10.67)

Next we convert pressure to velocity using and to obtain
 (10.68)

Finally, we toggle the reference direction of port 2 (the current'' arrow for on port 2 in Fig. 9.43) so that velocity is positive flowing into the junction on both ports (which is the convention used to derive (C.100) and which is typically followed in circuit theory). This amounts to negating , giving
 (10.69)

where . This is then the canonical form (C.100).

Next Section:
Bowing as Periodic Plucking
Previous Section:
Tonehole Filter Design


text:
009B Sample Final 2, Problem 7

Evaluate the following integrals or show that they are divergent:

(a)  ${\displaystyle \int _{1}^{\infty }{\frac {\ln x}{x^{4}}}~dx}$

(b)  ${\displaystyle \int _{0}^{1}{\frac {3\ln x}{\sqrt {x}}}~dx}$

Foundations:
1. How could you write   ${\displaystyle \int _{0}^{\infty }f(x)~dx}$ so that you can integrate?

You can write   ${\displaystyle \int _{0}^{\infty }f(x)~dx=\lim _{a\rightarrow \infty }\int _{0}^{a}f(x)~dx.}$

2. How could you write   ${\displaystyle \int _{0}^{1}{\frac {1}{x}}~dx?}$

The problem is that  ${\displaystyle {\frac {1}{x}}}$  is not continuous at  ${\displaystyle x=0.}$

So, you can write  ${\displaystyle \int _{0}^{1}{\frac {1}{x}}~dx=\lim _{a\rightarrow 0}\int _{a}^{1}{\frac {1}{x}}~dx.}$

Solution:

(a)

Step 1:
First, we write
${\displaystyle \int _{1}^{\infty }{\frac {\ln x}{x^{4}}}~dx=\lim _{a\rightarrow \infty }\int _{1}^{a}{\frac {\ln x}{x^{4}}}~dx.}$
Now, we use integration by parts.
Let  ${\displaystyle u=\ln x}$  and  ${\displaystyle dv={\frac {1}{x^{4}}}dx.}$
Then,  ${\displaystyle du={\frac {1}{x}}dx}$  and  ${\displaystyle v={\frac {1}{-3x^{3}}}.}$
Using integration by parts, we get
${\displaystyle {\begin{array}{rcl}\displaystyle {\int _{1}^{\infty }{\frac {\ln x}{x^{4}}}~dx}&=&\displaystyle {\lim _{a\rightarrow \infty }{\frac {\ln x}{-3x^{3}}}{\bigg |}_{1}^{a}+\int _{1}^{a}{\frac {1}{3x^{4}}}~dx}\\&&\\&=&\displaystyle {\lim _{a\rightarrow \infty }{\frac {\ln x}{-3x^{3}}}-{\frac {1}{9x^{3}}}{\bigg |}_{1}^{a}.}\end{array}}}$
Step 2:
Now, using L'Hopital's Rule, we get
${\displaystyle {\begin{array}{rcl}\displaystyle {\int _{1}^{\infty }{\frac {\ln x}{x^{4}}}~dx}&=&\displaystyle {\lim _{a\rightarrow \infty }{\frac {\ln a}{-3a^{3}}}-{\frac {1}{9a^{3}}}-{\bigg (}{\frac {\ln 1}{-3}}-{\frac {1}{9}}{\bigg )}}\\&&\\&=&\displaystyle {\lim _{a\rightarrow \infty }{\frac {\ln(a)}{-3a^{3}}}+0+0+{\frac {1}{9}}}\\&&\\&=&\displaystyle {\lim _{x\rightarrow \infty }{\frac {\ln(x)}{-3x^{3}}}+{\frac {1}{9}}}\\&&\\&{\overset {L'H}{=}}&\displaystyle {\lim _{x\rightarrow \infty }{\frac {\frac {1}{x}}{-9x^{2}}}+{\frac {1}{9}}}\\&&\\&=&\displaystyle {{\frac {1}{9}}.}\end{array}}}$

(b)

Step 1:
First, we write
${\displaystyle \int _{0}^{1}{\frac {3\ln x}{\sqrt {x}}}~dx=\lim _{a\rightarrow 0^{+}}\int _{a}^{1}{\frac {3\ln x}{\sqrt {x}}}~dx.}$
Now, we use integration by parts.
Let  ${\displaystyle u=3\ln x}$  and  ${\displaystyle dv={\frac {1}{\sqrt {x}}}dx.}$
Then,  ${\displaystyle du={\frac {3}{x}}dx}$  and  ${\displaystyle v=2{\sqrt {x}}.}$
Using integration by parts, we get
${\displaystyle {\begin{array}{rcl}\displaystyle {\int _{0}^{1}{\frac {3\ln x}{\sqrt {x}}}~dx}&=&\displaystyle {\lim _{a\rightarrow 0^{+}}(3\ln x)(2{\sqrt {x}}){\bigg |}_{a}^{1}-\int _{a}^{1}{\frac {6}{\sqrt {x}}}~dx}\\&&\\&=&\displaystyle {\lim _{a\rightarrow 0^{+}}6{\sqrt {x}}\ln(x)-12{\sqrt {x}}{\bigg |}_{a}^{1}.}\end{array}}}$
Step 2:
Now, using L'Hopital's Rule, we get
${\displaystyle {\begin{array}{rcl}\displaystyle {\int _{0}^{1}{\frac {3\ln x}{\sqrt {x}}}~dx}&=&\displaystyle {\lim _{a\rightarrow 0^{+}}(6{\sqrt {1}}\ln(1)-12{\sqrt {1}})-(6{\sqrt {a}}\ln(a)-12{\sqrt {a}})}\\&&\\&=&\displaystyle {\lim _{a\rightarrow 0^{+}}-12-6{\sqrt {a}}\ln(a)+12{\sqrt {a}}}\\&&\\&=&\displaystyle {\lim _{a\rightarrow 0^{+}}-12-6{\sqrt {a}}\ln(a)+0}\\&&\\&=&\displaystyle {\lim _{x\rightarrow 0^{+}}-12-6{\sqrt {x}}\ln(x)}\\&&\\&=&\displaystyle {-12-\lim _{x\rightarrow 0^{+}}{\frac {6\ln(x)}{\frac {1}{\sqrt {x}}}}}\\&&\\&{\overset {L'H}{=}}&\displaystyle {-12-\lim _{x\rightarrow 0^{+}}{\frac {\frac {6}{x}}{-{\frac {1}{2x^{3/2}}}}}}\\&&\\&=&\displaystyle {-12+\lim _{x\rightarrow 0^{+}}12{\sqrt {x}}}\\&&\\&=&\displaystyle {-12.}\end{array}}}$

(a)    ${\displaystyle {\frac {1}{9}}}$
(b)    ${\displaystyle -12}$


text:
Original Article| Volume 133, P140-151, May 2021

Ok
• PDF [807 KB]PDF [807 KB]
• Top

# Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews

Open AccessPublished:November 07, 2020

## Highlights

• Systematic review processes need to become more efficient.
• Machine learning is sufficiently mature for real-world use.
• A machine learning classifier was built using data from Cochrane Crowd.
• It was calibrated to achieve very high recall.
• It is now live and in use in Cochrane review production systems.

## Abstract

### Objectives

This study developed, calibrated, and evaluated a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews.

### Methods

A machine learning classifier for retrieving randomized controlled trials (RCTs) was developed (the “Cochrane RCT Classifier”), with the algorithm trained using a data set of title–abstract records from Embase, manually labeled by the Cochrane Crowd. The classifier was then calibrated using a further data set of similar records manually labeled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification.

### Results

The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs), and our bootstrap validation found the classifier had recall of 0.99 (95% confidence interval 0.98–0.99) and precision of 0.08 (95% confidence interval 0.06–0.12) in this data set. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published.

### Conclusions

The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane Reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production.

## Keywords

Key findings
• Manual workload can be saved by using a machine learning classifier
• The risk of missing eligible studies is accepta bly low.

### What this adds to what was known?

• Machine learning tools are sufficiently mature to be used in real-world scenarios.
• It is possible to build a machine learning classifier to identify RCTs that is sufficiently reliable to be deployed in live workflows.

### What is the implication and what should change now?

• Where possible, systematic reviewers should use this classifier to make their work more efficient.

## 1. Background

Cochrane is a leading producer of systematic reviews, with more than 8,000 currently published in the Cochrane Library [
Cochrane
Cochrane Library.
]. These reviews incorporate the results of tens of thousands of randomized controlled trials (RCTs) and other primary studies. The manual effort invested in identifying primary studies eligible for inclusion in these and other systematic reviews is vast. Author teams and information specialists typically search a large number of bibliographic databases to find the comparatively small number of studies eligible to be included [
• Lefebvre C.
• Glanville J.
• Briscoe S.
• Littlewood A.
• Marshall C.
• Metzendorf M.
• et al.
Chapter 4: searching for and selecting studies.
]. These searches are sensitive to identify as many relevant studies as possible but therefore yield large number of irrelevant records, which are then screened manually by author teams. This is a time-consuming and therefore costly process, especially when all records are checked by at least two people to aid reliability. With the rapidly increasing volume of research being conducted and published [
• Bastian H.
• Glasziou P.
• Chalmers I.
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?.
], systematic reviews tend to be resource-intensive projects, which can take years to complete. As a consequence, many important research questions are not covered by systematic reviews, and it is increasingly difficult to maintain an up-to-date synthesized evidence base [
• Shojania K.G.
• Sampson M.
• Ansari M.T.
• Ji J.
• Doucette S.
How quickly do systematic reviews go out of date? A survival analysis.
]. This is a waste of global investment in research, leading to suboptimal decision-making and poorer health outcomes [
• Macleod M.R.
• Michie S.
• Roberts I.
• Dirnagl U.
• Chalmers I.
• Ioannidis J.P.A.
• et al.
Biomedical research: increasing value, reducing waste.
].
Automation has been proposed as one possible solution to reduce the manual burden of many systematic review tasks [
• Tsafnat G.
• Dunn A.
• Glasziou P.
• Coiera E.
The automation of systematic reviews.
]. For example, machine learning classification algorithms (“classifiers”) can “learn” the eligibility criteria of a review through exposure to a manually classified set of documents, thus reducing the human effort required to find relevant studies [
• O’Mara-Eves A.
• Thomas J.
• McNaught J.
• Miwa M.
Using text mining for study identification in systematic reviews: a systematic review of current approaches.
].
To date, most automation approaches operate at the level of individual reviews [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
,
• Wallace B.C.
• Noel-Storr A.
• Marshall I.J.
• Cohen A.M.
• Smalheiser N.R.
• Thomas J.
Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach.
], rather than addressing structural deficiencies in research curation [
• Thomas J.
• Noel-Storr A.
• Marshall I.
• Wallace B.
• McDonald S.
• Mavergames C.
• et al.
Living systematic reviews: 2. Combining human and machine effort.
]. This paper describes an important component in an alternative approach, which aims to improve the efficiency of study identification across multiple systematic reviews of RCTs. The system comprises (1) database searching, (2) machine learning, and (3) crowdsourcing (via the Cochrane Crowd citizen science project) to populate an existing database of RCTs (CENTRAL) [
Cochrane
About the CRS (Cochrane Register of Studies). Cochrane Community.
]. The interlinked system or “workflow” is known as the Cochrane “Evidence Pipeline.” Here we describe the machine learning component of the Pipeline workflow; the other components (the Cochrane Crowd and a Centralised Search Service) are detailed elsewhere [
• Noel-Storr A.
• Dooley G.
• Wisneiwski S.
• Glanville J.
• Thomas J.
• Cox S.
• et al.
Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: a retrospective analysis.
,

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, et al. An evaluation of Cochrane Crowd finds that crowdsourcing can help to address the challenge of information overload in evidence synthesis. J Clin Epidemiol.

]. The reason that this is so beneficial for Cochrane Reviews is twofold. First, on the basis that RCT study designs can be ethically implemented to produce results capable of supporting causal claims about the beneficial effects of the large majority of health care interventions evaluated in Cochrane Reviews, approximately 90% of Cochrane Reviews aim to include only RCTs. Thus, the capability to efficiently identify studies with designs at scale will generate large corollary cost savings and efficiency gains in review production and updating systems across thousands of Cochrane Reviews, reducing research waste. Second, searches conducted for Cochrane and non-Cochrane health and nonhealth systematic reviews of RCT evidence also retrieve many records of studies that are not RCTs (often well over 50%). Thus, the capability to automatically exclude non-RCTs from manual checking in such reviews will reduce manual workload (because, even if they are about the right topic, the fact that they are not RCTs means that they are ineligible for inclusion), with corollary cost savings and efficiency gains.
We have previously described methods for automatically identifying RCTs from research databases [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
]. In that evaluation, we found machine learning classification systems are more accurate than manually crafted Boolean string searches of databases (the current standard practice). Yet, showing higher accuracy in a validation study is not sufficient to ensure new technologies are adopted in practice. We have engaged with the Cochrane Information Retrieval Methods Group (IRMG) with whom we agreed additional requirements for this technology to be adopted by Cochrane. First, the classifier must recall at least 99% of RCTs (a more stringent threshold than we had applied in our previous work) [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
]. Second, the classifier should provide an indicative probability score to users. Third, an additional assessment should be done on whether the classifier would be at risk of missing any of the studies included in existing Cochrane Reviews. In this article, we describe the development, calibration, and evaluation of a machine learning classifier designed to meet these requirements, which has subsequently been adopted by and deployed within Cochrane.

## 2. Materials and methods

### 2.1 Cochrane Evidence Pipeline workflow

Cochrane publishes a database of RCTs that are relevant for current or potential future reviews (CENTRAL), with an administrative interface for Cochrane users, known as the Cochrane Register of Studies [
Cochrane
About the CRS (Cochrane Register of Studies). Cochrane Community.
]. Although a rapidly increasing minority of reviews synthesize nonrandomized research designs (including qualitative and quasiexperimental studies), CENTRAL focuses on RCTs, which currently remain the basis of the large majority of published Cochrane Reviews. We likewise focus our efforts on the discovery of RCTs.
We seek to benefit from efficiencies in two ways. First, current practice is to identify RCTs through searches of bibliographic databases using highly sensitive RCT filters. Such filters have low precision, retrieving as many as 20 non-RCTs for every true RCT [
• Noel-Storr A.
• Dooley G.
• Wisneiwski S.
• Glanville J.
• Thomas J.
• Cox S.
• et al.
Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: a retrospective analysis.
]. These irrelevant articles then need to be manually screened and removed. Second, the same studies are retrieved and assessed multiple times by different people across the global systematic review workforce. The Pipeline therefore aims to avoid this duplication of effort by facilitating the reuse of previous assessments as to whether a given report describes, or does not describe, an RCT.
Fig. 1 depicts the role of machine learning within the Pipeline. To populate CENTRAL, Cochrane regularly searches a range of online resources (e.g., biomedical literature databases) through the “Centralised Search Service,” which is described elsewhere [
• Noel-Storr A.
• Dooley G.
• Wisneiwski S.
• Glanville J.
• Thomas J.
• Cox S.
• et al.
Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: a retrospective analysis.
]. Abstracts of these candidate articles (of which the majority are not RCTs) are “fed” into the top of the Pipeline. (The scope and detail of these searches is described here: https://www.cochranelibrary.com/central/central-creation). The machine learning classifier (described in this article) is used to filter out records that are highly unlikely to be an RCT study report. The remaining articles are then handed over to the Cochrane Crowd, which filters out all further records that do not report an RCT [

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, et al. An evaluation of Cochrane Crowd finds that crowdsourcing can help to address the challenge of information overload in evidence synthesis. J Clin Epidemiol.

]. Finally, the remaining articles (which should all describe RCTs) are stored in CENTRAL. Crowd “labels” are also used to update the machine learning algorithm so that it becomes more accurate at distinguishing between relevant and irrelevant records (see Section 2.7).

### 2.2 Data sets and their role in this study

High-quality data sets are vital for the development, calibration, and evaluation of reliable machine learning classifiers. Most evaluations of such classifiers use a single data set, which is split at random between “training” and “test” data (e.g., with 70% of the data reserved for training). The training data are used to estimate the model parameters, and the test data are used to evaluate its performance. However, although a single data set evaluation can provide estimates of classifier performance that have strong internal validity, it cannot tell us how well a classifier will perform in the real world, where data may come from sources that differ in important ways from those used to produce this data set. As outlined by Nevin, it is important to consider the external validity of machine learning models before deployment [
• Nevin L.
Advancing the beneficial use of machine learning in health care and medicine: toward a community understanding.
]. Here, we examined external validity in terms of whether the use of our machine learning model would risk missing RCTs included in Cochrane Reviews.
We therefore used three distinct data sets in the present study:
• Training data, from which the machine learning models are built;
• Calibration data, on which the threshold for determining the cutoff between “RCT” and “non-RCT” classifications was based; and
• Validation data, on which the calibrated classifier was evaluated.
Fig. 2 summarizes the contribution made by each data set that we now describe in detail.

### 2.3 Training data

The data set used to train the classifier comprises a corpus of 280,620 title–abstract records retrieved from Embase using a highly sensitive search for RCT (https://www.cochranelibrary.com/central/central-creation). This search has been carried out each month since January 2014 for the purpose of identifying relevant studies for inclusion in CENTRAL (see Section 2). In this study, we used records retrieved between January 2014 and July 2016 inclusive. During this period, any records indexed with the Emtree headings “Randomized controlled trial” or “Controlled clinical study” were automatically marked for inclusion in CENTRAL, without any manual checking, on the basis that this rule produced a false positive rate for identifying reports of RCTs that was judged sufficiently low. To account for the historical use of this rule, records with these specific Emtree headings were also excluded from our training data set. Because obvious RCTs and obvious non-RCTs had already been filtered out of this data set (using Emtree headings and the sensitive search filter for RCTs, respectively) before we used it to train the classifier, the data set therefore comprises records that are, on average, more difficult to classify according to whether or not they report an RCT, compared with an unfiltered sample from the raw database.
Next, each record in the training data set was labeled by Cochrane Crowd members according to whether it reported an RCT (n = 20,454) or not (n = 260,166). Each record was labeled by multiple Crowd members, with the final Crowd decision being determined by an agreement algorithm; Noel-Storr et al. report that the Crowd recall and precision for identifying RCTs both exceeded 99% [

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, et al. An evaluation of Cochrane Crowd finds that crowdsourcing can help to address the challenge of information overload in evidence synthesis. J Clin Epidemiol.

].
This data set (“Cochrane Crowd data” in Fig. 2) has characteristics that make it highly suitable for training a machine learning classifier: it is both large—so represents a wide range of instances of both the positive and negative classes (i.e., RCTs and non-RCTs)—and also very accurately labeled. However, it also comprised records added to Embase during a relatively short (<3 years) period, which could limit the generalizability of the resulting machine learning classifier.

### 2.4 Calibration data

When machine learning classifiers are used for prediction, they output a score (often scaled to be bounded by 0 and 1) that gets assigned to each title–abstract record, with a higher value representing an increased likelihood that the record reports an RCT. However, to use the classifier to reduce manual screening workload, we also needed to set a threshold score below which records (unlikely to be RCTs) are discarded and conversely above which records (possible RCTs) are retained for manual screening. Higher score thresholds can be expected to lead to a higher prevalence of reports of RCTs (true positives) among fewer retained records (i.e., higher precision) but at the expense of having discarded some reports of RCTs (false negatives) with scores below the threshold (i.e., lower recall). Conversely, a lower threshold can be expected to lead to lower precision but higher recall.
We sought the advice from the Cochrane IRMG and were advised that the classifier would need to have a threshold score calibrated to retrieve at least 99% of relevant RCT study reports to be adopted for use in Cochrane and also that achieving this high level of recall should be prioritized over any reductions in manual screening workload. These specifications reflect the strong aversion that we have, when conducting systematic reviews, to inadvertently failing to identify studies that should be included.
The Clinical Hedges data set (Fig. 2) was built during 2000 and 2001 for the purposes of testing and validating sensitive search filters for RCTs [
• Wilczynski N.L.
• Douglas M.
• Haynes R.B.
Team and the H
An overview of the design and methods for retrieving high-quality studies for clinical care.
]. It contains 49,028 title–abstract records manually identified and selected by information specialists using a combination of hand search and electronic search methods. Corresponding full-text reports of all records were manually checked to ascertain with confidence whether or not each reported an RCT, making this a highly accurate data set for our current purpose. Three records from this data set were no longer available in PubMed, so our final calibration data set comprised 49,025 PubMed title–abstract records, of which 1,587 reported an RCT (and the remaining 47,438 did not report an RCT).
It was more demanding to calibrate our RCT classifier on this data set (compared with using a proportion of records held back from the Cochrane Crowd data set) because (1) the records are older and are less likely to have a consistent reporting structure for RCTs because the study reports were published only a few years after the CONSORT statement (and before the latter became widely used) [
• Schulz K.F.
• Altman D.C.
• Moher D.
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
] and (2) the Clinical Hedges Team's assessments were based on full-text reports, and there is no indication in some of the corresponding titles and abstracts that they actually report an RCT. We used this data set to identify the threshold for achieving 99% recall and thereby calibrate our classifier, and we also present results concerning the precision with which these records can be identified.

### 2.5 Validation data

As described previously, this machine learning classifier was primarily designed to identify records of study reports potentially eligible for inclusion in Cochrane Reviews. We therefore validated the classifier using a third data set to determine whether the desired level of 99% recall (calibrated using the Clinical Hedges dataset) could be achieved in practice. This validation data set (“Cochrane Included Studies” in Fig. 2) comprises title and abstract records of all study reports included in Cochrane Reviews in which eligible study designs are restricted to “RCTs only,” published up to April 2017. The data set comprises 94,305 records of 58,283 included studies across 4,296 Cochrane Reviews. Although it could be assumed that the vast majority of these records report an RCT, in practice, we found that some records of included study reports did not report an RCT (e.g., they reported a meta-analysis of RCTs, a related editorial, or personal correspondence). These records were retained in the validation set, as removing them all would have required the manual screening of all records.

### 2.6 Excluded data

Articles without an abstract (i.e., title-only records) may contain insufficient information for accurate machine (or human) classification. However, title-only records that include the words “randomised controlled trial” (as per CONSORT guidance) should be labeled correctly by a classifier. In consultation with the IRMG and in the light of manual assessment of records with some content in their abstract field (but not a full abstract), we determined that pragmatic cutoffs for including a record in the training, calibration, or validation data sets would be set at 400 characters as a minimum abstract length and 15 characters as a minimum title length. These cutoffs aimed to balance the need for sufficient text to be present in the abstract field for the machine learning to operate while not referring too many records for manual assessment. It is important to note that, in the Cochrane Evidence Pipeline workflow (mentioned previously), all records with title and/or abstract fields that have fewer characters than the minimum cutoff are referred for manual screening by members of the Cochrane Crowd. When the minimum character cutoff is applied to the Cochrane Included Studies data set, the final number of studies in the evaluation falls to 44,007.

### 2.7 Machine learning methods for RCT identification

Machine learning describes a group of algorithms that seek to “learn” to do a task by exposure to (typically large amounts of) data. The approach we used here can be described as supervised machine learning: meaning that the algorithm is “trained” on articles for which the true label is already known. Although the current state-of-the-art approach for text classification is the use of neural network models, we have previously found that support vector machine (SVM) models (and specifically ensembles [ensembling describes a strategy of using multiple machine learning models together, with the aim of improving performance compared with any model individually] of multiple of SVM models) were similarly accurate for high-sensitivity document recall [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
]. SVMs are less computationally intensive than neural models and therefore can run quickly and without the need for any special computer hardware. The final Cochrane RCT Classifier model also needed to be deployed in a live Web service that might need to cope with heavy user demand. For these reasons, we chose SVMs for the present study. We refer the interested reader to a detailed description of machine learning methods as applied to abstract classification [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
]. In our previous work, we incorporated metadata from the database describing study design into our models (the Publication Type tag in MEDLINE, which is manually added by MEDLINE staff, often several months after publication). However, as the Evidence Pipeline retrieves mainly very new records, which usually lack this metadata, we used a model that uses titles and abstract text without additional metadata.
We used the bag-of-words approach, in which each title–abstract record is represented as a vector of 0s and 1s, depending on the presence or absence of each unique word from the article set vocabulary [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
]. These vector representations are then used to “train” (i.e., find optimal parameters for) an SVM model. We preprocessed the records to remove commonly used words (e.g., “and,” “the”) that appear on the PubMed “stopword” list [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
]. During our initial development phase, we found that an ensemble of two SVM models with minor differences resulted in greatest accuracy when evaluated on the training data. We therefore selected an ensemble of two SVM variants for use in the study (see Fig. 2). The first classifier (SVM1) represented the texts as individual words, pairs of words, and triplets of words (uni-, bi-, and tri-grams). This accounts for situations in which adjacent words affect document class (e.g., the text “randomized controlled trial” might be more strongly indicative of an RCT than any word individually). The second classifier (SVM2) used a unigram model (i.e., each word is considered individually) and used a strategy of oversampling. This strategy aims to reduce the likelihood of missing a “rare class” (here the “rare class” is RCTs, which account for ~5% of the data set) by artificially increasing the number of RCTs in the training data set by random sampling with replacement (this process is not repeated with the calibration or validation data sets, which are left in their original state).
The source code for building SVM2 is available at https://github.com/ijmarshall/robotsearch.

### 2.8 Generating calibrated probability estimates

SVMs estimate the distance between a given record and a “hyperplane” [
• Sain S.R.
• Vapnik V.N.
The nature of statistical learning theory.
], which, in the current use, separates RCTs (the positive class) from non-RCTs (the negative class). The hyperplane distance metric is not readily interpretable (in our data set, this metric had a numeric value approximately between −1 and +8), and we therefore sought to add probability estimates to meet the needs of Cochrane users, who have found this feature to be particularly useful in understanding the output. To achieve this, we calibrated the ensemble SVM scores on the Clinical Hedges data set using a logistic regression model (known as Platt scaling) [
• Platt J.C.
Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.
]. This generated a score for each “unseen” record in the calibration or validation data sets that is bounded by 0 and 1. These scores are closer to representing the true probability that a given record reports an RCT; as such they are readily interpretable, with higher scores representing a higher likelihood that the record reports an RCT (and vice versa). When viewed graphically, the distribution of scores is often U-shaped, with most records being assigned either a high (close to 1) or low (close to 0) probability score, and a smaller number of records in the middle that are more ambiguous in terms of class membership.

### 2.9 Evaluation metrics

In this article, we use the conventional information retrieval terminology recall and precision, which are synonymous with sensitivity and positive predictive value, respectively. As outlined previously, the recall statistic is of primary concern in the current use scenario—that is, that eligible study reports are not incorrectly discarded from the Evidence Pipeline workflow. Cochrane required the system to have at least 99% recall. Recall is calculated as the proportion of relevant items (i.e., records describing an RCT) that are correctly identified by the Evidence Pipeline workflow compared with the total number of records genuinely reporting an RCT that should have been identified.
To evaluate the discriminative performance and quality of calibration of our machine learning strategy on the Clinical Hedges data, we used bootstrap sampling as described by Steyerberg et al. [
• Steyerberg E.W.
• Harrell F.E.
• Borsboom G.J.J.M.
• Eijkemans M.J.C.
• Vergouwe Y.
• Habbema J.D.F.
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.
]. In short, a series of artificial new data sets were “bootstrapped” by random sampling with replacement from the Clinical Hedges dataset. Logistic regression models (which served the dual purposes of ensembling the individual SVM models and producing calibrated probability outputs) were trained on each sampled data set and evaluated on the original data set. This process was repeated 5,000 times and used to estimate performance metrics with 95% confidence intervals. Although the primary use of the system is for binary classification, a key secondary use is providing indicative probability scores to users. We evaluate the quality of the probabilities via a calibration plot and by calculation of the Brier score and C statistics [
• Brier G.
Verification of forecasts expressed in terms of probability.
].
Common practice in Cochrane Reviews is to find, use, and cite all published (and unpublished) reports of each included study (“study reports”). Many studies included in Cochrane Reviews comprised multiple study reports. This means that if the classifier “misses” one of several study reports of the same RCT, this does not necessarily mean the RCT study has been “lost.” We therefore adopted the following approach. We first classified all study reports in Cochrane Reviews of RCTs using the machine learning classifier and then we considered a study to be “lost” only if all reports of that study fell below the threshold. As such, the “study” is our unit of analysis rather than the “study report.” We made this decision because we found many secondary citations in reviews referred to indirectly related non-RCT studies and also because we would expect the retrieval of a single article would alert the review team to the existence of the trial.
Precision is also a metric of interest because it can be used to compute the number of articles requiring manual screening by Cochrane Crowd in the Evidence Pipeline workflow. Here, we were concerned with the number of irrelevant records (i.e., records not reporting an RCT) that are incorrectly classified by machine learning as relevant (i.e., records with an assigned probability score above the identified threshold score), which must then be filtered out manually by the Cochrane Crowd. Precision is calculated as the proportion of retrieved records, which genuinely report an RCT.
Recall and precision were calculated from a 2 × 2 table representing positive/negative (relevant/irrelevant) classes and whether they were correctly or incorrectly classified (Table 1).
Table 12 × 2 table from which precision and recall are calculated
RCTs (gold standard)Non-RCTs (gold standard)
Machine learning classed RCTsTrue positivesFalse positives
Machine learning classed non-RCTsFalse negativesTrue negatives
Abbreviation: RCT, randomized controlled trial.
Formulas used to calculate precision and recall are as follows:
$precision=truepositivestruepositives+falsepositivesrecall=truepositivestruepositives+falsenegatives$

We computed statistics for precision at 99% recall against the Clinical Hedges calibration data set. As specified previously, recall was set at 99% by the IRMG. As the Cochrane Reviews we examined contain only RCTs (and the non-RCTs excluded during searches are not usually comprehensively recorded), we were not able to calculate precision on the Cochrane Reviews dataset and report recall only.
For the primary analysis, the denominator was all articles in Cochrane Reviews meeting the minimum character length criteria described previously (i.e., very short titles and abstracts were excluded). We assume that manual assessment will yield 100% recall of these records. We also report results on the full data set, without removing articles with small or nonexistent abstracts, as a secondary analysis. The first figure can be interpreted as the recall of the overall workflow because it takes account of our decision to remove records with insufficient information for machine classification from the workflow.
We present absolute values of the total number of eligible studies “lost” to Cochrane Reviews. Finally, we also present the distribution of “lost” study reports according to the year of publication because we hypothesize that the classifier may perform less well on older study reports because (1) it has been trained on newer reports and (2) trial reporting may have improved as a result of the CONSORT statement.

## 3. Results

The machine learning classifier for identifying reports of randomized trials (Cochrane RCT Classifier) was built as per the previously mentioned methods from the screening of 280,620 Embase records (January 2014 to July 2016) by Cochrane Crowd. Of these, 20,454 (7.3%) were deemed to be RCTs.

### 3.1 Threshold setting and binary classification performance

The 49,025 records from the Clinical Hedges data set were scored by the machine learning classifier. The records were ordered according to classifier score, and precision and recall statistics were calculated for every record in sequence. The classifier probability, which corresponded with 99% recall, was recorded and used as the classification threshold for the later validation (and the deployed system). The discriminative and calibrative performance of this strategy, estimated using bootstrap sampling is presented in Table 2. We estimate that precision was 8.3%, meaning that one in every 12 records retrieved described an RCT. Setting the classifier at this level of recall resulted in 58% of records in this data set being automatically discarded as highly unlikely to be reporting a randomized trial.
Table 2Bootstrap estimates of model performance on Clinical Hedges data set, with 95% confidence intervals
Validation precisionValidation recallValidation specificityC statisticBrier score
0.08 (0.06, 0.12)0.99 (0.98, 0.99)0.63 (0.48, 0.76)0.98 (0.98, 0.98)0.05 (0.05, 0.05)
Estimates of the C statistic and Brier score were 0.978 and 0.048, respectively, indicating excellent discriminative performance. We present a calibration plot showing point estimates from the bootstrap evaluation and the final model (trained on the whole dataset) in Fig. 3. We show how the predicted scores are distributed for RCTs and non-RCTs in Fig. 4.

### 3.2 Validating the classifier recall on studies included in Cochrane Reviews

The title and abstract records of 58,283 studies included in 4,296 Cochrane Reviews were fed through the classifier. Records with a score equal to or above the threshold identified in the previous step were automatically classified as potentially reporting an RCT; those scoring below this threshold were automatically classified as not reporting an RCT.
Table 3 summarizes the number of eligible studies that are “lost” to reviews as a result of all of their corresponding study reports scoring lower than the threshold. When records that contain insufficient information for machine classification are excluded from machine classification and assumed to be manually assessed (see Section 2), the classifier correctly identifies 99.5% (43,783 out of 44,007) of studies.
Table 3Number of included studies in Cochrane Reviews classified as RCTs
RCTS correctly identified by the classifier (recall)RCTS not identified by the classifier
All studies (N = 58,283)54,683 (93.8%)3,600 (6.2%)
Studies with sufficient information for machine classification (N = 44,007 studies)43,783 (99.5%)224 (0.5%)
Abbreviation: RCT: randomized controlled trial.
In our secondary analysis, when we include for machine classification data for all studies (including the subset of studies, which contain insufficient information for accurate machine classification [see Section 2]), we find that 3,396 studies would be potentially “lost” to reviews (compared with 224 studies when only those with sufficient information are included).
Fig. 5 shows the 224 randomized trials “lost” by the classifier per 1,000 published, by year of publication, for all but one of the publications (the age of one publication could not be ascertained). These results show that older reports are much more likely to be misclassified by the machine learning classifier.

## 4. Discussion

### 4.1 Summary of findings

We conducted a three-stage study that involved training, calibrating, and evaluating a machine learning classifier designed to distinguish between bibliographic title–abstract records that report an RCT and those that do not. Recall falls to an unacceptably low level (94%) if records with limited information in their titles and/or abstracts are submitted for machine classification. However, when these records are excluded, the classifier exceeds the standard required by Cochrane with recall at 99.5% of all those records scored. It should be noted that this means that some records are unsuitable for machine learning and so must necessarily be checked manually; however, this mirrors current practice, whereby records with limited information in their titles and abstracts are retained for further assessment on the basis of their corresponding full-text reports.
We deem the recall level as “acceptable” for use in “live” reviews on the basis that (1) this exceeds the recall of validated RCT search filters that have been used in systematic review production for many years and (b) this threshold was agreed by methodologists in Cochrane for use in Cochrane Reviews.
Although the precision of 8% estimated against the Clinical Hedges dataset appears low, this is partly because of the age of that data set and relatively low prevalence of RCTs. In the Cochrane Evidence Pipeline workflow (Fig. 1), the classifier saved Cochrane Crowd from needing to check 185,039 records manually (out of a total of 449,480) during the 2018 calendar year, a very large saving in manual workload [

Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, et al. An evaluation of Cochrane Crowd finds that crowdsourcing can help to address the challenge of information overload in evidence synthesis. J Clin Epidemiol.

].
Systematic reviews are frequently used to support decision-making processes for both policymakers and practitioners and are also key sources of evidence in drug licensing regulation. Reviews need to be accurate representations of the state of current knowledge, as decisions that are based on their findings can affect people's lives. Reviews also need to be demonstrably correct, as the way in which evidence is synthesized can have implications, for example, for drug licensing, and can therefore be open to legal challenge. These joint imperatives—for systematic reviews to be correct and to be seen to be correct—generate the normative expectation that they should contain all relevant research evidence and the corollary concern that review findings based on bodies of evidence that inadvertently exclude some eligible studies are potentially unreliable. To this end, our study provides data demonstrating the reliability of implementing what could be seen as a major innovation in study identification methods for systematic reviews, the automatic eligibility assessment of study reports, and the exclusion of a portion without any manual checking by humans, rolled out at scale across Cochrane: the largest producer of systematic reviews globally and an organization committed to minimizing the risk of bias in review production through methodological and editorial rigor. We note that the recall threshold set by Cochrane (99%) exceeds the performance of conventional search methods (e.g., the Cochrane Highly Sensitive Search Strategy was found to have recall of 98.5% by the Clinical Hedges team) [
• McKibbon K.A.
• Lou W.N.
• Haynes R.B.
Retrieving randomized controlled trials from MEDLINE: a comparison of 38 published search filters.
], and we have demonstrated in previous work that our machine learning approach can exceed the precision achieved by conventional search filters [
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
].
Although our results indicated that 0.5% of studies could have been “lost” to Cochrane Reviews if authors had used this classifier (affecting 178 reviews, leaving 4,118 reviews unaffected), this is almost certainly an overestimate when considering the prospective use of this classifier to support the identification of newly published RCTs for new, updated, and/or living systematic reviews. First, other means of finding studies are routinely used in Cochrane Reviews alongside conventional electronic searching—such as checking reference lists and citing records or contacting researchers who are active in the topic area—so some of these “lost” studies would likely be found using these complementary search methods. Second, studies that are potentially lost are overwhelmingly older reports. Although we do not dismiss the potential importance of identifying older trials for consideration in systematic reviews, it is reassuring that more recent studies (relevant especially for newer treatments and review updates) are far less likely to be missed. One reason the classifier performs better for more recent studies could be improvement in the reporting of RCTs over time, for example, in response to the CONSORT statement [
• Schulz K.F.
• Altman D.C.
• Moher D.
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
,
• Turner L.
• Shamseer L.
• Altman D.G.
• Schulz K.F.
• Moher D.
Does use of the CONSORT statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review.
]. Trialists are now widely expected to detail trial methodology in the report's abstract and to include the fact that they are reporting an RCT in its title.

### 4.2 Strengths and weaknesses of this evaluation

In this article, we have described a robust evaluation of the performance of an RCT classifier in a large data set of systematic reviews. We were fortunate in having three large, independently generated, high-quality data sets available to train, calibrate, and validate the classifier. This is an unusual position to be in, and there are probably few study designs other than RCTs with comparably high-quality data sets available. We note that this may limit the potential to evaluate the performance of similar workflows, created to identify other types of study design, using the same three-stage process.
The current classifier has been trained almost exclusively on records published in English, so it does not necessarily generalize to other languages. However, this important limitation is, in principle, surmountable, as machine learning technology is language agnostic and would therefore be capable of modeling any language, so long as sufficient training data were available.
The focus of this work has been to build a machine learning classifier for deployment in a specific workflow. The machine learning classifier we have developed meets required levels of recall but inevitably results in some studies being “lost” to reviews. This study does not attempt to ascertain the impact of these losses on the affected reviews' statistical and narrative results and findings, and a future extension of this study will investigate this important question. We also note that only 178 of 4,296 reviews were affected, leaving results unchanged in at least 96% of the reviews.

### 4.3 Next steps: the “Screen4Me” service

We are currently piloting an extension to the Evidence Pipeline for use with individual Cochrane Reviews. Authors using this service will compile their set of potentially eligible records from searches of multiple databases (including CENTRAL) as is typical for any systematic review. Given that the RCT classifier and Cochrane Crowd have already classified more than 800,000 study records (and increasing by >10,000 per month), it is likely that a proportion of the records retrieved and uploaded to the Classifier by authors have already been classified according to whether they report an RCT or not. Where this is the case, the records that are already known not to describe RCTs will be removed from the workflow. The remaining studies will then be sent to the RCT classifier, and those records classified as not reporting an RCT (i.e., that fall below the 99% recall threshold) will be discarded. Finally, the records classified as potentially reporting an RCT will be screened by Cochrane Crowd. The review team is then left with a much smaller pool of records to examine, containing only RCTs. In early pilots, this new workflow reduced manual screening workload by between 40% and 70% depending on the prevalence of RCTs in the search results of individual reviews.

## 5. Conclusions

The Cochrane RCT Classifier is now deployed by Cochrane for reducing screening workload in review production. As part of a wider workflow that includes prospective database searches and crowdsourcing to build a comprehensive database of RCTs, machine learning can reduce the manual screening burden associated with research synthesis while ensuring a very high level of recall that is acceptable for an organization, which depends on having comprehensive access to the published research that falls within health care topics relevant to its scope.

## Acknowledgments

The authors acknowledge the vital contribution that members of Cochrane Crowd have made to this work. Without their patient, accurate labeling of hundreds of thousands of records, this work would have been impossible. Likewise, the authors acknowledge the contribution that McMaster University has made to this field in compiling the Hedges data set and making it available for methodological development. The availability of these data means that successive developments can be measured against the same baseline. The authors would like to thank the members of the Cochrane Information Retrieval Methods Group Executive for their helpful engagement, as they determined the classifier thresholds and discussed its implementation within Cochrane. Their input enabled the authors to calibrate the classifier at a threshold considered viable for use in “live” reviews and also to construct the validation portion of the study to test that our threshold would be sufficiently generalizable. The authors also acknowledge the contribution that Byron Wallace (Northeastern University) has made to the field. As well as collaborating with us on many projects, he carried out a code review of the SVM1 classifier, which is derived from his own work available at https://github.com/bwallace/curious_snake. The authors acknowledge with thanks permission to use Cochrane data and Gordon Dooley from Metaxis for supplying these data.

## References

• Cochrane
Cochrane Library.
(Available at)
https://www.cochranelibrary.com/
Date: 2019
Date accessed: October 26, 2019
• Lefebvre C.
• Glanville J.
• Briscoe S.
• Littlewood A.
• Marshall C.
• Metzendorf M.
• et al.
Chapter 4: searching for and selecting studies.
in: Higgins J. Thomas J. Chandler J. Cumpston M. Li T. Page M. Cochrane Handbook for Systematic Reviews of Interventions. 2nd ed. John Wiley & Sons, Chichester, UK2019: 67-99
• Bastian H.
• Glasziou P.
• Chalmers I.
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?.
PLoS Med. 2010; 7: e1000326
• Shojania K.G.
• Sampson M.
• Ansari M.T.
• Ji J.
• Doucette S.
How quickly do systematic reviews go out of date? A survival analysis.
Ann Intern Med. 2007; 147: 224-233
• Macleod M.R.
• Michie S.
• Roberts I.
• Dirnagl U.
• Chalmers I.
• Ioannidis J.P.A.
• et al.
Biomedical research: increasing value, reducing waste.
Lancet. 2014; 383: 101-104
• Tsafnat G.
• Dunn A.
• Glasziou P.
• Coiera E.
The automation of systematic reviews.
BMJ. 2013; 346: f139
• O’Mara-Eves A.
• Thomas J.
• McNaught J.
• Miwa M.
Using text mining for study identification in systematic reviews: a systematic review of current approaches.
Syst Rev. 2015; 4: 5
• Marshall I.
• Storr A.N.
• Kuiper J.
• Thomas J.
• Wallace B.C.
Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide.
Res Synth Methods. 2018; 9: 602-614
• Wallace B.C.
• Noel-Storr A.
• Marshall I.J.
• Cohen A.M.
• Smalheiser N.R.
• Thomas J.
Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach.
J Am Med Inform Assoc. 2017; 24: 1165-1168
• Thomas J.
• Noel-Storr A.
• Marshall I.
• Wallace B.
• McDonald S.
• Mavergames C.
• et al.
Living systematic reviews: 2. Combining human and machine effort.
J Clin Epidemiol. 2017; 91: 31-37
• Cochrane
About the CRS (Cochrane Register of Studies). Cochrane Community.
(Available at)
• Noel-Storr A.
• Dooley G.
• Wisneiwski S.
• Glanville J.
• Thomas J.
• Cox S.
• et al.
Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: a retrospective analysis.
J Clin Epidemiol. 2020; 127: 142-150
1. Noel-Storr A, Dooley G, Elliott J, Steele E, Shemilt I, Mavergames C, et al. An evaluation of Cochrane Crowd finds that crowdsourcing can help to address the challenge of information overload in evidence synthesis. J Clin Epidemiol.

• Nevin L.
Advancing the beneficial use of machine learning in health care and medicine: toward a community understanding.
PLoS Med. 2018; 15: 4-7
• Wilczynski N.L.
• Douglas M.
• Haynes R.B.
• Team and the H
An overview of the design and methods for retrieving high-quality studies for clinical care.
BMC Med Inform Decis Mak. 2005; 5: 1-15
• Schulz K.F.
• Altman D.C.
• Moher D.
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
BMC Med. 2010; 8: 1-9
• Sain S.R.
• Vapnik V.N.
The nature of statistical learning theory.
Technometrics, 2006: 131
• Platt J.C.
Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.
• Steyerberg E.W.
• Harrell F.E.
• Borsboom G.J.J.M.
• Eijkemans M.J.C.
• Vergouwe Y.
• Habbema J.D.F.
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis.
J Clin Epidemiol. 2001;
• Brier G.
Verification of forecasts expressed in terms of probability.
Mon Weather Rev. 1950;
• McKibbon K.A.
• Lou W.N.
• Haynes R.B.
Retrieving randomized controlled trials from MEDLINE: a comparison of 38 published search filters.
Health Info Libr J. 2009; 26: 187-202
• Turner L.
• Shamseer L.
• Altman D.G.
• Schulz K.F.
• Moher D.
Does use of the CONSORT statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review.
Syst Rev. 2012; 29: 60


text:
# More on linear independence

## Video transcript

I think by now we have a reasonable sense of what linear dependence means. So let's just do a slightly more formal definition of linear dependence. So we're going to say that a set of vectors-- Let me just define my set of vectors. Let me call my set s of vectors, v1, v2, all the way to vn. I'm going to say that they are linearly dependent. If and only if. So sometimes it's written if, if with a lot of f's in there. So sometimes it's written if and only if. Sometimes it's shown like an arrow in two directions. If and only if I can satisfy this equation, I can find a set of constants c1 times v1. I can take a linear combination of my vectors all the way to cn vn, that satisfy the equation that I can create this into the 0 vector. Sometimes it's just written as a bold 0, and sometimes you could just write it -- I mean we don't know the dimensionality of this vector. It would be a bunch of 0's. We don't know how many actual elements are in each of these vectors, but you get the idea. My set of vectors is linearly dependent-- remember I'm saying dependent, not independent --is linearly dependant, if and only if I can satisfy this equation for some ci's where not all of them are equal to 0. This is key, not all are 0. Or you could say it the other way. You could say at least one is non-zero. So how does this gel with what we were talking about in the previous video where I said look, a set is linearly dependent if one of the of vectors can be represented by the combination of the other vectors? Let me write that down. In the last few I said, look, one vector can be -- Let me write it this way. One vector being represented by the some of the other vectors, I can just write it like this. I can write it a little bit more math-y. In the last video, I said that linear dependence means that-- let me just pick an arbitrary vector, v1. Let's say that v1, you know this is arbitrary, v1 one could be represented by some combination of the other vectors. Let me call them a1 times v -- let me be careful -- a2 times v2 plus a3 times v3 plus all the way up to an times vn. This is what we said in the previous video. If this is linear dependence, any one of these guys can be represented as some combination of the other ones. So how does this imply that? In order show this if and only if, I have to show that this implies that and I have to show that that implies this. So this is almost a trivially easy proof. Because if I subtract v1 from both sides of this equation I get 0 is equal to minus 1 v1 plus a2 v2 plus a3 v3 all the way to an vn. And clearly I've just said, well, this is linearly dependent. That means that I can represent this vector as a sum of the other vectors, which means that minus 1 times v1 plus some combination of the other vectors is equal to 0, which means that I've been able to satisfy this equation, and at least one of my constants is non-zero. So I've shown you that, if I can represent one of the vectors by a sum of the other ones, then this condition is definitely going to be true. Now let me go the other way. Let me show you if I have this situation that I can definitely represent one of the vectors as the sum of the others. So let's say that this is true. And one of these constants, remember it's not just this, it's at least one, is non-zero. So let me just assume, just for the sake of simplicity-- I mean these are all arbitrary. I'll do it in a new color. Let me do it in the magenta. Let me assume that c1 is not equal to 0. If c1 is not equal to 0, then I can divide both sides of this equation by c1. And what do I get? I get v1 plus c2 over c1 v2 plus all the way up to cn over c1 is equal to 0. And then I can multiply both sides of this, or I could add negative v1 to both sides of this equation or subtract v1 from both sides. And I get c2 over c1 v2 plus all the way up to cn over c1 vn-- there's a vn here --is equal to minus v1. Now if I just multiply both sides of this by negative 1, I get a minus, and all these become minuses and this becomes a plus. And I just showed you that if at least one of these constants is non-zero, that I can represent my vector v1 as some combination of the other vectors. So we're able to go this way too. If this condition is true, then I can represent one of the vectors as a combination of the others. If I can represent one of those vectors as a combination of the others, then this condition is true. Hopefully that kind of proves that these two definitions are equivalent. Maybe it's a little bit of overkill. Let's apply that definition now, to actually test. You might say, hey Sal, why'd you go through all of this effort? I went through all of this effort because this is actually a really useful way to test whether things are linearly independent or dependent. Let's try it out. Let's use our newly found tool. Let's say I have the set of vectors-- Let me do it up here. I want to be efficient with my space usage. So let's say I have the set of vectors 2,1 and 3,2. And my question to you is, are these linearly independent or are they linearly dependent? In order for them to be linearly dependent, that means that if some constant times 2,1 plus some other constant times this second vector, 3,2 where this should be equal to 0. Where these both aren't necessarily 0. Before I go up for this problem, let's remember what we're going to find out. If either of these are non-zero, if c1 or c2 are non-zero, then this implies that we are dealing with a dependent, linearly dependent set. If c1 and c2 are both 0, if the only way to satisfy this equation -- I mean you can always satisfy it by sitting everything equal to 0. But if the only way to satisfy it is by making both of these guys 0, then we're dealing with a linearly independent set. Let's try to do some math. And this'll just take us back to our Algebra 1 days. In order for this to be true, that means 2 times c1 plus 3 times c2 is equal to -- when I say this is equal to 0, it's really the 0 vector. I can rewrite this as 0,0. So 2 times c1 plus 3 times c2 would be equal to that 0 there. And then we'd have 1 times c1 plus 2 times c2 is equal to that 0. And now this is just a system, two equations, two unknowns. A couple of things we could do. Let's just multiply this top equation by 1/2. If you multiply it by 1/2 you get c1 plus 3/2 plus 3/2 c2 is equal to 0. And then if we subtract the green equation from the red equation this becomes 0. 2 minus 1 and 1/2-- 3/2 is 1 and 1/2 --of this is just 1/2 c2 is equal to 0. And this is easy to solve. c2 is equal to 0. So what's c1? Well, just substitute this back in. c2 is equal to 0. So this is equal to 0. So c1 plus 0 is equal to 0. So c1 is also equal to 0. We could have substituted it back into that top equation as well. So the only solution to this equation involves both c1 and c2 being equal to 0. So they both have to be 0. So this is a linearly independent set of vectors. Which means that neither of them are redundant of the other one. You can't represent one as a combination of the other. And since we have two vectors here, and they're linearly independent, we can actually know that this will span r2. The span of my r vectors is equal to r2. If one of these vectors was just some multiple of the other, than the span would have been some line within r2, not all of. But now I can represent any vector in r2 as some combination of those. Let's do another example. Let me scroll to the right, because sometimes this thing, when I go too far down, I haven't figured out why, when I go too far down it starts messing up. So my next example is the set of vectors. So I have the vector 2,1. I have the vector 3,2. And I have the vector 1,2. And I want to know are these linearly dependent or linearly independent. So I go to through the same drill. I use that little theorem that I proved at the beginning of this video. In order for them to be linearly dependent there must be some set of weights that I can multiply these guys. So c1 times this vector plus c2 times this vector plus c3 times that vector, that will equal the 0 vector. And if one of these is non-zero then we're dealing with a linearly dependent set of vectors. And if all of them are 0, then it's independent. Let's just do our linear algebra. So this means that 2 times c1 plus 3 times c2 plus c3 is equal to that 0 up there. And then if we do the bottom rows-- Remember when you multiply a scalar times a vector you multiply it by each of these terms. So c1 times 1. 1c1 plus 2c2 plus 2c3 is equal to 0. There's a couple of giveaways on this problem. If you have three two-dimensional vectors, one of them is going to be redundant. Because, in the very best case, even if you assume that that vector and that vector are linearly independent, then these would span r2. Which means that any point, any vector, in your two-dimensional space can be represented by some combination of those two. In which case, this is going to be one of them because this is just a vector in two-dimensional space. So it would be linearly dependent. And then, if you say, well, these aren't linearly independent then, they're just multiples of each other. In which case, this would definitely be a linearly dependent set. When you see three vectors that are each only vectors in r2, that are each two-dimensional vectors, it's a complete giveaway that this is linearly dependent. But I'm going to show it to you using our dependent, using our little theorem here. So I'm going to show you that I can get non-zero c3's, c2's, and c1's such that I can get a 0 here. If all of these had to be 0-- I mean you can always set them equal to 0. But if they had to be equal to 0, then it would be linearly independent. Let me just show you. I can just pick some random c3. Let me pick c3 to be equal to negative 1. So what would these two equations reduce to? I mean you have just three unknowns and two equations, it means you don't have enough constraints on your system. So if I just set c3-- I just pick that out of a hat. I could have picked c3 to be anything. But if I set c3 to be equal to negative 1, what do these equations become? You get 2c1 plus 3c2 minus 1 is equal to 0. And you get c1 plus 2c2 minus 2 is equal to 0. Right? 2 times minus 1. What can I do here? If I multiply this second equation by 2, what do I get? I get 2 plus 4c2 minus 4 is equal to 0. And now let's subtract this equation from that equation. So the c1's cancel out. 3c2 minus 4c2 is minus c2. And then minus 1 minus minus 4, so that's minus 1 plus 4. That's plus 3 is equal to 0. And so we get our -- Let me make sure I got that right. We have a minus 1 minus a minus 4. So plus 4. So we have a plus 3. So that is a minus 2. So minus c2 is equal to minus 3 or c2 is equal to 3. And if c2 is equal to 3 and c3 is equal to minus 1-- Let's just substitute here, so we get c1 plus 2 times c2, so plus 6, plus 2 times c3. So minus 2 is equal to 0. c1 plus 4 is equal to 0. c1 is equal to minus 4. I'm giving you a combination of c's that will give us a 0 vector. If I multiply minus 4 times our first vector, 2,1, that's c1, plus 3 times our second vector, 3,2 minus 1 times our third vector, 1,2 this should be equal to 0. Let's verify it just for fun. Minus 4 times 2 is minus 8 plus 9 minus 1. Yeah, that's minus 9 plus 9. That's 0. Minus 4 times plus 6 minus 2 that's also 0. So we've just shown a linear combination of these vectors, where actually none of the constants are 0. But all we had to show was that at least one of the constants had to be non-zero, and we actually showed all three of them were. But at least one of these had to be non-zero. And I was able to satisfy this equation, I was able to make them into the zero vector. So this shows, this proves, that this is a linearly dependent set of vectors. Which means one of the vectors is redundant. And you can never just say, oh, this is the redundant vector, because I can represent this as combination of those two. You could just as easily pick this guy as the redundant vector, and say, hey, I can represent this guy as the sum of those two. There's not one bad apple in the bunch. Any of them can be represented by the combination of some other, by all of the rest of them. So hopefully you have a better intuition of linear dependence and independence. Maybe I'll continue. I'll do a few more examples in the next video.


text:
# A Bit More on the z-transform and the LPC Spectrum

We start with a linear constant-coefficient difference equation, relating a linear combination of output values to a linear combination of input values:

 Eq. 1 $$\;\;\;\;\;a_1 y[n] + a_2 y[n-1] + ... + a_{M+1} y[n-M] = b_1 x[n] + b_2x[n-1] + ... + b_{N+1} x[n-N]$$

(The input is x and the output is y, and here we've started indexing the coefficients from 1 upwards, rather than from 0 upwards, to accord with the usage in Octave's filter() function.)

Or equivalently

 Eq. 2 $$\sum_{k=0}^N a_{k+1} y[n-k] = \sum_{k-0}^M b_{k+1} x[n-k] \;\;\;\;\;\;for \;\;1 <=n<=length(x)$$

A little algebra gives us a form of the equation expressing the current output y[n] as a linear combination of inputs and outputs:

 Eq. 2 $$y[n] = -\sum_{k=1}^N c_{k+1} y[n-k] + \sum_{k=0}^M d_{k+1} x[n-k] \;\;\;\;\;\;for \;\;1 <=n<=length(x)$$

where $$c = a/a_1 \;\; and \;\; d = b/a_1$$.

Notice that for an Nth-order autoregressive model -- predicting each output as a linear combination of N previous samples -- there are N+1 $$a$$ coefficients, as in equations 1 and 2 above, with the first coefficient by convention equal to 1. And for an "all-pole" filter -- a purely autoregressive model, with no moving-average part -- we still need a dummy $$b$$ vector, with 1 as the first coefficient (thereby passing in the current input sample, to be combined with the weighted combination of earlier outputs.

The Octave/Matlab function filter() is invoked as

y = filter (b, a, x);

where x is the input vector, b and a are the input and output coefficients as in equations 1 and 2, and y is the output vector.

In the lecture notes, you learned how to calculate a as the solution to a problem in linear regression. Here we'll just use the arburg() function from the signal package, which we can invoke as

a = arburg(x, npoles);

Let's analyze this sentence from the TIMIT corpus:

The vertical line in the first syllable of "outgrew" is at about 0.823 seconds. So we do this:

[X, FS] = wavread("SX48a.wav");
windowT = 0.03;
windowN = round(windowT*FS);
T1 =  0.823;
T1n = round((T1-windowT/2)*FS);
S1 = X(T1n:(T1n+windowN-1));
NOrder = 10;
A1 = arburg(S1,NOrder);

This gives us 11 a coefficients calculated for the cited region of the audio clip:

A1 =
1 -1.4778 1.0066 -0.17666 0.33168 0.029883 -0.21203 -0.19413 0.91318 -0.63106 0.18872

We can ask Octave for the (complex) roots of the polynomial, of which there are 5 complex-conjugate pairs:

roots(A1)
ans =
-0.82266 +    0.37935i    -0.82266 -    0.37935i    -0.28849 +    0.92907i    -0.28849 -    0.92907i     0.61941 +    0.72331i     0.61941 -    0.72331i      0.8242 +    0.49916i      0.8242 -    0.49916i     0.40646 +    0.35125i     0.40646 -    0.35125i

Or more helpfully, we can get the 5 frequency and amplitudes (translating the frequencies from radians into Hz):

R1 = roots(A1)(1:2:9);
Freqs = angle(R1)*(FS/2)/(2*pi);
Amps = abs(R1);
[Freqs Amps]
ans =
1724.9     0.90591
1191.7     0.97282
549.17     0.95229
346.67     0.96357
453.7      0.5372

We can calculate and plot the impulse response:

NFFT=256;
Impulse = zeros(NFFT,1); Impulse(NOrder+1)=1;
ImpulseResponse = filter([1 0],A1,Impulse);
plot(1:NFFT,ImpulseResponse)

And we can compare the power spectrum of the impulse response with the power spectrum of the original signal:

NFFT=512; NFFT2 = NFFT/2;
freqs = (0:(NFFT2-1))*(FS/2)/NFFT2;
% Power Spectrum of LPC Impulse Response:
IR1 = zeros(NFFT,1); IR1(1:length(ImpulseResponse)) = ImpulseResponse;
psd1 = 20*log10(abs(fft(IR1)(1:NFFT2)));  psd1 = psd1-max(psd1);
% Power Spectrum of input signal (with Hamming window):
S1a = hamming(length(S1)).*S1;
xx = zeros(NFFT,1); xx(1:length(S1a)) = S1a;
psd2 = 20*log10(abs(fft(xx)(1:NFFT2)));  psd2 = psd2-max(psd2);
% Plot both:
plot(freqs, psd1,'b',freqs,psd2,'r')

As we discussed in the lecture notes on the z-transform, we can also estimate the spectrum of the autoregressive filter directly from the z-transform of $$H(z)$$, which in the case of a purely autoregressive filter is

$$\frac{Y(z)}{X(z)} = \frac{1}{ 1 + a_1 z^{(-1)} + ... + a_M z^{(-M)} }$$

Since we know that the z-transform reduces to the DTFT for $$z=e^{iw}$$, and we know how to calculate the z-transform of any causal LTI (i.e. the z-transform of its impulse response) from the coefficients of the difference equation, we can write down an expression for the spectrum of the filter (i.e. the ratio of the spectrum of the output to the spectrum of the input) by simply setting $$z=e^{iw}$$ in the z-transform.

This expression will be a periodic, continuous function of the variable w, which is frequency in radians. The period is obviously $$2pi$$. We can get N evenly-spaced samples of one period of the spectrum (which is also what the DFT gives us) by setting (in Matlab-ish) w = ((0:(N-1))/N)*2*pi, i.e.

$$z=e^{i(n/N)2\pi}\;\;\;\;for\;0<=n<=N-1$$

Of course, given that we only want the positive frequencies, we should limit the evaluation to w = ((0:(N-1)/N)*pi.

So let's do it:

N=256;
w = ((0:(N-1))/N)*pi;
Hz = zeros(N,1);
for n=1:N
z = exp(i*w(n));
Hz(n) = 1/(A1*z.^(-(0:NOrder))');
end
psd3 = 20*log10(abs(Hz)); psd3 = psd3-max(psd3);
plot(freqs,psd3,'mo',freqs,psd1,'b')

We see that the two ways of calculating the LDC spectrum -- by taking the FFT of the impulse response of the filter, and by evaluating the z-transform at suitable points around the unit circle -- produce identical results. You should be able to figure out for yourself how to evaluate the z-transform in its factored form.