text:
# Extraneous Solutions – Part 2 of 3

## Solving an Equation as a Sequence of Equation Replacement Operations

Part 1 was so long because I wanted to be extremely thorough and to present things to an audience that perhaps hadn’t thought much about the logic of equation solving at all. Since we’re now all experts, perhaps it’s worth it to summarize everything very succinctly.

Given an equation in one free variable, we want to find the solution set. To do this, we replace that equation with an equivalent equation whose solution set is more obvious.

(1) $8x - 5 = 5x + 1$

(2) $8x = 5x + 6$

(3) $3x = 6$

(4) $x = 2$

If in the transition from (1)-(2), from (2)-(3), and from (3)-(4) we are careful to replace each equation with an equivalent equation, then by the transitivity of equivalence, the original equation and terminal equation are guaranteed to be equivalent. Since the solution set of the terminal equation is obvious, we know the solution set of the original equation, as well. Thus solving an equation requires establishing that certain equation replacement operations are indeed equivalence preserving and having the creativity and experience to know which ones to apply and in what order.

## What are the Equivalence-Preserving Operations on Equations?

If $a = b$, then $f(a) = f(b)$ for any well-defined function $f$. If $a$ and $b$ are expressions containing a free-variable, then any value of that variable which satisfies $a = b$ will also satisfy$f(a) = f(b)$. In other words, if you find it useful, feel free to replace any equation with a new equation which is the result of applying any function to both sides of the original equation. Any solution to the original equation will also be a solution to the new equation.

If the function $f$ is also one-to-one, then by definition, $f(a) = f(b) \Rightarrow a = b$ so any solution of $f(a) = f(b)$ will also be a solution to $a = b$. Thus applying $f$ to both sides of an equation is equivalence-preserving. If $f$ is not one-to-one, then in general, the operation is not equivalence-preserving.

In solving equation (1), we applied $f(n) = n + 5, g(n) = n - 5x$ and $h(n) = n/2$ in that order. Since all three of the functions are one-to-one, we are assured that (1) and (4) are equivalent. If we had cause to apply a non-one-to-one function, then we should be vigilant for extraneous solution.

## A More Interesting Example

Consider

(5) $\sqrt{6x-2} - \sqrt{x+1} = 2$

As I mentioned in the other post, these square roots are begging to be squared, but since there are two of them, one squaring will not be enough. Even though it’s not necessary to do so, it’s helpful to move one radical expression to the other side.

(6) $\sqrt{6x-2} = 2 + \sqrt{x+1}$

(7) $6x - 2 = 4 + 4\sqrt{x + 1} + x + 1$ We squared!

(8) $5x - 7 = 4\sqrt{x+1}$

(9) $25x^2 - 70x + 49 = 16x + 16$ We squared again!

(10) $25x^2 - 86x + 33 = 0$

(11) $(25x - 11)(x - 3) = 0$

So $x \in \{\frac{11}{25}, 3\}$

Since in the transition from (6)-(7) and again in the transition from (8)-(9) we had reason to apply the non-one-to-one function $f(n) = n^2$, we should be vigilant for extraneous solutions. [Note: since both sides of (6) are necessarily positive, applying $f(n) = n^2$ is equivalence-preserving, so no extraneous roots will be created there.] By checking back in the original equation, we see that 3 is a solution, but $\frac{11}{25}$ is not. I am more or less content to leave it at that. But some may ask for more clarity as to exactly what happened and when, so let’s indulge them.

I will now list each equation in reverse order along with its solution set:

(11) $(25x - 11)(x - 3) = 0$                            $\{\frac{11}{25}, 3\}$

(10) $25x^2 - 86x + 33 = 0$                             $\{\frac{11}{25}, 3\}$

(9) $25x^2 - 70x + 49 = 16x + 16$                 $\{\frac{11}{25}, 3\}$

(8) $5x - 7 = 4\sqrt{x+1}$                                     $\{3\}$

Since $5\cdot\frac{11}{25} - 7 = \frac{11}{5} - \frac{35}{5} = -\frac{24}{5} \neq 4\sqrt{\frac{11}{25} + 1} = 4\sqrt{\frac{11}{25} + \frac{25}{25}} = 4\sqrt{\frac{36}{25}} = 4\cdot\frac{6}{5} = \frac{24}{5}$

So we have isolated the precise moment when the extraneous solution $x = \frac{11}{25}$ is created and it appears exactly where we would expect it, in the transition from (8) to (9) as we replaced (8) with the result of applying the non-one-to-one function $f(n) = n^2$ to both sides.

More specifically, if $x = \frac{11}{25}$, (8) reads $- \frac{24}{5} = \frac{24}{5}$, which is false, but (9) reads $(- \frac{24}{5})^2 = ( \frac{24}{5})^2$, which is true. For this particular value of $x$, we squared both sides and replaced a false statement with a true statement. In retrospect, we can say that $x =\frac{11}{25}$ is not a solution to (8) or to any previous equation in the solving sequence, but is a solution to (9) and thus to all subsequent equations in the solving sequence.

(7) $6x - 2 = 4 + 4\sqrt{x + 1} + x + 1$                          $\{3\}$

Since both sides of (7) are positive when $x = 3$, it does not surprise us that,

(6) $\sqrt{6x-2} = 2 + \sqrt{x+1}$                               $\{3\}$

(5) $\sqrt{6x-2} - \sqrt{x+1} = 2$                                $\{3\}$

By fully analyzing the logic behind each step of our equation replacement sequence, we not only:

• confirm that $x = 3$ is a solution and that $x = \frac{11}{25}$ is not and
• understand that squaring both sides may produce an extraneous solution

but also

• isolate the precise step in the solving sequence in which this extraneous solution was created answering the why, how, and when for this problem
• confirm that the non-solution status of $x = \frac{11}{25}$ is not merely due to an error of algebra or arithmetic, but is a direct result of that fact that this value produces an equation (8) of the form $a = -a$

That last point is crucial in distinguishing the phenomenon of extraneous roots from the phenomenon of user error in algebra or arithmetic. If our equation solving sequence consists solely of equivalence-preserving operations, we do not even need to check to see if solutions to our terminal equation are also solutions to our original equation. If we do decide to check, perhaps out of an abundance of caution, and find a discrepancy, then user error must be to blame.

On the other hand, if a solver does employ solution-set-enlarging operations in the solving sequence and finds that a solution to the terminal equation is not a solution to the original equation, is this because the solution is extraneous or due to user error? One could perform an analysis like I did above and confirm that the non-solution is not due to user error, but instead to the logic of the process.

# Extraneous Solutions – Part 1 of 3?

## Disclaimer

Within my small inner circle of math teachers, the mystery of extraneous solutions seems to be the issue of the year. I have so much to say on this topic (algebraic, logical, pedagogical, historical, linguistic) that I don’t really know where to begin. My only disclaimer is that I’m not really sure if this topic is all that important.

## Solving an Equation with a Radical Expression

Consider the following equation:

(1) $2\sqrt{x+8} +5 = 11$

One hardly needs algebra skills or prior knowledge to solve this, but prior experience suggests trying to isolate $x$.

(2) $2\sqrt{x+8} = 6$ (we subtract 5 from both sides)

(3) $\sqrt{x+8} = 3$ (we divide both sides by 2)

Now, if the square root of something is 3, then that something must be 9, so it immediately follows that

(4) $x+8 = 9$

(5) $x = 1$ (we subtract 8 from both sides)

## Squaring Both Sides

In my transition from (3) to (4), I used a bit of reasoning. Some conversational common sense told me that “if the square root of something is 3, then that something must be 9”. But that logic is usually just reduced to an algebraic procedure: “squaring both sides”. If we square both sides of equation (3), we get equation (4).

On the one hand, this seems like a natural move. Since the meaning of $\sqrt{a}$ is “the (positive) quantity which when squared is $a$“, the expression $\sqrt{a}$ is practically begging us to square it. Only then can we recover what lies inside. A quantity “which when squared is $a$” is like a genie “which when summoned will grant three wishes”. In both cases you know exactly what to do next.

Unfortunately, squaring both sides of an equation is problematic. If $a = b$ is true, then $a^2 = b^2$ is also true. But the converse does not hold. If $a^2 = b^2$, we cannot conclude that $a = b$, because opposites have the same square.

This leads to problems when solving an equation if one squares both sides indiscriminately.

## A Silly Equation Leads to Extraneous Solutions

Consider the equation,

(6) $x = 4$

This is an equation with one free variable. It’s a statement, but it’s a statement whose truth is impossible to determine. So it’s not quite a proposition. Logicians would call it a predicate. Linguistically, it’s comparable to a sentence with an unresolved anaphor. If someone begins a conversation with the sentence “He is 4 years old”, then without context we can’t process it. Depending on who “he” refers to, the sentence may be true or false. The goal of solving an equation is to find the solution set, the set of all values for the free variable(s) which make the sentence true.

Equation (6) is only true if $x$ has value 4. So the solution set is $\left\{4 \right\}$. But if we square both sides for some reason…

(7) $x^2 = 16$ has solution set $\left\{4, -4\right\}$

We began with $x = 4$, “did some algebra”, and ended up with $x^2 = 16$. By inspection, $-4$ is a solution to $x^2 = 16$, but not to the original equation which we were solving, so we call $-4$ an “extraneous solution”. [Extraneous – irrelevant or unrelated to the subject being dealt with]

Note that the appearance of the extraneous solution in the algebra of (6)-(7) did not involve the square root operation at all. But this example was also a bit silly because no one would square both sides when presented with equation (6), so let’s look at a slightly less silly example.

(8) $2\sqrt{x+8} + 5 = -1$

(9) $2\sqrt{x+8} = -6$

(10) $\sqrt{x+8} = -3$

People paying attention might stop here and conclude (correctly) that (10) has no solutions, since the square root of a number can not be negative. Closer inspection of the logic of the algebraic operations in (8)-(10) enables us to conclude that the original equation (8) has no solutions either. Since $a = b \iff a - 5 = b -5$, any solution to (8) will also be a solution to (9) and vice versa. Since $a = b \iff a/2 = b/2$, any solution to (9) will also be a solution to (10) and vice versa. So equations (8), (9), and (10) are all “equivalent” in the sense that they have the same solution set.

But what if the equation solver does not notice this fact about (10) and decides to square both sides to get at that information hidden inside the square root?

(11) $x+8 = 9$

(12) $x = 1$

Again we have an extraneous solution. $x = 1$ is a solution to (12), but not to the original equation (8). Where did everything go wrong? By the previous logic, (8), (9), and (10) are all equivalent. (11) and (12) are also equivalent. So the extraneous solution somehow arose in the transition from (10) to (11), by squaring both sides.

So unlike subtracting 5 from both sides or dividing both sides by 2, squaring both sides is not an equivalence-preserving operation. But we tolerate this operation because the implication goes in the direction that matters. If $a = b$, then $a^2 = b^2$, so if $a$ and $b$ are expressions containing a free variable $x$, any value of $x$ that makes $a = b$ true will also make $a^2 = b^2$ true.

In other words, squaring both sides can only enlarge the solution set. So if one is vigilant when squaring both sides to the possible creation of extraneous solutions, and is willing to test solutions to the terminal equation back into the original equation, the process of squaring both sides is innocent and unproblematic.

## Those Who are Still Not Satisfied

Still there are some who are not satisfied with this explanation: “Why does this happen? What is really going on? Where do the extraneous solutions come from? What do they mean?”

One source of the problem is the square root operation itself. $\sqrt{a}$ is, by the conventional definition, the positive quantity which when squared is $a$. The reason that we have to stress the positive quantity is that there are always two real numbers that when squared equal any given positive real number. There are a few slightly different ways of making this same point. The operation of squaring a number erases the evidence of whether that number was positive or negative, so information is lost and we are not able to reverse the squaring process.

We can also phrase the phenomenon in the language of functions. Since squaring is a common and useful mathematical practice, information will often come to us squared and we’ll need an un-squaring process to unpack that information. $f(x) = x^2$, for all the reasons just mentioned, is not a one-to-one function, so strictly speaking, it is not invertible. But un-squaring is too important, so we persevere. As with all non-one-to-one functions, we first restrict the domain of $f(x) = x^2$ to $[0, \infty)$ to make it one-to-one. This inverse, $f^{-1}(x) = \sqrt{x}$ thus has a positive range and so the convention that $\sqrt{a} \geq 0$ is born. So every use of the square root symbol comes with the proviso that we mean the positive root, not the negative root. We inevitably lose track of this information when squaring both sides.

[Note: Students can easily lose track of these conventions. After a lot of practice solving quadratic equations, moving from $x^2 = 9$ effortlessly to $x = \pm 3$, students will often start to report that $\sqrt{9} = \pm 3$.]

The convention that we choose the positive root is totally arbitrary. In a world in which we restricted the domain of  $f(x) = x^2$ to $(-\infty, 0]$ before inverting, $\sqrt{9}$ would be $-3$. In that world, $x = 1$ is a perfectly good solution to $2\sqrt{x+8} + 5 = -1$, not extraneous at all.

## A Trigonometric Equation which Yields an Extraneous Solution

For parallelism, consider the (somewhat artificial) equation:

(13) $\arccos(2x-1) = \frac{4\pi}{3}$

Like in (10), careful and observant solvers might notice that the range of the $\arccos(x)$ function is $[0, \pi]$ and correctly conclude that the equation has no solutions. But there seems to be a lot going on inside that $\arccos$ expression, so many will rush ahead and try to unpack it by “cosineing”. Indeed, since $a=b \Rightarrow \cos(a) = \cos(b)$, this seems innocent.

(14) $2x - 1 = -\frac{1}{2}$

(15) $2x = \frac{1}{2}$

(16) $x = \frac{1}{4}$

But $x = \frac{1}{4}$ is an extraneous solution since $\arccos(-\frac{1}{2}) = \frac{2\pi}{3}$ not $\frac{4\pi}{3}$.

The explanation for this extraneous solution will be similar to the logic we used above. If $a = b$, then $\cos(a) = \cos(b)$, so if $a$ and $b$ are expressions containing a free variable $x$, any value of $x$ that makes $a = b$ true will also make $\cos(a) = \cos(b)$ true. So we will not lose any solutions by “taking the cosine of both sides”. But as the cosine function is not one-to-one, $\cos(a) = \cos(b)$ does not imply that $a = b$. So taking the cosine of both sides, just like squaring both sides, can enlarge the solution set.

The above paragraph explains why extraneous solutions could appear in the solution of (13), but maybe not why they do appear. For that, we again must look to the presence of the $\arccos$ function. Since $\cos$ is not one-to-one, we had to arbitrarily restrict its domain to $[0, \pi]$ prior to inverting. So every use of the $\arccos$ symbol comes with its own proviso that we are referring to a number in a particular interval of values. In a world in which we had restricted the domain of $\cos$ to $[\pi, 2\pi]$ prior to inverting, $x = \frac{1}{4}$ would be a perfectly good solution to $\arccos(2x-1) = \frac{4\pi}{3}$, not extraneous at all.

The above examples seem to suggest that one can avoid dealing with extraneous solutions by carefully examining one’s equations at each step. But in practice, this really isn’t possible. I saved the fun examples for the end, but as this post is already way way too long, they will have to wait for a bit later.

-Will Rose

## Thanks

Thanks to John Chase for letting me guest post on his blog. Thanks to James Key for encouraging me again and again to think about extraneous solutions.

# Proving identities – what’s your philosophy?

What happens in your classroom when you give students the following task?

Prove $1+\frac{1}{\cos{\theta}}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$.

Sometimes the command is Verify or Show instead of Prove, but the intent is the same.

# Two non-examples

Here are two ways that a student might work the problem.

Method 1

$1+\frac{1}{\cos{\theta}}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$1+\sec{\theta}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$(\sec{\theta}-1)(1+\sec{\theta})=\tan^2{\theta}$

$\sec^2{\theta}-1=\tan^2{\theta}$

$\tan^2{\theta}=\tan^2{\theta}$

Method 2

$1+\frac{1}{\cos{\theta}}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$1+\sec{\theta}=\frac{\sec^2{\theta}-1}{\sec{\theta}-1}$

$1+\sec{\theta}=\frac{(\sec{\theta}-1)(\sec{\theta}+1)}{\sec{\theta}-1}$

$\sec{\theta}+1=\sec{\theta}+1$

How do you feel about these methods? In my opinion, both methods represent a fundamental misunderstanding of the prompt. Method 1 is especially grotesque, but Method 2 also leaves a lot to be desired. Let me explain. And if you think the above methods are perfectly fine, please be patient and hear me out.

This is the crux of the issue:

The prompt was to prove the statement. But if the first line of our work is the very thing we’re out to prove, then we are already assuming the thing we want to prove. We’re Begging the Question.

It’s as if someone demands,

“Well, let’s first start by assuming that Statement X is true.”

This is nonsense.

# What went wrong?

So what is the proper way to engage this proof? Let’s roll back a bit.

The error in these approaches seems to stem from a desire to perform algebraic operations on both sides of an equation in the same way that you might if you were solving an equation.

When we “do algebra” and write Equation B below another Equation A without any words, we always mean that Equation A implies Equation B. That is, when we write

Equation A

Equation B

Equation C

etc…

we mean that Equation C follows from Equation B, which follows from Equation A.

Some might claim that each line should be equivalent to the last. But, again, when we “do algebra” by performing algebraic manipulations to both sides of an equation to transform it from equation A into equation B, we always mean $A\Rightarrow B$, we don’t mean $A\iff B$. Take, for example, the following algebra which results in an extraneous solution:

$\sqrt{x+2}=x$

$(\sqrt{x+2})^2=x^2$

$x+2=x^2$

$0=x^2-x-2$

$0=(x-2)(x+1)$

$x=2 \text{ or } x=-1$

In this example, each line follows from the previous, however reversing the logic doesn’t work. But we accept that this is the usual way we do algebra ($A\Rightarrow B\Rightarrow C\Rightarrow \cdots$). Here the last line doesn’t hold because only one solution satisfies the original equation ($x=2$). Remember that our logic is still flawless, though. Our logic just says that IF $\sqrt{x+2}=x$ for a given $x$THEN $(\sqrt{x+2})^2=x^2$.

As we move through the algebra line by line, we either preserve the solution set or increase its size. In the case above, the solution set for the original equation is {2}, and as we go to line 2 and beyond, the solution set is {2,-1}.

For more, James Tanton has a nice article about extraneous solutions and why they arise, which I highly recommend.

So if this is the universal way we interpret algebraic work, which is what I argue, then it is wrong to construct an argument of the form $A\Rightarrow B\Rightarrow C$ in order to prove statement A is true from premise C. The argument begs the question.

Both Method 1 and Method 2 make this mistake.

# How does a proof go again?

I want to actually make a more general statement. The argument I gave above regarding how we “do algebra” is actually how we present any sort of deductive argument. We always present such an argument in order, where later statements are supported by earlier statements.

ANY time we see a sequence of statements (not just equations) A, B, C that is being put forward as a proof, if logical connectives are missing, the mathematical community agrees that “$\Rightarrow$” is the missing logical connection.

That is, if we see the proof A,B,C as a proof of statement C from premise A, we assume that the argument really means $A\Rightarrow B\Rightarrow C$.

This is usually the interpretation in the typical two-column proof, as well. We just provide the next step with a supporting theorem/definition/axiom, but we don’t also go out of our way to say “oh, and line #7 follows from the previous lines.”

Example: Given a non-empty set $E$ with lower bound $a$ and upper bound $b$, show that $a\leq b$.

1. $E$ is non-empty and $a$ and $b$ are lower and upper bounds for $E$. (given)
2. Set $E$ contains at least one element $x$. (definition of non-empty)
3. $a\leq x$ and $x\leq b$. (definitions of lower and upper bound)
4. $a\leq b$. (transitive property of inequality)

Notice I never say that one line follows from the next. And also notice that it would be a mistake to interpret the logical connectives as biconditional.

# The path of righteousness

I encourage my students to work with only ONE side of the expression and manipulate it independently, in its own little dark box, and when it comes out into the light, if it looks the same as the other side, you’ve proved the equivalence of the expressions.

For example, to show that $\log\left(\frac{1}{t-2}\right)-\log\left(\frac{10}{t}\right)=-1+\log\left(\frac{t}{t-2}\right)$ for $t>2$, I would expect this kind of work for “full credit”:

$\text{LHS }=\log\left(\frac{1}{t-2}\right)-\log\left(\frac{10}{t}\right)$

$=-\log(t-2)-\log(10)+\log(t)$

$=-\log(10)+\log(t)-\log(t-2)$

$= -1 + \log\left(\frac{t}{t-2}\right)$

$=\text{ RHS}$

Interestingly, I WOULD also accept an argument of the form $A\iff B\iff C$ as justification for conclusion A from premise C, but I would want a student to say “A is true if and only if B is true, which is true if and only if C is true.” Even though it provides a valid proof, I discourage students from using this somewhat cumbersome construction.

So let’s return to the original problem and show a few ways a student could do it correctly.

# Three examples

Method A – A direct proof by manipulating only one side

$\text{LHS}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$=\frac{\sec^2{\theta}-1}{\sec{\theta}-1}$

$=\frac{(\sec{\theta}-1)(\sec{\theta}+1)}{\sec{\theta}-1}$

$=\sec{\theta}+1$

$=\text{RHS}$

Method B – A proof starting with a known equality

$\tan^2{\theta}=\tan^2{\theta}$

$\sec^2{\theta}-1=\tan^2{\theta}$

$(\sec{\theta}-1)(1+\sec{\theta})=\tan^2{\theta}$

$1+\sec{\theta}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$1+\frac{1}{\cos{\theta}}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

Method C – Carefully specifying biconditional implications

$1+\frac{1}{\cos{\theta}}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$\text{if and only if}$

$1+\sec{\theta}=\frac{\tan^2{\theta}}{\sec{\theta}-1}$

$\text{if and only if}$

$(\sec{\theta}-1)(1+\sec{\theta})=\tan^2{\theta}$

$\text{if and only if}$

$\sec^2{\theta}-1=\tan^2{\theta}$

$\text{if and only if}$

$\tan^2{\theta}=\tan^2{\theta}$

While all of these are now technically correct, I think we all prefer Method A. The other methods are cool too. But please, please, promise me you won’t use Methods 1 or 2 which I presented in my introduction.

# In conclusion

Some might argue that the heavy criticism I’ve leveled against Methods 1 and 2 is nitpicking. But I disagree. This kind of careful reasoning is exactly the business of mathematicians. It’s not good enough to just produce “answers,” our job is to produce good reasoning. Mathematics, remember, is a sense-making discipline.

Thanks for staying with me to the end of this long-winded post. Can you tell I’ve had this conversation with a lot of students over the last ten years?

1. Dave Richeson has a similar rant with a similar thesis here.
2. This article was originally inspired by this recent post on Patrick Honner’s blog. A bunch of us fought about this topic in the comments, and in the end, Patrick encouraged me to write my own post on the subject. So here I am. Thanks for pushing me in the right direction, Mr. Honner!

# What does it mean to truly prove something?

Let me point you to the following recent blog post from Prof Keith Devlin, entitled “What is a proof, really?”

After a lifetime in professional mathematics, during which I have read a lot of proofs, created some of my own, assisted others in creating theirs, and reviewed a fair number for research journals, the one thing I am sure of is that the definition of proof you will find in a book on mathematical logic or see on the board in a college level introductory pure mathematics class doesn’t come close to the reality.

For sure, I have never in my life seen a proof that truly fits the standard definition. Nor has anyone else.

The usual maneuver by which mathematicians leverage that formal notion to capture the arguments they, and all their colleagues, regard as proofs is to say a proof is a finite sequence of assertions that could be filled in to become one of those formal structures.

It’s not a bad approach if the goal is to give someone a general idea of what a proof is. The trouble is, no one has ever carried out that filling-in process. It’s purely hypothetical. How then can anyone know that the purported proof in front of them really is a proof?

(more)

I won’t be shy in saying that I disagree with Keith Devlin. Maybe I misunderstand the subtle nuance of his argument. Maybe I haven’t done enough advanced mathematics. Please help me understand.

Devlin says that proofs created by the mathematical community (on the blackboard, and in journals) are informal and non-rigorous. I think we all agree with him on this point.

But the main point of his article seems to be that these proofs are non-rigorous and can never be made rigorous. That is, he’s suggesting that there could be holes in the logic of even the most vetted & time-tested proofs. He says that these proofs need to be filled in at a granular level, from first principles. Devlin writes, “no one has ever carried out that filling-in process.”

The trouble is, there is a whole mathematical community devoted to this filling-in process. Many high-level results have been rigorously proven going all the way back to first principles. That’s the entire goal of the metamath project. If you haven’t ever stumbled on this site, it will blow your mind. Click on the previous link, but don’t get too lost. Come back and read the rest of my post!

I’ve reread his blog post multiple times, and the articles he linked to. And I just can’t figure out what he could possibly mean by this. It sounds like Devlin thoroughly understands what the metamath project is all about, and he’s very familiar with proof-checking and mathematical logic. So he definitely isn’t writing his post out of ignorance–he’s a smart guy! Again, I ask, can anyone help me understand?

I know that a statement is only proven true relative to the axioms of the formal system. If you change your axioms, different results arise (like changing Euclid’s Fifth Postulate or removing the Axiom of Choice). And I’ve read enough about Gödel to understand the limits of formal systems. As mathematicians, we choose to make our formal systems consistent at the expense of completeness.

Is Devlin referring to one of these things?

I don’t usually make posts that are so confrontational. My apologies! I didn’t really want to post this to my blog. I would have much rather had this conversation in the comments section of Devlin’s blog. I posted two comments but neither one was approved. I gather that many other comments were censored as well.

Here’s the comment I left on his blog, which still hasn’t shown up. (I also left one small comment saying something similar.)

Prof. Devlin,

You said you got a number of comments like Steven’s. Can you approve those comments for public viewing? (one of those comments was mine!)

I think Steven’s comment has less to do with computer *generated* proofs as it does with computer *checked* proofs, like those produced by the http://us.metamath.org/ community.

There’s a big difference between the proof of the Four Color Theorem, which doesn’t really pass our “elegance” test, and the proof of $e^{i\pi}=-1$ which can be found here: http://us.metamath.org/mpegif/efipi.html

A proof like the one I just linked to is done by humans, but is so rigorous that it can be *checked* by a computer. For me, it satisfies both my hunger for truth AND my hunger to understand *why* the statement is true.

I don’t understand how the metamath project doesn’t meet your criteria for the filling in process. I’ll quote you again, “The trouble is, no one has ever carried out that filling-in process. It’s purely hypothetical. How then can anyone know that the purported proof in front of them really is a proof?”

What is the metamath project, if not the “filling in” process?

John

If anyone wants to continue this conversation here at my blog, uncensored, please feel free to contribute below :-). Maybe Keith Devlin will even stop by!

# Math on Quora

I may not have been very active on my blog recently (sorry for the three-month hiatus), but it’s not because I haven’t been actively doing math. And in fact, I’ve also found other outlets to share about math.

Have you used Quora yet?

Quora, at least in principle, is a grown-up version of yahoo answers. It’s like stackoverflow, but more philosophical and less technical. You’ll (usually) find thoughtful questions and thoughtful answers. Like most question-answer sites, you can ‘up-vote’ an answer, so the best answers generally appear at the top of the feed.

The best part about Quora is that it somehow attracts really high quality respondents, including: Ashton Kutcher, Jimmy Wales, Jermey Lin, and even Barack Obama. Many other mayors, famous athletes, CEOs, and the like, seem to darken the halls of Quora. For a list of famous folks on Quora, check out this Quora question (how meta!).

Also contributing quality answers is none other than me. It’s still a new space for me, but I’ve made my foray into Quora in a few small ways. Check out the following questions for which I’ve contributed answers, and give me some up-votes, or start a comment battle with me or something :-).

And here are a few posts where my comments appear:

# A TOK Lecture on Mathematical Thinking

Students in our International Baccalaureate program here at RM are required to take a core class called Theory of Knowledge (TOK) which is kind of a philosophy class for high school students–or, at least the epistemology piece.

In some schools, this course is taught by math teachers. Here at RM, no math teachers currently teach TOK, which is too bad. So I volunteered to put together a guest lecture on Mathematical Thinking. I’ve tried it out once with a TOK class and I gave the lecture for some of my math teacher colleagues today after school. I plan to give the lecture to more TOK classes this spring.

I thought I’d share it with the MTBoS as well, so here it is. Feel free to read, comment on, or borrow my materials. I think other IB math teachers would especially benefit:

# Why Calculus still belongs at the top

AP Calculus is often seen as the pinnacle of the high school mathematics curriculum*–or the “summit” of the mountain as Professor Arthur Benjamin calls it. Benjamin gave a compelling TED talk in 2009 making the case that this is the wrong summit and the correct summit should be AP Statistics. The talk is less than 3 minutes, so if you haven’t yet seen it, I encourage you to check it out here and my first blog post about it here.

I love Arthur Benjamin and he makes a lot of good points, but I’d like to supply some counter-points in this post, which I’ve titled “Why Calculus still belongs at the top.”

Full disclosure: I teach AP Calculus and I’ve never taught AP Statistics. However I DO know and love statistics–I just took a grad class in Stat and thoroughly enjoyed it. But I wouldn’t want to teach it to high school students. Here’s why: For high school students, non-Calculus based Statistics seems more like magic than mathematics.

When I teach math I try, to the extent that it’s possible, to never provide unjustified statements or unproven claims. (Of course this is not always possible, but I try.) For example, in my Algebra 2 class I derive the quadratic formula. In my Precalculus class, I derive all the trig identities we ask the students to know. And in my Calculus class, I “derive” the various rules for differentiation or integration. I often tell the students that copying down the proof is completely optional and the proof will not be tested–“just sit back and relax and enjoy the show!”

But such an approach to mathematical thinking can rarely be applied in a high school Statistics course because statistics rests SO heavily on calculus and so the ‘proofs’ are inaccessible. I’d like to make a startling claim: I claim that 99.99% of AP Statistics students and 99% of AP Statistics teachers cannot even give the function-rule for the normal distribution.

Image used by permission from Interactive Mathematics. Click the image to go there and learn all about the normal distribution!

In what other math class would you talk about a function ALL YEAR and never give its rule? The normal distribution is the centerpiece (literally!) of the Statistics curriculum. And yet we never even tell them its equation nor where it comes from. That should be some kind of mathematical crime. We might as well call the normal distribution the “magic curve.”

Furthermore, a kid can go through all of AP Statistics and never think about integration, even though that’s what their doing every single time they look up values in those stat tables in the back of the book.

I agree that statistics is more applicable to the ‘real world’ of most of these kids’ lives, and on that point, I agree with Arthur Benjamin. But I would argue that application is not the most important reason we teach mathematics. The most important thing we teach kids is mathematical thinking.

The same thing is true of every other high school subject area. Will most students ever need to know particular historical facts? No. We aim to train them in historical thinking. What about balancing an equation in Chemistry? Or dissecting a frog? They’ll likely never do that again, but they’re getting a taste of what scientists do and how they think. In general, two of our aims as secondary educators are to (1) provide a liberal education for students so they can engage in intelligent conversations with all people in all subject areas in the adult world and (2) to open doors for a future career in a more narrow field of study.

So where does statistics fit into all of this? I think it’s still worth teaching, of course. It’s very important and has real world meaning. But the value I find in teaching statistics feels VERY different than the value I find in teaching every other math class. Like I said before, it feels a bit more like magic than mathematics.**

I argue that Calculus does a better job of training students to think mathematically.

But maybe that’s just how I feel. Maybe we can get Art Benjamin to stop by and weigh in!

.

….

*In our school, and in many other schools, we actually have many more class options beyond Calculus for those students who take Calculus in their Sophomore or Junior year and want to be exposed to even more math.

** Many parts of basic Probability and Statistics can be taught with explanations and proof, namely the discrete portions–and this should be done. But working with continuous distributions can only be justified using Calculus.


text:
# Induced voltages and step-up transformers

1. ### kitkat2950

7
I have been trying, but I still have 3 problems I can't get.

1)A simple generator has a 600 loop square coil 18.0 cm. on a side. How fast (in rev/s.) must it turn in a 0.730 T. field to produce a 120-V. peak output?

I think I need to use the equation Emax=NBAw. N=600 B=.730 A=.0324 Emax=120
I found w to be 8.456 but I didn't know if it was in rev/min or rev/s. So I tried that answer and then tried dividing it by 60 for seconds, but neither answer worked. Any help?

2)A step-up transformer increases 50 V. to 110 V. What is the current in amperes in the secondary as compared to the primary? Assume 100 percent efficiency.

I know the primary voltage is 50V and the secondary voltage is 110V. And I think Vp/Vs=Ip/Is, but I'm not sure how to find the answer w/o one of the currents.

3)A transformer has 32 turns in the primary and 181 turns in the secondary. Assuming 100 percent efficiency, by what factor does it change the voltage?

I think this one is basically the same as number 2, and I just don't understand what I am supposed to do. I think I'm supposed to divide one by the other, but when i put those answers in, they're not right.

2. ### alphysicist

2,248
Hi kitkat2950,

The $\omega$ is angular frequency, which is not in rev/min or rev/s; what are the units of angular frequency?

Once you find $\omega$, you'll have to convert it to rev/s.

It's strange that the ask for the current in the secondary in amperes. When they say "as compared to the primary" that suggests that what they are looking for is the ratio of the currrents. Was there a diagram or anything else in the problem?

Your equation is not correct. If the transformer is 100% efficient, what is not lost by the transformer?

What is the relationship between voltage and turns in a transformer? (Without knowing what numbers you used and got, it's difficult to determine what you might have done wrong.)

Last edited: Jul 21, 2008
3. ### annythewitch

2
2. the ratio you want is:

Vs/Vv=Ns/Np: Is/Ip=Np/Ns >>> Vs/Vp=Ip/Is >>> Is/Ip=Vp/Vs = 50/100= .5

4. ### alphysicist

2,248
No, I don't believe that's the answer. You seem to have put in a wrong number.

Also, you have put in a few extra steps. There is 100% efficiency, so the power is the same on both sides, so you can immediately write down:

$$I_p V_p = I_s V_s$$

and get the ratio in the next step.

5. ### annythewitch

2
you are right. I mistyped the fraction. it should have been 50/110, and yeah, your way is easier.

Last edited: Nov 8, 2008
Know someone interested in this topic? Share a link to this question via email, Google+, Twitter, or Facebook

Have something to add?

format: plain


text:
## Local distributed rate limiting

Suppose our job is to design a rate limiting system. It could be embedded in a service or be a service of its own. Given some details about a request from which we build a source identifier (an IP address is popular and we may assume that in what follows) our system should check how many requests have been made from that identifier in some interval of time and allow or deny the request based on that information.

I’ve seen a handful of such systems implemented at work. More precisely, I’ve read outage reports on a handful of such systems. A common theme in their design seems to be that the system keeps the request count per identifier in some central database, like Cassandra or MySQL. I can only assume this is done to ensure the system is in some way fair or correct, where those terms are taken to mean that as far as we know the rate limits we impose are global across all requests to a given availability zone or data center or unit of compute area.

I am but a simple country developer, and am probably missing something obvious, but it has never been clear to me why we care about fairness or correctness in these systems. Those come at the price of coordination, which is evil and we hate it. But we should also be able to do without it. Each node in our system can keep its own count of requests per identifier, and make its decisions based on that local information only.

There are a couple of reasons we can do this. The first is that requests are presumably load-balanced across our nodes. That load balancing is either done in as uniform a way as possible, or in a "sticky" way where each request identifier gets sent to the same node (or small group of nodes, then again as uniformly as possible). In either case each node sees either zero requests per identifier or a representative sample of them. So having the nodes make local decisions about identifiers will in aggregate reflect the global behavior of the system.

The second is that source behavior is usually binary. A given source is either well-behaved at a given moment, or completely batshit insane. Either behavior is perfectly visible at the local level and needs no coordination to figure out. If we need correctness for later accounting or observability, the nodes can report their local counts to a central authority on their own time.

So the next time you need distributed rate limiting, think locally. Reach for an in-memory SQLite database, or something horrifying like a ring buffer you roll yourself and definitely test no part of before shipping. Have fun with it. But don’t coordinate when you don’t have to.

## Estimating the norm of the exterior product

Let $$V$$ be a real finite-dimensional vector space, and let $$\bigwedge^\bullet V$$ be the exterior algebra of $$V$$. An inner product on $$V$$ induces inner products on each component $$\bigwedge^k V$$ of the exterior algebra. Recall that we have the exterior product $\bigwedge{}^{p} V \times \bigwedge{}^{q} V \to \bigwedge{}^{p+q} V, \quad (u, v) \mapsto u \wedge v.$ Since all the spaces here are finite-dimensional, there exist constants $$C = C(p,q,n)$$ such that $|u \wedge v| \leq C |u| \, |v|$ for all $$u \in \bigwedge{}^ p V$$ and $$v \in \bigwedge{}^ q V$$. Can we estimate these constants?

The answer apparently has applications to physics. I don’t understand the connection to physics or the physical motivation, but in a 1961 paper the physicist Yang Chen-Ning conjectured that for even-dimensional spaces the optimal bound is achieved on powers of the standard symplectic form. As

$\frac{\omega^p}{p!} \wedge \frac{\omega^q}{q!} = \binom{p+q}p \frac{\omega^{p+q}}{(p+q)!},$ where $$\omega = \sum_{i=1}^n dx_{2i-1} \wedge dx_{2i}$$ and $$|\omega^p / p!|^2 = \binom np$$, the conjecture is that $C(2p,2q,2n) = \tbinom{p+q}p\sqrt{\frac{\binom{n}{p+q}}{\binom np \binom nq}}.$ Yang claims to prove this for the case of $$p = 1$$ in his 1961 paper, though I don’t understand his proof. A preprint from 2014 claims to prove more cases of the conjecture; again the details of the proof defeat me.

There are two ways of getting rather brutal estimates for these constants. Let $$(v_1, \ldots, v_n)$$ be an orthonormal basis of $$V$$. Then $$(v_I)$$ where $$I \subset \{1, \ldots, n\}$$ with $$|I| = p$$ is an orthonormal basis of $$\bigwedge^p V$$, and similar for $$(v_J)$$ with $$|J| = q$$. If $$u$$ and $$v$$ are elements of these bases, then $$u \wedge v$$ is either 0 or has norm 1.

For our first attempt, let $$x = \sum_{|I|=p} x_I v_I$$ and $$y = \sum_{|J|=q} y_J v_J$$ be elements of $$\bigwedge^p V$$ and $$\bigwedge^q V$$. Then $|x \wedge y| = \biggl| \sum_{|I|=p, |J|=q} x_I y_J v_I \wedge v_J \biggr| \leq \sum_{|I|=p, |J|=q} |x_I y_J| = \sum_{|I|=p} |x_I| \cdot \sum_{|J|=q} |y_J|.$ By taking the inner product of $$x$$ and the vector whose elements are the signs of each $$x_I$$ and applying Cauchy—Schwarz, we get $\sum_{|I|=p} |x_I| \leq \sqrt{\tbinom np} |x|.$ This gives $|x \wedge y| \leq \sqrt{\tbinom np \tbinom nq} |x| |y|.$ Our first estimate is thus $$C(p,q,n) \leq \sqrt{\binom np \binom nq}$$. This is likely to be a bad estimate, because when applying the triangle inequality above we assigned equal weight to all expressions $$|v_I \wedge v_J|$$, even though many of them were zero. We also applied the Cauchy—Schwarz inequality on top of the triangle one. The conditions for those inequalities to be exact are orthogonal — the triangle one is exact when all the vectors lie on a straight line, while Cauchy—Schwarz is exact when all the vectors are orthogonal — which should yield suboptimal bounds.

For our second attempt, we consider the bilinear map $b : \bigwedge{}^ p V \otimes \bigwedge{}^ q V \to \bigwedge{}^ {p+q} V, \quad u \otimes v \mapsto u \wedge v.$ Its operator norms satisfy $$\|b\|^2 \leq |b|^2$$, where $\| b \|^2 = \sup_{x,y} \frac{ |b(x,y)|^2 }{ |x|^2 |y|^2 }$ and where $$|b|$$ is the Hilbert—Schmidt norm. We can calculate that norm by counting the number of non-zero entries in the matrix for $$b$$.

Let then $$I = (i_1, \ldots, i_p)$$ be a multiindex. We can choose $$\binom{n-p}{q}$$ indices $$J = (j_1, \ldots, j_q)$$ such that the corresponding basis elements $$v_I$$ and $$v_J$$ satisfy $$b(v_I \otimes v_J) = v_I \wedge v_J \not= 0$$. As there are $$\binom np$$ ways of picking $$I$$, we conclude that the Hilbert—Schmidt norm of $$b$$ is $|b|^2 = \tbinom np \tbinom {n-p}q = \tbinom n{n-p} \tbinom {n-p}q.$ We then get a slightly better estimate $C(p,q,n) \leq \sqrt{\tbinom n{n-p} \tbinom {n-p}q} \leq \sqrt{\tbinom np \tbinom nq}$ with equality in the second place if and only if $$p = 0$$ or $$p = n$$.

A fun fact is that these estimates are arbitrarily bad. When $$p + q = n$$, the Hodge star operator gives an isometry $$\bigwedge^q V \cong \bigwedge^p V$$ and the Cauchy—Schwarz inequality gives $|u \wedge v| \leq |u| |v|,$ so $$C(p,n-p,n) = 1$$, which agrees with Yang’s conjectured value when $$p$$ and $$n$$ are even. Our best estimate in this case is $C(2p,2n-2p,2n) \leq \sqrt{\tbinom {2n-2p}{2p}},$ which goes to infinity as $$n$$ grows.

I think this is a very fun instance of a problem that’s easy to state and whose solution is somewhat difficult. The simple attempts at a solution run into basic problems with the optimal conditions for different inequalities on the one hand, and the fact that the things we can easily calculate are not the things we’re interested in on the other hand.

## Emacs is my favorite Emacs package

In the category of "people whose work ethic and output I can only envy from afar while failing to do anything of value", Protesilaos Stavrou takes the lead with his video Emacs is my "favourite Emacs package" in which he demonstrates how bananas well he has come to understand Emacs in the single year (!!!) he has used it.

## Anscombe's quartet and monitoring

Anscombe’s quartet is a collection of four data sets of points in the plane. When graphed the data sets are obviously very different, but they have identical sets of descriptive statistics. Newer variations on the same theme include the delightful Datasaurus dozen, which includes animations of deformations from one step into another where the descriptive statistics are kept identical during the animation.

I’ve seen Anscombe’s quartet cited in discussions around monitoring and observability to assert that the average and standard deviation (two of the statistical tools used in the quartet) are not useful for monitoring and alerting; instead one should use percentiles. I agree with the conclusion but think it does not follow from the premise, and would like to offer some thoughts on the theme from the perspective of a reformed geometer.

To do so, I’d like to work out some examples involving the average and standard deviation. The data sets above involve collections of points in the plane but I’d like us to discuss data modeled on timeseries instead. The difference isn’t important, everything we say applies to both theaters, but the geometry we want to point out is easier to see in the latter.

## Setup

We’ll model a portion of a timeseries as a tuple of $$n$$ real numbers $$(x_1, \ldots, x_n)$$. We can think of these as $$n$$ measurements recorded at one-second (or minute, hour, …) intervals. The set of all possible such timeseries is familiar; it is the real vector space $$\mathbb{R}^n$$ of dimension $$n$$. (Timeseries can be added and subtracted, there is a zero-timeseries that serves as an identity, and we can multiply a timeseries by a real number.)

The main characters of our story are the average and standard deviation. If $$x = (x_1, \ldots, x_n)$$ is a timeseries, its average is $\mu(x) = (x_1 + \cdots + x_n) / n$ and its standard deviation is defined by the scary formula[1] $\sigma(x) = \sqrt{ ( (x_1 - \mu(x) )^2 + \cdots + (x_n - \mu(x) )^2 ) / n }.$

We can now look at some examples. In every one of these, we’ll fix the average and standard deviation to some values and look at the collection of timeseries that have those values. The first couple of examples are just warm-up.

## n = 1

We first look at $$n = 1$$, that is, timeseries that just have one value $$x = (x_1)$$. For such a timeseries we have $$\mu(x) = x_1$$ and $$\sigma(x) = 0$$ so each timeseries is uniquely determined by its average and they all have the same standard deviation.

## n = 2

The next case is $$n = 2$$, of timeseries with two values $$x = (x_1, x_2)$$. This is slightly more interesting; we have the average $$\mu(x) = (x_1 + x_2) / 2$$ and the standard deviation \begin{align} \sigma(x) &= \sqrt{( (x_1 - \mu(x) )^2 + (x_2 - \mu(x) )^2 )/2} \\ &= \sqrt{((x_1 - x_2)^2 + (x_2 - x_1)^2)/8} = |x_1 - x_2|/2. \end{align} The second equality here is by substituting the definition of $$\mu(x)$$.

This is more interesting because different timeseries can have different averages and standard deviations. However, I claim that if we fix both the values of the average and standard deviations, say to $$\mu$$ and $$\sigma$$, there is at most one timeseries that has those values.

To see this, observe that the formula for the average implies that $$x_2 = 2 \mu - x_1$$, so if we fix the value of the average the value of $$x_2$$ is determined by the choice of $$x_1$$. If we substitute this into the formula for the standard deviation we get $\sigma(x) = |x_1 - 2 \mu + x_1|/2 = |x_1 - \mu|.$ We can solve this equation by splitting into cases where $$x_1 > \mu$$ and $$x_1 < \mu$$. In the former we get $$x_1 = \sigma + \mu$$ and in the latter we get $$x_1 = \mu - \sigma$$. Notice that in both cases, the value of $$x_1$$ is determined by the values of the average and standard deviations, and the value of $$x_2$$ is determined by that of $$x_1$$ and the average, so both $$x_1$$ and $$x_2$$ are determined once we fix the others.

## n = 3

We now get into the first case that has a hint of the general picture. Our timeseries have three points, $$x = (x_1, x_2, x_3)$$. We could write down equations like in the previous case and calculate, but it’s worth it to take a step back and think about what’s going on.

Let’s say we have fixed the value of the average of our timeseries to a constant $$\mu$$. The formula for the standard deviation of our series is then $\sigma(x) = \sqrt{((x_1 - \mu)^2 + (x_2 - \mu)^2 + (x_3 - \mu)^2)/2}.$ If we form the point $$(\mu, \mu, \mu)$$, we see that $\sigma(x) = \|(x_1, x_2, x_3) - (\mu, \mu, \mu)\|/\sqrt{2}$ is just a multiple of the Euclidean distance between our timeseries and the fixed point $$(\mu, \mu, \mu)$$. If we also fix $$\sigma(x) = \sigma$$, the set of points that satisfy this equation is $S := \{ (x_1, x_2, x_3) \mid \|(x_1, x_2, x_3) - (\mu, \mu, \mu)\| = \sqrt{2} \sigma \}.$ This should look familiar, it is the sphere whose center is $$(\mu, \mu, \mu)$$ and whose radius is $$\sqrt{2}\sigma$$.

Similarly, the set of timeseries whose average is equal to $$\mu$$ is $P := \{ (x_1, x_2, x_3) \mid (x_1 + x_2 + x_3)/2 = \mu \}.$ For those who know a little linear algebra, we can write the condition as being that $$(x_1, x_2, x_3) \cdot (1, 1, 1) = 2 \mu$$, that is, that the inner product with $$(1, 1, 1)$$ must be equal to $$2 \mu$$. The set of points that satisfies such a condition is a plane; a flat two-dimensional space.

We’re interested in the set of points where both of these conditions are true, which is the intersection $$S \cap P$$. We could work out what that is by calculation, but it’s easiest to visualize it. The intersection of a sphere and a plane in three dimensions is either empty (when they don’t meet), a single point (when the plane is tangent to the sphere), or a circle that lies on the sphere.

In three dimensions, if we pick our values for the average and standard deviations right, there is thus a whole circle of timeseries that have that average and standard deviation. They form a kind of Anscombe set.

## Higher dimensions

In higher dimensions, basically the same thing happens as in dimension 3. The set of timeseries that have a fixed average is a hyperplane and the set of timeseries that have a fixed standard deviation and average is a sphere. Their intersection, when it is not empty or a single point, is not as easily described but it is a subspace of dimension $$n-2$$. As $$n$$ grows higher, this space grows as well, and in general the space of timeseries that have a fixed average and standard deviation is enormous.

## Other prescriptive statistics

The final point I’d like to make, and one you’ll have to take my word for to some extent, is that there is nothing special about the average or standard deviation here. Very similar things will happen for every collection of statistical tools we pick.

Every statistical tool can be viewed as a function that takes a timeseries (or perhaps a couple of timeseries, which complicates things a little but not a lot) and returns a real number. These functions are very well behaved (continuous, differentiable, and so on).

Given a collection of such functions $$f_1, \ldots, f_k$$ on the space of timeseries of $$n$$ points, we can ask whether there is an Anscombe set of fixed values of these functions? That is, if we fix values $$y_1, \ldots, y_k$$, can we say anything about the set $$\{x = (x_1, \ldots, x_n) \mid f_1(x) = y_1, \ldots, f_k(x) = y_k \}$$?

With some hand waving, we can actually say this: Fixing the value of a single tool $$f_j$$ creates a hypersurface $$X_j = \{x \mid f_j(x) = y_j\}$$ inside the space of timeseries, that is, a subspace of dimension one less than the ambient space. All timeseries in $$X_j$$ will have the same value with respect to this tool. If the intersection of all of these $$X_1 \cap \cdots \cap X_k$$ is not empty, it will generally be a subspace of dimension $$n - k$$. If $$n$$ is much bigger than $$k$$, the dimension of this space, the space of timeseries that all have the same statistical values with respect to our tools, is enormous.

Making the hand waving precise and figuring out when this intersection will be non-empty is the domain of algebraic geometry over the real numbers. Saying anything at all about that is beyond our scope here.

## Outro

I hope I have convinced you that the statement "the average and standard deviation are useless for monitoring because of Anscombe’s quartet" is not true. The moral of Anscombe’s quartet, which boils down to intersection theory in algebraic geometry, is that the space of data that fits any given collection of statistical instruments (including any collection of percentiles) is gigantic and we have to look at the data to make sense of it.

In general, percentiles are more valuable for analyzing performance or traffic data than the average and standard deviation. The reason is not Anscombe’s quartet, the robustness of percentiles, or anything to do with normal distributions, but that the average and standard deviation are fundamentally tools for answering the questions "what would this be if it were constant?" and "how far away is this from being constant?".[2] For performance and traffic analysis, we don’t care about either of those questions, but about peaks in demand and how we can handle them. We should pick tools that can help with that, but other tools have their place.

1. Some people divide by $$n-1$$ in the formula for the standard deviation. I’m sure they have their reasons, but the factor of $$n$$ is more natural for geometers. The reason involves interpreting timeseries as functions and is a bit of a detour from what we really care about.
2. Exactly how involves some linear algebra and this post was already long. If people care, I can write another one explaining how.

## How Silicon Valley will solve the trolley problem

The trolley problem asks how to decide between the lives of people in two groups. At the moment, it comes up in our industry in discussions around self-driving cars: Suppose a car gets into a situation where it must risk injuring either its passengers or pedestrians; which ones should it prioritize saving?

Always choosing one group leads to suboptimal outcomes. If we always save the pedestrians, they may perform attacks on car riders by willfully stepping into traffic. If we always save the passengers, we may run over a pedestrian whose life happens to be of greater worth than that of the passengers.

The crux of the trolley problem is how we should make the choice of what lives to save. As with all hard problems without clear metrics for success, it’s best to solve this by having the participants decide this for themselves.

Every time we view a website, our friendly ad networks must decide what ads we should see. This is done by having our user profile put up on auction for prospective advertisers. During a handful of milliseconds, participants may inspect our profile and place a bid for our attention according to what they see.

The same technology could be trivially repurposed for deciding the trolley problem in the context of self-driving cars.

Assume the identity of every participant in the trolley scenario is known. Practically, we know the identity of the passengers; that of the pedestrians could be known if their phones broadcast a special short-range identification signal. An incentive for broadcasting such a signal could be that we would have to assume that a person without one were one of no means.

Given this information, a car about to be involved in a collision could take a few milliseconds to send the identities of the people involved to an auction service. Participants who had the foresight of purchasing accident insurance would have agents bidding on their behalf. The winner of the bid would be safe from the forthcoming accident, and their insurance would pay some sum to the car manufacturer post-hoc as a reward for conforming to the rules.

The greater a person’s individual wealth, the better insurance coverage and latencies they could purchase, and the more accidents they could expect to survive. This aligns nicely with the values of societies like the United States, where the worth of a person’s life is proportional to their wealth.

As this system lets self-driving car manufacturers off the hook for any decisions taken, and would need coordinated ethical action on the behalf of software engineers to not be implemented, we expect to see this system in action in the world in short order.

## Infosec: A board game

I’m pleased to announce the release of my new card-drawing game Infosec.

The rules of the game are simple. It is for two or more players. The player with the fewest friends is the Infosec Expert; the other players are the Coworkers.

To start, the Infosec Expert deals three cards face down from a standard deck of cards. The Coworker on the Infosec Expert’s right hand should draw one card.

"Which card?" the Coworker may ask.

"This is a simple phishing exercise," the Infosec Expert should reply. "Just pick a card."

"But they all look the same," the Coworker may object.

"Draw one. And get it right."

This exchange should go on in increasingly hostile tones until the Coworker agrees to draw a card. It will have been the wrong card. The Infosec Expert should inform the Coworker:

"You got phished. You moron. You fucking idiot. You’re such a goddamn waste of space and time. How could you have gotten this so wrong? Were you even trying? Answer me. What the fuck was that?"

Feel free to ad-lib along these lines, or draw as many cards as you want from the accompanying Admonishment Deck (expansion packs available). Include ad-hominem attacks and use as many personal details as you know. Interrupt the Coworker if they try to reply. Don’t hold back.

Once the Coworker is silent, the Infosec Expert should collect the cards, deal new ones, and proceed to the next Coworker in line.

The game ends with the Infosec Expert’s victory when all the Coworkers have left.

## Backing up data like the adult I supposedly am

Like so many things I’m supposed to do but don’t — getting exercise, eating right, sleeping well, standing up for women and minorities in public spaces — backing up my data has always been something I’ve half-assed at best.

I’ve lugged around an external hard drive with a few hundred gigabytes of data for the last 10 years, and made backups to it once every three or four years or so. Every time I’ve tried restoring anything from those backups I’ve regretted it, because of course I just bought the drive, plugged it in and copied stuff to it, so it is a FAT32 drive while I have mostly had EXT4 filesystems, which means all my file permissions get lost during the process.

I’ve written shameful little shell scripts to set file permissions to 0644 and directory permissions to 0755, recursively, many many times.

Part of my problem was that I both know just enough rsync to be dangerous and have a credit card so I can provision cloud VMs, so forever just around the corner was my perfect backup solution that I’d write myself and maintain and actually do instead of dealing with whatever I had going on in my life. I’ve come to accept that this will never happen, or perhaps more definitively, that I’d rather cut myself than write and maintain another piece of ad-hoc software for myself.

Luckily I recently found two things that have solved this whole problem for me: borg and rsync.net.

Borg is backup software. It compresses and deduplicates data at the block level, and strongly encourages (but does not force) you to encrypt data before backing it up. It is everything I’d want from my half-assed rsync and shell script abomination.

I read its documentation a couple of times and was impressed. I then set about comparing different VM hosts to see which one would give me the cheapest block storage option, when the result of some random google search led me to rsync.net. They are a company that stores backups, pretty cheaply, and even more cheaply if you use borg to take them. I guess they just really love borg and want us to love it too.

I signed up for their cheapest plan, which starts at 100GB stored for $18 per year. They have no network in- or egress costs, and the storage amount can be adjusted at any time. Once my account had been activated, I did a little password reset dance, and uploaded a public SSH key. I wanted to back up my $HOME directory, so after installing borg I ran:

export BORG_REMOTE_PATH="borg1"
borg init --encryption repokey-blake2 UID@ch-s011.rsync.net:home

This created a remote borg repository called "home" on rsync.net’s servers. The environment variable is so we use a more recent version of borg on the remote server (version 1.1.11 at the time of writing), as the default version is rather old (version 0.29.0).

When choosing what encryption method to use, one can choose between a "repokey" or a "keyfile". They both create a private key locked with a passphrase; the difference is that with "repokey" the key is stored in the borg repo, while with "keyfile" it is stored outside of it. This boils down to whether we think a passphrase is enough security for our data, or whether we think having a secret keyfile is necessary. I figured my password manager could create a strong enough passphrase for my needs, and I didn’t want to think about losing the keyfile, so I chose "repokey-blake2".

To create my first backup, I ran

borg create --exclude "$HOME/.cache" UID@ch-s011.rsync.net:home::backup-1 "$HOME"

which created the archive "backup-1" in my "home" borg repository. I didn’t change the compression algorithm from the default one.

By default borg compresses data with lz4. It can use other compression methods (xz, zlib, zstd). I compared their compression ratios on some binary files I had and found no difference between them. I think this is because the large binary files I have are mostly audio and video files in lossy formats, which don’t seem to benefit very much from further compression. I have a lot of text files as well, but text takes up so little relative space on today’s hardware that it makes no sense to spend CPU cycles on compressing it better than lz4 does.

This backup command hummed along for a good while, and through a couple of reboot cycles. Doing a second backup right after it finished (or the day after) took a lot less time because of the deduplication:

borg create --exclude "$HOME/.cache" UID@ch-s011.rsync.net:home::backup-2 "$HOME"

Restoring from backup is also easy:

borg extract UID@ch-s011.rsync.net:home::backup-2

I set this up to run as a daily timed systemd service at noon (very easy on NixOS, which every Linux user should be using unless they hate themselves), and will never, ever think about this again. For a handful of bucks a year, that is a good deal.

## Review of home manager

A common theme among the people who fall in love with Nix (or its cousin NixOS) is that they want it to manage everything for them. Finding out the limits of that capability and where it breaks down is part of each’s journey.

An obvious enough limit to Nix’s reach is secret management. It can easily handle public keys, SSH or cryptographic ones, but the private keys cannot be managed by its store as it is readable by all.

The promise of Nix is that we will never break our system by updating it. We can always roll back to a previous working version, once we have a working version at all. This is in contrast to, say, every other Linux distribution. I have borked Ubuntu, Arch, Fedora and others during system updates. Often the safest way to update the system is to back up /home and reinstall the OS.

From somewhere comes the idea that a user on a Linux system should be able to install their own programs and manage their own system services. (See Chris Wellons for where you can run with this idea to if your sysadmin gives you a C compiler.) This seems odd from a historical perspective. In a true multi-user system, the system administrators normally do not want users to be able to install arbitrary software or run services. On a modern "multi"-user system, where there is a only single user, avoiding system packages and services seems like some kind of theater.

Yet we do it anyway. An argument I sometimes make to myself is that this is a cleaner separation between what I need the system to do versus what I do on it. I may want to be able to run traceroute as root, but I don’t care about root being able to run the Go compiler.

NixOS has facilities to enforce this separation. It happily creates users and their home directories, and can fill in some of the bits that go there, like public SSH keys. It can install packages for specific users, and allow them to define systemd services. It will not manage the configuration of specific user packages (like dotfiles) without some coercing. One can presumably create custom derivations of, say, ZSH with all the configuration one wants, but who has the time?

Home manager wants to fill this gap and bring the power of Nix to user environment and configuration management. It lets individual users say what packages they want installed; what services they want run; and what configuration files should go where. On the surface it seems like something I should love, but after using it for a month I wrote it out of my system and now use plain NixOS.

I used Home manager for three things:

1. Installing packages for my user.

2. Scheduling services for my user.

3. Installing configuration files for my software.

The first one was never a big attraction. Home manager lets us define a list of packages in home.packages of packages to install. If we control the system configuration, we can acheive the same by defining those packages in users.users.\$username.packages.

If a user controls the system configuration (directly or through a sysadmin), they can also define their own user services via systemd.user. Home manager’s selling point is that it comes with a large list of already defined services that we can enable with a boolean flag, instead of having to write our own service configuration. This is admittedly nice. In the end, I found that learning the idiosyncrazies of each home manager service definition was a less useful use of my time than learning how to define NixOS systemd services once and for all. The latter is after all where the former end up.

As a long-time sufferer of dotfile management, I had high hopes for the third point. And indeed, home manager will manage dotfiles just fine. It can do this in two modes: it can generate a config file from various options we fill out if someone has written a home manager module for the program we’re trying to configure, or it can plomp a file on the system verbatim from a source. I used the latter, as I didn’t feel like learning a configuration language to be able to partially configure program dotfiles was a good idea.

This works well, until we want to change anything in a dotfile. This experiment with home manager coincided with a regular low point in my life in which I try to use emacs. This comes with quite a lot of .emacs changes as I use Lisp for the only thing it’s ever been good for; configuring a text editor in the most complicated way imaginable. Now, the dotfiles that home manager (or Nix) puts on our systems are read-only, so every change would involve changing the source file and running home-manager switch. This seems like unnecessarily many steps, especially after I saw this brilliant Hacker news comment, which after a week of use is a much better solution for this problem.

All in all home manager is nice software. I can see it being useful for people who either don’t control the system they run on but want to use Nix in user mode to run their corner of it, or for those Nix users who gamers (a well-adjusted group of humans if there ever was one) would call "filthy casuals", that is, people who just want things to work and don’t care very much about learning how to write enough Nix to make that happen.

I’m not included in those groups, as I run this system and explicitly want to learn to use Nix in anger so I can try and fail to convince people to run it in production at work. Home manager is fine software and if it makes you happy, then please use it.

## Use ad hoc structs for command-line flags in Go

The path of least resistance to commandline flag parsing in Go is to use the flag package from the standard library. A lot of times the result looks like this:

func main() {
help := flags.Bool("help", false, "HALP")
frobinate := flags.Int("frobinate", 0, "Amount to frobinate by")
blargalarg := flags.String("blargalarg", "", "Social media comment")
// [713 variables later]
flag.Parse()

// Much later
if *blargalarg != "" {
// Do things. We may or may not remember what this variable is.
}
}

That is, we have a bunch of variables lying around we don’t really care about and take up perfectly good names. If we see one of them later in the program, we don’t have any context on where it comes from, so we have to start jumping around in the source.

In my projects I’ve used a little accounting trick to hold these flags. I find it helps me deal with them. We just define an anonymous struct to hold the flags:

func main() {
flags := struct{
help *bool
frobinate *int
blargalarg *string
// [713 field definitions]
}{
help: flags.Bool("help", false, "HALP"),
frobinate: flags.Int("frobinate", 0, "Amount to frobinate by"),
blargalarg: flags.String("blargalarg", "", "Social media comment"),
// [713 field instantiations]
}
flag.Parse()

// Much later
if *flags.blargalarg != "" {
// AAAAAH YES IT'S A FLAG
}
}

If I really need to, I can pull the anonymous struct out into its own global variable or type definition or whatever, and pass it around as arguments to functions that deal with its contents. That is not as handy with a litter of flag variables. But really I just find that defining all these flags clearly in one place makes the program easier to read later once I’ve forgotten what it does.

## Kleroteria

I got picked in Kleroteria a couple of months ago. This is my contribution.

Three years ago, my wife and I moved to Amsterdam. We wanted to have kids, and thought the Netherlands was a better place to do that than Mexico, where my wife is from and we lived.

We’ve both moved around a lot, and have picked up the languages of the places we’ve lived. That’s how we speak French and Spanish, along with English. Learning those was always a necessity; no one in France is going to speak English voluntarily, and my in-laws don’t speak the best English so I had to learn Spanish.

So it was a change to come to the Netherlands. Everyone here speaks perfect English. They also have no patience for your attempts at Dutch pronunciation or verb conjugation. The second they hear you’re not from here, they switch. On paper this is great, but it means that after three years here I still don’t speak Dutch. I can never get anyone to have a whole conversation with me in it.

A couple of weeks after we came here, we got pregnant. We had a boy, who has grown up into a little man who loves cars and trains, makes funny faces at the dinner table, and keeps trying to show our dog his books. ("Pancho! See!", but Pancho doesn’t care.)

We speak to him each in our own language, and by now he understands what we say. He’s been going to daycare here since he was five months old, where they speak Dutch, so he understands that as well. Kids who grow up with more than one language start speaking later, which is can be frustrating for everyone. He knows what he wants to say, but can’t figure out what words to use, and we have to keep guessing at sounds while other parents don’t.

He’s finally speaking now. He tells us everything he wants and knows. In Dutch. Every time. Every word comes out in Dutch. And we still don’t really speak it. It’s not really any less frustrating for anyone. There’s a lot of guessing at pronunciation and Google translate.

But what do you know. I did finally get someone to speak Dutch to me.