text:
## Correction, Sigma Algebras, and Mass functions

(by MarkCC) Sep 02 2013

So, I messed up a bit in the previous post. Let me get that out of the way before we move forward!

In measure theory, you aren't just working with sets. You're working with something called σ-algebras. It's a very important distinction.

The problem is, our intuition of sets doesn't always work. Sets, as defined formally, are really pretty subtle. We expect certain things to be true, because they make sense. But in fact, they are not implied by the definition of sets. A σ-algebra is, essentially, a well-behaved set - a set whose behavior matches our usual expectations.

To be formal, a sigma algebra over a set S is a collection Σ of subsets of S such that:

1. Σ is closed over set complement.
2. Σ is closed over countable union.

The reason why you need to make this restriction is, ultimately, because of the axiom of choice. Using the axiom of choice, you can create sets which are unmeasurable. They're clearly subsets of a measurable set, and supersets of other measurable sets - and yet, they are, themselves, not measurable. This leads to things like the Banach-Tarski paradox: you can take a measurable set, divide it into non-measurable subsets, and then combine those non-measurable subsets back into measurable sets whose size seem to make no sense. You can take a sphere the size of a baseball, slice it into pieces, and then re-assemble those pieces into a sphere the size of the earth, without stretching them!

These non-measurable sets blow away our expectations about how things should behave. The restriction to σ algebras is just a way of saying that we need to be working in a space where all sets are measurable. When we're looking at measure theory (or probability theory, where we're building on measures), we need to exclude non-measurable sets. If we don't, we're seriously up a creek without a paddle. If we allowed non-measurable sets, then the probability theory we're building would be inconsistent, and that's the kiss of death in mathematics.

Ok. So, with that out of the way, how do we actually use Kolmogorov's axioms? It all comes down to the idea of a sample space. You need to start with an experiment that you're going to observe. For that experiment, there are a set of possible outcomes. The set of all possible outcomes is the sample space.

Here's where, sadly, even axiomatized probability theory gets a bit handwavy. Given the sample space, you can define the structure of the sample space with a function, called the probability mass function, f, which maps each possible event in the sample space to a probability. To be a valid mass function for a sample space S, it's got to have the following properties:

1. For each event e in S, f(e) ≥ 0 and f(e) <= 1..
2. The sum of the probabilities in the sample space must be 1:

So we wind up with a sort of circularity: in order to describe the probability of events, we need to start by knowing the probability of events. In fact, this isn't really a problem: we're talking about taking something than we observe in the real world, and mapping it into the abstract space of math. Whenever we do that, we need to take our observations of the real world and create an approximation as a mathematical model.

The point of probability theory isn't to do that primitive mapping. In general, we already understand how rolling a single die works. We know how it should behave, and we know how and why its actual behavior can vary from our expectation. What we want to know is really how many events combine.

We don't need any special theory to figure out what the probability of rolling a 3 on a six-sided die is: that's easy, and it's obvious: it's 1 in 6. But what's the probability of winning a game of craps?

If all days of the year 2001 are equally likely, then we don't need anything fancy to ask what the probability of someone born in 2001's birthday being July 21st. It's easy: 1 in 365. But if I've got a group of 35 people, what's the probability of two of them sharing the same birthday?

Both of those questions start with the assignment of a probability mass function, which is trivial. But they involve combining the probabilities given by those mass functions, and use them with Kolmogorov's axioms to figure out the probabilities of the complicated events.

## Kolmogorov's Axioms of Probability

(by MarkCC) Aug 24 2013

The way that I've talked about probability so far is mostly informal. That's the way that probability theory was treated for a long time. You defined probability spaces over collections of equal probability sets. You combined probability spaces by combining their events into other kinds of equally probable events.

The problem with that should be obvious: it's circular. You want to define the probability of events; to do that, you need to start with equally probable events, which means that on some level, you already know the probabilities. If you don't know the probabilities, you can't talk about them. The reality is somewhat worse than that, because this way of looking at things completely falls apart when you start trying to think about infinite probability spaces!

So what can you do?

The answer is to reformulate probability. Mathematicians knew about this kind of problem for a very long time, but what they mostly just ignored it: probability wasn't considered a terribly interesting field.

Then, along came Kolmogorov - the same brilliant guy who's theory of computational complexity is so fascinating to me! Kolmogorov created a new formulation of probability theory. Instead of starting with a space of equally probable discrete events, you start with a measure space.

Before we can look at how Kolmogorov reformulated probability (the Kolmogorov axioms), we need to look at just what a measure space is.

A measure space is just a set with a measure function. So let X be a set. A measure μ on X is a function from a subset of X to a real number: with the following properties:

• Measures are non-negative:
• The measure of the empty set is always 0:
• The measure of a finite sequence of unions is the sum of the individual measures

So the idea is pretty simple: a measure space is just a way of defining the size of a subset in a consistent way.

To work with probability, you need a measure space where the measure of the entire set is 1. With that idea in mind, we can put together a proper, formal definition of a probability space that will really allow us to work with, and to combine probabilities in a rigorous way.

Like our original version, a probability space has a set of events, called its event space. We'll use F to represent the set of all possible events, and e to represent an event in that set.

There are three fundamental axioms of probability, which are going to look really similar to the three axioms of a measure space:

1. Basic measure: the probability of any event is a positive real number: .
2. Unit measure: the probability that some event will occur is 1, which we write as ( is called the unit event, and is the union of all possible events.) Alternatively, the probability of no event occurring is 0: .
3. Combination: For any two distinct events or sets of events and , the probability of or is : . This can be extended to any countable sequence of unions.

This is very similar to the informal version we used earlier. But as we'll see later, this simple formulation from measure theory will give us a lot of additional power.

It's worth taking a moment to point out two implications of these axioms. (In fact, I've seen some presentations that treat some of these as additional axioms, but they're provable from the first three.

• Monotonicity: if , then .
• Upper Bound: for any event or set of events , .

The brilliance of Kolmogorov was realizing that these rules were everything you need to work out any probability you want - in both finite and infinite spaces. We'll see that there's a lot of complexity in the combinatorics of probability, but it will all always ultimately come back to these three rules.

## Infinite Cantor Crankery

(by MarkCC) Jul 29 2013

I recently got yet another email from a Cantor crank.

Sadly, it's not a particularly interesting letter. It contains an argument that I've seen more times than I can count. But I realized that I don't think I've ever written about this particular boneheaded nonsense!

I'm going to paraphrase the argument: the original is written in broken english and is hard to follow.

• Cantor's diagonalization creates a magical number ("Cantor's number") based on an infinitely long table.
• Each digit of Cantor's number is taken from one row of the table: the Nth digit is produced by the Nth row of the table.
• This means that the Nth digit only exists after processing N rows of the table.
• Suppose it takes time t to get the value of a digit from a row of the table.
• Therefore, for any natural number N, it takes N*t time to get the first N digits of Cantor's number.
• Any finite prefix of Cantor's number is a rational number, which is clearly in the table.
• The full Cantor's number doesn't exist until an infinite number of steps has been completed, at time &infinity;*t.
• Therefore Cantor's number never exists. Only finite prefixes of it exist, and they are all rational numbers.

The problem with this is quite simple: Cantor's proof doesn't create a number; it identifies a number.

It might take an infinite amount of time to figure out which number we're talking about - but that doesn't matter. The number, like all numbers, exists, independent of
our ability to compute it. Once you accept the rules of real numbers as a mathematical framework, then all of the numbers, every possible one, whether we can identify it, or describe it, or write it down - they all exist. What a mechanism like Cantor's diagonalization does is just give us a way of identifying a particular number that we're interested in. But that number exists, whether we describe it or identify it.

The easiest way to show the problem here is to think of other irrational numbers. No irrational number can ever be written down completely. We know that there's got to be some number which, multiplied by itself, equals 2. But we can't actually write down all of the digits of that number. We can write down progressively better approximations, but we'll never actually write the square root of two. By the argument above against Cantor's number, we can show that the square root of two doesn't exist. If we need to create the number by writing down all af its digits,s then the square root of two will never get created! Nor will any other irrational number. If you insist on writing numbers down in decimal form, then neither will many fractions. But in math, we don't create numbers: we describe numbers that already exist.

But we could weasel around that, and create an alternative formulation of mathematics in which all numbers must be writeable in some finite form. We wouldn't need to say that we can create numbers, but we could constrain our definitions to get rid of the nasty numbers that make things confusing. We could make a reasonable argument that those problematic real numbers don't really exist - that they're an artifact of a flaw in our logical definition of real numbers. (In fact, some mathematicians like Greg Chaitin have actually made that argument semi-seriously.)

By doing that, irrational numbers could be defined out of existence, because they
can't be written down. In essence, that's what my correspondant is proposing: that the definition of real numbers is broken, and that the problem with Cantor's proof is that it's based on that faulty definition. (I don't think that he'd agree that that's what he's arguing - but either numbers exist that can't be written in a finite amount of time, or they don't. If they do, then his argument is worthless.)

You certainly can argue that the only numbers that should exist are numbers that can be written down. If you do that, there are two main paths. There's the theory of computable numbers (which allows you to keep π and the square roots), and there's the theory of rational numbers (which discards everything that can't be written as a finite fraction). There are interesting theories that build on either of those two approaches. In both, Cantor's argument doesn't apply, because in both, you've restricted the set of numbers to be a countable set.

But that doesn't say anything about the theory of real numbers, which is what Cantor's proof is talking about. In the real numbers, numbers that can't be written down in any form do exist. Numbers like the number produced by Cantor's diagonalization definitely do. The infinite time argument is a load of rubbish because it's based on the faulty concept that Cantor's number doesn't exist until we create it.

The interesting thing about this argument to be, is its selectivity. To my correspondant, the existence of an infinitely long table isn't a problem. He doesn't think that there's anything wrong with the idea of an infinite process creating an infinite table containing a mapping between the natural numbers and the real numbers. He just has a problem with the infinite process of traversing that table. Which is really pretty silly when you think about it.

## Recipe: Sous Vide Braised Pork Belly with Chevre Polenta

(by MarkCC) Jul 01 2013

I really outdid myself with tonight's dinner. It was a total ad-lib - not recipe written in advance, just randomly trying to make something good. It turned out so good that I need to write down what I did, so that I can make it again!

## Part 1: the pork

• 2 1/2 pounds pork belly. I'm picky about pork; if I'm going to eat it, I want it to be good. I didn't grow up eating pork. My family didn't keep kosher, but we didn't bring pork into the house. To this day, I don't like most pork. Grocery store pork is, typically, bland, greasy, and generally nasty stuff. But the first real pork that I ate was at Momofuku in Manhattan. It was Berkshire pork, from a farm in upstate NY. That was delicious. Since then I've experimented, and I really think that nothing compares to fresh Berkshire. It costs a lot more than grocery store pork, but it's worth it. I order it direct from Flying Pig Farm.
• 4 cloves garlic.
• 1 teaspoon salt.
• 1 1/2 teaspoons fennel pollen.
• 1 teaspoon dried rosemary.
• 1 tablespoons olive oil.
• pepper
• 1/4 cup salt.
• 1/4 cup sugar.
1. Prepare the pork belly: trim off the skin, and any egregiously extra fat from the skin side.
2. Put the garlic, fennel pollen, rosemary, 1 teaspoon of salt, and the olive oil into a mortar and pestle, and crush them to a paste.
3. Coat the pork with the herb paste.
4. Add fresh-ground black pepper to the pork.
5. Mix together 1/4 cup each of sugar and salt, and coat the pork with it.
6. Put the pork into the fridge overnight.
7. In the morning, remove the pork from the fridge, and discard any liquids that were drawn out by the salt.
8. Sealed the pork in a sous vide bag, and cook at 190 degrees for
5 hours. (If you don't have a sous vide machine, you could probably
do it covered in a 200 degree oven. You'll probably want to add a bit
of water.)
9. Take out the pork, and separate the meat from the liquid that's collected in the bags. (Do NOT discard it; that's pure flavor!) Put
both into the fridge for a couple of hours to cool.
10. When it's cool, the fat that rendered out of the pork will have solidifed - remove it, and discard it. (Or keep it for something else.)
11. Cut the pork into 2 inch thick chunks.
12. In a smoking hot cast iron pan, brown the pork chunks on all sides.
13. Add in the reserved liquids, along with 1/4 cup of port wine.
Reduce until it forms a glaze over the pork. Remove the pork to a
plate - it's done!

## Part 2: the Polenta

• 1 cup polenta. I use very coarse polenta - I like my polenta to have some texture. (My friend Anoop teases me, insisting that I'm making grits.)
• 4 cups chicken stock.
• 1 cup water.
• 1 teaspoon salt.
• 1 tablespoon butter.
• 2 ounces chevre goat cheese.
1. Put the salt, water, and chicken stock into a pan, and bring to a boil.
2. Reduce the heat to medium low, and stir in the polenta.
3. Cook the polenta on medium low to low heat for 1 1/2 hours.
4. Remove from heat, add in the butter, and stir until it's all melted and blended in.
5. crumble the goat cheese in, and stir it in.

### Part 3: the assembly.

1. Put a big pile of the polenta in the middle of a plate.
2. Put a couple of chunks of the glazed pork onto the polenta.
3. Put sauteed asparagus around the outside.

## Independence and Combining probabilities.

(by MarkCC) Jun 24 2013

As I alluded to in my previous post, simple probability is really a matter of observation, to produce a model. If you roll a six sided die over and over again, you'll see that each face comes up pretty much equally often, and so you model it as 1/6 probability for each face. There's not really a huge amount of principle behind the basic models: it's really just whatever works. This is the root of the distinction between interpretations: a frequentist starts with an experiment, and builds a descriptive model based on it, and says that the underlying phenomena being tested has the model as g, a property; a Bayesian does almost the same thing, but says that the model describes the state of their knowledge.

Where probability starts to become interesting in when you combine things. I know the probability of outcomes for rolling one die: how can I use that to see what happens when I roll five dice together? I know the probability of drawing a specific card from a deck: what are the odds of being dealt a particular poker hand?

We'll start with the easiest part: combining independent probabilities. The probability of two events are independent when there's no way for the outcome of one to influence the outcome of the other. For example, if you're flipping a coin several times, the result of one coin flip has no effect on the result of a subsequent flip. On the other hand, dealing 10 cards from a deck is a sequence of dependent events: once you've dealt one card, the next deal must come from the remaining cards: you can't deal the same card twice.

If you know the probability space of your trials, then recognizing an independent situation is easy: if the outcome of one trial doesn't alter the probability space of other trials, then they're independent.

Look back at the coin flip example: we know what the probability space of a coin flip looks like: it's got two, equally probable outcomes. If you've flipped a coin once, and you're going to flip another coin, the result of the first flip can't do anything that alters the probability space of a subsequent flip.

But if you think about dealing cards, that's not true. With a standard deck of cards, the initial probability space has 52 outcomes, each of which is equally likely. So the odds of being dealt the 5 of spades is exactly 1/52.

Now, suppose that you got lucky, and you did get dealt the 5 of spades on the first card. What's the probability of being dealt the 5 of spades's on the second? If they were independent events, it would still be 1/52. But once you've dealt one card, you can't deal it again. The probability of being dealt the 5 of spades as the second card is 0: it's impossible. The probability space only has 51 possible outcomes, and the 5 of spades is not one of them. The space has changed. That's the definition of a dependent event.

When you're faced with dependent probabilities, you need to figure out how the probability space will be changed, and incorporate that into your computation. Once you've incorporated the change in the probability space of the second test, then you've got a new independent probability, and you can combine them. Figuring out how to alter the probability space can be extremely difficult, but that's what makes it interesting.

When you're dealing with independent events, it's easy to combine them. There are two basic ways of combining event probabilities,
and they should be familiar from logic: event1 AND event2, and event1 OR event2.

Suppose you're looking at two test with independent outcomes. I know that the probability of event e is P(e), and the probability of event f is P(f) Then the outcome of e & f - that is, of having e as the outcome of the first trial, and f as the outcome of the second, is P(e)×P(f). The odds of rolling HTTH on a coin is (1/2)*(1/2)*(1/2)*(1/2)=(1/16).

If you're looking at independent alternatives - that is, the probability of e OR F, you combine the probabilities of the event with addition: P(e) + P(f). So, the odds of drawing any heart from a deck: for each card, it's 1/52. There are thirteen different hearts. So the odds of drawing a red are 1/52 + 1/52 + ... = 13/52 = 1/4.

That still doesn't get us to the really interesting stuff. We still can't quite work out something like the odds of being dealt a flush. To get there, we need to learn some combinatorics, which will allow us to formulate the probability spaces that we need for an interesting probability.

## Probability Spaces

(by MarkCC) Jun 19 2013

Sorry for the slowness of the blog lately. I finally got myself back onto a semi-regular schedule when I posted about the Adria Richards affair, and that really blew up. The amount of vicious, hateful bile that showed up, both in comments (which I moderated) and in my email was truly astonishing. I've written things which pissed people off before, and I've gotten at least my fair share of hatemail. But nothing I've written before came close to preparing me for the kind of unbounded hatred that came in response to that post.

I really needed some time away from the blog after that.

Anyway, I'm back, and it's time to get on with some discrete probability theory!

I've already written a bit about interpretations of probability. But I haven't said anything about what probability means formally. When I say that the probability of rolling a 3 with a pair of fair six-sided dice is 1/18, how do I know that? Where did that 1/6th figure come from?

The answer lies in something called a probability space. I'm going to explain the probability space in frequentist terms, because I think that that's easiest, but there is (of course) an equivalent Bayesian description.) Suppose I'm looking at a particular experiment. In classic mathematical form, a probability space consists of three components (Ω, E, P), where:

1. Ω, called the sample space, is a set containing all possible outcomes of the experiment. For a pair of dice, Ω would be the set of all possible rolls: {(1,1), (1,2), (1,3), (1,4), (1,5), (1, 6), (2,1), ..., (6, 5), (6,6)}.
2. E is an equivalence relation over Ω, which partitions Ω into a set of events. Each event is a set of outcomes that are equivalent. For rolling a pair of dice, an event is a total - each event is the set of outcomes that have the same total. For the event "3" (meaning a roll that totalled three), the set would be {(1, 2), (2, 1)}.
3. P is a probability assignment. For each event e in E, P(e) is a value between 0 and 1, where:

(That is, the sum of the probabilities of all of the possible events in the space is exactly 1.)

The probability of an event e being the outcome of a trial is P(e).

So the probability of any particular event as the result of a trial is a number between 0 and 1. What's it mean? If the probability of event e is p, then if we repeat the trial N times, we expect N*p of those trials to have e as their result. If the probability of e is 1/4, and we repeat the trial 100 times, we'd expect e to be the result 25 times.

But in an important sense, that's a cop-out. We've defined probability in terms of this abstract model, where the third component is the probability. Isn't that circular?

Not really. For a given trial, we create the probability assignment by observation and/or analysis. The important point is that this is really just a bare minimum starting point. What we really care about in probability isn't the change associated with a single, simple, atomic event. What we want to do is take the probability associated with a group of single events, and use our understanding of that to allow us to explore a complex event.

If I give you a well-shuffled deck of cards, it's easy to show that the odds of drawing the 3 of diamonds is 1/52. What we want to do with probability is things like ask: What are the odds of being dealt a flush in a poker hand?

The construction of a probability space gives us a well-defined platform to use for building probabilistic models of more interesting things. Give a probability space of two single dice, we can combine them together to create the probability space of the two dice rolled together. Given the probability space of a pair of dice, we can construct the probability space of a game of craps. And so on.

## Probability and Interpretations

(by MarkCC) May 12 2013

I'm going to do some writing about discrete probability theory. Probability is an extremely important area of math. We encounter aspects of it every day. It's also a very poorly understood area - it's one that we see abused or just fouled up every day.

I'm going to focus on discrete probability theory. What that means is that we're going to look at things where the space containing the things that we're going to look at contains a countable number of elements. The probability of getting a certain sequence of coin flips, or of getting a certain hand of cards are described by discrete probability theory. On the other hand, the odds of a radioactive isotope decaying at a particular time requires continuous probability theory.

Before getting into the details, there's one important thing to mention. When you're talking about probability, there are two fundamental schools of interpretetation. There are frequentist interpretations, and there are Bayesian interpretations.

In a frequentist interpretation, when you say the probability of an event is 0.6, what you mean is that if you were to perform a series of experiments precisely reproducing the event, then on average, if you did 100 experiments, the event would occur 60 times. In the frequentist interpretation, the probability is an intrinsic property of the event. For a frequentist, it makes sense to say that there is a "real" probability associated with an event.

In a Bayesian interpretation, when you say that the probability of an event is 0.6, what you mean is that based on your current state of knowledge about the event, you have a 60% certainty that the event will occur. In a strict Bayesian interpretation, the event doesn't have any kind of intrinsic probability associated with it. The specific event that you're interested in either will occur, or it won't. There's no real probability involved. What probability measures is how certain you are about whether or not it will occur.

For example, think about flipping a fair coin.

A frequentist would say that you can flip a coin many times, and half of the time, it will land on heads. So the probability of a coin flip landing on the head of the coin is 0.5. A Bayesian would say that the coin will land either on heads or on tails. Since you don't know which, and you have no other information to use to be able to make a better prediction, you can have a certainty of 0.5 that it will land on the head of the coin.

In the real world, I think that most people are really somewhere in between.

I think that all but the most fervent Bayesians do rely on an intuitive notion of the "intrinsic" probability of an event. They may describe it in different terms, but when it comes down to it, they're using the basic frequentist notion. And I don't think that you can find a sane frequentist anywhere who won't use Bayes theorem to update their priors in the face of new information - which is the most fundamental notion in the Bayesian interpretation.

One note before I finish this, and get started on the real meaty posts. In the past, when I've talked about probability, people have started stupid flamewars in the comments. People get downright religious about interpretations of probability. There are religious Bayesians, who think that all frequentists are stupid idiots who should be banished from the field of math; likewise, there are religious frequentists who think that Bayesians are all a crop of arrogant know-it-alls who should be sent to Siberia. I am not going to tolerate any of that nonsense. If you feel that you cannot read posts on probability without going into a diatribe about those stupid frequentists/Bayesians and their deliberately stupid ideas, please go away and don't even read these posts. If you do go into such a diatribe, I will delete your comments without any hesitation.

## Speed-Crankery

(by MarkCC) May 05 2013

A fun game to play with cranks is: how long does it take for the crank to contradict themselves?

When you're looking at a good example of crankery, it's full of errors. But for this game, it's not enough to just find an error. What we want is for them to say something so wrong that one sentence just totally tears them down and demonstrates that what they're doing makes no sense.

"The color of a clear sky is green" is, most of the time, wrong. If a crank makes some kind of argument based on the alleged fact that the color of a clear daytime sky is green, the argument is wrong. But as a statement, it's not nonsensical. It' just wrong.

On th other hand, "The color of a clear sky is steak frite with bernaise sauce and a nice side of roasted asparagus", well... it's not even wrong. It's just nonsense.

Today's crank is a great example of this. If, that is, it's legit. I'm not sure that this guy is serious. I think this might be someone playing games, pretending to be a crank. But even if it is, it's still fun.

About a week ago, I got en mail titled "I am a Cantor crank" from a guy named Chris Cuellar. The contents were:

...AND I CHALLENGE YOU TO A DUEL!! En garde!

Haha, ok, not exactly. But you really seem to be interested in this stuff. And so am I. But I think I've nailed Cantor for good this time. Not only have I come up with algorithms to count some of these "uncountable" things, but I have also addressed the proofs directly. The diagonalization argument ends up failing spectacularly, and I believe I have a good explanation for why the whole thing ends up being invalid in the first place.

And then I also get to the power set of natural numbers... I really hope my arguments can be followed. The thing I have to emphasize is that I am working on a different system that does NOT roll up cardinality and countability into one thing! As it will turn out, rational numbers are bigger than integers, integers are bigger than natural numbers... but they are ALL countable, nonetheless!

Anyway, I had started a little blog of my own a while ago on these subjects. The first post is here:

http://laymanmath.blogspot.com/2012/09/the-purpose-and-my-introduction.html

Have fun... BWAHAHAHA

So. We've got one paragraph of intro. And then everything crashes and burns in an instant.

"Rational numbers are bigger than integers, integers are bigger than natural numbers, but they are all countable". This is self-evident rubbish. The definition of "countable" say that an infinite set I is countable if, and only if, you can create a one-to-one mapping between the members of I and the natural numbers. The definition of cardinality says that if you can create a one-to-one mapping between two sets, the sets are the same size.

When Mr. Cuellar says that the set of rational numbers is bigger that the set of natural numbers, but that they are still countable... he's saying that there is not a one-to-one mapping between the two sets, but that there is a one-to-one mapping between the two sets.

Look - you don't get to redefine terms, and then pretend that your redefined terms mean the same thing as the original terms.

If you claim to be refuting Cantor's proof that the cardinality of the real numbers is bigger than the cardinality of the natural numbers, then you have to use Cantor's definition of cardinality.

You can change the definition of the size of a set - or, more precisely, you can propose an alternative metric for how to compare the sizes of sets. But any conclusions that you draw about your new metric are conclusions about your new metric - they're not conclusions about Cantor's cardinality. You can define a new notion of set size in which all infinite sets are the same size. It's entirely possible to do that, and to do that in a consistent way. But it will say nothing about Cantor's cardinality. Cantor's proof will still work.

What my correspondant is doing is, basically, what I did above in saying that the color of the sky is steak frites. I'm using terms in a completely inconsistent meaningless way. Steak frites with bernaise sauce isn't a color. And what Mr. Cuellar does is similar: he's using the word "cardinality", but whatever he means by it, it's not what Cantor meant, and it's not what Cantor's proof meant. You can draw whatever conclusions you want from your new definition, but it has no bearing on whether or not Cantor is correct. I don't even need to visit his site: he's demonstrated, in record time, that he has no idea what he's doing.

## The Gravitational Force of Rubbish

(by MarkCC) May 01 2013

Imagine, for just a moment, that you were one a group of scientists that had proven the most important, the most profound, the most utterly amazing scientific discovery of all time. Where would you publish it?

Maybe Nature? Science? Or maybe you'd prefer to go open-access, and go with PLOS ONE? Or more mainstream, and send a press release to the NYT?

Well, in the case of today's crackpots, they bypassed all of those boring journals. They couldn't be bothered with a pompous rag like the Times. No, they went for the really serious press: America Now with Leeza Gibbons.

What did they go to this amazing media outlet to announce? The most amazing scientific discovery of all time: gravity is an illusion! There's no gravity. In fact, not just is there no gravity, but all of that quantum physics stuff? It's utter rubbish. You don't need any of that complicated stuff! No - you need only one thing: the solar wind.

A new theory on the forces that control planetary orbit refutes the 400-year old assumptions currently held by the scientific community. Scientific and engineering experts Gerhard and Kevin Neumaier have established a relationship between solar winds and a quantized order in both the position and velocity of the solar system's planets, and movement at an atomic level, with both governed by the same set of physics.

The observations made bring into question the Big Bang Theory, the concept of black holes, gravitational waves and gravitons. The Neumaiers' paper, More Than Gravity, is available for review at MoreThanGravity.com

Pretty damned impressive, huh? So let's follow their instructions, and go over to their website.

Ever since humankind discovered that the Earth and the planets revolved around the Sun, there was a question about what force was responsible for this. Since the days of Newton, science has held onto the notion that an invisible force, which we have never been able to detect, controls planetary motion. There are complicated theories about black holes that have never been seen, densities of planets that have never been measured, and subatomic particles that have never been detected.

However, it is simpler than all of that and right in front of us. The Sun and the solar wind are the most powerful forces in our solar system. They are physically moving the planets. In fact, the solar wind spins outward in a spiral at over a million miles per hour that controls the velocity and distances that planets revolve around the Sun. The Sun via the solar wind quantizes the orbits of the planets – their position and speed.

The solar wind also leads to the natural log and other phenomenon from the very large scale down to the atomic level. This is clearly a different idea than the current view that has been held for over 400 years. We have been working on this for close 50 years and thanks to satellite explorations of space have data that just was not available when theories long ago were developed. We think that we have many of the pieces but there are certainly many more to be found. We set this up as a web site, rather as some authoritative book so that there would be plenty of opportunity for dialog. The name for this web site, www.MorethanGravity.com was chosen because we believe there is far more to this subject than is commonly understood. Whether you are a scientific expert in your field or just have a general interest in how our solar system works, we appreciate your comments.

See, it's all about the solar wind. There's no such thing as gravity - that's just nonsense. The sun produces the solar wind, which does absolutely everything. The wind comes out of the sun, and spirals out from the sun. That spiral motion has eddies in it an quantized intervals, and that's where the planets are. Amazing, huh?

Remember my mantra: the worst math is no math. This is a beautiful demonstration
of that.

Of course... why does the solar wind move in a spiral? Everything we know says that in the absence of a force, things move in a straight line. It can't be spiraling because of gravity, because there is no gravity. So why does it spiral? Our brilliant authors don't bother to say. What makes it spiral, instead of just move straight? Mathematically, spiral motion is very complicated. It requires a centripetal force which is smaller than the force that would produce an orbit. Where's that force in this framework? There isn't any. They just say that that's how the solar wind works, period. There are many possible spirals, with different radial velocities - which one does the solar wind follow according to this, and why? Again, no answer from the authors.

Or... why is the sun producing the solar wind at all? According to those old, stupid theories that this work of brilliance supercedes, the sun produces a solar wind because it's fusing hydrogen atoms into helium. That's happening because gravity is causing the atoms of the sun to be compressed together until they fuse. Without gravity, why is fusion happening at all? And given that it's happening, why does the sun not just explode into a supernova? We know, from direct observation, that the energy produced by fusion creates an outward force. But gravity can't be holding the sun together - so why is the sun there at all? Still, no answers.

They do, eventually, do some math. One of the big "results" of this hypothesis is about the "quantization" of the orbits of planets around the sun. They were able to develop a simple equation which predicts the locations where planets could exist in their "solar wind" system.

Let’s start with the distance between the planets and the Sun. We guessed that if the solar system was like an atom, that planetary distance would be quantized. This is to say that we thought that the planets would have definite positions and that they would be either in the position or it would be empty. In a mathematical sense, this would be represented by a numerical integer ordering (0,1,2,3,…). If the first planet, Mercury was in the 0 orbital, how would the rest of the planets line up? Amazingly well we found.

If we predict the distance from the surface of the Sun to each planet in this quantized approach, the results are astounding. If D equals the mean distance to the surface of the Sun, and d0 as the distance to Mercury, we can describe the relationship that orders the planets mathematically as:

Each planetary position can be predicted from this equation in a simple calculation as we increase the integer (or planet number) n. S is the solar factor, which equals 1.387. The solar factor is found in the differential rotation of the Sun and the profile of the solar wind which we will discuss later.

Similar to the quantized orbits that exist within an atom, the planetary bodies are either there or not. Mercury is in the zero orbital. The next orbital is missing a planet. The second, third, and fourth orbitals are occupied by Venus, Earth, and Mars respectively. The fifth orbital is missing. The sixth orbital is filled with Ceres. Ceres is described as either the largest of all asteroids or a minor planet (with a diameter a little less than half that of Pluto), depending on who describes it. Ceres was discovered in 1801 as astronomers searched for the missing planets that the Titius-Bode Law predicted would exist.

So. What they found was an exponential equation which products very approximate versions of the size of first 8 planets' orbits, as well as a couple of missing ones.

This is, in its way, interesting. Not because they found anything, but rather because they think that this is somehow profound.

We've got 8 data points (or 9, counting the asteroid belt). More precisely, we have 9 ranges, because all of the orbits are elliptical,but the authors of this junk are producing a single number for the size of the orbits, and they can declare success if their number falls anywherewithin the range from perihelion to aphelion in each of the orbits.

It would be shocking if there weren't any number of simple equations that described exactly the 9 data points of the planet's orbits.

But they couldn't even make that work directly. They only manage to get a partial hit - getting an equation that hits the right points, but which also generates a bunch of misses. There's nothing remotely impressive about that.

From there, they move on to the strawmen. For example, they claim that their "solar wind" hypothesis explains why the planets all orbit in the same direction on the same plane. According to them, if orbits were really gravitational, then planets would orbit in random directions on random planes around the sun. But their theory is better than gravity, because it says why the planets are in the same plane, and why they're all orbiting in the same direction.

The thing is, this is a really stupid argument. Why are the planets in the same plane, orbiting in the same direction? Because the solar system was formed out of a rotating gas cloud. There's a really good, solid, well-supported explanation of why the planets exist, and why they orbit the sun the way they do. Gravity doesn't explain all of it, but gravity is a key piece of it.

What they don't seem to understand is how amazingly powerful the theory of gravity is as a predictive tool. We've sent probes to the outer edges of the solar system. To do that, we didn't just aim a rocket towards Jupiter and fire it off. We've done things like the Cassini probe, where we launched a rocket towards Venus. It used the gravitational field of Venus twice to accelerate it with a double-slingshot maneuver, and send it back towards earth, using the earth's gravity to slingshot it again, to give it the speed it needed to get to Jupiter.

This wasn't a simple thing to do. It required an extremely deep understanding of gravity, with extremely accurate predictions of exactly how gravity behaves.

How do our brilliant authors answer this? By handwaving. The extend of their response is:

Gravitational theory works for things like space travel because it empirically measures the force of a planet, rather than predicting it.

That's a pathetic handwave, and it's not even close to true. The gravitational slingshot is a perfect answer to it. A slingshot doesn't just use some "empirically measured" force of a planet. It's a very precise prediction of what the forces will be at different distances, how that force will vary, and what effects that force will have.

They do a whole lot more handwaving of very much the same order. Pure rubbish.

## What the heck is a DNS amplification DoS attack?

(by MarkCC) Apr 08 2013

A couple of weeks ago, there was a bunch of news about a major DOS attack on Spamhaus. Spamhaus is an online service that maintains a blacklist of mail servers that are known for propagating spam. I've been getting questions about what a DoS attack is, and more specifically what a "DNS amplification attack" (the specific attack at the heart of last week's news) is. This all became a bit more relevant to me last week, because some asshole who was offended by my post about the Adria Richards affair launched a smallish DoS attack against scientopia. (This is why we were interrmitently very slow last week, between tuesday and thursday. Also, to be clear, the DNS amplification attack was used on Spamhaus. Scientopia was hit by a good old fashioned DDoS attack.)

So what is a DoS attack? And what specifically is a DNS amplification attack?

Suppose that you're a nastly person who wants to take down a website like scientopia. How could you do it? You could hack in to the server, and delete everything. That would kill it pretty effectively, right?

It certainly would. But from the viewpoint of an attacker, that's not a particularly easy thing to do. You'd need to get access to a privileged account on our server. Even if we're completely up to date on patches and security fixes, it's probably possible to do that, but it's still probably going to be a lot of work. Even for a dinky site like scientopia, getting that kind of access probably isn't trivial. For a big security-focused site like spamhaus, that's likely to be close to impossible: there are layers of security that you'd need to get through, and there are people constantly watching for attacks. Even if you got through, if the site has reliable backups, it won't be down for long, and once they get back up, they'll patch whatever hole you used to get in, so you'd be back to square one. It's a lot of work, and there are much easier ways to take down a site.

What you, as an attacker, want is a way to take the site down without having any kind of access to the system. You want a way that keeps the site down for as long as you want it down. And you want a way that doesn't leave easily traced connections to you.

That's where the DoS attack comes in. DoS stands for "denial of service". The idea of a DoS attack is to take a site down without really taking it down. You don't actually kill the server; you just make it impossible for legitimate users to access it. If the sites users can't access the site even though the server is technically still up and running, you've still effectively killed it.

How do you do that? You overwhelm the server. You target some finite resource on the server, and force it to use up that resource just dealing with requests or traffic that you sent to the server, leaving it with nothing for its legitimate users.

In terms of the internet, the two resources that people typically target are CPU and network bandwidth.

Every time that you send a request to a webserver, the server has to do some computation to process that request. The server has a finite amount of computational capability. If you can hit it with enough requests that it spends all of its time processing your requests, then the site becomes unusable, and it effectively goes down. This is the simplest kind of DoS attack. It's generally done in a form called a DDoS - distributed denial of server attack, where the attacker users thousands or millions of virus-infected computers to send requests. The server gets hit by a vast storm of requests, and it can't distinguish the legitimate requests from the ones generated by the attacker. This is the kind of attack that hit Scientopia last week. We were getting streams of a couple of thousands malformed requests per second.

This kind of attack can be very effective. It's hard - not impossible, but hard - to fight. You need to identify the common traits of the attackers, and set up some kind of filter to discard those requests. From the attacker's point of view, it's got one problem: price. Most people don't have a personal collection of virus-infected machines that they can use to mount an attack. What they actually do is rent machines! Virus authors run services where they'll use the machines that they've to run an attack for you, for a fee. They typically charge per machine-hour. So to keep a good attack going for a long time is expensive! Another problem with this kind of attack is that the amount of traffic that you can inflict on the server per attacker is also used by the client. The client needs to establish a connection to the server. That consumes CPU, network connections, and bandwidth on the client.

The other main DoS vector is network bandwidth. Every server running a website is connected to the network by a connection with a fixed capacity, called it's bandwidth. A network connection can only carry a certain quantity of information. People love to make fun of the congressman who said that the internet is like a series of tubes, but that's not really a bad analogy. Any given connection is a lot like a pipe. You can only cram so much information through that pipe in a given period of time. If you can send enough traffic to completely fill that pipe, then the computer on the other end is, effectively, off the network. It can't receive any requests.

For a big site like spamhaus, it's very hard to get enough machines attacking to effectively kill the site. The amount of bandwidth, and the number of different network paths connecting spamhaus to the internet is huge! The number of infected machines available for an attack is limited, and the cost of using all of them is prohibitive.

What an attacker would like for killing something like Spamhaus is an attack where the amount of work/cpu/traffic used to generate the attack is much smaller than the amount of work/cpu/traffic used by the server to combat the attack. That's where amplification comes in. You want to find some way of using a small amount of work/traffic on your attacker machines to cause your target to lost a large amount of work/traffic.

In this recent attack on Spamhaus, they used an amplification attack, that was based on a basic internet infrastructure service called the Domain Name Service (DNS). DNS is the service which is used to convert between the name of a server (like scientopia.org), and its numeric internet address (184.106.221.182). DNS has some technical properties that make it idea for this kind of attack:

1. It's not a connection-based service. In most internet services, you establish a connection to a server, and send a request on that connection. The server responds on the same connection. In a connection-based service, that means two things. First, you need to use just as much bandwidth as the target, because if you drop the connection, the server sees the disconnect and stops processing your request. Second, the server knows who it's connected to, and it always sends the results of a request to the client that requested it. But DNS doesn't work that way. In DNS, you send a request without a connection, and in the request, you provide an address that the response should be sent to. So you can fake a DNS request, by putting someone else's address as the "respond-to" address in the request.
2. It's possible to set up DNS to create very large responses to very small requests. There are lots of ways to do this. The important thing is that it's really easy to use DNS in a way that allows you to amplify the amount of data being sent to a server by a factor of 100. In one common form of DNS amplification, you send 60 byte requests, which generate responses larger than 6,000 bytes.

Put these two properties together, and you get a great attack vector: you can send tiny, cheap requests to a server, which don't cause any incoming traffic on your attacker machine, and which send large quantities of data to your target. Doing this is called a DNS amplification attack: it's an amplification attack which uses properties of DNS to generate large quantities of data send to your server, using small quantities of data sent by your attackers.

That's exactly what happened to Spamhaus last week. The attackers used a very common DNS extension, which allowed them to amplify 60 byte requests into 4,000 byte responses, and to send the responses to the spamhaus servers.

There are, of course, more details. (For example, when direct attacks didn't work, they tried an indirect attack that didn't target the spamhaus servers, but instead tried to attack other servers that spamhaus relied on.) But this is the gist.

• Scientopia Blogs


text:
# RC filtering circuits

Heya! I've got a lab practical tomorrow to do with RC circuits, what do the terminals at the top and bottom of a filter circuit diagram mean, I'm assuming Vin and Vout is the voltage in and then 0V/ground, but what do the bottom two terminals correspond to?

Cheers

The bottom left and right are the return only. $V_{in}\;$ is input voltage across the left two terminals from the top terminal to the bottom terminal on the left. $V_{out}\;$ is the output voltage across the right two terminals from the top terminal to the bottom terminal on the right.

format: plain


text:
»  Home »  Products »  Features »  Panel data »  Generalized estimating equations

## Generalized estimating equations: xtgee

The use of panel-data models has exploded in the past ten years as analysts more often need to analyze richer data structures. Some examples of panel data are nested datasets that contain observations of smaller units nested within larger units. An example might be counties (the replication) in various states (the panel identifier). Other examples of panel data are longitudinal, having multiple observations (the replication) on the same experimental unit (the panel identifier) over time. xtgee allows either type of panel data.

Stata estimates extensions to generalized linear models in which you can model the structure of the within-panel correlation. This extension allows users to fit GLM-type models to panel data.

xtgee offers a rich collection of models for analysts. These models correspond to population-averaged (or marginal) models in the panel-data literature.

What makes xtgee useful is the number of statistical models that it generalizes for use with panel data, the richer correlation structure with models available in other commands, and the availability of robust standard errors, which do not always exist in the equivalent command.

In this example, we consider a probit model in which we wish to model whether a worker belongs to the union based on the person's age and whether they are living outside of an SMSA. The people in the study appear multiple times in the dataset (this type of panel dataset is commonly referred to as a longitudinal dataset), and we assume that the observations on a given person are more correlated than those between different persons.

. webuse nlswork
(National Longitudinal Survey.  Young Women 14-26 years of age in 1968)

. xtset idcode
panel variable:  idcode (unbalanced)

. xtgee union age not_smsa, family(binomial) link(probit) corr(exchangeable)

Iteration 1: tolerance = .05859927
Iteration 2: tolerance = .00346479
Iteration 3: tolerance = .0001277
Iteration 4: tolerance = 4.486e-06
Iteration 5: tolerance = 1.548e-07

GEE population-averaged model                   Number of obs     =     19,226
Group variable:                     idcode      Number of groups  =      4,150
Family:                           binomial                    min =          1
Correlation:                  exchangeable                    avg =        4.6
max =         12
Wald chi2(2)      =      30.23
Scale parameter:                         1      Prob > chi2       =     0.0000

union        Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]

age      .0045624   .0013959     3.27   0.001     .0018264    .0072984
not_smsa     -.1440246   .0318838    -4.52   0.000    -.2065156   -.0815336
_cons     -.8770284   .0479603   -18.29   0.000    -.9710288   -.7830279

### xtgee options

xtgee allows these options:

 Families Bernoulli/binomial gamma Gaussian inverse Gaussian negative binomial Poisson Links cloglog identity log logit negative binomial odds power power probit reciprocal Correlation structures independent exchangeable autoregressive stationary nonstationary unstructured user-specified

Assume an independent correlation structure that ignores the panel structure of the data. Under this assumption, xtgee will produce answers already provided by Stata’s nonpanel estimation commands. Examples of situations when xtgee provides the same answers are given in the table shown below.

 Family Link Correlation Equivalent Stata gaussian identity independent regress gaussian identity exchangeable xtreg, re (see note 1) gaussian identity exchangeable xtreg, pa binomial cloglog independent cloglog (see note 2) binomial cloglog exchangeable xtcloglog, pa binomial logit independent logit or logistic binomial logit exchangeable xtlogit, pa binomial probit independent probit (see note 3) binomial probit exchangeable xtprobit, pa nbinomial nbinomial independent nbreg (see note 4) poisson log independent poisson poisson log exchangeable xtpoisson, pa gamma log independent streg, dist(exp) nohr (see note 5) family link independent glm, irls (see note 6)
 Note 1 These methods produce the same results only for balanced panels. Note 2 For cloglog estimation, xtgee with corr(independent) and cloglog will produce the same coefficients, but the standard errors will be only asymptotically equivalent because cloglog is not the canonical link for the binomial family. Note 3 For probit estimation, xtgee with corr(independent) and probit will produce the same coefficients, but the standard errors will be only asymptotically equivalent because probit is not the canonical link for the binomial family. If the binomial denominator is not 1, the equivalent maximum-likelihood command is bprobit. Note 4 Fitting a negative binomial model using xtgee (or glm) will yield results conditional on the specified value of alpha. nbreg, however, estimates that parameter and provides unconditional estimates. Note 5 xtgee with corr(independent) can be used to fit exponential regressions, but this requires specifying scale(1). As with probit, the xtgee-reported standard errors will be only asymptotically equivalent to those produced by streg, dist(exp) nohr because log is not the canonical link for the gamma family. xtgee cannot be used to fit exponential regressions on censored data. Using the independent correlation structure, xtgee will fit the same model as the glm, irls command if the family–link combination is the same. Note 6 If xtgee is equivalent to another command, using corr(independent) and the vce(robust) option with xtgee corresponds to using vce(cluster clustvar) option in the equivalent command, where clustvar corresponds to the panel variable.

If you choose to model the intracluster correlation as an identity matrix (by specifying the name of an existing identity matrix in the option corr), GEE estimation reduces to a generalized linear model, and the results will be identical to estimation by glm.

. glm union age not_smsa, family(gauss) link(identity)

Iteration 0:   log likelihood = -10713.086

Generalized linear models                         No. of obs      =     19,226
Optimization     : ML                             Residual df     =     19,223
Scale parameter =   .1784791
Deviance         =  3430.904127                   (1/df) Deviance =   .1784791
Pearson          =  3430.904127                   (1/df) Pearson  =   .1784791

Variance function: V(u) = 1                       [Gaussian]
Link function    : g(u) = u                       [Identity]

AIC             =   1.114749
Log likelihood   = -10713.08631                   BIC             =  -186185.1

OIM
union        Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]

age     .0018369   .0004926     3.73   0.000     .0008714    .0028024
not_smsa    -.0648492   .0067672    -9.58   0.000    -.0781126   -.0515858
_cons     .1950571   .0158061    12.34   0.000     .1640777    .2260365

. xtgee union age not_smsa, family(gauss) link(identity) corr(indep)

Iteration 1: tolerance = 6.777e-16

GEE population-averaged model                   Number of obs     =     19,226
Group variable:                     idcode      Number of groups  =      4,150
Family:                           Gaussian                    min =          1
Correlation:                   independent                    avg =        4.6
max =         12
Wald chi2(2)      =     103.63
Scale parameter:                  .1784513      Prob > chi2       =     0.0000

Pearson chi2(19226):               3430.90      Deviance          =    3430.90
Dispersion (Pearson):             .1784513      Dispersion        =   .1784513

union        Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]

age     .0018369   .0004926     3.73   0.000     .0008715    .0028023
not_smsa    -.0648492   .0067666    -9.58   0.000    -.0781116   -.0515869
_cons     .1950571   .0158049    12.34   0.000     .1640801    .2260341

We could fill up lots of space demonstrating other ways that xtgee is equivalent to other features of Stata, but the real power is in using it for its intended use and modeling the correlation that exists in the panels.


text:
# Questions about the validity of a certain model for tidal resonance at the Bay of Fundy

I'm currently working on a (relatively-simple) fluid-dynamics model of the tidal resonance at the Bay of Fundy, a place which is known for his world-breaking tidal range (15 meters on average, while the historical record is 21.6 meters - at year 1869). After a lot of effort, I succeeded in calculating the $$M2$$ periodic-component (semi-diurnal component) of the component of the horizontal tidal force along the north-east; from looking at maps of the Bay of Fundy, I gained the impression that the longitudinal axis of its body of water is directed approximately at 45 degrees north to the eastern direction, so it's in the north-east direction (which is 45 degrees to both the north and the east).

In the process of calculation I took the following values for the relevant sizes:

• Declination of the Moon's orbit relative to the ecliptic - $$5$$ degrees.
• Declination of the Earth's axis relative to the perpendicular to the ecliptic - $$23.5$$ degrees.
• Latitude of the Bay of Fundy - $$45$$ degrees.

Taking into account only the lunar tidal force (without the solar), the result is:

$$T_\text{NE}=\frac{3GM_\text{moon}R_\text{earth}}{2L^3}\cdot 0.473\cos(\frac{2\pi}{\Delta T_\text{tide}}t),$$

where $$\Delta T_\text{tide} = 12 \text{h} 25 \text{m}$$.

Now, I wanted to estimate the horizontal movement of a vertical slice of water at the Bay of Fundy using the model of a forced harmonic oscillator with the northern-eastern tidal force as the forcing function. Under the assumption of negligible damping parameter, the result is supposed to be (on basis of the solution to the forced harmonic oscillator):

$$\Delta x = \frac{\frac{3GM_\text{moon}R_\text{earth}}{2L^3}\cdot 0.473}{\omega^2_\text{tide}-\omega^2_\text{seiche}}$$

The horizontal translation of the water slice together with its depth and width at the entry to the bay determines the amount of water flowing into the bay, and together with the length of the bay enables one to get a rough approximation of the tidal range at the narrow part of the bay (where the tides reach their peak).

My problem is about getting accurate data on the geometric dimensions of the bay as well as its resonance period - taking the estimate $$\Delta T_\text{seich}= 12\text{h} 40 \text{m}$$ gives too low values for the the tidal range - it amounts to only about $$3$$ meters! In addition, according to Wikipedia, in each tide cycle an amount of water of approximately $$100,000,000,000\ m^3$$ flows into the bay; if we combine this with the depth of water at the entrance of the bay ($$200\ m$$) and the width of its mouth ($$52\ \text{km}$$) we can conclude that the horizontal movement of a vertical slice of water (which is twice the amplitude) is about $$9.5\ \text{km}$$, much more than the result of my formula. Also, the peak depth-averaged tidal current velocity at the bay (at its narrow part) is about $$4.5\ m/s$$, and if we compare it with $$\Delta x \cdot \omega_{tide}$$ than it's about 50-60 times larger (and still significantly larger if we also take into account the fact that the channel becomes narrower and shallower).

Therefore, as a first step of checking this model, I'd like to get accurate data on the geometric dimensions of the Bay of Fundy:

• What is the exact length of the Bay of Fundy? I mean the length of that part of the bay with the resonant properties. I ask because there are inconsistencies between the different sources - Encyclopedia Britannica says it's 151 km long, another source states 220 km, another 290 km, and I even found a source claiming 400 km.
• What is the depth of water at the entrance to the bay? What is the average depth at the bay?
• I'm aware that this isn't an easy question to answer experimentally, but what is the resonance period (seiche period) at the bay? I'll be glad if anyone will redirect me to good sources on the resonance period.

Finally, if the problem isn't with the accuracy of the data available on web, I want to know what is the flaw in my model.

• Does your model include location and tides levels from ports around the bay? Actual recordings may be of course better, than «just» the forecast of a tide chart like this about Port Lorne. May 30, 2021 at 21:15
• @Buttonwood - in my model, i divided the total tidal range into two parts: 1. "global" tidal range - the water rise around the bay (without the resonance effect), which i assumed is about 1 meter, and the resonant part which is calculated by the procedure outlined in my post, and is equal to only about 2 meters. If we take into account that the bay becomes narrower (at a ratio of about 1 to 3) it might add a factor of two (so tidal range is supposed to be 5 meters), but its still far from regular measurements there (about 14 meters). May 30, 2021 at 21:34
• Just a comment: Tsunami wave in open ocean are not that high, sometimes 1-2 meters, when they arrive on the coast then they rise to the spectacular 10/15 meters. Is your simplified model providing 3 meters taking into account such an effect? Ps: if you can provide in the comment pictures of such a tide I would be happy :) ! Jun 1, 2021 at 5:07
• @EarlGrey - in my model i didn't take into the account the effect of shallowing of the water in the bay, and thanks for the comment. Anyway, i believe it to be a marginal effect - there are many shallow water coasts in the world, but only those with the resonant effect experience amplification of tides. I think this effect might amplify the tides a bit more. Meanwhile, i found an article which says that the M2 component of tidal force isn't the only one with resonance with the bay - there is an additional N2 component. Jun 1, 2021 at 12:47
• And a pic of such a tide can be found in wikipedia article "tidal resonance" - en.m.wikipedia.org/wiki/Tidal_resonance. Jun 1, 2021 at 12:49


text:
# VS Code online and networked

Here be tips for using VS Code as a collaborative cloud-friendly editor, in several senses.

## “Remote” editing

Remote edits code in particular environments from a local VS Code instance - including spinning up containers or SSH sessions, so that your editor and execution environments can be different. The remote execution environments can be elsewhere on the internet, or they can be local VMs, which is why I have “scare quotes” around the word “remote”. VS Code tries hard to make this seamless, installing some helper files on the remote side that interact smoothly with the local side. Occasionally some extensions will not work gracefully across the barrier between local and remote practices but for most of my workflows this problem does not arise. This is generally the easiest and most mainstream option, and I use it all the time.

This is best explained by using the tutorial examples.

TutorialDescription
Remote via SSHConnect to remote and virtual machines with Visual Studio Code via SSH.
Work in WSLRun Visual Studio Code in Windows Subsystem for Linux.
Develop in ContainersRun Visual Studio Code in a Docker Container.
GitHub CodespacesConnect to a codespace with Visual Studio Code.

Also I wrote one about getting remote debugging for python working, see below.

Pro tip: remote editing via unreliable or intermittent network connection can be a drag. There is no universal Persistent SSH session solution at time of writing. It would be nice if it worked over eternal terminal but it does not. However, Christopher LaPointe’s solution using tmux seems OK for now.

tl;dr:

{
"terminal.integrated.profiles.linux": {
"tmux": {
"path": "/usr/bin/tmux",
"args": [
"new-session",
"-A",
"-s",
"vscode-${workspaceFolderBasename}" ], }, }, "terminal.integrated.defaultProfile.linux": "tmux", "remote.SSH.defaultForwardedPorts": [ { "localPort": 9002, "remotePort": 9002, "name": "SA" } ], } ## VS Code server Slightly weirder: run a server process that makes VS Code accessible to you in a browser. No local copy of VS Code is required; you use a browser window for everything. I have not had need of this yet, because any scenario where I have HTTPS browser access to a remote server, I have also had SSH access and so it has been easier to use the VS Code Remote setup. However, a completely browser-based IDE might be useful in some circumstances: in exotic devops scenarios, through painful firewalls, when working from a thin client, something like that? There are at least two forks of VS code designed to be a server-side editor. Code-server by Coder: The Developer Workspace Platform seems actively maintained and popular. AFAICT it is Linux-only. Alternatively, gitpod-io/openvscode-server seems very similar and has an explanatory blurb: VS Code has traditionally been a desktop IDE built with web technologies. A few years back, people started patching it in order to run it in a remote context and to make it accessible through web browsers. These efforts have been complex and error prone, because many changes had to be made across the large code base of VS Code. Luckily, in 2019 the VS Code team started to refactor its architecture to support a browser-based working mode. While this architecture has been adopted by Gitpod and GitHub, the important bits have not been open-sourced, until now. As a result, many people in the community still use the old, hard to maintain and error-prone approach. At Gitpod, we’ve been asked a lot about how we do it. So we thought we might as well share the minimal set of changes needed so people can rely on the latest version of VS Code, have a straightforward upgrade path and low maintenance effort. ## Google Cloud Does this use the same stack as the previous ones? Maybe. GoogleCloudPlatform/cloud-code-vscode: Cloud Code for Visual Studio Code: Issues, Documentation and more Cloud Code for VS Code brings the power and convenience of IDEs to cloud-native Kubernetes and Cloud Run application development. Cloud Code works with Google’s command-line container tools like skaffold and kubectl under the hood, providing local, continuous feedback on your project as you build, edit, and run your applications locally or in the cloud. ## Collaborative editing A different type of networking: Not using my IDE to access remote files, but using my local IDE to share the editing experience with others. Many users can access the same VS code session at the same time and collaborate on the same code edits (not just, say, the same files). There is a real-time collaboration extension called liveshare Welcome to Visual Studio Live Share! Live Share enables you to collaboratively edit and debug with others in real time, regardless of the programming languages you're using or app types you're building. You can instantly and securely share your current project, start a joint debugging session, share terminal instances, forward localhost web apps, have voice calls, and more! Unlike traditional pair programming, Visual Studio Live Share allows developers to work together, while retaining their personal editor preferences (e.g. theme, keybindings), as well as having their own cursor. This allows you to seamlessly transition between following one another, and being able to explore ideas/tasks on your own. This ability to work together and independently provides a collaboration experience that feels much like in-person collaboration. Live Share is built using revolutionary architecture and concepts that manifest as powerful features for our users. To learn about those features, see Live Share features. TIP: Did you know that you can now join a Live Share session from the browser? This means, you no longer need to install a desktop client to collaborate! Just click on the link shared with you and you can get a full-fidelity VS Code editor experience in the browser. Learn more here. ## Case study: Remote python debugger Having said that everything is seamless, there are in fact edge cases where there divide between what is here on the client and what is there on the remote server, becomes important. One of those cases is interactive debugging. Here is a worked example of what I needed to do to get a remote python debugger working from my local machine. Finding out why code that works locally does not work in the actual target deploy environment. First need to install the debugpy package in our project. How to install depends how we are managing your python environment, but it will probably be one of these: pip install debugy ## 👈🏻 this one for me conda install debugpy poetry add debugpy Now I choose a network port for the debugger.1 This port needs to be unique to me (in the sense that no-one else on the remote machine should be using it) and it should be between 1024 and 65535. For simplicity’s sake I will assume the port is 48888, but I would recommend against that number for you because someone else might have copy-pasted these instructions and got there first, if you are using a shared machine. (Although not me; I use different port numbers in reality, this is just a tutorial example). Next, we do a one-time setup of VS Code to know that it should look for that port by finding the debug tab in VS Code and adding a configuration. This will open launch.json into which we can add a configuration for a remote attach debugger: { "name": "Attach", "type": "python", "request": "attach", "connect": { "host": "127.0.0.1", "port": 48888 }, "pathMappings": [ { "localRoot": "${workspaceFolder}",
"remoteRoot": "."
}
]
}

This part is clunky, but after one-time setup everything goes smoothly. VS Code will do all the remaining necessary networking by magic.

So NOW WE ARE READY TO DEBUG. Almost. We still need to load up our code on the server to USE the debugpy module. Two options here.

Firstly, we could invoke the code from the vs code terminal like this:

python -m debugpy --listen 0.0.0.0:48888 ./myscript.py

Alternatively, we could load and execute the debugging system inside myscript.py:

import debugpy
debugpy.listen(('0.0.0.0', debug))
debugpy.wait_for_client()

Either way, VS code will detect what I am doing and magically open up an SSH tunnel from here to there. Now, if you did the wait_for_client thing, the code will pause and wait for for a debugger connection.

OK great, so now I select that debug configuration I created earlier in order to get my debugger to connect

I was stuck for ages on this. but now I ALSO NEED TO CLICK THE GREEN TRIANGLE or it doesn’t start. And then… That’s it. we are done.

The debugger will show what is happening in general, but it is especially useful when

1. we hit a line that raises an exception, or
2. the code hits the line breakpoint() which will cause the debugger to drop into the code right there.

How does the debugger work? I do not need to explain because microsoft has already done that. But tl;dr there is now a lot of information about my running code, a graph of the current call stack, a little tab to inspect the current value of variables etc.

1. Technically we need to choose a port on the local AND remote machine, and these can be different, but why do that?↩

### No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.


text:
# Approximating a discrete probability distribution with a standard normal distribution

Let us approximate a discrete distribution by a standard normal distribution, without using a continuity correction factor. Let $X$ be a random variable with discrete distribution, and $Y$ be a random variable with standard normal distribution. Since we did not use a continuity correction factor, can we say that the $P(X \geq x)$ is always greater than or equal to its approximated probability by the standard normal distribution?

• Yes, for normal approximations to binomial. Details and an example in my Answer below. – BruceET Jun 18 '18 at 9:29
• No (did you intend less than or equal?) - take the simple example of tossing two fair coins and letting $X$ be the number of heads. Then $P(X \gt 1) = \frac14$ for two heads, while the normal approximation at the expectation of $1$ would suggest $\frac12$ – Henry Jun 18 '18 at 14:28

If the discrete random variable $X$ takes integer values, then $$P(X > x)= P(X \ge x+1) = P(X \ge x+.5)$$ The continuity correction would use the third expression when using a continuous distribution as an approximation.

Ordinarily, the approximating continuous distribution would have positive probability in the interval $[x, x+.5].$ In that case using the continuity correction will give you a smaller approximated value.

Example: Suppose $X \sim \mathsf{Binom}(n = 64, p = 1/2)$ and you seek $P(X > 30).$ The exact value is $P(X > 30) = 1 - P(X \le 30) = 0.6460096.$

1 - pbinom(30, 64, .5)
##  0.6460096

If you use $P(X^\prime > 30) = 1 - P(X^\prime \le 30)$ as an approximation, where $X^\prime \sim \mathsf{Norm}(\mu = 32, \sigma=4),$ you will get $P(X > 300) \approx 0.6914625.$

1 - pnorm(30, 32, 4)
## 0.6914625

But if you use the continuity correction, you will use $P(X^\prime > 30.5) = 1 - P(X^\prime \le 30.5) = 0.6461698.$ Hence, your approximation will be $P(X > 30) \approx 0.6461698.$ This is smaller than the value 0.6914625 without the continuity correction. It is also closer to the exact binomial probability.

1 - pnorm(30.5, 32, 4)
##  0.6461698

Usually in textbook examples you can expect about two decimal places of accuracy from a continuity-corrected normal approximation to a binomial distribution. To four decimal places, the exact value in this example is 0.6460 and the continuity-corrected normal approximation is 0.6462. (Here we get three-place accuracy; approximations are often best when $p \approx 1/2.$)

The figure below shows relevant binomial probabilities (vertical bars) and the approximating normal density curve. Notice that be binomial probability $P(X = 31)$ is approximated by the area under the normal curve above the interval $[30.5, 31.5].$ The uncorrected approximation wrongly includes the vertical strip between $x = 30.0$ and $x=30.5$ under the normal curve.

Note: The values I have shown are from R statistical software. If your normal approximations are obtained by standardization and using a printed normal table, then results will be slightly different because of the rounding entailed in the use of the table.

• Thank you. So you mean that we can't say for sure that P(X>=n) is greater than its approximated value by normal distribution? – Carol Jun 18 '18 at 17:31
• I checked P(X>=n) and normal distribution always gives the smaller value. Is there any intuition or proof behind that? – Carol Jun 18 '18 at 17:50
• I'm agreeing with you. The difference in my example is the area under the normal curve above the interval $[30.0, 30.5],$ as mentioned just before the figure. To four places, compare normal probabilities $P(X^\prime > 30) = 0.6915$ without continuity correction with $P(X^\prime > 30.5) = 0.6462$ with continuity correction. For intuition, start with obvious fact that $P(X^\prime > 30) > P(X^\prime > 30.5).$ (Maybe shade in the two areas under the normal curve.) – BruceET Jun 18 '18 at 21:47
• Thank you. I mean the probability value when we don't use continuity correction factor is always less than the "true probability value" for P(X >= x). In fact, without using correction factor we are underestimating the true probabilities. – Carol Jun 19 '18 at 1:42
• Now you've changed it to $P(X \ge x),$ which is a slightly different situation. I have tried to set the principles out clearly in my answer, and I'll leave the variations to you. – BruceET Jun 19 '18 at 2:17

Hint:

Consider a binomial distribution with parameters $n, p$.

• What is $P(X \gt n)$ for the actual binomial distribution?
• Is the normal approximation less than or equal to this?


text:
# warp v0.1.0

0

0th

Percentile

## Group Dates

Tooling to group dates by a variety of periods including: yearly, monthly, by second, by week of the month, and more. The groups are defined in such a way that they also represent the distance between dates in terms of the period. This extracts valuable information that can be used in further calculations that rely on a specific temporal spacing between observations.

# warp

The goal of warp is to provide tooling to group dates by a variety of periods, such as: yearly, monthly, by second, by week of the month, and more.

library(warp)

## Installation

You can install the release version from CRAN with:

install.package("warp")

You can install the development version from GitHub with:

# install.packages("devtools")
devtools::install_github("DavisVaughan/warp")

## Example

One of the core functions in warp is warp_distance(), which allows you to provide a date time vector and compute the “distance” from an origin. For example, this computes the number of months from the unix epoch.

x <- as.Date("1970-01-01") + -2:2
x
#> [1] "1969-12-30" "1969-12-31" "1970-01-01" "1970-01-02" "1970-01-03"

warp_distance(x, period = "month")
#> [1] -1 -1  0  0  0

The values that warp_distance() returns correspond to the distance from x to the origin, in units defined by the period and the width defined by every. The origin defaults to the unix epoch of 1970-01-01 00:00:00 in the time zone of x, but you can change that. In this case the distances are saying that, for example, "1970-01-02" is in the same month as the origin, and "1969-12-31" is 1 month group away.

You can also compute daily distances. Rather than grouping by 1 day, let’s lump every 2 days together, starting from the default origin.

# Groups 1970-01-01 and 1970-01-02 together
warp_distance(x, period = "day", every = 2)
#> [1] -1 -1  0  0  1

You will often want to set your own origin date. Let’s shift it forward 1 to 1970-01-02.

origin <- as.Date("1970-01-02")
origin
#> [1] "1970-01-02"

# Groups 1970-01-02 and 1970-01-03 together
warp_distance(x, period = "day", every = 2, origin = origin)
#> [1] -2 -1 -1  0  0

Another interesting period to group by is the "mweek", i.e. the week of the month. Notice that days 1-7 of January 1970 are grouped into the same bucket. Also note that days 29-31 of December 1969 fell at the end of their corresponding month. This irregular week of size 3 is treated as the 5th week of that month, but the offset value of -1 is still the number of week buckets from the origin of 1970-01-01.

y <- as.Date("1969-12-28") + 0:14

tibble::tibble(
y = y,
mweek = warp_distance(y, "mweek")
)
#> # A tibble: 15 x 2
#>    y          mweek
#>    <date>     <dbl>
#>  1 1969-12-28    -2
#>  2 1969-12-29    -1
#>  3 1969-12-30    -1
#>  4 1969-12-31    -1
#>  5 1970-01-01     0
#>  6 1970-01-02     0
#>  7 1970-01-03     0
#>  8 1970-01-04     0
#>  9 1970-01-05     0
#> 10 1970-01-06     0
#> 11 1970-01-07     0
#> 12 1970-01-08     1
#> 13 1970-01-09     1
#> 14 1970-01-10     1
#> 15 1970-01-11     1

## Inspiration

The algorithm for warp_distance() was inspired by xts::endpoints().

## Functions in warp

 Name Description warp_distance Compute distances from a date time origin warp_change Detect changes in a date time vector warp_boundary Locate period boundaries for a date vector warp-package warp: Group Dates No Results!


text:
# The dynamic allocation problem

### Avoiding memory allocation

caution

Smart contracts must avoid dynamic allocation. Due to the performance penalty incurred by dynamic allocation, the MultiversX Virtual Machine is configured with hard limits and will stop a contract that attempts too much allocation.

Here are a few simple guidelines you can use to ensure your contract performs efficiently. By following them, you might notice a considerable reduction of gas consumption when your contract is called. It is also likely that the WASM binary resulting from compilation may become smaller in size, thus faster and cheaper to call overall.

### It's all about the types

Many basic Rust types (like String and Vec<T>) are dynamically allocated on the heap. In simple terms, it means the program (in this case, the smart contract) keeps asking for more and more memory from the runtime environment (the VM). For small collections, this doesn't matter much, but for bigger collection, this can become slow and the VM might even stop the contract and mark the execution as failed.

The main issue is that basic Rust types are quite eager with dynamic memory allocation: they ask for more memory than they actually need. For ordinary programs, this is great for performance, but for smart contracts, where every instruction costs gas, can be quite impactful, on both cost and even runtime failures.

The alternative is to use managed types instead of the usual Rust types. All managed types, like BigUint, ManagedBuffer etc. store all their contents inside the VM's memory, as opposed to the contract memory, so they have a great performance advantage. But you don't need to be concerned with "where" the contents are, because managed types automatically keep track of the contents with help from the VM.

The managed types work by only storing a handle within the contract memory, which is a u32 index, while the actual payload resides in reserved VM memory. So whenever you have to add two BigUints for example, the + operation in your code will only pass the three handles: the result, the first operand, and the second operand. This way, there is very little data being passed around, which in turn makes everything cheaper. And since these types only store a handle, their memory allocation is fixed in size, so it can be allocated on the stack instead of having to be allocated on the heap.

caution

If you need to update older code to take advantage of managed types, please take the time to understand the changes you need to make. Such an update is important and cannot be done automatically.

### Base Rust types vs managed types

Below is a table of unmanaged types (basic Rust types) and their managed counterparts, provided by the MultiversX framework:

Unmanaged (safe to use)Unmanaged (allocates on the heap)Managed
--BigUint
&[u8]-&ManagedBuffer
-BoxedBytesManagedBuffer
ArrayVec<u8, CAP>1Vec<u8>ManagedBuffer
-StringManagedBuffer
--TokenIdentifier
-MultiValueVecMultiValueEncoded / MultiValueManagedVec
ArrayVec<T, CAP>1Vec<T>ManagedVec<T>
[T; N]2Box<[T; N]>ManagedByteArray<N>
-AddressManagedAddress
-H256ManagedByteArray<32>
--EsdtTokenData
--EsdtTokenPayment

In most cases, the managed types can be used as drop-in replacements for the basic Rust types. For a simple example, see BigUint Operations.

We also recommend allocating Rust arrays directly on the stack (as local variables) whenever a contiguous area of useful memory is needed. Moreover, avoid allocating mutable global buffers for this purpose, which require unsafe code to work with.

Also, consider using ArrayVec, which provides the functionality of a Vec, but without allocation on the heap. Instead, it requires allocation of a block of memory directly on the stack, like a basic Rust local array, but retains the flexibility of Vec.

caution

Make sure you migrate to the managed types incrementally and thoroughly test your code before even considering deploying to the mainnet.

tip

You can use the mxpy contract report command to verify whether your contract still requires dynamic allocation or not.

1. Be careful when passing arrays around, since they get copied when returned from functions. This can add a lot of expensive memory copies in your contract.
2. ArrayVec allocates on the stack, and so it has a fixed capacity - it cannot grow indefinitely. You can make it as large as you please, but be warned that adding beyond this capacity results in a panic. Use try_push instead of push for more graceful error handling.