I would hesitate a great deal before entering an argument with either Larry Wasserman or Eliezer Yudkowsky, but here goes.

You're right that if, for whatever reason, someone is fascinated by "coverage" then confidence intervals will answer their questions better than Bayesian posteriors. But I think Eliezer's right that there's something very wrong with thinking that "coverage" in this sense is what matters.

Let's consider your example again. In what circumstances is the following actually a useful problem to solve? "Given an observation of one thing from a box, tell me a set of box-types in such a way that for each box-type you'll choose a set including the right one at least 70% of the time."

I can think of some. For example: a mad scientist starts sending you boxes, with instructions to start guessing; he's going to monitor your results on each box-type and if he sees you getting any type of box wrong more than about 30% of the time he'll kill you. Otherwise he'll reward you for nominating fewer box-types each time. But (1) that's a desperately contrived situation and (2) the most diehard Bayesian, in that situation, will produce something like "confidence intervals" because that's what Bayesian decision theory says to do.

Is there any not-so-contrived situation where the problem solved by confidence intervals is actually an important one?

By the way, my best guess about the Rule / Theorem thing is that he's distinguishing between a theorem about conditional probabilities, and a normative rule saying "when you get new information, update your beliefs like so".

> the most diehard Bayesian, in that situation, will produce something like "confidence intervals" because that's what Bayesian decision theory says to do.

I disagree that this is what "Bayesian" decision theory says to do. It's what decision theory says to do, and it's what math says to do, and it's what the constraints require. It's not particularly "Bayesian" -- it's just what you have to do.

If everything that happens to be the correct answer (including frequentist confidence intervals when called for) is now described as Bayesian, then the term has no meaning and we are all Bayesians. :-)

> Is there any not-so-contrived situation where the problem solved by confidence intervals is actually an important one?

What would you do in the case of my 100 robots, where you want 70 of them to come to the correct decision, and they have to make their decisions independently? Having them all calculate a posterior independently works terribly (as I showed, 80% of them come to the wrong conclusion with >73% belief). Confidence intervals work a heck of a lot better.

The optimal approach would consider what single algorithm works best when run independently. Finding these solutions (on, e.g., a decentralized POMDP) is an open problem.

I called it Bayesian because it describes what Bayesians will do. It does indeed also describe what anyone else sane will do. The point is that in order to make confidence intervals the right answer you need a situation weird enough to make even Bayesians use confidence intervals.

In the case of your 100 robots, why am I supposed to want 70 of them to come to the correct decision? This seems just like my mad-scientist example: contrived to force confidence intervals (or something very like them) to be the right answer. Can you explain in what sort of situation this would be a sensible thing to care about?

So explicitly state the property you do want to optimize for in your robot example, and state your prior belief, and then crank the handle on the Bayesian reasoning machine to obtain your optimal answer.

You may get an intractable problem that you can't solve exactly, and for a carefully cherry picked objective the frequentist answer might even be a good approximation.

What if you're an Italian seismologist, and you want to produce a prediction which is in some way useful, but still expresses an appropriate level of doubt to a lay audience?

> All mathematicians and engineers have access to the whole world of theorems and algorithms and techniques, all of them true, or meeting their specifications.

Only crazy mathematicians would think nothing of a set of tools that contradict each other. If I recall correctly, frequentist methods often yield different (and therefore contradictory) results depending on the way you look at the data. That's crazy.

I also have read your link: the prior problem can easily be solved by just communicating the likelihood ratios. Those are pretty much indisputable, and the actual contribution of the paper. Let the others start from their own priors. We have to chose one anyway.

Bayesian methods are also inconsistent wrt infinitely small changes in priors. See Wasserman's blog for more details. If you're worried about the Likelihood Principle causing "frequenting" inconsistency then you're taking a bitter pill in that not everyone actually believes the LP, the assertions leading to it can be debated, and, further, Likelihoodist methods (like most of Fisher's work) are not affected by it while still worrying themselves with coverage instead of posteriors.

> Bayesian methods are also inconsistent wrt infinitely small changes in priors. See Wasserman's blog for more details.

That looks like it would falsify the method instantly. Or are you talking about merely small changes in priors? Anyway, I can't find the details you speak of, do you have a link?

As for the likelihood principle, my math is rusty at the moment, I'll check. I expect however that I will just accept this principle as obvious. People don't believe it? They probably had the wrong teachings, just like most religious people. I'm not sure I know on a gut level how hopelessly wrong people often are. I'm not sure I properly feel the fear that should come with the possibility that I am hopelessly wrong. But I have an idea.

I agree with you and Larry and want to elaborate. Dogmatic Bayesians often come armed with assertions that probability calculus is the "correct" way of doing inference and, further, that there is some kind of ethical maximum achieved when you make your decisions exclusively via inference.

They have strong reason to feel this way, of course. Probability calculus is a very elegant theory that can be constructed from rather sane "desiderata" a la Cox's Theorem. Furthermore, if you think of ethics as an optimization problem, which is fair, then you're likely to be attracted to decision theory as a powerful tool for being more ethical. QED?

Unfortunately, neither of these reasons have the conclusive force that they appear to. The sanity of Cox's postulates and ethics of optimization are still (arbitrary) choices which serve mostly to create a formalized world for us to live in and study with efficacy. There is no doubt that people have derived great power from similar use of powerful models, but models they are nonetheless.

Asserting that modeling the world based on these postulates is ethically mandatory is weird. Feeling confident that you can become more powerful than others by doing so is a bet.

So, assuming you're a betting type instead of a religious type, would you rather use models that are elegant but incomplete or practical and battle-tested? I believe both Sampling philosophy and Bayesian philosophy to be mathematically elegant. I also have seen them both very practically used.

And once you're interested in measuring practical performance of these methods, once you decide to pick things like loss functions and philosophies of measurement, then it's easy to find places where either framework fails.

---

So really, in my mind, there are Bayesian methods and Frequentist methods and Bayesians who, probably following Jaynes' hyperbolic and recommended work, feel that there is a certain theory of mind which is correct due to its elegance. Most Engineers can pick between the methods and most mathematicians can pick between the theories.

And having a favorite theory isn't so bad either. I quite like some "Likelihoodists" as well and learn from them every time we talk.

---

I think the best model of the situation is that of logic. It's like we have "First-order logicians" and "Higher-order logicians", each of which espousing a viewpoint that a particular Theory is "more true" than the other. Truly, they both have fortes and foibles, though, and the practical mathematician studies both and uses whichever one lets them prove what they need to. The problem comes into play when "higher-order logicians" suggest that "first-order logicians" are inherently wrong since their very theory prevents them from being able to conceive of a better theory and this reticence is leading to disarray. But then the "higher-order logicians" can't come to a consensus on how to fix things and also take a potentially infinite amount of time to reach a conclusion.

Really, though, our thought is neither bound to first or second order logic. Perhaps this causes us pain from time to time, but it also lets us pick and choose.

> Dogmatic Bayesians often come armed with assertions that probability calculus is the "correct" way of doing inference and, further, that there is some kind of ethical maximum achieved when you make your decisions exclusively via inference.

Put it more crudely, "Bayesians are a bit extremist, therefore we shouldn't trust them too much".

> Unfortunately, neither of these reasons have the conclusive force that they appear to. The sanity of Cox's postulates and ethics of optimization are still (arbitrary) choices…

I have read the first two chapters of Jayne's Book, so I must ask: do you know of any other choice that isn't completely insane? You need very few assumptions to get to probability theory.

> …which serve mostly to create a formalized world for us to live in and study with efficacy.

Last time I checked, it looked like our world runs on math (which math is the big question). But even if it doesn't, do you expect we can find anything better to study it? Even if the world is chaotic, it doesn't mean our thinking shouldn't be lawful.

Haynes is intuitively convincing, but intuition isn't everything. If you buy his interpretation of his desiderata and buy his assertions (like: plausibility is a real) then its convincing to believe that his argument leads to probability---though I think Cox's Theorem as presented by Jaynes has been disproved? I think it still holds in another form though.

The dogmatism is not that we should forsake math, but instead that we should accept any One True Math. History is littered with the broken careers of those who did so.

Disproved?! frantic search… Oh, you meant for infinite sets. Somehow I'm not surprised. When one can make a 1-1 correspondence between even numbers and natural numbers… Maybe that's one reason why Nick Bostrom says Infinite Ethics is hard? http://www.nickbostrom.com/ethics/infinite.html

Plausibility is a real… What else could it be? I still need something continuous, quantifiable, and totally ordered. Why, might one say? I'm not sure I can explain. Some things we just take for granted. Only a rock would take nothing for granted.