I wrote a bit about the unconvinceability issue here.

For example, you say “In real life, extreme likelihood ratios for extreme improbabilities are rather common, e.g., what was the prior probability of my typing this exact paragraph?” But the magic in that case was that the hypothesis that you would write this exact paragraph is a very complex hypothesis. It is easy to get lots of evidence for complex hypotheses, and much harder (and in extreme cases impossible) to get lots of evidence for simple hypotheses. My intuition is that 33 bits is not many for complicated hypotheses, but it is an awful lot for simple hypotheses. Maybe you disagree? I’m not sure if I’m slicing things up the right way, and it would be cool if my views shifted a lot.

I think one-sided skepticism is justified based on anthropic considerations, for particular simple indexical assertions like “I am super special and my decisions significantly affect 10^100 other decision-makers.” This is pretty much just the simulation argument—if you think are the one in 10^100, you need to think about how many of the 10^100 are delusional, and how many are the one.

You can’t stretch “but lots people could come up with some other clever-sounding justification” that far. The richest man in the world really is special, and he knows that while maybe there are O(10) other people who can come up with similarly compelling arguments for their own specialness, there aren’t O(100). Similarly, the world’s most impressive academic by popular vote knows there are maybe O(100) similarly poised people but not O(1000). [I made up numbers, but hopefully the idea rings true.]

Anyway, the problem with PM is that under computational-complexity formulations of priors, the probability decreases vastly more slowly than the utilities increase in size. In your introduction, the problem is that people trying to apply a calibration-overconfidence principle will find themselves unable to drive down their priors very far. Hanson’s reply, if valid, is a solution to PM because it restores the balance of prior probability falloff vs. utility increase. It would work the same way for the problem you introduced if we just decided that we’re allowed to actually say “seven billion to one” for prior odds when there’s a reference class of known size that large, the same way we’re actually allowed to say “125 million to one” for lottery tickets.

In Hanson’s original solution to PM, though, we get the problem that we are basically *never* allowed to believe in the Mugger even if they part the heavens with a gesture, show us the machine running reality and give us a careful explanation of exactly why they decided to present us with this problem. I am still not sure how to resolve this one.

A similar but not quite analogous problem of unconvinceability would apply if we allowed “seven billion to one” without overconfidence adjustments for the prior of helping the world, but then started applying lots of clever-sounding overconfidence adjustments whenever somebody tried to build up a likelihood ratio in favor of being able to help the world – e.g. “Oh, sure, you scored over a million to one on that test of mathematical ability, but maybe someone else has some other clever-sounding justification for thinking they can help the world.” In this case the problem seems to stem from a one-sided skepticism in which we’re allowed to assign very extreme prior odds without worrying about overconfidence, but then we’re not allowed to use any extreme likelihood ratios to climb back up. In real life, extreme likelihood ratios for extreme improbabilities are rather common, e.g., what was the prior probability of my typing this exact paragraph? Making it more difficult to climb out of the prior improbability of saving the world must imply some special skeptical burden beyond that involved in a mere -33 bit prior or so – unlike the PM case, 33 bits of info wouldn’t ordinarily be difficult to obtain unless there was some special epistemic difficulty associated with getting extreme likelihood ratios. We can find candidates for what these special epistemic difficulties might be, but then the situation has moved beyond what’s analogous to Pascal’s Mugging.