Intuitively, it seems that any simple hypothesis should receive a reasonable prior probability. It seems unreasonable to assign a simple hypothesis a ridiculously low probability. In some sense this is a matter of preference, but here I’ll briefly argue:
If H is a simple hypothesis and the prior probability of H is very small, then there is nothing you could possibly see which would convince you of H.
Which suggests that this is at a minimum a very dangerous thing to do (though I’m open to the possibility that there are things you should simply never believe, no matter what you see).
The argument is basically a formalization of the platitude that you can only ever get so much evidence, because hey, it’s not that unlikely that you are crazy. In fact that claim is false, as Jacob Steinhardt was quick to point out when I made it recently. The possibility that you are crazy or in a simulation doesn’t preclude you making arbitrarily large updates. It just precludes making arbitrarily large updates in favor of simple hypotheses.
First, by “simple hypothesis” I mean a hypothesis which has a short description in whatever internal language you use to represent hypotheses. So “God exists” and “God doesn’t exist” count, so long as your brain assigns them subjective probabilities.
Second, by “prior probability” I really mean the probability you assign at some arbitrary point in time, in particular after updating on all of your introspective access. Then the result will imply that at no future time can you come to believe H, without forgetting the evidence you have already encountered.
Third, by “very small” I mean much smaller than 2-|H|. And by “much smaller” in that sentence I mean by a large factor (which doesn’t depend on H). 2100 should be plenty.
Fourth by “convince of X” I mean “cause you to believe X as the result of a correct Bayesian update.”
Finally, I’m going to make the assumption that you don’t assign negligible probability to skeptical scenarios (like the simulation hypothesis, insanity, or your experiences being an elaborate hoax). This can be justified by observing that such skeptical scenarios apply to a non-negligible fraction of observers in many reasonable worlds (for example, physical universes like ours). In fact, in many reasonable worlds skeptical scenarios apply to all observers. So in order to assign skeptical scenarios negligible probabilities, you’ve got to assign a whole lot of otherwise reasonable worlds negligible probabilities.
Having made those definitions the claim is now pretty straightforward. Consider some arbitrary evidence E you might receive, which might constitute substantial evidence in favor of H. I’ll exhibit a hypothesis Q such that P(H) << P(Q) and P(E|H) = P(E|Q), where P is your prior. This implies that P(H|E) / P(Q|E) = P(H) / P(Q) << 1, so in particular P(H|E) << 1.
Q is the hypothesis: H is false, however your experiences are manufactured according to the distribution P( * | H). Clearly then P(E|Q) = P(E|H). To see that P(Q) >> P(H), consider the claim S: a hypothesis H’ was selected uniformly at random using my internal representation of hypotheses, and then my experiences were manufactured according to the distribution P( * | H’).
If you assign skeptical hypotheses reasonable probability, P(S) should not be not too small. Moreover, P(Q|S) is at least 2-|H|. Thus P(Q) >> P(H), as desired.