This blog contains a lot of material that is very weird. A natural question is: why write about such weird things? Many of the situations I’ve considered or problems I’ve written about involve improbable hypotheticals. We probably won’t ever have a powerful utility maximizer which we must provide a precisely defined utility function, for example. Why not focus on designing systems that are likely to actually get built?
The reason should not be so unfamiliar to a mainstream academic: though these problems may be too simple to be directly applicable, thinking about them sheds light on fundamental philosophical and mathematical issues. The heuristic is something like: where there are simple, fundamental problems on which it is possible to make progress, you should do it, even if you can’t see exactly why or how that progress will be useful.
I don’t know what the future will look like. But I’d guess it will be influenced profoundly by the development of machine intelligences of some sort. What kind of understanding is necessary, to ensure that this influence is positive? I don’t know that either, but it seems quite likely that we will have to face much hairier, real-world analogs of questions like “if we had a powerful utility maximizer, what utility function might we want to give it?” or “if we had an oracle that could answer formally specified questions, what would we do with it?” Maybe working on these simpler questions is a waste of time–maybe we should wait until we understand better what the real questions will be–but there aren’t that many simple questions in the world. As long as we only waste time on fundamental problems, we at least won’t waste too much time. And it seems to me that there may be relatively deep theory underlying these questions–if that’s the case, we may not want to count on our ability to figure it all out when the time comes.
This isn’t so different from the state of affairs in any other theoretical field. Why do we care about having a polynomial time algorithm for secure multi-party computation? It probably isn’t going to ever get used (and I could use more extreme examples, which definitely won’t get used). But because the question is so simple, and seems fundamentally related to the sort of thing we are likely to want to do in the future, we expect (I think rightly) that a deeper understanding is likely to be independently useful, and to suggest fruitful directions for further study.