Comments on: Hazards for Formal Specifications

By: Specifying a human precisely (reprise) | Ordinary Ideas

Specifying a human precisely (reprise) | Ordinary Ideas — Sun, 24 Aug 2014 21:43:36 +0000

[…] significant concern with this procedure is the one I discussed before, essentially that Solomonoff induction might end up believing the simulation […]

By: Solomonoff Induction and Simulations « Ordinary Ideas

Solomonoff Induction and Simulations « Ordinary Ideas — Thu, 24 May 2012 16:56:34 +0000

[…] desired information, and then applying Solomonoff induction to pinpoint that continuation. In some earlier posts I have written about the following objection: Solomonoff induction applied to sequence of […]

By: On the Difficulty of AI Boxing « Ordinary Ideas

On the Difficulty of AI Boxing « Ordinary Ideas — Fri, 27 Apr 2012 03:12:59 +0000

[…] likely to be manufactured by computationally simple agents trying to mess with our universe. (See hazards.) For example: we control one copy of this AI, and we’ll reward it if it answers our question […]

By: jsteinhardt

jsteinhardt — Tue, 24 Jan 2012 06:55:09 +0000

“One possibility is interference from an alternative Everett branch in which a singularity went badly.”

How would interference occur on macroscopic scales? That seems like it would be extraordinarily difficult.

“Indeed, to the extent that we as humans believe that our own recollections are enough to provide substantial evidence about the world, and to the extent that we believe that the Solomonoff prior is a reasonable model for our own predictive frameworks, we must believe that our own brains (complete with our recollections) are most concisely described by modeling the universe that produced those recollections.”

I’m confused by this. Could you indicate concretely how you would specify a brain more concisely by first describing the universe?

I also don’t really follow the TDT part at all. Why is the TDT agent being run on the input given to the human in the box? Is it supposed to be the AI that locates a human decision theory? In that case, why would it have such an arbitrary goal, and why would it choose to take over the universe?

By: Avoiding Simulation Warfare with Bounded Complexity Measures « Ordinary Ideas

Thu, 22 Dec 2011 01:36:15 +0000

[…] some decisions and conditioning the universal prior on agreement with those decisions (see here). I have argued that the behavior of the result on new decisions is going to be dominated by the winner of a […]

By: Formal Instructions « Ordinary Ideas

Formal Instructions « Ordinary Ideas — Thu, 22 Dec 2011 01:00:08 +0000

[…] We’ve maybe gotten some leverage on the first parts (though right now the difficulties here loom pretty large), which involve precisely defining certain concepts for an AI, but it isn’t […]