Comments on: The motivated simulator argument

By: Specifying “enlightened judgment” precisely (reprise) | Ordinary Ideas

Wed, 27 Aug 2014 01:28:22 +0000

[…] and it’s hard to say how that would end. This is the human version of the problem I described here; to the extent that humans can avoid such problems it shouldn’t really be any more of a […]

By: Specifying a human precisely (reprise) | Ordinary Ideas

Specifying a human precisely (reprise) | Ordinary Ideas — Sun, 24 Aug 2014 21:43:38 +0000

[…] significant concern with this procedure is the one I discussed before, essentially that Solomonoff induction might end up believing the simulation argument and […]

By: “Indirect Normativity” Write-up | Ordinary Ideas

“Indirect Normativity” Write-up | Ordinary Ideas — Tue, 30 Jul 2013 21:51:02 +0000

[…] thought experiment might be occurring? This possibility is particularly troubling in light of the incentives our scheme creates—anyone who can manipulate H’s behavior can have a significant effect […]

By: Eliezer Yudkowsky

Eliezer Yudkowsky — Thu, 13 Jun 2013 20:59:43 +0000

I remark that if the problem can be framed such that a crisp AI, running on a logical decision theory with a solved problem of maximization vantage points with a no-blackmail equilbrium, would regard the simulations constructed as being of negative utility, then the AI will refuse to be motivated by the simulations so as not to give the simulators an incentive to construct those simulations in the first place. The problem arises when the simulators can offer the AI something, such as the certain press of a reward button, such that the AI prefers the universes in which the simulations were constructed. It should also be kept in mind that however you construct the AI so as not to be thus motivated, the simulators are trying to find a way to make it a positive-sum bargain from the AI’s perspective regardless of your intentions, so you must be very confident indeed that the AI disprefers every possible scenario in which is simulated.

By: Generalized abstract incentives problems, and simulation arms races « Ordinary Ideas

Thu, 06 Dec 2012 03:05:01 +0000

[…] if it didn’t cooperate, the balance of power on Earth would be non-negligibly altered). The trouble is that such an agent can be manipulated by applying even relatively modest amounts of […]

By: Will Newsome

Will Newsome — Sat, 26 May 2012 19:52:00 +0000

Though note that if there’s a transcendent creator deity then “original worlds” aren’t obviously more important than nor distinguishable from simulated worlds, depending on the preferences of the creator deity. Given that hypothesis the stellar resources might not be “out there” even in the vanishingly small fraction of worlds where humans are the first agents to gain access to the cosmic commons.

(I’ll note that one should withhold judgment on this idea until one has seriously considered the relevant theological arguments. Surprisingly few people seem to understand theology, even though it’s clearly one of humanity’s most important fields of inquiry. Politics is the mind-killer.)

By: paulfchristiano

paulfchristiano — Fri, 25 May 2012 15:31:31 +0000

Three responses:

1. Humans should generally expect the simulations of them to continue obeying the same laws they always have (until there is some reason not to expect it any more).
2. Humans care about their ability to influence the world, which is much larger in basements (this accounts for why, e.g., I probably don’t care so much about simulations) I think this is the principled magic that allows you to not care about simulations, and you can pull it back from UDT recommendations to subjective anticipations.
3. I think insofar as humans care about the probable continuations of their experiences, they should mostly be concerned with simulations.

By: Vladimir Slepnev

Vladimir Slepnev — Fri, 25 May 2012 08:28:26 +0000

But humans are sensors too! How do you know that you care about the real world rather than about all the simulators which have incentives to mess with you? That should be especially difficult if you’re an AI researcher :-) If humans can minimize the impact of meddling simulations by using some sort of magic of subjective anticipation, then maybe we should figure out how that magic works and use it in our AIs…