Comments on: Approval-seeking

By: MIRI's September Newsletter | Machine Intelligence Research Institute

MIRI's September Newsletter | Machine Intelligence Research Institute — Thu, 09 Oct 2014 20:06:07 +0000

[…] to MIRI’s Friendly AI interests: “Three impacts of machine intelligence,” “Approval-seeking [agents],” “Straightforward vs. goal-oriented communication,” “Specifying a human […]

By: Keith Henson

Keith Henson — Wed, 03 Sep 2014 04:38:24 +0000

I have talked about approval or status seeking as motivation for AIs for a long time, perhaps ten years and about that being a prime and relatively benign human motivation for even longer. Happy to see the meme getting attention/expansion.

Old stuff here, http://www.cryonet.org/cgi-bin/dsp.cgi?msg=30170

Here is how I treated it in fiction:

“As soon as the seed finished the dish (after consulting its clock,
its GPS location and the place of the sun), it aligned the dish on
the African net communication transponder attached to the
geosynchronous ring and asked for a permanently assigned address on
the net. Up to that point the clinic seed was a generic
product. The address it was assigned was just a string of
hexadecimal numbers but it was a _unique number_! The clinic’s
personality was human in that it could feel happy, even smug, about
acquiring its very own _unique identification_.

“The clinic had other carefully selected human personality
characteristics such as seeking the good opinion of its peers (humans
and other of its kind alike). It also had a few unhuman limits.

More if you google “keith henson” status AI evolutionary psychology

The way you presented it is more rigorous than what I did.

Thanks.

Keith Henson

By: Specifying “enlightened judgment” precisely (reprise) | Ordinary Ideas

Wed, 27 Aug 2014 01:28:18 +0000

[…] Approval-seeking […]

By: Specifying a human precisely (reprise) | Ordinary Ideas

Specifying a human precisely (reprise) | Ordinary Ideas — Sun, 24 Aug 2014 21:43:29 +0000

[…] “maximize the extent to which I would approve of your decision upon reflection.” (I have suggested that we would be happy with a powerful AI that made decisions according to this […]

By: Assorted links

Assorted links — Sun, 27 Jul 2014 00:26:24 +0000

[…] “Approval-seeking“: Christiano’s latest post on indirect normativity for AGI. […]

By: Wei Dai

Wei Dai — Wed, 23 Jul 2014 11:26:28 +0000

I’m not saying it’s a deal-breaker either, since that of course depends on what alternatives are available. But one seemingly novel practical implication is that if some form of indirect normativity is likely to be our best bet for controlling AIs, we need to prioritize efforts to figure out whether we would adopt selfish values if we were to undergo a lot of reflection, and if so exactly what form those selfish values would take. (See http://lesswrong.com/lw/8gk/where_do_selfish_values_come_from/ for a related post that tries to make some progress in this direction, but mainly just points out how much we don’t understand about selfishness.)

>for example choosing a person who we think is willing to implement a particular strategy that would respect our egoist values, rather than pursuing their own values

This may not work, no matter who we choose, if for example egoism is objectively correct in some sense, or if all humans just have a psychological tendency to converge towards egoism after extensive reflection.

By: paulfchristiano

paulfchristiano — Wed, 23 Jul 2014 03:38:40 +0000

I agree this is a concern, perhaps a serious one, and it probably warrants more discussion. But the damage is bounded to the extent of our personal preferences for a very precise form of egoism, rather than concern for the people we might have become under the counterfactual. I am not too concerned about huge divergence (so that I no longer sympathize with the simulated person, or am very uncertain about the extent to which they are “like me”), because I can make critical decisions (in the counterfactual simulation) prior to such divergence occurring. For example, I can settle on a definition of “upon reflection” which I think is more likely to preserve the egoist parts of my values. So for the most part, this is something I am willing to accept. I’m also willing to sacrifice a lot on very narrow egoist values (e.g. those that value me over the person I would become upon reflection) in exchange for gains to people generally like me.

There are other approaches to dealing with this if we thought it was a big deal, for example choosing a person who we think is willing to implement a particular strategy that would respect our egoist values, rather than pursuing their own values. Or perhaps choosing a group that would be willing to do so. On this reading, the only point of referring to a person is as a kind of bootstrapping to refer to the conceptual procedure which is locked up in their head.

So overall I am happy to keep this in mind as a pain point, but I don’t see it as a dealbreaker. And as you observe, it doesn’t seem to be a big distinction between this account and CEV, or any other plausible-seeming accounts on the table.

By: Wei Dai

Wei Dai — Wed, 23 Jul 2014 00:26:43 +0000

Hi Paul, did you see the comment I left at http://lesswrong.com/r/discussion/lw/c0k/formalizing_value_extrapolation/b0rb? To generalize that point, I don’t currently know whether “I as an input/output process” actually has the same values as “I as a specific physical entity” so I’m not sure it’s safe to hand over control to the former.