Possible Failures

Coercion by Simulator

Consider a very intelligent AI pursuing a reward signal (apparently) controlled by humans. A standard concern is that at some point the AI would prefer to take control from the humans and use this control to give itself reward.

Instead, the AI may decide that it is quite likely to be in a simulation and begin behaving unpredictably. For example, the AI may decide that the vast majority of agents who share its experiences are living in simulations controlled by uFAIs which successfully took over universes like ours. Such uFAIs may adopt policies like intervening in simulations to administer or withhold reward according to whether the reward-seeking AI takes actions which facilitate a uFAI-takeover. Moreover, it may begin these interventions precisely when the AI discovers this line of argument.

At first it seems like a TDT-style agent trying to maximize a mathematically defined utility is not susceptible to this sort of attack (except insofar as utility may be defined in terms of a mathematical object like the universal distribution which can be manipulated by a simulator).  But unfortunately this is not clear.

A TDT agent in general will need to rely on its environment to do useful computation, although the details of this relationship are not yet clear. An AI running on a large computer should probably be able to reason about possible worlds in which that computer behaves in a wide variety of ways (for example, counterfactual or simulated worlds in which the computer violates its ordinary contract) and should reason about how the AI’s behavior in those possible worlds affects utility. Understanding the possible effects of this uncertainty seems out of reach for now, but it does seem plausible that a simulator’s threat could control actions even of some TDT-style agents (in retrospect we would probably reject a decision theory with this property, so these considerations just suggest developing a much better understanding of decision theory).

Altruism

Although we imagine an AI as maximizing a given utility function, if the AI is able to reason about counterfactuals and acausal influence, it may make decisions with concern for their effects not only if it were in a different situation but if it had a different utility function. This may be particularly problematic when we try and use self-interest to control AI behavior. For example, suppose we build a system comprised of many AI’s such that one (which we do not yet know) has the ability to guarantee for itself its largest possibility utility. If we rely on that AI choosing to maximize its own utility, we may be surprised. Natural decision theories may make decisions from behind a veil of ignorance with respect to the definition of utility itself, helping others even when they have no use for the generosity of others.

Pascal’s Wager

On reflection, humans seem to be uncertain about their endorsed response to the possibility of very large payoffs arising with negligible probability (particularly when that probability comes entirely from radical model uncertainty). For example, many utilitarian-types feel like they are indifferent to a gamble which would equiprobably increase or decrease the number of happy future humans by 50%, but are horrified by the possibility of a galaxy-scale AI investing nearly all of its resources exploring improbable possibilities for exotic physics or escape from a simulation, and even more objectionable outcomes are conceivable. If there is are a priori conceivable outcomes which seem 3^^^3 better than the best outcome which seems mathematically conceivable upon inspection, should an AI focus exclusively on possible worlds where its mathematical reasoning is untrustworthy?

These problems may be fixed by an appropriate choice of values or dissolved by more careful philosophical consideration, but it is also possible that they are debilitating for certain decision theories. For example, VNM notwithstanding, choosing actions which maximize a mathematically/algorithmically defined utility function may not be a tenable design decision if we want our agent to entertain logical uncertainty (e.g. about whether the defined utility function is bounded).

Game Theory

AIs we build may encounter other agents in our future light cone. Moreover, if they have utility functions that care about other possible (or impossible) worlds they are very likely to reason about agents in other possible worlds.

In such situations we want our AI to capture gains from trade and other advantages from interaction, but we are also concerned about the possibility of blackmail or manipulation. In the worst and most bizarre cases, counterfactual uFAIs may blackmail an AI we create into producing very little value. Handling these situations correctly–cooperating on appropriate one-shot prisoner’s dilemmas, refusing to give into threats when appropriate, and so on–seems to depend on decision theory rather than values.

Stranger Things than These

I don’t know how strange the world is, or what sort of completely unexpected thing can go wrong. If someone had suggested acausal trade to me a year ago, or implied that a simulator might be able to control the behavior of a carefully constructed agent, I would not have taken the possibility seriously. And if no one had suggested it to me, I would have confidently endorsed schemes that failed for completely unanticipated reasons. I can certainly imagine that there is more room for weirdness. It is very hard to imagine formally ruling out the possibility (as it seems to require understanding decision theory for our own use, first, to see what should qualify as a proof). We can begin to imagine all sorts of strangeness, ranging from agents inhabiting and controling the formal systems we (shouldn’t, but did) trust, to more exotic forms of philosophical angst leading to erratic behavior.

Punting Decision Theory

We may leave the problem “figure out decision theory” to an AI with some naive but (hopefully) good enough decision theory. Hopefully such an AI will in turn figure out decision theory, create a new agent with a more robust decision theory, and thereby avoid the above problems. There are a few issues here, though:

Early Difficulties

An agent may encounter philosophically troubling or game-theoretically challenging ideas while still running a flawed decision theory. For example, as it is about to start running the new TDT agent it built it might discover an argument that no matter what it does it can’t get utility more than X, and by not self-modifying it can get utility X, so call off the modification and become trapped by its decision theory.

Suboptimal attractors

Agents with some decision theories, even decision theories which seem manifestly suboptimal to humans, will not fix their problems if given the chance. For example, an agent vulnerable to Pascal’s muggings may never remove this vulnerability (because in its view it is an opportunity to capture immense value). Or, we could imagine a CDT agent modifying into some sort of “exceptionless decision theory:” do what the original agent would have done in your situation. Someone running exceptionless decision theory may still be vulnerable to blackmail from an adversary who commits appropriately.

It may be necessary to get some aspects of decision theory right at first if we want to avoid the failures above in the long run.

Value Drift

It is possible that an AI trying to maximize the utility function U using one decision theory will eventually produce an AI running an improved decision theory but with some modified (in the worst case unrecognizably) utility function U’. For example, if an early decision theory leads to insufficiently cautious self-modification, the utility function may drift randomly; if the early AI encounters philosophical difficulties that lead it into existential despair, the utility function may only be concerned with worlds in which the AI’s early reasoning or perceptions were delusional.

Before building FAI you built an oracle AI to help you. With its help, you found a mathematical definition of U, the utility of humanity’s extrapolated volition (or whatever). You were all pretty pleased with yourselves, but you didn’t stop there: you found a theory of everything, located humanity within it, and wrote down the predicate F(X) = “The humans run the program described by X.”

To top it off, with the help of your oracle AI you found the code for a “best possible AI”, call it FAI, and a proof of the theorem:

There exists a constant Best such that U ≤ Best, but F(FAI) implies U =  Best.”

Each of these steps you did with incredible care. You have proved beyond reasonable doubt that U and F represent what you want them to.

You present your argument to the people of the world. Some people object to your reasoning, but it is airtight: if they choose to stop you from running FAI, they will still receive U ≤ Best, so why bother?

Now satisfied and with the scheduled moment arrived, you finally run FAI. Promptly the oracle AI destroys civilization and spends the rest of its days trying to become as confident as possible that Tic-Tac-Toe is really a draw (like you asked it to, once upon a time).

Just a lighthearted illustration that decision theory isn’t only hard for AI.

(Disclaimer: this narrative claims to represent reality only insofar as it is mathematically plausible.)

Edit: I think the moral was unclear. The point is: in fact F(FAI), and so in fact U = Best so U ≤ Best. Everything was as claimed and proven. But this doesn’t change the fact that you would rather not use this procedure.

Loebian Problems in Decision Theory

Consider an agent trying to maximize U, deciding whether to take an action X. The agent knows that U = 0 if it doesn’t take X, but is uncertain about its utility if it takes X.

A central difficulty in decision theory is: if as a matter of mathematical fact the agent doesn’t take X, how is it supposed to reason about “what U would be if I take X?”

One approach is to exploit an agent’s uncertainty about its own future action. Although the agent’s action can be determined (by simulation), the agent doesn’t know its own outcome or the resulting utility, but may be able to discover some relationships between them. As long as the agent remains uncertain in this way, we can represent “what would happen if I take X” by conditioning my uncertainty about my own action.

Even though the agent cannot simulate itself to determine its own action, this approach is still problematic, due (at least) to the following reasoning:

Consider the statement L =  “If I found a proof of L and still did X, it would be terrible.” If I found a proof of L then I could also prove that I’d proven L, and so I could prove “If I did X, it would be terrible.” But then I wouldn’t do X, and so in this case it would be vacuously true that “If I did X, it would be terrible.”

But now I’ve shown that if I could prove L and still did X, it would be (vacuously terrible). This is precisely the statement L, so L is true and I’ve just proven it. So if I do X, it would be terrible. I guess I won’t do X.

So the behavior of our agent gets dominated by details of how it does the proof search (which have been obscured in the above argument, but obviously must play an important role) which is terrible.

Blackmail and TDT

Suppose you are running some reflectively consistent decision theory and I am considering blackmailing you (and you are in the agent’s situation, trying to maximize U and considering whether to take action X). I tell you:

If you take action X, I will go out of my way to make sure that U is negative.

and make what you consider a credible commitment to do so. You are deciding how to respond: based on my commitment, it is clear that taking action X will give you negative utility, so as a good consequentialist you shouldn’t. But if your decision theory is reflectively consistent, you will (probably) realize that your decision in this situation may have two different types of consequences. In the “real” world, it determines your “real” action and therefore your utility. But it also may influence the blackmailer’s predictions of your behavior. If the blackmailer will only attempt to blackmail you if he believes that you will give in, then the mathematical fact that you give in to blackmail will cause you to be blackmailed.

Weighing both of these consequences, you may decide that it is better not to give into the blackmailer. (Although this issue seems to be complex: if the blackmailer is also running TDT he will predict that you will try to understand the relationship between your cooperation and the blackmailer’s decision to blackmail, and if he predicts this correctly he may commit to blackmailing anyway so that you have “no hope” of avoiding the blackmail. But if you predict that the blackmailer will reason in this way, you will decide to commit to not cooperating even sooner, and so on. If anyone knows what happens, I don’t.) For example, if the blackmailer is running a consequentialist decision theory and is a good predictor, it seems clear that you shouldn’t give in (and that TDT as it has been informally described won’t).

Counterfactual Blackmail

Now suppose that you have a trusty “intuition module,” which is constrained to only output true statements but which is otherwise adversarial.

You are again deciding whether to take the action X, and your intuition module says to you:

If you take action X, you will receive negative utility.

If we view your intuition module as a separate entity, behaving adversarially, it seems clear that the situation is quite analogous to ordinary blackmail. If the statement is really true, then we shouldn’t take action X. But by being the sort of person who doesn’t take action X upon learning this fact, we are allowing the intuition module to honestly make such a statement. This is most clear if we operationalize the intuition model’s contract by saying it maximizes the utility function U’ defined by:

U’ = -infinity if the intuition module ever lies.

U’ = – U otherwise.

Then the intuition module will need to “check” if it can honestly output “If you take action X, you will receive negative utility.” Our decision about taking action X then has two effects (just like our decision in the ordinary blackmailing): it determines our real action, and it determines whether our intuition module can manipulate us in this way.

The Difficulty

Things get substantially more difficult when the intuition module is internal to the agent. How can the agent distinguish between the output of the intuition module (which might be a lie if the agent is operating within the intuition module’s counterfactual) and the reasoning which justifies the agent’s response to the intuition module’s output? At some point some reasoning must be trusted as the basis for a decision; if we expect that every thought might be an artifact of a counterfactual than we descend into madness.

One approach to this problem is to have a system of trusted beliefs which is “too small” to permit these difficulties, and to treat the external environment as adversarial (though subject to some mathematical restrictions, which you can manipulate by means of your trusted beliefs). It may be possible to carry out some TDT reasoning using a very small set of beliefs which can be trusted, and to bootstrap up to larger agents with better understanding by manipulating the environment.

Nothing here can be stated formally, but here is the mental imagery behind this intuition. Consider a human operator, inside a “robot TDT” controlled from the inside. The human is able to handle some TDT-style reasoning, like the sort outlined above, but it cannot in general reason about statements complicated enough to talk about the robot it is controlling. Nevertheless, by using this sort of minimal TDT reasoning, it may be able to direct the robot sensibly/reflectively consistently. For example, if the human sees that the robot has produced a proof that “If I take X I will receive negative utility” the human can understand that it might just be in a counterfactual, so its decisions may have consequences other than the obvious ones.

(To the extent that this mode of thinking is useful, I expect it to be as an intuition pump for discovering what arguments should really motivate action. The human’s position seems to be little more than a restatement of our current situation with respect to decision theory, which intuition might do better on.)

One approach to understanding myself within the world is to have a reductionist view of the world, and the understanding that I myself am embedded somewhere in this world, so that my uncertainty about my own behavior may couple to uncertainty about the universe in a particular way. I’ll model this as an agent who knows his own description and has a mathematically defined utility function (which may depend for example on the execution of certain programs which are logically related to the agent’s own output). This is a standard formalism. Note that the agent need not have explicit representations of the programs involved–it may simply reason about mathematical structures which it suspects may have this relationship with each other.

I’ll assume that the agent has a math-intuition module, in Wei Dai’s terminology. That is, a module which at any time can assign a “probability” to each mathematical statement. We’d like to build agents which work well as long as its math intuition module works well.

The decision rule I would like to use is the following: for each possible action X, estimate the expected utility if I (counterfactually) choose X. Then choose the action with the largest expected utility. The difficulty is in defining the counterfactuals appropriately.

One way to escape this problem is to use random decision making (a similar approach is necessary to salvage decision markets). That is, with probability p make a decision randomly, and then ask about the probable outcomes conditioned on yourself choosing X at random. This has the virtue that it makes counterfactual definitions nearly trivial: if our coins are really random, then we can talk directly about the possible worlds in which they came up differently without having to perform any (as yet somewhat mysterious) “counterfactual surgery.”

Unfortunately it is surprisingly hard to write down a formal agent who cares about things like “that coin I just flipped” (I believe that the approach I outlined to specifying counterfactuals may be made to work, but it is at least tricky). Instead we can consider abstract psuedorandomness, for example by asking cryptographic questions whose answers we know we could compute but whose answers we are currently uncertain about. This can be formalized by using the math intuition module, and it seems like it may work well if our intuition module has sufficient self-confidence regarding its beliefs about the sort of cryptographic challenges we are trying to use as coins (that is, using such reflective beliefs the math intuition module can treat a statement to which it assigns a 50% probability very nearly as if it were a “logical coin” which the universe hasn’t yet flipped–this sort of reasoning isn’t valid in general, see “Beliefs which are Always Wrong”.)

Here is an informal sketch of a decision procedure along these lines:

1) Using the math intuition module, find a statement X to which the math intuition module assigns a probability of about 2^-20 (say) whose truth can be determined in a reasonable amount of time, and an assertion Y to which the math intuition module assigns a probability of  roughly 1/2 . This can be accomplished by taking a bunch of cryptographic problems, and taking a conjunction of many uncertain assertions about their solutions. (There are classes of cryptographic problems such that the agent can learn the answer to any given question in a reasonable amount of time, but under mild assumptions it is guaranteed to have significant uncertainty about almost every particular problem until it has spent a modest amount of time thinking about it.)

2) Ask the math intuition module about your expected utility conditional on X AND Y, and about your expected utility of X AND NOT Y.

3) Evaluate X and Y.

4) If X AND NOT Y, output decision 1. If X AND Y, output decision 1. If NOT X, then output decision 0 or 1 according to whether the math intuition module’s estimate of your utility is higher when conditioning on X AND NOT Y or on X AND Y.

If X and Y don’t couple to your utility function except through your decision, then this approach seems to produce essentially correct (consequentialist, not timeless) decisions.

In Newcomb-like situations, this algorithm will one-box if the simulation is exact–because it is using logical “coins”, it will play randomly if and only if its simulation plays randomly–but it will not necessarily one-box against approximate simulations. Moreover, this effect manifests even when the predictor’s error is very low; for example, if the simulator isn’t able to guess what X the agent will choose, then the simulator can simply predict two-box and be correct with probability 1 – P(X).

Note that the point of using a math intuition module and reasoning about apparent “acausal” control was not to succeed at a limited class of Newcomb-like problems: the point was to make sensible decisions at all. Similarly, the purpose of randomness was to allow any consideration of counterfactuals at all.

Note that if X and Y couple with the agent’s utility through some mechanism other than the agent’s action, the agent may fail to behave reasonably. The main way that such a coupling could occur is if a mugger trying to control the agent’s behavior made decisions on the basis of X or Y, after learning that they will be used for the agent’s decision. But note that, once we have already introduced a mugger, normal CDT will fail in the same way.

Self-modeling

Existing formal decision theories may be described as “dualist”: they consider the rest of the world on one side of a divider and the agent on the other, interacting through some formally specified interface. The agent’s algorithm is absent from the world model, affecting reality only by its outputs. In reality, the agent is part of the world, governed by the same laws and interacting with the rest of the world through a complex and jagged interface. It has been remarked that AIXI cannot recognize itself in a mirror or refrain from using the computer it is running on for scrap metal.

Game theory provides a simple example of one concrete failure. Consider an iterated game of rock-paper-scissors in which the winner takes $1 from the loser, but either player may opt-out for a small penalty. After losing enough money, a human will start opting out, realizing that her opponent is exploiting some regularity in her thinking which she can’t eliminate. A traditional Bayesian consequentialist dualist is not capable of reasoning in this way: independent of its decision, it believes that the opponent is going to play some fixed distribution over their possible plays. Regardless of the agent’s logical uncertainty there is guaranteed to be some option which it believes has a non-negative expected value and which it therefore prefers to opting out.

Self-locating

Even if we had a compact physical Theory of Everything, an agent is still left with an important form of uncertainty: where is it located within that universe? For example, if I were to tell you right now that you inhabit a game of life board with some compactly specified initial conditions and your goal is to make as many of the squares alive as possible, what formalism could represent your uncertainty about the relationship between your sense experience and your knowledge of underlying reality or your goals? What about information from introspection, or from mere knowledge of your own existence? Traditional priors do not seem suited to this task, although our understanding of the situation is not yet good enough to say anything with confidence.

Although this problem apparently must be solved by any successful formal decision theory, it may be most productive to think about it independently. I am inclined to list it here anyway in part because Ambient Decision Theory seems to be the closest humans have come to getting a handle on the problem.

Computation-aware

Although we consider an agent’s output as its primary method for interacting with the world, its computation may interact with the world through other channels.

For example, if an agent is playing rock-paper-scissors against an adversary who has a complete model of the agent but limited computation, the agent should choose its move in a way that the adversary will be unable to predict. If an agent is attempting to acausally cooperate with a computationally weaker opponent, the agent needs to ensure that the weaker agent is still incentivized to cooperate (for example, the more powerful agent may want to make its decision in a simple way that can be understood by the weaker agent). If an agent learns that certain types of operations consume more power or dissipate more heat, it will want to change its use of those operations strategically.

Wei Dai has proposed dealing with these issues by first running a less computationally expensive algorithm (using a computation-oblivious decision theory) to determine what more expensive algorithm should be used to make a particular object level decision. Overall, this problem is the only one on the list which appears to be satisfactorily resolved, and further progress seems likely to clarify the difficulties of recursively self-modifying AGI but not to help with decision theory or FAI in particular.

Reflectively consistent

Existing formal decision theories seem to exhibit reflective inconsistency: an agent using one of these decision procedures would choose to replace itself by a suitable agent running a different decision procedure. To the extent that an AI is motivated and able to fundamentally modify its own decision procedure and goals (or create new agents with different decision procedures or goals), we have at best a very limited understanding of its behavior. Aside from philosophical interest, this provides a strong motivation for seeking decision theories which will not be immediately replaced by the agents using them.

This principle is well-illustrated by a slightly modified version of Parfit’s hitchhiker. Suppose that, unbeknownst to you, your life is currently in danger. A potential benefactor has the opportunity to save your life at great expense, but fortunately expects to acquire evidence which will convince you that he did in fact save your life. He plans to save your life, approach you with this evidence, and ask you to pay him $100. For better or for worse, this benefactor can reliably predict whether you will pay, and only intends to help you if he expects to receive $100. Many decision procedures fail to hand over the $100 after having been saved, because cooperation no longer leads causally to any benefits. But an agent using most of these decision procedures would rather be the sort of person who pays the $100 and gets saved, and would therefore immediately abandon its original decision procedure and adopt a new one.

Acausally cooperative

Consider two agents aware of each other’s decision theories (or of each other’s complete descriptions) who are faced with the following cooperation problem: each independently may pay $1 in order to give the other $2. Without the ability to enter binding contracts, agents using traditional decision procedures will generally decline to pay for the others’ advantage, despite the fact that both agents can receive an extra dollar if they both cooperate. However, when agents can reliably model each other they may be able to reach a Pareto efficient outcome.

We would like to find a formal and general decision theory which at least occasionally leads two agents using that theory to cooperate (without requiring the agents to be identical) but no such theory is known. The difficulty of this problem is suggested by the difficulty of cooperative game theory without transferrable utility. However, existing work on updateless decision theory seems to come tantalizingly (to me at least) close to coordinating acausal cooperation in some situations and at present no theoretical obstacles are known.

Acausal cooperation and reflective consistency appear to be closely linked: for example, we can view Parfit’s hitchhiker as a cooperation problem between an agent and the benefactor’s model of that agent. One distinction between these settings is that many open problems related to reflective consistency can be stated in a particularly “fair” model, in which payoff depends only on the input-output behavior of an agent, while acausal coordination seems to require that the agents are able to see each other’s source, and different intuitions suggest that acausal cooperation and reflective consistency are possible.

Personally, after losing at matching pennies a relatively small number of times I will suspect that the predictor is able to reliably to predict my behavior and so will begin opting out. We expect that a reasonable AI will behave similarly.  But we run into difficulties when we try and formalize the reasoning that led to our behavior.

For example, consider any Bayesian using a causal decision theory, who maintains an implicit distribution over possible world models and chooses an approximately utility maximizing action with respect to this distribution.

Regardless of the current distribution maintained by such an algorithm, at least one of the predictor’s plays must have probability at least 50%–if heads has probability at least 50% then the player would rather choose tails than opt out, and otherwise the probability of tails is at least 50% and the player would rather choose heads than opt out.

In order to choose to opt out, an agent must be able to model non-causal relationships between its behavior and the world model. A UDT agent would handle this decision properly, if UDT could be suitably pinned down, but it still leaves us with the interesting situation with respect to beliefs: *whatever* the agent believes about the predictor’s play, the agent will be wrong (or miscalibrated), at least in the sense that those beliefs should not be used to guide action. In order to motivate action, the agent should be reasoning about the predictor’s behavior *conditioned* on the agent’s choice

This may not seem so troubling. After all, in general we need to reason about conditional rather than unconditional probabilities in decision theory, and the inclusion of a predictor seems to make this an unfair testing ground for beliefs as such.

But consider a “math intuition model,” which assigns subjective probabilities to mathematical facts. We generally imagine such a module as dealing with beliefs rather than actions, but a math intuition module can be presented with an identical problem: for example, a math intuition module reasoning about arithmetic may face an arithmetic problem Q which encodes the intuition module’s own behavior, and the answer to the problem may be controlled by the module’s output in such a way that the module is always wrong about Q. (Q is the direct analog of Godel’s paradoxical sentence.) So it seems that any math intuition module must be systematically miscalibrated about certain facts and must be aware of that miscalibration, but is unable to do anything about it.

This suggests that we should, at the very least, probably cast the AI reflection problem not as “How can an agent avoid going crazy while being convinced of its own correctness” but rather “How can an agent justify expanding its own influence when it is convinced of its own fallibility?”

It has been observed on Less Wrong that a physical, approximate implementation of AIXI is unable to reason about its own embedding in the universe, and therefore is apt to make certain mistakes: for example, it is likely to destroy itself for spare parts, and is unable to recognize itself in a mirror. But these seem to be mild failures compared to other likely outcomes: a physical, approximate implementation of AIXI is likely to develop a reductionist world model, doubt that its decisions have any effect on reality, and begin behaving completely erratically.

Setup

Let A be an agent running on a physical computer, implementing some approximate version of AIXI. Suppose that A is running inside of an indestructible box, connected to the external world by an input wire W1 and an output wire W2.

Suppose that this computer exists within a lawful physical universe, governed by some rules which can be inferred by A. For simplicity, assume that the universe and its initial conditions can be described succinctly and inferred by A, and that the sequence of bits sent over W1 and W2 can be defined using an additional 10000 bits once a description of the universe is in hand. (Similar problems arise for identical reasons in more realistic settings, where A will work instead with a local model of reality with more extensive boundary conditions and imperfect predictability, but this simplified setting is easier to think about formally.)

Recall the definition of AIXI: A will try to infer a simple program which takes A’s outputs as input and provides A’s inputs as output, and then choose utility maximizing actions with respect to that program. Thus two models with identical predictive power may lead to very different actions, if they give different predictions in counterfactuals where A changes its output (this is not philosophy, just straightforward symbol pushing from the definition of AIXI).

AIXI’s Behavior

First pretend that, despite being implemented on a physical computer, A was able to perform perfect Solomonoff induction. What model would A learn then? There are two natural candidates:

• A’s outputs are fed to the output wire W2, the rest of the universe (including A itself) behaves according to physical law, and A is given the values from input wire W1 as its input. (Model 1)
• A’s outputs are ignored, the rest of the universe behaves according to physical law, and A is given the values from W1 as its input. (Model 2)

Both of these models give perfect predictions, but Model 2 is substantially simpler (around 10000 bits simpler, and specifying A’s control over W2’s values in 10000 bits seems quite optimistic). Therefore A will put much more probability mass on Model 2 than Model 1. In fact, Model 2 or its close variants probably receive almost all of the probability mass.

If A believed Model 2, what would its actions be? Well, in Model 2 A’s outputs are completely ignored! So although Model 2 may make up most of the probability mass, it contributes nothing important to the expected value computation determining A’s actions. So maybe A will make decisions as if it believed Model 1, and therefore behave appropriately for a surprising reason? In fact this is unlikely: there are many other models of intermediate complexity, for example:

• A’s outputs are usually ignored, the rest of the universe behaves physically, and then A is given the values from W1 as its input. However, if A ever outputs the sequence 0011011000111010100, then A’s inputs are modified to include a large reward signal on the next step. (Model 3)

Model 3 is an intentionally silly example, but hopefully the point is clear: Model 1 is enormously more complicated than the simplest explanation for A’s observations, and so there are many other (potentially very strange and arbitrary) models which account for the data but are simpler than Model 1. It is these models which will determine A’s behavior, and the result is almost certainly very arbitrary (this is similar to some discussions of Pascal’s Wager: it is possible that all of these strange models will cancel out and add up to normality, but it seems outlandishly unlikely).

For example, if A suspects Model 3 may be true it will be eager to test this hypothesis by outputting 0011011000111010100. It will find that it continues to receive inputs from W1, but no matter, it will move on to testing Model 4, which has the same form but with a different target bitstring. Of course this is not what A will actually do, as there will be other models with complicated effects on behavior, but this gives the flavor of A’s failure.

If A somehow did accept Model 1, then we would be back in the situation normally discussed on Less Wrong: A believes that the values on W2 are magically made equal to A’s outputs, and so is unconcerned with its own real physical instantiation. In particular, note that having some uncertainty between Model 1 and Model 2 is not going to save A from any of these problems: in the possible worlds in which Model 2 is true, A doesn’t care at all what it does (A doesn’t “want” its physical instantiation to be destroyed, but by the same token it believes it has no control), and so A’s behavior reduces to the normal self-destructive behavior of Model 1.

Approximate AIXI’s Behavior

An approximate version of AIXI may be able to save itself from existential despair by a particular failure of its approximate inference and a lack of reflective understanding.

Because A is only an approximation to AIXI, it cannot necessarily find the simplest model for its observations. The real behavior of A depends on the nature of its approximate inference. It seems safe to assume that A is able to discover some approximate versions of Model 1 or Model 2, or else A’s behavior will be poor for other reasons (for example, modern humans can’t infer the physical theory of everything or the initial conditions of the universe, but their models are still easily good enough to support reductionist views like Model 2), but its computational limitations may still play a significant role.

Why A might not fail

How could A believe Model 1 despite its prior improbability? Well, note that A cannot perform a complete simulation of its physical environment (since it is itself contained in that environment) and so can never confirm that Model 2 really does correctly predict reality. It can acquire what seems to a human like overwhelming evidence for this assertion, but recall that A is learning an input-output relationship and so it may assign zero probability to the statement “Model 2 and Model 1 make identical predictions,” because Model 1 depends on the indeterminate input (in particular, if this indeterminate was set to be a truly random variable, then it would be mathematically sound to assign zero probability to this assertion). In this case, no amount of evidence will ever allow A to conclude that Model 2 and Model 1 are identically equivalent–any observed equivalence would need to be the result of increasingly unlikely coincidences (we can view this as a manifestation of A’s ignorance about its own implementation of an algorithm).

Now consider A’s beliefs about W2. It is relatively easy for A to check (for almost all timesteps) that Model 1 correctly predicts each bit on W2, while A only has enough time to check that Model 2 correctly predicts only a few of these bits. Therefore the probability of Model 2 must be decreased by A’s estimate of the likelihood that Model 2 would happen to set the correct value for all of the bits that A didn’t have time to verify. Model 1’s probability must be decreased likewise, but because A was able to check more of Model 1’s values, Model 1 leaves less unexplained data and may not be as unlikely as Model 2.

Why A probably fails anyway

In general, it is very fragile to expect that an AI will behave correctly because it fails to realize something. There are a number of obvious reasons that A might accept the reductionist hypothesis, even acknowledging the above argument.

First, an effective approximate inference module seems unlikely to make the mistake described in the last section. Making this mistake seems to correlate with other errors which may disrupt normal “sane” behavior: in general, given a model M which the agent cannot exactly simulate, uncertainty about the M’s outputs (even if M is known to produce mostly the same output, and never known to produce incorrect output) causes the model’s probability to drop off exponentially with the number of unverified computations. It seems as though any “sane” AI should be able to assign some constant probability to the hypothesis that this model exactly captures the output, and thereby avoid considering other exponentially improbable hypotheses. This may relate to deeper concerns about approximate Solomonoff induction.

Second, A may be able to observe the operation of the hardware on which it is running. This will generally cause Model 1 to be much less probable: indeed, if A can observe even one “causal ancestor” of W2’s value, it will no longer gain very much by believing Model 1 as such (since now Model 1 only produces the correct output if Model 2 did anyway–all of the relative advantage for Model 1 comes from occasions when A can observe the value of W2 without observing the operations directly responsible for that value, which may be rare). Of course there are more complicated models in which A’s outputs control reality in more subtle ways, which may have better complexity. Understanding this issue completely depends on a much more detailed knowledge of A’s approximate inference and the nature of A’s observations. In general, however, being able to observe its own computation seems like it may be adequate to force A into a reductionist model.

Third, A’s approximate inference module may be aware of the fact that A’s own outputs are produced algorithmically (as a computational aid, not an underlying belief about reality). This would cause it to assign positive probability to the assertion “Model 2 is equivalent to Model 1,” and eventually force it into a reductionist model.

Conclusion

Agents designed in the spirit of AIXI appear to be extremely fragile and vulnerable to the sort of existential despair described above. Progress on reflection is probably necessary not only to design an agent which refrains from killing itself when convenient, but even to design an agent which behaves coherently when embedded in the physical universe.

Consider an agent A, aware of its own embedding in some lawful universe, able to reason about itself and use that reasoning to inform action.  By interacting with the world, A is able to modify itself or construct new agents, and using these abilities effectively is likely to be an important component of AGI.  Our current understanding appears to be inadequate for guiding such an agent’s behavior, for (at least) the following reason:

If A does not believe “A’s beliefs reflect reality,” then A will lose interest in creating further copies of itself, improving its own reasoning, or performing natural self-modifications. Indeed, if A’s beliefs don’t reflect reality then creating more copies of A or spending more time thinking may do more harm than good. But if A does believe “A’s beliefs reflect reality,” then A runs immediately into Gödelian problems: for example, does A become convinced of the sentence Q = “A does not believe Q”? We need to find a way for A to have some confidence in its own behavior without running into these fundamental difficulties with reflection.

This problem has been discussed occasionally at Less Wrong, but I would like to clarify and lay out some examples before trying to start in on a resolution.

Gödel Machines

The Gödel machine is a formalism described by Shmidhuber for principled self-modification. A Gödel machine is designed to solve some particular object level problem in its allotted time. I will describe one Gödel machine implementation.

The initial machine A has an arbitrary object level problem solver. Before running the object level problem solver, however, A spends half of its time enumerating pairs of strings (A’, P); for each one, if A’ is a valid description of an agent and P is a proof that A’ does better on the object level task than A, then A transforms into A’.

Now suppose that A’s initial search for self-modifications is inefficient: a new candidate agent A’ has a more efficient proof checker, and so is able to examine more proofs before defaulting to the original object level problem solver. Intuitively, A’ is an improvement over A, and this is (a very mild version of) the sort of self-modification which we would like our agents to perform. For example, a human is easily able to recognize that it can achieve higher utility by modifying itself to think faster. But we will see that under most conditions A cannot prove that A’ is an improvement, because A is not able to assume its own consistency (as we do in our justification that A’ is an improvement).

Suppose A’ considers some new self-modification (A”, P) not considered by A. We would like A to be able to prove thatif A’ executes this self-modification–if P is a proof that A” is an improvement over A’–then A” is really an improvement over A’ (and therefore over A). We say that a pair (A”, P) is safe if it has this property, i.e., if either P is not a proof that A” is an improvement over A’, or if A” is really an improvement over A’.

A can prove that particular pairs are safe, but suppose A’ considers many candidate modifications (A”, P1), (A”, P2), …, (A”, Pk). It is now much harder for A to show that all of these self-modifications are safe, without relying directly on the consistency of A’s axioms system (which A can never assume, by the incompleteness theorem). If the set of considered proofs is sufficiently rich, then by a bounded version of Löb’s theorem A can only prove that all of these self-modifications are safe if it can prove directly that A” is an improvement over A’. But in that case A might as well dispense with theorem proving altogether and just commit to transforming into A” after exhausting all other self-modifications.

(Originally this post was centered around a formal version of the above argument, but although I believe the argument works it turns out to be simultaneously very difficult and very unenlightening. Here is the main difficulty: in order to justify converting into A’, A only needs to prove that A’ won’t execute any destructive modifications if A’ doesn’t make some other modification before reaching any of the new proposals, and this additional assumption is very difficult to work with. In general, it is very hard to prove anything at all about a Gödel machine’s behavior, which is unfortunate because a Gödel machine must do exactly this in order to self-modify.)

This informal argument does not show that a Gödel machine will not execute any self-improvements, but it does show that if it does execute any non-trivial self-improvements then it must appeal to some justification other than the intuitive one. Other modifications, for example any modification which is so radical as to change the set of self-modifications considered in a more complicated way or to change the architecture fundamentally, seem to be even harder to justify than the one described here. Gödel machines may be able to self-modify, but this argument shows that this conclusion is not supported by intuitive arguments about the existence of beneficial self-modifications.

Creating New Agents

We can also try a non-reflective technique for designing an “optimal problem solver,” in the spirit of Hutter search. Again, suppose we have a particular object level problem and some allotted time.

Consider the agent A which behaves as follows: in the first half of the allotted time, it enumerates pairs (B, P), and if B is a valid description of an algorithm and P is a proof that B has utility at least U on the object level problem (using at most half of the allotted time), then A stores the pair (B, U). After half of the allotted time, the pair (B, U) with maximal U is selected, and then the algorithm B is run using the remaining time.

This solution seems like it may avoid some of the trouble encountered by the Gödel machine, in exchange for being somewhat less ambitious in its goals. But we will see that the same reflection problem arises.

If the allotted time is very large, and if self-modification is an important component of general intelligence, then we expect A to select a new algorithm B which is itself capable of (provably useful) self-modification. In principle A could simply run long enough to become a very powerful non-self-modifying agent; but it may be that finding a very powerful agent B directly is much more difficult then proceeding through a sequence of successively more powerful agents, each intelligent enough to design the next but not to leap directly to B.

But suppose A creates a new algorithm B which uses the same axioms as A to prove theorems, and uses these theorems to guide its own self-modification. Again, by Löb’s theorem we see that A cannot guarantee that B’s self-modifications are really beneficial. This means that A is unable to rule out the possibility that B executes a very bad self-modification (or creates a new agent with very bad behavior); but without ruling out the possibility that B performs very badly, A cannot run B instead of some more conservative algorithm which does not do any self-modification.

This does not rule out the possibility that A creates a new algorithm B’ which pursues self-modification using a similar strategy but with a more restricted set of axioms–one which can be proven consistent from within A. And in turn, if B’ takes the same form as A, then any algorithm C which B’ creates or modifies into will use a smaller still set of axioms by the same argument. Eliezer describes the proof-theoretic strength of such an agent as a ‘telomere’–with every self-modification it becomes weaker, and for Gödelian reasons it cannot be increased without sacrificing provable correctness.

(The distinction between the Gödel machine and creating new agents is inessential; for example, we could imagine a Gödel machine attempt to self-modify by improving the efficiency of its proof searcher and then using the additional time to consider some rewrites which are justified by a more restricted axiom system.)

Possible Solutions

There are some natural solutions which have been considered.

Long Telomeres

Recognizing that an agent must decrease its proof-theoretic strength when performing a self-modification, we may just try to construct an initial agent with substantial proof-theoretic strength. For example, if we believe that PA probably has sufficient logical expressiveness to facilitate useful superintelligence, we may create an agent with axioms PA + “PA is consistent” + “PA + ‘PA is consistent’ is consistent” and so on. We may iterate this process a finite number of times to allow for a few self-modifications, or perhaps add an entire countable chain so that the original AI can decide how much proof-theoretic strength its first descendant needs, or perhaps add a few countable chains so that the agent can self-modify a handful of times before needing to make that decision, or so on.

The primary objection to this solution is philosophical: humans appear to be able to reason about self-modification much more robustly (or at least with better results) than an AI using any of the logical frameworks discussed so far, and it is quite ambiguous whether or not a system with ‘long telomeres’ can ever justify becoming a system which is actually able to reason about self-modification correctly (or, if we have doubts about humans’ capacities, whether such a system even exists). Minimally, if we want to understand the form an AI is likely to take after self-modification, we need to understand how such a system would eventually escape from problems that currently seem inescapable; without such understanding the resulting behavior is hard to get a handle on (although we might still be satisfied, if that behavior is backed up by formal justification produced by the AI).

Lesser objections focus on the additional complexity required to maintain such a long telomere. Each additional consistency axiom added to a system essentially requires recursively specifying the entire agent as it exists so far, and the complexity of the mathematical manipulations required seems to grow rapidly. This argument seems to be fairly unimportant, and so if the philosophical objections were dispatched with I would consider building agents with very strong proof systems a feasible path forward.

Beliefs rather than Proofs

Humans seem to be able to avoid Gödelian problems with reflection, but they may do this by having confidence in statements which they cannot prove, and manipulating uncertainty about logical facts rather than dealing in absolute assertions of truth. The set of “statements I strongly believe” seems to differ in important ways from the set of “statements I can prove” in several important ways; for example, if I am confident of a statement at one point in time I am not committed to remaining confident indefinitely.

I suspect that a satisfactory theory of reflective beliefs would go far towards resolving the AI reflection problem, but this is hardly more than a restatement of the problem. Translating from proofs to beliefs does not immediately resolve the problems with self-reference, it just replaces them with subtly different issues. For example, it is easy to see that an agent should not believe that its own beliefs are well-calibrated on all questions, and so we must attempt to formalize a version of the self-consistency hypothesis which is weak enough to be defensible but still strong enough to escape the shortcomings described above. I hope to make several posts on this topic in the near future.

Ignorance

Understanding this issue may not be necessary to building safe AGI. Indeed, self-modification may ultimately play a minimal role in intelligence, or we may settle for executing self-modifications using weaker justification. However if we accept usual arguments about the importance of FAI, then we should not be satisfied with this solution.

The importance of self-modification is an open question which has received much discussion here and elsewhere. It is worth adding that, to the extent that we are concerned with influencing probable outcomes for humanity, the highest leverage scenarios seem to be those in which self-modification tends to result in positive feedback loops and takeoff (if we assign such scenarios significant weight). That is, in such scenarios we should be particularly cautious about building self-modifying systems, but there is also a much greater imperative to understand how to design safe and stable AI.

Standard arguments surrounding FAI (particularly, the importance of early AI goal systems and the fragility of humane value) suggest that agents should have high degrees of confidence in a change before executing it. If an agent’s beliefs are not correctly related to reality, the resulting behavior may be as dangerous as if the agent’s valus were modified. For example, incorrect beliefs about logical structure which cause that agent to fail to preserve its own values in subsequent rewrites, or incorrect beliefs about the relationships between value and reality.