A separate issue is “Given a thing that responds to stimuli in this way, try to extract some abstract features of interest.”

A human in a box, communicating through a keyboard, is going to offer a strongly biased picture of how the human brain works. The use of heuristics by the human decision procedure is undoubtedly context-dependent, and this is a highly artificial context. Important chunks of the human utility function may never be called upon, and so may be overlooked by a simulation.

Also, if we are trying to extract a generic human utility function, relying on one person may lead to overfitting. What we get here is surely much much more complicated than what we ultimately want, if the aim is to extract a human-friendly value system. We don’t want an AI to believe that all its choices need to be measured against the preferences of someone who was stuck in a box once, possibly having a bad day, and whose brain was full of all sorts of idiosyncrasies and irrelevant microphysical complexities.

So ultimately you *do* have to solve the arcane problems like identifying the “internal mechanism our brains use for giving our intentions control over motor function”. Or perhaps you personally don’t have to solve them, but your algorithm for determining the nature of the human decision procedure will be performing an equivalent analysis. “Overfitting” is a nice familiar way to pose the question: In inferring the human decision procedure, by observing and simulating human beings, how do we avoid overfitting? Answering *that* question should take you a long way.

If the objective is just to simulate a particular human being… again, a microphysically exact physical simulation would not be the simplest way to simulate a person. It might be one of the simplest functions to ostensively *specify* (“simulate what’s happening in that box”), but it would be full of unnecessary details about ions.

On a practical level, an efficient way to develop a whole-brain model may involve repeated high-resolution fMRI, and the progressive development of finite-state machine models for the voxels, in the context of interactive experiments. That is, you’ll be modeling the brain as a lattice of finite state machines coupled to their neighbors. You get the data from the fMRI, and there will be a protocol of interaction with the subject designed to reveal the dynamics in ever-greater detail, until diminishing returns set in.