Modelling Natural Action Selection

Cover......Page 1
Modelling Natural Action Selection......Page 2
Title......Page 4
Copyright......Page 5
Contents......Page 6
Foreword......Page 9
Preface......Page 12
Contributors......Page 13
1 General introduction......Page 18
Acknowledgements......Page 20
Part I Rational and optimal decision makings......Page 22
2.1 Suboptimality and ‘matching’......Page 24
2.3 Optimal perceptual selection......Page 25
2.4 Bayesian approaches to action selection......Page 26
References......Page 27
3.1 Introduction......Page 29
Matching......Page 32
3.2.1 The delay reduction hypothesis......Page 35
3.2.1.1 DRH and optimal foraging theory......Page 36
3.3.1 The outcome is a side-effect......Page 37
3.3.2.1 Uncontrolled variation in state......Page 38
Possible future states......Page 40
3.3.2.3 There may be more freedom in behaviour than originally anticipated......Page 41
1. Changes in optimal behaviour......Page 42
2. Changes in the performance......Page 43
3.3.2.4 Fluctuating environments and biased probabilities......Page 44
3.4 Discussion......Page 45
Acknowledgements......Page 46
References......Page 47
4.1 Introduction......Page 54
4.1.1 Embeddedness and optimal foraging theory......Page 55
4.1.3 Organisation......Page 56
4.2 Optimised agent-based modelling......Page 57
4.3 Embedded models of action selection......Page 59
4.4 Probability matching and inter-forager interference......Page 63
4.5 Matching and the ideal free distribution......Page 66
4.6 Discussion......Page 71
References......Page 72
5.1 Introduction......Page 78
5.2.1 Definitions of compromise......Page 79
5.2.2 Ethology......Page 80
5.2.4 Optimal biological approaches......Page 81
5.2.5 Artificial intelligence and planning......Page 82
5.2.6 Behaviour-based robotics......Page 83
5.3 Experiments......Page 84
5.3.1 Prescriptive experiments......Page 85
5.3.1.1 Formal model......Page 86
5.3.1.2 Prescriptive results......Page 88
5.3.1.3 Prescriptive discussion......Page 90
5.3.1.4 Prescriptive analysis......Page 91
5.3.1.5 Analysis experiments......Page 92
5.3.1.6 Prescriptive analysis discussion......Page 93
5.3.2 Proscriptive experiments......Page 94
5.3.2.1 Formal model......Page 95
5.3.2.2 Proscriptive results......Page 96
5.3.2.3 Proscriptive discussion......Page 99
5.4.1 High versus low-level actions......Page 101
5.4.2 Compromise behaviour hypothesis......Page 102
References......Page 103
6.1 Introduction......Page 108
6.2 Review of the LCA model......Page 109
6.2.2 Biologically inspired models of perceptual choice......Page 110
6.2.3 Linear LCA model......Page 111
6.2.4 Dynamics of the model......Page 112
6.2.5 Performance of linear LCA model......Page 114
6.2.6 Nonlinear LCA model......Page 116
6.2.7 Performance of bounded LCA model......Page 117
6.3 The advantage of nonlinearity in multiple choice......Page 119
6.3.1 Case 1: only two accumulators receive input and noise......Page 120
6.3.2 Case 2: biologically realistic input parameters for choice with continuous variables......Page 121
6.4 Optimisation of performance of bounded LCA model in the interrogation paradigm......Page 123
6.5 Value-based decisions......Page 124
6.5.2 Violations of expected-value and preference reversals......Page 125
6.5.3 Nonlinear utility functions and the Weber law......Page 127
6.5.4 Modelling value-based choice in the LCA framework......Page 129
6.5.4.1 Risk-aversion in probabilistic choice......Page 130
6.5.4.2 Multidimensional choice: reference effects and preference reversal......Page 131
6.6 Discussion......Page 133
References......Page 134
7.1 Introduction......Page 137
7.2.1 Combining prior knowledge with new evidence......Page 140
7.2.2 Combining multiple pieces of information......Page 141
7.3 Bayesian estimation across time......Page 142
7.4 Bayesian estimation of structure......Page 144
7.5 Bayesian decision theory and inverse decision theory......Page 146
7.6.1 The optimal control problem......Page 148
7.6.2 Solution strategies in optimal control......Page 149
7.6.3 Optimal control as Bayesian inference......Page 150
7.7 Experimental investigation of learnt statistics......Page 151
7.8 Discussion......Page 155
References......Page 157
8.1 Introduction......Page 161
8.2.1 Competitive queuing......Page 163
8.2.2 Resampling models......Page 165
8.3.1 Recall latencies......Page 166
8.4 Generic model......Page 167
8.4.1 Model architecture......Page 168
8.4.1.4 Inhibition layer......Page 170
8.4.2.1 Simulation 1: RS versus CQ......Page 171
Procedure......Page 173
8.4.3.1 Simulation 2: setsize effects......Page 176
8.5 Discussion......Page 177
8.5.1 Implications for dynamical models of memory......Page 178
8.5.2 Implications for dynamical models of sequential selection......Page 179
8.5.4 Conclusion......Page 180
References......Page 181
Part II Computational neuroscience models......Page 184
9.1 Neural substrates for action selection in cortico-basal ganglia loops......Page 186
9.2 The decomposition of control for behavioural sequencing......Page 188
9.3 Subcortical substrates for action selection......Page 189
9.4 Disorders of action selection: clinical implications of action selection modelling......Page 190
References......Page 191
10.1 Introduction......Page 193
10.2 Selection and initiation of corrective submovements......Page 195
10.2.1 Abstract model of how corrective submovements are generated......Page 198
10.2.2 Neural mechanisms......Page 199
10.2.4 Integrative control of hand movement and on-line error correction......Page 202
10.3.1 The Replicate task......Page 205
10.3.2 Functional neuroimaging (fMRI) of Replicate in human subjects......Page 206
10.3.3 Action selection in the loop through the basal ganglia......Page 207
10.3.4 Model of competitive pattern classification......Page 209
10.3.5 Pattern classification in phylogeny......Page 211
10.4 Discussion......Page 212
10.4.1 Serial order processing, basal ganglia loops and recursion......Page 213
10.4.2 Networks of DPMs and agent-based modelling......Page 215
10.4.3 Understanding action selection deficits in brain disorders......Page 216
10.4.3.1 A model of the etiology of schizophrenia......Page 217
References......Page 219
11.1 Introduction......Page 225
11.2 The affordance competition hypothesis......Page 228
11.3 A computational model of reaching decisions......Page 234
11.4 Discussion......Page 243
References......Page 247
12.1 Introduction......Page 256
12.2.1 Six key functional demands for working memory......Page 266
12.2.2 Dynamic updating via basal ganglia gating......Page 268
12.2.3 Learning when to gate in the BG......Page 273
12.2.4 Empirical tests of the model......Page 275
12.3.1 The full MT model......Page 276
12.3.2 Recent progress: the task contingency-shifting paradox in the WCST......Page 278
12.4 Conclusion and future model development......Page 280
Summary......Page 289
13.1 Temporal abstraction and the scaling problem......Page 291
13.2.1 Fundamentals of RL: temporal difference learning in actor–critic models......Page 293
13.2.2 Incorporating temporally abstract actions......Page 295
13.2.4 Illustrations of performance......Page 297
13.3.1 Relation to previous work in psychology......Page 299
13.3.2 Negative transfer......Page 301
13.3.3 The option discovery problem......Page 303
13.4.1.2 Potential neural correlates......Page 305
13.4.1.4 Potential neural correlates......Page 307
13.4.2.3 Extension 4: Temporal scope of the prediction error......Page 308
13.5 Discussion......Page 309
13.5.2 Strict versus quasi-hierarchical structure......Page 310
13.5.3 Directions for further research......Page 311
Appendix A......Page 312
Training......Page 313
Training......Page 314
Acknowledgements......Page 315
References......Page 316
14.1 Introduction......Page 325
14.2 Where and what is the mRF?......Page 328
14.4 Inputs to the mRF......Page 330
14.5 Outputs of the mRF......Page 331
14.6 Internal circuitry of the mRF......Page 332
14.6.2 An anatomical model of the mRF......Page 333
14.6.3 Structural properties of the mRF......Page 335
14.7.1 Functional organisation of mRF......Page 337
14.7.2 Action selection at the cluster level......Page 338
14.7.2.2 Subaction configuration......Page 339
14.7.3 Distributed action representation in the mRF......Page 340
14.7.4 Reconciling cluster-based and distributed action selection......Page 342
14.8 Integration of the action selection systems......Page 344
14.9 Further questions about the mRF......Page 346
14.10 Final remarks......Page 347
Appendix A Basic form of the computational models......Page 348
References......Page 349
15.1 Introduction......Page 355
15.2 Neurocomputational framework......Page 356
15.3.1 Parkinson’s disease and dopamine manipulations......Page 359
15.3.1.2 Deep brain stimulation in Parkinson’s disease......Page 361
15.3.2 Decision–making deficits in ventromedialor/bitofrontal patients......Page 362
15.3.3 Attention deficit/hyperactivity as a disorder of action selection......Page 364
15.3.3.1 Structural, functional, and dopamine effects in attention deficit/hyperactivity disorder......Page 365
15.3.3.2 Reward anticipation and temporal discounting in attention deficit/hyperactivity disorder......Page 366
15.3.3.3 Norepinephrine in attention deficit/hyperactivity disorder and action selection......Page 367
15.3.3.4 Simulating norepinephrine function in action selection and attention deficit/hyperactivity disorder......Page 368
15.4 Clinical implications of basal ganglia modelling......Page 372
Implementational details......Page 374
Connectivity and mechanics of the basal ganglia model......Page 375
Subthalmic nucleus connectivity with other basal ganglia and cortical structures......Page 376
Inhibition within and between layers......Page 377
Learning......Page 378
Appendix B Additional details for locus coeruleus/norepinephrine simulations......Page 379
References......Page 380
16.1 Introduction......Page 388
16.2 Modelling the Stroop task......Page 389
16.2.1 Limitations of the Cohen model with stimulus-onset asynchrony are failures of response selection......Page 393
16.3.1 The basal ganglia and action selection......Page 396
16.3.2 Including the thalamic complex......Page 398
16.3.3 Combining the Cohen model with the basal ganglia response mechanism......Page 399
16.3.4 Simulations I: matching basic empirical data......Page 401
16.3.5 Simulations II: dynamic attentional inhibition......Page 403
16.4 Discussion......Page 405
16.4.1 Why the basal ganglia model successfully simulates reaction times......Page 406
16.4.2 Weaknesses of the diffusion model......Page 407
16.4.4 Benefits of the basal ganglia model......Page 408
16.4.6 Conclusions and future work......Page 410
References......Page 411
17.1 Introduction......Page 415
17.1.1 The oculomotor system......Page 417
17.1.1.2 The frontal eye field......Page 418
17.1.1.3 The saccadic generator......Page 419
17.1.1.4 The visuo-motor response......Page 420
17.1.2 Accumulation by positive feedback......Page 421
17.1.3 Competition in the oculomotor system......Page 424
17.2 Methods......Page 428
17.3.1 Accumulation dynamics in the oculomotor model......Page 430
17.3.1.2 Competitive accumulation......Page 432
17.3.1.3 Selection, enaction, and accumulator reset......Page 433
17.4 Conclusions......Page 434
17.4.1 Accuracy, response time, and the effect of learning......Page 435
17.5.1.1 Algorithm refinement......Page 438
17.5.1.2 Adaptive learning......Page 439
References......Page 440
Part III Action selection in social contexts......Page 444
18.1 Simulations as explanations of pro-social behaviour......Page 446
18.2 Levels of selection: ultimate goals and action-selection mechanisms......Page 447
18.3 Levels of selection, emergence, and cognition in social behaviour......Page 448
18.4 Precise matches to quantitative data: voting mechanisms in animals and humans......Page 449
References......Page 450
19.1 Introduction......Page 452
19.2 Analysing agent-based modelling......Page 453
19.2.1 Validation and analytic solutions......Page 454
19.2.2 Agent-based models as scientific hypotheses......Page 455
19.3.1 Primate social structure......Page 456
19.3.2 The DomWorld model......Page 457
19.3.2.2 DomWorld agents......Page 458
19.3.2.3 DomWorld agent behaviour......Page 459
19.3.3.1 Gradients of dominance hierarchy......Page 460
19.3.3.3 Sexual attraction during tumescence......Page 461
19.4.1 Replication......Page 462
19.4.1.1 Results......Page 463
19.4.2 Analysis of model correlates......Page 464
19.4.3 Evaluating the model......Page 467
19.4.4 Partial recovery of the model......Page 471
19.5 Conclusions and discussion......Page 472
19.5.2 Agent-based models as hypotheses and vectors of scientific communication......Page 473
References......Page 475
20.1 Introduction......Page 479
20.2 Decision making in animal groups......Page 481
20.3 Agent-based modelling......Page 482
20.4 Field data......Page 485
20.5 The model......Page 486
20.6 Sensitivity analysis......Page 490
20.7 Discussion......Page 495
Acknowledgements......Page 497
References......Page 498
21.1 ‘Spatial’ models of political competition......Page 502
21.2 Agent-based spatial models of party competition......Page 504
21.3 Birth and death of political parties......Page 505
21.3.1 The distribution of voter ideal points......Page 506
21.3.2 Voter dissatisfaction at tick t......Page 507
21.3.5 The birth of political parties......Page 508
21.3.6 The fitness and death of political parties......Page 509
21.3.7 ‘Campaign’ ticks and ‘election’ ticks......Page 510
21.3.8.1 Specifying the model......Page 511
21.3.8.2 Party system sizes......Page 512
21.4 Performance of sets of parties using different decision rules......Page 514
21.4.1 Policy locations of party births, deaths, and survivors......Page 515
21.4.3 Representativeness of the party system......Page 517
21.5 Discussion and further work......Page 521
References......Page 522
22.1 Introduction......Page 525
22.2 Optimal decision making......Page 526
22.4 The Usher–McClelland model......Page 528
22.5 Decision making in social insect colonies......Page 530
22.6 Models of house-hunting by social insect colonies......Page 531
22.6.1 House-hunting in Temnothorax albipennis......Page 532
22.6.2 House-hunting with indirect switching in Apis mellifera......Page 533
22.6.3 House-hunting with direct switching in Apis mellifera......Page 534
22.7 Numerical simulation......Page 535
22.8 Discussion......Page 536
Acknowledgements......Page 539
Appendix B Reduction of the Temnothorax albipennis model to one dimension......Page 540
Appendix C Reduction of the Apis mellifera indirect-switching model to one dimension......Page 541
Appendix D Reduction of the Apis mellifera direct-switching model to one dimension......Page 543
References......Page 544
23.1 Introduction......Page 548
23.2.1 Outline of the model......Page 550
23.2.2 Model exploration......Page 552
23.3 Results......Page 554
23.4 Discussion......Page 556
References......Page 560
Index......Page 563