Modeling Generalization in the Acquired Equivalence Paradigm with the Successor Representation

Authors: Troy M. Houser

Abstract:

The successor representation balances flexible and efficient reinforcement learning by learning to predict the future, given the present. As such, the successor representation models stimuli as what future states they lead to. Therefore, two stimuli that are perceptually dissimilar but lead to the same future state will come to be represented more similarly. This is very similar to an older behavioral paradigm -the acquired equivalence paradigm, which measures the generalization of learned associations. Here, we test, via computational modeling, the plausibility that the successor representation is the mechanism by which people generalize knowledge learned in the acquired equivalence paradigm. Computational evidence suggests that this is a plausible mechanism for acquired equivalence and thus can guide future empirical work on individual differences in associative-based generalization.

Keywords: Acquired equivalence, successor representation, generalization, decision-making.

References:

[1] R. N. Shepard, “Toward a universal law of generalization for psychological science,” Science (1979), vol. 237, no. 4820, 1987, doi: 10.1126/science.3629243.
[2] D. Zeithamova and C. R. Bowman, “Generalization and the hippocampus: More than one story?,” 2020. doi: 10.1016/j.nlm.2020.107317.
[3] C. R. Bowman, T. Iwashita, and D. Zeithamova, “Tracking prototype and exemplar representations in the brain across learning,” Elife, vol. 9, pp. 1–47, 2020, doi: 10.7554/eLife.59360.
[4] Ian Goodfellow, Yoshua Bengio, Aaron Courville. “Deep learning,” Nat Methods, vol. 13, no. 1, 2015.
[5] Y. LeCun, G. Hinton, and Y. Bengio,.“Deep learning (2015), Nature, vol. 521, 2015.
[6] A. R. Vaidya and D. Badre, “Abstract task representations for inference and control,” 2022. doi: 10.1016/j.tics.2022.03.009.
[7] C. R. Bowman and D. Zeithamova, “High coherence among training exemplars promotes broad generalization of face families,” PsyArXiv Preprints, 2023.
[8] C. R. Bowman and D. Zeithamova, “Coherent category training enhances generalization in prototype-based categories.,” J Exp Psychol Learn Mem Cogn, May 2023, doi: 10.1037/xlm0001243.
[9] C. R. Bowman and D. Zeithamova, “Abstract memory representations in the ventromedial prefrontal cortex and hippocampus support concept generalization,” Journal of Neuroscience, vol. 38, no. 10, pp. 2605–2614, 2018, doi: 10.1523/JNEUROSCI.2811-17.2018.
[10] M. I. Posner and S. W. Keele, “On the Genesis of Abstract Ideas,” J Exp Psychol, vol. 77, no. 3 PART 1, 1968, doi: 10.1037/h0025953.
[11] J. D. Smith and J. P. Minda, “Distinguishing Prototype-Based and Exemplar-Based Processes in Dot-Pattern Category Learning,” J Exp Psychol Learn Mem Cogn, vol. 28, no. 4, 2002, doi: 10.1037/0278-7393.28.4.800.
[12] J. P. Minda and J. D. Smith, “Comparing Prototype-Based and Exemplar-Based Accounts of Category Learning and Attentional Allocation,” J Exp Psychol Learn Mem Cogn, vol. 28, no. 2, 2002, doi: 10.1037/0278-7393.28.2.275.
[13] J. D. Smith and J. P. Minda, “Thirty Categorization Results in Search of a Model,” J Exp Psychol Learn Mem Cogn, vol. 26, no. 1, 2000, doi: 10.1037/0278-7393.26.1.3.
[14] J. D. Smith and J. P. Minda, “Prototypes in the mist: The early epochs of category learning,” J Exp Psychol Learn Mem Cogn, vol. 24, no. 6, 1998, doi: 10.1037/0278-7393.24.6.1411.
[15] N. E. Miller and Dollard, “Social Learning and Imitation,” J Nerv Ment Dis, vol. 99, no. 2, 1941, doi: 10.1097/00005053-194402000-00051.
[16] R. C. Honey and G. Hall, “Acquired Equivalence and Distinctiveness of Cues Using a Sensory-Preconditioning Procedure,” The Quarterly Journal of Experimental Psychology Section B, vol. 43, no. 2, 1991, doi: 10.1080/14640749108401263.
[17] R. C. Honey and G. Hall, “Acquired Equivalence and Distinctiveness of Cues,” J Exp Psychol Anim Behav Process, vol. 15, no. 4, 1989, doi: 10.1037/0097-7403.15.4.338.
[18] E. Coutureau, A. S. Killcross, M. Good, V. J. Marshall, J. Ward-Robinson, and R. C. Honey, “Acquired equivalence and distinctiveness of cues: II. Neural manipulations and their implications,” J Exp Psychol Anim Behav Process, vol. 28, no. 4, 2002, doi: 10.1037/0097-7403.28.4.388.
[19] D. Shohamy and A. D. Wagner, “Integrating Memories in the Human Brain: Hippocampal-Midbrain Encoding of Overlapping Events,” Neuron, vol. 60, no. 2, 2008, doi: 10.1016/j.neuron.2008.09.023.
[20] M. Meeter, D. Shohamy, and C. E. Myers, “Acquired Equivalence Changes Stimulus Representations,” J Exp Anal Behav, vol. 91, no. 1, 2009, doi: 10.1901/jeab.2009.91-127.
[21] C. E. Myers et al., “Dissociating hippocampal versus basal ganglia contributions to learning and transfer,” J Cogn Neurosci, vol. 15, no. 2, 2003, doi: 10.1162/089892903321208123.
[22] M. A. de Araujo Sanchez and D. Zeithamova, “Generalization and false memory in acquired equivalence,” Cognition, vol. 234, 2023, doi: 10.1016/j.cognition.2023.105385.
[23] K. J. Friston, “Waves of prediction,” PLoS Biol, vol. 17, no. 10, 2019, doi: 10.1371/journal.pbio.3000426.
[24] M. J. D. Ramstead, P. B. Badcock, and K. J. Friston, “Answering Schrödinger’s question: A free-energy formulation,” 2018. doi: 10.1016/j.plrev.2017.09.001.
[25] W. Schultz, P. Dayan, and P. R. Montague, “A neural substrate of prediction and reward,” Science (1979), vol. 275, no. 5306, 1997, doi: 10.1126/science.275.5306.1593.
[26] R. S. Sutton and A. G. Barto, “Reinforcement Learning: An Introduction,” IEEE Trans Neural Netw, vol. 9, no. 5, 1998, doi: 10.1109/tnn.1998.712192.
[27] P. Dayan, “Improving Generalization for Temporal Difference Learning: The Successor Representation,” Neural Comput, vol. 5, no. 4, 1993, doi: 10.1162/neco.1993.5.4.613.
[28] S. J. Gershman, “The successor representation: Its computational logic and neural substrates,” Journal of Neuroscience, vol. 38, no. 33, pp. 7193–7200, 2018, doi: 10.1523/JNEUROSCI.0151-18.2018.
[29] H. Eichenbaum, “Hippocampus: Cognitive processes and neural representations that underlie declarative memory,” 2004. doi: 10.1016/j.neuron.2004.08.028.
[30] H. Eichenbaum, “On the Integration of Space, Time, and Memory,” 2017. doi: 10.1016/j.neuron.2017.06.036.
[31] J. A. Dusek and H. Eichenbaum, “The hippocampus and memory for orderly stimulus relations,” Proc Natl Acad Sci U S A, vol. 94, no. 13, pp. 7109–7114, 1997, doi: 10.1073/pnas.94.13.7109.
[32] H. Eichenbaum, T. Otto, and N. J. Cohen, “The hippocampus-what does it do?,” 1992. doi: 10.1016/0163-1047(92)90724-I.
[33] H. Eichenbaum, T. Otto, and N. J. Cohen, “Two functional components of the hippocampal memory system,” Behavioral and Brain Sciences, vol. 17, no. 3, pp. 449–472, 1994, doi: 10.1017/S0140525X00035391.
[34] S. Wientjes and C. B. Holroyd, “The successor representation subserves hierarchical abstraction for goal-directed behavior,” PLoS Comput Biol, vol. 20, no. 2, 2024, doi: 10.1371/journal.pcbi.1011312.
[35] J. Y. Son, A. Bhandari, and O. FeldmanHall, “Abstract cognitive maps of social network structure aid adaptive inference,” Proc Natl Acad Sci U S A, vol. 120, no. 47, 2023, doi: 10.1073/pnas.2310801120.
[36] J. A. Kostek et al., “Acquired Equivalence in U.S. Veterans With Symptoms of Posttraumatic Stress: Reexperiencing Symptoms Are Associated With Greater Generalization,” J Trauma Stress, vol. 27, no. 6, pp. 717–720, Dec. 2014, doi: 10.1002/jts.21974.