Deep Learning, Neuroscience, and Psychology

Postdoctoral Research Associate

Department of Experimental Psychology, University of Oxford

**Supervisors:** Christopher Summerfield and Tim Behrens

**Education:**

PhD in Electrical Engineering, Stanford University

Thesis: Deep linear neural networks: A theory of learning in the brain and mind

Advisers: Jay McClelland (primary), Andrew Ng, Christoph Schreiner, and Surya Ganguli

BSE in Electrical Engineering, Princeton University (summa cum laude)

**Curriculum Vitae:** pdf

The theory of deep learning and its applications to phenomena in neuroscience and psychology.

Saxe, A.M., McClelland, J.L., Ganguli, S. (2018). **A mathematical theory of semantic development in deep neural networks**. *arXiv.*

pdf | arxiv

Bansal, Y., Advani, M., Cox, D. D., & Saxe, A. M. (2018). **Minnorm training: an algorithm for training over-parameterized deep neural networks**. *arXiv.*

pdf | arxiv

Masís, J., Saxe, A. M., & Cox, D. D. (2018). **Rats optimize reward rate & learning speed in a 2-AFC task.** Poster at *COSYNE.* Denver.

pdf

Saxe*, A. M., & Advani*, M. (2018). **A theory of memory replay and generalization performance in neural networks.** Poster at *COSYNE*. Denver. *Equal contributions.

pdf

Zhang, Y., Saxe, A. M., Advani, M. S., & Lee, A. A. (2018). **Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning**. *Molecular Physics*, 1–10.

pdf | arxiv

Earle, A. C., Saxe, A. M., & Rosman, B. (2018). **Hierarchical Subtask Discovery with Non-Negative Matrix Factorization**. In Y. Bengio & Y. LeCun (Eds.), *the International Conference on Learning Representations*. Vancouver, Canada.

pdf | workshop version

Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2018). **On the Information Bottleneck Theory of Deep Learning**. In Y. Bengio & Y. LeCun (Eds.), *the International Conference on Learning Representations*. Vancouver, Canada.

pdf | code

Nye, M., & Saxe, A. (2018). **Are Efficient Deep Representations Learnable?** In Y. Bengio & Y. LeCun (Eds.), *Workshop Track at the International Conference on Learning Representations*. Vancouver, Canada.

pdf

Advani*, M., & Saxe*, A.M. (2017). **High-dimensional dynamics of generalization error in neural networks.** *arXiv*. *Equal contributions.

pdf | arxiv

Musslick, S., Saxe, A.M., Ozcimder, K., Dey, B., Henselman, G., & Cohen, J.D. (2017, July). **Multitasking Capability Versus Learning Efficiency in Neural Network Architectures.** In M. Knauff, M. Paulen, N. Sebanz, & I. Wachsmuth (Eds.), *Proceedings of the 39th annual meeting of the Cognitive Science Society.* (pp. 829-834). Austin, TX: Cognitive Science Society.

pdf

Saxe, A.M., Earle, A.C., & Rosman, B. (2017). **Hierarchy Through Composition with Multitask LMDPs.** In *ICML 2017*.

pdf | supplementary material

Tsai*, C.Y., Saxe*, A., & Cox, D. (2016) **Tensor Switching Networks.** In *NIPS 2016*. *Equal contributions.

pdf | supplementary material

McClelland, J.L., Sadeghi, Z. & Saxe, A.M. (2016). **A Critique of Pure Hierarchy: Uncovering Cross-Cutting Structure in a Natural Dataset.** *Neurocomputational Models of Cognitive Development and Processing (pp 51-68).* World Scientific.

pdf

Saxe, A.M. (2016, August). **Inferring actions, intentions, and causal relations in a neural network.** Poster at *CogSci 2016*, Quebec City, Canada.

pdf

Saxe, A.M., & Norman, K. (2016, February). **Optimal storage capacity associative memories exhibit retrieval-induced forgetting.** Poster at *COSYNE 2016*. Salt Lake City.

pdf

Baldassano*, C., & Saxe*, A.M. (2016, February). **A theory of learning dynamics in perceptual decision-making.** Poster at *COSYNE 2016*. Salt Lake City. *Equal contributions.

pdf

Saxe, A.M. (2015, March). **A deep learning theory of perceptual learning dynamics.** Poster at *COSYNE 2015*. Salt Lake City.

pdf

Lee, R., & Saxe, A.M. (2015, March). **The effect of pooling in a deep learning model of perceptual learning.** Poster at *COSYNE 2015*. Salt Lake City.

pdf

Goodfellow, I.J., Vinyals, O., & Saxe, A.M. (2015). **Qualitatively characterizing neural network optimization problems.** In Y. Bengio & Y. LeCun (Eds.), *International Conference on Learning Representations.* San Diego, CA.

pdf | arxiv

Saxe, A.M. (2014, July) **Multitask model-free reinforcement learning.** Poster at *CogSci 2014*, Quebec City, Canada.

pdf | code upon request

Lee, R., Saxe, A., & McClelland, J.L. (2014, July). **Modeling perceptual learning with deep networks.** Poster at *CogSci 2014*, Quebec City, Canada.

pdf

Saxe, A.M., McClelland, J.L., & Ganguli, S. (2014). **Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.** In Y. Bengio & Y. LeCun (Eds.), *International Conference on Learning Representations.* Banff, Canada.

pdf | arxiv

Saxe, A.M., McClelland, J.L., and Ganguli, S. (2013) **Dynamics of learning in deep linear neural networks.** In *NIPS Workshop on Deep Learning 2013*.

pdf | supplementary material

Saxe, A.M., McClelland, J.L., and Ganguli, S. (2013) **Learning hierarchical category structure in deep networks.** In M. Knauff, M. Paulen, N. Sebanz, & I. Wachsmuth (Eds.), *Proceedings of the 35th annual meeting of the Cognitive Science Society.* (pp. 1271-1276). Austin, TX: Cognitive Science Society.

pdf

Saxe, A.M., McClelland, J.L., and Ganguli, S. (2013) **A Mathematical Theory of Semantic Development.** Poster at *COSYNE 2013*, Salt Lake City.

pdf

Saxe, A., Bhand, M., Mudur, R., Suresh, B., & Ng, A. (2011) **Unsupervised learning models of primary cortical receptive fields and receptive field plasticity.** In *NIPS 2011*.

pdf | supplementary material | data upon request

Saxe, A., Koh, P.W., Chen, Z., Bhand, M., Suresh, B., & Ng, A. (2011). **On random weights and unsupervised feature learning.** In *ICML 2011*.

pdf | code

Saxe, A., Bhand, M., Mudur, R., Suresh, B., & Ng, A. (2011, February). **Modeling cortical representational plasticity with unsupervised feature learning.** Poster at *COSYNE 2011*, Salt Lake City.

pdf

Saxe, A., Koh, P.W., Chen, Z., Bhand, M., Suresh, B., & Ng, A. (2010). **On random weights and unsupervised feature learning.** In *NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning*.

pdf | supplementary material | code

Balci, F., Simen, P., Niyogi, R., Saxe, A., Hughes, J.A., Holmes, P., Cohen, J.D. (2010). **Acquisition of decision making criteria: reward rate ultimately beats accuracy.** *Attention, Perception, & Psychophysics, 1–18.*

pdf

Goodfellow, I. J., Le, Q. V., Saxe, A. M., Lee, H., & Ng, A.Y. (2009). **Measuring invariances in deep networks.** In *NIPS 2009*.

pdf

Baldassano, C.A., Franken, G.H., Mayer, J.R., Saxe, A.M., & Yu, D.D. (2009). **Kratos: Princeton University’s entry in the 2008 Intelligent Ground Vehicle Competition.** *Proceedings of SPIE*.

pdf

Atreya, A.R., Cattle, B.C., Collins, B.M., Essenburg, B., Franken, G.H., Saxe, A.M., et al. (2006). **Prospect Eleven: Princeton University’s entry in the 2005 DARPA Grand Challenge.** *Journal of Field Robotics, 23(9), 745-753.*

pdf

Lecture slides on backpropagation

Object recognition with features from random weight TCNNs