Neural Models of the Psychosemantics of ‘Most’

Lewis O’Sullivan, Shane Steinert-ThrelkeldNAACL 2019

TL;DR

This paper explores how neural networks can model human understanding of the quantifier "most." By training convolutional and recurrent neural networks on a visual verification task, the authors found that these models exhibit human-like behaviors, such as sensitivity to set ratios and image types, suggesting potential for neural models to serve as cognitive models for psycholinguistic phenomena.

What Problem Does This Paper Address?

The paper tackles the question of how the meaning of linguistic expressions, specifically quantifiers like "most," is represented and utilized in human cognition. While semantics traditionally focuses on truth conditions, this work investigates the "psychosemantics" – how these meanings are processed in real-time cognitive tasks. Previous research by Pietroski et al. (2009) showed that human participants exhibit variations in verifying sentences like "Most of the dots are yellow," suggesting different underlying representations and verification strategies. The authors aim to build neural network models that can replicate these human behaviors, serving as a bridge between computational models and cognitive theories of language understanding. They are particularly interested in whether neural networks can capture the nuances of human performance, including sensitivity to visual cues and potential trade-offs between speed and accuracy.

Methods

The researchers employed two main types of neural network architectures:

  • Convolutional Neural Networks (CNNs): Specifically, they used VGG networks (VGG7, VGG9, VGG11, VGG13) of varying depths. Network depth was used as an operationalization of "task duration," simulating increased information processing capacity.
  • Recurrent Models of Visual Attention (RAM): This model, inspired by human visual attention, processes images by taking a series of "glimpses." The number of glimpses was used to operationalize "task duration," reflecting the number of fixations and saccades a human might make.

Both network types were trained on a visual verification task similar to Pietroski et al. (2009), where they had to determine if "Most of the dots are blue" was true for various dot arrangements. They also introduced a novel manipulation of "task duration" by varying network depth (CNNs) or the number of glimpses (RAM).

Datasets

The models were trained and evaluated on a dataset of dot matrix stimuli. These stimuli consisted of two sets of dots in various ratios and spatial arrangements.

  • Stimuli Generation: Images were 128x128 pixels, converted to grayscale. Ratios ranged from 1:2 to 9:10. Four arrangement types were used: column pairs sorted, column pairs mixed, scattered pairs, and scattered random.
  • Dataset Split: The data was split into training (18,000 images), validation, and test sets (3,600 images each), with all sets balanced across ratio, image type, and truth value.

Key Insights and Ideas

  • Neural Networks Mimic Human Sensitivity: Both CNNs and RAM models showed human-like sensitivity to the ratio of dots (performing worse with more balanced ratios) and to image type (performing better on structured arrangements like column pairs compared to scattered dots).
  • "Task Duration" Matters Differently: The operationalization of "task duration" had a more pronounced effect on CNNs (depth) than on RAM models (number of glimpses), suggesting different architectural sensitivities to processing constraints.
  • Approximate Number System (ANS) Fit: The models' performance, particularly their accuracy across different dot ratios, could be well-fitted by a psychophysical model of the Approximate Number System (ANS), mirroring findings from human participants. This suggests that neural models might be learning ANS-like representations.
  • Subtle Differences Remain: While models showed remarkable similarity to human performance, there were subtle differences, such as how models treated "column sorted" vs. "column mixed" image types, indicating that human strategies might still be more nuanced.

Key Results

  • Accuracy Trends: All networks showed decreasing accuracy as dot ratios became more balanced. VGG7 performed significantly worse than VGG9-13, suggesting that deeper CNNs generally improve performance. RAM models showed more variability but generally performed well.
  • Image Type Effects: CNNs performed worse on scattered images than column types. RAM models showed a similar pattern, though less pronounced. Both model types performed better on image types with paired dots.
  • ANS Model Fit: The psychophysical model of the ANS provided a good fit to the network data, with Weber fractions (a measure of discriminability) comparable to those found for human participants. For example, VGG7 had a Weber fraction of 0.363 for scattered random trials, while human participants showed a w of roughly 0.3.
  • Task Duration Impact: Increasing CNN depth (VGG11) led to better performance. For RAM models, increasing glimpses had a less consistent effect on overall accuracy.

Limitations and Future Work

  • Model vs. Human Strategy: While models replicate behavioral traces, the underlying strategies might differ. For instance, humans show a stronger preference for "column sorted" over "column mixed" images than the models do.
  • Hyper-parameter Tuning: More extensive hyper-parameter searches could potentially improve model fit to human data and allow for more robust predictions.
  • RAM Model Improvement: Providing RAM models with a low-resolution overview of the entire image could aid in location selection for glimpses.
  • Investigating Strategies: Techniques like transfer learning and diagnostic classifiers could be used to better understand the internal strategies employed by the models.
  • Neuron-Level Analysis: Investigating whether individual neurons or groups of neurons in the models exhibit activation patterns consistent with Weber's law could provide deeper insights into ANS-like processing.
  • Broader Stimuli: Extending the task to include more complex image manipulations (e.g., more colors) or different quantifiers could further test the models' cognitive plausibility.
  • Operationalizing Task Duration: Developing more nuanced ways to operationalize "task duration" for different model architectures remains an area for future work.

Summary updated

BibTeX
@inproceedings{osullivan2019neural,
  author = {Lewis O’Sullivan and Shane Steinert-Threlkeld},
  title = {{Neural Models of the Psychosemantics of ‘Most’}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}

Loading related content...

Figure