Abstract

     Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.

Introduction

     Every day, various types of sensory information from the external environment are transferred to the brain through different modalities and then processed to generate a series of coping behaviours. Among these perceptual modalities, vision is arguably the dominant contributor to the interactions between the external environment and the brain. Approximately 70 percent of human perception information is derived from vision[1] , far more than the auditory system, tactile system, and other sensory systems combined. The visual system is the part of the central nervous system that is required for visual perception, processing, and interpreting visual information to build a representation of the visual environment. It consists of the eye, retina, fibers that conduct visual information to the thalamus, the superior colliculus, and parts of the cerebral cortex. Today, researchers can collect neural signals using different recording modalities, e.g., spikes, electroencephalography (EEG), and functional magnetic resonance imaging (fMRI), from brain activity in different parts of the visual system, such as the retina, lateral geniculate nucleus (LGN), and primary visual cortex (V1) cortex, etc. Depending on the corresponding collecting devices, different recording modalities differ in their invasiveness, scale, and precision.

Conclusions

     In this paper, we first briefly analyzed the evolution of decoding tasks, i.e., classification, identification, and reconstruction, as this research field has developed. And we introduced the main neural recording modalities used in visual neural decoding and analyzed the characteristics of the data they acquire. Then we reviewed the main types of decoding approaches that researchers have proposed in recent decades in this field. Open data resources of data and toolkits, as well as open challenges and potential future directions of visual neural decoding, are suggested as well. The ultimate purpose of visual decoding is to decode the content of our experience in the absence of visual input. However, the scarcity of pairwise neurophysiological stimulus datasets and accurate, large-scale recording neural modalities continue to hinder the development of this discipline. Nevertheless, the importance of visual neural decoding cannot be understated. The development of neural decoding technology will promote the development of neural prostheses and brain-computer interface devices. We hope that our brief review will inspire ideas for future work in the cross-disciplinary field of brain science and neural computing.