Self-organizing multimodal codes

Holographic Attention: Part I 


<img className="rounded-xl object-contain mt-8 mb-1 mx-auto " src="/images/work/show/walk.gif" width="200" height="200" />

<a href="https://redwood.berkeley.edu/wp-content/uploads/2020/05/paiton2020subspace.pdf" target="_blank" className="flex items-center border border-[0.5px] border-slate-200 justify-center w-full px-4 py-1 mt-4 mb-4 text-base text-center text-black bg-slate-800 rounded-lg shadow-lg dark:bg-gray-200 dark:text-white">
  <span>Link to Paper </span>
</a>

<h2 className="mt-4 border-slate-100 text-left font-bold text-xl justify-start">Natural video scene statistics </h2>

<p>
The brain is an inference machine, yet universal computational algorithms supporting this inference remain largely unknown. The efficient coding hypothesis suggests that the brain extracts statistical structure from noisy sensory data to construct a coherent model of the natural world [1][2]. Research has demonstrated that natural visual scenes have distinct spatiotempal statistical structure at several different scales [2][3]. Here we explore a bio-inspired generative model that uses sparse coding to learn this spatiotemporal structure in natural video scenes. 
</p>

<div className="flex flex-row justify-center gap-4 object-contain mt-8 mb-1 w-full max-w-[620px]">

<div className="flex flex-col gap-4 justify-between">
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_dongAtick1.png" width="400" height="400" />
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_dongAtick2.png" width="400" height="400" />
</div>

<div className="flex flex-col gap-4 justify-between">
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_cooper1.png" width="280" height="200" />
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_cooper2.png" width="280" height="200" />
</div>

</div>
<em className="text-xs mb-8">
Figure 1. Spatial and temporal frequency statistics of natural scenes. (Left) Seminal research from Dong & Atick revealing the power-law distribution relationship between signal intensity and signal frequency in spatial and temporal domains. (Image Source: [1]) (Right) Recent research from Cai, Cooper et al [2] has also found this statistical relationship to exist similarly in aquatic and terrestrial environments. (Image Source: [2]) 
</em>

<div>
<img className="mx-auto rounded-xl object-contain p-4 mb-1" src="/images/work/show/slca_dongAtick3.png" width="400" height="400" />
</div>
<em className="text-xs mb-8">
Figure 2. Measured pairwise probability distribution for two similar spatial locations separated by 33 ms in time. White (black) represents high (low) probability. This illustrates high correlation between the two light intensities within small separations of time, suggesting a general spatio-temporal statistical structure for natural moving scenes(Image Source: [1])
</em>


<h2 className="mt-4 border-slate-100 text-left font-bold text-xl justify-start">Sparse coding with LCA</h2>
<p>
Sparse coding is a well-explored method for unsupervised learning of statistical structure in data especially in high dimensional spaces. Unlike typical compression or bottle-neck autoencoders which simply map high-dimensional data to low dimensional spaces, sparse coding instead maps high-dimensional data to an overcomplete, higher dimensional embedding space but constrains the active components representing a single signal to a low dimension. The theory posits that efficient representations are formed under resource constraints and thus latent variables undergo competition to create encodings that are maximally expressive with the minimal active (non-zero) variables. The higher dimensional representational space adds fidelity, expressivity, and robustness to the encoded information as each representation becomes nearly orthogonal to the others (maximally uncorrelated). Image data is a notoriously high dimensional domain where sparse coding has also been applied extensively. Remarkably, it was shown in Olshausen et al. [4] that when applied to natural images, sparse coding learned latent representations that matched the tuning properties of simple cells in the V1 area of the primate visual cortex. When visualized in the image domain, these representations resemble "edge-dectors" or wavelet filters. Importantly, unlike hand-engineered wavelet filters, sparse coding's representation are learned from data in an unsupervised manner based on a generative model. 
</p>
<br/>
 
 Sparse coding can be modeled as a linear generative equation as the following:
<br/>
<div className="text-center mt-8">

<span style="display: inline-block;"> $s = a\Phi + v$  </span>

</div>
<br/>

The additional term 
<span style="display: inline-block;">$v$ </span> 
represents the noise in the model which is assumed to be 
<span style="display: inline-block;"> $\backsim N(0,\Sigma)$</span> 
, drawn from a Gaussian with 0 mean and diagonal variance 
<span style="display: inline-block;">$\Sigma$</span>.
<br/>
The objective function for denoising the input with a sparsity constraint is the following:
<br/>
<div className="text-center mt-8">

<span style="display: inline-block;"> $\underset{a}{\operatorname{argmin}}(E_{sc} = \frac{1}{2} \|s - \Phi \|+ (\lambda \sum_{k=1} C(a_k)))$ </span>

</div>
<br/>
<p>
Sparse coding can be implemented as an autoencoder with shared encoder-decoder weights or as a single-layer recurrent neural network with all-to-all connections, like a Hopfield network. Sparse coding models are typically trained to denoise and compress data. Signals are fed to the network which then under goes an interative winner-takes-all greedy algorithm to select the most useful feature vectors to reconstruct the signal. 
</p>

<img className="rounded-xl object-contain mx-auto mt-8 mb-1" src="/images/work/show/lca_vanilla_vid_crop.gif" width="400" height="400" />
 <em className="text-xs mb-4">
Figure 3. LCA animated as a recurrent neural network, with all-to-all connections. Neurons surpassing the activation threshold, supress the activity of all other neurons, encouraging a sparse encoding. Connections between neurons omitted for clarity.
</em>

<p>
LCA is a class of neural networks that implement sparse coding effciently as a dynamical system, taking inspiration from leaky-integrate and fire models of biological neurons which have internal states and external activations. Neurons integrate their internal state value, until an activation threshold is surpassed, increasing the value of the external state variable  a. Active neurons supress the activity of all other neurons, encouraging a sparse encoding. These models are neat, because they can also be implemented in low power analog circuits. 
</p>

<br/>

The internal state of neurons over time for LCA can be modeled as a nonlinear ordinary differential equaiton (ODE) with the following dynamical equation: 
<br/>
<div className="text-center mt-8">

<span style="display: inline-block;"> $\dot{u_{m}}(t) = \frac{1}{\tau}[b_{m}(t) - u_{m}(t) - \sum_{k \neq m} G_{m,n}(a_{n}(t))]$ </span>  

</div>
<br/>
The full dynamic system equationn for LCA can then be written as:

<div className="text-center">

<span style="display: inline-block;"> $\hat{s}(t)$  </span>

</div>
<br/>

where 

<div className="text-center">

<span style="display: inline-block;"> $a(t) = T_{\lambda}(u(t))$ </span>

</div>

<br/>
This ODE, has an equivalent mathematical expression as the continuous Hopfield Network model.

<img className="rounded-xl object-contain mt-8 mb-1" src="/images/work/show/SLCA_LCAcircuit.png" width="600" height="300" />
 <em className="text-xs mb-4">
Figure 4. The original locally competitive algorithm (LCA) represented as an analog circuit. Where u represents the internal state of the neurons and a represenst the external state. Connections between neurons omitted for clarity. (Image source:[5])
</em>

<h2 className="mt-8 text-left font-bold text-xl justify-start">Subspace LCA</h2>
<p>
In our study, we extend the LCA to investigate efficient encoding of natural video. We augment LCA into a heirarchical, two-layer model. The second layer design was  inspired by latent variable models, factorized representations, and disentanglement theories of perception. We hypothesized that first layer neuron representations could be further composed into a representation that disentangled motion from form via a second layer computation. The second layer neurons could be then potentially learn an efficient, higher order statistic of the data based on the subspaces formed by the features learned in the first layer. 
</p>
<br/>
<p>
The second layer neurons group the responses of the first layer neurons, introducing an aspect of locality. Instead of all first layers neurons having equal influence on other first layer neurons, groups of neurons can now only co-influence their second layer neuron to become active. The second layer neurons, however, can influence any other second layer neurons. This result of this design shares some similarity to block-sparse coding whereby groups of cells are turned on or turned off together at the same time. The activation values of the second layer neurons are determined by the energy of within group first layer neuron activities. Here we chose the L2 norm, although there are many other possibilities that could be interesting to explore. 
</p>
<br/>
<img className="rounded-xl object-contain mx-auto mt-8 mb-1" src="/images/work/show/slca_fig2.gif" width="400" height="200" />
<em className="text-xs mb-4">
Figure 5. The subspace locally competitive algorithm (SLCA) illustrated as two-layer recurrent neural network. Latent nodes in first layer have all-to-all,reccurent, lateral connections. Additionally, a second layer of neurons form subspaces over the first layer via pooling first layer responses. When input data is sequential or time-varying, the network is encouraged to learn different representations in the first and second layers.
</em>

<p>
We consider the pooling second layer as subspaces spanned by basis vectors learned in the first layer. One could also interpret Subspace LCA (SLCA) as introducing a topological ordering constraint on the first layer neurons in LCA via a neighborhood, pooling function. The topological ordering here refers to one that is more general than the x-y spatial domain of the data input, and instead refers to the topology of an abstracted feature space of high dimensions.
</p>
<br/>
<p>
The training objective for the network was to reconstruct time-varying, natural video data. The new SLCA objective function can be expressed as the following: 
</p>
<br/>

<div className="text-center">
 
<span style="display: inline-block;"> $\underset{a}{\operatorname{argmin}}(E_{slca} = \frac{1}{2} \|s - \sum_{i=1}\sum_{j=1}\sigma_{j}z_{ij}\Phi_{ij} \|+$ </span>
<span style="display: inline-block;">$\lambda \sum_{j=1}^{N}\sigma_{j}+ \beta\sum\limits_{i=1}^{N}\sum\limits_{j=1}^{M}\sum\limits_{l=1}^{M}\Phi_{i}\Phi_{i}-I_{N})$ </span>

 
</div>

Our dataset was sourced from the Hyvarinen video dataset of natural scenes (e.g. ducks in a pond or birds in a tree) and head-mounted camera footage from a student walking on UC Berkeley college campus. We extracted 16x16 patches of each frame across various time lengths in the video to train and evaluate the network. 


<img className="rounded-xl mx-auto object-contain mt-8 mb-1" src="/images/work/show/slca_figure1_vid.gif" width="600" height="300" />
<em className="text-xs mb-4">
Figure 6. Experimental setup for encoding natural video with SLCA. 16 x 16 patches were first extracted from the video and then fed into the recurrent SLCA network. The dimensionality of the latent representation was varied from 1x to 5x of the input dimension. 
</em>


<h3 className="text-left font-bold text-lg justify-start">Results</h3>

<img className="rounded-xl object-contain mx-auto mt-8 mb-1" src="/images/work/show/SLCA_Figure3_Features.png" width="500" height="400" />
<em className="text-xs mb-4">
Figure 7. Visualized learned features grouped by subspace. The subspaces appear to learn to span some transformations space on a feature vector which allows the second layer neuron activations to be invariant to those within-group transformations.
</em>

After applying the subspace topological constraint on the SLCA model, we discovered interesting insights into the characteristics of the learned latent features and their subspaces. We found that the first layer learned "edge detector"-like features similar to previous research [1], while the second layer neurons learned to group first layer features that shared general shape and orientation but varied in phase and spatial translations (Fig 3). These results suggest that the learned subspaces can be invariant to spatial translations and phase modulations in the observed data. These translations and modulations correspond to motion in visual perception [6] 

<br/>

The temporal behavior of the second layer neurons compared to the first is also revealing. For video, these second layer neurons activations were also more stable over time compared to the first layer, suggesting an adherence to a manifold.  

<img className="rounded-xl object-contain mt-8 mb-1 mx-auto" src="/images/work/show/SLCA_Figure5_Stability.png" width="500" height="400" />
<em className="text-xs mb-4">
Figure 8. Temporal stability appears stronger for second layer neurons compared to first layer neurons. 
</em>



<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/SLCA_Figure4_GroupSize.png" width="400" height="400" />
<em className="text-xs mb-4">
Figure 9. Visualized learned features grouped by subspace. The subspaces appear to learn to span some transformations space on a feature vector which allows the second layer neuron activations to be invariant to those within-group transformations.
</em>


<h3 className="font-bold pt-2 pb-2">
Conclusion
</h3>
There are several interesting takeways from this experiment for unsupervised encoding of video in a overcomplete sparse coding network. First, heiarachical structure appears to be necessary to express the high-order dependencies intrinsic in video. Additionally, the higher-order statistical structure can be extracted with topological grouping introduced in a layer beyond the first layer of processing. Data reconstruction also appears to be benefitted by increasing the number of first-layer units within a group. However, as the group size grows, there  will also be a growth in the fraction of neurons that will need to be active to maintain a low reconstruction loss. Our experiments show that a good efficiency trade-off occurs arround group sizes of four where roughly a third of the neurons within the group are active. Future interesting research directions could explore scaling this network to process larger frames and on full color video. 

<br/>

<h4 className="font-bold pt-2 pb-2 mt-8">References</h4>
<ul className="list-decimal text-sm">

<li>
1. Dong, D. W. & Atick, J. J. Statistics of natural time-varying images. Netw. Comput. Neural Syst. 6(3), 345–358 (1995).
</li>
<br/>
<li>
2. Cai L., Krishna V., Hladnik T., Guilbeault N., Vijayakumar C., Arunachalam M., Junti S., Arrenberg A., Thiele T., Cooper E. Spatiotemporal visual statistics of quatic environments in the natural habitates of zerbrafish. 2023. doi: 10.1038/s41598-023-36099-z.
</li>
<br/>

<li>
3. van Hateren, J. H. A theory of maximizing sensory information. Biol. Cybern. 68(1), 23–29 (1992).
</li>
<br/>
<li>
4. Olshausen BA, Field DJ. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Res. 1997 Dec;37(23):3311-25. doi: 10.1016/s0042-6989(97)00169-7. PMID: 9425546.
</li>
<br/>

<li>
5.  Rozell C., Johnson D., Baraniuk R., Olshausen B. 2008 Sparse Coding via Thresholding and Local Competition in Neural Circuits. 
</li>
<br/>

<li>
6.  Cadieu C., Olshausen B. 2011 Learning Intermediate-Level Representations of Form and Motion from Natural Movies
</li>
<br/>

</ul>

blog

Self-organizing multimodal codes

Holographic Attention: Part I

Unsupervised Video Encoding