
<img className="rounded-xl object-contain mt-8 mb-1 mx-auto " src="/images/work/show/walk.gif" width="200" height="200" />



<h2 className="mt-8 border-slate-100 text-left font-bold text-xl justify-start">Natural video scene statistics </h2>

<p>
The brain is an inference machine, yet universal computational algorithms supporting this inference remain largely unknown. The efficient coding hypothesis suggests that the brain extracts statistical structure from noisy sensory data to construct a coherent model of the natural world [1][2]. Research has demonstrated that natural visual scenes have distinct spatiotempal statistical structure at several different scales [2][3]. Here we explore a bio-inspired generative model that uses sparse coding to learn this spatiotemporal structure in natural video scenes. 
</p>

<div className="flex flex-row justify-center gap-4 object-contain mt-8 mb-1 w-full max-w-[620px]">

<div className="flex flex-col gap-4 justify-between">
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_dongAtick1.png" width="400" height="400" />
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_dongAtick2.png" width="400" height="400" />
</div>

<div className="flex flex-col gap-4 justify-between">
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_cooper1.png" width="280" height="200" />
<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/slca_cooper2.png" width="280" height="200" />
</div>

</div>
<em className="text-xs mb-8">
Figure 1. Spatial and temporal frequency statistics of natural scenes. (Left) Seminal research from Dong & Atick revealing the power-law distribution relationship between signal intensity and signal frequency in spatial and temporal domains. (Image Source: [1]) (Right) Recent research from Cai, Cooper et al [2] has also found this statistical relationship to exist similarly in aquatic and terrestrial environments. (Image Source: [2]) 
</em>

<div>
<img className="mx-auto rounded-xl object-contain p-4 mb-1" src="/images/work/show/slca_dongAtick3.png" width="400" height="400" />
</div>
<em className="text-xs mb-8">
Figure 2. Measured pairwise probability distribution for two similar spatial locations separated by 33 ms in time. White (black) represents high (low) probability. This illustrates high correlation between the two light intensities within small separations of time, suggesting a general spatio-temporal statistical structure for natural moving scenes(Image Source: [1])
</em>


<h2 className="mt-4 border-slate-100 text-left font-bold text-xl justify-start">Sparse coding with LCA</h2>
<p>
Sparse coding is a well-explored method for unsupervised learning of statistical structure in data especially in high dimensional spaces. Unlike typical compression or bottle-neck autoencoders which simply map hi-dimensional data to low dimensional spaces, sparse coding instead maps hi-dimensional data to an overcomplete, higher dimensional embedding space but constrains the active components representing a single signal to a low dimension. The theory posits that efficient representations are formed under resource constraints and thus latent variables undergo competition to create encodings that are maximally expressive with the minimal active (non-zero) variables. The higher dimensional representational space adds fidelity, expressivity, and robustness to the encoded information as each representation becomes nearly orthogonal to the others (maximally uncorrelated). Image data is a notoriously high dimensional domain where sparse coding has also been applied extensively. Remarkably, it was shown in Olshausen et al. [4] that when applied to natural images, sparse coding learned latent representations that matched the tuning properties of simple cells in the V1 area of the primate visual cortex. When visualized in the image domain, these representations resemble "edge-dectors" or wavelet filters. Importantly, unlike hand-engineered wavelet filters, sparse coding's representation are learned from data in an unsupervised manner based on a generative model. 
</p>
<br/>
 
 Sparse coding can be modeled as a linear generative equation as the following:
<br/>
<div className="text-center mt-8">

<span style="display: inline-block;"> $s = a\Phi + v$  </span>

</div>
<br/>

The additional term 
<span style="display: inline-block;">$v$ </span> 
represents the noise in the model which is assumed to be 
<span style="display: inline-block;"> $\backsim N(0,\Sigma)$</span> 
, drawn from a Gaussian with 0 mean and diagonal variance 
<span style="display: inline-block;">$\Sigma$</span>.
<br/>
The objective function for denoising the input with a sparsity constraint is the following:
<br/>
<div className="text-center mt-8">

<span style="display: inline-block;"> $\underset{a}{\operatorname{argmin}}(E_{sc} = \frac{1}{2} \|s - \Phi \|+ (\lambda \sum_{k=1} C(a_k)))$ </span>

</div>
<br/>
<p>
Sparse coding can be implemented as an autoencoder with shared encoder-decoder weights or as a single-layer recurrent neural network with all-to-all connections, like a Hopfield network. Sparse coding models are typically trained to denoise and compress data. Signals are fed to the network which then under goes an interative winner-takes-all greedy algorithm to select the most useful feature vectors to reconstruct the signal. 
</p>

<img className="rounded-xl object-contain mx-auto mt-8 mb-1" src="/images/work/show/lca_vanilla_vid_crop.gif" width="400" height="400" />
 <em className="text-xs mb-4">
Figure 3. LCA animated as a recurrent neural network, with all-to-all connections. Neurons surpassing the activation threshold, supress the activity of all other neurons, encouraging a sparse encoding. Connections between neurons omitted for clarity.
</em>

<p>
LCA is a class of neural networks that implement sparse coding effciently as a dynamical system, taking inspiration from leaky-integrate and fire models of biological neurons which have internal states and external activations. Neurons integrate their internal state value, until an activation threshold is surpassed, increasing the value of the external state variable  a. Active neurons supress the activity of all other neurons, encouraging a sparse encoding. These models are neat, because they can also be implemented in low power analog circuits. 
</p>

<br/>

The internal state of neurons over time for LCA can be modeled as a nonlinear ordinary differential equaiton (ODE) with the following dynamical equation: 
<br/>
<div className="text-center mt-8">

<span style="display: inline-block;"> $\dot{u_{m}}(t) = \frac{1}{\tau}[b_{m}(t) - u_{m}(t) - \sum_{k \neq m} G_{m,n}(a_{n}(t))]$ </span>  

</div>
<br/>
The full dynamic system equationn for LCA can then be written as:

<div className="text-center">

<span style="display: inline-block;"> $\hat{s}(t)$  </span>

</div>
<br/>

where 

<div className="text-center">

<span style="display: inline-block;"> $a(t) = T_{\lambda}(u(t))$ </span>

</div>

<br/>
This ODE, has an equivalent mathematical expression as the continuous Hopfield Network model.

<img className="rounded-xl object-contain mt-8 mb-1" src="/images/work/show/SLCA_LCAcircuit.png" width="600" height="300" />
 <em className="text-xs mb-4">
Figure 4. The original locally competitive algorithm (LCA) represented as an analog circuit. Where u represents the internal state of the neurons and a represenst the external state. Connections between neurons omitted for clarity. (Image source:[5])
</em>

<h2 className="mt-8 text-left font-bold text-xl justify-start">Subspace LCA</h2>
<p>
In our study, we extend the LCA to investigate efficient encoding of natural video. We augment LCA into a heirarchical, two-layer model. The second layer design was  inspired by latent variable models, factorized representations, and disentanglement theories of perception. We hypothesized that first layer neuron representations could be further composed into a representation that disentangled motion from form via a second layer computation. The second layer neurons could be then potentially learn an efficient, higher order statistic of the data based on the subspaces formed by the features learned in the first layer. 
</p>
<br/>
<p>
The second layer neurons group the responses of the first layer neurons, introducing an aspect of locality. Instead of all first layers neurons having equal influence on other first layer neurons, groups of neurons can now only co-influence their second layer neuron to become active. The second layer neurons, however, can influence any other second layer neurons. This result of this design shares some similarity to block-sparse coding whereby groups of cells are turned on or turned off together at the same time. The activation values of the second layer neurons are determined by the energy of within group first layer neuron activities. Here we chose the L2 norm, although there are many other possibilities that could be interesting to explore. 
</p>
<br/>
<img className="rounded-xl object-contain mx-auto mt-8 mb-1" src="/images/work/show/slca_fig2.gif" width="400" height="200" />
<em className="text-xs mb-4">
Figure 5. The subspace locally competitive algorithm (SLCA) illustrated as two-layer recurrent neural network. Latent nodes in first layer have all-to-all,reccurent, lateral connections. Additionally, a second layer of neurons form subspaces over the first layer via pooling first layer responses. When input data is sequential or time-varying, the network is encouraged to learn different representations in the first and second layers.
</em>

<p>
We consider the pooling second layer as subspaces spanned by basis vectors learned in the first layer. One could also interpret Subspace LCA (SLCA) as introducing a topological ordering constraint on the first layer neurons in LCA via a neighborhood, pooling function. The topological ordering here refers to one that is more general than the x-y spatial domain of the data input, and instead refers to the topology of an abstracted feature space of high dimensions.
</p>
<br/>
<p>
The training objective for the network was to reconstruct time-varying, natural video data. The objective function can be expressed as the following: 
</p>
<br/>

<div className="text-center">
 
<span style="display: inline-block;"> $\underset{a}{\operatorname{argmin}}(E_{slca} = \frac{1}{2} \|s - \sum_{i=1}\sum_{j=1}\sigma_{j}z_{ij}\Phi_{ij} \|+$ </span>
<span style="display: inline-block;">$\lambda \sum_{j=1}^{N}\sigma_{j}+ \beta\sum\limits_{i=1}^{N}\sum\limits_{j=1}^{M}\sum\limits_{l=1}^{M}\Phi_{i}\Phi_{i}-I_{N})$ </span>

 
</div>

Our dataset was sourced from the Hyvarinen video dataset of natural scenes (e.g. ducks in a pond or birds in a tree) and head-mounted camera footage from a student walking on UC Berkeley college campus. We extracted 16x16 patches of each frame across various time lengths in the video to train and evaluate the network. 


<img className="rounded-xl mx-auto object-contain mt-8 mb-1" src="/images/work/show/slca_figure1_vid.gif" width="600" height="300" />
<em className="text-xs mb-4">
Figure 6. Experimental setup for encoding natural video with SLCA. 16 x 16 patches were first extracted from the video and then fed into the recurrent SLCA network. The dimensionality of the latent representation was varied from 1x to 5x of the input dimension. 
</em>


<h3 className="text-left font-bold text-lg justify-start">Results</h3>

<img className="rounded-xl object-contain mx-auto mt-8 mb-1" src="/images/work/show/SLCA_Figure3_Features.png" width="500" height="400" />
<em className="text-xs mb-4">
Figure 7. Visualized learned features grouped by subspace. The subspaces appear to learn to span some transformations space on a feature vector which allows the second layer neuron activations to be invariant to those within-group transformations.
</em>

After applying the subspace topological constraint on the SLCA model, we discovered interesting insights into the characteristics of the learned latent features and their subspaces. We found that the first layer learned "edge detector"-like features similar to previous research [1], while the second layer neurons learned to group first layer features that shared general shape and orientation but varied in phase and spatial translations (Fig 3). These results suggest that the learned subspaces can be invariant to spatial translations and phase modulations in the observed data. These translations and modulations correspond to motion in visual perception [6] 

<br/>

The temporal behavior of the second layer neurons compared to the first is also revealing. For video, these second layer neurons activations were also more stable over time compared to the first layer, suggesting an adherence to a manifold.  

<img className="rounded-xl object-contain mt-8 mb-1 mx-auto" src="/images/work/show/SLCA_Figure5_Stability.png" width="500" height="400" />
<em className="text-xs mb-4">
Figure 8. Temporal stability appears stronger for second layer neurons compared to first layer neurons. 
</em>



<img className="mx-auto rounded-xl object-contain p-4" src="/images/work/show/SLCA_Figure4_GroupSize.png" width="400" height="400" />
<em className="text-xs mb-4">
Figure 9. Visualized learned features grouped by subspace. The subspaces appear to learn to span some transformations space on a feature vector which allows the second layer neuron activations to be invariant to those within-group transformations.
</em>


<h3 className="font-bold pt-2 pb-2">
Conclusion
</h3>
There are several interesting takeways from this experiment for unsupervised encoding of video in a overcomplete sparse coding network. First, heiarachical structure appears to be necessary to express the high-order dependencies intrinsic in video. Additionally, the higher-order statistical structure can be extracted with topological grouping introduced in a layer beyond the first layer of processing. Data reconstruction also appears to be benefitted by increasing the number of first-layer units within a group. However, as the group size grows, there  will also be a growth in the fraction of neurons that will need to be active to maintain a low reconstruction loss. Our experiments show that a good efficiency trade-off occurs arround group sizes of four where roughly a third of the neurons within the group are active. Future interesting research directions could explore scaling this network to process larger frames and on full color video. 

<br/>

<h4 className="font-bold pt-2 pb-2 mt-8">References</h4>
<ul className="list-decimal text-sm">

<li>
1. Dong, D. W. & Atick, J. J. Statistics of natural time-varying images. Netw. Comput. Neural Syst. 6(3), 345–358 (1995).
</li>
<br/>
<li>
2. Cai L., Krishna V., Hladnik T., Guilbeault N., Vijayakumar C., Arunachalam M., Junti S., Arrenberg A., Thiele T., Cooper E. Spatiotemporal visual statistics of quatic environments in the natural habitates of zerbrafish. 2023. doi: 10.1038/s41598-023-36099-z.
</li>
<br/>

<li>
3. van Hateren, J. H. A theory of maximizing sensory information. Biol. Cybern. 68(1), 23–29 (1992).
</li>
<br/>
<li>
4. Olshausen BA, Field DJ. Sparse coding with an overcomplete basis set: a strategy employed by V1? Vision Res. 1997 Dec;37(23):3311-25. doi: 10.1016/s0042-6989(97)00169-7. PMID: 9425546.
</li>
<br/>

<li>
5.  Rozell C., Johnson D., Baraniuk R., Olshausen B. 2008 Sparse Coding via Thresholding and Local Competition in Neural Circuits. 
</li>
<br/>

<li>
6.  Cadieu C., Olshausen B. 2011 Learning Intermediate-Level Representations of Form and Motion from Natural Movies
</li>
<br/>

</ul>

Unsupervised Video Encoding

Self-organizing multimodal codes

Holographic Attention: Part I 


<div className="flex justify-center grid grid-cols-2 gap-4 max-w-[620px] w-full rounded-xl">

<img src="/images/work/show/terroir_seg.gif" alt="drawing" width="300" className="rounded-xl"/>


<img src="/images/work/show/terroirClusterCount.jpg" alt="drawing" width="300" className="rounded-xl" />

</div>
<em className="text-xs mt-1 mb-4">
Figure 1. (Left) Automated video grape segmentation from data collected using RGB and IR machine vision cameras, an NVIDA Jetson, and deep convolutional neural networks.(Right) Canopy coverage and leaf-grape color similarity presented the some of the largest challenges for imaging. Sonoma, California (2018)
</em>

In 2018, I co-founded an agtech startup with college classmates called Terroir AI. Our mission was to help vineyard farmers minimize their frequent annual crop losses by improving the coverage and precision of their crop monitoring and management. Our solution was an autonomous AI computer vision imaging system...

<br/> 
Crop yields are uncertain for farmers every season, due to both environmental factors and lack of precise data oversight over crop growth. Wine grape farmers, who supply a niche market segment, are particularly susceptible to variations in crop yield.
We set out to reduce some of this uncertainty with AI and machine learning. AI infrastructure and models were progressing and the time was ripe for applying new advances in deep-learning to help vineyards, but in our journey we realized there was no low hanging fruit.🍇

<br/>

<img src="/images/work/show/terroirAI_1.jpeg" alt="TerroirAI sensor system" className="rounded-xl mt-8 mb-8" width="500" height="300"/>

Initially, we explored using drones for image capture, but we soon discovered that aerial imaging was unsuitable for crops covered by a canopy. Additionally, the imaging system we required was a heavy payload for commercial UAV drones at the time. We utimately narrowed our focus to real-time data capture from a ground vehicle. Although the autonomous aspect of the system wasn't fully realized, we designed it with the potential for gradual autonomy.

<div className="flex flex-row gap-4 justify-between mt-8 ">

<img src="/images/work/show/terroirAI_4.jpg" alt="Aaron and steven with sensor system" className="mx-auto rounded-xl" width="250" height="200"/>


<img src="/images/work/show/terroirAI_5.jpg" alt="Sensor system components" className="mx-auto rounded-xl" width="250" height="200"/>

</div>
<em className="text-xs mt-1 mb-8">
Figure 3. (Left) Aaron and I collecting data in Napa Valley, California. (Right) Internals of imaging system. The sensor system included multiple cameras with infrared and RGB sensors with synced capture. The system required careful consideration for variables like focal distance, sensor pitch, shutter speed that could be sufficient in highly variable landscapes. Building a field-rad computer and camera system was an obstacle as the hot environment quickly overheated the system and dust from the ground often impacted visibility. We engineered an enclosure with aluminum and a transparent acrylic viewing window that did improve the system's durability and reliability to some degree.
</em>
<br/>

I led technical product development for the field-ready computer vision system that could segment and count grape clusters. The system also integrated IMU data with the segmented image data to achieve SLAM and link location coordinates to the grape cluster counts. We collaborated with and received guidance from engineers at Stanford Research Institute (SRI) in both Palo Alto, California and Princeton, New Jersey.

<br/>


From the outset, I was responsible for making both software and hardware design decisions concurrently to design our product end-to-end. The project required applying diverse engineering skills, ranging from physical optics to machine learning and deep learning. Our cameras and computers needed to function in the hot, arid environment of vineyards while also withstanding the bumpy roads and added occlusion from dust, rocks, and vines flying around. Simultaneously, the computer vision algorithm needed the clearest views possible to detect the fruiting grapes at various angles. The system also needed substantial memory to store, transform, and analyze hundreds of thousands of images on the edge. Even minute mechanical details such as the appropiate camera lens power, their orientation or their pitch impacted the video signal suitability for the performance neural network. Additionally, the shutter speed and syncing of cameras impacted software decisions for data batching and memory management for the video data analysis. Designing such a machine vision system end-to-end was an exciting challenge, and it highlighted the interdependence of all the engineered parts.

<br/>

It wasn't long before we realized the need for efficient edge compute since internet connection on large acres of vineyards is very rare to get. We opted for the Nvidia Jestson computer for our main compute. I trained several image segmentation deep neural networks on the grapevine image data we collected. Through much trial and error, we were eventually able to run real-time image segmentation. The other challenge we ran into was image data compression. For image compression, we developed several techniques to utilize sparse coding and we had to experiment with a variety of methods for batching images and quickly freeing up more space for long continous collections. 



<img src="/images/work/show/terroirErrors.jpg" alt="Imaging errors" className=" mx-auto rounded-xl mt-8" width="400" height="200"/>
<em className="text-xs mt-1 mb-4">
Figure 4. Grape and leaf color similarity and occulsion posed some of the largest challenges for the imaging system. We built an adaptive statistical model to estimate average miss rate to calculate full yield. 
</em>

<br/>

The engineering and design process was demanding, but it was fulfilling to see the components come together after months of planning and prototyping. In this multifaceted domain, numerous engineering and economic tradeoffs were necessary.

<br/>

<div className="flex justify-center grid grid-cols-2 gap-4 max-w-[620px] w-full rounded-xl mb-4 mt-4"> 
<img src="/images/work/show/terroirAI_3.jpeg" alt="Sensor system components" className=" mx-auto rounded-xl" width="400" height="200"/>


<img src="/images/work/show/terroirAI_2.jpeg" alt="Sensor system components" className=" mx-auto rounded-xl" width="400" height="200"/>

</div>

Our diligence and dedication led us to a successful round of field tests in Napa Valley in California and in the Mendoza wine region in Argentina. Unfortunately our demos came to an abrupt stop when the wildfires of 2020 hit Northern California and we had to adjust our data collection and model training game plan. The long harvest season cycles constrained the speed of product development of our product, so we had to restructure and pivot the project after some time. Nonetheless, it was an incredibly rich learning experience that revealed many of the critical components involved in providing reliable AI-powered solutions in the real world.

Terroir AI


<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/sk_cover_v1.png" width="600" height="300" />

<h2 className="mt-8 text-left font-bold text-xl justify-start">Expanding human potential for entrepreneurship</h2>
<p className="mt-4">
Every company starts from an idea, but transforming an idea to a profitable business can be a long, treacherous journey. Founders need to build a team, build product, deliver product, market to customers, get feedback, and let's not forget to stay on top of the accounting and legal paperwork. 
</p>
<br/>
<p>
It's a true superhero task, and every superhero needs a sidekick...
</p>
<br/>
<p className="">
Sidekiik is here to the rescue for the next generation of ambitious companies just getting started! Whether starting a e-commerce brand, podcast, or the next social media unicorn, Sidekiik can help founding team individuals cover all the bases of their business operations. What makes Sidekiik special is its multi-agent system that helps guide users through their business development journey. These agents can integrate with automations for the many facets of business operations from accounting, to customer support, content creation and marketing. 
</p>
<br/>


Sidekiik


<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/saka-cover2.png" width="600" height="300" />

Sakajawea is an AI web application to practice conversation in multiple langauges with text or voice. Sakajawea aims to make learning languages fun and accessible, and help contribute to the preservation of endangered languages. All models are open source and run locally using WebGPU, making language practice more accessible than ever!

<br/>

Scienctific research suggests that language learning in humans can often be optimized by learning multiple languages at once, yet most language learning apps only emphasize one language at a time. Sakajawea can be used to practice multiple language switching within the same conversation. Learners can tune the speed and mix and match languages. Try two at a time, then three... how many can you do? 

<br/>

Update [Feb 2025]

Current testing underway for integrating text-to-speech (TTS) models so that the AI chatbot's responses can be vocalized. Still 100% running in the browser!





Sakajawea

<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/beezneez-projectFig1.png" width="600" height="300" />

<h2 className="mt-8 text-left font-bold text-xl justify-start">Business intelligence driven by agentic AI</h2>
<p className="mt-8 mb-8">
Capture signal, cut the noise. Beezneez is a modern, flexible, business intelligence platform for investors powered by AI agents. Beezneez is the new way to get into the details of a business to make important investment decisions. It's Value Line, but better. Users can create their own dashboard, curate their own indicators, ratios, risk metric, and management profiling, all from one knowledge base.
</p>

<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/beezneez-projectFig2.png" width="600" height="300" />
<br/>
<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/beezneez-projectFig3.png" width="600" height="300" />
<br/>
<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/beezneez-projectFig4.png" width="600" height="300" />
<br/>
<img className="rounded-xl object-contain mt-4 mb-1" src="/images/work/show/beezneez-projectFig5.png" width="600" height="300" />


Beezneez


<img className="rounded-xl object-contain mt-4 mb-1 mx-auto" src="/images/work/show/quasar-app-cover.png" width="600" height="300" />

<p className="mt-2 mb-8">

I co-founded Quasar, a company that developed a digital asset management platform without borders. A major challenge in blockchain ecosystems is the fragmentation between different blockchains, which limits smart contracts' ability to operate across the broader market. We introduced key innovations, including core contributions to the inter-blockchain communication (IBC) protocol, enabling communication between separate and disparate blockchains.
</p>

<br/>

<p className="mt-8 mb-8">
Leading product and technical development, I helped raise over $11M in funding and expand our global team 3x. Building a product and culture in a remote-first setting was a great learning experience. I honed my skills for rapid prototyping, program management, and software deployment.  
</p>


hi, i'm steve!🖖🏾

blogs. some of my thinkings