Terroir AI

iOS App Figure 1. (Left) Automated video grape segmentation from data collected using RGB and IR machine vision cameras, an NVIDA Jetson, and deep convolutional neural networks.(Right) Cluster count estimates by acre using probabilistic machine learning conditioned on data collected from our imaging system. Sonoma, California (2018)

In 2018, I co-founded an agtech startup with college classmates called Terroir AI. Our mission was to help vineyard farmers improve the coverage and precision of their crop monitoring to minimize their frequent annual crop losses due to pest and diseases. Our solution was an autonomous AI computer vision imaging system…

Crop yields are uncertain for farmers every season, due to both environmental factors and lack of precise data oversight over crop growth. Wine grape farmers, who supply a niche market segment, are particularly susceptible to variations in crop yield. We set out to reduce some of this uncertainty with AI and machine learning. AI infrastructure and models were progressing and the time was ripe for applying new advances in deep-learning to help vineyards, but in our journey we realized there was no low hanging fruit 🍇

Initially, we explored using drones for image capture, but we soon discovered that aerial imaging was unsuitable for crops covered by a canopy. Additionally, the imaging system we required was a heavy payload for commercial UAV drones at the time. We utimately narrowed our focus to real-time data capture from a ground vehicle. Although the autonomous aspect of the system wasn’t fully realized, we designed it with the potential for gradual autonomy.

Figure 3. (Left) Aaron and I collecting data in Napa Valley, California. (Right) Internals of imaging system. The sensor system included multiple cameras with infrared and RGB sensors with synced capture. The system required careful consideration for variables like focal distance, sensor pitch, and shutter speed that also had to adapt to highly variable landscapes. Building a field-rad computer and camera system was an obstacle as the hot environment quickly overheated the system and dust from the ground often impacted visibility. We engineered an enclosure with aluminum and a transparent acrylic viewing window that improved the system's durability and reliability.

I led technical product development for the field-ready computer vision system that could segment and count grape clusters. The system also integrated IMU data with the segmented image data to achieve SLAM and link location coordinates to the grape cluster counts. We collaborated with and received guidance from engineers at Stanford Research Institute (SRI) in both Palo Alto, California and Princeton, New Jersey.

From the outset, I was responsible for making both software and hardware design decisions concurrently to design our product end-to-end. The project required applying diverse engineering skills, ranging from physical optics to machine learning and deep learning. Our cameras and computers needed to function in the hot, arid environment of vineyards while also withstanding the bumpy roads and added occlusion from dust, rocks, and vines flying around. Simultaneously, the computer vision algorithm needed the clearest views possible to detect the fruiting grapes at various angles. The system also needed substantial memory to store, transform, and analyze hundreds of thousands of images on the edge. Even minute mechanical details such as the appropiate camera lens power, their orientation or their pitch impacted the video signal suitability for the performance neural network. Additionally, the shutter speed and syncing of cameras impacted software decisions for data batching and memory management for the video data analysis. Designing such a machine vision system end-to-end was an exciting challenge, and it highlighted the interdependence of all the engineered parts.

It wasn’t long before we realized the need for efficient edge compute since internet connection on large acres of vineyards is very rare to get. We opted for the Nvidia Jestson computer for our main compute. I trained several image segmentation deep neural networks on the grapevine image data we collected. Through much trial and error, we were eventually able to run real-time image segmentation. The other challenge we ran into was image data compression. For image compression, we developed several techniques to utilize sparse coding and we had to experiment with a variety of methods for batching images and quickly freeing up more space for long continous collections.

Figure 4. Grape and leaf color similarity and occulsion posed some of the largest challenges for the imaging system. We built an adaptive statistical model to estimate average miss rate to calculate full yield.

The engineering and design process was demanding, but it was fulfilling to see the components come together after months of planning and prototyping. In this multifaceted domain, numerous engineering and economic tradeoffs were necessary.

Our diligence and dedication led us to a successful round of field tests in Napa Valley in California and in the Mendoza wine region in Argentina. Unfortunately our demos came to an abrupt stop when the wildfires of 2020 hit Northern California and we had to adjust our data collection and model training game plan. The long harvest season cycles constrained the speed of product development of our product, so we had to restructure and pivot the project after some time. Nonetheless, it was an incredibly rich learning experience that revealed many of the critical components involved in providing reliable AI-powered solutions in the real world.