Tamara Todić

Junior Researcher, BioSense Institute

In the fields we often see some undesirable patterns, like weed clusters, planter skip, double plant, standing water or waterway. All of this has a significant effect on the amount and quality of yields, so there exists interest in detecting these patterns in order to take appropriate actions timely and that way reduce or even eliminate possible losses.

Images captured by drone are suitable for detection of these patterns. They provide a view from bird’s perspective, the image can be taken whenever it’s needed (no need to wait for the satellite to come over the field of interest), and the clouds are easier to avoid – drones fly on much lower altitudes than satellites.

Database we used is given for the Agriculture-Vision Challenge that was organized for the first time by the CVPR conference in 2020. The example from the dataset is given on the image above. It consists of 12901 images of corn and soy fields from Illinois and Iowa territories. The images are captured by drone, from a bird’s perspective. They contain RGB and NIR (Near InfraRed) channels. As well as the visible part of electromagnetic spectra, NIR bands are used often in image processing in agriculture. It corresponds to the health of the plants – healthy plants have stronger reflectance in NIR bands than the unhealthy. All the images from the dataset are labeled by agronomy experts. There are 6 types of patterns present in the dataset:

• Cloud shadow
• Planter skip
• Double plant
• Standing water
• Waterway
• Weed cluster

For every image in the database there is a mask that shows valid pixels where no aberrations happened during the acquisition, as well as a mask that shows pixels of interest. When evaluating the model, only the valid pixels of interest are considered.

The problem was first solved for four out of six present classes:
• Planter skip
• Double plant
• Standing water
• Waterway

In the dataset there is the imbalance in the number of images per class – the most examples belong to weed clusters and cloud shadow classes. Because of that, these two classes were not included in the first solution.
Also there is a problem of very small areas of certain pattern classes, so that there is a very small number of anomaly pixels compared to the normal pixels, which makes detection even harder.

The problem is being solved using deep convolutional neural networks. More specifically, architectures for semantic segmentation are used, where every pixel is assigned to one of all possible classes. These networks have a huge number of parameters to be learned in the process of training. The training requires lots of time and memory resources. It is also necessary to get as many pictures as possible, because the model gives better results if trained on many different examples.

Some initial results are shown on the following image. The first column contains RGB images, the second NIR images, the third labels and the fourth model predictions.