Find instances by segmenting object boundaries
Introduction¶
Today we are going to consider wery interesting, exotic and novel concept: how to adopt any segmentation neural network to find instances. While others use Mask-RCNN (and of course it is good enought for many tasks), today we are going to propose another approach.
The main idea is the following: we will add additional class (object boundary). It will allow us to train model to segment objects with their boundaries, and then we will be able to use these boundaries to split objects instances via DTL query.
Task description¶
Deep Learning has a lot of applications in agriculture. Today we will build system to perform instance segmentation of plant leafs. We will use Aberystwyth Leaf Evaluation Dataset.
Here is the example of image:
Reproduce results
You can download the tutorial data and reproduce entire experiment yourself.
Combine described approaches with other tutorials
We recommend to combine these techniques with other tutorials to obtain better results.
To solve this task we should combine a few approaches: sliding window and adding the additional class (automatically using DTL)
Step 1. Upload Aberystwyth Leaf Evaluation Dataset¶
a) Download the original dataset images_and_annotations.zip
b) Choose the aberystwyth
import option
c) Drag images_and_annotations
dir to the upload window
d) Name this project aberystwyth
As you can see, the annotations look unusual. Here is an example of a single object:
Step 2. DTL #1 to split leafs to separate instances¶
-
Layer #1 (
"action": "data"
) takes all data from projectaberystwyth
and keeps classes as they are. -
Layer #2 (
"action": "split_masks"
) splits masks of classleaf
into connected components. As a result from one mask with a few leafs we will get separate masks for each leaf. -
Layer #3 (
"action": "supervisely"
) saves results to the new projectaberystwyth_splitted
.
[ { "dst": "$sample", "src": [ "aberystwyth/*" ], "action": "data", "settings": { "classes_mapping": "default" } }, { "dst": "$sample1", "src": [ "$sample" ], "action": "split_masks", "settings": { "classes": [ "leaf" ] } }, { "dst": "aberystwyth_splitted", "src": [ "$sample1" ], "action": "supervisely", "settings": {} } ]
Step 3. Dtl #2 to apply augmentations and prepare training dataset¶
In this DTL query we apply transformations, filter images and split them into train and validation sets.
-
Layer #1 (
"action": "data"
) takes all data from the projectaberystwyth_splitted
and keeps classes as they are. -
Layer #2 (
"action": "flip"
) flips the data horisontally. -
Layer #3 (
"action": "multiply"
) creates 10 copies for each image. -
Layer #6 (
"action": "crop"
) performs random crops (from 20% to 25% in width and from 25% to 30% in height with respect to the image size) -
Layer #7 (
"action": "if"
) filters the data (sends the data containing at least one object to the first branch, all other data will be sent tonull
). -
Layer #8 (
"action": "if"
) randomly splits the data into two branches: first branch - 95% (will be tagged astrain
) and second branch - 5% (will be tagged asval
). -
Layer #9 (
"action": "tag"
) adds the tagtrain
to all input images. -
Layer #10 (
"action": "tag"
) adds the tagval
to all input images. -
Layer #11 (
"action": "rename"
) makes copies of objects of the classleaf
and renames them astmp-contour
-
Layer #12 (
"action": "line2bitmap"
) transforms the objects into their contours. -
Layer #13 (
"action": "crop"
) crops 3px stripes from each side to drop the boundary contours. -
Layer #14 (
"action": "supervisely"
) saves results to the new projectaberystwyth_random_crop_train
.
[ { "dst": "$sample", "src": [ "aberystwyth_splitted/*" ], "action": "data", "settings": { "classes_mapping": "default" } }, { "dst": "$vflip", "src": [ "$sample" ], "action": "flip", "settings": { "axis": "vertical" } }, { "dst": "$multi", "src": [ "$sample", "$vflip" ], "action": "multiply", "settings": { "multiply": 10 } }, { "dst": "$crop", "src": [ "$multi" ], "action": "crop", "settings": { "random_part": { "width": { "max_percent": 25, "min_percent": 20 }, "height": { "max_percent": 30, "min_percent": 25 } } } }, { "dst": [ "$filter", "null" ], "src": [ "$crop" ], "action": "if", "settings": { "condition": { "min_objects_count": 1 } } }, { "dst": [ "$to_train", "$to_val" ], "src": [ "$filter" ], "action": "if", "settings": { "condition": { "probability": 0.95 } } }, { "dst": "$train", "src": [ "$to_train" ], "action": "tag", "settings": { "tag": "train", "action": "add" } }, { "dst": "$val", "src": [ "$to_val" ], "action": "tag", "settings": { "tag": "val", "action": "add" } }, { "dst": "$100", "src": [ "$train", "$val" ], "action": "duplicate_objects", "settings": { "classes_mapping": { "leaf": "tmp-contour" } } }, { "dst": "$101", "src": [ "$100" ], "action": "line2bitmap", "settings": { "width": 3, "classes_mapping": { "tmp-contour": "contour" } } }, { "dst": "$102", "src": [ "$101" ], "action": "crop", "settings": { "sides": { "top": "3px", "left": "3px", "right": "3px", "bottom": "3px" } } }, { "dst": "zenodo_random_crop_train", "src": [ "$102" ], "action": "supervisely", "settings": {} } ]
Detailed description.
Here are a few examples of images after this DTL query.



Step 4. Train neural network¶
Basic step by step training guide is here. It is the same for all models inside Supervisely. Detailed information regarding training configs is here.
UNetV2 weights were initialized from the corresponding model from the Models list (UNetV2 with VGG weigths that was pretrained on ImageNet).
Resulting model will be named unet-leafs
. Project zenodo_random_crop_train
is used for training.
{ "lr": 0.001, "epochs": 5, "val_every": 1, "batch_size": { "val": 3, "train": 3 }, "input_size": { "width": 512, "height": 512 }, "gpu_devices": [ 0, 1, 2 ], "data_workers": { "val": 0, "train": 3 }, "dataset_tags": { "val": "val", "train": "train" }, "special_classes": { "neutral": "neutral", "background": "bg" }, "weights_init_type": "transfer_learning" }
Training takes 15 minutes on three GPU devices.
After training the last model checkpoint is saved to "Neural Networks" list.
Monitor training charts and test various checkpoints
We recommend to carefully monitor the training charts to prevent overfitting or underfitting. Especially it is very important when we use a small training dataset. In this case restoring checkpoints is a key component to successful research.
Step 5. Apply NN to test images.¶
Basic step by step inference guide is here. It is the same for all models inside Supervisely. Detailed information regarding inference configs is here.
We apply the model unet-leafs
to the project with test images.
Config for sliding window inference:
{ "mode": { "save": false, "source": "sliding_window", "window": { "width": 512, "height": 512 }, "min_overlap": { "x": 0, "y": 0 } }, "gpu_devices": [ 0 ], "model_classes": { "add_suffix": "_dl", "save_classes": [ "contour", "leaf" ] }, "existing_objects": { "add_suffix": "", "save_classes": [] } }
Afterwards leaf instances can be extracted from the segmentation mask using DTL query like in this tutorial.
Conclusion¶
As you can see the results are good enough.
Use any segmentation model
You can use any segmentation model with the approach described above.