FruitNeRF++: A Generalized Multi-Fruit Counting Method Utilizing Contrastive Learning and Neural Radiance Fields

Authors: Lukas Meyer, Andrei-Timotei Ardelean, Tim Weyrich, Marc Stamminger · IROS 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Adds a contrastively trained instance field to a fruit NeRF so fruits of any type can be counted by clustering a 3D point cloud, without fruit-specific shape templates.

Why it matters

One pipeline handles six fruit types without per-fruit templates, using unordered photographs and off-the-shelf foundation models for the masks. For orchard yield estimation this removes the per-fruit, per-growth-stage tuning that template-based clustering required.

Problem

Fruit counting supports harvest workforce planning, packaging, storage and processing, but most existing methods are built for one fruit type. The predecessor FruitNeRF lifts RGB and semantic masks into a neural field, but its cascaded clustering relies on fruit-specific templates and several sensitive hyper-parameters, which must change with fruit type and growth stage.

Proposed solution

FruitNeRF++ adds a neural instance field to the NeRF and trains it with an InfoNCE-style contrastive loss on 2D instance masks from Grounded-SAM or Detic, using the mean feature of each fruit as the positive and neighbouring fruits as hard negatives. The density and colour fields are trained first, then the semantic field, then the instance field alone with the others frozen. The scene is sampled into a fruit point cloud carrying 32-dimensional instance features, split into partitions with k-means, and each partition is clustered with HDBSCAN using a weighted sum of cosine feature distance and Euclidean distance.

Key insights

Using one prototype (mean) feature per fruit as the positive, instead of contrasting every pixel pair, gave a better training signal for the instance field.
Uniform pixel sampling suits NeRF colour training but not instance learning, because background pixels dominate; sampling groups of neighbouring fruits supplies hard local negatives.
Clustering on cosine feature distance plus Euclidean position distance worked better than either alone; on synthetic data most separation came from position because touching fruits are rare. Section IV-E gives λe = 5 as the Euclidean weight, while Section V states λe = 1 for the synthetic experiments and λe = 5 for FUJI.
A contrastive temperature τ between 0.1 and 0.3, peaking at 0.2, worked on synthetic data; outside that range the clustering could not separate instances.
Mask quality limits counts: on synthetic data Grounded-SAM masks reached a mean IoU of 0.561 and Detic 0.49, and a leaf or branch across a fruit can split it into two instances.

Main results

Average F1 over six synthetic fruit types: 0.925 with ground-truth masks, 0.832 with Grounded-SAM masks and 0.776 with Detic masks.
With ground-truth masks, per-fruit F1 for FruitNeRF++ vs FruitNeRF (Table II, FruitNeRF values taken from its own paper): apple 0.996 vs 1, plum 0.859 vs 0.885, lemon 0.978 vs 0.978, pear 0.996 vs 0.971, peach 0.996 vs 0.987, mango 0.713 vs 0.873.
Mango was the hardest case (649 of 1150 counted with ground-truth masks), which the authors attribute to the large number of fruits and heavy occlusion.
Real FUJI apple dataset (582 images, 12 trees): F1 0.765 with Grounded-SAM masks, against 0.881 for the fruit-specific method of Gené-Mola et al.
On the Messy Rooms dataset of 100 household objects, 99 objects were detected.
For apples, F1 reaches 1 at an embedding size of D = 16; plums and mangoes need larger embeddings.

Limitations

Training on a larger scene such as FUJI takes about 8 hours on an Nvidia A5000 (24 GB), so the method is not real-time; slow convergence of the instance field accounts for more than half of that time.
Results depend on good coverage of the tree; noisy data, mainly inaccurate camera poses, lets one fruit's embeddings contaminate its neighbours.
Instance segmentation tends to predict two instances when a leaf or branch visually splits a fruit, pulling the two halves' embeddings apart.
Fruits that grow in clusters, such as bananas, berries or dates, need further research to estimate cluster sizes and counts.
Real-world evaluation covers a single apple dataset whose camera poses needed COLMAP plus manual registration of both tree sides; the authors attribute the gap to Gené-Mola et al. to these noisy poses.

Field impact

FruitNeRF++ was an IROS 2025 award finalist and has 5 citations. The authors give a code repository, github.com/meyerls/FruitNeRFpp, with the code to be released on acceptance, and extend the FruitNeRF multi-fruit dataset with instance masks.

Abstract

We introduce FruitNeRF++, a novel fruit-counting approach that combines contrastive learning with neural radiance fields to count fruits from unstructured input photographs of orchards. Our work is based on FruitNeRF, which employs a neural semantic field combined with a fruit-specific clustering approach. The requirement for adaptation for each fruit type limits the applicability of the method, and makes it difficult to use in practice. To lift this limitation, we design a shape-agnostic multi-fruit counting framework, that complements the RGB and semantic data with instance masks predicted by a vision foundation model. The masks are used to encode the identity of each fruit as instance embeddings into a neural instance field. By volumetrically sampling the neural fields, we extract a point cloud embedded with the instance features, which can be clustered in a fruit-agnostic manner to obtain the fruit count. We evaluate our approach using a synthetic dataset containing apples, plums, lemons, pears, peaches, and mangoes, as well as a real-world benchmark apple dataset. Our results demonstrate that FruitNeRF++ is easier to control and compares favorably to other state-of-the-art methods.

Open Source Robotics