CLUST 2015

MICCAI 2015 Challenge on Liver Ultrasound Tracking
Munich, October 9, 2015 (Full Day)

MICCAI 2015 CLUST is now closed! In this page you can find a summary of the MICCAI workshop, including the final program and the proceedings.

However, we have an open challenge based on the CLUST 2015 data. Results submitted to this challenge will be included in the open challenge leaderboard.


Data usage

Participants agree that the data provided to them shall be used solely for the purpose of participating in the CLUST challenge and shall not be used, disclosed or distributed to any other person under any circumstance.

The tracking performance will be evaluated by the organizers after submission of the tracking results. Only this evaluation provides the official CLUST tracking performance and shall be reported in any publication to allow comparison.

When publishing the results based on the challenge data, participants shall cite the challenge and the references listed at the bottom of this page.

Method's parameters

To avoid unrealistic adjustment of the method's parameters per sequence, parameters should either be automatically determined or should be fixed for at least all sequences per ultrasound scanner type and task. It should be clear from the description of the method that this rule was followed by either specifying the automatic procedure or by stating the values of the fixed parameters.


Participants shall submit results and a description of the method, following the instructions listed in Submission. The description will serve the organizers to identify and discriminate between methods, and to verify that the rules regarding method's parameters were fulfilled. The description will only be released as informal submission (see below) after explicit request from the participant. The evaluation of the submitted results will be done by the challenge organizers.

By submitting their results, participants agree that these will be included on the online leaderboard webpage. There are three result categories:

Anonymous submissions, which have not progressed to the other two categories within 12 months after submission, will be removed from the leaderboard.

Results should only be compared to official submissions.

Participants shall notify the challenge organizers about any publication which is (partly) based on CLUST data.



This challenge contains of a total of 86 independent datasets

The data are anonymized and in the format of sequences of 2D images (.png) or 3D images (.mha).

The data are split into a training (40% of all sequences) and a test set (60%). Annotations are provided for the training set, to allow for some tuning of the tracking algorithm. For the test set, the annotations of the first images are provided. These need to be tracked over time.

The tracking performance of the test set will be evaluated by the organizers and summary results will be provided to the participants and published in Results, as informal or official submission, according to the Submission rules.

Part of the test data (20% of all data) will be distributed on October 9, 2015, for on-site processing during the event before noon.

For details on the submission formats, check the Submission page.


The training and test data will be available from April 9, 2015 until July 5, 2015 here.

A username and password are requested to access the data. Please contact us to receive the login details.

The test data for the on-site challenge is available here ONLY during the CLUST 2015 event, on October 9, 2015, from 08:30 to 11:30, Central European Time. Login credentials will be provided at the event.

Submission guidelines

Participants are invited to submit the following files:

  1. Tracking results for the annotations, which were provided for the first frame.
  2. Description of the tracking method.

1. Tracking results

It is possible to submit results for all data or for at least one of the following subsets:

Partial results for these subsets will not be assessed.

Data format

Please submit your tracking results in a zip archive to be named as the provided username. Additionally, name the tracking results according to the corresponding sequences and as done for the annotation of the first image. Please submit the tracking results in the same format as provided for the testing sequences of each test set subgroup. In details:

Point tracking results:

Results evaluation

The organizers will quantitatively evaluate the submitted tracking results with respect to ground truth annotations. For the evaluation we will consider the following measurement:

Error statistics will be summarized by mean, standard deviation and 95%.

We will provide the authors with the outcome of the aforementioned evaluation. After approval of the authors, we will include the quantitative evaluation of the tracking results in the MICCAI 2015 CLUST leaderboard.

2. Paper format

Participants shall provide a 4-8 page paper describing their method. Papers should be formatted in Lecture Notes in Computer Science style. The file format for submissions is Adobe Portable Document Format (PDF). Other formats will not be accepted. For further info, please refer to the MICCAI conference submission format.

Authors of accepted papers shall share their publication, as a reference to the results published in the official Results.

Upload your submission

It is possible to upload your submission here.


The results listed on this webpage are based on the first test set (40% of all data) of the CLUST 2015 challenge, as in the referenced publications. The results of all test data (60%), including the MICCAI 2015 CLUST on-site challenge, at listed here.

Results of 2D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.

Tracking error [mm]
Submission date Remark
Mean Standard deviation 95th percentile
Hallack A., et al. 0.91 1.66 2.20 Jul 2015
Ozkan E., et al. 1.04 1.48 2.26 Nov 2016
Makhinya M. and Goksel O. 1.09 1.75 2.42 Jul 2015
Kondo S. 1.09 1.35 3.07 Jul 2015
Chen et al. 1.63 1.99 4.36 Jun 2016 +300ms latency & temporal prediction
Chen et al. 2.20 2.04 5.10 Jun 2016 +300ms latency (no temporal prediction)
Nouri D. and Rothberg, A. 2.83 4.86 13.13 Jul 2015

Results of 3D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.

Tracking error [mm]
Submission date
Mean Standard deviation 95th percentile
Royer L., et al. 1.56 0.69 2.91 Jul 2015
Banerjee J., et al. 1.47 0.58 2.38 Jul 2015

* Informal submission.

Important dates

All deadlines are 23:59 Pacific Standard Time and in the year 2015.


Apr 13 Release of training data

May 26 Release of test data

May 26 Call for short papers

Jul 6 Tracking results submission deadline

Jul 8 Tracking performance notification

Jul 13 Paper submission deadline

Jul 28 Notification of acceptance

Aug 1 Workshop registration due

Aug 5 Submission deadline for accepted papers

Aug Final program

Oct 9 Workshop date

Final Program

Friday, October 9, 2015 (9.00 am - 5.30 pm)

Download the detailed schedule here.

9.15 - 9.30 am Welcome and Introductions for on-site challenge V. De Luca, E. Harris, M. Lediju Bell, C. Tanner
9.30 - 10.30 am On-site Challenge Chair: E. Harris, Institute of Cancer Research, London, UK
10.30 - 11.00 am Coffee Break
11.00 am - 12.30 pm On-site Challenge (continue) Chair: E. Harris, Institute of Cancer Research, London, UK
12.30 - 1.30 pm Lunch Break
2.00 - 2.10 pm Introduction C. Tanner, ETH Zurich, Switzerland
2.10 - 3.30 pm Oral Session 1 - 2D Tracking Chair: M. A. Lediju Bell, Johns Hopkins University, Baltimore, USA
Liver Ultrasound Tracking using a Learned Distance Metric. D. Nouri, A. Rothberg, 4Catalyzer, Inc., Guilford CT 06437, USA
Liver Ultrasound Tracking Using Kernelized Correlation Filter With Adaptive Window Size Selection. S. Kondo, Konica Minolta Inc., Osaka, Japan
Motion Tracking in 2D Ultrasound Using Vessel Models and Robust Optic-Flow. M. Makhinya, O. Goksel, Computer Vision Lab, ETH Zurich, Switzerland
Robust Liver Ultrasound Tracking using Dense Distinctive Image Features. A. Hallack1, B.W. Papiez1, A. Cifor2, M.J. Gooding2, J.A. Schnabel3, 1 Institute of Biomedical Engineering (Department of Engineering Science), University of Oxford, UK; 2 Mirada Medical, Oxford, UK; 3 Division of Imaging Sciences & Biomedical Engineering, King's College London, UK
3.30 - 4.00 pm Coffee Break
4.00 - 4.40 pm Oral Session 2 - 3D Tracking Chair: C. Tanner, ETH Zurich, Switzerland
Tracking of Non-rigid Targets in 3D US Images: Results on CLUST 2015. L. Royer1,2,3, G. Dardenne1, A. Le Bras1,4, M. Marchal1,3, A. Krupa1,2, 1 Institut de Recherche Technologique b-com, Rennes, France; 2 Inria Rennes - Bretagne Atlantique, France; 3 INSA de Rennes, France; 4 CHU de Rennes, France
A combined tracking and registration approach for tracking anatomical landmarks in 4D ultrasound of the liver. J. Banerjee1,2, C. Klink1, E. Vast1,2, W.J. Niessen1,2, A. Moelker1, T. van Walsum1,2, 1 Dept. of Radiology, Erasmus MC, Rotterdam, The Netherlands; 2 Dept. of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands
4.40 - 5.00 pm Overall Results and Conclusions V. De Luca, ETH Zurich, Switzerland
5.00 - 5.25 pm Discussion and Outlook Chair: E. Harris, Institute of Cancer Research, London, UK


The event was held in Forum 9, in the Holiday Inn Hotel, Munich.


The proceeding of MICCAI CLUST 2015 are available for download here.

A summary of the results is published in the Results listed above. In the proceedings, authors reported tracking results only for the initial test data (40% of all data).

The results are obtained from all test data (60% of all data), including the on-site challenge results (20%) are not included in the proceeding and are listed here.


Valeria De Luca, ETH Zurich, Switzerland

Emma Harris, Institute of Cancer Research, Sutton, UK

Muyinatu A. Lediju Bell, Johns Hopkins University, USA

Christine Tanner, ETH Zurich, Switzerland

ETH Zurich



LCSR Johns Hopkins