Participants agree that the data provided to them shall be used solely for the purpose of participating in the CLUST challenge and shall not be used, disclosed or distributed to any other person under any circumstance.
The tracking performance will be evaluated by the organizers after submission of the tracking results. Only this evaluation provides the official CLUST tracking performance and shall be reported in any publication to allow comparison.
When publishing the results based on the challenge data, participants shall cite the challenge and the references listed at the bottom of this page.
To avoid unrealistic adjustment of the method's parameters per sequence, parameters should either be automatically determined or should be fixed for at least all sequences per ultrasound scanner type and task. It should be clear from the description of the method that this rule was followed by either specifying the automatic procedure or by stating the values of the fixed parameters.
Participants shall submit results and a description of the method, following the instructions listed in Submission. The description will serve the organizers to identify and discriminate between methods, and to verify that the rules regarding method's parameters were fulfilled. The description will only be released as informal submission (see below) after explicit request from the participant. The evaluation of the submitted results will be done by the challenge organizers.
By submitting their results, participants agree that these will be included on the online leaderboard webpage. There are three result categories:
Results should only be compared to official submissions.
Participants shall notify the challenge organizers about any publication which is (partly) based on CLUST data.
This challenge contains of a total of 86 independent datasets
The data are split into a training (40% of all sequences) and a test set (60%). Annotations are provided for the training set, to allow for some tuning of the tracking algorithm. For the test set, the annotations of the first images are provided. These need to be tracked over time.
The tracking performance of the test set will be evaluated by the organizers and summary results will be provided to the participants and published in Results, as informal or official submission, according to the Submission rules.
Part of the test data (20% of all data) will be distributed on October 9, 2015, for on-site processing during the event before noon.
For details on the submission formats, check the Submission page.
The training and test data will be available from April 9, 2015 until July 5, 2015 here.
A username and password are requested to access the data. Please contact us to receive the login details.
The test data for the on-site challenge is available here ONLY during the CLUST 2015 event, on October 9, 2015, from 08:30 to 11:30, Central European Time. Login credentials will be provided at the event.
Participants are invited to submit the following files:
It is possible to submit results for all data or for at least one of the following subsets:
Please submit your tracking results in a zip archive to be named as the provided username. Additionally, name the tracking results according to the corresponding sequences and as done for the annotation of the first image. Please submit the tracking results in the same format as provided for the testing sequences of each test set subgroup. In details:
Point tracking results:
The organizers will quantitatively evaluate the submitted tracking results with respect to ground truth annotations. For the evaluation we will consider the following measurement:
We will provide the authors with the outcome of the aforementioned evaluation. After approval of the authors, we will include the quantitative evaluation of the tracking results in the MICCAI 2015 CLUST leaderboard.
Participants shall provide a 4-8 page paper describing their method. Papers should be formatted in Lecture Notes in Computer Science style. The file format for submissions is Adobe Portable Document Format (PDF). Other formats will not be accepted. For further info, please refer to the MICCAI conference submission format.
Authors of accepted papers shall share their publication, as a reference to the results published in the official Results.
It is possible to upload your submission here.
The results listed on this webpage are based on the first test set (40% of all data) of the CLUST 2015 challenge, as in the referenced publications. The results of all test data (60%), including the MICCAI 2015 CLUST on-site challenge, at listed here.
Results of 2D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.
|Mean||Standard deviation||95th percentile|
|Hallack A., et al.||0.91||1.66||2.20||Jul 2015|
|Ozkan E., et al.||1.04||1.48||2.26||Nov 2016|
|Makhinya M. and Goksel O.||1.09||1.75||2.42||Jul 2015|
|Kondo S.||1.09||1.35||3.07||Jul 2015|
|Chen et al.||1.63||1.99||4.36||Jun 2016||+300ms latency & temporal prediction|
|Chen et al.||2.20||2.04||5.10||Jun 2016||+300ms latency (no temporal prediction)|
|Nouri D. and Rothberg, A.||2.83||4.86||13.13||Jul 2015|
Results of 3D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.
|Mean||Standard deviation||95th percentile|
|Royer L., et al.||1.56||0.69||2.91||Jul 2015|
|Banerjee J., et al.||1.47||0.58||2.38||Jul 2015|
* Informal submission.
All deadlines are 23:59 Pacific Standard Time and in the year 2015.
|Apr 13||Release of training data|
|May 26||Release of test data|
|May 26||Call for short papers|
|Jul 6||Tracking results submission deadline|
|Jul 8||Tracking performance notification|
|Jul 13||Paper submission deadline|
|Jul 28||Notification of acceptance|
|Aug 1||Workshop registration due|
|Aug 5||Submission deadline for accepted papers|
|Oct 9||Workshop date|
Friday, October 9, 2015 (9.00 am - 5.30 pm)
Download the detailed schedule here.
|9.15 - 9.30 am||Welcome and Introductions for on-site challenge V. De Luca, E. Harris, M. Lediju Bell, C. Tanner|
|9.30 - 10.30 am||On-site Challenge Chair: E. Harris, Institute of Cancer Research, London, UK|
|10.30 - 11.00 am||Coffee Break|
|11.00 am - 12.30 pm||On-site Challenge (continue) Chair: E. Harris, Institute of Cancer Research, London, UK|
|12.30 - 1.30 pm||Lunch Break|
|2.00 - 2.10 pm||Introduction C. Tanner, ETH Zurich, Switzerland|
|2.10 - 3.30 pm||Oral Session 1 - 2D Tracking Chair: M. A. Lediju Bell, Johns Hopkins University, Baltimore, USA|
|Liver Ultrasound Tracking using a Learned Distance Metric. D. Nouri, A. Rothberg, 4Catalyzer, Inc., Guilford CT 06437, USA|
|Liver Ultrasound Tracking Using Kernelized Correlation Filter With Adaptive Window Size Selection. S. Kondo, Konica Minolta Inc., Osaka, Japan|
|Motion Tracking in 2D Ultrasound Using Vessel Models and Robust Optic-Flow. M. Makhinya, O. Goksel, Computer Vision Lab, ETH Zurich, Switzerland|
|Robust Liver Ultrasound Tracking using Dense Distinctive Image Features. A. Hallack1, B.W. Papiez1, A. Cifor2, M.J. Gooding2, J.A. Schnabel3, 1 Institute of Biomedical Engineering (Department of Engineering Science), University of Oxford, UK; 2 Mirada Medical, Oxford, UK; 3 Division of Imaging Sciences & Biomedical Engineering, King's College London, UK|
|3.30 - 4.00 pm||Coffee Break|
|4.00 - 4.40 pm||Oral Session 2 - 3D Tracking Chair: C. Tanner, ETH Zurich, Switzerland|
|Tracking of Non-rigid Targets in 3D US Images: Results on CLUST 2015. L. Royer1,2,3, G. Dardenne1, A. Le Bras1,4, M. Marchal1,3, A. Krupa1,2, 1 Institut de Recherche Technologique b-com, Rennes, France; 2 Inria Rennes - Bretagne Atlantique, France; 3 INSA de Rennes, France; 4 CHU de Rennes, France|
|A combined tracking and registration approach for tracking anatomical landmarks in 4D ultrasound of the liver. J. Banerjee1,2, C. Klink1, E. Vast1,2, W.J. Niessen1,2, A. Moelker1, T. van Walsum1,2, 1 Dept. of Radiology, Erasmus MC, Rotterdam, The Netherlands; 2 Dept. of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands|
|4.40 - 5.00 pm||Overall Results and Conclusions V. De Luca, ETH Zurich, Switzerland|
|5.00 - 5.25 pm||Discussion and Outlook Chair: E. Harris, Institute of Cancer Research, London, UK|
The event was held in Forum 9, in the Holiday Inn Hotel, Munich.
The proceeding of MICCAI CLUST 2015 are available for download here.
A summary of the results is published in the Results listed above. In the proceedings, authors reported tracking results only for the initial test data (40% of all data).
The results are obtained from all test data (60% of all data), including the on-site challenge results (20%) are not included in the proceeding and are listed here.
Valeria De Luca, ETH Zurich, Switzerland firstname.lastname@example.org
Emma Harris, Institute of Cancer Research, Sutton, UK email@example.com
Muyinatu A. Lediju Bell, Johns Hopkins University, USA firstname.lastname@example.org
Christine Tanner, ETH Zurich, Switzerland email@example.com