CLUST 2015

MICCAI 2015 Challenge on Liver Ultrasound Tracking
Munich, October 9, 2015 (Full Day)

MICCAI 2015 CLUST is now closed! In this page you can find a summary of the MICCAI workshop, including the final program and the proceedings.

However, we have an open challenge based on the CLUST 2015 data. Results submitted to this challenge will be included in the open challenge leaderboard.

---

Rules

Data usage

Participants agree that the data provided to them shall be used solely for the purpose of participating in the CLUST challenge and shall not be used, disclosed or distributed to any other person under any circumstance.

The tracking performance will be evaluated by the organizers after submission of the tracking results. Only this evaluation provides the official CLUST tracking performance and shall be reported in any publication to allow comparison.

When publishing the results based on the challenge data, participants shall cite the challenge and the references listed at the bottom of this page.

Method's parameters

To avoid unrealistic adjustment of the method's parameters per sequence, parameters should either be automatically determined or should be fixed for at least all sequences per ultrasound scanner type and task. It should be clear from the description of the method that this rule was followed by either specifying the automatic procedure or by stating the values of the fixed parameters.

Submission

Participants shall submit results and a description of the method, following the instructions listed in Submission. The description will serve the organizers to identify and discriminate between methods, and to verify that the rules regarding method's parameters were fulfilled. The description will only be released as informal submission (see below) after explicit request from the participant. The evaluation of the submitted results will be done by the challenge organizers.

By submitting their results, participants agree that these will be included on the online leaderboard webpage. There are three result categories:

• Anonymous informal submission without description, to support anonymous manuscript submission for at most 12 months
• Non-anonymous informal submission with description, to support method descriptions not intended for publication
• Official submission, linked to an already published method or to an informal submission, being now accepted for publication

Anonymous submissions, which have not progressed to the other two categories within 12 months after submission, will be removed from the leaderboard.

Results should only be compared to official submissions.

Participants shall notify the challenge organizers about any publication which is (partly) based on CLUST data.

References

De Luca, V., Benz, T., Kondo, S., König, L., Lübke, D., Rothlübbers, S., Somphone, O., Allaire, S., Lediju Bell, M.A., Chung, D.Y.F., Cifor, A., Grozea, C., Günther, M., Jenne, J., Kipshagen, T., Kowarschik, M., Navab, N., Rühaak, J., Schwaab, J., Tanner, C.: The 2014 liver ultrasound tracking benchmark. Physics in Medicine and Biology 60(14), 5571 (2015)
De Luca, V., Tschannen, M., Szekely, G., Tanner, C.: A Learning-Based Approach for Fast and Robust Vessel Tracking in Long Ultrasound Sequences. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2013, Lecture Notes in Computer Science, vol. 8149, pp. 518–525 (2013)
Preiswerk, F., De Luca, V., Arnold, P., Celicanin, Z., Petrusca, L., Tanner, C., Bieri, O., Salomir, R., Cattin, P. C.: Model-guided respiratory organ motion prediction of the liver from 2D ultrasound. Medical Image Analysis 18(5), 740-751 (2014)
Lediju, M.A., Byram, B.C., Harris, E.J., Evans, P.M, Bamber, J.C.: 3D Liver tracking using a matrix array: Implications for ultrasonic guidance of IMRT. In: IEEE Ultrasonics Symposium - IUS 2010. pp. 1628–1631 (Oct 2010)
Lediju Bell, M.A., Byram, B.C., Harris, E.J., Evans, P.M., Bamber, J.C.: In vivo liver tracking with a high volume rate 4D ultrasound scanner and a 2D matrix array probe. Physics in Medicine and Biology 57(5), 1359 (2012)
Vijayan, S., Klein, S., Hofstad, E., Lindseth, F., Ystgaard, B., Lango, T.: Validation of a non-rigid registration method for motion compensation in 4D ultrasound of the liver. In: IEEE 10th International Symposium on Biomedical Imaging - ISBI 2013. pp. 792–795 (April 2013)
Banerjee, J., Klink, C., Peters, E.D., Niessen, W.J., Moelker, A., van Walsum, T.: 4D Liver Ultrasound Registration. In: Biomedical Image Registration, Lecture Notes in Computer Science, vol. 8545, pp 194-202 (2014).

---

Data

This challenge contains of a total of 86 independent datasets

• 64 2D+t sequences
• 22 4D sequences

The data are anonymized and in the format of sequences of 2D images (.png) or 3D images (.mha).

The data are split into a training (40% of all sequences) and a test set (60%). Annotations are provided for the training set, to allow for some tuning of the tracking algorithm. For the test set, the annotations of the first images are provided. These need to be tracked over time.

The tracking performance of the test set will be evaluated by the organizers and summary results will be provided to the participants and published in Results, as informal or official submission, according to the Submission rules.

Part of the test data (20% of all data) will be distributed on October 9, 2015, for on-site processing during the event before noon.

For details on the submission formats, check the Submission page.

Download

The training and test data will be available from April 9, 2015 until July 5, 2015 here.

A username and password are requested to access the data. Please contact us to receive the login details.

The test data for the on-site challenge is available here ONLY during the CLUST 2015 event, on October 9, 2015, from 08:30 to 11:30, Central European Time. Login credentials will be provided at the event.

---

Submission guidelines

Participants are invited to submit the following files:

1. Tracking results for the annotations, which were provided for the first frame.
2. Description of the tracking method.

1. Tracking results

It is possible to submit results for all data or for at least one of the following subsets:

• all 2D point-tracking test data (CIL, ETH, ICR, MED)
• all 3D point-tracking test data (EMC, ICR, SMT)

Partial results for these subsets will not be assessed.

Data format

Please submit your tracking results in a zip archive to be named as the provided username. Additionally, name the tracking results according to the corresponding sequences and as done for the annotation of the first image. Please submit the tracking results in the same format as provided for the testing sequences of each test set subgroup. In details:

Point tracking results:

• File name SequenceName_PointNumber, file format .txt, e.g. ETH-01-1_1.txt
• File content: 3 (4) columns for 2D (3D) sequences indicating, in each row, frame number, x-coordinate, y-coordinate(, z-coordinate) in pixels (voxels) of the tracked point. Please include results for all frames

Results evaluation

The organizers will quantitatively evaluate the submitted tracking results with respect to ground truth annotations. For the evaluation we will consider the following measurement:

• Euclidean distance between the tracked points and manual annotation

Error statistics will be summarized by mean, standard deviation and 95%.

We will provide the authors with the outcome of the aforementioned evaluation. After approval of the authors, we will include the quantitative evaluation of the tracking results in the MICCAI 2015 CLUST leaderboard.

2. Paper format

Participants shall provide a 4-8 page paper describing their method. Papers should be formatted in Lecture Notes in Computer Science style. The file format for submissions is Adobe Portable Document Format (PDF). Other formats will not be accepted. For further info, please refer to the MICCAI conference submission format.

Authors of accepted papers shall share their publication, as a reference to the results published in the official Results.

Upload your submission

It is possible to upload your submission here.

---

Results

The results listed on this webpage are based on the first test set (40% of all data) of the CLUST 2015 challenge, as in the referenced publications. The results of all test data (60%), including the MICCAI 2015 CLUST on-site challenge, at listed here.

Results of 2D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.

Participant	Tracking error [mm]			Submission date	Remark
	Mean	Standard deviation	95th percentile
Hallack A., et al.	0.91	1.66	2.20	Jul 2015
Ozkan E., et al.	1.04	1.48	2.26	Nov 2016
Makhinya M. and Goksel O.	1.09	1.75	2.42	Jul 2015
Kondo S.	1.09	1.35	3.07	Jul 2015
Chen et al.	1.63	1.99	4.36	Jun 2016	+300ms latency & temporal prediction
Chen et al.	2.20	2.04	5.10	Jun 2016	+300ms latency (no temporal prediction)
Nouri D. and Rothberg, A.	2.83	4.86	13.13	Jul 2015

Results of 3D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.

Participant	Tracking error [mm]			Submission date
	Mean	Standard deviation	95th percentile
Royer L., et al.	1.56	0.69	2.91	Jul 2015
Banerjee J., et al.	1.47	0.58	2.38	Jul 2015

^* Informal submission.

---

Important dates

All deadlines are 23:59 Pacific Standard Time and in the year 2015.

	Apr 13	Release of training data
	May 26	Release of test data
	May 26	Call for short papers
	Jul 6	Tracking results submission deadline
	Jul 8	Tracking performance notification
	Jul 13	Paper submission deadline
	Jul 28	Notification of acceptance
	Aug 1	Workshop registration due
	Aug 5	Submission deadline for accepted papers
	Aug	Final program
	Oct 9	Workshop date

---

Final Program

Friday, October 9, 2015 (9.00 am - 5.30 pm)

Download the detailed schedule here.

9.15 - 9.30 am	Welcome and Introductions for on-site challenge V. De Luca, E. Harris, M. Lediju Bell, C. Tanner
9.30 - 10.30 am	On-site Challenge Chair: E. Harris, Institute of Cancer Research, London, UK
10.30 - 11.00 am	Coffee Break
11.00 am - 12.30 pm	On-site Challenge (continue) Chair: E. Harris, Institute of Cancer Research, London, UK
12.30 - 1.30 pm	Lunch Break
2.00 - 2.10 pm	Introduction C. Tanner, ETH Zurich, Switzerland
2.10 - 3.30 pm	Oral Session 1 - 2D Tracking Chair: M. A. Lediju Bell, Johns Hopkins University, Baltimore, USA
	Liver Ultrasound Tracking using a Learned Distance Metric. D. Nouri, A. Rothberg, 4Catalyzer, Inc., Guilford CT 06437, USA
	Liver Ultrasound Tracking Using Kernelized Correlation Filter With Adaptive Window Size Selection. S. Kondo, Konica Minolta Inc., Osaka, Japan
	Motion Tracking in 2D Ultrasound Using Vessel Models and Robust Optic-Flow. M. Makhinya, O. Goksel, Computer Vision Lab, ETH Zurich, Switzerland
	Robust Liver Ultrasound Tracking using Dense Distinctive Image Features. A. Hallack1, B.W. Papiez1, A. Cifor2, M.J. Gooding2, J.A. Schnabel3, 1 Institute of Biomedical Engineering (Department of Engineering Science), University of Oxford, UK; 2 Mirada Medical, Oxford, UK; 3 Division of Imaging Sciences & Biomedical Engineering, King's College London, UK
3.30 - 4.00 pm	Coffee Break
4.00 - 4.40 pm	Oral Session 2 - 3D Tracking Chair: C. Tanner, ETH Zurich, Switzerland
	Tracking of Non-rigid Targets in 3D US Images: Results on CLUST 2015. L. Royer1,2,3, G. Dardenne1, A. Le Bras1,4, M. Marchal1,3, A. Krupa1,2, 1 Institut de Recherche Technologique b-com, Rennes, France; 2 Inria Rennes - Bretagne Atlantique, France; 3 INSA de Rennes, France; 4 CHU de Rennes, France
	A combined tracking and registration approach for tracking anatomical landmarks in 4D ultrasound of the liver. J. Banerjee1,2, C. Klink1, E. Vast1,2, W.J. Niessen1,2, A. Moelker1, T. van Walsum1,2, 1 Dept. of Radiology, Erasmus MC, Rotterdam, The Netherlands; 2 Dept. of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands
4.40 - 5.00 pm	Overall Results and Conclusions V. De Luca, ETH Zurich, Switzerland
5.00 - 5.25 pm	Discussion and Outlook Chair: E. Harris, Institute of Cancer Research, London, UK

---

Venue

The event was held in Forum 9, in the Holiday Inn Hotel, Munich.

---

Proceedings

The proceeding of MICCAI CLUST 2015 are available for download here.

A summary of the results is published in the Results listed above. In the proceedings, authors reported tracking results only for the initial test data (40% of all data).

The results are obtained from all test data (60% of all data), including the on-site challenge results (20%) are not included in the proceeding and are listed here.

---

Organizers

Valeria De Luca, ETH Zurich, Switzerland vdeluca@vision.ee.ethz.ch

Emma Harris, Institute of Cancer Research, Sutton, UK emma.harris@icr.ac.uk

Muyinatu A. Lediju Bell, Johns Hopkins University, USA muyinatu.ledijubell@jhu.edu

Christine Tanner, ETH Zurich, Switzerland tannerch@vision.ee.ethz.ch