CLUST 2014

MICCAI Challenge on Liver Ultrasound Tracking
Boston, September 14, 2014 (PM)

MICCAI 2014 CLUST is now closed! In this page you can find a summary of the MICCAI workshop, including the final program and the proceedings.

However, we have an open challenge based on the CLUST 2014 data. Results submitted to this challenge will be included in the leaderboard shown below.

Rules

Data usage

Participants agree that the data provided to them shall be used solely for the purpose of participating in the challenge and shall not be used, disclosed or distributed to any other person under any circumstance.

When publishing the results based on the challenge data, participants shall cite the challenge and the references listed at the bottom of this page.

Submission

Participants shall submit results and a description of the method, following the instructions listed in the Submission guidelines. The description will serve the organizers to identify and discriminate between methods, but will only be released as informal submission (see below) after explicit request from the participant. The evaluation of the submitted results will be done by the challenge organizers.

By submitting their results, participants agree that these will be included on the online leaderboard. There are two result categories:

Informal submission, which is valid until the related paper has been published
Official submission, linked to an already published method or to an informal submission, being now accepted for publication

Results should only be compared to official submissions.

Participants shall notify the challenge organizers about any publication which is (partly) based on CLUST data.

References

De Luca, V., Benz, T., Kondo, S., König, L., Lübke, D., Rothlübbers, S., Somphone, O., Allaire, S., Lediju Bell, M.A., Chung, D.Y.F., Cifor, A., Grozea, C., Günther, M., Jenne, J., Kipshagen, T., Kowarschik, M., Navab, N., Rühaak, J., Schwaab, J., Tanner, C.: The 2014 liver ultrasound tracking benchmark. Physics in Medicine and Biology 60(14), 5571 (2015)

Cifor, A., Risser, L., Chung, D., Anderson, E., Schnabel, J.: Hybrid Feature-Based Diffeomorphic Registration for Tumor Tracking in 2-D Liver Ultrasound Images. IEEE Transaction on Medical Imaging, 32(9), 1647–1656 (Sept 2013)

De Luca, V., Tschannen, M., Szekely, G., Tanner, C.: A Learning-Based Approach for Fast and Robust Vessel Tracking in Long Ultrasound Sequences. In: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2013, Lecture Notes in Computer Science, vol. 8149, pp. 518–525 (2013)

Lediju, M.A., Byram, B.C., Harris, E.J., Evans, P.M, Bamber, J.C.: 3D Liver tracking using a matrix array: Implications for ultrasonic guidance of IMRT. In: IEEE Ultrasonics Symposium - IUS 2010. pp. 1628–1631 (Oct 2010)

Lediju Bell, M.A., Byram, B.C., Harris, E.J., Evans, P.M., Bamber, J.C.: In vivo liver tracking with a high volume rate 4D ultrasound scanner and a 2D matrix array probe. Physics in Medicine and Biology 57(5), 1359 (2012)

Vijayan, S., Klein, S., Hofstad, E., Lindseth, F., Ystgaard, B., Lango, T.: Validation of a non-rigid registration method for motion compensation in 4D ultrasound of the liver. In: IEEE 10th International Symposium on Biomedical Imaging - ISBI 2013. pp. 792–795 (April 2013)

Banerjee, J., Klink, C., Peters, E.D., Niessen, W.J., Moelker, A., van Wal- sum, T.: 4D Liver Ultrasound Registration. In: Biomedical Image Registration, Lecture Notes in Computer Science, vol. 8545, pp 194-202 (2014).

Data

This challenge contains of a total of 54 independent datasets

38 2D+t sequences
16 4D sequences

The data are anonymized and in the format of sequences of 2D images (.png, .jpeg, NFTI) or 3D images (.mha).

The data are split into a training (10% of the sequences) and a test set (90%). Annotations are provided for the training set, to allow for some tuning of the tracking algorithm. For the test set, the annotations of the first images are provided. These need to be tracked over time.

The tracking performance of the test set will be evaluated by the organizers and summary results will be provided to the participants and published in Results, as informal or official submission, according to the Submission rules.

For details on the submission formats, check the Submission section in this page.

Download

The training and test data are available here.

A username and password are requested to access the data. Please contact us to receive the login details.

Submission guidelines

Participants are invited to submit the following files:

Tracking results for the annotations, which were provided for the first frame.
Description of the tracking method.

1. Tracking results

It is possible to submit results for all data or for at least one of the following subsets:

all 2D point-tracking test data (ETH, MED)
all 2D segmentation test data (OX)
all 3D point-tracking test data (ICR, SMT, EMC)

Partial results for these subset will not be assessed.

Data format

Please submit your tracking results in a zip archive to be named as the provided username. Additionally, name the tracking results according to the corresponding sequences and as done for the annotation of the first image. Please submit the tracking results in the same format as provided for the testing sequences of each test set subgroup. In details:

Point tracking results:
- File name SequenceName_PointNumber, file format .txt, e.g. ETH-01_1.txt
- File content: 3 (4) columns for 2D (3D) sequences indicating, in each row, frame number, x-coordinate, y-coordinate(, z-coordinate) in pixels (voxels) of the tracked point. Please include results for all frames
Segmentation results:
- Folder name SequenceName, e.g OX-01
- Folder content: sequence of (binary) mask images with the resulting segmentation. Name each mask image with the correspondent frame number and file format, e.g. 010.nii for frame number 10 in the sequence
Algorithm run-time. Report in a separate text file:
- the average run-time for all cases per data subset
- the implementation details (hardware/software)

Please include in the archive a text file listing the submitted files and a brief description. Do NOT submit the original images or other tracking results, e.g. deformation fields or tracking results of features which were not provided by the organizers.

Results evaluation

The organizers will quantitatively evaluate the submitted tracking results with respect to ground truth annotations. For the evaluation we will consider the following measurements:

Euclidean distance between the tracked points and manual annotation
Overlap ratio (Dice coefficient) between segmentation results and manual segmentation

Error statistics will be summarized by mean, standard deviation and 95%.

We will provide the authors with the outcome of the aforementioned evaluation. After approval of the authors, we will include the quantitative evaluation of the tracking results in Results.

The publication of the results can be of two types:

Informal, for not yet accepted papers
Official, which will have full authorship information and description of the method from a paper, which has been accepted for publication

2. Paper format

Participants shall provide a short paper describing their method. Papers should be formatted in Lecture Notes in Computer Science style. The file format for submissions is Adobe Portable Document Format (PDF). Other formats will not be accepted.

Authors of accepted papers shall share their publication, as a reference to the results published in the official Results.

Upload your submission

It is possible to upload your submission here.

Results

The results listed on this webpage include all test data (90%) of CLUST14 challenge, while the referenced publications for Sept 2014 state the results for the first test set (70%).

Results of 2D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.

Participant	Tracking error [mm]			Submission date
Participant	Mean	Standard deviation	95th percentile	Submission date
König L., et al.	1.51	1.88	4.06	Sept 2014
Rothlübbers S., et al.	1.52	1.38	4.08	Sept 2014
Kondo S.	1.83	3.16	4.82	Sept 2014
Benz T., et al.	1.84	2.42	5.34	Sept 2014
Lübke D., and Grozea C.	1.91	2.47	5.32	Sept 2014
Somphone O., et al.	2.00	2.87	5.59	Sept 2014
O'Shea T., et al.	2.61	3.78	7.98	Sept 2014

Results of 2D tumor-segmentation tracking. The results are summarized by the range of the mean DICE coefficient, computed for each segmentation and sequence.

Participant	Mean Dice [%]		Submission date
Participant	Min	Max	Submission date
Somphone O., et al.	76.7	92.3	Sept 2014

Results of 3D point-landmark tracking. The results are in millimetres and ranked according to increasing mean tracking error.

Participant	Tracking error [mm]			Submission date
Participant	Mean	Standard deviation	95th percentile	Submission date
Royer L., et al.	1.62	2.19	4.81	Jan 2015
Somphone O., et al.	2.55	2.46	7.98	Sept 2014
Rothlübbers S., et al.	2.80	2.96	7.94	Sept 2014
Lübke D., and Grozea C.	4.63	4.03	12.44	Sept 2014

Important dates

All deadlines are 23:59 Pacific Standard Time.

	Mar 1	Workshop acceptance notification
	Mar 25	Release of 80% of the data
	Mar 30	Call for papers
	Jun 18	Tracking results submission deadline
	Jun 19	Tracking performance notification
	Jun 24	Paper submission deadline
	Jul 5	Acceptance/rejection notification
	Jul 22	Submission deadline for accepted papers
	Jul 22	Workshop registration due
	Aug 1	Final program
	Sep 8	Release of final 20% of the data
	Sep 14 - PM	Workshop date

Final Program

Download the detailed schedule here.

1.15 - 1.40 pm	Welcome and Introduction C. Tanner, V. De Luca, ETH Zurich, Switzerland
1.40 - 3.00 pm	Oral Session 1 - 2D Tracking Chair: A. Cifor, University of Oxford, UK
	Liver Feature Motion Estimation in Long High Frame Rate 2D Ultrasound Sequences. T. O'Shea, J. Bamber, E. Harris, ICR, London, UK
	Liver Ultrasound Tracking Using Long-term and Short-term Template Matching. S. Kondo, Konica Minolta Inc., Osaka, Japan
	Kernel-based Tracking in Ultrasound Sequences of Liver. T. Benz¹, M. Kowarschik^1,2, N. Navab¹, ¹ CAMP TU Munich, Germany; ² Angiography & Interventional X-Ray Systems, Siemens AG, Healthcare Sector, Forchheim, Germany
	A Non-Linear Image Registration Scheme for Real-Time Liver Ultrasound Tracking using Normalized Gradient Fields. L. König, T. Kipshagen, J. Rühaak, Fraunhofer MEVIS, Lübeck, Germany
3.00 - 3.30 pm	Coffee Break
3.30 - 4.30 pm	Oral Session 2 - 2D and 3D Tracking Chair: M. A. Lediju Bell, Johns Hopkins University, Baltimore, USA
	High Performance Online Motion Tracking in Abdominal Ultrasound Imaging. D. Lübke, C. Grozea, Fraunhofer MEVIS, Bremen, Germany; Fraunhofer FOKUS, Berlin, Germany
	MICCAI CLUST 2014 - Bayesian Real-Time Liver Feature Ultrasound Tracking. S. Rothlübbers, J. Schwaab, J. Jenne, M. Günther, Fraunhofer MEVIS, Bremen, Germany; Mediri GmbH, Heidelberg, Germany
	Live Feature Tracking in Ultrasound Liver Sequences with Sparse Demons. O. Somphone, S. Allaire, B. Mory, C. Dufour, Medisys Lab Philips Research, Suresnes, France
4.30 - 5.00 pm	Overall Results and Conclusions V. De Luca, ETH Zurich, Switzerland

Venue

The workshop was held in the Stata Center 36, in room 428. A map of the MIT campus can be found here.

Further information can be found on the MICCAI Workshop page.

Proceedings

The proceeding of MICCAI CLUST 2014 are available for download here.

A summary of the results is published in Results, see Submission date: Sept 2014. These results are obtained from all test data. In the proceedings, authors reported tracking results only for the initial test data (70% of all data).

Organizers

Valeria De Luca, ETH Zurich, Switzerland vdeluca@vision.ee.ethz.ch

Amalia Cifor, University of Oxford, UK amalia.cifor@eng.ox.ac.uk

Muyinatu A. Lediju Bell, Johns Hopkins University, USA muyinatu.ledijubell@jhu.edu

Christine Tanner, ETH Zurich, Switzerland tannerch@vision.ee.ethz.ch