Competition

Truth-based CT reconstruction challenge

Organized by challenge-organizer - Current server time: June 6, 2023, 8:46 a.m. UTC

First phase

Data Available and Begin Submissions to Dropbox
March 1, 2022, midnight UTC

End

Competition Ends
Oct. 20, 2022, midnight UTC

Truth-based CT (TrueCT) reconstruction challenge

In partnership with the American Association of Physicists in Medicine (AAPM), the Center for Virtual Imaging Trials (CVIT) is offering the TrueCT reconstruction challenge. The Center will provide participants with simulated sinogram data of various anatomies and disease types for them to reconstruct into high quality CT slices using any type of algebraic, statistical, iterative, and ML-based algorithms as trained on clinical data. Reconstructions will be quantitatively evaluated versus the known truth inherent in the Center’s simulated data.

Overview

Increased use of “approximating” reconstruction methods in CT, including iterative reconstruction (IR) and deep learning (DL) methods, creates uncertainty in how much of the native information might be distorted in the reconstruction process. If one had access to ground truth, i.e., the true underlying anatomy and physiology, the precise limitation of the reconstruction process could be objectively quantified. However, such information is not available in patient images. Ground truth is known in physical phantoms, but these phantoms have limited utility for this purpose as they do not model patient complex anatomy, motion, abnormalities, or variability, factors that are known to impact the performance of non-linear, scene-dependent reconstruction methods. Virtual imaging toolsets consisting of realistic computational phantoms and accurate imaging simulation techniques can provide sets of imaging data with a digitally-defined ground truth with which CT reconstruction methods can be objectively and quantitatively evaluated and compared. This challenge will take advantage of the resources of the CVIT to create a dataset of realistic CT images of virtual patients with known ground truth to provide an objective evaluation of CT reconstruction methods.

The Challenge

Challenge participants will be provided with 200 sets of simulated sinogram data generated from computational anthropomorphic phantoms, each unique in its anatomy, sampling the ranges of weight, age, and sex of adult patients. Each phantom includes a particular pathology or disease: 67 COPD, 67 lung nodule, and 66 abdominal cases. The data includes a mixture of pathologies of undisclosed details. All cases are virtually imaged with a representative CT system with ranges of radiation dose level to represent clinical variability in techniques. Each case includes a meta-description file for data format and any other pertinent case information. Challenge participants may use any type of algebraic, statistical, iterative, and ML-based algorithms as trained on clinical data, to reconstruct the sinograms into high quality CT slices. Reconstructions will be evaluated by the Center through comparison to the computational known truth. The nature of this challenge precludes provision of a training dataset.

Participation

Upon registration, an e-mail will be sent to Challenge participants including:

  • A link to download the sinogram data with metadata
  • A unique link to upload results
  • Detailed instructions for how to submit results, including required formats and file organization.

Results

Results of the TrueCT challenge will be presented at the AAPM Grand Challenges Symposium at the 2022 AAPM Annual Meeting. An individual from each of the two top-performing teams will receive a waiver of the meeting registration fee to present their methods during this session. Top performing participants will also be offered the opportunity for their algorithm to be integrated with Center for Virtual Imaging Trials resources with licensing option.

 

Publication

We will seek the publication of challenge data and results in the Medical Physics journal. In addition, we encourage the participants to prepare manuscripts focusing on their methodologies.

 

Important Dates

  • Feb 24, 2022: Grand challenge website launch
  • March 1, 2022: Sinogram data made available
  • May 7, 2022: Last day to submit sample results for format validation
  • May 31st, 2022 May 17, 2022: Final submission of results (11:59PM EDT)
  • June 10, 2022: Scores posted and top two teams invited to present at challenge symposium
  • July 10-14, 2022: Grand Challenge Symposium, AAPM 2022 Annual Meeting
  • August 2022: Teams are invited to participate in the challenge publication and datasets made public

Organizers

  • Ehsan Samei
  • Ehsan Abadi
  • Paul Segars
  • Joseph Lo
  • Samuel Armato and the AAPM Working Group on Grand Challenges

Contact

For further information, please contact cvit-challenge@duke.edu.

Evaluation

For all the analysis, the reconstructed data will be registered to the ground truth. Ground truth will be established based on the mono-energetic representation of the phantoms. Data will be evaluated based on the similarity to the ground truth of the phantoms. For ranking purposes, we will only use task-generic (e.g., structural similarity index, root mean squared error) metrics. In addition to task-generic metrics, we will also evaluate the images based on task-specific (e.g., detectability index, radi-omics, density quantification) metrics, per case, per pathology type, and across the entire dataset. We will provide all the metrics to the participants and devise a collective score and report that in the coming publication. Detailed information on the acquired data and evaluation metrics will be shared along with the projection data.

 

Terms and Conditions: 

Once a team submits their images for evaluation, they will not be permitted to withdraw from the challenge, and their data and anonymous performance will be included in the aggregated results for publication.

Data Available and Begin Submissions to Dropbox

Start: March 1, 2022, midnight

Description: Go to the Get Data tab and upload your data via sftp as per the instructions, rather than use this submission section.

Competition Ends

Oct. 20, 2022, midnight

You must be logged in to participate in competitions.

Sign In