> Details of distance metric

Thanks for putting together this challenge! Reading through the website, I wondered if there is an exact definition or reference implementation of the "per-image distance metric". Would it be possible to provide that? It could be useful for model development.

Posted by: mzenk @ June 17, 2022, 7:26 a.m.

The distance-based metric is only planned to be used in the final evaluation/ranking of the challenge (we might use several variations for statistical ranking using https://github.com/wiesenfa/challengeR.

During the validation phase, we will use several "global" metrics, which will be available on the leaderboard.

For example, these are the current "global" metrics based on the reference implementation:

Evaluating SRV_best_FL_global_model.pt on test1:

===site-1===
evaluating 3327 predictions: lin. kappa 0.541, quad. kappa 0.647, auc. 0.893

===site-2===
evaluating 1128 predictions: lin. kappa 0.454, quad. kappa 0.563, auc. 0.915

===site-3===
evaluating 5952 predictions: lin. kappa 0.626, quad. kappa 0.731, auc. 0.914

===overall===
evaluating 10407 predictions: lin. kappa 0.584, quad. kappa 0.692, auc. 0.901

Posted by: hroth @ June 17, 2022, 6:27 p.m.

Thanks for your answer, I understand that you don't want to publish per-image results for the distance-based metric on the test set on a live leaderboard. Since the evaluation metrics are usually defined before the actual challenge evaluation begins, would it be possible to provide the mathematical definition or an implementation for the training/validation set? This may also be helpful in experiments with public data that is not part of this challenge.

Posted by: mzenk @ June 20, 2022, 9:57 a.m.

Thank you again for your comment, we published the evaluation script as part of the reference repo here: https://github.com/Project-MONAI/tutorials/blob/main/federated_learning/breast_density_challenge/challenge_evaluate.py

The computation is very straightforward. We are planning to use two metrics, the linear and the quadratic distance:

# per-image distance metrics
dist = np.abs(np.squeeze(gt_labels) - np.squeeze(pred_labels))
lin_dist = -dist
quad_dist = -(dist ** 2)
avg_lin_dist = np.mean(lin_dist)
avg_quad_dist = np.mean(quad_dist)

Posted by: hroth @ June 29, 2022, 4:11 p.m.

Thanks for the clarification :)

Posted by: mzenk @ June 30, 2022, 10:50 a.m.
Post in this thread