> How is communication budget considered in this challenge?

Since limited communication budget is an important part of FL, I'm wondering how it is reflected in this challenge. Is it simulated by limiting the total runtime to 8h (number I found on the website)? If yes, is there a way to find out how much of the runtime is used for communication and, for example, how this depends on the model size?

Posted by: mzenk @ June 17, 2022, 7:38 a.m.

In this year's instance of the challenge, we only plan to evaluate the final accuracy of the trained models. Any FL training & and inference workflow that can complete within the 8-hour time limit will qualify for the final ranking. We will ask participants can record information such as the number of communications and message sizes for their final algorithms to study the impact of such design choices.

Related to this topic, we recently tuned the num_workers used in the PyTorch dataloaders to optimize the run-time within the challenge system (see https://github.com/Project-MONAI/tutorials/pull/756).
Please use this number of workers (4) to achieve the best runtime performance within the challenge system.

Here are the tuning results for reference: https://docs.google.com/spreadsheets/d/10y9pNYIpyVnn9MgIkwjyjZ8DzP33KdDWNyOLZX18Fq4/edit?usp=sharing

Posted by: hroth @ June 17, 2022, 6:41 p.m.

FYI, we should also be able to collect the log information that will tell us the communication times for each submission.

Posted by: hroth @ June 17, 2022, 6:43 p.m.

Alright, thanks for the additional information. Will participants be able to inspect these logs as well?

Posted by: mzenk @ June 20, 2022, 9:59 a.m.
Post in this thread