> tb_events empty after running run_all_fl.sh

I went through the reference_implementation and ran ./run_all_fl.sh as instructed. The process seems to have completed normally; however, when I tried visualizing on the tensorboard I got nothing. I checked the folder and it seems result_server/run_1/tb_events is empty. Is there something I need to adjust for the tb_events to be stored?

Posted by: Linardos @ June 26, 2022, 11:28 a.m.

The default settings should generate the tb event files on the server. The expected folder structure of the result is for example the one below.
Please make sure you are running the latest version from the repo https://github.com/Project-MONAI/tutorials/tree/main/federated_learning/breast_density_challenge.
If the issue persists, please file a ticket in the repo and attach your log files.

tree result_server
result_server
├── log.txt
├── predictions.json
├── run_1
│   ├── app_server
│   │   ├── best_FL_global_model.pt
│   │   ├── config
│   │   │   ├── config_fed_client.json
│   │   │   └── config_fed_server.json
│   │   └── FL_global_model.pt
│   ├── cross_site_val
│   │   ├── cross_val_results.json
│   │   ├── model_shareables
│   │   └── result_shareables
│   │   ├── site-1_SRV_best_FL_global_model.pt
│   │   ├── site-1_SRV_FL_global_model.pt
│   │   ├── site-2_SRV_best_FL_global_model.pt
│   │   ├── site-2_SRV_FL_global_model.pt
│   │   ├── site-3_SRV_best_FL_global_model.pt
│   │   └── site-3_SRV_FL_global_model.pt
│   ├── fl_app.txt
│   └── tb_events
│   ├── site-1
│   │   └── events.out.tfevents.1656347708.hroth-X299-A.101.2
│   ├── site-2
│   │   └── events.out.tfevents.1656347708.hroth-X299-A.101.1
│   └── site-3
│   └── events.out.tfevents.1656347708.hroth-X299-A.101.0
├── site-1_log.txt
├── site-2_log.txt
└── site-3_log.txt

Posted by: hroth @ June 27, 2022, 4:44 p.m.

A related issue has been created here https://github.com/Project-MONAI/tutorials/issues/764

Posted by: hroth @ July 5, 2022, 5:33 p.m.
Post in this thread