Commit Graph
1 Commits
Author SHA1 Message Date
cc232e3e35 Increase NCCL timeout to 3 hours (#12345)
* Increase NCCL timeout to 3 hours

When training on a large dataset using DDP, the scanning process will be very long, and it will raise NCCL timeout error. Change the default timeout 30min to 3 hours, same as ultralytics yolov8 (https://github.com/ultralytics/ultralytics/pull/3343)

Signed-off-by: Troy <wudashuo@vip.qq.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Troy <wudashuo@vip.qq.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2023-11-23 13:12:05 +01:00