Ankle/yolov5_birdproject
1
0 Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Ultralytics Refactor https://ultralytics.com/actions (#13553)

Browse Source 
* Refactor code for speed and clarity

* Auto-format by https://ultralytics.com/actions

* Update README.md

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

---------

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: UltralyticsAssistant <web@ultralytics.com>
This commit is contained in:
Paula DerrengerGlenn JocherUltralyticsAssistant
 authored and GitHub committed 2025-03-28 03:35:11 +01:00
1 parent 3fee72b5e8
commit e9ab205ef6
13 files changed
+484 -390
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

utils/loggers/comet/README.md
+122 -94
View File
@@ -1,104 +1,114 @@
<a href="https://www.ultralytics.com/"><img src="https://raw.githubusercontent.com/ultralytics/assets/main/logo/Ultralytics_Logotype_Original.svg" width="320" alt="Ultralytics logo"></a>
<img src="https://cdn.comet.ml/img/notebook_logo.png">
# YOLOv5 with Comet
# Using YOLOv5 with Comet
This guide will cover how to use YOLOv5 with [Comet](https://bit.ly/yolov5-readme-comet2)
Welcome to the guide on integrating [Ultralytics YOLOv5](https://github.com/ultralytics/yolov5) with [Comet](https://www.comet.com/site/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme)! Comet provides powerful tools for experiment tracking, model management, and visualization, enhancing your machine learning workflow. This document details how to leverage Comet to monitor training, log results, manage datasets, and optimize hyperparameters for your YOLOv5 models.
# About Comet
## 🧪 About Comet
Comet builds tools that help data scientists, engineers, and team leaders accelerate and optimize machine learning and deep learning models.
[Comet](https://www.comet.com/site/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme) builds tools that help data scientists, engineers, and team leaders accelerate and optimize machine learning and deep learning models.
Track and visualize model metrics in real time, save your hyperparameters, datasets, and model checkpoints, and visualize your model predictions with [Comet Custom Panels](https://www.comet.com/docs/v2/guides/comet-dashboard/code-panels/about-panels/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github)! Comet makes sure you never lose track of your work and makes it easy to share results and collaborate across teams of all sizes!
Track and visualize model metrics in real-time, save your hyperparameters, datasets, and model checkpoints, and visualize your model predictions with [Comet Custom Panels](https://www.comet.com/docs/v2/guides/comet-dashboard/code-panels/about-panels/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme)! Comet ensures you never lose track of your work and makes it easy to share results and collaborate across teams of all sizes.
# Getting Started
## 🚀 Getting Started
## Install Comet
Follow these steps to set up Comet for your YOLOv5 projects.
### Install Comet
Install the necessary [Python package](https://pypi.org/project/comet-ml/) using pip:
```shell
pip install comet_ml
```
## Configure Comet Credentials
### Configure Comet Credentials
There are two ways to configure Comet with YOLOv5.
You can configure Comet in two ways:
You can either set your credentials through environment variables
1. **Environment Variables:** Set your credentials directly in your environment.
**Environment Variables**
```shell
export COMET_API_KEY=<Your Comet API Key>
export COMET_PROJECT_NAME=<Your Comet Project Name> # Defaults to 'yolov5' if not set
```
```shell
export COMET_API_KEY=<Your Comet API Key>
export COMET_PROJECT_NAME=<Your Comet Project Name> # This will default to 'yolov5'
```
Find your API key in your [Comet Account Settings](https://www.comet.com/docs/v2/guides/getting-started/quickstart/#get-an-api-key?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme).
Or create a `.comet.config` file in your working directory and set your credentials there.
2. **Configuration File:** Create a `.comet.config` file in your working directory with the following content:
```ini
[comet]
api_key=<Your Comet API Key>
project_name=<Your Comet Project Name> # Defaults to 'yolov5' if not set
```
**Comet Configuration File**
### Run the Training Script
```
[comet]
api_key=<Your Comet API Key>
project_name=<Your Comet Project Name> # This will default to 'yolov5'
```
## Run the Training Script
Execute the YOLOv5 training script. Comet will automatically start logging your run.
```shell
# Train YOLOv5s on COCO128 for 5 epochs
python train.py --img 640 --batch 16 --epochs 5 --data coco128.yaml --weights yolov5s.pt
```
That's it! Comet will automatically log your hyperparameters, command line arguments, training and validation metrics. You can visualize and analyze your runs in the Comet UI
That's it! Comet automatically logs hyperparameters, command-line arguments, and training/validation metrics. Visualize and analyze your runs in the Comet UI. For more details on training, see the [YOLOv5 Train documentation](https://docs.ultralytics.com/yolov5/tutorials/train_custom_data/).
<img width="1920" alt="yolo-ui" src="https://user-images.githubusercontent.com/26833433/202851203-164e94e1-2238-46dd-91f8-de020e9d6b41.png">
<img width="1920" alt="Comet UI showing YOLOv5 training metrics" src="https://user-images.githubusercontent.com/26833433/202851203-164e94e1-2238-46dd-91f8-de020e9d6b41.png">
# Try out an Example!
## ✨ Try an Example!
Check out an example of a [completed run here](https://www.comet.com/examples/comet-example-yolov5/a0e29e0e9b984e4a822db2a62d0cb357?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step&utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github)
Explore a completed YOLOv5 training run tracked with Comet:
Or better yet, try it out yourself in this Colab Notebook
- **[View Example Run on Comet](https://www.comet.com/examples/comet-example-yolov5/a0e29e0e9b984e4a822db2a62d0cb357?experiment-tab=chart&showOutliers=true&smoothing=0&transformY=smoothing&xAxis=step&utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme)**
Run the example yourself using this [Google Colab](https://colab.research.google.com/) Notebook:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/comet-examples/blob/master/integrations/model-training/yolov5/notebooks/Comet_and_YOLOv5.ipynb)
# Log automatically
## 📊 Automatic Logging
By default, Comet will log the following items
Comet automatically logs the following information by default:
## Metrics
### Metrics
- Box Loss, Object Loss, Classification Loss for the training and validation data
- mAP_0.5, mAP_0.5:0.95 metrics for the validation data.
- Precision and Recall for the validation data
- **Losses:** Box Loss, Object Loss, Classification Loss (Training and Validation).
- **Performance:** [mAP@0.5](https://docs.ultralytics.com/guides/yolo-performance-metrics/), mAP@0.5:0.95 (Validation).
- **Precision and Recall:** Validation data metrics.
## Parameters
### Parameters
- Model Hyperparameters
- All parameters passed through the command line options
- **Model Hyperparameters:** Configuration used for the model.
- **Command Line Arguments:** All arguments passed via the CLI.
## Visualizations
### Visualizations
- Confusion Matrix of the model predictions on the validation data
- Plots for the PR and F1 curves across all classes
- Correlogram of the Class Labels
- **[Confusion Matrix](https://en.wikipedia.org/wiki/Confusion_matrix):** Model predictions on validation data.
- **Curves:** PR and F1 curves across all classes.
- **Label Correlogram:** Correlation visualization of class labels.
# Configure Comet Logging
## ⚙️ Advanced Configuration
Comet can be configured to log additional data either through command line flags passed to the training script or through environment variables.
Customize Comet's logging behavior using command-line flags or environment variables.
```shell
export COMET_MODE=online # Set whether to run Comet in 'online' or 'offline' mode. Defaults to online
export COMET_MODEL_NAME=<your model name> #Set the name for the saved model. Defaults to yolov5
export COMET_LOG_CONFUSION_MATRIX=false # Set to disable logging a Comet Confusion Matrix. Defaults to true
export COMET_MAX_IMAGE_UPLOADS=<number of allowed images to upload to Comet> # Controls how many total image predictions to log to Comet. Defaults to 100.
export COMET_LOG_PER_CLASS_METRICS=true # Set to log evaluation metrics for each detected class at the end of training. Defaults to false
export COMET_DEFAULT_CHECKPOINT_FILENAME=<your checkpoint filename> # Set this if you would like to resume training from a different checkpoint. Defaults to 'last.pt'
export COMET_LOG_BATCH_LEVEL_METRICS=true # Set this if you would like to log training metrics at the batch level. Defaults to false.
export COMET_LOG_PREDICTIONS=true # Set this to false to disable logging model predictions
# Environment Variables for Comet Configuration
export COMET_MODE=online # 'online' or 'offline'. Default: online
export COMET_MODEL_NAME=<your_model_name> # Name for the saved model. Default: yolov5
export COMET_LOG_CONFUSION_MATRIX=false # Disable confusion matrix logging. Default: true
export COMET_MAX_IMAGE_UPLOADS=<number> # Max prediction images to log. Default: 100
export COMET_LOG_PER_CLASS_METRICS=true # Log metrics per class. Default: false
export COMET_DEFAULT_CHECKPOINT_FILENAME=<checkpoint_file.pt> # Checkpoint for resuming. Default: 'last.pt'
export COMET_LOG_BATCH_LEVEL_METRICS=true # Log training metrics per batch. Default: false
export COMET_LOG_PREDICTIONS=true # Disable prediction logging if set to false. Default: true
```
## Logging Checkpoints with Comet
Refer to the [Comet documentation](https://www.comet.com/docs/v2/guides/experiment-logging/configure-comet/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme) for more configuration options.
Logging Models to Comet is disabled by default. To enable it, pass the `save-period` argument to the training script. This will save the logged checkpoints to Comet based on the interval value provided by `save-period`
### Logging Checkpoints with Comet
Model checkpoint logging to Comet is disabled by default. Enable it using the `--save-period` argument during training. This saves checkpoints to Comet at the specified epoch interval.
```shell
python train.py \
@@ -107,18 +117,18 @@ python train.py \
--epochs 5 \
--data coco128.yaml \
--weights yolov5s.pt \
--save-period 1
--save-period 1 # Save checkpoint every epoch
```
## Logging Model Predictions
Checkpoints will appear in the "Assets & Artifacts" tab of your Comet experiment. Learn more about model management in the [Comet Model Registry](https://www.comet.com/docs/v2/guides/model-registry/overview/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme).
By default, model predictions (images, ground truth labels and bounding boxes) will be logged to Comet.
### Logging Model Predictions
You can control the frequency of logged predictions and the associated images by passing the `bbox_interval` command line argument. Predictions can be visualized using Comet's Object Detection Custom Panel. This frequency corresponds to every Nth batch of data per epoch. In the example below, we are logging every 2nd batch of data for each epoch.
By default, model predictions (images, ground truth labels, bounding boxes) for the validation set are logged. Control the logging frequency using the `--bbox_interval` argument, which specifies logging every Nth batch per epoch.
**Note:** The YOLOv5 validation dataloader will default to a batch size of 32, so you will have to set the logging frequency accordingly.
**Note:** The YOLOv5 validation dataloader defaults to a batch size of 32. Adjust `--bbox_interval` accordingly.
Here is an [example project using the Panel](https://www.comet.com/examples/comet-example-yolov5?shareable=YcwMiJaZSXfcEXpGOHDD12vA1&utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github)
Visualize predictions using Comet's [Object Detection Custom Panel](https://www.comet.com/docs/v2/guides/comet-dashboard/built-in-panels/object-detection-panel/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme). See an [example project using the Panel here](https://www.comet.com/examples/comet-example-yolov5?shareable=YcwMiJaZSXfcEXpGOHDD12vA1&utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme).
```shell
python train.py \
@@ -127,12 +137,12 @@ python train.py \
--epochs 5 \
--data coco128.yaml \
--weights yolov5s.pt \
--bbox_interval 2
--bbox_interval 2 # Log predictions every 2nd validation batch per epoch
```
### Controlling the number of Prediction Images logged to Comet
#### Controlling the Number of Prediction Images
When logging predictions from YOLOv5, Comet will log the images associated with each set of predictions. By default a maximum of 100 validation images are logged. You can increase or decrease this number using the `COMET_MAX_IMAGE_UPLOADS` environment variable.
Adjust the maximum number of validation images logged using the `COMET_MAX_IMAGE_UPLOADS` environment variable.
```shell
env COMET_MAX_IMAGE_UPLOADS=200 python train.py \
@@ -141,12 +151,12 @@ env COMET_MAX_IMAGE_UPLOADS=200 python train.py \
--epochs 5 \
--data coco128.yaml \
--weights yolov5s.pt \
--bbox_interval 1
--bbox_interval 1 # Log every batch
```
### Logging Class Level Metrics
Use the `COMET_LOG_PER_CLASS_METRICS` environment variable to log mAP, precision, recall, f1 for each class.
Enable logging of mAP, precision, recall, and F1-score for each class using the `COMET_LOG_PER_CLASS_METRICS` environment variable.
```shell
env COMET_LOG_PER_CLASS_METRICS=true python train.py \
@@ -157,11 +167,13 @@ env COMET_LOG_PER_CLASS_METRICS=true python train.py \
--weights yolov5s.pt
```
## Uploading a Dataset to Comet Artifacts
## 💾 Dataset Management with Comet Artifacts
If you would like to store your data using [Comet Artifacts](https://www.comet.com/docs/v2/guides/data-management/using-artifacts/#learn-more?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github), you can do so using the `upload_dataset` flag.
Use [Comet Artifacts](https://www.comet.com/docs/v2/guides/data-management/using-artifacts/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme) to version, store, and manage your datasets.
The dataset be organized in the way described in the [YOLOv5 documentation](https://docs.ultralytics.com/yolov5/tutorials/train_custom_data/). The dataset config `yaml` file must follow the same format as that of the `coco128.yaml` file.
### Uploading a Dataset
Upload your dataset using the `--upload_dataset` flag. Ensure your dataset follows the structure described in the [YOLOv5 documentation](https://docs.ultralytics.com/yolov5/tutorials/train_custom_data/) and the dataset config [YAML](https://yaml.org/) file matches the format of `coco128.yaml`.
```shell
python train.py \
@@ -170,25 +182,32 @@ python train.py \
--epochs 5 \
--data coco128.yaml \
--weights yolov5s.pt \
--upload_dataset
--upload_dataset # Upload the dataset specified in coco128.yaml
```
You can find the uploaded dataset in the Artifacts tab in your Comet Workspace <img width="1073" alt="artifact-1" src="https://user-images.githubusercontent.com/7529846/186929193-162718bf-ec7b-4eb9-8c3b-86b3763ef8ea.png">
View the uploaded dataset in the Artifacts tab of your Comet Workspace.
<img width="1073" alt="Comet Artifacts tab showing uploaded dataset" src="https://user-images.githubusercontent.com/7529846/186929193-162718bf-ec7b-4eb9-8c3b-86b3763ef8ea.png">
You can preview the data directly in the Comet UI. <img width="1082" alt="artifact-2" src="https://user-images.githubusercontent.com/7529846/186929215-432c36a9-c109-4eb0-944b-84c2786590d6.png">
Preview data directly in the Comet UI.
<img width="1082" alt="Comet UI previewing dataset images" src="https://user-images.githubusercontent.com/7529846/186929215-432c36a9-c109-4eb0-944b-84c2786590d6.png">
Artifacts are versioned and also support adding metadata about the dataset. Comet will automatically log the metadata from your dataset `yaml` file <img width="963" alt="artifact-3" src="https://user-images.githubusercontent.com/7529846/186929256-9d44d6eb-1a19-42de-889a-bcbca3018f2e.png">
Artifacts are versioned and support metadata. Comet automatically logs metadata from your dataset `yaml` file.
<img width="963" alt="Comet Artifact metadata view" src="https://user-images.githubusercontent.com/7529846/186929256-9d44d6eb-1a19-42de-889a-bcbca3018f2e.png">
### Using a saved Artifact
### Using a Saved Artifact
If you would like to use a dataset from Comet Artifacts, set the `path` variable in your dataset `yaml` file to point to the following Artifact resource URL.
To use a dataset stored in Comet Artifacts, update the `path` in your dataset `yaml` file to the Artifact resource URL:
```
# contents of artifact.yaml file
path: "comet://<workspace name>/<artifact name>:<artifact version or alias>"
```yaml
# contents of artifact.yaml
path: "comet://<workspace_name>/<artifact_name>:<artifact_version_or_alias>"
train: images/train # Adjust subdirectory if needed
val: images/val # Adjust subdirectory if needed
# Other dataset configurations...
```
Then pass this file to your training script in the following way
Then, pass this configuration file to your training script:
```shell
python train.py \
@@ -199,35 +218,36 @@ python train.py \
--weights yolov5s.pt
```
Artifacts also allow you to track the lineage of data as it flows through your Experimentation workflow. Here you can see a graph that shows you all the experiments that have used your uploaded dataset. <img width="1391" alt="artifact-4" src="https://user-images.githubusercontent.com/7529846/186929264-4c4014fa-fe51-4f3c-a5c5-f6d24649b1b4.png">
Artifacts track data lineage, showing which experiments used specific dataset versions.
<img width="1391" alt="Comet Artifact lineage graph" src="https://user-images.githubusercontent.com/7529846/186929264-4c4014fa-fe51-4f3c-a5c5-f6d24649b1b4.png">
## Resuming a Training Run
## 🔄 Resuming Training Runs
If your training run is interrupted for any reason, e.g. disrupted internet connection, you can resume the run using the `resume` flag and the Comet Run Path.
If a training run is interrupted (e.g., due to connection issues), you can resume it using the `--resume` flag with the Comet Run Path (`comet://<your_workspace>/<your_project>/<experiment_id>`).
The Run Path has the following format `comet://<your workspace name>/<your project name>/<experiment id>`.
This will restore the run to its state before the interruption, which includes restoring the model from a checkpoint, restoring all hyperparameters and training arguments and downloading Comet dataset Artifacts if they were used in the original run. The resumed run will continue logging to the existing Experiment in the Comet UI
This restores the model state, hyperparameters, arguments, and downloads necessary Artifacts, continuing logging to the existing Comet Experiment. Learn more about [resuming runs in Comet](https://www.comet.com/docs/v2/guides/experiment-logging/resume-experiment/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme).
```shell
python train.py \
--resume "comet://<your run path>"
--resume "comet://<your_workspace>/<your_project>/<experiment_id>"
```
## Hyperparameter Search with the Comet Optimizer
## 🔍 Hyperparameter Optimization (HPO)
YOLOv5 is also integrated with Comet's Optimizer, making is simple to visualize hyperparameter sweeps in the Comet UI.
YOLOv5 integrates with the [Comet Optimizer](https://www.comet.com/docs/v2/guides/hyperparameter-optimization/overview/?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme) for easy hyperparameter sweeps and visualization. This helps in finding the best set of parameters for your model, a process often referred to as [Hyperparameter Tuning](https://docs.ultralytics.com/guides/hyperparameter-tuning/).
### Configuring an Optimizer Sweep
To configure the Comet Optimizer, you will have to create a JSON file with the information about the sweep. An example file has been provided in `utils/loggers/comet/optimizer_config.json`
Create a [JSON](https://www.json.org/json-en.html) configuration file defining the sweep parameters, search strategy, and objective metric. An example is provided at `utils/loggers/comet/optimizer_config.json`.
Run the sweep using the `hpo.py` script:
```shell
python utils/loggers/comet/hpo.py \
--comet_optimizer_config "utils/loggers/comet/optimizer_config.json"
```
The `hpo.py` script accepts the same arguments as `train.py`. If you wish to pass additional arguments to your sweep simply add them after the script.
The `hpo.py` script accepts the same arguments as `train.py`. Pass additional fixed arguments for the sweep:
```shell
python utils/loggers/comet/hpo.py \
@@ -238,13 +258,21 @@ python utils/loggers/comet/hpo.py \
### Running a Sweep in Parallel
Execute multiple sweep trials concurrently using the `comet optimizer` command:
```shell
comet optimizer -j <set number of workers> utils/loggers/comet/hpo.py \
utils/loggers/comet/optimizer_config.json"
comet optimizer -j \
utils/loggers/comet/optimizer_config.json < num_workers > utils/loggers/comet/hpo.py
```
### Visualizing Results
Replace `<num_workers>` with the desired number of parallel processes.
Comet provides a number of ways to visualize the results of your sweep. Take a look at a [project with a completed sweep here](https://www.comet.com/examples/comet-example-yolov5/view/PrlArHGuuhDTKC1UuBmTtOSXD/panels?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github)
### Visualizing HPO Results
<img width="1626" alt="hyperparameter-yolo" src="https://user-images.githubusercontent.com/7529846/186914869-7dc1de14-583f-4323-967b-c9a66a29e495.png">
Comet offers various visualizations for analyzing sweep results, such as parallel coordinate plots and parameter importance plots. Explore a [project with a completed sweep here](https://www.comet.com/examples/comet-example-yolov5/view/PrlArHGuuhDTKC1UuBmTtOSXD/panels?utm_source=yolov5&utm_medium=partner&utm_campaign=partner_yolov5_2022&utm_content=github_readme).
<img width="1626" alt="Comet HPO visualization" src="https://user-images.githubusercontent.com/7529846/186914869-7dc1de14-583f-4323-967b-c9a66a29e495.png">
## 🤝 Contributing
Contributions to enhance the YOLOv5-Comet integration are welcome! Please see the [Ultralytics Contributing Guide](https://docs.ultralytics.com/help/contributing/) for more information on how to get involved. Thank you for helping improve this integration!