Detect.py supports running against a Triton container (#9228)

* update coco128-seg comments

* Enables detect.py to use Triton for inference

Triton Inference Server is an open source inference serving software
that streamlines AI inferencing.
https://github.com/triton-inference-server/server

The user can now provide a "--triton-url" argument to detect.py to use
a local or remote Triton server for inference.
For e.g., http://localhost:8000 will use http over port 8000
and grpc://localhost:8001 will use grpc over port 8001.
Note, it is not necessary to specify a weights file to use Triton.

A Triton container can be created by first exporting the Yolov5 model
to a Triton supported runtime. Onnx, Torchscript, TensorRT are
supported by both Triton and the export.py script.

The exported model can then be containerized via the OctoML CLI.
See https://github.com/octoml/octo-cli#getting-started for a guide.

* added triton client to requirements

* fixed support for TFSavedModels in Triton

* reverted change

* Test CoreML update

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update ci-testing.yml

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Use pathlib

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Refacto DetectMultiBackend to directly accept triton url as --weights http://...

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Deploy category

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update detect.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update common.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update common.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update predict.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update predict.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update predict.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update triton.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update triton.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add printout and requirements check

* Cleanup

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* triton fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed triton model query over grpc

* Update check_requirements('tritonclient[all]')

* group imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix likely remote URL bug

* update comment

* Update is_url()

* Fix 2x download attempt on http://path/to/model.pt

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: glennjocher <glenn.jocher@ultralytics.com>
Co-authored-by: Gaz Iqbal <giqbal@octoml.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

detect.py

@@ -49,7 +49,7 @@ from utils.torch_utils import select_device, smart_inference_mode
 
 @smart_inference_mode()
 def run(
-        weights=ROOT / 'yolov5s.pt',  # model.pt path(s)
+        weights=ROOT / 'yolov5s.pt',  # model path or triton URL
         source=ROOT / 'data/images',  # file/dir/URL/glob/screen/0(webcam)
         data=ROOT / 'data/coco128.yaml',  # dataset.yaml path
         imgsz=(640, 640),  # inference size (height, width)
@@ -108,11 +108,11 @@ def run(
     vid_path, vid_writer = [None] * bs, [None] * bs
 
     # Run inference
-    model.warmup(imgsz=(1 if pt else bs, 3, *imgsz))  # warmup
+    model.warmup(imgsz=(1 if pt or model.triton else bs, 3, *imgsz))  # warmup
     seen, windows, dt = 0, [], (Profile(), Profile(), Profile())
     for path, im, im0s, vid_cap, s in dataset:
         with dt[0]:
-            im = torch.from_numpy(im).to(device)
+            im = torch.from_numpy(im).to(model.device)
             im = im.half() if model.fp16 else im.float()  # uint8 to fp16/32
             im /= 255  # 0 - 255 to 0.0 - 1.0
             if len(im.shape) == 3:
@@ -214,7 +214,7 @@ def run(
 
 def parse_opt():
     parser = argparse.ArgumentParser()
-    parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s.pt', help='model path(s)')
+    parser.add_argument('--weights', nargs='+', type=str, default=ROOT / 'yolov5s.pt', help='model path or triton URL')
     parser.add_argument('--source', type=str, default=ROOT / 'data/images', help='file/dir/URL/glob/screen/0(webcam)')
     parser.add_argument('--data', type=str, default=ROOT / 'data/coco128.yaml', help='(optional) dataset.yaml path')
     parser.add_argument('--imgsz', '--img', '--img-size', nargs='+', type=int, default=[640], help='inference size h,w')