- 블로그 방문을 환영합니다. (255)
Notice
Recent Posts
Recent Comments
Link
| 일 | 월 | 화 | 수 | 목 | 금 | 토 |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | ||||
| 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| 11 | 12 | 13 | 14 | 15 | 16 | 17 |
| 18 | 19 | 20 | 21 | 22 | 23 | 24 |
| 25 | 26 | 27 | 28 | 29 | 30 | 31 |
Tags
- #undef
- bootstrap4 패키지
- 콘솔 가상환경 # 콘솔 #가상환경
- MySQL 세팅
- #if
- #endif
- 성능지표
- heroku
- Django
- jinja2
- 사이트 도메인
- PyQt5
- 튜토리얼
- #ifdef
- DB 데이터
- 환경변수 설정
- Action
- javascript
- bootstrap
- #ifndef
- OpenCV
- 명령어
- href
- VS Code
- DB 데이터 저장
- OpenCV + Flask
- flask
- #else
- 실시간 시계
- openweathermap
Archives
- Today
- Total
PROGRAMMING
YOLO (gum, chocolate, milk / 실습) 본문
1. Mount Google Drive¶
In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [ ]:
!ls -la "/content/drive/MyDrive/darknet"
total 32 drwx------ 2 root root 4096 Dec 23 03:16 backup drwx------ 2 root root 4096 Dec 23 02:33 bin drwx------ 2 root root 4096 Dec 23 03:04 cfg drwx------ 2 root root 4096 Dec 23 02:16 cuDNN drwx------ 2 root root 4096 Dec 23 03:04 custom drwx------ 2 root root 4096 Dec 23 03:04 data drwx------ 2 root root 4096 Dec 23 02:16 .ipynb_checkpoints drwx------ 2 root root 4096 Dec 23 02:36 weights
In [ ]:
!ls -la /usr/local/cuda/include/cudnn.h
-r--r--r-- 1 root root 134857 Sep 26 2019 /usr/local/cuda/include/cudnn.h
2. Install CUDA related modules, cuDNN¶
Unzip the cuDNN from Google Drive to Colab /usr/local/cuda folder
In [ ]:
!tar -xzvf /content/drive/MyDrive/darknet/cuDNN/cudnn-10.1-linux-x64-v7.6.5.32.tgz -C /usr/local/
!chmod a+r /usr/local/cuda/include/cudnn.h
!cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
cuda/include/cudnn.h cuda/NVIDIA_SLA_cuDNN_Support.txt cuda/lib64/libcudnn.so cuda/lib64/libcudnn.so.7 cuda/lib64/libcudnn.so.7.6.5 cuda/lib64/libcudnn_static.a #define CUDNN_MAJOR 7 #define CUDNN_MINOR 6 #define CUDNN_PATCHLEVEL 5 -- #define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL) #include "driver_types.h"
3. Load DarkNet¶
In [ ]:
import os
if not os.path.exists('darknet'):
os.makedirs('darknet')
%cd darknet
%ls
/content/darknet
In [ ]:
!ls -la '/content/drive/MyDrive/darknet/bin/darknet'
!cp /content/drive/MyDrive/darknet/bin/darknet ./darknet
!chmod +x ./darknet
-rw------- 1 root root 3213624 Dec 23 02:35 /content/drive/MyDrive/darknet/bin/darknet
4. Test Yolo¶
In [14]:
!cp -r '/content/drive/MyDrive/darknet/weights' .
!cp -r '/content/drive/MyDrive/darknet/cfg' .
!cp -ar '/content/drive/MyDrive/darknet/data' .
%ls
cfg/ custom/ darknet* data/ weights/
In [15]:
def imShow(path):
import cv2
import matplotlib.pyplot as plt
%matplotlib inline
image = cv2.imread(path)
height, width = image.shape[:2]
resized_image = cv2.resize(image,(3*width, 3*height), interpolation = cv2.INTER_CUBIC)
fig = plt.gcf()
fig.set_size_inches(18, 10)
plt.axis("off")
#plt.rcParams['figure.figsize'] = [10, 5]
plt.imshow(cv2.cvtColor(resized_image, cv2.COLOR_BGR2RGB))
plt.show()
def upload():
from google.colab import files
uploaded = files.upload()
for name, data in uploaded.items():
with open(name, 'wb') as f:
f.write(data)
print ('saved file', name)
def download(path):
from google.colab import files
files.download(path)
In [ ]:
!./darknet detect cfg/yolov3.cfg weights/yolov3.weights data/fruit10.jpg
imShow('predictions.jpg')
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 255 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 255 0.088 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 255 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 255 0.177 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 255 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 255 0.353 BF 106 yolo Total BFLOPS 65.864 Allocate additional workspace_size = 52.43 MB Loading weights from weights/yolov3.weights... seen 64 Done! data/fruit10.jpg: Predicted in 41.922000 milli-seconds. orange: 99% orange: 59% apple: 85% orange: 99% Unable to init server: Could not connect: Connection refused (predictions:333): Gtk-WARNING **: 00:24:53.984: cannot open display:
5. Custom Yolo¶
In [32]:
!cp -r "/content/drive/MyDrive/darknet/custom" .
아래를 누르면 아주 긴 시간 동안 학습하게 됨¶
In [19]:
!./darknet detector train custom/custom_data.data custom/custom-train-yolo.cfg weights/darknet53.conv.74 -dont_show
스트리밍 출력 내용이 길어서 마지막 5000줄이 삭제되었습니다.
Loaded: 0.000039 seconds
4535: 0.016981, 0.026326 avg loss, 0.001000 rate, 2.478710 seconds, 145120 images
Loaded: 0.000066 seconds
4536: 0.027410, 0.026435 avg loss, 0.001000 rate, 2.473895 seconds, 145152 images
Loaded: 0.000074 seconds
4537: 0.015900, 0.025381 avg loss, 0.001000 rate, 2.475527 seconds, 145184 images
Loaded: 0.000038 seconds
4538: 0.005233, 0.023367 avg loss, 0.001000 rate, 2.480626 seconds, 145216 images
Loaded: 0.000069 seconds
4539: 0.025539, 0.023584 avg loss, 0.001000 rate, 2.472172 seconds, 145248 images
Loaded: 0.000036 seconds
4540: 0.037033, 0.024929 avg loss, 0.001000 rate, 2.483267 seconds, 145280 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.050323 seconds
4541: 0.021305, 0.024566 avg loss, 0.001000 rate, 1.794421 seconds, 145312 images
Loaded: 0.000036 seconds
4542: 0.015077, 0.023617 avg loss, 0.001000 rate, 1.871934 seconds, 145344 images
Loaded: 0.000043 seconds
4543: 0.021076, 0.023363 avg loss, 0.001000 rate, 1.871767 seconds, 145376 images
Loaded: 0.000039 seconds
4544: 0.017245, 0.022751 avg loss, 0.001000 rate, 1.864361 seconds, 145408 images
Loaded: 0.000053 seconds
4545: 0.020677, 0.022544 avg loss, 0.001000 rate, 1.861999 seconds, 145440 images
Loaded: 0.000042 seconds
4546: 0.020260, 0.022316 avg loss, 0.001000 rate, 1.864988 seconds, 145472 images
Loaded: 0.000037 seconds
4547: 0.012198, 0.021304 avg loss, 0.001000 rate, 1.869541 seconds, 145504 images
Loaded: 0.000049 seconds
4548: 0.026619, 0.021835 avg loss, 0.001000 rate, 1.866041 seconds, 145536 images
Loaded: 0.000043 seconds
4549: 0.020161, 0.021668 avg loss, 0.001000 rate, 1.868986 seconds, 145568 images
Loaded: 0.000042 seconds
4550: 0.025344, 0.022035 avg loss, 0.001000 rate, 1.873055 seconds, 145600 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000041 seconds
4551: 0.030547, 0.022887 avg loss, 0.001000 rate, 2.926625 seconds, 145632 images
Loaded: 0.000044 seconds
4552: 0.017839, 0.022382 avg loss, 0.001000 rate, 3.036515 seconds, 145664 images
Loaded: 0.000069 seconds
4553: 0.032114, 0.023355 avg loss, 0.001000 rate, 3.032995 seconds, 145696 images
Loaded: 0.000060 seconds
4554: 0.027320, 0.023752 avg loss, 0.001000 rate, 3.034624 seconds, 145728 images
Loaded: 0.000038 seconds
4555: 0.023832, 0.023760 avg loss, 0.001000 rate, 3.034686 seconds, 145760 images
Loaded: 0.000062 seconds
4556: 0.039567, 0.025340 avg loss, 0.001000 rate, 3.032339 seconds, 145792 images
Loaded: 0.000035 seconds
4557: 0.026659, 0.025472 avg loss, 0.001000 rate, 3.037356 seconds, 145824 images
Loaded: 0.000052 seconds
4558: 0.029107, 0.025836 avg loss, 0.001000 rate, 3.033708 seconds, 145856 images
Loaded: 0.000038 seconds
4559: 0.013822, 0.024634 avg loss, 0.001000 rate, 3.038497 seconds, 145888 images
Loaded: 0.000038 seconds
4560: 0.021207, 0.024292 avg loss, 0.001000 rate, 3.037793 seconds, 145920 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.032191 seconds
4561: 0.015445, 0.023407 avg loss, 0.001000 rate, 2.927680 seconds, 145952 images
Loaded: 0.000053 seconds
4562: 0.019315, 0.022998 avg loss, 0.001000 rate, 3.035527 seconds, 145984 images
Loaded: 0.000037 seconds
4563: 0.014747, 0.022173 avg loss, 0.001000 rate, 3.038748 seconds, 146016 images
Loaded: 0.000042 seconds
4564: 0.017306, 0.021686 avg loss, 0.001000 rate, 3.034865 seconds, 146048 images
Loaded: 0.000039 seconds
4565: 0.016130, 0.021131 avg loss, 0.001000 rate, 3.044429 seconds, 146080 images
Loaded: 0.000037 seconds
4566: 0.014629, 0.020480 avg loss, 0.001000 rate, 3.039007 seconds, 146112 images
Loaded: 0.000040 seconds
4567: 0.016071, 0.020039 avg loss, 0.001000 rate, 3.033477 seconds, 146144 images
Loaded: 0.000036 seconds
4568: 0.014674, 0.019503 avg loss, 0.001000 rate, 3.043834 seconds, 146176 images
Loaded: 0.000040 seconds
4569: 0.034496, 0.021002 avg loss, 0.001000 rate, 3.040313 seconds, 146208 images
Loaded: 0.000039 seconds
4570: 0.011565, 0.020058 avg loss, 0.001000 rate, 3.031688 seconds, 146240 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000036 seconds
4571: 0.012699, 0.019323 avg loss, 0.001000 rate, 4.355851 seconds, 146272 images
Loaded: 0.000041 seconds
4572: 0.050827, 0.022473 avg loss, 0.001000 rate, 4.515842 seconds, 146304 images
Loaded: 0.000038 seconds
4573: 0.015143, 0.021740 avg loss, 0.001000 rate, 4.507013 seconds, 146336 images
Loaded: 0.000065 seconds
4574: 0.014344, 0.021000 avg loss, 0.001000 rate, 4.511007 seconds, 146368 images
Loaded: 0.000056 seconds
4575: 0.019296, 0.020830 avg loss, 0.001000 rate, 4.507022 seconds, 146400 images
Loaded: 0.000041 seconds
4576: 0.024402, 0.021187 avg loss, 0.001000 rate, 4.520625 seconds, 146432 images
Loaded: 0.000041 seconds
4577: 0.030953, 0.022164 avg loss, 0.001000 rate, 4.511074 seconds, 146464 images
Loaded: 0.000057 seconds
4578: 0.033407, 0.023288 avg loss, 0.001000 rate, 4.527636 seconds, 146496 images
Loaded: 0.000064 seconds
4579: 0.021813, 0.023141 avg loss, 0.001000 rate, 4.507432 seconds, 146528 images
Loaded: 0.000089 seconds
4580: 0.023223, 0.023149 avg loss, 0.001000 rate, 4.526392 seconds, 146560 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000056 seconds
4581: 0.009041, 0.021738 avg loss, 0.001000 rate, 3.177253 seconds, 146592 images
Loaded: 0.000065 seconds
4582: 0.015864, 0.021151 avg loss, 0.001000 rate, 3.301490 seconds, 146624 images
Loaded: 0.000041 seconds
4583: 0.040551, 0.023091 avg loss, 0.001000 rate, 3.284777 seconds, 146656 images
Loaded: 0.000039 seconds
4584: 0.025775, 0.023359 avg loss, 0.001000 rate, 3.291185 seconds, 146688 images
Loaded: 0.000038 seconds
4585: 0.023549, 0.023378 avg loss, 0.001000 rate, 3.287970 seconds, 146720 images
Loaded: 0.000070 seconds
4586: 0.011869, 0.022227 avg loss, 0.001000 rate, 3.295671 seconds, 146752 images
Loaded: 0.000072 seconds
4587: 0.044688, 0.024473 avg loss, 0.001000 rate, 3.282383 seconds, 146784 images
Loaded: 0.000031 seconds
4588: 0.031347, 0.025161 avg loss, 0.001000 rate, 3.282576 seconds, 146816 images
Loaded: 0.000041 seconds
4589: 0.012742, 0.023919 avg loss, 0.001000 rate, 3.286892 seconds, 146848 images
Loaded: 0.000045 seconds
4590: 0.021886, 0.023716 avg loss, 0.001000 rate, 3.297603 seconds, 146880 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.036213 seconds
4591: 0.022176, 0.023562 avg loss, 0.001000 rate, 3.172481 seconds, 146912 images
Loaded: 0.000043 seconds
4592: 0.019917, 0.023197 avg loss, 0.001000 rate, 3.288412 seconds, 146944 images
Loaded: 0.000044 seconds
4593: 0.012454, 0.022123 avg loss, 0.001000 rate, 3.278471 seconds, 146976 images
Loaded: 0.000062 seconds
4594: 0.016865, 0.021597 avg loss, 0.001000 rate, 3.278401 seconds, 147008 images
Loaded: 0.000040 seconds
4595: 0.015294, 0.020967 avg loss, 0.001000 rate, 3.270882 seconds, 147040 images
Loaded: 0.000037 seconds
4596: 0.009475, 0.019818 avg loss, 0.001000 rate, 3.289982 seconds, 147072 images
Loaded: 0.000052 seconds
4597: 0.009283, 0.018764 avg loss, 0.001000 rate, 3.277177 seconds, 147104 images
Loaded: 0.000038 seconds
4598: 0.009820, 0.017870 avg loss, 0.001000 rate, 3.289301 seconds, 147136 images
Loaded: 0.000039 seconds
4599: 0.022317, 0.018314 avg loss, 0.001000 rate, 3.285630 seconds, 147168 images
Loaded: 0.000051 seconds
4600: 0.008114, 0.017294 avg loss, 0.001000 rate, 3.279302 seconds, 147200 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.337665 seconds
4601: 0.009588, 0.016524 avg loss, 0.001000 rate, 2.613137 seconds, 147232 images
Loaded: 0.000057 seconds
4602: 0.011991, 0.016070 avg loss, 0.001000 rate, 2.716970 seconds, 147264 images
Loaded: 0.000067 seconds
4603: 0.028131, 0.017277 avg loss, 0.001000 rate, 2.713239 seconds, 147296 images
Loaded: 0.000042 seconds
4604: 0.012343, 0.016783 avg loss, 0.001000 rate, 2.723734 seconds, 147328 images
Loaded: 0.000040 seconds
4605: 0.020426, 0.017147 avg loss, 0.001000 rate, 2.720687 seconds, 147360 images
Loaded: 0.000040 seconds
4606: 0.012190, 0.016652 avg loss, 0.001000 rate, 2.719213 seconds, 147392 images
Loaded: 0.000065 seconds
4607: 0.017344, 0.016721 avg loss, 0.001000 rate, 2.716291 seconds, 147424 images
Loaded: 0.000035 seconds
4608: 0.008887, 0.015938 avg loss, 0.001000 rate, 2.717518 seconds, 147456 images
Loaded: 0.000033 seconds
4609: 0.010017, 0.015346 avg loss, 0.001000 rate, 2.723553 seconds, 147488 images
Loaded: 0.000053 seconds
4610: 0.010416, 0.014853 avg loss, 0.001000 rate, 2.718100 seconds, 147520 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.081018 seconds
4611: 0.013044, 0.014672 avg loss, 0.001000 rate, 2.386254 seconds, 147552 images
Loaded: 0.000052 seconds
4612: 0.026213, 0.015826 avg loss, 0.001000 rate, 2.481883 seconds, 147584 images
Loaded: 0.000040 seconds
4613: 0.013768, 0.015620 avg loss, 0.001000 rate, 2.480923 seconds, 147616 images
Loaded: 0.000054 seconds
4614: 0.008494, 0.014907 avg loss, 0.001000 rate, 2.477652 seconds, 147648 images
Loaded: 0.000038 seconds
4615: 0.010186, 0.014435 avg loss, 0.001000 rate, 2.477541 seconds, 147680 images
Loaded: 0.000067 seconds
4616: 0.012525, 0.014244 avg loss, 0.001000 rate, 2.477972 seconds, 147712 images
Loaded: 0.000079 seconds
4617: 0.009621, 0.013782 avg loss, 0.001000 rate, 2.479758 seconds, 147744 images
Loaded: 0.000066 seconds
4618: 0.022814, 0.014685 avg loss, 0.001000 rate, 2.478120 seconds, 147776 images
Loaded: 0.000044 seconds
4619: 0.008896, 0.014106 avg loss, 0.001000 rate, 2.475675 seconds, 147808 images
Loaded: 0.000074 seconds
4620: 0.009927, 0.013688 avg loss, 0.001000 rate, 2.487789 seconds, 147840 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000041 seconds
4621: 0.012848, 0.013604 avg loss, 0.001000 rate, 2.619973 seconds, 147872 images
Loaded: 0.000048 seconds
4622: 0.009526, 0.013196 avg loss, 0.001000 rate, 2.725613 seconds, 147904 images
Loaded: 0.000042 seconds
4623: 0.017964, 0.013673 avg loss, 0.001000 rate, 2.729453 seconds, 147936 images
Loaded: 0.000064 seconds
4624: 0.012926, 0.013599 avg loss, 0.001000 rate, 2.715061 seconds, 147968 images
Loaded: 0.000039 seconds
4625: 0.006916, 0.012930 avg loss, 0.001000 rate, 2.717646 seconds, 148000 images
Loaded: 0.000036 seconds
4626: 0.006634, 0.012301 avg loss, 0.001000 rate, 2.728958 seconds, 148032 images
Loaded: 0.000038 seconds
4627: 0.008016, 0.011872 avg loss, 0.001000 rate, 2.719002 seconds, 148064 images
Loaded: 0.000052 seconds
4628: 0.013128, 0.011998 avg loss, 0.001000 rate, 2.722141 seconds, 148096 images
Loaded: 0.000038 seconds
4629: 0.011229, 0.011921 avg loss, 0.001000 rate, 2.718335 seconds, 148128 images
Loaded: 0.000040 seconds
4630: 0.010113, 0.011740 avg loss, 0.001000 rate, 2.718722 seconds, 148160 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.030468 seconds
4631: 0.007791, 0.011345 avg loss, 0.001000 rate, 2.616016 seconds, 148192 images
Loaded: 0.000039 seconds
4632: 0.009700, 0.011181 avg loss, 0.001000 rate, 2.721707 seconds, 148224 images
Loaded: 0.000034 seconds
4633: 0.008975, 0.010960 avg loss, 0.001000 rate, 2.719533 seconds, 148256 images
Loaded: 0.000045 seconds
4634: 0.009584, 0.010823 avg loss, 0.001000 rate, 2.721990 seconds, 148288 images
Loaded: 0.000043 seconds
4635: 0.007206, 0.010461 avg loss, 0.001000 rate, 2.723115 seconds, 148320 images
Loaded: 0.000038 seconds
4636: 0.011404, 0.010555 avg loss, 0.001000 rate, 2.721917 seconds, 148352 images
Loaded: 0.000038 seconds
4637: 0.010136, 0.010513 avg loss, 0.001000 rate, 2.722216 seconds, 148384 images
Loaded: 0.000039 seconds
4638: 0.017893, 0.011251 avg loss, 0.001000 rate, 2.719756 seconds, 148416 images
Loaded: 0.000037 seconds
4639: 0.006959, 0.010822 avg loss, 0.001000 rate, 2.719140 seconds, 148448 images
Loaded: 0.000034 seconds
4640: 0.007548, 0.010495 avg loss, 0.001000 rate, 2.718306 seconds, 148480 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000044 seconds
4641: 0.010306, 0.010476 avg loss, 0.001000 rate, 2.020813 seconds, 148512 images
Loaded: 0.000040 seconds
4642: 0.012027, 0.010631 avg loss, 0.001000 rate, 2.114562 seconds, 148544 images
Loaded: 0.000042 seconds
4643: 0.009789, 0.010547 avg loss, 0.001000 rate, 2.110603 seconds, 148576 images
Loaded: 0.000042 seconds
4644: 0.020119, 0.011504 avg loss, 0.001000 rate, 2.109310 seconds, 148608 images
Loaded: 0.000078 seconds
4645: 0.012989, 0.011653 avg loss, 0.001000 rate, 2.102388 seconds, 148640 images
Loaded: 0.000038 seconds
4646: 0.027466, 0.013234 avg loss, 0.001000 rate, 2.100353 seconds, 148672 images
Loaded: 0.000034 seconds
4647: 0.013626, 0.013273 avg loss, 0.001000 rate, 2.106977 seconds, 148704 images
Loaded: 0.000036 seconds
4648: 0.008882, 0.012834 avg loss, 0.001000 rate, 2.103270 seconds, 148736 images
Loaded: 0.000087 seconds
4649: 0.008199, 0.012370 avg loss, 0.001000 rate, 2.104576 seconds, 148768 images
Loaded: 0.000061 seconds
4650: 0.007462, 0.011880 avg loss, 0.001000 rate, 2.103088 seconds, 148800 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.098215 seconds
4651: 0.037507, 0.014442 avg loss, 0.001000 rate, 1.785837 seconds, 148832 images
Loaded: 0.000038 seconds
4652: 0.018993, 0.014897 avg loss, 0.001000 rate, 1.870742 seconds, 148864 images
Loaded: 0.000065 seconds
4653: 0.013758, 0.014783 avg loss, 0.001000 rate, 1.863203 seconds, 148896 images
Loaded: 0.000056 seconds
4654: 0.013240, 0.014629 avg loss, 0.001000 rate, 1.872749 seconds, 148928 images
Loaded: 0.000042 seconds
4655: 0.009555, 0.014122 avg loss, 0.001000 rate, 1.868932 seconds, 148960 images
Loaded: 0.000062 seconds
4656: 0.019275, 0.014637 avg loss, 0.001000 rate, 1.873420 seconds, 148992 images
Loaded: 0.000034 seconds
4657: 0.025136, 0.015687 avg loss, 0.001000 rate, 1.864131 seconds, 149024 images
Loaded: 0.000037 seconds
4658: 0.012100, 0.015328 avg loss, 0.001000 rate, 1.863548 seconds, 149056 images
Loaded: 0.000039 seconds
4659: 0.031504, 0.016946 avg loss, 0.001000 rate, 1.865270 seconds, 149088 images
Loaded: 0.000035 seconds
4660: 0.020996, 0.017351 avg loss, 0.001000 rate, 1.867151 seconds, 149120 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000069 seconds
4661: 0.010300, 0.016646 avg loss, 0.001000 rate, 3.148724 seconds, 149152 images
Loaded: 0.000036 seconds
4662: 0.025556, 0.017537 avg loss, 0.001000 rate, 3.282660 seconds, 149184 images
Loaded: 0.000053 seconds
4663: 0.007441, 0.016527 avg loss, 0.001000 rate, 3.277648 seconds, 149216 images
Loaded: 0.000042 seconds
4664: 0.012108, 0.016085 avg loss, 0.001000 rate, 3.273338 seconds, 149248 images
Loaded: 0.000037 seconds
4665: 0.013677, 0.015844 avg loss, 0.001000 rate, 3.279836 seconds, 149280 images
Loaded: 0.000058 seconds
4666: 0.012982, 0.015558 avg loss, 0.001000 rate, 3.287041 seconds, 149312 images
Loaded: 0.000044 seconds
4667: 0.012170, 0.015219 avg loss, 0.001000 rate, 3.273963 seconds, 149344 images
Loaded: 0.000044 seconds
4668: 0.010815, 0.014779 avg loss, 0.001000 rate, 3.300842 seconds, 149376 images
Loaded: 0.000038 seconds
4669: 0.011536, 0.014455 avg loss, 0.001000 rate, 3.271884 seconds, 149408 images
Loaded: 0.000062 seconds
4670: 0.010430, 0.014052 avg loss, 0.001000 rate, 3.289307 seconds, 149440 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000042 seconds
4671: 0.021745, 0.014822 avg loss, 0.001000 rate, 1.798360 seconds, 149472 images
Loaded: 0.000032 seconds
4672: 0.030528, 0.016392 avg loss, 0.001000 rate, 1.865156 seconds, 149504 images
Loaded: 0.000031 seconds
4673: 0.012844, 0.016037 avg loss, 0.001000 rate, 1.862582 seconds, 149536 images
Loaded: 0.000044 seconds
4674: 0.019933, 0.016427 avg loss, 0.001000 rate, 1.867171 seconds, 149568 images
Loaded: 0.000040 seconds
4675: 0.023035, 0.017088 avg loss, 0.001000 rate, 1.874530 seconds, 149600 images
Loaded: 0.000040 seconds
4676: 0.028874, 0.018266 avg loss, 0.001000 rate, 1.866847 seconds, 149632 images
Loaded: 0.000060 seconds
4677: 0.023245, 0.018764 avg loss, 0.001000 rate, 1.868565 seconds, 149664 images
Loaded: 0.000042 seconds
4678: 0.016192, 0.018507 avg loss, 0.001000 rate, 1.867229 seconds, 149696 images
Loaded: 0.000030 seconds
4679: 0.040311, 0.020687 avg loss, 0.001000 rate, 1.864486 seconds, 149728 images
Loaded: 0.000037 seconds
4680: 0.031979, 0.021817 avg loss, 0.001000 rate, 1.866103 seconds, 149760 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.091294 seconds
4681: 0.021351, 0.021770 avg loss, 0.001000 rate, 1.790640 seconds, 149792 images
Loaded: 0.000033 seconds
4682: 0.024442, 0.022037 avg loss, 0.001000 rate, 1.863432 seconds, 149824 images
Loaded: 0.000040 seconds
4683: 0.039644, 0.023798 avg loss, 0.001000 rate, 1.864570 seconds, 149856 images
Loaded: 0.000037 seconds
4684: 0.014174, 0.022835 avg loss, 0.001000 rate, 1.867064 seconds, 149888 images
Loaded: 0.000038 seconds
4685: 0.017415, 0.022293 avg loss, 0.001000 rate, 1.869198 seconds, 149920 images
Loaded: 0.000057 seconds
4686: 0.019777, 0.022042 avg loss, 0.001000 rate, 1.863468 seconds, 149952 images
Loaded: 0.000034 seconds
4687: 0.016456, 0.021483 avg loss, 0.001000 rate, 1.868875 seconds, 149984 images
Loaded: 0.000063 seconds
4688: 0.017924, 0.021127 avg loss, 0.001000 rate, 1.861884 seconds, 150016 images
Loaded: 0.000035 seconds
4689: 0.029272, 0.021942 avg loss, 0.001000 rate, 1.865097 seconds, 150048 images
Loaded: 0.000039 seconds
4690: 0.030781, 0.022826 avg loss, 0.001000 rate, 1.864711 seconds, 150080 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000041 seconds
4691: 0.015059, 0.022049 avg loss, 0.001000 rate, 2.392122 seconds, 150112 images
Loaded: 0.000039 seconds
4692: 0.020137, 0.021858 avg loss, 0.001000 rate, 2.478756 seconds, 150144 images
Loaded: 0.000036 seconds
4693: 0.013870, 0.021059 avg loss, 0.001000 rate, 2.475681 seconds, 150176 images
Loaded: 0.000057 seconds
4694: 0.009527, 0.019906 avg loss, 0.001000 rate, 2.491620 seconds, 150208 images
Loaded: 0.000035 seconds
4695: 0.018563, 0.019772 avg loss, 0.001000 rate, 2.482657 seconds, 150240 images
Loaded: 0.000074 seconds
4696: 0.016707, 0.019465 avg loss, 0.001000 rate, 2.480303 seconds, 150272 images
Loaded: 0.000035 seconds
4697: 0.021527, 0.019671 avg loss, 0.001000 rate, 2.480349 seconds, 150304 images
Loaded: 0.000040 seconds
4698: 0.013613, 0.019065 avg loss, 0.001000 rate, 2.480315 seconds, 150336 images
Loaded: 0.000045 seconds
4699: 0.029008, 0.020060 avg loss, 0.001000 rate, 2.482213 seconds, 150368 images
Loaded: 0.000034 seconds
4700: 0.011094, 0.019163 avg loss, 0.001000 rate, 2.479018 seconds, 150400 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000040 seconds
4701: 0.013490, 0.018596 avg loss, 0.001000 rate, 4.353388 seconds, 150432 images
Loaded: 0.000057 seconds
4702: 0.018190, 0.018555 avg loss, 0.001000 rate, 4.504595 seconds, 150464 images
Loaded: 0.000066 seconds
4703: 0.022301, 0.018930 avg loss, 0.001000 rate, 4.506723 seconds, 150496 images
Loaded: 0.000036 seconds
4704: 0.019201, 0.018957 avg loss, 0.001000 rate, 4.515001 seconds, 150528 images
Loaded: 0.000039 seconds
4705: 0.017219, 0.018783 avg loss, 0.001000 rate, 4.512085 seconds, 150560 images
Loaded: 0.000038 seconds
4706: 0.014434, 0.018348 avg loss, 0.001000 rate, 4.509695 seconds, 150592 images
Loaded: 0.000076 seconds
4707: 0.017301, 0.018243 avg loss, 0.001000 rate, 4.515263 seconds, 150624 images
Loaded: 0.000036 seconds
4708: 0.037822, 0.020201 avg loss, 0.001000 rate, 4.517042 seconds, 150656 images
Loaded: 0.000041 seconds
4709: 0.011030, 0.019284 avg loss, 0.001000 rate, 4.504755 seconds, 150688 images
Loaded: 0.000039 seconds
4710: 0.011667, 0.018522 avg loss, 0.001000 rate, 4.522858 seconds, 150720 images
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000040 seconds
4711: 0.018918, 0.018562 avg loss, 0.001000 rate, 4.728459 seconds, 150752 images
Loaded: 0.000040 seconds
4712: 0.038974, 0.020603 avg loss, 0.001000 rate, 4.899990 seconds, 150784 images
Loaded: 0.000054 seconds
4713: 0.018660, 0.020409 avg loss, 0.001000 rate, 4.900235 seconds, 150816 images
Loaded: 0.000036 seconds
4714: 0.014994, 0.019867 avg loss, 0.001000 rate, 4.898347 seconds, 150848 images
Loaded: 0.000031 seconds
4715: 0.010947, 0.018975 avg loss, 0.001000 rate, 4.916114 seconds, 150880 images
Loaded: 0.000038 seconds
4716: 0.011109, 0.018189 avg loss, 0.001000 rate, 4.922256 seconds, 150912 images
Loaded: 0.000063 seconds
4717: 0.011770, 0.017547 avg loss, 0.001000 rate, 4.921569 seconds, 150944 images
Loaded: 0.000035 seconds
4718: 0.029145, 0.018707 avg loss, 0.001000 rate, 4.913769 seconds, 150976 images
Loaded: 0.000036 seconds
4719: 0.021133, 0.018949 avg loss, 0.001000 rate, 4.905687 seconds, 151008 images
Loaded: 0.000037 seconds
4720: 0.011224, 0.018177 avg loss, 0.001000 rate, 4.906643 seconds, 151040 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000074 seconds
4721: 0.014130, 0.017772 avg loss, 0.001000 rate, 3.178054 seconds, 151072 images
Loaded: 0.000043 seconds
4722: 0.026381, 0.018633 avg loss, 0.001000 rate, 3.293584 seconds, 151104 images
Loaded: 0.000035 seconds
4723: 0.015386, 0.018308 avg loss, 0.001000 rate, 3.293902 seconds, 151136 images
Loaded: 0.000068 seconds
4724: 0.007318, 0.017209 avg loss, 0.001000 rate, 3.295394 seconds, 151168 images
Loaded: 0.000066 seconds
4725: 0.011181, 0.016606 avg loss, 0.001000 rate, 3.293364 seconds, 151200 images
Loaded: 0.000050 seconds
4726: 0.008530, 0.015799 avg loss, 0.001000 rate, 3.296285 seconds, 151232 images
Loaded: 0.000030 seconds
4727: 0.011498, 0.015369 avg loss, 0.001000 rate, 3.288164 seconds, 151264 images
Loaded: 0.000038 seconds
4728: 0.018212, 0.015653 avg loss, 0.001000 rate, 3.278705 seconds, 151296 images
Loaded: 0.000061 seconds
4729: 0.012259, 0.015314 avg loss, 0.001000 rate, 3.295674 seconds, 151328 images
Loaded: 0.000038 seconds
4730: 0.010144, 0.014797 avg loss, 0.001000 rate, 3.300560 seconds, 151360 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000065 seconds
4731: 0.029685, 0.016285 avg loss, 0.001000 rate, 1.793888 seconds, 151392 images
Loaded: 0.000061 seconds
4732: 0.029358, 0.017593 avg loss, 0.001000 rate, 1.857357 seconds, 151424 images
Loaded: 0.000049 seconds
4733: 0.036600, 0.019493 avg loss, 0.001000 rate, 1.858327 seconds, 151456 images
Loaded: 0.000064 seconds
4734: 0.019563, 0.019500 avg loss, 0.001000 rate, 1.874170 seconds, 151488 images
Loaded: 0.000051 seconds
4735: 0.024940, 0.020044 avg loss, 0.001000 rate, 1.865534 seconds, 151520 images
Loaded: 0.000033 seconds
4736: 0.018558, 0.019896 avg loss, 0.001000 rate, 1.861323 seconds, 151552 images
Loaded: 0.000062 seconds
4737: 0.038210, 0.021727 avg loss, 0.001000 rate, 1.865842 seconds, 151584 images
Loaded: 0.000041 seconds
4738: 0.026711, 0.022225 avg loss, 0.001000 rate, 1.865755 seconds, 151616 images
Loaded: 0.000038 seconds
4739: 0.015495, 0.021552 avg loss, 0.001000 rate, 1.865289 seconds, 151648 images
Loaded: 0.000039 seconds
4740: 0.020338, 0.021431 avg loss, 0.001000 rate, 1.860484 seconds, 151680 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000043 seconds
4741: 0.022481, 0.021536 avg loss, 0.001000 rate, 2.012835 seconds, 151712 images
Loaded: 0.000048 seconds
4742: 0.014265, 0.020809 avg loss, 0.001000 rate, 2.097345 seconds, 151744 images
Loaded: 0.000039 seconds
4743: 0.038285, 0.022557 avg loss, 0.001000 rate, 2.097920 seconds, 151776 images
Loaded: 0.000036 seconds
4744: 0.012406, 0.021541 avg loss, 0.001000 rate, 2.098388 seconds, 151808 images
Loaded: 0.000036 seconds
4745: 0.013997, 0.020787 avg loss, 0.001000 rate, 2.107028 seconds, 151840 images
Loaded: 0.000067 seconds
4746: 0.019384, 0.020647 avg loss, 0.001000 rate, 2.095077 seconds, 151872 images
Loaded: 0.000064 seconds
4747: 0.014867, 0.020069 avg loss, 0.001000 rate, 2.104619 seconds, 151904 images
Loaded: 0.000056 seconds
4748: 0.008242, 0.018886 avg loss, 0.001000 rate, 2.103233 seconds, 151936 images
Loaded: 0.000069 seconds
4749: 0.013503, 0.018348 avg loss, 0.001000 rate, 2.099749 seconds, 151968 images
Loaded: 0.000038 seconds
4750: 0.014233, 0.017936 avg loss, 0.001000 rate, 2.099776 seconds, 152000 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000042 seconds
4751: 0.011295, 0.017272 avg loss, 0.001000 rate, 2.626786 seconds, 152032 images
Loaded: 0.000071 seconds
4752: 0.017091, 0.017254 avg loss, 0.001000 rate, 2.727096 seconds, 152064 images
Loaded: 0.000053 seconds
4753: 0.027834, 0.018312 avg loss, 0.001000 rate, 2.716632 seconds, 152096 images
Loaded: 0.000037 seconds
4754: 0.011683, 0.017649 avg loss, 0.001000 rate, 2.716614 seconds, 152128 images
Loaded: 0.000066 seconds
4755: 0.010969, 0.016981 avg loss, 0.001000 rate, 2.716621 seconds, 152160 images
Loaded: 0.000044 seconds
4756: 0.015393, 0.016822 avg loss, 0.001000 rate, 2.724960 seconds, 152192 images
Loaded: 0.000041 seconds
4757: 0.010061, 0.016146 avg loss, 0.001000 rate, 2.725111 seconds, 152224 images
Loaded: 0.000043 seconds
4758: 0.014385, 0.015970 avg loss, 0.001000 rate, 2.728468 seconds, 152256 images
Loaded: 0.000044 seconds
4759: 0.017292, 0.016102 avg loss, 0.001000 rate, 2.733047 seconds, 152288 images
Loaded: 0.000047 seconds
4760: 0.011033, 0.015595 avg loss, 0.001000 rate, 2.735746 seconds, 152320 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.077306 seconds
4761: 0.012565, 0.015292 avg loss, 0.001000 rate, 2.028130 seconds, 152352 images
Loaded: 0.000078 seconds
4762: 0.011310, 0.014894 avg loss, 0.001000 rate, 2.112349 seconds, 152384 images
Loaded: 0.000045 seconds
4763: 0.010642, 0.014469 avg loss, 0.001000 rate, 2.111498 seconds, 152416 images
Loaded: 0.000069 seconds
4764: 0.015575, 0.014579 avg loss, 0.001000 rate, 2.109556 seconds, 152448 images
Loaded: 0.000082 seconds
4765: 0.012028, 0.014324 avg loss, 0.001000 rate, 2.115496 seconds, 152480 images
Loaded: 0.000042 seconds
4766: 0.017108, 0.014603 avg loss, 0.001000 rate, 2.109149 seconds, 152512 images
Loaded: 0.000038 seconds
4767: 0.014084, 0.014551 avg loss, 0.001000 rate, 2.103950 seconds, 152544 images
Loaded: 0.000053 seconds
4768: 0.012048, 0.014300 avg loss, 0.001000 rate, 2.113768 seconds, 152576 images
Loaded: 0.000042 seconds
4769: 0.008556, 0.013726 avg loss, 0.001000 rate, 2.106067 seconds, 152608 images
Loaded: 0.000043 seconds
4770: 0.030362, 0.015390 avg loss, 0.001000 rate, 2.107251 seconds, 152640 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000039 seconds
4771: 0.018764, 0.015727 avg loss, 0.001000 rate, 3.169482 seconds, 152672 images
Loaded: 0.000063 seconds
4772: 0.009921, 0.015146 avg loss, 0.001000 rate, 3.289473 seconds, 152704 images
Loaded: 0.000044 seconds
4773: 0.013784, 0.015010 avg loss, 0.001000 rate, 3.288559 seconds, 152736 images
Loaded: 0.000047 seconds
4774: 0.011826, 0.014692 avg loss, 0.001000 rate, 3.300953 seconds, 152768 images
Loaded: 0.000041 seconds
4775: 0.008303, 0.014053 avg loss, 0.001000 rate, 3.290723 seconds, 152800 images
Loaded: 0.000047 seconds
4776: 0.012566, 0.013904 avg loss, 0.001000 rate, 3.293787 seconds, 152832 images
Loaded: 0.000045 seconds
4777: 0.009034, 0.013417 avg loss, 0.001000 rate, 3.291643 seconds, 152864 images
Loaded: 0.000040 seconds
4778: 0.023496, 0.014425 avg loss, 0.001000 rate, 3.294765 seconds, 152896 images
Loaded: 0.000043 seconds
4779: 0.008914, 0.013874 avg loss, 0.001000 rate, 3.288091 seconds, 152928 images
Loaded: 0.000068 seconds
4780: 0.006729, 0.013159 avg loss, 0.001000 rate, 3.296685 seconds, 152960 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000043 seconds
4781: 0.014504, 0.013294 avg loss, 0.001000 rate, 4.365134 seconds, 152992 images
Loaded: 0.000044 seconds
4782: 0.026325, 0.014597 avg loss, 0.001000 rate, 4.508883 seconds, 153024 images
Loaded: 0.000066 seconds
4783: 0.013983, 0.014536 avg loss, 0.001000 rate, 4.525286 seconds, 153056 images
Loaded: 0.000067 seconds
4784: 0.014485, 0.014531 avg loss, 0.001000 rate, 4.513702 seconds, 153088 images
Loaded: 0.000042 seconds
4785: 0.013526, 0.014430 avg loss, 0.001000 rate, 4.511817 seconds, 153120 images
Loaded: 0.000060 seconds
4786: 0.017449, 0.014732 avg loss, 0.001000 rate, 4.533312 seconds, 153152 images
Loaded: 0.000046 seconds
4787: 0.011206, 0.014379 avg loss, 0.001000 rate, 4.516173 seconds, 153184 images
Loaded: 0.000044 seconds
4788: 0.012485, 0.014190 avg loss, 0.001000 rate, 4.515123 seconds, 153216 images
Loaded: 0.000040 seconds
4789: 0.010266, 0.013798 avg loss, 0.001000 rate, 4.513279 seconds, 153248 images
Loaded: 0.000042 seconds
4790: 0.010126, 0.013430 avg loss, 0.001000 rate, 4.514035 seconds, 153280 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000038 seconds
4791: 0.015156, 0.013603 avg loss, 0.001000 rate, 2.020337 seconds, 153312 images
Loaded: 0.000032 seconds
4792: 0.009613, 0.013204 avg loss, 0.001000 rate, 2.104487 seconds, 153344 images
Loaded: 0.000036 seconds
4793: 0.021995, 0.014083 avg loss, 0.001000 rate, 2.106719 seconds, 153376 images
Loaded: 0.000064 seconds
4794: 0.016686, 0.014343 avg loss, 0.001000 rate, 2.113602 seconds, 153408 images
Loaded: 0.000041 seconds
4795: 0.017979, 0.014707 avg loss, 0.001000 rate, 2.104715 seconds, 153440 images
Loaded: 0.000082 seconds
4796: 0.008406, 0.014077 avg loss, 0.001000 rate, 2.112383 seconds, 153472 images
Loaded: 0.000066 seconds
4797: 0.024958, 0.015165 avg loss, 0.001000 rate, 2.112179 seconds, 153504 images
Loaded: 0.000041 seconds
4798: 0.009897, 0.014638 avg loss, 0.001000 rate, 2.109117 seconds, 153536 images
Loaded: 0.000067 seconds
4799: 0.016874, 0.014862 avg loss, 0.001000 rate, 2.105064 seconds, 153568 images
Loaded: 0.000077 seconds
4800: 0.008582, 0.014234 avg loss, 0.000100 rate, 2.105123 seconds, 153600 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.295434 seconds
4801: 0.021568, 0.014967 avg loss, 0.000100 rate, 1.575581 seconds, 153632 images
Loaded: 0.000043 seconds
4802: 0.027872, 0.016258 avg loss, 0.000100 rate, 1.650286 seconds, 153664 images
Loaded: 0.000070 seconds
4803: 0.030772, 0.017709 avg loss, 0.000100 rate, 1.646016 seconds, 153696 images
Loaded: 0.000040 seconds
4804: 0.021900, 0.018128 avg loss, 0.000100 rate, 1.647312 seconds, 153728 images
Loaded: 0.000052 seconds
4805: 0.041422, 0.020458 avg loss, 0.000100 rate, 1.637196 seconds, 153760 images
Loaded: 0.000042 seconds
4806: 0.023623, 0.020774 avg loss, 0.000100 rate, 1.652405 seconds, 153792 images
Loaded: 0.000076 seconds
4807: 0.032030, 0.021900 avg loss, 0.000100 rate, 1.648166 seconds, 153824 images
Loaded: 0.000039 seconds
4808: 0.010123, 0.020722 avg loss, 0.000100 rate, 1.646150 seconds, 153856 images
Loaded: 0.000047 seconds
4809: 0.029239, 0.021574 avg loss, 0.000100 rate, 1.644510 seconds, 153888 images
Loaded: 0.000045 seconds
4810: 0.015514, 0.020968 avg loss, 0.000100 rate, 1.657991 seconds, 153920 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000042 seconds
4811: 0.024321, 0.021303 avg loss, 0.000100 rate, 3.182373 seconds, 153952 images
Loaded: 0.000044 seconds
4812: 0.009725, 0.020145 avg loss, 0.000100 rate, 3.289898 seconds, 153984 images
Loaded: 0.000066 seconds
4813: 0.010025, 0.019133 avg loss, 0.000100 rate, 3.293288 seconds, 154016 images
Loaded: 0.000042 seconds
4814: 0.009743, 0.018194 avg loss, 0.000100 rate, 3.287128 seconds, 154048 images
Loaded: 0.000041 seconds
4815: 0.016541, 0.018029 avg loss, 0.000100 rate, 3.301872 seconds, 154080 images
Loaded: 0.000041 seconds
4816: 0.009215, 0.017147 avg loss, 0.000100 rate, 3.290444 seconds, 154112 images
Loaded: 0.000038 seconds
4817: 0.011036, 0.016536 avg loss, 0.000100 rate, 3.293124 seconds, 154144 images
Loaded: 0.000069 seconds
4818: 0.009813, 0.015864 avg loss, 0.000100 rate, 3.289441 seconds, 154176 images
Loaded: 0.000040 seconds
4819: 0.012256, 0.015503 avg loss, 0.000100 rate, 3.285738 seconds, 154208 images
Loaded: 0.000050 seconds
4820: 0.009777, 0.014931 avg loss, 0.000100 rate, 3.297041 seconds, 154240 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.034532 seconds
4821: 0.017927, 0.015230 avg loss, 0.000100 rate, 2.618915 seconds, 154272 images
Loaded: 0.000044 seconds
4822: 0.009624, 0.014670 avg loss, 0.000100 rate, 2.724424 seconds, 154304 images
Loaded: 0.000070 seconds
4823: 0.009721, 0.014175 avg loss, 0.000100 rate, 2.724397 seconds, 154336 images
Loaded: 0.000045 seconds
4824: 0.009653, 0.013723 avg loss, 0.000100 rate, 2.719130 seconds, 154368 images
Loaded: 0.000067 seconds
4825: 0.007103, 0.013061 avg loss, 0.000100 rate, 2.717277 seconds, 154400 images
Loaded: 0.000056 seconds
4826: 0.010818, 0.012836 avg loss, 0.000100 rate, 2.720634 seconds, 154432 images
Loaded: 0.000044 seconds
4827: 0.007575, 0.012310 avg loss, 0.000100 rate, 2.723110 seconds, 154464 images
Loaded: 0.000044 seconds
4828: 0.006908, 0.011770 avg loss, 0.000100 rate, 2.727253 seconds, 154496 images
Loaded: 0.000038 seconds
4829: 0.018542, 0.012447 avg loss, 0.000100 rate, 2.721231 seconds, 154528 images
Loaded: 0.000054 seconds
4830: 0.008711, 0.012074 avg loss, 0.000100 rate, 2.728004 seconds, 154560 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.143741 seconds
4831: 0.007097, 0.011576 avg loss, 0.000100 rate, 2.386720 seconds, 154592 images
Loaded: 0.000040 seconds
4832: 0.006038, 0.011022 avg loss, 0.000100 rate, 2.488842 seconds, 154624 images
Loaded: 0.000052 seconds
4833: 0.012317, 0.011152 avg loss, 0.000100 rate, 2.483572 seconds, 154656 images
Loaded: 0.000082 seconds
4834: 0.010611, 0.011098 avg loss, 0.000100 rate, 2.490616 seconds, 154688 images
Loaded: 0.000045 seconds
4835: 0.017267, 0.011715 avg loss, 0.000100 rate, 2.482157 seconds, 154720 images
Loaded: 0.000045 seconds
4836: 0.008784, 0.011421 avg loss, 0.000100 rate, 2.482799 seconds, 154752 images
Loaded: 0.000047 seconds
4837: 0.015020, 0.011781 avg loss, 0.000100 rate, 2.480716 seconds, 154784 images
Loaded: 0.000043 seconds
4838: 0.009273, 0.011531 avg loss, 0.000100 rate, 2.484944 seconds, 154816 images
Loaded: 0.000085 seconds
4839: 0.007861, 0.011164 avg loss, 0.000100 rate, 2.490755 seconds, 154848 images
Loaded: 0.000067 seconds
4840: 0.007957, 0.010843 avg loss, 0.000100 rate, 2.486435 seconds, 154880 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000044 seconds
4841: 0.005225, 0.010281 avg loss, 0.000100 rate, 2.614844 seconds, 154912 images
Loaded: 0.000065 seconds
4842: 0.007055, 0.009959 avg loss, 0.000100 rate, 2.719841 seconds, 154944 images
Loaded: 0.000050 seconds
4843: 0.008718, 0.009835 avg loss, 0.000100 rate, 2.722033 seconds, 154976 images
Loaded: 0.000063 seconds
4844: 0.007652, 0.009616 avg loss, 0.000100 rate, 2.720405 seconds, 155008 images
Loaded: 0.000044 seconds
4845: 0.009371, 0.009592 avg loss, 0.000100 rate, 2.718714 seconds, 155040 images
Loaded: 0.000050 seconds
4846: 0.010058, 0.009638 avg loss, 0.000100 rate, 2.718943 seconds, 155072 images
Loaded: 0.000070 seconds
4847: 0.006722, 0.009347 avg loss, 0.000100 rate, 2.725226 seconds, 155104 images
Loaded: 0.000040 seconds
4848: 0.009596, 0.009372 avg loss, 0.000100 rate, 2.720625 seconds, 155136 images
Loaded: 0.000042 seconds
4849: 0.005244, 0.008959 avg loss, 0.000100 rate, 2.719983 seconds, 155168 images
Loaded: 0.000038 seconds
4850: 0.007442, 0.008807 avg loss, 0.000100 rate, 2.721007 seconds, 155200 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000049 seconds
4851: 0.006925, 0.008619 avg loss, 0.000100 rate, 4.009329 seconds, 155232 images
Loaded: 0.000067 seconds
4852: 0.006174, 0.008374 avg loss, 0.000100 rate, 4.167091 seconds, 155264 images
Loaded: 0.000043 seconds
4853: 0.005763, 0.008113 avg loss, 0.000100 rate, 4.160253 seconds, 155296 images
Loaded: 0.000042 seconds
4854: 0.008869, 0.008189 avg loss, 0.000100 rate, 4.158725 seconds, 155328 images
Loaded: 0.000041 seconds
4855: 0.004021, 0.007772 avg loss, 0.000100 rate, 4.164536 seconds, 155360 images
Loaded: 0.000044 seconds
4856: 0.007262, 0.007721 avg loss, 0.000100 rate, 4.165193 seconds, 155392 images
Loaded: 0.000064 seconds
4857: 0.009543, 0.007903 avg loss, 0.000100 rate, 4.167638 seconds, 155424 images
Loaded: 0.000046 seconds
4858: 0.007480, 0.007861 avg loss, 0.000100 rate, 4.161746 seconds, 155456 images
Loaded: 0.000045 seconds
4859: 0.005247, 0.007600 avg loss, 0.000100 rate, 4.171420 seconds, 155488 images
Loaded: 0.000042 seconds
4860: 0.006599, 0.007500 avg loss, 0.000100 rate, 4.165973 seconds, 155520 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.060986 seconds
4861: 0.010910, 0.007841 avg loss, 0.000100 rate, 2.932260 seconds, 155552 images
Loaded: 0.000057 seconds
4862: 0.009303, 0.007987 avg loss, 0.000100 rate, 3.044296 seconds, 155584 images
Loaded: 0.000050 seconds
4863: 0.011403, 0.008328 avg loss, 0.000100 rate, 3.040454 seconds, 155616 images
Loaded: 0.000056 seconds
4864: 0.004560, 0.007952 avg loss, 0.000100 rate, 3.040353 seconds, 155648 images
Loaded: 0.000072 seconds
4865: 0.007661, 0.007922 avg loss, 0.000100 rate, 3.045050 seconds, 155680 images
Loaded: 0.000044 seconds
4866: 0.007575, 0.007888 avg loss, 0.000100 rate, 3.036361 seconds, 155712 images
Loaded: 0.000041 seconds
4867: 0.005236, 0.007623 avg loss, 0.000100 rate, 3.050007 seconds, 155744 images
Loaded: 0.000044 seconds
4868: 0.006089, 0.007469 avg loss, 0.000100 rate, 3.047964 seconds, 155776 images
Loaded: 0.000045 seconds
4869: 0.004823, 0.007205 avg loss, 0.000100 rate, 3.048422 seconds, 155808 images
Loaded: 0.000066 seconds
4870: 0.005023, 0.006986 avg loss, 0.000100 rate, 3.037203 seconds, 155840 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000042 seconds
4871: 0.012843, 0.007572 avg loss, 0.000100 rate, 2.026233 seconds, 155872 images
Loaded: 0.000042 seconds
4872: 0.009467, 0.007762 avg loss, 0.000100 rate, 2.108544 seconds, 155904 images
Loaded: 0.000041 seconds
4873: 0.008969, 0.007882 avg loss, 0.000100 rate, 2.113851 seconds, 155936 images
Loaded: 0.000044 seconds
4874: 0.005400, 0.007634 avg loss, 0.000100 rate, 2.108341 seconds, 155968 images
Loaded: 0.000037 seconds
4875: 0.025427, 0.009413 avg loss, 0.000100 rate, 2.110470 seconds, 156000 images
Loaded: 0.000039 seconds
4876: 0.009184, 0.009390 avg loss, 0.000100 rate, 2.103523 seconds, 156032 images
Loaded: 0.000042 seconds
4877: 0.017803, 0.010232 avg loss, 0.000100 rate, 2.110119 seconds, 156064 images
Loaded: 0.000066 seconds
4878: 0.008930, 0.010102 avg loss, 0.000100 rate, 2.104303 seconds, 156096 images
Loaded: 0.000057 seconds
4879: 0.006795, 0.009771 avg loss, 0.000100 rate, 2.108602 seconds, 156128 images
Loaded: 0.000042 seconds
4880: 0.013066, 0.010100 avg loss, 0.000100 rate, 2.115263 seconds, 156160 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000044 seconds
4881: 0.007122, 0.009803 avg loss, 0.000100 rate, 4.016310 seconds, 156192 images
Loaded: 0.000071 seconds
4882: 0.007854, 0.009608 avg loss, 0.000100 rate, 4.173831 seconds, 156224 images
Loaded: 0.000044 seconds
4883: 0.007557, 0.009403 avg loss, 0.000100 rate, 4.168210 seconds, 156256 images
Loaded: 0.000042 seconds
4884: 0.006712, 0.009134 avg loss, 0.000100 rate, 4.164692 seconds, 156288 images
Loaded: 0.000041 seconds
4885: 0.028685, 0.011089 avg loss, 0.000100 rate, 4.167956 seconds, 156320 images
Loaded: 0.000044 seconds
4886: 0.011135, 0.011093 avg loss, 0.000100 rate, 4.188654 seconds, 156352 images
Loaded: 0.000073 seconds
4887: 0.011630, 0.011147 avg loss, 0.000100 rate, 4.185026 seconds, 156384 images
Loaded: 0.000034 seconds
4888: 0.012199, 0.011252 avg loss, 0.000100 rate, 4.174962 seconds, 156416 images
Loaded: 0.000040 seconds
4889: 0.010310, 0.011158 avg loss, 0.000100 rate, 4.172942 seconds, 156448 images
Loaded: 0.000043 seconds
4890: 0.007388, 0.010781 avg loss, 0.000100 rate, 4.182154 seconds, 156480 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000060 seconds
4891: 0.014859, 0.011189 avg loss, 0.000100 rate, 1.794493 seconds, 156512 images
Loaded: 0.000068 seconds
4892: 0.008208, 0.010891 avg loss, 0.000100 rate, 1.871215 seconds, 156544 images
Loaded: 0.000043 seconds
4893: 0.017451, 0.011547 avg loss, 0.000100 rate, 1.866420 seconds, 156576 images
Loaded: 0.000044 seconds
4894: 0.012543, 0.011646 avg loss, 0.000100 rate, 1.868819 seconds, 156608 images
Loaded: 0.000069 seconds
4895: 0.008921, 0.011374 avg loss, 0.000100 rate, 1.869198 seconds, 156640 images
Loaded: 0.000043 seconds
4896: 0.005552, 0.010792 avg loss, 0.000100 rate, 1.864681 seconds, 156672 images
Loaded: 0.000072 seconds
4897: 0.017668, 0.011479 avg loss, 0.000100 rate, 1.866954 seconds, 156704 images
Loaded: 0.000044 seconds
4898: 0.007397, 0.011071 avg loss, 0.000100 rate, 1.872741 seconds, 156736 images
Loaded: 0.000070 seconds
4899: 0.009965, 0.010960 avg loss, 0.000100 rate, 1.869073 seconds, 156768 images
Loaded: 0.000043 seconds
4900: 0.019623, 0.011827 avg loss, 0.000100 rate, 1.875489 seconds, 156800 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000066 seconds
4901: 0.015668, 0.012211 avg loss, 0.000100 rate, 2.632163 seconds, 156832 images
Loaded: 0.000074 seconds
4902: 0.004254, 0.011415 avg loss, 0.000100 rate, 2.716154 seconds, 156864 images
Loaded: 0.000066 seconds
4903: 0.005996, 0.010873 avg loss, 0.000100 rate, 2.717818 seconds, 156896 images
Loaded: 0.000044 seconds
4904: 0.007032, 0.010489 avg loss, 0.000100 rate, 2.727784 seconds, 156928 images
Loaded: 0.000056 seconds
4905: 0.006186, 0.010059 avg loss, 0.000100 rate, 2.730428 seconds, 156960 images
Loaded: 0.000041 seconds
4906: 0.005191, 0.009572 avg loss, 0.000100 rate, 2.725840 seconds, 156992 images
Loaded: 0.000040 seconds
4907: 0.006750, 0.009290 avg loss, 0.000100 rate, 2.727835 seconds, 157024 images
Loaded: 0.000040 seconds
4908: 0.028046, 0.011165 avg loss, 0.000100 rate, 2.721566 seconds, 157056 images
Loaded: 0.000047 seconds
4909: 0.003194, 0.010368 avg loss, 0.000100 rate, 2.723026 seconds, 157088 images
Loaded: 0.000054 seconds
4910: 0.005772, 0.009909 avg loss, 0.000100 rate, 2.719187 seconds, 157120 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.059740 seconds
4911: 0.007185, 0.009636 avg loss, 0.000100 rate, 2.388500 seconds, 157152 images
Loaded: 0.000073 seconds
4912: 0.006274, 0.009300 avg loss, 0.000100 rate, 2.482436 seconds, 157184 images
Loaded: 0.000080 seconds
4913: 0.008614, 0.009232 avg loss, 0.000100 rate, 2.491122 seconds, 157216 images
Loaded: 0.000039 seconds
4914: 0.012989, 0.009607 avg loss, 0.000100 rate, 2.488751 seconds, 157248 images
Loaded: 0.000041 seconds
4915: 0.006626, 0.009309 avg loss, 0.000100 rate, 2.483070 seconds, 157280 images
Loaded: 0.000084 seconds
4916: 0.006792, 0.009057 avg loss, 0.000100 rate, 2.486882 seconds, 157312 images
Loaded: 0.000036 seconds
4917: 0.004679, 0.008620 avg loss, 0.000100 rate, 2.484924 seconds, 157344 images
Loaded: 0.000039 seconds
4918: 0.006202, 0.008378 avg loss, 0.000100 rate, 2.484156 seconds, 157376 images
Loaded: 0.000051 seconds
4919: 0.007804, 0.008320 avg loss, 0.000100 rate, 2.481850 seconds, 157408 images
Loaded: 0.000064 seconds
4920: 0.009587, 0.008447 avg loss, 0.000100 rate, 2.482717 seconds, 157440 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.074267 seconds
4921: 0.012148, 0.008817 avg loss, 0.000100 rate, 2.029328 seconds, 157472 images
Loaded: 0.000082 seconds
4922: 0.006370, 0.008572 avg loss, 0.000100 rate, 2.110650 seconds, 157504 images
Loaded: 0.000066 seconds
4923: 0.016942, 0.009409 avg loss, 0.000100 rate, 2.111845 seconds, 157536 images
Loaded: 0.000037 seconds
4924: 0.014633, 0.009932 avg loss, 0.000100 rate, 2.106443 seconds, 157568 images
Loaded: 0.000072 seconds
4925: 0.012473, 0.010186 avg loss, 0.000100 rate, 2.118379 seconds, 157600 images
Loaded: 0.000042 seconds
4926: 0.008524, 0.010020 avg loss, 0.000100 rate, 2.108960 seconds, 157632 images
Loaded: 0.000044 seconds
4927: 0.006824, 0.009700 avg loss, 0.000100 rate, 2.112355 seconds, 157664 images
Loaded: 0.000059 seconds
4928: 0.007567, 0.009487 avg loss, 0.000100 rate, 2.104200 seconds, 157696 images
Loaded: 0.000065 seconds
4929: 0.007577, 0.009296 avg loss, 0.000100 rate, 2.104771 seconds, 157728 images
Loaded: 0.000042 seconds
4930: 0.007820, 0.009148 avg loss, 0.000100 rate, 2.102139 seconds, 157760 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.123851 seconds
4931: 0.011059, 0.009339 avg loss, 0.000100 rate, 1.797085 seconds, 157792 images
Loaded: 0.000039 seconds
4932: 0.018525, 0.010258 avg loss, 0.000100 rate, 1.861564 seconds, 157824 images
Loaded: 0.000057 seconds
4933: 0.009092, 0.010141 avg loss, 0.000100 rate, 1.874519 seconds, 157856 images
Loaded: 0.000042 seconds
4934: 0.008515, 0.009979 avg loss, 0.000100 rate, 1.867215 seconds, 157888 images
Loaded: 0.000041 seconds
4935: 0.012965, 0.010277 avg loss, 0.000100 rate, 1.865020 seconds, 157920 images
Loaded: 0.000039 seconds
4936: 0.012199, 0.010470 avg loss, 0.000100 rate, 1.866804 seconds, 157952 images
Loaded: 0.000039 seconds
4937: 0.007551, 0.010178 avg loss, 0.000100 rate, 1.860456 seconds, 157984 images
Loaded: 0.000037 seconds
4938: 0.008944, 0.010054 avg loss, 0.000100 rate, 1.866592 seconds, 158016 images
Loaded: 0.000040 seconds
4939: 0.008064, 0.009855 avg loss, 0.000100 rate, 1.868452 seconds, 158048 images
Loaded: 0.000046 seconds
4940: 0.008161, 0.009686 avg loss, 0.000100 rate, 1.864296 seconds, 158080 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000046 seconds
4941: 0.005722, 0.009289 avg loss, 0.000100 rate, 2.393628 seconds, 158112 images
Loaded: 0.000042 seconds
4942: 0.015708, 0.009931 avg loss, 0.000100 rate, 2.477826 seconds, 158144 images
Loaded: 0.000039 seconds
4943: 0.015223, 0.010460 avg loss, 0.000100 rate, 2.478451 seconds, 158176 images
Loaded: 0.000054 seconds
4944: 0.006997, 0.010114 avg loss, 0.000100 rate, 2.479634 seconds, 158208 images
Loaded: 0.000065 seconds
4945: 0.013818, 0.010484 avg loss, 0.000100 rate, 2.475168 seconds, 158240 images
Loaded: 0.000066 seconds
4946: 0.008453, 0.010281 avg loss, 0.000100 rate, 2.478943 seconds, 158272 images
Loaded: 0.000034 seconds
4947: 0.006463, 0.009900 avg loss, 0.000100 rate, 2.486519 seconds, 158304 images
Loaded: 0.000098 seconds
4948: 0.007507, 0.009660 avg loss, 0.000100 rate, 2.476084 seconds, 158336 images
Loaded: 0.000052 seconds
4949: 0.009840, 0.009678 avg loss, 0.000100 rate, 2.475568 seconds, 158368 images
Loaded: 0.000038 seconds
4950: 0.004775, 0.009188 avg loss, 0.000100 rate, 2.473293 seconds, 158400 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000041 seconds
4951: 0.005664, 0.008835 avg loss, 0.000100 rate, 2.924959 seconds, 158432 images
Loaded: 0.000086 seconds
4952: 0.006623, 0.008614 avg loss, 0.000100 rate, 3.033309 seconds, 158464 images
Loaded: 0.000043 seconds
4953: 0.008666, 0.008619 avg loss, 0.000100 rate, 3.033075 seconds, 158496 images
Loaded: 0.000062 seconds
4954: 0.018904, 0.009648 avg loss, 0.000100 rate, 3.039977 seconds, 158528 images
Loaded: 0.000081 seconds
4955: 0.010150, 0.009698 avg loss, 0.000100 rate, 3.032355 seconds, 158560 images
Loaded: 0.000040 seconds
4956: 0.010103, 0.009739 avg loss, 0.000100 rate, 3.032673 seconds, 158592 images
Loaded: 0.000041 seconds
4957: 0.004861, 0.009251 avg loss, 0.000100 rate, 3.033589 seconds, 158624 images
Loaded: 0.000045 seconds
4958: 0.008110, 0.009137 avg loss, 0.000100 rate, 3.035347 seconds, 158656 images
Loaded: 0.000053 seconds
4959: 0.011374, 0.009360 avg loss, 0.000100 rate, 3.034461 seconds, 158688 images
Loaded: 0.000036 seconds
4960: 0.006926, 0.009117 avg loss, 0.000100 rate, 3.040500 seconds, 158720 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000044 seconds
4961: 0.016353, 0.009840 avg loss, 0.000100 rate, 1.791867 seconds, 158752 images
Loaded: 0.000045 seconds
4962: 0.007413, 0.009598 avg loss, 0.000100 rate, 1.872851 seconds, 158784 images
Loaded: 0.000043 seconds
4963: 0.020900, 0.010728 avg loss, 0.000100 rate, 1.862641 seconds, 158816 images
Loaded: 0.000039 seconds
4964: 0.008051, 0.010460 avg loss, 0.000100 rate, 1.868372 seconds, 158848 images
Loaded: 0.000039 seconds
4965: 0.016799, 0.011094 avg loss, 0.000100 rate, 1.860330 seconds, 158880 images
Loaded: 0.000048 seconds
4966: 0.004732, 0.010458 avg loss, 0.000100 rate, 1.868949 seconds, 158912 images
Loaded: 0.000043 seconds
4967: 0.008171, 0.010229 avg loss, 0.000100 rate, 1.867945 seconds, 158944 images
Loaded: 0.000039 seconds
4968: 0.011356, 0.010342 avg loss, 0.000100 rate, 1.867551 seconds, 158976 images
Loaded: 0.000048 seconds
4969: 0.009441, 0.010252 avg loss, 0.000100 rate, 1.871327 seconds, 159008 images
Loaded: 0.000041 seconds
4970: 0.013245, 0.010551 avg loss, 0.000100 rate, 1.869428 seconds, 159040 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000040 seconds
4971: 0.005317, 0.010028 avg loss, 0.000100 rate, 2.929483 seconds, 159072 images
Loaded: 0.000061 seconds
4972: 0.013192, 0.010344 avg loss, 0.000100 rate, 3.045530 seconds, 159104 images
Loaded: 0.000065 seconds
4973: 0.010766, 0.010386 avg loss, 0.000100 rate, 3.043263 seconds, 159136 images
Loaded: 0.000093 seconds
4974: 0.021794, 0.011527 avg loss, 0.000100 rate, 3.043444 seconds, 159168 images
Loaded: 0.000057 seconds
4975: 0.005977, 0.010972 avg loss, 0.000100 rate, 3.039762 seconds, 159200 images
Loaded: 0.000043 seconds
4976: 0.005945, 0.010469 avg loss, 0.000100 rate, 3.051582 seconds, 159232 images
Loaded: 0.000072 seconds
4977: 0.007905, 0.010213 avg loss, 0.000100 rate, 3.051904 seconds, 159264 images
Loaded: 0.000065 seconds
4978: 0.018671, 0.011059 avg loss, 0.000100 rate, 3.047197 seconds, 159296 images
Loaded: 0.000041 seconds
4979: 0.009826, 0.010936 avg loss, 0.000100 rate, 3.046445 seconds, 159328 images
Loaded: 0.000042 seconds
4980: 0.006757, 0.010518 avg loss, 0.000100 rate, 3.051528 seconds, 159360 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.048243 seconds
4981: 0.015263, 0.010992 avg loss, 0.000100 rate, 1.800841 seconds, 159392 images
Loaded: 0.000042 seconds
4982: 0.009851, 0.010878 avg loss, 0.000100 rate, 1.872717 seconds, 159424 images
Loaded: 0.000042 seconds
4983: 0.008686, 0.010659 avg loss, 0.000100 rate, 1.875855 seconds, 159456 images
Loaded: 0.000042 seconds
4984: 0.013132, 0.010906 avg loss, 0.000100 rate, 1.878221 seconds, 159488 images
Loaded: 0.000041 seconds
4985: 0.021403, 0.011956 avg loss, 0.000100 rate, 1.870819 seconds, 159520 images
Loaded: 0.000033 seconds
4986: 0.007064, 0.011467 avg loss, 0.000100 rate, 1.871463 seconds, 159552 images
Loaded: 0.000044 seconds
4987: 0.013017, 0.011622 avg loss, 0.000100 rate, 1.869943 seconds, 159584 images
Loaded: 0.000057 seconds
4988: 0.019568, 0.012416 avg loss, 0.000100 rate, 1.869411 seconds, 159616 images
Loaded: 0.000050 seconds
4989: 0.015284, 0.012703 avg loss, 0.000100 rate, 1.867634 seconds, 159648 images
Loaded: 0.000083 seconds
4990: 0.008244, 0.012257 avg loss, 0.000100 rate, 1.869347 seconds, 159680 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.203121 seconds
4991: 0.007743, 0.011806 avg loss, 0.000100 rate, 1.788974 seconds, 159712 images
Loaded: 0.000067 seconds
4992: 0.006117, 0.011237 avg loss, 0.000100 rate, 1.860080 seconds, 159744 images
Loaded: 0.000040 seconds
4993: 0.008869, 0.011000 avg loss, 0.000100 rate, 1.862237 seconds, 159776 images
Loaded: 0.000047 seconds
4994: 0.006779, 0.010578 avg loss, 0.000100 rate, 1.869833 seconds, 159808 images
Loaded: 0.000060 seconds
4995: 0.018241, 0.011344 avg loss, 0.000100 rate, 1.863412 seconds, 159840 images
Loaded: 0.000035 seconds
4996: 0.006447, 0.010855 avg loss, 0.000100 rate, 1.859156 seconds, 159872 images
Loaded: 0.000042 seconds
4997: 0.010143, 0.010783 avg loss, 0.000100 rate, 1.872669 seconds, 159904 images
Loaded: 0.000071 seconds
4998: 0.010197, 0.010725 avg loss, 0.000100 rate, 1.870458 seconds, 159936 images
Loaded: 0.000079 seconds
4999: 0.011774, 0.010830 avg loss, 0.000100 rate, 1.865100 seconds, 159968 images
Loaded: 0.000036 seconds
5000: 0.011438, 0.010890 avg loss, 0.000100 rate, 1.863660 seconds, 160000 images
Saving weights to backup/custom-train-yolo_5000.weights
Saving weights to backup/custom-train-yolo_last.weights
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5001: 0.009035, 0.010705 avg loss, 0.000100 rate, 2.644133 seconds, 160032 images
Loaded: 0.000043 seconds
5002: 0.008932, 0.010528 avg loss, 0.000100 rate, 2.717371 seconds, 160064 images
Loaded: 0.000042 seconds
5003: 0.005788, 0.010054 avg loss, 0.000100 rate, 2.716518 seconds, 160096 images
Loaded: 0.000034 seconds
5004: 0.006983, 0.009747 avg loss, 0.000100 rate, 2.718708 seconds, 160128 images
Loaded: 0.000043 seconds
5005: 0.005928, 0.009365 avg loss, 0.000100 rate, 2.716268 seconds, 160160 images
Loaded: 0.000057 seconds
5006: 0.005420, 0.008970 avg loss, 0.000100 rate, 2.719257 seconds, 160192 images
Loaded: 0.000036 seconds
5007: 0.007809, 0.008854 avg loss, 0.000100 rate, 2.716557 seconds, 160224 images
Loaded: 0.000052 seconds
5008: 0.006836, 0.008652 avg loss, 0.000100 rate, 2.734574 seconds, 160256 images
Loaded: 0.000046 seconds
5009: 0.015225, 0.009309 avg loss, 0.000100 rate, 2.726177 seconds, 160288 images
Loaded: 0.000070 seconds
5010: 0.005534, 0.008932 avg loss, 0.000100 rate, 2.731062 seconds, 160320 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000045 seconds
5011: 0.007593, 0.008798 avg loss, 0.000100 rate, 3.174321 seconds, 160352 images
Loaded: 0.000052 seconds
5012: 0.010073, 0.008926 avg loss, 0.000100 rate, 3.294355 seconds, 160384 images
Loaded: 0.000070 seconds
5013: 0.005071, 0.008540 avg loss, 0.000100 rate, 3.300724 seconds, 160416 images
Loaded: 0.000041 seconds
5014: 0.006504, 0.008336 avg loss, 0.000100 rate, 3.292372 seconds, 160448 images
Loaded: 0.000045 seconds
5015: 0.004957, 0.007999 avg loss, 0.000100 rate, 3.305334 seconds, 160480 images
Loaded: 0.000041 seconds
5016: 0.009514, 0.008150 avg loss, 0.000100 rate, 3.298304 seconds, 160512 images
Loaded: 0.000056 seconds
5017: 0.006244, 0.007959 avg loss, 0.000100 rate, 3.290856 seconds, 160544 images
Loaded: 0.000039 seconds
5018: 0.004314, 0.007595 avg loss, 0.000100 rate, 3.298679 seconds, 160576 images
Loaded: 0.000060 seconds
5019: 0.006671, 0.007502 avg loss, 0.000100 rate, 3.293303 seconds, 160608 images
Loaded: 0.000043 seconds
5020: 0.005144, 0.007267 avg loss, 0.000100 rate, 3.289003 seconds, 160640 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.054438 seconds
5021: 0.004327, 0.006973 avg loss, 0.000100 rate, 3.165155 seconds, 160672 images
Loaded: 0.000044 seconds
5022: 0.011534, 0.007429 avg loss, 0.000100 rate, 3.286630 seconds, 160704 images
Loaded: 0.000042 seconds
5023: 0.007351, 0.007421 avg loss, 0.000100 rate, 3.292587 seconds, 160736 images
Loaded: 0.000035 seconds
5024: 0.005824, 0.007261 avg loss, 0.000100 rate, 3.284465 seconds, 160768 images
Loaded: 0.000088 seconds
5025: 0.007425, 0.007278 avg loss, 0.000100 rate, 3.294223 seconds, 160800 images
Loaded: 0.000042 seconds
5026: 0.005414, 0.007091 avg loss, 0.000100 rate, 3.283979 seconds, 160832 images
Loaded: 0.000042 seconds
5027: 0.004542, 0.006836 avg loss, 0.000100 rate, 3.292580 seconds, 160864 images
Loaded: 0.000044 seconds
5028: 0.003874, 0.006540 avg loss, 0.000100 rate, 3.277415 seconds, 160896 images
Loaded: 0.000070 seconds
5029: 0.004567, 0.006343 avg loss, 0.000100 rate, 3.295846 seconds, 160928 images
Loaded: 0.000074 seconds
5030: 0.005775, 0.006286 avg loss, 0.000100 rate, 3.292387 seconds, 160960 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.039401 seconds
5031: 0.008295, 0.006487 avg loss, 0.000100 rate, 2.024529 seconds, 160992 images
Loaded: 0.000042 seconds
5032: 0.010532, 0.006892 avg loss, 0.000100 rate, 2.105540 seconds, 161024 images
Loaded: 0.000041 seconds
5033: 0.007674, 0.006970 avg loss, 0.000100 rate, 2.106612 seconds, 161056 images
Loaded: 0.000046 seconds
5034: 0.005639, 0.006837 avg loss, 0.000100 rate, 2.104897 seconds, 161088 images
Loaded: 0.000068 seconds
5035: 0.007712, 0.006924 avg loss, 0.000100 rate, 2.103749 seconds, 161120 images
Loaded: 0.000067 seconds
5036: 0.006553, 0.006887 avg loss, 0.000100 rate, 2.107874 seconds, 161152 images
Loaded: 0.000052 seconds
5037: 0.009655, 0.007164 avg loss, 0.000100 rate, 2.108354 seconds, 161184 images
Loaded: 0.000042 seconds
5038: 0.006198, 0.007067 avg loss, 0.000100 rate, 2.101767 seconds, 161216 images
Loaded: 0.000039 seconds
5039: 0.006542, 0.007015 avg loss, 0.000100 rate, 2.116160 seconds, 161248 images
Loaded: 0.000040 seconds
5040: 0.006575, 0.006971 avg loss, 0.000100 rate, 2.109603 seconds, 161280 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.064892 seconds
5041: 0.007760, 0.007050 avg loss, 0.000100 rate, 2.023400 seconds, 161312 images
Loaded: 0.000045 seconds
5042: 0.006147, 0.006959 avg loss, 0.000100 rate, 2.103724 seconds, 161344 images
Loaded: 0.000043 seconds
5043: 0.005736, 0.006837 avg loss, 0.000100 rate, 2.106499 seconds, 161376 images
Loaded: 0.000041 seconds
5044: 0.005946, 0.006748 avg loss, 0.000100 rate, 2.105365 seconds, 161408 images
Loaded: 0.000079 seconds
5045: 0.008645, 0.006938 avg loss, 0.000100 rate, 2.104403 seconds, 161440 images
Loaded: 0.000055 seconds
5046: 0.005174, 0.006761 avg loss, 0.000100 rate, 2.103297 seconds, 161472 images
Loaded: 0.000039 seconds
5047: 0.008054, 0.006891 avg loss, 0.000100 rate, 2.098027 seconds, 161504 images
Loaded: 0.000047 seconds
5048: 0.027915, 0.008993 avg loss, 0.000100 rate, 2.107574 seconds, 161536 images
Loaded: 0.000044 seconds
5049: 0.008078, 0.008902 avg loss, 0.000100 rate, 2.105723 seconds, 161568 images
Loaded: 0.000040 seconds
5050: 0.004226, 0.008434 avg loss, 0.000100 rate, 2.111647 seconds, 161600 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5051: 0.006885, 0.008279 avg loss, 0.000100 rate, 2.922200 seconds, 161632 images
Loaded: 0.000043 seconds
5052: 0.006110, 0.008062 avg loss, 0.000100 rate, 3.033848 seconds, 161664 images
Loaded: 0.000044 seconds
5053: 0.006069, 0.007863 avg loss, 0.000100 rate, 3.042845 seconds, 161696 images
Loaded: 0.000042 seconds
5054: 0.004944, 0.007571 avg loss, 0.000100 rate, 3.032392 seconds, 161728 images
Loaded: 0.000083 seconds
5055: 0.006407, 0.007455 avg loss, 0.000100 rate, 3.039060 seconds, 161760 images
Loaded: 0.000065 seconds
5056: 0.006828, 0.007392 avg loss, 0.000100 rate, 3.039691 seconds, 161792 images
Loaded: 0.000085 seconds
5057: 0.004710, 0.007124 avg loss, 0.000100 rate, 3.033313 seconds, 161824 images
Loaded: 0.000040 seconds
5058: 0.014989, 0.007910 avg loss, 0.000100 rate, 3.040359 seconds, 161856 images
Loaded: 0.000065 seconds
5059: 0.006734, 0.007793 avg loss, 0.000100 rate, 3.048813 seconds, 161888 images
Loaded: 0.000068 seconds
5060: 0.005850, 0.007598 avg loss, 0.000100 rate, 3.029371 seconds, 161920 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.055045 seconds
5061: 0.006818, 0.007520 avg loss, 0.000100 rate, 2.390857 seconds, 161952 images
Loaded: 0.000041 seconds
5062: 0.005831, 0.007351 avg loss, 0.000100 rate, 2.475272 seconds, 161984 images
Loaded: 0.000040 seconds
5063: 0.009845, 0.007601 avg loss, 0.000100 rate, 2.485718 seconds, 162016 images
Loaded: 0.000042 seconds
5064: 0.007859, 0.007627 avg loss, 0.000100 rate, 2.480485 seconds, 162048 images
Loaded: 0.000037 seconds
5065: 0.004025, 0.007266 avg loss, 0.000100 rate, 2.476427 seconds, 162080 images
Loaded: 0.000043 seconds
5066: 0.006709, 0.007211 avg loss, 0.000100 rate, 2.481816 seconds, 162112 images
Loaded: 0.000052 seconds
5067: 0.006554, 0.007145 avg loss, 0.000100 rate, 2.481628 seconds, 162144 images
Loaded: 0.000047 seconds
5068: 0.003271, 0.006758 avg loss, 0.000100 rate, 2.478395 seconds, 162176 images
Loaded: 0.000043 seconds
5069: 0.007292, 0.006811 avg loss, 0.000100 rate, 2.479087 seconds, 162208 images
Loaded: 0.000042 seconds
5070: 0.005981, 0.006728 avg loss, 0.000100 rate, 2.484100 seconds, 162240 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000039 seconds
5071: 0.008944, 0.006950 avg loss, 0.000100 rate, 4.374516 seconds, 162272 images
Loaded: 0.000040 seconds
5072: 0.011535, 0.007408 avg loss, 0.000100 rate, 4.504816 seconds, 162304 images
Loaded: 0.000061 seconds
5073: 0.005766, 0.007244 avg loss, 0.000100 rate, 4.499410 seconds, 162336 images
Loaded: 0.000042 seconds
5074: 0.009476, 0.007467 avg loss, 0.000100 rate, 4.510350 seconds, 162368 images
Loaded: 0.000039 seconds
5075: 0.007318, 0.007452 avg loss, 0.000100 rate, 4.500457 seconds, 162400 images
Loaded: 0.000078 seconds
5076: 0.008618, 0.007569 avg loss, 0.000100 rate, 4.517183 seconds, 162432 images
Loaded: 0.000070 seconds
5077: 0.009288, 0.007741 avg loss, 0.000100 rate, 4.508322 seconds, 162464 images
Loaded: 0.000084 seconds
5078: 0.006112, 0.007578 avg loss, 0.000100 rate, 4.515522 seconds, 162496 images
Loaded: 0.000047 seconds
5079: 0.017182, 0.008538 avg loss, 0.000100 rate, 4.521163 seconds, 162528 images
Loaded: 0.000047 seconds
5080: 0.006822, 0.008367 avg loss, 0.000100 rate, 4.536676 seconds, 162560 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000037 seconds
5081: 0.008071, 0.008337 avg loss, 0.000100 rate, 2.926284 seconds, 162592 images
Loaded: 0.000040 seconds
5082: 0.006525, 0.008156 avg loss, 0.000100 rate, 3.038748 seconds, 162624 images
Loaded: 0.000080 seconds
5083: 0.004974, 0.007838 avg loss, 0.000100 rate, 3.035182 seconds, 162656 images
Loaded: 0.000041 seconds
5084: 0.004764, 0.007530 avg loss, 0.000100 rate, 3.043459 seconds, 162688 images
Loaded: 0.000040 seconds
5085: 0.006642, 0.007442 avg loss, 0.000100 rate, 3.040124 seconds, 162720 images
Loaded: 0.000037 seconds
5086: 0.009086, 0.007606 avg loss, 0.000100 rate, 3.042274 seconds, 162752 images
Loaded: 0.000043 seconds
5087: 0.005933, 0.007439 avg loss, 0.000100 rate, 3.034099 seconds, 162784 images
Loaded: 0.000065 seconds
5088: 0.003996, 0.007094 avg loss, 0.000100 rate, 3.032421 seconds, 162816 images
Loaded: 0.000043 seconds
5089: 0.005165, 0.006901 avg loss, 0.000100 rate, 3.035028 seconds, 162848 images
Loaded: 0.000043 seconds
5090: 0.011170, 0.007328 avg loss, 0.000100 rate, 3.041500 seconds, 162880 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5091: 0.011719, 0.007767 avg loss, 0.000100 rate, 1.784550 seconds, 162912 images
Loaded: 0.000045 seconds
5092: 0.015078, 0.008498 avg loss, 0.000100 rate, 1.871836 seconds, 162944 images
Loaded: 0.000041 seconds
5093: 0.015795, 0.009228 avg loss, 0.000100 rate, 1.859770 seconds, 162976 images
Loaded: 0.000042 seconds
5094: 0.009796, 0.009285 avg loss, 0.000100 rate, 1.863872 seconds, 163008 images
Loaded: 0.000057 seconds
5095: 0.027634, 0.011120 avg loss, 0.000100 rate, 1.867192 seconds, 163040 images
Loaded: 0.000067 seconds
5096: 0.008807, 0.010889 avg loss, 0.000100 rate, 1.865435 seconds, 163072 images
Loaded: 0.000043 seconds
5097: 0.012734, 0.011073 avg loss, 0.000100 rate, 1.865039 seconds, 163104 images
Loaded: 0.000040 seconds
5098: 0.007588, 0.010725 avg loss, 0.000100 rate, 1.865508 seconds, 163136 images
Loaded: 0.000045 seconds
5099: 0.008115, 0.010464 avg loss, 0.000100 rate, 1.865716 seconds, 163168 images
Loaded: 0.000036 seconds
5100: 0.010720, 0.010489 avg loss, 0.000100 rate, 1.861648 seconds, 163200 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.223147 seconds
5101: 0.011389, 0.010579 avg loss, 0.000100 rate, 1.572362 seconds, 163232 images
Loaded: 0.000040 seconds
5102: 0.007015, 0.010223 avg loss, 0.000100 rate, 1.638366 seconds, 163264 images
Loaded: 0.000048 seconds
5103: 0.025040, 0.011705 avg loss, 0.000100 rate, 1.639497 seconds, 163296 images
Loaded: 0.000058 seconds
5104: 0.013876, 0.011922 avg loss, 0.000100 rate, 1.639910 seconds, 163328 images
Loaded: 0.000040 seconds
5105: 0.030888, 0.013818 avg loss, 0.000100 rate, 1.644983 seconds, 163360 images
Loaded: 0.000072 seconds
5106: 0.018792, 0.014316 avg loss, 0.000100 rate, 1.638221 seconds, 163392 images
Loaded: 0.000039 seconds
5107: 0.013320, 0.014216 avg loss, 0.000100 rate, 1.640663 seconds, 163424 images
Loaded: 0.000042 seconds
5108: 0.018555, 0.014650 avg loss, 0.000100 rate, 1.639434 seconds, 163456 images
Loaded: 0.000057 seconds
5109: 0.009300, 0.014115 avg loss, 0.000100 rate, 1.649142 seconds, 163488 images
Loaded: 0.000052 seconds
5110: 0.048505, 0.017554 avg loss, 0.000100 rate, 1.638502 seconds, 163520 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5111: 0.009180, 0.016717 avg loss, 0.000100 rate, 4.013055 seconds, 163552 images
Loaded: 0.000039 seconds
5112: 0.006638, 0.015709 avg loss, 0.000100 rate, 4.150281 seconds, 163584 images
Loaded: 0.000077 seconds
5113: 0.008590, 0.014997 avg loss, 0.000100 rate, 4.156958 seconds, 163616 images
Loaded: 0.000064 seconds
5114: 0.007779, 0.014275 avg loss, 0.000100 rate, 4.168051 seconds, 163648 images
Loaded: 0.000034 seconds
5115: 0.006112, 0.013459 avg loss, 0.000100 rate, 4.158345 seconds, 163680 images
Loaded: 0.000043 seconds
5116: 0.010749, 0.013188 avg loss, 0.000100 rate, 4.163183 seconds, 163712 images
Loaded: 0.000063 seconds
5117: 0.022484, 0.014117 avg loss, 0.000100 rate, 4.170468 seconds, 163744 images
Loaded: 0.000052 seconds
5118: 0.011387, 0.013844 avg loss, 0.000100 rate, 4.166426 seconds, 163776 images
Loaded: 0.000067 seconds
5119: 0.023731, 0.014833 avg loss, 0.000100 rate, 4.165575 seconds, 163808 images
Loaded: 0.000042 seconds
5120: 0.004472, 0.013797 avg loss, 0.000100 rate, 4.166473 seconds, 163840 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.064420 seconds
5121: 0.007011, 0.013118 avg loss, 0.000100 rate, 2.385012 seconds, 163872 images
Loaded: 0.000116 seconds
5122: 0.006110, 0.012418 avg loss, 0.000100 rate, 2.474501 seconds, 163904 images
Loaded: 0.000059 seconds
5123: 0.012288, 0.012405 avg loss, 0.000100 rate, 2.478517 seconds, 163936 images
Loaded: 0.000069 seconds
5124: 0.015127, 0.012677 avg loss, 0.000100 rate, 2.470212 seconds, 163968 images
Loaded: 0.000068 seconds
5125: 0.012252, 0.012634 avg loss, 0.000100 rate, 2.480347 seconds, 164000 images
Loaded: 0.000035 seconds
5126: 0.016549, 0.013026 avg loss, 0.000100 rate, 2.473114 seconds, 164032 images
Loaded: 0.000042 seconds
5127: 0.006440, 0.012367 avg loss, 0.000100 rate, 2.470217 seconds, 164064 images
Loaded: 0.000037 seconds
5128: 0.007281, 0.011859 avg loss, 0.000100 rate, 2.472887 seconds, 164096 images
Loaded: 0.000051 seconds
5129: 0.012907, 0.011963 avg loss, 0.000100 rate, 2.477269 seconds, 164128 images
Loaded: 0.000037 seconds
5130: 0.008694, 0.011636 avg loss, 0.000100 rate, 2.468843 seconds, 164160 images
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000062 seconds
5131: 0.016916, 0.012164 avg loss, 0.000100 rate, 1.579870 seconds, 164192 images
Loaded: 0.000074 seconds
5132: 0.011068, 0.012055 avg loss, 0.000100 rate, 1.636277 seconds, 164224 images
Loaded: 0.000041 seconds
5133: 0.010762, 0.011925 avg loss, 0.000100 rate, 1.644432 seconds, 164256 images
Loaded: 0.000070 seconds
5134: 0.017991, 0.012532 avg loss, 0.000100 rate, 1.640258 seconds, 164288 images
Loaded: 0.000069 seconds
5135: 0.011323, 0.012411 avg loss, 0.000100 rate, 1.644270 seconds, 164320 images
Loaded: 0.000065 seconds
5136: 0.027913, 0.013961 avg loss, 0.000100 rate, 1.635687 seconds, 164352 images
Loaded: 0.000074 seconds
5137: 0.015102, 0.014075 avg loss, 0.000100 rate, 1.651892 seconds, 164384 images
Loaded: 0.000075 seconds
5138: 0.023598, 0.015028 avg loss, 0.000100 rate, 1.634946 seconds, 164416 images
Loaded: 0.000040 seconds
5139: 0.035372, 0.017062 avg loss, 0.000100 rate, 1.647813 seconds, 164448 images
Loaded: 0.000041 seconds
5140: 0.011752, 0.016531 avg loss, 0.000100 rate, 1.644461 seconds, 164480 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000033 seconds
5141: 0.007682, 0.015646 avg loss, 0.000100 rate, 2.380312 seconds, 164512 images
Loaded: 0.000069 seconds
5142: 0.007681, 0.014850 avg loss, 0.000100 rate, 2.477404 seconds, 164544 images
Loaded: 0.000040 seconds
5143: 0.004054, 0.013770 avg loss, 0.000100 rate, 2.470117 seconds, 164576 images
Loaded: 0.000073 seconds
5144: 0.005393, 0.012932 avg loss, 0.000100 rate, 2.473922 seconds, 164608 images
Loaded: 0.000041 seconds
5145: 0.007417, 0.012381 avg loss, 0.000100 rate, 2.476115 seconds, 164640 images
Loaded: 0.000041 seconds
5146: 0.010377, 0.012180 avg loss, 0.000100 rate, 2.470557 seconds, 164672 images
Loaded: 0.000072 seconds
5147: 0.008155, 0.011778 avg loss, 0.000100 rate, 2.482115 seconds, 164704 images
Loaded: 0.000074 seconds
5148: 0.006512, 0.011251 avg loss, 0.000100 rate, 2.468780 seconds, 164736 images
Loaded: 0.000066 seconds
5149: 0.006015, 0.010728 avg loss, 0.000100 rate, 2.478588 seconds, 164768 images
Loaded: 0.000065 seconds
5150: 0.006368, 0.010292 avg loss, 0.000100 rate, 2.476680 seconds, 164800 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000037 seconds
5151: 0.005662, 0.009829 avg loss, 0.000100 rate, 4.342996 seconds, 164832 images
Loaded: 0.000066 seconds
5152: 0.007773, 0.009623 avg loss, 0.000100 rate, 4.502471 seconds, 164864 images
Loaded: 0.000060 seconds
5153: 0.009147, 0.009575 avg loss, 0.000100 rate, 4.502118 seconds, 164896 images
Loaded: 0.000039 seconds
5154: 0.008999, 0.009518 avg loss, 0.000100 rate, 4.502262 seconds, 164928 images
Loaded: 0.000054 seconds
5155: 0.006666, 0.009233 avg loss, 0.000100 rate, 4.497398 seconds, 164960 images
Loaded: 0.000074 seconds
5156: 0.015485, 0.009858 avg loss, 0.000100 rate, 4.511370 seconds, 164992 images
Loaded: 0.000046 seconds
5157: 0.007088, 0.009581 avg loss, 0.000100 rate, 4.496966 seconds, 165024 images
Loaded: 0.000041 seconds
5158: 0.006253, 0.009248 avg loss, 0.000100 rate, 4.513711 seconds, 165056 images
Loaded: 0.000045 seconds
5159: 0.008496, 0.009173 avg loss, 0.000100 rate, 4.514357 seconds, 165088 images
Loaded: 0.000064 seconds
5160: 0.006059, 0.008862 avg loss, 0.000100 rate, 4.514635 seconds, 165120 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.057580 seconds
5161: 0.005894, 0.008565 avg loss, 0.000100 rate, 4.350122 seconds, 165152 images
Loaded: 0.000040 seconds
5162: 0.007546, 0.008463 avg loss, 0.000100 rate, 4.518444 seconds, 165184 images
Loaded: 0.000042 seconds
5163: 0.009527, 0.008569 avg loss, 0.000100 rate, 4.502995 seconds, 165216 images
Loaded: 0.000041 seconds
5164: 0.005147, 0.008227 avg loss, 0.000100 rate, 4.530501 seconds, 165248 images
Loaded: 0.000042 seconds
5165: 0.005457, 0.007950 avg loss, 0.000100 rate, 4.499946 seconds, 165280 images
Loaded: 0.000043 seconds
5166: 0.007292, 0.007884 avg loss, 0.000100 rate, 4.505175 seconds, 165312 images
Loaded: 0.000069 seconds
5167: 0.006633, 0.007759 avg loss, 0.000100 rate, 4.497275 seconds, 165344 images
Loaded: 0.000066 seconds
5168: 0.007268, 0.007710 avg loss, 0.000100 rate, 4.522577 seconds, 165376 images
Loaded: 0.000068 seconds
5169: 0.006989, 0.007638 avg loss, 0.000100 rate, 4.509789 seconds, 165408 images
Loaded: 0.000053 seconds
5170: 0.006999, 0.007574 avg loss, 0.000100 rate, 4.523139 seconds, 165440 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000044 seconds
5171: 0.008875, 0.007704 avg loss, 0.000100 rate, 2.031479 seconds, 165472 images
Loaded: 0.000039 seconds
5172: 0.007630, 0.007697 avg loss, 0.000100 rate, 2.101497 seconds, 165504 images
Loaded: 0.000044 seconds
5173: 0.007918, 0.007719 avg loss, 0.000100 rate, 2.112248 seconds, 165536 images
Loaded: 0.000043 seconds
5174: 0.008882, 0.007835 avg loss, 0.000100 rate, 2.105169 seconds, 165568 images
Loaded: 0.000041 seconds
5175: 0.008631, 0.007915 avg loss, 0.000100 rate, 2.107011 seconds, 165600 images
Loaded: 0.000041 seconds
5176: 0.005655, 0.007689 avg loss, 0.000100 rate, 2.110476 seconds, 165632 images
Loaded: 0.000058 seconds
5177: 0.014652, 0.008385 avg loss, 0.000100 rate, 2.123030 seconds, 165664 images
Loaded: 0.000034 seconds
5178: 0.020525, 0.009599 avg loss, 0.000100 rate, 2.102063 seconds, 165696 images
Loaded: 0.000058 seconds
5179: 0.006815, 0.009321 avg loss, 0.000100 rate, 2.110669 seconds, 165728 images
Loaded: 0.000037 seconds
5180: 0.008854, 0.009274 avg loss, 0.000100 rate, 2.109897 seconds, 165760 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000045 seconds
5181: 0.006680, 0.009014 avg loss, 0.000100 rate, 2.379000 seconds, 165792 images
Loaded: 0.000044 seconds
5182: 0.006683, 0.008781 avg loss, 0.000100 rate, 2.481535 seconds, 165824 images
Loaded: 0.000043 seconds
5183: 0.011548, 0.009058 avg loss, 0.000100 rate, 2.484325 seconds, 165856 images
Loaded: 0.000056 seconds
5184: 0.008682, 0.009020 avg loss, 0.000100 rate, 2.498957 seconds, 165888 images
Loaded: 0.000069 seconds
5185: 0.008801, 0.008998 avg loss, 0.000100 rate, 2.484300 seconds, 165920 images
Loaded: 0.000076 seconds
5186: 0.005926, 0.008691 avg loss, 0.000100 rate, 2.486812 seconds, 165952 images
Loaded: 0.000068 seconds
5187: 0.011421, 0.008964 avg loss, 0.000100 rate, 2.487547 seconds, 165984 images
Loaded: 0.000061 seconds
5188: 0.006983, 0.008766 avg loss, 0.000100 rate, 2.486389 seconds, 166016 images
Loaded: 0.000048 seconds
5189: 0.008430, 0.008732 avg loss, 0.000100 rate, 2.481893 seconds, 166048 images
Loaded: 0.000039 seconds
5190: 0.019148, 0.009774 avg loss, 0.000100 rate, 2.486345 seconds, 166080 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.087158 seconds
5191: 0.009660, 0.009763 avg loss, 0.000100 rate, 2.035449 seconds, 166112 images
Loaded: 0.000048 seconds
5192: 0.005470, 0.009333 avg loss, 0.000100 rate, 2.113633 seconds, 166144 images
Loaded: 0.000043 seconds
5193: 0.004601, 0.008860 avg loss, 0.000100 rate, 2.106132 seconds, 166176 images
Loaded: 0.000036 seconds
5194: 0.006231, 0.008597 avg loss, 0.000100 rate, 2.111805 seconds, 166208 images
Loaded: 0.000067 seconds
5195: 0.005996, 0.008337 avg loss, 0.000100 rate, 2.114505 seconds, 166240 images
Loaded: 0.000039 seconds
5196: 0.009635, 0.008467 avg loss, 0.000100 rate, 2.127872 seconds, 166272 images
Loaded: 0.000043 seconds
5197: 0.005333, 0.008154 avg loss, 0.000100 rate, 2.101651 seconds, 166304 images
Loaded: 0.000067 seconds
5198: 0.009051, 0.008243 avg loss, 0.000100 rate, 2.115023 seconds, 166336 images
Loaded: 0.000042 seconds
5199: 0.013438, 0.008763 avg loss, 0.000100 rate, 2.118236 seconds, 166368 images
Loaded: 0.000043 seconds
5200: 0.006221, 0.008509 avg loss, 0.000100 rate, 2.113420 seconds, 166400 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5201: 0.004841, 0.008142 avg loss, 0.000100 rate, 2.647246 seconds, 166432 images
Loaded: 0.000082 seconds
5202: 0.007827, 0.008110 avg loss, 0.000100 rate, 2.724631 seconds, 166464 images
Loaded: 0.000037 seconds
5203: 0.007017, 0.008001 avg loss, 0.000100 rate, 2.719417 seconds, 166496 images
Loaded: 0.000036 seconds
5204: 0.004752, 0.007676 avg loss, 0.000100 rate, 2.714566 seconds, 166528 images
Loaded: 0.000042 seconds
5205: 0.004797, 0.007388 avg loss, 0.000100 rate, 2.729412 seconds, 166560 images
Loaded: 0.000043 seconds
5206: 0.005642, 0.007214 avg loss, 0.000100 rate, 2.718634 seconds, 166592 images
Loaded: 0.000039 seconds
5207: 0.010985, 0.007591 avg loss, 0.000100 rate, 2.716330 seconds, 166624 images
Loaded: 0.000044 seconds
5208: 0.005867, 0.007418 avg loss, 0.000100 rate, 2.720200 seconds, 166656 images
Loaded: 0.000042 seconds
5209: 0.006533, 0.007330 avg loss, 0.000100 rate, 2.722739 seconds, 166688 images
Loaded: 0.000050 seconds
5210: 0.005987, 0.007195 avg loss, 0.000100 rate, 2.719300 seconds, 166720 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5211: 0.013747, 0.007851 avg loss, 0.000100 rate, 4.372947 seconds, 166752 images
Loaded: 0.000036 seconds
5212: 0.010657, 0.008131 avg loss, 0.000100 rate, 4.515339 seconds, 166784 images
Loaded: 0.000075 seconds
5213: 0.010353, 0.008353 avg loss, 0.000100 rate, 4.518411 seconds, 166816 images
Loaded: 0.000043 seconds
5214: 0.012120, 0.008730 avg loss, 0.000100 rate, 4.520946 seconds, 166848 images
Loaded: 0.000049 seconds
5215: 0.008957, 0.008753 avg loss, 0.000100 rate, 4.522524 seconds, 166880 images
Loaded: 0.000073 seconds
5216: 0.007327, 0.008610 avg loss, 0.000100 rate, 4.536319 seconds, 166912 images
Loaded: 0.000043 seconds
5217: 0.007871, 0.008536 avg loss, 0.000100 rate, 4.546288 seconds, 166944 images
Loaded: 0.000075 seconds
5218: 0.004930, 0.008176 avg loss, 0.000100 rate, 4.536781 seconds, 166976 images
Loaded: 0.000042 seconds
5219: 0.005366, 0.007895 avg loss, 0.000100 rate, 4.539199 seconds, 167008 images
Loaded: 0.000069 seconds
5220: 0.004877, 0.007593 avg loss, 0.000100 rate, 4.539573 seconds, 167040 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.014140 seconds
5221: 0.005446, 0.007378 avg loss, 0.000100 rate, 2.621347 seconds, 167072 images
Loaded: 0.000071 seconds
5222: 0.007578, 0.007398 avg loss, 0.000100 rate, 2.726062 seconds, 167104 images
Loaded: 0.000041 seconds
5223: 0.006257, 0.007284 avg loss, 0.000100 rate, 2.719017 seconds, 167136 images
Loaded: 0.000042 seconds
5224: 0.007053, 0.007261 avg loss, 0.000100 rate, 2.728196 seconds, 167168 images
Loaded: 0.000078 seconds
5225: 0.005615, 0.007096 avg loss, 0.000100 rate, 2.733926 seconds, 167200 images
Loaded: 0.000037 seconds
5226: 0.004316, 0.006818 avg loss, 0.000100 rate, 2.721622 seconds, 167232 images
Loaded: 0.000052 seconds
5227: 0.007286, 0.006865 avg loss, 0.000100 rate, 2.736815 seconds, 167264 images
Loaded: 0.000072 seconds
5228: 0.011852, 0.007364 avg loss, 0.000100 rate, 2.729255 seconds, 167296 images
Loaded: 0.000057 seconds
5229: 0.007614, 0.007389 avg loss, 0.000100 rate, 2.723631 seconds, 167328 images
Loaded: 0.000043 seconds
5230: 0.007649, 0.007415 avg loss, 0.000100 rate, 2.725229 seconds, 167360 images
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000040 seconds
5231: 0.010426, 0.007716 avg loss, 0.000100 rate, 1.576916 seconds, 167392 images
Loaded: 0.000070 seconds
5232: 0.014572, 0.008402 avg loss, 0.000100 rate, 1.651365 seconds, 167424 images
Loaded: 0.000054 seconds
5233: 0.015545, 0.009116 avg loss, 0.000100 rate, 1.641603 seconds, 167456 images
Loaded: 0.000041 seconds
5234: 0.009787, 0.009183 avg loss, 0.000100 rate, 1.640962 seconds, 167488 images
Loaded: 0.000082 seconds
5235: 0.015152, 0.009780 avg loss, 0.000100 rate, 1.655091 seconds, 167520 images
Loaded: 0.000045 seconds
5236: 0.009478, 0.009750 avg loss, 0.000100 rate, 1.643532 seconds, 167552 images
Loaded: 0.000066 seconds
5237: 0.013578, 0.010132 avg loss, 0.000100 rate, 1.657125 seconds, 167584 images
Loaded: 0.000044 seconds
5238: 0.042214, 0.013341 avg loss, 0.000100 rate, 1.643543 seconds, 167616 images
Loaded: 0.000059 seconds
5239: 0.022109, 0.014217 avg loss, 0.000100 rate, 1.636909 seconds, 167648 images
Loaded: 0.000051 seconds
5240: 0.015691, 0.014365 avg loss, 0.000100 rate, 1.649870 seconds, 167680 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5241: 0.005901, 0.013518 avg loss, 0.000100 rate, 2.923242 seconds, 167712 images
Loaded: 0.000044 seconds
5242: 0.006885, 0.012855 avg loss, 0.000100 rate, 3.043306 seconds, 167744 images
Loaded: 0.000044 seconds
5243: 0.006597, 0.012229 avg loss, 0.000100 rate, 3.030505 seconds, 167776 images
Loaded: 0.000038 seconds
5244: 0.004941, 0.011500 avg loss, 0.000100 rate, 3.034215 seconds, 167808 images
Loaded: 0.000044 seconds
5245: 0.005235, 0.010874 avg loss, 0.000100 rate, 3.027008 seconds, 167840 images
Loaded: 0.000068 seconds
5246: 0.004282, 0.010215 avg loss, 0.000100 rate, 3.039283 seconds, 167872 images
Loaded: 0.000047 seconds
5247: 0.004804, 0.009674 avg loss, 0.000100 rate, 3.030805 seconds, 167904 images
Loaded: 0.000040 seconds
5248: 0.007149, 0.009421 avg loss, 0.000100 rate, 3.043459 seconds, 167936 images
Loaded: 0.000073 seconds
5249: 0.005642, 0.009043 avg loss, 0.000100 rate, 3.028748 seconds, 167968 images
Loaded: 0.000041 seconds
5250: 0.007244, 0.008863 avg loss, 0.000100 rate, 3.040575 seconds, 168000 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.174516 seconds
5251: 0.008956, 0.008873 avg loss, 0.000100 rate, 2.938471 seconds, 168032 images
Loaded: 0.000047 seconds
5252: 0.005673, 0.008553 avg loss, 0.000100 rate, 3.050854 seconds, 168064 images
Loaded: 0.000045 seconds
5253: 0.008709, 0.008568 avg loss, 0.000100 rate, 3.032991 seconds, 168096 images
Loaded: 0.000056 seconds
5254: 0.005051, 0.008217 avg loss, 0.000100 rate, 3.047561 seconds, 168128 images
Loaded: 0.000058 seconds
5255: 0.006625, 0.008057 avg loss, 0.000100 rate, 3.035385 seconds, 168160 images
Loaded: 0.000062 seconds
5256: 0.006026, 0.007854 avg loss, 0.000100 rate, 3.052479 seconds, 168192 images
Loaded: 0.000057 seconds
5257: 0.003785, 0.007447 avg loss, 0.000100 rate, 3.031185 seconds, 168224 images
Loaded: 0.000069 seconds
5258: 0.015566, 0.008259 avg loss, 0.000100 rate, 3.052528 seconds, 168256 images
Loaded: 0.000069 seconds
5259: 0.004374, 0.007871 avg loss, 0.000100 rate, 3.026362 seconds, 168288 images
Loaded: 0.000060 seconds
5260: 0.007036, 0.007787 avg loss, 0.000100 rate, 3.054855 seconds, 168320 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.118938 seconds
5261: 0.008129, 0.007821 avg loss, 0.000100 rate, 1.796595 seconds, 168352 images
Loaded: 0.000040 seconds
5262: 0.010826, 0.008122 avg loss, 0.000100 rate, 1.867166 seconds, 168384 images
Loaded: 0.000061 seconds
5263: 0.011478, 0.008457 avg loss, 0.000100 rate, 1.872532 seconds, 168416 images
Loaded: 0.000035 seconds
5264: 0.009326, 0.008544 avg loss, 0.000100 rate, 1.880520 seconds, 168448 images
Loaded: 0.000043 seconds
5265: 0.012524, 0.008942 avg loss, 0.000100 rate, 1.878398 seconds, 168480 images
Loaded: 0.000044 seconds
5266: 0.009373, 0.008985 avg loss, 0.000100 rate, 1.866286 seconds, 168512 images
Loaded: 0.000050 seconds
5267: 0.016117, 0.009699 avg loss, 0.000100 rate, 1.870081 seconds, 168544 images
Loaded: 0.000034 seconds
5268: 0.008455, 0.009574 avg loss, 0.000100 rate, 1.872075 seconds, 168576 images
Loaded: 0.000043 seconds
5269: 0.010148, 0.009632 avg loss, 0.000100 rate, 1.876993 seconds, 168608 images
Loaded: 0.000040 seconds
5270: 0.009892, 0.009658 avg loss, 0.000100 rate, 1.864496 seconds, 168640 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000039 seconds
5271: 0.006052, 0.009297 avg loss, 0.000100 rate, 2.933491 seconds, 168672 images
Loaded: 0.000040 seconds
5272: 0.006859, 0.009053 avg loss, 0.000100 rate, 3.042376 seconds, 168704 images
Loaded: 0.000044 seconds
5273: 0.007673, 0.008915 avg loss, 0.000100 rate, 3.030855 seconds, 168736 images
Loaded: 0.000063 seconds
5274: 0.007442, 0.008768 avg loss, 0.000100 rate, 3.048138 seconds, 168768 images
Loaded: 0.000040 seconds
5275: 0.006647, 0.008556 avg loss, 0.000100 rate, 3.028862 seconds, 168800 images
Loaded: 0.000038 seconds
5276: 0.004966, 0.008197 avg loss, 0.000100 rate, 3.034761 seconds, 168832 images
Loaded: 0.000070 seconds
5277: 0.004763, 0.007854 avg loss, 0.000100 rate, 3.023083 seconds, 168864 images
Loaded: 0.000054 seconds
5278: 0.003698, 0.007438 avg loss, 0.000100 rate, 3.035615 seconds, 168896 images
Loaded: 0.000061 seconds
5279: 0.004838, 0.007178 avg loss, 0.000100 rate, 3.018237 seconds, 168928 images
Loaded: 0.000066 seconds
5280: 0.004680, 0.006928 avg loss, 0.000100 rate, 3.035158 seconds, 168960 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.092145 seconds
5281: 0.013527, 0.007588 avg loss, 0.000100 rate, 2.619714 seconds, 168992 images
Loaded: 0.000069 seconds
5282: 0.003964, 0.007226 avg loss, 0.000100 rate, 2.718170 seconds, 169024 images
Loaded: 0.000048 seconds
5283: 0.007199, 0.007223 avg loss, 0.000100 rate, 2.703108 seconds, 169056 images
Loaded: 0.000072 seconds
5284: 0.005493, 0.007050 avg loss, 0.000100 rate, 2.717474 seconds, 169088 images
Loaded: 0.000046 seconds
5285: 0.005187, 0.006864 avg loss, 0.000100 rate, 2.710834 seconds, 169120 images
Loaded: 0.000038 seconds
5286: 0.007576, 0.006935 avg loss, 0.000100 rate, 2.720014 seconds, 169152 images
Loaded: 0.000042 seconds
5287: 0.008412, 0.007083 avg loss, 0.000100 rate, 2.713924 seconds, 169184 images
Loaded: 0.000042 seconds
5288: 0.008435, 0.007218 avg loss, 0.000100 rate, 2.722012 seconds, 169216 images
Loaded: 0.000060 seconds
5289: 0.005668, 0.007063 avg loss, 0.000100 rate, 2.713425 seconds, 169248 images
Loaded: 0.000038 seconds
5290: 0.005754, 0.006932 avg loss, 0.000100 rate, 2.723554 seconds, 169280 images
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000039 seconds
5291: 0.013402, 0.007579 avg loss, 0.000100 rate, 4.730455 seconds, 169312 images
Loaded: 0.000040 seconds
5292: 0.005785, 0.007399 avg loss, 0.000100 rate, 4.900947 seconds, 169344 images
Loaded: 0.000035 seconds
5293: 0.008394, 0.007499 avg loss, 0.000100 rate, 4.912349 seconds, 169376 images
Loaded: 0.000068 seconds
5294: 0.011535, 0.007903 avg loss, 0.000100 rate, 4.906174 seconds, 169408 images
Loaded: 0.000056 seconds
5295: 0.005577, 0.007670 avg loss, 0.000100 rate, 4.916762 seconds, 169440 images
Loaded: 0.000043 seconds
5296: 0.023355, 0.009239 avg loss, 0.000100 rate, 4.908979 seconds, 169472 images
Loaded: 0.000066 seconds
5297: 0.008033, 0.009118 avg loss, 0.000100 rate, 4.919742 seconds, 169504 images
Loaded: 0.000061 seconds
5298: 0.012267, 0.009433 avg loss, 0.000100 rate, 4.944154 seconds, 169536 images
Loaded: 0.000064 seconds
5299: 0.006332, 0.009123 avg loss, 0.000100 rate, 4.906219 seconds, 169568 images
Loaded: 0.000070 seconds
5300: 0.012035, 0.009414 avg loss, 0.000100 rate, 4.905409 seconds, 169600 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000039 seconds
5301: 0.006215, 0.009094 avg loss, 0.000100 rate, 2.021733 seconds, 169632 images
Loaded: 0.000037 seconds
5302: 0.007656, 0.008950 avg loss, 0.000100 rate, 2.106940 seconds, 169664 images
Loaded: 0.000038 seconds
5303: 0.008163, 0.008872 avg loss, 0.000100 rate, 2.103679 seconds, 169696 images
Loaded: 0.000042 seconds
5304: 0.007106, 0.008695 avg loss, 0.000100 rate, 2.103439 seconds, 169728 images
Loaded: 0.000054 seconds
5305: 0.006489, 0.008474 avg loss, 0.000100 rate, 2.109924 seconds, 169760 images
Loaded: 0.000047 seconds
5306: 0.025401, 0.010167 avg loss, 0.000100 rate, 2.109867 seconds, 169792 images
Loaded: 0.000043 seconds
5307: 0.007951, 0.009946 avg loss, 0.000100 rate, 2.113894 seconds, 169824 images
Loaded: 0.000040 seconds
5308: 0.013894, 0.010340 avg loss, 0.000100 rate, 2.114497 seconds, 169856 images
Loaded: 0.000050 seconds
5309: 0.015115, 0.010818 avg loss, 0.000100 rate, 2.099915 seconds, 169888 images
Loaded: 0.000038 seconds
5310: 0.008480, 0.010584 avg loss, 0.000100 rate, 2.107119 seconds, 169920 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5311: 0.004972, 0.010023 avg loss, 0.000100 rate, 2.617495 seconds, 169952 images
Loaded: 0.000032 seconds
5312: 0.008279, 0.009848 avg loss, 0.000100 rate, 2.720034 seconds, 169984 images
Loaded: 0.000064 seconds
5313: 0.004536, 0.009317 avg loss, 0.000100 rate, 2.718154 seconds, 170016 images
Loaded: 0.000044 seconds
5314: 0.004891, 0.008875 avg loss, 0.000100 rate, 2.720339 seconds, 170048 images
Loaded: 0.000050 seconds
5315: 0.006280, 0.008615 avg loss, 0.000100 rate, 2.723701 seconds, 170080 images
Loaded: 0.000053 seconds
5316: 0.005983, 0.008352 avg loss, 0.000100 rate, 2.721060 seconds, 170112 images
Loaded: 0.000040 seconds
5317: 0.010933, 0.008610 avg loss, 0.000100 rate, 2.724290 seconds, 170144 images
Loaded: 0.000055 seconds
5318: 0.008712, 0.008620 avg loss, 0.000100 rate, 2.715690 seconds, 170176 images
Loaded: 0.000042 seconds
5319: 0.003280, 0.008086 avg loss, 0.000100 rate, 2.718350 seconds, 170208 images
Loaded: 0.000045 seconds
5320: 0.006634, 0.007941 avg loss, 0.000100 rate, 2.721786 seconds, 170240 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000041 seconds
5321: 0.007452, 0.007892 avg loss, 0.000100 rate, 4.013898 seconds, 170272 images
Loaded: 0.000040 seconds
5322: 0.009966, 0.008100 avg loss, 0.000100 rate, 4.160094 seconds, 170304 images
Loaded: 0.000041 seconds
5323: 0.007100, 0.008000 avg loss, 0.000100 rate, 4.160145 seconds, 170336 images
Loaded: 0.000042 seconds
5324: 0.006401, 0.007840 avg loss, 0.000100 rate, 4.172369 seconds, 170368 images
Loaded: 0.000034 seconds
5325: 0.008542, 0.007910 avg loss, 0.000100 rate, 4.164141 seconds, 170400 images
Loaded: 0.000035 seconds
5326: 0.004201, 0.007539 avg loss, 0.000100 rate, 4.160684 seconds, 170432 images
Loaded: 0.000040 seconds
5327: 0.005191, 0.007304 avg loss, 0.000100 rate, 4.162598 seconds, 170464 images
Loaded: 0.000044 seconds
5328: 0.006436, 0.007217 avg loss, 0.000100 rate, 4.178044 seconds, 170496 images
Loaded: 0.000043 seconds
5329: 0.004350, 0.006931 avg loss, 0.000100 rate, 4.165986 seconds, 170528 images
Loaded: 0.000066 seconds
5330: 0.006977, 0.006935 avg loss, 0.000100 rate, 4.169490 seconds, 170560 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5331: 0.013696, 0.007611 avg loss, 0.000100 rate, 1.796143 seconds, 170592 images
Loaded: 0.000040 seconds
5332: 0.011949, 0.008045 avg loss, 0.000100 rate, 1.863349 seconds, 170624 images
Loaded: 0.000079 seconds
5333: 0.020089, 0.009249 avg loss, 0.000100 rate, 1.871027 seconds, 170656 images
Loaded: 0.000053 seconds
5334: 0.009440, 0.009268 avg loss, 0.000100 rate, 1.872079 seconds, 170688 images
Loaded: 0.000039 seconds
5335: 0.009942, 0.009336 avg loss, 0.000100 rate, 1.872692 seconds, 170720 images
Loaded: 0.000040 seconds
5336: 0.011962, 0.009599 avg loss, 0.000100 rate, 1.875580 seconds, 170752 images
Loaded: 0.000041 seconds
5337: 0.010367, 0.009675 avg loss, 0.000100 rate, 1.867768 seconds, 170784 images
Loaded: 0.000063 seconds
5338: 0.011346, 0.009842 avg loss, 0.000100 rate, 1.870156 seconds, 170816 images
Loaded: 0.000037 seconds
5339: 0.007460, 0.009604 avg loss, 0.000100 rate, 1.871335 seconds, 170848 images
Loaded: 0.000063 seconds
5340: 0.009437, 0.009587 avg loss, 0.000100 rate, 1.869779 seconds, 170880 images
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000039 seconds
5341: 0.012515, 0.009880 avg loss, 0.000100 rate, 4.729508 seconds, 170912 images
Loaded: 0.000040 seconds
5342: 0.008266, 0.009719 avg loss, 0.000100 rate, 4.887729 seconds, 170944 images
Loaded: 0.000035 seconds
5343: 0.006457, 0.009393 avg loss, 0.000100 rate, 4.882718 seconds, 170976 images
Loaded: 0.000045 seconds
5344: 0.016881, 0.010141 avg loss, 0.000100 rate, 4.891298 seconds, 171008 images
Loaded: 0.000042 seconds
5345: 0.005169, 0.009644 avg loss, 0.000100 rate, 4.892621 seconds, 171040 images
Loaded: 0.000055 seconds
5346: 0.013343, 0.010014 avg loss, 0.000100 rate, 4.873226 seconds, 171072 images
Loaded: 0.000040 seconds
5347: 0.011908, 0.010203 avg loss, 0.000100 rate, 4.884735 seconds, 171104 images
Loaded: 0.000083 seconds
5348: 0.006355, 0.009819 avg loss, 0.000100 rate, 4.894476 seconds, 171136 images
Loaded: 0.000069 seconds
5349: 0.016453, 0.010482 avg loss, 0.000100 rate, 4.883648 seconds, 171168 images
Loaded: 0.000069 seconds
5350: 0.012457, 0.010680 avg loss, 0.000100 rate, 4.913781 seconds, 171200 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.003135 seconds
5351: 0.006645, 0.010276 avg loss, 0.000100 rate, 2.620888 seconds, 171232 images
Loaded: 0.000070 seconds
5352: 0.006678, 0.009916 avg loss, 0.000100 rate, 2.723877 seconds, 171264 images
Loaded: 0.000064 seconds
5353: 0.006808, 0.009606 avg loss, 0.000100 rate, 2.720593 seconds, 171296 images
Loaded: 0.000049 seconds
5354: 0.004779, 0.009123 avg loss, 0.000100 rate, 2.716648 seconds, 171328 images
Loaded: 0.000039 seconds
5355: 0.006895, 0.008900 avg loss, 0.000100 rate, 2.719529 seconds, 171360 images
Loaded: 0.000040 seconds
5356: 0.010813, 0.009091 avg loss, 0.000100 rate, 2.717102 seconds, 171392 images
Loaded: 0.000036 seconds
5357: 0.005545, 0.008737 avg loss, 0.000100 rate, 2.718114 seconds, 171424 images
Loaded: 0.000037 seconds
5358: 0.005676, 0.008431 avg loss, 0.000100 rate, 2.727588 seconds, 171456 images
Loaded: 0.000044 seconds
5359: 0.005518, 0.008139 avg loss, 0.000100 rate, 2.722142 seconds, 171488 images
Loaded: 0.000039 seconds
5360: 0.009285, 0.008254 avg loss, 0.000100 rate, 2.721924 seconds, 171520 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5361: 0.008094, 0.008238 avg loss, 0.000100 rate, 2.390460 seconds, 171552 images
Loaded: 0.000042 seconds
5362: 0.011183, 0.008533 avg loss, 0.000100 rate, 2.478599 seconds, 171584 images
Loaded: 0.000066 seconds
5363: 0.004071, 0.008086 avg loss, 0.000100 rate, 2.485840 seconds, 171616 images
Loaded: 0.000055 seconds
5364: 0.006388, 0.007917 avg loss, 0.000100 rate, 2.493929 seconds, 171648 images
Loaded: 0.000065 seconds
5365: 0.004634, 0.007588 avg loss, 0.000100 rate, 2.477838 seconds, 171680 images
Loaded: 0.000032 seconds
5366: 0.005222, 0.007352 avg loss, 0.000100 rate, 2.485090 seconds, 171712 images
Loaded: 0.000039 seconds
5367: 0.010688, 0.007685 avg loss, 0.000100 rate, 2.480667 seconds, 171744 images
Loaded: 0.000046 seconds
5368: 0.004170, 0.007334 avg loss, 0.000100 rate, 2.485727 seconds, 171776 images
Loaded: 0.000074 seconds
5369: 0.012904, 0.007891 avg loss, 0.000100 rate, 2.481586 seconds, 171808 images
Loaded: 0.000076 seconds
5370: 0.005364, 0.007638 avg loss, 0.000100 rate, 2.480475 seconds, 171840 images
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5371: 0.009171, 0.007791 avg loss, 0.000100 rate, 4.729544 seconds, 171872 images
Loaded: 0.000057 seconds
5372: 0.008875, 0.007900 avg loss, 0.000100 rate, 4.890131 seconds, 171904 images
Loaded: 0.000045 seconds
5373: 0.016881, 0.008798 avg loss, 0.000100 rate, 4.892477 seconds, 171936 images
Loaded: 0.000044 seconds
5374: 0.008416, 0.008760 avg loss, 0.000100 rate, 4.908640 seconds, 171968 images
Loaded: 0.000038 seconds
5375: 0.009766, 0.008860 avg loss, 0.000100 rate, 4.911878 seconds, 172000 images
Loaded: 0.000034 seconds
5376: 0.007360, 0.008710 avg loss, 0.000100 rate, 4.903490 seconds, 172032 images
Loaded: 0.000066 seconds
5377: 0.008155, 0.008655 avg loss, 0.000100 rate, 4.905753 seconds, 172064 images
Loaded: 0.000040 seconds
5378: 0.005669, 0.008356 avg loss, 0.000100 rate, 4.903433 seconds, 172096 images
Loaded: 0.000064 seconds
5379: 0.008913, 0.008412 avg loss, 0.000100 rate, 4.902807 seconds, 172128 images
Loaded: 0.000043 seconds
5380: 0.008148, 0.008386 avg loss, 0.000100 rate, 4.914197 seconds, 172160 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000036 seconds
5381: 0.008640, 0.008411 avg loss, 0.000100 rate, 2.042445 seconds, 172192 images
Loaded: 0.000037 seconds
5382: 0.013376, 0.008908 avg loss, 0.000100 rate, 2.104590 seconds, 172224 images
Loaded: 0.000039 seconds
5383: 0.006881, 0.008705 avg loss, 0.000100 rate, 2.112280 seconds, 172256 images
Loaded: 0.000066 seconds
5384: 0.006225, 0.008457 avg loss, 0.000100 rate, 2.107955 seconds, 172288 images
Loaded: 0.000060 seconds
5385: 0.005246, 0.008136 avg loss, 0.000100 rate, 2.103733 seconds, 172320 images
Loaded: 0.000038 seconds
5386: 0.007865, 0.008109 avg loss, 0.000100 rate, 2.109500 seconds, 172352 images
Loaded: 0.000080 seconds
5387: 0.006182, 0.007916 avg loss, 0.000100 rate, 2.106301 seconds, 172384 images
Loaded: 0.000039 seconds
5388: 0.007165, 0.007841 avg loss, 0.000100 rate, 2.114769 seconds, 172416 images
Loaded: 0.000039 seconds
5389: 0.007571, 0.007814 avg loss, 0.000100 rate, 2.109415 seconds, 172448 images
Loaded: 0.000044 seconds
5390: 0.009322, 0.007965 avg loss, 0.000100 rate, 2.104845 seconds, 172480 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000059 seconds
5391: 0.004627, 0.007631 avg loss, 0.000100 rate, 3.168080 seconds, 172512 images
Loaded: 0.000038 seconds
5392: 0.005300, 0.007398 avg loss, 0.000100 rate, 3.294816 seconds, 172544 images
Loaded: 0.000065 seconds
5393: 0.004309, 0.007089 avg loss, 0.000100 rate, 3.271218 seconds, 172576 images
Loaded: 0.000051 seconds
5394: 0.015531, 0.007933 avg loss, 0.000100 rate, 3.300750 seconds, 172608 images
Loaded: 0.000042 seconds
5395: 0.006748, 0.007815 avg loss, 0.000100 rate, 3.286344 seconds, 172640 images
Loaded: 0.000043 seconds
5396: 0.005939, 0.007627 avg loss, 0.000100 rate, 3.316648 seconds, 172672 images
Loaded: 0.000082 seconds
5397: 0.006422, 0.007507 avg loss, 0.000100 rate, 3.295775 seconds, 172704 images
Loaded: 0.000045 seconds
5398: 0.006129, 0.007369 avg loss, 0.000100 rate, 3.287366 seconds, 172736 images
Loaded: 0.000055 seconds
5399: 0.004646, 0.007097 avg loss, 0.000100 rate, 3.303603 seconds, 172768 images
Loaded: 0.000066 seconds
5400: 0.006839, 0.007071 avg loss, 0.000010 rate, 3.293831 seconds, 172800 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.259817 seconds
5401: 0.004621, 0.006826 avg loss, 0.000010 rate, 2.391659 seconds, 172832 images
Loaded: 0.000060 seconds
5402: 0.010744, 0.007218 avg loss, 0.000010 rate, 2.478466 seconds, 172864 images
Loaded: 0.000037 seconds
5403: 0.006954, 0.007191 avg loss, 0.000010 rate, 2.482733 seconds, 172896 images
Loaded: 0.000036 seconds
5404: 0.014010, 0.007873 avg loss, 0.000010 rate, 2.473327 seconds, 172928 images
Loaded: 0.000042 seconds
5405: 0.005500, 0.007636 avg loss, 0.000010 rate, 2.471594 seconds, 172960 images
Loaded: 0.000043 seconds
5406: 0.005174, 0.007390 avg loss, 0.000010 rate, 2.482448 seconds, 172992 images
Loaded: 0.000080 seconds
5407: 0.006452, 0.007296 avg loss, 0.000010 rate, 2.489540 seconds, 173024 images
Loaded: 0.000070 seconds
5408: 0.006043, 0.007171 avg loss, 0.000010 rate, 2.475807 seconds, 173056 images
Loaded: 0.000051 seconds
5409: 0.009807, 0.007434 avg loss, 0.000010 rate, 2.487325 seconds, 173088 images
Loaded: 0.000064 seconds
5410: 0.007790, 0.007470 avg loss, 0.000010 rate, 2.486281 seconds, 173120 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.083273 seconds
5411: 0.007690, 0.007492 avg loss, 0.000010 rate, 2.021083 seconds, 173152 images
Loaded: 0.000039 seconds
5412: 0.005830, 0.007326 avg loss, 0.000010 rate, 2.106124 seconds, 173184 images
Loaded: 0.000055 seconds
5413: 0.009392, 0.007532 avg loss, 0.000010 rate, 2.102053 seconds, 173216 images
Loaded: 0.000070 seconds
5414: 0.005446, 0.007324 avg loss, 0.000010 rate, 2.114120 seconds, 173248 images
Loaded: 0.000040 seconds
5415: 0.006860, 0.007277 avg loss, 0.000010 rate, 2.117658 seconds, 173280 images
Loaded: 0.000079 seconds
5416: 0.005954, 0.007145 avg loss, 0.000010 rate, 2.111670 seconds, 173312 images
Loaded: 0.000044 seconds
5417: 0.017430, 0.008174 avg loss, 0.000010 rate, 2.106821 seconds, 173344 images
Loaded: 0.000042 seconds
5418: 0.006731, 0.008029 avg loss, 0.000010 rate, 2.107116 seconds, 173376 images
Loaded: 0.000036 seconds
5419: 0.007287, 0.007955 avg loss, 0.000010 rate, 2.113955 seconds, 173408 images
Loaded: 0.000036 seconds
5420: 0.006380, 0.007798 avg loss, 0.000010 rate, 2.110141 seconds, 173440 images
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000036 seconds
5421: 0.009145, 0.007932 avg loss, 0.000010 rate, 4.728645 seconds, 173472 images
Loaded: 0.000071 seconds
5422: 0.004381, 0.007577 avg loss, 0.000010 rate, 4.893568 seconds, 173504 images
Loaded: 0.000065 seconds
5423: 0.013655, 0.008185 avg loss, 0.000010 rate, 4.906369 seconds, 173536 images
Loaded: 0.000042 seconds
5424: 0.005745, 0.007941 avg loss, 0.000010 rate, 4.899779 seconds, 173568 images
Loaded: 0.000051 seconds
5425: 0.006422, 0.007789 avg loss, 0.000010 rate, 4.875939 seconds, 173600 images
Loaded: 0.000049 seconds
5426: 0.010197, 0.008030 avg loss, 0.000010 rate, 4.881861 seconds, 173632 images
Loaded: 0.000033 seconds
5427: 0.012226, 0.008449 avg loss, 0.000010 rate, 4.882357 seconds, 173664 images
Loaded: 0.000079 seconds
5428: 0.014589, 0.009063 avg loss, 0.000010 rate, 4.893724 seconds, 173696 images
Loaded: 0.000042 seconds
5429: 0.015060, 0.009663 avg loss, 0.000010 rate, 4.903140 seconds, 173728 images
Loaded: 0.000054 seconds
5430: 0.010395, 0.009736 avg loss, 0.000010 rate, 4.916135 seconds, 173760 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.039211 seconds
5431: 0.005018, 0.009264 avg loss, 0.000010 rate, 2.928263 seconds, 173792 images
Loaded: 0.000040 seconds
5432: 0.006584, 0.008996 avg loss, 0.000010 rate, 3.034297 seconds, 173824 images
Loaded: 0.000041 seconds
5433: 0.005730, 0.008670 avg loss, 0.000010 rate, 3.040492 seconds, 173856 images
Loaded: 0.000044 seconds
5434: 0.010193, 0.008822 avg loss, 0.000010 rate, 3.037435 seconds, 173888 images
Loaded: 0.000041 seconds
5435: 0.006526, 0.008593 avg loss, 0.000010 rate, 3.033296 seconds, 173920 images
Loaded: 0.000043 seconds
5436: 0.005287, 0.008262 avg loss, 0.000010 rate, 3.047520 seconds, 173952 images
Loaded: 0.000041 seconds
5437: 0.005110, 0.007947 avg loss, 0.000010 rate, 3.045152 seconds, 173984 images
Loaded: 0.000041 seconds
5438: 0.006281, 0.007780 avg loss, 0.000010 rate, 3.049535 seconds, 174016 images
Loaded: 0.000057 seconds
5439: 0.006040, 0.007606 avg loss, 0.000010 rate, 3.042529 seconds, 174048 images
Loaded: 0.000042 seconds
5440: 0.003751, 0.007221 avg loss, 0.000010 rate, 3.041402 seconds, 174080 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000046 seconds
5441: 0.006273, 0.007126 avg loss, 0.000010 rate, 1.799879 seconds, 174112 images
Loaded: 0.000054 seconds
5442: 0.009088, 0.007322 avg loss, 0.000010 rate, 1.862316 seconds, 174144 images
Loaded: 0.000054 seconds
5443: 0.007491, 0.007339 avg loss, 0.000010 rate, 1.864867 seconds, 174176 images
Loaded: 0.000064 seconds
5444: 0.019692, 0.008574 avg loss, 0.000010 rate, 1.863272 seconds, 174208 images
Loaded: 0.000047 seconds
5445: 0.010747, 0.008791 avg loss, 0.000010 rate, 1.864662 seconds, 174240 images
Loaded: 0.000056 seconds
5446: 0.016650, 0.009577 avg loss, 0.000010 rate, 1.862100 seconds, 174272 images
Loaded: 0.000064 seconds
5447: 0.036849, 0.012304 avg loss, 0.000010 rate, 1.863443 seconds, 174304 images
Loaded: 0.000060 seconds
5448: 0.032472, 0.014321 avg loss, 0.000010 rate, 1.862365 seconds, 174336 images
Loaded: 0.000060 seconds
5449: 0.010062, 0.013895 avg loss, 0.000010 rate, 1.861530 seconds, 174368 images
Loaded: 0.000044 seconds
5450: 0.012753, 0.013781 avg loss, 0.000010 rate, 1.866120 seconds, 174400 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.076609 seconds
5451: 0.007802, 0.013183 avg loss, 0.000010 rate, 1.786748 seconds, 174432 images
Loaded: 0.000040 seconds
5452: 0.013257, 0.013191 avg loss, 0.000010 rate, 1.870746 seconds, 174464 images
Loaded: 0.000041 seconds
5453: 0.009393, 0.012811 avg loss, 0.000010 rate, 1.862533 seconds, 174496 images
Loaded: 0.000038 seconds
5454: 0.007684, 0.012298 avg loss, 0.000010 rate, 1.869422 seconds, 174528 images
Loaded: 0.000060 seconds
5455: 0.013436, 0.012412 avg loss, 0.000010 rate, 1.867495 seconds, 174560 images
Loaded: 0.000037 seconds
5456: 0.011462, 0.012317 avg loss, 0.000010 rate, 1.878384 seconds, 174592 images
Loaded: 0.000055 seconds
5457: 0.010597, 0.012145 avg loss, 0.000010 rate, 1.859462 seconds, 174624 images
Loaded: 0.000039 seconds
5458: 0.012799, 0.012210 avg loss, 0.000010 rate, 1.865530 seconds, 174656 images
Loaded: 0.000041 seconds
5459: 0.012172, 0.012206 avg loss, 0.000010 rate, 1.863065 seconds, 174688 images
Loaded: 0.000038 seconds
5460: 0.015776, 0.012563 avg loss, 0.000010 rate, 1.866172 seconds, 174720 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000051 seconds
5461: 0.006759, 0.011983 avg loss, 0.000010 rate, 2.619240 seconds, 174752 images
Loaded: 0.000061 seconds
5462: 0.006591, 0.011444 avg loss, 0.000010 rate, 2.715346 seconds, 174784 images
Loaded: 0.000072 seconds
5463: 0.007130, 0.011012 avg loss, 0.000010 rate, 2.724255 seconds, 174816 images
Loaded: 0.000068 seconds
5464: 0.004789, 0.010390 avg loss, 0.000010 rate, 2.714747 seconds, 174848 images
Loaded: 0.000034 seconds
5465: 0.006716, 0.010023 avg loss, 0.000010 rate, 2.715256 seconds, 174880 images
Loaded: 0.000042 seconds
5466: 0.004316, 0.009452 avg loss, 0.000010 rate, 2.719839 seconds, 174912 images
Loaded: 0.000041 seconds
5467: 0.003574, 0.008864 avg loss, 0.000010 rate, 2.725750 seconds, 174944 images
Loaded: 0.000037 seconds
5468: 0.006384, 0.008616 avg loss, 0.000010 rate, 2.720041 seconds, 174976 images
Loaded: 0.000040 seconds
5469: 0.018866, 0.009641 avg loss, 0.000010 rate, 2.723282 seconds, 175008 images
Loaded: 0.000041 seconds
5470: 0.004113, 0.009088 avg loss, 0.000010 rate, 2.729618 seconds, 175040 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.090733 seconds
5471: 0.010190, 0.009199 avg loss, 0.000010 rate, 1.792477 seconds, 175072 images
Loaded: 0.000038 seconds
5472: 0.011810, 0.009460 avg loss, 0.000010 rate, 1.870814 seconds, 175104 images
Loaded: 0.000039 seconds
5473: 0.012468, 0.009761 avg loss, 0.000010 rate, 1.874427 seconds, 175136 images
Loaded: 0.000036 seconds
5474: 0.011496, 0.009934 avg loss, 0.000010 rate, 1.868340 seconds, 175168 images
Loaded: 0.000071 seconds
5475: 0.015738, 0.010514 avg loss, 0.000010 rate, 1.871192 seconds, 175200 images
Loaded: 0.000083 seconds
5476: 0.018505, 0.011314 avg loss, 0.000010 rate, 1.878838 seconds, 175232 images
Loaded: 0.000040 seconds
5477: 0.011488, 0.011331 avg loss, 0.000010 rate, 1.866970 seconds, 175264 images
Loaded: 0.000040 seconds
5478: 0.011236, 0.011322 avg loss, 0.000010 rate, 1.873190 seconds, 175296 images
Loaded: 0.000071 seconds
5479: 0.006288, 0.010818 avg loss, 0.000010 rate, 1.864199 seconds, 175328 images
Loaded: 0.000034 seconds
5480: 0.014182, 0.011155 avg loss, 0.000010 rate, 1.871399 seconds, 175360 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.110456 seconds
5481: 0.024397, 0.012479 avg loss, 0.000010 rate, 1.792213 seconds, 175392 images
Loaded: 0.000036 seconds
5482: 0.005717, 0.011803 avg loss, 0.000010 rate, 1.863986 seconds, 175424 images
Loaded: 0.000050 seconds
5483: 0.007681, 0.011390 avg loss, 0.000010 rate, 1.863983 seconds, 175456 images
Loaded: 0.000050 seconds
5484: 0.007421, 0.010993 avg loss, 0.000010 rate, 1.862947 seconds, 175488 images
Loaded: 0.000038 seconds
5485: 0.006181, 0.010512 avg loss, 0.000010 rate, 1.870282 seconds, 175520 images
Loaded: 0.000069 seconds
5486: 0.007968, 0.010258 avg loss, 0.000010 rate, 1.871785 seconds, 175552 images
Loaded: 0.000037 seconds
5487: 0.008692, 0.010101 avg loss, 0.000010 rate, 1.868418 seconds, 175584 images
Loaded: 0.000054 seconds
5488: 0.007295, 0.009821 avg loss, 0.000010 rate, 1.866432 seconds, 175616 images
Loaded: 0.000034 seconds
5489: 0.028185, 0.011657 avg loss, 0.000010 rate, 1.875428 seconds, 175648 images
Loaded: 0.000040 seconds
5490: 0.011415, 0.011633 avg loss, 0.000010 rate, 1.864802 seconds, 175680 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5491: 0.004926, 0.010962 avg loss, 0.000010 rate, 2.618432 seconds, 175712 images
Loaded: 0.000037 seconds
5492: 0.004681, 0.010334 avg loss, 0.000010 rate, 2.722623 seconds, 175744 images
Loaded: 0.000051 seconds
5493: 0.005137, 0.009814 avg loss, 0.000010 rate, 2.742287 seconds, 175776 images
Loaded: 0.000066 seconds
5494: 0.005098, 0.009343 avg loss, 0.000010 rate, 2.721708 seconds, 175808 images
Loaded: 0.000058 seconds
5495: 0.006066, 0.009015 avg loss, 0.000010 rate, 2.729662 seconds, 175840 images
Loaded: 0.000048 seconds
5496: 0.004153, 0.008529 avg loss, 0.000010 rate, 2.728023 seconds, 175872 images
Loaded: 0.000049 seconds
5497: 0.004808, 0.008157 avg loss, 0.000010 rate, 2.723726 seconds, 175904 images
Loaded: 0.000060 seconds
5498: 0.006907, 0.008032 avg loss, 0.000010 rate, 2.719062 seconds, 175936 images
Loaded: 0.000042 seconds
5499: 0.011116, 0.008340 avg loss, 0.000010 rate, 2.720728 seconds, 175968 images
Loaded: 0.000063 seconds
5500: 0.003256, 0.007832 avg loss, 0.000010 rate, 2.718237 seconds, 176000 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000039 seconds
5501: 0.005707, 0.007619 avg loss, 0.000010 rate, 4.730841 seconds, 176032 images
Loaded: 0.000047 seconds
5502: 0.008450, 0.007702 avg loss, 0.000010 rate, 4.906101 seconds, 176064 images
Loaded: 0.000065 seconds
5503: 0.006256, 0.007558 avg loss, 0.000010 rate, 4.873301 seconds, 176096 images
Loaded: 0.000041 seconds
5504: 0.007918, 0.007594 avg loss, 0.000010 rate, 4.898949 seconds, 176128 images
Loaded: 0.000033 seconds
5505: 0.005675, 0.007402 avg loss, 0.000010 rate, 4.885367 seconds, 176160 images
Loaded: 0.000039 seconds
5506: 0.007164, 0.007378 avg loss, 0.000010 rate, 4.905036 seconds, 176192 images
Loaded: 0.000065 seconds
5507: 0.014183, 0.008059 avg loss, 0.000010 rate, 4.911575 seconds, 176224 images
Loaded: 0.000059 seconds
5508: 0.008464, 0.008099 avg loss, 0.000010 rate, 4.908417 seconds, 176256 images
Loaded: 0.000045 seconds
5509: 0.005945, 0.007884 avg loss, 0.000010 rate, 4.917051 seconds, 176288 images
Loaded: 0.000039 seconds
5510: 0.006636, 0.007759 avg loss, 0.000010 rate, 4.912139 seconds, 176320 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.024129 seconds
5511: 0.005205, 0.007504 avg loss, 0.000010 rate, 2.932672 seconds, 176352 images
Loaded: 0.000041 seconds
5512: 0.005568, 0.007310 avg loss, 0.000010 rate, 3.041670 seconds, 176384 images
Loaded: 0.000049 seconds
5513: 0.004591, 0.007038 avg loss, 0.000010 rate, 3.038647 seconds, 176416 images
Loaded: 0.000038 seconds
5514: 0.003628, 0.006697 avg loss, 0.000010 rate, 3.037722 seconds, 176448 images
Loaded: 0.000067 seconds
5515: 0.006266, 0.006654 avg loss, 0.000010 rate, 3.044970 seconds, 176480 images
Loaded: 0.000037 seconds
5516: 0.006143, 0.006603 avg loss, 0.000010 rate, 3.047219 seconds, 176512 images
Loaded: 0.000040 seconds
5517: 0.004652, 0.006408 avg loss, 0.000010 rate, 3.045265 seconds, 176544 images
Loaded: 0.000064 seconds
5518: 0.002618, 0.006029 avg loss, 0.000010 rate, 3.042985 seconds, 176576 images
Loaded: 0.000077 seconds
5519: 0.006008, 0.006027 avg loss, 0.000010 rate, 3.057369 seconds, 176608 images
Loaded: 0.000065 seconds
5520: 0.004925, 0.005917 avg loss, 0.000010 rate, 3.039305 seconds, 176640 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.031058 seconds
5521: 0.009088, 0.006234 avg loss, 0.000010 rate, 2.937195 seconds, 176672 images
Loaded: 0.000042 seconds
5522: 0.003784, 0.005989 avg loss, 0.000010 rate, 3.043175 seconds, 176704 images
Loaded: 0.000044 seconds
5523: 0.005449, 0.005935 avg loss, 0.000010 rate, 3.041585 seconds, 176736 images
Loaded: 0.000038 seconds
5524: 0.004619, 0.005803 avg loss, 0.000010 rate, 3.042889 seconds, 176768 images
Loaded: 0.000062 seconds
5525: 0.003771, 0.005600 avg loss, 0.000010 rate, 3.043623 seconds, 176800 images
Loaded: 0.000053 seconds
5526: 0.005400, 0.005580 avg loss, 0.000010 rate, 3.045808 seconds, 176832 images
Loaded: 0.000064 seconds
5527: 0.005268, 0.005549 avg loss, 0.000010 rate, 3.052601 seconds, 176864 images
Loaded: 0.000038 seconds
5528: 0.003833, 0.005377 avg loss, 0.000010 rate, 3.056582 seconds, 176896 images
Loaded: 0.000051 seconds
5529: 0.007361, 0.005576 avg loss, 0.000010 rate, 3.051930 seconds, 176928 images
Loaded: 0.000045 seconds
5530: 0.005532, 0.005571 avg loss, 0.000010 rate, 3.047936 seconds, 176960 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.030926 seconds
5531: 0.006833, 0.005697 avg loss, 0.000010 rate, 2.624965 seconds, 176992 images
Loaded: 0.000052 seconds
5532: 0.004407, 0.005568 avg loss, 0.000010 rate, 2.719154 seconds, 177024 images
Loaded: 0.000068 seconds
5533: 0.006213, 0.005633 avg loss, 0.000010 rate, 2.728896 seconds, 177056 images
Loaded: 0.000063 seconds
5534: 0.005097, 0.005579 avg loss, 0.000010 rate, 2.730071 seconds, 177088 images
Loaded: 0.000046 seconds
5535: 0.009442, 0.005966 avg loss, 0.000010 rate, 2.725980 seconds, 177120 images
Loaded: 0.000045 seconds
5536: 0.008092, 0.006178 avg loss, 0.000010 rate, 2.723021 seconds, 177152 images
Loaded: 0.000040 seconds
5537: 0.005399, 0.006100 avg loss, 0.000010 rate, 2.725682 seconds, 177184 images
Loaded: 0.000055 seconds
5538: 0.004884, 0.005979 avg loss, 0.000010 rate, 2.735016 seconds, 177216 images
Loaded: 0.000054 seconds
5539: 0.003211, 0.005702 avg loss, 0.000010 rate, 2.737669 seconds, 177248 images
Loaded: 0.000040 seconds
5540: 0.005562, 0.005688 avg loss, 0.000010 rate, 2.723442 seconds, 177280 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000040 seconds
5541: 0.009692, 0.006088 avg loss, 0.000010 rate, 4.021595 seconds, 177312 images
Loaded: 0.000038 seconds
5542: 0.009897, 0.006469 avg loss, 0.000010 rate, 4.165518 seconds, 177344 images
Loaded: 0.000058 seconds
5543: 0.005330, 0.006355 avg loss, 0.000010 rate, 4.163161 seconds, 177376 images
Loaded: 0.000063 seconds
5544: 0.006230, 0.006343 avg loss, 0.000010 rate, 4.177964 seconds, 177408 images
Loaded: 0.000043 seconds
5545: 0.003041, 0.006012 avg loss, 0.000010 rate, 4.166366 seconds, 177440 images
Loaded: 0.000041 seconds
5546: 0.005394, 0.005951 avg loss, 0.000010 rate, 4.169175 seconds, 177472 images
Loaded: 0.000056 seconds
5547: 0.005279, 0.005883 avg loss, 0.000010 rate, 4.176443 seconds, 177504 images
Loaded: 0.000038 seconds
5548: 0.005283, 0.005823 avg loss, 0.000010 rate, 4.172395 seconds, 177536 images
Loaded: 0.000080 seconds
5549: 0.005303, 0.005771 avg loss, 0.000010 rate, 4.176924 seconds, 177568 images
Loaded: 0.000034 seconds
5550: 0.008461, 0.006040 avg loss, 0.000010 rate, 4.176374 seconds, 177600 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.025677 seconds
5551: 0.004294, 0.005866 avg loss, 0.000010 rate, 2.926373 seconds, 177632 images
Loaded: 0.000039 seconds
5552: 0.004920, 0.005771 avg loss, 0.000010 rate, 3.038681 seconds, 177664 images
Loaded: 0.000041 seconds
5553: 0.004641, 0.005658 avg loss, 0.000010 rate, 3.030916 seconds, 177696 images
Loaded: 0.000061 seconds
5554: 0.009968, 0.006089 avg loss, 0.000010 rate, 3.034210 seconds, 177728 images
Loaded: 0.000038 seconds
5555: 0.005076, 0.005988 avg loss, 0.000010 rate, 3.032449 seconds, 177760 images
Loaded: 0.000038 seconds
5556: 0.007606, 0.006150 avg loss, 0.000010 rate, 3.033226 seconds, 177792 images
Loaded: 0.000040 seconds
5557: 0.005190, 0.006054 avg loss, 0.000010 rate, 3.040494 seconds, 177824 images
Loaded: 0.000060 seconds
5558: 0.007034, 0.006152 avg loss, 0.000010 rate, 3.033129 seconds, 177856 images
Loaded: 0.000076 seconds
5559: 0.003338, 0.005870 avg loss, 0.000010 rate, 3.040737 seconds, 177888 images
Loaded: 0.000068 seconds
5560: 0.008284, 0.006112 avg loss, 0.000010 rate, 3.036012 seconds, 177920 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5561: 0.007490, 0.006250 avg loss, 0.000010 rate, 1.781107 seconds, 177952 images
Loaded: 0.000072 seconds
5562: 0.013920, 0.007017 avg loss, 0.000010 rate, 1.863888 seconds, 177984 images
Loaded: 0.000057 seconds
5563: 0.015400, 0.007855 avg loss, 0.000010 rate, 1.869447 seconds, 178016 images
Loaded: 0.000060 seconds
5564: 0.009766, 0.008046 avg loss, 0.000010 rate, 1.873637 seconds, 178048 images
Loaded: 0.000042 seconds
5565: 0.027667, 0.010008 avg loss, 0.000010 rate, 1.872947 seconds, 178080 images
Loaded: 0.000068 seconds
5566: 0.011291, 0.010136 avg loss, 0.000010 rate, 1.868032 seconds, 178112 images
Loaded: 0.000048 seconds
5567: 0.011002, 0.010223 avg loss, 0.000010 rate, 1.874766 seconds, 178144 images
Loaded: 0.000066 seconds
5568: 0.014455, 0.010646 avg loss, 0.000010 rate, 1.876756 seconds, 178176 images
Loaded: 0.000071 seconds
5569: 0.010667, 0.010648 avg loss, 0.000010 rate, 1.876036 seconds, 178208 images
Loaded: 0.000058 seconds
5570: 0.009718, 0.010555 avg loss, 0.000010 rate, 1.870148 seconds, 178240 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000041 seconds
5571: 0.005287, 0.010028 avg loss, 0.000010 rate, 2.635745 seconds, 178272 images
Loaded: 0.000069 seconds
5572: 0.005957, 0.009621 avg loss, 0.000010 rate, 2.722512 seconds, 178304 images
Loaded: 0.000042 seconds
5573: 0.012662, 0.009925 avg loss, 0.000010 rate, 2.728106 seconds, 178336 images
Loaded: 0.000043 seconds
5574: 0.004734, 0.009406 avg loss, 0.000010 rate, 2.726320 seconds, 178368 images
Loaded: 0.000064 seconds
5575: 0.004832, 0.008949 avg loss, 0.000010 rate, 2.736009 seconds, 178400 images
Loaded: 0.000054 seconds
5576: 0.005520, 0.008606 avg loss, 0.000010 rate, 2.735526 seconds, 178432 images
Loaded: 0.000040 seconds
5577: 0.004979, 0.008243 avg loss, 0.000010 rate, 2.728311 seconds, 178464 images
Loaded: 0.000068 seconds
5578: 0.005046, 0.007924 avg loss, 0.000010 rate, 2.736266 seconds, 178496 images
Loaded: 0.000043 seconds
5579: 0.003968, 0.007528 avg loss, 0.000010 rate, 2.731393 seconds, 178528 images
Loaded: 0.000059 seconds
5580: 0.006710, 0.007446 avg loss, 0.000010 rate, 2.732944 seconds, 178560 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5581: 0.004811, 0.007183 avg loss, 0.000010 rate, 2.927617 seconds, 178592 images
Loaded: 0.000043 seconds
5582: 0.004850, 0.006949 avg loss, 0.000010 rate, 3.040261 seconds, 178624 images
Loaded: 0.000057 seconds
5583: 0.004856, 0.006740 avg loss, 0.000010 rate, 3.035438 seconds, 178656 images
Loaded: 0.000042 seconds
5584: 0.005731, 0.006639 avg loss, 0.000010 rate, 3.047834 seconds, 178688 images
Loaded: 0.000057 seconds
5585: 0.005117, 0.006487 avg loss, 0.000010 rate, 3.039981 seconds, 178720 images
Loaded: 0.000063 seconds
5586: 0.003315, 0.006170 avg loss, 0.000010 rate, 3.044710 seconds, 178752 images
Loaded: 0.000041 seconds
5587: 0.002957, 0.005849 avg loss, 0.000010 rate, 3.033157 seconds, 178784 images
Loaded: 0.000037 seconds
5588: 0.005609, 0.005825 avg loss, 0.000010 rate, 3.041533 seconds, 178816 images
Loaded: 0.000061 seconds
5589: 0.004087, 0.005651 avg loss, 0.000010 rate, 3.035460 seconds, 178848 images
Loaded: 0.000048 seconds
5590: 0.006411, 0.005727 avg loss, 0.000010 rate, 3.039806 seconds, 178880 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.052078 seconds
5591: 0.007306, 0.005885 avg loss, 0.000010 rate, 2.397361 seconds, 178912 images
Loaded: 0.000036 seconds
5592: 0.007119, 0.006008 avg loss, 0.000010 rate, 2.476284 seconds, 178944 images
Loaded: 0.000082 seconds
5593: 0.006194, 0.006027 avg loss, 0.000010 rate, 2.484638 seconds, 178976 images
Loaded: 0.000034 seconds
5594: 0.007161, 0.006140 avg loss, 0.000010 rate, 2.479460 seconds, 179008 images
Loaded: 0.000042 seconds
5595: 0.007059, 0.006232 avg loss, 0.000010 rate, 2.486533 seconds, 179040 images
Loaded: 0.000044 seconds
5596: 0.005815, 0.006190 avg loss, 0.000010 rate, 2.479562 seconds, 179072 images
Loaded: 0.000067 seconds
5597: 0.004928, 0.006064 avg loss, 0.000010 rate, 2.480029 seconds, 179104 images
Loaded: 0.000082 seconds
5598: 0.008128, 0.006271 avg loss, 0.000010 rate, 2.481603 seconds, 179136 images
Loaded: 0.000046 seconds
5599: 0.012125, 0.006856 avg loss, 0.000010 rate, 2.479561 seconds, 179168 images
Loaded: 0.000043 seconds
5600: 0.004730, 0.006643 avg loss, 0.000010 rate, 2.477726 seconds, 179200 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5601: 0.006535, 0.006633 avg loss, 0.000010 rate, 3.199697 seconds, 179232 images
Loaded: 0.000082 seconds
5602: 0.010017, 0.006971 avg loss, 0.000010 rate, 3.283008 seconds, 179264 images
Loaded: 0.000043 seconds
5603: 0.007837, 0.007058 avg loss, 0.000010 rate, 3.277618 seconds, 179296 images
Loaded: 0.000040 seconds
5604: 0.005010, 0.006853 avg loss, 0.000010 rate, 3.294511 seconds, 179328 images
Loaded: 0.000039 seconds
5605: 0.004663, 0.006634 avg loss, 0.000010 rate, 3.305562 seconds, 179360 images
Loaded: 0.000037 seconds
5606: 0.006381, 0.006609 avg loss, 0.000010 rate, 3.301388 seconds, 179392 images
Loaded: 0.000080 seconds
5607: 0.004724, 0.006420 avg loss, 0.000010 rate, 3.292823 seconds, 179424 images
Loaded: 0.000042 seconds
5608: 0.011865, 0.006965 avg loss, 0.000010 rate, 3.303413 seconds, 179456 images
Loaded: 0.000043 seconds
5609: 0.004731, 0.006741 avg loss, 0.000010 rate, 3.299009 seconds, 179488 images
Loaded: 0.000043 seconds
5610: 0.005710, 0.006638 avg loss, 0.000010 rate, 3.300415 seconds, 179520 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5611: 0.006757, 0.006650 avg loss, 0.000010 rate, 4.029755 seconds, 179552 images
Loaded: 0.000042 seconds
5612: 0.005420, 0.006527 avg loss, 0.000010 rate, 4.172148 seconds, 179584 images
Loaded: 0.000058 seconds
5613: 0.011102, 0.006985 avg loss, 0.000010 rate, 4.170359 seconds, 179616 images
Loaded: 0.000056 seconds
5614: 0.016369, 0.007923 avg loss, 0.000010 rate, 4.188637 seconds, 179648 images
Loaded: 0.000067 seconds
5615: 0.003542, 0.007485 avg loss, 0.000010 rate, 4.162270 seconds, 179680 images
Loaded: 0.000045 seconds
5616: 0.003871, 0.007124 avg loss, 0.000010 rate, 4.182176 seconds, 179712 images
Loaded: 0.000046 seconds
5617: 0.007665, 0.007178 avg loss, 0.000010 rate, 4.161049 seconds, 179744 images
Loaded: 0.000040 seconds
5618: 0.006963, 0.007156 avg loss, 0.000010 rate, 4.183623 seconds, 179776 images
Loaded: 0.000044 seconds
5619: 0.004140, 0.006855 avg loss, 0.000010 rate, 4.157897 seconds, 179808 images
Loaded: 0.000041 seconds
5620: 0.004837, 0.006653 avg loss, 0.000010 rate, 4.183301 seconds, 179840 images
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000044 seconds
5621: 0.009800, 0.006968 avg loss, 0.000010 rate, 1.576823 seconds, 179872 images
Loaded: 0.000065 seconds
5622: 0.022245, 0.008495 avg loss, 0.000010 rate, 1.653551 seconds, 179904 images
Loaded: 0.000069 seconds
5623: 0.021046, 0.009750 avg loss, 0.000010 rate, 1.640960 seconds, 179936 images
Loaded: 0.000041 seconds
5624: 0.010310, 0.009806 avg loss, 0.000010 rate, 1.643928 seconds, 179968 images
Loaded: 0.000037 seconds
5625: 0.014484, 0.010274 avg loss, 0.000010 rate, 1.648870 seconds, 180000 images
Loaded: 0.000040 seconds
5626: 0.011022, 0.010349 avg loss, 0.000010 rate, 1.647500 seconds, 180032 images
Loaded: 0.000041 seconds
5627: 0.015779, 0.010892 avg loss, 0.000010 rate, 1.636375 seconds, 180064 images
Loaded: 0.000066 seconds
5628: 0.012673, 0.011070 avg loss, 0.000010 rate, 1.644999 seconds, 180096 images
Loaded: 0.000041 seconds
5629: 0.037108, 0.013674 avg loss, 0.000010 rate, 1.646559 seconds, 180128 images
Loaded: 0.000042 seconds
5630: 0.063805, 0.018687 avg loss, 0.000010 rate, 1.644420 seconds, 180160 images
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.095475 seconds
5631: 0.012199, 0.018038 avg loss, 0.000010 rate, 1.575601 seconds, 180192 images
Loaded: 0.000041 seconds
5632: 0.019335, 0.018168 avg loss, 0.000010 rate, 1.645076 seconds, 180224 images
Loaded: 0.000070 seconds
5633: 0.019117, 0.018263 avg loss, 0.000010 rate, 1.639452 seconds, 180256 images
Loaded: 0.000034 seconds
5634: 0.011564, 0.017593 avg loss, 0.000010 rate, 1.643092 seconds, 180288 images
Loaded: 0.000041 seconds
5635: 0.012236, 0.017057 avg loss, 0.000010 rate, 1.647856 seconds, 180320 images
Loaded: 0.000041 seconds
5636: 0.010123, 0.016364 avg loss, 0.000010 rate, 1.647566 seconds, 180352 images
Loaded: 0.000040 seconds
5637: 0.010999, 0.015827 avg loss, 0.000010 rate, 1.649937 seconds, 180384 images
Loaded: 0.000081 seconds
5638: 0.018971, 0.016142 avg loss, 0.000010 rate, 1.639794 seconds, 180416 images
Loaded: 0.000049 seconds
5639: 0.006619, 0.015189 avg loss, 0.000010 rate, 1.642741 seconds, 180448 images
Loaded: 0.000042 seconds
5640: 0.011586, 0.014829 avg loss, 0.000010 rate, 1.643423 seconds, 180480 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.000045 seconds
5641: 0.004612, 0.013807 avg loss, 0.000010 rate, 2.621810 seconds, 180512 images
Loaded: 0.000038 seconds
5642: 0.007613, 0.013188 avg loss, 0.000010 rate, 2.713115 seconds, 180544 images
Loaded: 0.000042 seconds
5643: 0.004713, 0.012340 avg loss, 0.000010 rate, 2.718024 seconds, 180576 images
Loaded: 0.000043 seconds
5644: 0.004836, 0.011590 avg loss, 0.000010 rate, 2.718946 seconds, 180608 images
Loaded: 0.000035 seconds
5645: 0.004744, 0.010905 avg loss, 0.000010 rate, 2.713210 seconds, 180640 images
Loaded: 0.000040 seconds
5646: 0.005377, 0.010353 avg loss, 0.000010 rate, 2.714790 seconds, 180672 images
Loaded: 0.000061 seconds
5647: 0.004213, 0.009739 avg loss, 0.000010 rate, 2.723647 seconds, 180704 images
Loaded: 0.000053 seconds
5648: 0.005576, 0.009322 avg loss, 0.000010 rate, 2.719023 seconds, 180736 images
Loaded: 0.000043 seconds
5649: 0.006092, 0.008999 avg loss, 0.000010 rate, 2.722152 seconds, 180768 images
Loaded: 0.000043 seconds
5650: 0.008186, 0.008918 avg loss, 0.000010 rate, 2.716837 seconds, 180800 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.049859 seconds
5651: 0.003880, 0.008414 avg loss, 0.000010 rate, 2.389298 seconds, 180832 images
Loaded: 0.000042 seconds
5652: 0.004350, 0.008008 avg loss, 0.000010 rate, 2.478160 seconds, 180864 images
Loaded: 0.000041 seconds
5653: 0.005102, 0.007717 avg loss, 0.000010 rate, 2.480637 seconds, 180896 images
Loaded: 0.000041 seconds
5654: 0.008343, 0.007780 avg loss, 0.000010 rate, 2.476857 seconds, 180928 images
Loaded: 0.000068 seconds
5655: 0.013060, 0.008308 avg loss, 0.000010 rate, 2.485043 seconds, 180960 images
Loaded: 0.000107 seconds
5656: 0.005868, 0.008064 avg loss, 0.000010 rate, 2.480761 seconds, 180992 images
Loaded: 0.000038 seconds
5657: 0.006481, 0.007906 avg loss, 0.000010 rate, 2.474494 seconds, 181024 images
Loaded: 0.000043 seconds
5658: 0.005336, 0.007649 avg loss, 0.000010 rate, 2.483727 seconds, 181056 images
Loaded: 0.000035 seconds
5659: 0.006710, 0.007555 avg loss, 0.000010 rate, 2.478576 seconds, 181088 images
Loaded: 0.000037 seconds
5660: 0.009716, 0.007771 avg loss, 0.000010 rate, 2.476114 seconds, 181120 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5661: 0.004089, 0.007403 avg loss, 0.000010 rate, 3.159714 seconds, 181152 images
Loaded: 0.000073 seconds
5662: 0.006677, 0.007330 avg loss, 0.000010 rate, 3.295353 seconds, 181184 images
Loaded: 0.000090 seconds
5663: 0.005409, 0.007138 avg loss, 0.000010 rate, 3.285899 seconds, 181216 images
Loaded: 0.000039 seconds
5664: 0.005764, 0.007001 avg loss, 0.000010 rate, 3.277212 seconds, 181248 images
Loaded: 0.000045 seconds
5665: 0.005662, 0.006867 avg loss, 0.000010 rate, 3.284935 seconds, 181280 images
Loaded: 0.000042 seconds
5666: 0.004961, 0.006676 avg loss, 0.000010 rate, 3.292465 seconds, 181312 images
Loaded: 0.000046 seconds
5667: 0.004063, 0.006415 avg loss, 0.000010 rate, 3.279019 seconds, 181344 images
Loaded: 0.000036 seconds
5668: 0.007473, 0.006521 avg loss, 0.000010 rate, 3.292541 seconds, 181376 images
Loaded: 0.000042 seconds
5669: 0.006317, 0.006500 avg loss, 0.000010 rate, 3.290667 seconds, 181408 images
Loaded: 0.000049 seconds
5670: 0.004903, 0.006341 avg loss, 0.000010 rate, 3.291405 seconds, 181440 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.040502 seconds
5671: 0.007047, 0.006411 avg loss, 0.000010 rate, 2.029815 seconds, 181472 images
Loaded: 0.000044 seconds
5672: 0.007718, 0.006542 avg loss, 0.000010 rate, 2.110269 seconds, 181504 images
Loaded: 0.000034 seconds
5673: 0.003901, 0.006278 avg loss, 0.000010 rate, 2.102885 seconds, 181536 images
Loaded: 0.000042 seconds
5674: 0.009048, 0.006555 avg loss, 0.000010 rate, 2.116227 seconds, 181568 images
Loaded: 0.000046 seconds
5675: 0.007231, 0.006622 avg loss, 0.000010 rate, 2.108495 seconds, 181600 images
Loaded: 0.000072 seconds
5676: 0.007087, 0.006669 avg loss, 0.000010 rate, 2.109308 seconds, 181632 images
Loaded: 0.000071 seconds
5677: 0.006874, 0.006689 avg loss, 0.000010 rate, 2.109238 seconds, 181664 images
Loaded: 0.000054 seconds
5678: 0.018610, 0.007881 avg loss, 0.000010 rate, 2.110718 seconds, 181696 images
Loaded: 0.000074 seconds
5679: 0.006002, 0.007693 avg loss, 0.000010 rate, 2.110865 seconds, 181728 images
Loaded: 0.000046 seconds
5680: 0.006610, 0.007585 avg loss, 0.000010 rate, 2.108969 seconds, 181760 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000051 seconds
5681: 0.007688, 0.007595 avg loss, 0.000010 rate, 2.393925 seconds, 181792 images
Loaded: 0.000044 seconds
5682: 0.006180, 0.007454 avg loss, 0.000010 rate, 2.487471 seconds, 181824 images
Loaded: 0.000040 seconds
5683: 0.007080, 0.007417 avg loss, 0.000010 rate, 2.491807 seconds, 181856 images
Loaded: 0.000050 seconds
5684: 0.008515, 0.007526 avg loss, 0.000010 rate, 2.481933 seconds, 181888 images
Loaded: 0.000043 seconds
5685: 0.004228, 0.007197 avg loss, 0.000010 rate, 2.482682 seconds, 181920 images
Loaded: 0.000039 seconds
5686: 0.008074, 0.007284 avg loss, 0.000010 rate, 2.484079 seconds, 181952 images
Loaded: 0.000073 seconds
5687: 0.005302, 0.007086 avg loss, 0.000010 rate, 2.477461 seconds, 181984 images
Loaded: 0.000069 seconds
5688: 0.004846, 0.006862 avg loss, 0.000010 rate, 2.484181 seconds, 182016 images
Loaded: 0.000071 seconds
5689: 0.004678, 0.006644 avg loss, 0.000010 rate, 2.477889 seconds, 182048 images
Loaded: 0.000043 seconds
5690: 0.009822, 0.006962 avg loss, 0.000010 rate, 2.483238 seconds, 182080 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.075101 seconds
5691: 0.011883, 0.007454 avg loss, 0.000010 rate, 2.402985 seconds, 182112 images
Loaded: 0.000066 seconds
5692: 0.009430, 0.007651 avg loss, 0.000010 rate, 2.486992 seconds, 182144 images
Loaded: 0.000043 seconds
5693: 0.008174, 0.007704 avg loss, 0.000010 rate, 2.486710 seconds, 182176 images
Loaded: 0.000040 seconds
5694: 0.005675, 0.007501 avg loss, 0.000010 rate, 2.496313 seconds, 182208 images
Loaded: 0.000061 seconds
5695: 0.005925, 0.007343 avg loss, 0.000010 rate, 2.486244 seconds, 182240 images
Loaded: 0.000045 seconds
5696: 0.008968, 0.007506 avg loss, 0.000010 rate, 2.484326 seconds, 182272 images
Loaded: 0.000058 seconds
5697: 0.007318, 0.007487 avg loss, 0.000010 rate, 2.485530 seconds, 182304 images
Loaded: 0.000074 seconds
5698: 0.005185, 0.007257 avg loss, 0.000010 rate, 2.493710 seconds, 182336 images
Loaded: 0.000099 seconds
5699: 0.004034, 0.006934 avg loss, 0.000010 rate, 2.492422 seconds, 182368 images
Loaded: 0.000046 seconds
5700: 0.006580, 0.006899 avg loss, 0.000010 rate, 2.487626 seconds, 182400 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5701: 0.009195, 0.007129 avg loss, 0.000010 rate, 4.729429 seconds, 182432 images
Loaded: 0.000044 seconds
5702: 0.006029, 0.007019 avg loss, 0.000010 rate, 4.902544 seconds, 182464 images
Loaded: 0.000072 seconds
5703: 0.008769, 0.007194 avg loss, 0.000010 rate, 4.903831 seconds, 182496 images
Loaded: 0.000049 seconds
5704: 0.019165, 0.008391 avg loss, 0.000010 rate, 4.911419 seconds, 182528 images
Loaded: 0.000045 seconds
5705: 0.008054, 0.008357 avg loss, 0.000010 rate, 4.914520 seconds, 182560 images
Loaded: 0.000071 seconds
5706: 0.008719, 0.008393 avg loss, 0.000010 rate, 4.931300 seconds, 182592 images
Loaded: 0.000044 seconds
5707: 0.009274, 0.008481 avg loss, 0.000010 rate, 4.893931 seconds, 182624 images
Loaded: 0.000047 seconds
5708: 0.008786, 0.008512 avg loss, 0.000010 rate, 4.897223 seconds, 182656 images
Loaded: 0.000071 seconds
5709: 0.006763, 0.008337 avg loss, 0.000010 rate, 4.905077 seconds, 182688 images
Loaded: 0.000069 seconds
5710: 0.009517, 0.008455 avg loss, 0.000010 rate, 4.904253 seconds, 182720 images
Resizing
448 x 448
try to allocate additional workspace_size = 59.01 MB
CUDA allocate done!
Loaded: 0.038909 seconds
5711: 0.017969, 0.009406 avg loss, 0.000010 rate, 2.627364 seconds, 182752 images
Loaded: 0.000042 seconds
5712: 0.004523, 0.008918 avg loss, 0.000010 rate, 2.725517 seconds, 182784 images
Loaded: 0.000045 seconds
5713: 0.006998, 0.008726 avg loss, 0.000010 rate, 2.730523 seconds, 182816 images
Loaded: 0.000072 seconds
5714: 0.004814, 0.008335 avg loss, 0.000010 rate, 2.720629 seconds, 182848 images
Loaded: 0.000048 seconds
5715: 0.006701, 0.008171 avg loss, 0.000010 rate, 2.721814 seconds, 182880 images
Loaded: 0.000039 seconds
5716: 0.005625, 0.007917 avg loss, 0.000010 rate, 2.731528 seconds, 182912 images
Loaded: 0.000041 seconds
5717: 0.004663, 0.007591 avg loss, 0.000010 rate, 2.733206 seconds, 182944 images
Loaded: 0.000037 seconds
5718: 0.004768, 0.007309 avg loss, 0.000010 rate, 2.723861 seconds, 182976 images
Loaded: 0.000041 seconds
5719: 0.009427, 0.007521 avg loss, 0.000010 rate, 2.725202 seconds, 183008 images
Loaded: 0.000044 seconds
5720: 0.006682, 0.007437 avg loss, 0.000010 rate, 2.725874 seconds, 183040 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5721: 0.005764, 0.007270 avg loss, 0.000010 rate, 2.932539 seconds, 183072 images
Loaded: 0.000049 seconds
5722: 0.007287, 0.007271 avg loss, 0.000010 rate, 3.054912 seconds, 183104 images
Loaded: 0.000043 seconds
5723: 0.006582, 0.007202 avg loss, 0.000010 rate, 3.050331 seconds, 183136 images
Loaded: 0.000043 seconds
5724: 0.006097, 0.007092 avg loss, 0.000010 rate, 3.042929 seconds, 183168 images
Loaded: 0.000067 seconds
5725: 0.003965, 0.006779 avg loss, 0.000010 rate, 3.040378 seconds, 183200 images
Loaded: 0.000071 seconds
5726: 0.003724, 0.006474 avg loss, 0.000010 rate, 3.040636 seconds, 183232 images
Loaded: 0.000070 seconds
5727: 0.004661, 0.006292 avg loss, 0.000010 rate, 3.041697 seconds, 183264 images
Loaded: 0.000070 seconds
5728: 0.005342, 0.006197 avg loss, 0.000010 rate, 3.039511 seconds, 183296 images
Loaded: 0.000070 seconds
5729: 0.005747, 0.006152 avg loss, 0.000010 rate, 3.037127 seconds, 183328 images
Loaded: 0.000086 seconds
5730: 0.006225, 0.006160 avg loss, 0.000010 rate, 3.036654 seconds, 183360 images
Resizing
512 x 512
try to allocate additional workspace_size = 77.07 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5731: 0.005953, 0.006139 avg loss, 0.000010 rate, 3.165895 seconds, 183392 images
Loaded: 0.000050 seconds
5732: 0.005697, 0.006095 avg loss, 0.000010 rate, 3.273405 seconds, 183424 images
Loaded: 0.000039 seconds
5733: 0.012748, 0.006760 avg loss, 0.000010 rate, 3.275866 seconds, 183456 images
Loaded: 0.000045 seconds
5734: 0.003949, 0.006479 avg loss, 0.000010 rate, 3.279959 seconds, 183488 images
Loaded: 0.000044 seconds
5735: 0.004205, 0.006252 avg loss, 0.000010 rate, 3.310323 seconds, 183520 images
Loaded: 0.000043 seconds
5736: 0.004660, 0.006092 avg loss, 0.000010 rate, 3.292207 seconds, 183552 images
Loaded: 0.000054 seconds
5737: 0.005903, 0.006073 avg loss, 0.000010 rate, 3.286154 seconds, 183584 images
Loaded: 0.000056 seconds
5738: 0.004725, 0.005939 avg loss, 0.000010 rate, 3.289945 seconds, 183616 images
Loaded: 0.000043 seconds
5739: 0.006887, 0.006034 avg loss, 0.000010 rate, 3.300289 seconds, 183648 images
Loaded: 0.000044 seconds
5740: 0.005523, 0.005982 avg loss, 0.000010 rate, 3.301097 seconds, 183680 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5741: 0.005446, 0.005929 avg loss, 0.000010 rate, 2.024319 seconds, 183712 images
Loaded: 0.000060 seconds
5742: 0.008031, 0.006139 avg loss, 0.000010 rate, 2.101646 seconds, 183744 images
Loaded: 0.000057 seconds
5743: 0.007043, 0.006229 avg loss, 0.000010 rate, 2.102471 seconds, 183776 images
Loaded: 0.000043 seconds
5744: 0.008924, 0.006499 avg loss, 0.000010 rate, 2.106489 seconds, 183808 images
Loaded: 0.000071 seconds
5745: 0.010715, 0.006921 avg loss, 0.000010 rate, 2.117764 seconds, 183840 images
Loaded: 0.000047 seconds
5746: 0.004950, 0.006723 avg loss, 0.000010 rate, 2.101509 seconds, 183872 images
Loaded: 0.000070 seconds
5747: 0.006702, 0.006721 avg loss, 0.000010 rate, 2.103675 seconds, 183904 images
Loaded: 0.000041 seconds
5748: 0.031847, 0.009234 avg loss, 0.000010 rate, 2.116091 seconds, 183936 images
Loaded: 0.000042 seconds
5749: 0.009670, 0.009277 avg loss, 0.000010 rate, 2.098529 seconds, 183968 images
Loaded: 0.000073 seconds
5750: 0.004650, 0.008815 avg loss, 0.000010 rate, 2.101417 seconds, 184000 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.141369 seconds
5751: 0.015434, 0.009477 avg loss, 0.000010 rate, 2.023196 seconds, 184032 images
Loaded: 0.000039 seconds
5752: 0.007137, 0.009243 avg loss, 0.000010 rate, 2.104461 seconds, 184064 images
Loaded: 0.000041 seconds
5753: 0.006413, 0.008960 avg loss, 0.000010 rate, 2.097220 seconds, 184096 images
Loaded: 0.000079 seconds
5754: 0.006892, 0.008753 avg loss, 0.000010 rate, 2.099033 seconds, 184128 images
Loaded: 0.000066 seconds
5755: 0.010691, 0.008947 avg loss, 0.000010 rate, 2.109118 seconds, 184160 images
Loaded: 0.000036 seconds
5756: 0.003968, 0.008449 avg loss, 0.000010 rate, 2.102119 seconds, 184192 images
Loaded: 0.000046 seconds
5757: 0.005958, 0.008200 avg loss, 0.000010 rate, 2.099649 seconds, 184224 images
Loaded: 0.000052 seconds
5758: 0.006660, 0.008046 avg loss, 0.000010 rate, 2.104734 seconds, 184256 images
Loaded: 0.000055 seconds
5759: 0.006657, 0.007907 avg loss, 0.000010 rate, 2.099388 seconds, 184288 images
Loaded: 0.000039 seconds
5760: 0.008237, 0.007940 avg loss, 0.000010 rate, 2.099418 seconds, 184320 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000063 seconds
5761: 0.005257, 0.007672 avg loss, 0.000010 rate, 2.918447 seconds, 184352 images
Loaded: 0.000060 seconds
5762: 0.007599, 0.007664 avg loss, 0.000010 rate, 3.028679 seconds, 184384 images
Loaded: 0.000063 seconds
5763: 0.004562, 0.007354 avg loss, 0.000010 rate, 3.028937 seconds, 184416 images
Loaded: 0.000066 seconds
5764: 0.005688, 0.007187 avg loss, 0.000010 rate, 3.033194 seconds, 184448 images
Loaded: 0.000039 seconds
5765: 0.005705, 0.007039 avg loss, 0.000010 rate, 3.030903 seconds, 184480 images
Loaded: 0.000041 seconds
5766: 0.006348, 0.006970 avg loss, 0.000010 rate, 3.030961 seconds, 184512 images
Loaded: 0.000084 seconds
5767: 0.005013, 0.006774 avg loss, 0.000010 rate, 3.036621 seconds, 184544 images
Loaded: 0.000042 seconds
5768: 0.003859, 0.006483 avg loss, 0.000010 rate, 3.043007 seconds, 184576 images
Loaded: 0.000060 seconds
5769: 0.008262, 0.006661 avg loss, 0.000010 rate, 3.037104 seconds, 184608 images
Loaded: 0.000038 seconds
5770: 0.003441, 0.006339 avg loss, 0.000010 rate, 3.040038 seconds, 184640 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000044 seconds
5771: 0.004595, 0.006164 avg loss, 0.000010 rate, 4.009379 seconds, 184672 images
Loaded: 0.000066 seconds
5772: 0.005929, 0.006141 avg loss, 0.000010 rate, 4.161186 seconds, 184704 images
Loaded: 0.000043 seconds
5773: 0.006641, 0.006191 avg loss, 0.000010 rate, 4.171951 seconds, 184736 images
Loaded: 0.000045 seconds
5774: 0.004086, 0.005980 avg loss, 0.000010 rate, 4.160495 seconds, 184768 images
Loaded: 0.000119 seconds
5775: 0.006507, 0.006033 avg loss, 0.000010 rate, 4.157638 seconds, 184800 images
Loaded: 0.000043 seconds
5776: 0.003875, 0.005817 avg loss, 0.000010 rate, 4.153179 seconds, 184832 images
Loaded: 0.000033 seconds
5777: 0.004788, 0.005714 avg loss, 0.000010 rate, 4.162686 seconds, 184864 images
Loaded: 0.000074 seconds
5778: 0.007372, 0.005880 avg loss, 0.000010 rate, 4.171062 seconds, 184896 images
Loaded: 0.000091 seconds
5779: 0.005567, 0.005849 avg loss, 0.000010 rate, 4.160939 seconds, 184928 images
Loaded: 0.000074 seconds
5780: 0.004417, 0.005705 avg loss, 0.000010 rate, 4.164880 seconds, 184960 images
Resizing
608 x 608
try to allocate additional workspace_size = 108.69 MB
CUDA allocate done!
Loaded: 0.000038 seconds
5781: 0.006142, 0.005749 avg loss, 0.000010 rate, 4.722070 seconds, 184992 images
Loaded: 0.000041 seconds
5782: 0.008465, 0.006021 avg loss, 0.000010 rate, 4.895733 seconds, 185024 images
Loaded: 0.000052 seconds
5783: 0.006640, 0.006083 avg loss, 0.000010 rate, 4.869369 seconds, 185056 images
Loaded: 0.000044 seconds
5784: 0.012985, 0.006773 avg loss, 0.000010 rate, 4.892428 seconds, 185088 images
Loaded: 0.000044 seconds
5785: 0.020586, 0.008154 avg loss, 0.000010 rate, 4.892742 seconds, 185120 images
Loaded: 0.000041 seconds
5786: 0.007974, 0.008136 avg loss, 0.000010 rate, 4.891329 seconds, 185152 images
Loaded: 0.000040 seconds
5787: 0.011498, 0.008472 avg loss, 0.000010 rate, 4.875001 seconds, 185184 images
Loaded: 0.000038 seconds
5788: 0.013935, 0.009019 avg loss, 0.000010 rate, 4.889070 seconds, 185216 images
Loaded: 0.000042 seconds
5789: 0.007655, 0.008882 avg loss, 0.000010 rate, 4.886663 seconds, 185248 images
Loaded: 0.000042 seconds
5790: 0.017801, 0.009774 avg loss, 0.000010 rate, 4.900342 seconds, 185280 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.038253 seconds
5791: 0.016109, 0.010408 avg loss, 0.000010 rate, 4.361328 seconds, 185312 images
Loaded: 0.000057 seconds
5792: 0.004820, 0.009849 avg loss, 0.000010 rate, 4.526647 seconds, 185344 images
Loaded: 0.000053 seconds
5793: 0.004601, 0.009324 avg loss, 0.000010 rate, 4.516652 seconds, 185376 images
Loaded: 0.000039 seconds
5794: 0.011072, 0.009499 avg loss, 0.000010 rate, 4.506857 seconds, 185408 images
Loaded: 0.000069 seconds
5795: 0.011670, 0.009716 avg loss, 0.000010 rate, 4.508055 seconds, 185440 images
Loaded: 0.000075 seconds
5796: 0.007090, 0.009453 avg loss, 0.000010 rate, 4.502858 seconds, 185472 images
Loaded: 0.000068 seconds
5797: 0.004492, 0.008957 avg loss, 0.000010 rate, 4.514461 seconds, 185504 images
Loaded: 0.000055 seconds
5798: 0.008610, 0.008922 avg loss, 0.000010 rate, 4.508892 seconds, 185536 images
Loaded: 0.000045 seconds
5799: 0.006555, 0.008686 avg loss, 0.000010 rate, 4.520134 seconds, 185568 images
Loaded: 0.000037 seconds
5800: 0.007756, 0.008593 avg loss, 0.000010 rate, 4.511670 seconds, 185600 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000058 seconds
5801: 0.023910, 0.010124 avg loss, 0.000010 rate, 1.791430 seconds, 185632 images
Loaded: 0.000036 seconds
5802: 0.007516, 0.009864 avg loss, 0.000010 rate, 1.863643 seconds, 185664 images
Loaded: 0.000042 seconds
5803: 0.007096, 0.009587 avg loss, 0.000010 rate, 1.869880 seconds, 185696 images
Loaded: 0.000044 seconds
5804: 0.009460, 0.009574 avg loss, 0.000010 rate, 1.873388 seconds, 185728 images
Loaded: 0.000068 seconds
5805: 0.011303, 0.009747 avg loss, 0.000010 rate, 1.869286 seconds, 185760 images
Loaded: 0.000053 seconds
5806: 0.006291, 0.009401 avg loss, 0.000010 rate, 1.868667 seconds, 185792 images
Loaded: 0.000064 seconds
5807: 0.010494, 0.009511 avg loss, 0.000010 rate, 1.871145 seconds, 185824 images
Loaded: 0.000044 seconds
5808: 0.009924, 0.009552 avg loss, 0.000010 rate, 1.870836 seconds, 185856 images
Loaded: 0.000044 seconds
5809: 0.006527, 0.009249 avg loss, 0.000010 rate, 1.882748 seconds, 185888 images
Loaded: 0.000045 seconds
5810: 0.010198, 0.009344 avg loss, 0.000010 rate, 1.871286 seconds, 185920 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000066 seconds
5811: 0.009284, 0.009338 avg loss, 0.000010 rate, 4.362702 seconds, 185952 images
Loaded: 0.000043 seconds
5812: 0.004623, 0.008867 avg loss, 0.000010 rate, 4.540426 seconds, 185984 images
Loaded: 0.000064 seconds
5813: 0.006377, 0.008618 avg loss, 0.000010 rate, 4.529375 seconds, 186016 images
Loaded: 0.000057 seconds
5814: 0.005356, 0.008292 avg loss, 0.000010 rate, 4.542343 seconds, 186048 images
Loaded: 0.000099 seconds
5815: 0.007884, 0.008251 avg loss, 0.000010 rate, 4.533254 seconds, 186080 images
Loaded: 0.000036 seconds
5816: 0.009823, 0.008408 avg loss, 0.000010 rate, 4.533301 seconds, 186112 images
Loaded: 0.000050 seconds
5817: 0.004189, 0.007986 avg loss, 0.000010 rate, 4.515797 seconds, 186144 images
Loaded: 0.000039 seconds
5818: 0.005073, 0.007695 avg loss, 0.000010 rate, 4.530287 seconds, 186176 images
Loaded: 0.000077 seconds
5819: 0.003982, 0.007324 avg loss, 0.000010 rate, 4.519427 seconds, 186208 images
Loaded: 0.000062 seconds
5820: 0.004213, 0.007012 avg loss, 0.000010 rate, 4.528563 seconds, 186240 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000068 seconds
5821: 0.007729, 0.007084 avg loss, 0.000010 rate, 1.802414 seconds, 186272 images
Loaded: 0.000066 seconds
5822: 0.022540, 0.008630 avg loss, 0.000010 rate, 1.863000 seconds, 186304 images
Loaded: 0.000055 seconds
5823: 0.011270, 0.008894 avg loss, 0.000010 rate, 1.873089 seconds, 186336 images
Loaded: 0.000084 seconds
5824: 0.006036, 0.008608 avg loss, 0.000010 rate, 1.865563 seconds, 186368 images
Loaded: 0.000042 seconds
5825: 0.008469, 0.008594 avg loss, 0.000010 rate, 1.878009 seconds, 186400 images
Loaded: 0.000040 seconds
5826: 0.012587, 0.008993 avg loss, 0.000010 rate, 1.870397 seconds, 186432 images
Loaded: 0.000067 seconds
5827: 0.013661, 0.009460 avg loss, 0.000010 rate, 1.868464 seconds, 186464 images
Loaded: 0.000047 seconds
5828: 0.027962, 0.011310 avg loss, 0.000010 rate, 1.871519 seconds, 186496 images
Loaded: 0.000072 seconds
5829: 0.007364, 0.010916 avg loss, 0.000010 rate, 1.861967 seconds, 186528 images
Loaded: 0.000079 seconds
5830: 0.007940, 0.010618 avg loss, 0.000010 rate, 1.864602 seconds, 186560 images
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.075133 seconds
5831: 0.016867, 0.011243 avg loss, 0.000010 rate, 1.573630 seconds, 186592 images
Loaded: 0.000041 seconds
5832: 0.008450, 0.010964 avg loss, 0.000010 rate, 1.642058 seconds, 186624 images
Loaded: 0.000042 seconds
5833: 0.014090, 0.011276 avg loss, 0.000010 rate, 1.649536 seconds, 186656 images
Loaded: 0.000034 seconds
5834: 0.018717, 0.012020 avg loss, 0.000010 rate, 1.650447 seconds, 186688 images
Loaded: 0.000045 seconds
5835: 0.010221, 0.011840 avg loss, 0.000010 rate, 1.649916 seconds, 186720 images
Loaded: 0.000070 seconds
5836: 0.010677, 0.011724 avg loss, 0.000010 rate, 1.646890 seconds, 186752 images
Loaded: 0.000055 seconds
5837: 0.014675, 0.012019 avg loss, 0.000010 rate, 1.636965 seconds, 186784 images
Loaded: 0.000041 seconds
5838: 0.011010, 0.011918 avg loss, 0.000010 rate, 1.642751 seconds, 186816 images
Loaded: 0.000041 seconds
5839: 0.011844, 0.011911 avg loss, 0.000010 rate, 1.642439 seconds, 186848 images
Loaded: 0.000043 seconds
5840: 0.009859, 0.011706 avg loss, 0.000010 rate, 1.651795 seconds, 186880 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000066 seconds
5841: 0.012840, 0.011819 avg loss, 0.000010 rate, 1.793350 seconds, 186912 images
Loaded: 0.000072 seconds
5842: 0.005026, 0.011140 avg loss, 0.000010 rate, 1.860402 seconds, 186944 images
Loaded: 0.000048 seconds
5843: 0.005597, 0.010585 avg loss, 0.000010 rate, 1.862202 seconds, 186976 images
Loaded: 0.000052 seconds
5844: 0.008608, 0.010388 avg loss, 0.000010 rate, 1.863802 seconds, 187008 images
Loaded: 0.000041 seconds
5845: 0.011490, 0.010498 avg loss, 0.000010 rate, 1.867829 seconds, 187040 images
Loaded: 0.000063 seconds
5846: 0.011728, 0.010621 avg loss, 0.000010 rate, 1.868322 seconds, 187072 images
Loaded: 0.000051 seconds
5847: 0.008465, 0.010405 avg loss, 0.000010 rate, 1.860274 seconds, 187104 images
Loaded: 0.000049 seconds
5848: 0.011848, 0.010550 avg loss, 0.000010 rate, 1.871162 seconds, 187136 images
Loaded: 0.000072 seconds
5849: 0.005833, 0.010078 avg loss, 0.000010 rate, 1.869753 seconds, 187168 images
Loaded: 0.000068 seconds
5850: 0.007316, 0.009802 avg loss, 0.000010 rate, 1.864882 seconds, 187200 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000066 seconds
5851: 0.007951, 0.009617 avg loss, 0.000010 rate, 2.018924 seconds, 187232 images
Loaded: 0.000044 seconds
5852: 0.008686, 0.009524 avg loss, 0.000010 rate, 2.103235 seconds, 187264 images
Loaded: 0.000068 seconds
5853: 0.007941, 0.009365 avg loss, 0.000010 rate, 2.102681 seconds, 187296 images
Loaded: 0.000064 seconds
5854: 0.007261, 0.009155 avg loss, 0.000010 rate, 2.108471 seconds, 187328 images
Loaded: 0.000068 seconds
5855: 0.008152, 0.009055 avg loss, 0.000010 rate, 2.107153 seconds, 187360 images
Loaded: 0.000042 seconds
5856: 0.006538, 0.008803 avg loss, 0.000010 rate, 2.107578 seconds, 187392 images
Loaded: 0.000046 seconds
5857: 0.015423, 0.009465 avg loss, 0.000010 rate, 2.107596 seconds, 187424 images
Loaded: 0.000039 seconds
5858: 0.006797, 0.009198 avg loss, 0.000010 rate, 2.101577 seconds, 187456 images
Loaded: 0.000037 seconds
5859: 0.006934, 0.008972 avg loss, 0.000010 rate, 2.103063 seconds, 187488 images
Loaded: 0.000041 seconds
5860: 0.013685, 0.009443 avg loss, 0.000010 rate, 2.102973 seconds, 187520 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.181041 seconds
5861: 0.012586, 0.009757 avg loss, 0.000010 rate, 2.028508 seconds, 187552 images
Loaded: 0.000041 seconds
5862: 0.007115, 0.009493 avg loss, 0.000010 rate, 2.108231 seconds, 187584 images
Loaded: 0.000090 seconds
5863: 0.012567, 0.009800 avg loss, 0.000010 rate, 2.106126 seconds, 187616 images
Loaded: 0.000046 seconds
5864: 0.007051, 0.009526 avg loss, 0.000010 rate, 2.103661 seconds, 187648 images
Loaded: 0.000043 seconds
5865: 0.010758, 0.009649 avg loss, 0.000010 rate, 2.103367 seconds, 187680 images
Loaded: 0.000039 seconds
5866: 0.006127, 0.009297 avg loss, 0.000010 rate, 2.110953 seconds, 187712 images
Loaded: 0.000041 seconds
5867: 0.008674, 0.009234 avg loss, 0.000010 rate, 2.112565 seconds, 187744 images
Loaded: 0.000070 seconds
5868: 0.006413, 0.008952 avg loss, 0.000010 rate, 2.106294 seconds, 187776 images
Loaded: 0.000046 seconds
5869: 0.007644, 0.008821 avg loss, 0.000010 rate, 2.109378 seconds, 187808 images
Loaded: 0.000071 seconds
5870: 0.004860, 0.008425 avg loss, 0.000010 rate, 2.110725 seconds, 187840 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.136003 seconds
5871: 0.005828, 0.008166 avg loss, 0.000010 rate, 2.025201 seconds, 187872 images
Loaded: 0.000068 seconds
5872: 0.007021, 0.008051 avg loss, 0.000010 rate, 2.109960 seconds, 187904 images
Loaded: 0.000048 seconds
5873: 0.005770, 0.007823 avg loss, 0.000010 rate, 2.110758 seconds, 187936 images
Loaded: 0.000067 seconds
5874: 0.006446, 0.007685 avg loss, 0.000010 rate, 2.107710 seconds, 187968 images
Loaded: 0.000046 seconds
5875: 0.004657, 0.007383 avg loss, 0.000010 rate, 2.111263 seconds, 188000 images
Loaded: 0.000040 seconds
5876: 0.005750, 0.007219 avg loss, 0.000010 rate, 2.107780 seconds, 188032 images
Loaded: 0.000042 seconds
5877: 0.006500, 0.007147 avg loss, 0.000010 rate, 2.118126 seconds, 188064 images
Loaded: 0.000065 seconds
5878: 0.006732, 0.007106 avg loss, 0.000010 rate, 2.108352 seconds, 188096 images
Loaded: 0.000043 seconds
5879: 0.007516, 0.007147 avg loss, 0.000010 rate, 2.108000 seconds, 188128 images
Loaded: 0.000053 seconds
5880: 0.006622, 0.007094 avg loss, 0.000010 rate, 2.110875 seconds, 188160 images
Resizing
416 x 416
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000052 seconds
5881: 0.008234, 0.007208 avg loss, 0.000010 rate, 2.389297 seconds, 188192 images
Loaded: 0.000045 seconds
5882: 0.005704, 0.007058 avg loss, 0.000010 rate, 2.481578 seconds, 188224 images
Loaded: 0.000040 seconds
5883: 0.005585, 0.006911 avg loss, 0.000010 rate, 2.483763 seconds, 188256 images
Loaded: 0.000069 seconds
5884: 0.006814, 0.006901 avg loss, 0.000010 rate, 2.485151 seconds, 188288 images
Loaded: 0.000055 seconds
5885: 0.005904, 0.006801 avg loss, 0.000010 rate, 2.480325 seconds, 188320 images
Loaded: 0.000042 seconds
5886: 0.005040, 0.006625 avg loss, 0.000010 rate, 2.482766 seconds, 188352 images
Loaded: 0.000067 seconds
5887: 0.006414, 0.006604 avg loss, 0.000010 rate, 2.484046 seconds, 188384 images
Loaded: 0.000064 seconds
5888: 0.007050, 0.006649 avg loss, 0.000010 rate, 2.490303 seconds, 188416 images
Loaded: 0.000046 seconds
5889: 0.010133, 0.006997 avg loss, 0.000010 rate, 2.486998 seconds, 188448 images
Loaded: 0.000044 seconds
5890: 0.003316, 0.006629 avg loss, 0.000010 rate, 2.484042 seconds, 188480 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000041 seconds
5891: 0.005189, 0.006485 avg loss, 0.000010 rate, 4.028205 seconds, 188512 images
Loaded: 0.000044 seconds
5892: 0.006553, 0.006492 avg loss, 0.000010 rate, 4.171325 seconds, 188544 images
Loaded: 0.000039 seconds
5893: 0.006142, 0.006457 avg loss, 0.000010 rate, 4.182443 seconds, 188576 images
Loaded: 0.000055 seconds
5894: 0.004652, 0.006276 avg loss, 0.000010 rate, 4.190545 seconds, 188608 images
Loaded: 0.000039 seconds
5895: 0.004489, 0.006098 avg loss, 0.000010 rate, 4.171586 seconds, 188640 images
Loaded: 0.000037 seconds
5896: 0.005535, 0.006041 avg loss, 0.000010 rate, 4.176603 seconds, 188672 images
Loaded: 0.000044 seconds
5897: 0.006799, 0.006117 avg loss, 0.000010 rate, 4.180616 seconds, 188704 images
Loaded: 0.000050 seconds
5898: 0.005926, 0.006098 avg loss, 0.000010 rate, 4.175778 seconds, 188736 images
Loaded: 0.000037 seconds
5899: 0.013232, 0.006811 avg loss, 0.000010 rate, 4.178586 seconds, 188768 images
Loaded: 0.000098 seconds
5900: 0.004756, 0.006606 avg loss, 0.000010 rate, 4.172848 seconds, 188800 images
Saving weights to backup/custom-train-yolo_last.weights
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000045 seconds
5901: 0.014385, 0.007384 avg loss, 0.000010 rate, 1.578540 seconds, 188832 images
Loaded: 0.000041 seconds
5902: 0.079146, 0.014560 avg loss, 0.000010 rate, 1.647149 seconds, 188864 images
Loaded: 0.000040 seconds
5903: 0.014602, 0.014564 avg loss, 0.000010 rate, 1.657139 seconds, 188896 images
Loaded: 0.000040 seconds
5904: 0.019005, 0.015008 avg loss, 0.000010 rate, 1.644495 seconds, 188928 images
Loaded: 0.000039 seconds
5905: 0.032294, 0.016737 avg loss, 0.000010 rate, 1.647701 seconds, 188960 images
Loaded: 0.000046 seconds
5906: 0.008163, 0.015879 avg loss, 0.000010 rate, 1.648732 seconds, 188992 images
Loaded: 0.000068 seconds
5907: 0.011989, 0.015490 avg loss, 0.000010 rate, 1.647219 seconds, 189024 images
Loaded: 0.000048 seconds
5908: 0.015163, 0.015458 avg loss, 0.000010 rate, 1.650336 seconds, 189056 images
Loaded: 0.000050 seconds
5909: 0.010262, 0.014938 avg loss, 0.000010 rate, 1.648806 seconds, 189088 images
Loaded: 0.000041 seconds
5910: 0.009404, 0.014385 avg loss, 0.000010 rate, 1.648098 seconds, 189120 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000044 seconds
5911: 0.008766, 0.013823 avg loss, 0.000010 rate, 1.803797 seconds, 189152 images
Loaded: 0.000047 seconds
5912: 0.008187, 0.013259 avg loss, 0.000010 rate, 1.872749 seconds, 189184 images
Loaded: 0.000042 seconds
5913: 0.008884, 0.012822 avg loss, 0.000010 rate, 1.868290 seconds, 189216 images
Loaded: 0.000043 seconds
5914: 0.012262, 0.012766 avg loss, 0.000010 rate, 1.869574 seconds, 189248 images
Loaded: 0.000082 seconds
5915: 0.018706, 0.013360 avg loss, 0.000010 rate, 1.869809 seconds, 189280 images
Loaded: 0.000069 seconds
5916: 0.010444, 0.013068 avg loss, 0.000010 rate, 1.876273 seconds, 189312 images
Loaded: 0.000089 seconds
5917: 0.009803, 0.012742 avg loss, 0.000010 rate, 1.871944 seconds, 189344 images
Loaded: 0.000058 seconds
5918: 0.010920, 0.012560 avg loss, 0.000010 rate, 1.865833 seconds, 189376 images
Loaded: 0.000041 seconds
5919: 0.010515, 0.012355 avg loss, 0.000010 rate, 1.867631 seconds, 189408 images
Loaded: 0.000041 seconds
5920: 0.011373, 0.012257 avg loss, 0.000010 rate, 1.865590 seconds, 189440 images
Resizing
320 x 320
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.207363 seconds
5921: 0.018774, 0.012909 avg loss, 0.000010 rate, 1.580077 seconds, 189472 images
Loaded: 0.000034 seconds
5922: 0.022767, 0.013894 avg loss, 0.000010 rate, 1.650352 seconds, 189504 images
Loaded: 0.000067 seconds
5923: 0.009157, 0.013421 avg loss, 0.000010 rate, 1.649726 seconds, 189536 images
Loaded: 0.000047 seconds
5924: 0.011046, 0.013183 avg loss, 0.000010 rate, 1.645709 seconds, 189568 images
Loaded: 0.000054 seconds
5925: 0.009307, 0.012796 avg loss, 0.000010 rate, 1.643926 seconds, 189600 images
Loaded: 0.000040 seconds
5926: 0.018209, 0.013337 avg loss, 0.000010 rate, 1.645726 seconds, 189632 images
Loaded: 0.000042 seconds
5927: 0.019032, 0.013906 avg loss, 0.000010 rate, 1.648842 seconds, 189664 images
Loaded: 0.000045 seconds
5928: 0.012172, 0.013733 avg loss, 0.000010 rate, 1.643539 seconds, 189696 images
Loaded: 0.000044 seconds
5929: 0.019621, 0.014322 avg loss, 0.000010 rate, 1.653563 seconds, 189728 images
Loaded: 0.000033 seconds
5930: 0.012199, 0.014109 avg loss, 0.000010 rate, 1.645201 seconds, 189760 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5931: 0.012561, 0.013955 avg loss, 0.000010 rate, 1.789404 seconds, 189792 images
Loaded: 0.000039 seconds
5932: 0.008353, 0.013394 avg loss, 0.000010 rate, 1.873012 seconds, 189824 images
Loaded: 0.000040 seconds
5933: 0.010876, 0.013143 avg loss, 0.000010 rate, 1.890003 seconds, 189856 images
Loaded: 0.000043 seconds
5934: 0.007527, 0.012581 avg loss, 0.000010 rate, 1.868885 seconds, 189888 images
Loaded: 0.000038 seconds
5935: 0.008632, 0.012186 avg loss, 0.000010 rate, 1.871384 seconds, 189920 images
Loaded: 0.000052 seconds
5936: 0.009198, 0.011887 avg loss, 0.000010 rate, 1.874765 seconds, 189952 images
Loaded: 0.000067 seconds
5937: 0.009816, 0.011680 avg loss, 0.000010 rate, 1.869917 seconds, 189984 images
Loaded: 0.000041 seconds
5938: 0.011389, 0.011651 avg loss, 0.000010 rate, 1.872658 seconds, 190016 images
Loaded: 0.000034 seconds
5939: 0.007349, 0.011221 avg loss, 0.000010 rate, 1.870539 seconds, 190048 images
Loaded: 0.000081 seconds
5940: 0.007573, 0.010856 avg loss, 0.000010 rate, 1.876417 seconds, 190080 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000044 seconds
5941: 0.008920, 0.010663 avg loss, 0.000010 rate, 2.023439 seconds, 190112 images
Loaded: 0.000047 seconds
5942: 0.010106, 0.010607 avg loss, 0.000010 rate, 2.103206 seconds, 190144 images
Loaded: 0.000051 seconds
5943: 0.013476, 0.010894 avg loss, 0.000010 rate, 2.114552 seconds, 190176 images
Loaded: 0.000046 seconds
5944: 0.004912, 0.010296 avg loss, 0.000010 rate, 2.105952 seconds, 190208 images
Loaded: 0.000042 seconds
5945: 0.025044, 0.011770 avg loss, 0.000010 rate, 2.112571 seconds, 190240 images
Loaded: 0.000038 seconds
5946: 0.006173, 0.011211 avg loss, 0.000010 rate, 2.105017 seconds, 190272 images
Loaded: 0.000044 seconds
5947: 0.006351, 0.010725 avg loss, 0.000010 rate, 2.102808 seconds, 190304 images
Loaded: 0.000037 seconds
5948: 0.008141, 0.010466 avg loss, 0.000010 rate, 2.116101 seconds, 190336 images
Loaded: 0.000055 seconds
5949: 0.007005, 0.010120 avg loss, 0.000010 rate, 2.112431 seconds, 190368 images
Loaded: 0.000062 seconds
5950: 0.006397, 0.009748 avg loss, 0.000010 rate, 2.110145 seconds, 190400 images
Resizing
544 x 544
try to allocate additional workspace_size = 87.01 MB
CUDA allocate done!
Loaded: 0.000041 seconds
5951: 0.016378, 0.010411 avg loss, 0.000010 rate, 4.020749 seconds, 190432 images
Loaded: 0.000042 seconds
5952: 0.008127, 0.010183 avg loss, 0.000010 rate, 4.167929 seconds, 190464 images
Loaded: 0.000037 seconds
5953: 0.007467, 0.009911 avg loss, 0.000010 rate, 4.163035 seconds, 190496 images
Loaded: 0.000064 seconds
5954: 0.003870, 0.009307 avg loss, 0.000010 rate, 4.168795 seconds, 190528 images
Loaded: 0.000067 seconds
5955: 0.006336, 0.009010 avg loss, 0.000010 rate, 4.167081 seconds, 190560 images
Loaded: 0.000044 seconds
5956: 0.006723, 0.008781 avg loss, 0.000010 rate, 4.170363 seconds, 190592 images
Loaded: 0.000068 seconds
5957: 0.006746, 0.008578 avg loss, 0.000010 rate, 4.155698 seconds, 190624 images
Loaded: 0.000039 seconds
5958: 0.007667, 0.008487 avg loss, 0.000010 rate, 4.168002 seconds, 190656 images
Loaded: 0.000052 seconds
5959: 0.007993, 0.008437 avg loss, 0.000010 rate, 4.159550 seconds, 190688 images
Loaded: 0.000056 seconds
5960: 0.005095, 0.008103 avg loss, 0.000010 rate, 4.172237 seconds, 190720 images
Resizing
480 x 480
try to allocate additional workspace_size = 67.74 MB
CUDA allocate done!
Loaded: 0.000036 seconds
5961: 0.003889, 0.007682 avg loss, 0.000010 rate, 2.940572 seconds, 190752 images
Loaded: 0.000038 seconds
5962: 0.009153, 0.007829 avg loss, 0.000010 rate, 3.038876 seconds, 190784 images
Loaded: 0.000038 seconds
5963: 0.003971, 0.007443 avg loss, 0.000010 rate, 3.028778 seconds, 190816 images
Loaded: 0.000064 seconds
5964: 0.003186, 0.007017 avg loss, 0.000010 rate, 3.035843 seconds, 190848 images
Loaded: 0.000040 seconds
5965: 0.016149, 0.007930 avg loss, 0.000010 rate, 3.032544 seconds, 190880 images
Loaded: 0.000039 seconds
5966: 0.003902, 0.007528 avg loss, 0.000010 rate, 3.040726 seconds, 190912 images
Loaded: 0.000040 seconds
5967: 0.004236, 0.007198 avg loss, 0.000010 rate, 3.033031 seconds, 190944 images
Loaded: 0.000056 seconds
5968: 0.005296, 0.007008 avg loss, 0.000010 rate, 3.042611 seconds, 190976 images
Loaded: 0.000063 seconds
5969: 0.005500, 0.006857 avg loss, 0.000010 rate, 3.027601 seconds, 191008 images
Loaded: 0.000039 seconds
5970: 0.004322, 0.006604 avg loss, 0.000010 rate, 3.028259 seconds, 191040 images
Resizing
384 x 384
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000042 seconds
5971: 0.015590, 0.007502 avg loss, 0.000010 rate, 2.021047 seconds, 191072 images
Loaded: 0.000060 seconds
5972: 0.004512, 0.007203 avg loss, 0.000010 rate, 2.100367 seconds, 191104 images
Loaded: 0.000039 seconds
5973: 0.005498, 0.007033 avg loss, 0.000010 rate, 2.099969 seconds, 191136 images
Loaded: 0.000042 seconds
5974: 0.012377, 0.007567 avg loss, 0.000010 rate, 2.101347 seconds, 191168 images
Loaded: 0.000041 seconds
5975: 0.004478, 0.007258 avg loss, 0.000010 rate, 2.100580 seconds, 191200 images
Loaded: 0.000064 seconds
5976: 0.005541, 0.007087 avg loss, 0.000010 rate, 2.104113 seconds, 191232 images
Loaded: 0.000039 seconds
5977: 0.007090, 0.007087 avg loss, 0.000010 rate, 2.106511 seconds, 191264 images
Loaded: 0.000040 seconds
5978: 0.006247, 0.007003 avg loss, 0.000010 rate, 2.105696 seconds, 191296 images
Loaded: 0.000036 seconds
5979: 0.007483, 0.007051 avg loss, 0.000010 rate, 2.103055 seconds, 191328 images
Loaded: 0.000050 seconds
5980: 0.010193, 0.007365 avg loss, 0.000010 rate, 2.114088 seconds, 191360 images
Resizing
576 x 576
try to allocate additional workspace_size = 97.55 MB
CUDA allocate done!
Loaded: 0.000037 seconds
5981: 0.011791, 0.007808 avg loss, 0.000010 rate, 4.341319 seconds, 191392 images
Loaded: 0.000063 seconds
5982: 0.011639, 0.008191 avg loss, 0.000010 rate, 4.531151 seconds, 191424 images
Loaded: 0.000072 seconds
5983: 0.015755, 0.008947 avg loss, 0.000010 rate, 4.497216 seconds, 191456 images
Loaded: 0.000065 seconds
5984: 0.005190, 0.008572 avg loss, 0.000010 rate, 4.515459 seconds, 191488 images
Loaded: 0.000077 seconds
5985: 0.008269, 0.008541 avg loss, 0.000010 rate, 4.498967 seconds, 191520 images
Loaded: 0.000068 seconds
5986: 0.023284, 0.010016 avg loss, 0.000010 rate, 4.503157 seconds, 191552 images
Loaded: 0.000066 seconds
5987: 0.008468, 0.009861 avg loss, 0.000010 rate, 4.500605 seconds, 191584 images
Loaded: 0.000075 seconds
5988: 0.006011, 0.009476 avg loss, 0.000010 rate, 4.506352 seconds, 191616 images
Loaded: 0.000046 seconds
5989: 0.003723, 0.008900 avg loss, 0.000010 rate, 4.502927 seconds, 191648 images
Loaded: 0.000073 seconds
5990: 0.008811, 0.008892 avg loss, 0.000010 rate, 4.516789 seconds, 191680 images
Resizing
352 x 352
try to allocate additional workspace_size = 52.43 MB
CUDA allocate done!
Loaded: 0.000043 seconds
5991: 0.009928, 0.008995 avg loss, 0.000010 rate, 1.797979 seconds, 191712 images
Loaded: 0.000044 seconds
5992: 0.014109, 0.009507 avg loss, 0.000010 rate, 1.864415 seconds, 191744 images
Loaded: 0.000042 seconds
5993: 0.011723, 0.009728 avg loss, 0.000010 rate, 1.871586 seconds, 191776 images
Loaded: 0.000078 seconds
5994: 0.022874, 0.011043 avg loss, 0.000010 rate, 1.860330 seconds, 191808 images
Loaded: 0.000029 seconds
5995: 0.008355, 0.010774 avg loss, 0.000010 rate, 1.870388 seconds, 191840 images
Loaded: 0.000040 seconds
5996: 0.006462, 0.010343 avg loss, 0.000010 rate, 1.867876 seconds, 191872 images
Loaded: 0.000060 seconds
5997: 0.011011, 0.010410 avg loss, 0.000010 rate, 1.871816 seconds, 191904 images
Loaded: 0.000041 seconds
5998: 0.006265, 0.009995 avg loss, 0.000010 rate, 1.873397 seconds, 191936 images
Loaded: 0.000063 seconds
5999: 0.007348, 0.009730 avg loss, 0.000010 rate, 1.865390 seconds, 191968 images
Loaded: 0.000041 seconds
6000: 0.011870, 0.009944 avg loss, 0.000010 rate, 1.793497 seconds, 192000 images
Saving weights to backup/custom-train-yolo_6000.weights
Saving weights to backup/custom-train-yolo_last.weights
Saving weights to backup/custom-train-yolo_final.weights
In [46]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_last.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_last.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 90.14 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 1.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [38]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_final.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_final.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 90.14 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 0.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [45]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_6000.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_6000.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 90.14 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 1.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [44]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_5000.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_5000.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 90.11 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 0.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [43]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_4000.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_4000.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 88.34 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 1.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [42]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_3000.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_3000.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 86.40 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 0.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [40]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_2000.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_2000.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 5, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 50.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 100.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 91.33 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 1.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [41]:
!./darknet detector map custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_1000.weights
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_1000.weights... seen 64 Done! calculation mAP (mean average precision)... 2 detections_count = 6, unique_truth_count = 5 class_id = 0, name = chocolate, ap = 100.00% (TP = 1, FP = 0) class_id = 1, name = gum, ap = 100.00% (TP = 2, FP = 0) class_id = 2, name = milk, ap = 50.00% (TP = 2, FP = 0) for thresh = 0.25, precision = 1.00, recall = 1.00, F1-score = 1.00 for thresh = 0.25, TP = 5, FP = 0, FN = 0, average IoU = 80.99 % IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.833333, or 83.33 % Total Detection Time: 0.000000 Seconds Set -points flag: `-points 101` for MS COCO `-points 11` for PascalVOC 2007 (uncomment `difficult` in voc.data) `-points 0` (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset
In [ ]:
if os.path.exists('./backup'):
!cp -r ./backup "/content/drive/MyDrive/darknet"
if not os.path.exists('./backup'):
os.makedirs('./backup')
!cp -r "/content/drive/MyDrive/darknet/backup/custom-train-yolo_final.weights" ./backup
In [30]:
!ls -al ./backup
total 1924624 drwxr-xr-x 3 root root 4096 Dec 24 07:26 . drwxr-xr-x 8 root root 4096 Dec 24 02:21 .. -rw-r--r-- 1 root root 246348468 Dec 24 03:10 custom-train-yolo_1000.weights -rw-r--r-- 1 root root 246348468 Dec 24 04:01 custom-train-yolo_2000.weights -rw-r--r-- 1 root root 246348468 Dec 24 04:51 custom-train-yolo_3000.weights -rw-r--r-- 1 root root 246348468 Dec 24 05:41 custom-train-yolo_4000.weights -rw-r--r-- 1 root root 246348468 Dec 24 06:31 custom-train-yolo_5000.weights -rw-r--r-- 1 root root 246348468 Dec 24 07:20 custom-train-yolo_6000.weights -rw-r--r-- 1 root root 246348468 Dec 24 07:20 custom-train-yolo_final.weights -rw-r--r-- 1 root root 246348468 Dec 24 07:20 custom-train-yolo_last.weights drwxr-xr-x 2 root root 4096 Dec 24 07:26 .ipynb_checkpoints
In [39]:
!./darknet detector test custom/custom_data.data custom/custom-train-yolo.cfg backup/custom-train-yolo_final.weights custom/pack5.jpg -dont-show
imShow('predictions.jpg')
layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BF 1 conv 64 3 x 3 / 2 416 x 416 x 32 -> 208 x 208 x 64 1.595 BF 2 conv 32 1 x 1 / 1 208 x 208 x 64 -> 208 x 208 x 32 0.177 BF 3 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BF 4 Shortcut Layer: 1 5 conv 128 3 x 3 / 2 208 x 208 x 64 -> 104 x 104 x 128 1.595 BF 6 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 7 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 8 Shortcut Layer: 5 9 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BF 10 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BF 11 Shortcut Layer: 8 12 conv 256 3 x 3 / 2 104 x 104 x 128 -> 52 x 52 x 256 1.595 BF 13 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 14 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 15 Shortcut Layer: 12 16 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 17 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 18 Shortcut Layer: 15 19 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 20 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 21 Shortcut Layer: 18 22 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 23 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 24 Shortcut Layer: 21 25 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 26 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 27 Shortcut Layer: 24 28 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 29 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 30 Shortcut Layer: 27 31 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 32 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 33 Shortcut Layer: 30 34 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 35 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 36 Shortcut Layer: 33 37 conv 512 3 x 3 / 2 52 x 52 x 256 -> 26 x 26 x 512 1.595 BF 38 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 39 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 40 Shortcut Layer: 37 41 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 42 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 43 Shortcut Layer: 40 44 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 45 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 46 Shortcut Layer: 43 47 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 48 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 49 Shortcut Layer: 46 50 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 51 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 52 Shortcut Layer: 49 53 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 54 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 55 Shortcut Layer: 52 56 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 57 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 58 Shortcut Layer: 55 59 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 60 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 61 Shortcut Layer: 58 62 conv 1024 3 x 3 / 2 26 x 26 x 512 -> 13 x 13 x1024 1.595 BF 63 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 64 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 65 Shortcut Layer: 62 66 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 67 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 68 Shortcut Layer: 65 69 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 70 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 71 Shortcut Layer: 68 72 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 73 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 74 Shortcut Layer: 71 75 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 76 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 77 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 78 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 79 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BF 80 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BF 81 conv 24 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 24 0.008 BF 82 yolo 83 route 79 84 conv 256 1 x 1 / 1 13 x 13 x 512 -> 13 x 13 x 256 0.044 BF 85 upsample 2x 13 x 13 x 256 -> 26 x 26 x 256 86 route 85 61 87 conv 256 1 x 1 / 1 26 x 26 x 768 -> 26 x 26 x 256 0.266 BF 88 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 89 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 90 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 91 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BF 92 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BF 93 conv 24 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 24 0.017 BF 94 yolo 95 route 91 96 conv 128 1 x 1 / 1 26 x 26 x 256 -> 26 x 26 x 128 0.044 BF 97 upsample 2x 26 x 26 x 128 -> 52 x 52 x 128 98 route 97 36 99 conv 128 1 x 1 / 1 52 x 52 x 384 -> 52 x 52 x 128 0.266 BF 100 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 101 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 102 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 103 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BF 104 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BF 105 conv 24 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 24 0.033 BF 106 yolo Total BFLOPS 65.304 Allocate additional workspace_size = 52.43 MB Loading weights from backup/custom-train-yolo_final.weights... seen 64 Done! custom/pack5.jpg: Predicted in 40.971000 milli-seconds. milk: 100% gum: 100% Unable to init server: Could not connect: Connection refused (predictions:30392): Gtk-WARNING **: 07:35:16.884: cannot open display:
'OpenCV > 머신비전 - 이미지 디텍팅' 카테고리의 다른 글
| 훈련 데이터 카메라 영상 적용 (0) | 2020.12.28 |
|---|---|
| YOLO (gum, milk, chocolate) 추가 자료 (0) | 2020.12.24 |
| YOLO (apple, orange / 교육) (0) | 2020.12.24 |
| YOLO에 관한 이론 (0) | 2020.12.24 |
| labelImg (0) | 2020.12.24 |
'OpenCV/머신비전 - 이미지 디텍팅' Related Articles
-
훈련 데이터 카메라 영상 적용
2020.12.28
-
YOLO (gum, milk, chocolate) 추가 자료
2020.12.24
- YOLO (apple, orange / 교육) 2020.12.24
-
YOLO에 관한 이론
2020.12.24
Comments