Run Tensorflow object detecion with Nvidia GTX650 Ti (2GB)? -
is there way let run object detection 2gb graphic card? have 24gb dd3 ram on main board, can't use gpu?
i did try add session_config.gpu_options.allow_growth=true in trainer.py don't help. seems graphic card doesn't have enough memory.
cardinfos:
0, name: geforce gtx 650, pci bus id: 0000:01:00.0) [name: "/cpu:0" device_type: "cpu" memory_limit: 268435456 locality { } incarnation: 4876955943962853047 , name: "/gpu:0" device_type: "gpu" memory_limit: 1375862784 locality { bus_id: 1 } incarnation: 4236842880144430162 physical_device_desc: "device: 0, name: geforce gtx 650, pci bus id: 0000:01:00.0" ]
train.py output:
limit: 219414528 inuse: 192361216 maxinuse: 192483072 numallocs: 6030 maxallocsize: 6131712 2017-09-13 13:47:13.429510: w tensorflow/core/common_runtime/bfc_allocator.cc:277] ****************************************************************************************____________ 2017-09-13 13:47:13.481829: w tensorflow/core/framework/op_kernel.cc:1192] internal: dst tensor not initialized. [[node: prefetch_queue_dequeue/_5471 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] info:tensorflow:error reported coordinator: <class 'tensorflow.python.framework.errors_impl.internalerror'>, dst tensor not initialized. [[node: prefetch_queue_dequeue/_5471 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2017-09-13 13:47:13.955327: w tensorflow/core/framework/op_kernel.cc:1192] internal: dst tensor not initialized. [[node: prefetch_queue_dequeue/_299 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3432_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2017-09-13 13:47:13.956056: w tensorflow/core/framework/op_kernel.cc:1192] internal: dst tensor not initialized. [[node: prefetch_queue_dequeue/_299 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3432_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] traceback (most recent call last): file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(*args) file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn status, run_metadata) file "/usr/lib/python3.5/contextlib.py", line 66, in __exit__ next(self.gen) file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.tf_getcode(status)) tensorflow.python.framework.errors_impl.internalerror: dst tensor not initialized. [[node: prefetch_queue_dequeue/_5471 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] during handling of above exception, exception occurred: traceback (most recent call last): file "train.py", line 198, in <module> tf.app.run() file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) file "train.py", line 194, in main worker_job_name, is_chief, flags.train_dir) file "/home/dee/documents/projects/tensor/models/object_detection/trainer.py", line 297, in train saver=saver) file "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 755, in train sess, train_op, global_step, train_step_kwargs) file "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 488, in train_step run_metadata=run_metadata) file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run run_metadata_ptr) file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run feed_dict_tensor, options, run_metadata) file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run options, run_metadata) file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.internalerror: dst tensor not initialized.
indeed dst tensor not initialized
message indicates gpu runs out of memory. can try decrease batch size minimum , decrease resolution of images feeding model. try use ssd mobilenet model, because lightweight.
to answer second part of question: have thought modern gpus run hybrid mode drivers/gpu start streaming resources system ram on pcie bus make "missing" vram. since system ram 3-5x slower gddr5 higher latency, running out of vram translate significant performance loss. faced same issue on gtx 1060 6gb vram, cuda process crashed because ran out of gpu.
Comments
Post a Comment