Run Tensorflow object detecion with Nvidia GTX650 Ti (2GB)? -


is there way let run object detection 2gb graphic card? have 24gb dd3 ram on main board, can't use gpu?

i did try add session_config.gpu_options.allow_growth=true in trainer.py don't help. seems graphic card doesn't have enough memory.

cardinfos:

0, name: geforce gtx 650, pci bus id: 0000:01:00.0) [name: "/cpu:0" device_type: "cpu" memory_limit: 268435456 locality { } incarnation: 4876955943962853047 , name: "/gpu:0" device_type: "gpu" memory_limit: 1375862784 locality {   bus_id: 1 } incarnation: 4236842880144430162 physical_device_desc: "device: 0, name: geforce gtx 650, pci bus id: 0000:01:00.0" ] 

train.py output:

limit:                   219414528 inuse:                   192361216 maxinuse:                192483072 numallocs:                    6030 maxallocsize:              6131712  2017-09-13 13:47:13.429510: w tensorflow/core/common_runtime/bfc_allocator.cc:277] ****************************************************************************************____________ 2017-09-13 13:47:13.481829: w tensorflow/core/framework/op_kernel.cc:1192] internal: dst tensor not initialized.      [[node: prefetch_queue_dequeue/_5471 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] info:tensorflow:error reported coordinator: <class 'tensorflow.python.framework.errors_impl.internalerror'>, dst tensor not initialized.      [[node: prefetch_queue_dequeue/_5471 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2017-09-13 13:47:13.955327: w tensorflow/core/framework/op_kernel.cc:1192] internal: dst tensor not initialized.      [[node: prefetch_queue_dequeue/_299 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3432_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] 2017-09-13 13:47:13.956056: w tensorflow/core/framework/op_kernel.cc:1192] internal: dst tensor not initialized.      [[node: prefetch_queue_dequeue/_299 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_3432_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]] traceback (most recent call last):   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call     return fn(*args)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1306, in _run_fn     status, run_metadata)   file "/usr/lib/python3.5/contextlib.py", line 66, in __exit__     next(self.gen)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status     pywrap_tensorflow.tf_getcode(status)) tensorflow.python.framework.errors_impl.internalerror: dst tensor not initialized.      [[node: prefetch_queue_dequeue/_5471 = _recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_5476_prefetch_queue_dequeue", tensor_type=dt_float, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]  during handling of above exception, exception occurred:  traceback (most recent call last):   file "train.py", line 198, in <module>     tf.app.run()   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 48, in run     _sys.exit(main(_sys.argv[:1] + flags_passthrough))   file "train.py", line 194, in main     worker_job_name, is_chief, flags.train_dir)   file "/home/dee/documents/projects/tensor/models/object_detection/trainer.py", line 297, in train     saver=saver)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 755, in train     sess, train_op, global_step, train_step_kwargs)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 488, in train_step     run_metadata=run_metadata)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 895, in run     run_metadata_ptr)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1124, in _run     feed_dict_tensor, options, run_metadata)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1321, in _do_run     options, run_metadata)   file "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1340, in _do_call     raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.internalerror: dst tensor not initialized. 

indeed dst tensor not initialized message indicates gpu runs out of memory. can try decrease batch size minimum , decrease resolution of images feeding model. try use ssd mobilenet model, because lightweight.

to answer second part of question: have thought modern gpus run hybrid mode drivers/gpu start streaming resources system ram on pcie bus make "missing" vram. since system ram 3-5x slower gddr5 higher latency, running out of vram translate significant performance loss. faced same issue on gtx 1060 6gb vram, cuda process crashed because ran out of gpu.


Comments

Popular posts from this blog

ZeroMQ on Windows, with Qt Creator -

unity3d - Unity SceneManager.LoadScene quits application -

python - Error while using APScheduler: 'NoneType' object has no attribute 'now' -