-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] RuntimeError: /io/build/temp.linux-x86_64-cpython-37/spconv/build/core_cc/src/csrc/sparse/all/SpconvOps/SpconvOps_get_indice_pairs.cc(65) not implemented for CPU ONLY build. #2061
Comments
Maybe you install the CPU-only spconv. Please install the spconv with CUDA. https://github.com/traveller59/spconv |
Ok, thank you, because my cuda version is 10.1.105, the latest spconv2.x supports duda10.2 at least, so I changed the original spconv version to 1.2.1. solved! |
After I replace it with spconv1.2.1, what effect will spconv display as False, and the result of training with pre-training weights is 0!!! 2022-11-28 23:59:39,479 - mmdet - INFO - Set random seed to 0, deterministic: False
|
Hi, we have support spconv1.x in mmcv, so you don't need to install spconv1.x. And can you use default mmcv's spconv to try again? |
And I suggest you check your env by PointPillars model. |
1. May I use when building a virtual environment: conda activate -n mmdtection3d python=3.7 2. When we set up the environment for testing, this error still popped up: **(mmdetection3d) ubuntu@ubuntu-ThinkStation-P920:~/zhangweili/mmdetection3d-master$ python demo/pcd_demo.py demo/data/kitti/kitti_000008.bin configs/second/hv_second_secfpn_6x8_80e_kitti-3d-car.py checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-car_20200620_230238-393f000c.pth --show May I ask how to solve this problem? Is the reason for this problem that all dependencies cannot be installed directly?
Name Version Build Channel_libgcc_mutex 0.1 main |
@VVsssssk @JingweiZhang12 @
TorchVision: 0.11.0+cu111 |
Is it done? I came across this issue recently. |
Prerequisite
Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
master branch https://github.com/open-mmlab/mmdetection3d
Environment
CUDA_HOME: /usr/local/cuda-10.1/
NVCC: Cuda compilation tools, release 10.1, V10.1.10
GCC: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
PyTorch: 1.8.0+cu101
PyTorch compiling details: PyTorch built with:
TorchVision: 0.9.0+cu101
OpenCV: 4.6.0
MMCV: 1.6.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMDetection: 2.25.3
MMSegmentation: 0.29.1
MMDetection3D: 1.0.0rc5+
spconv2.0: True
<<< conda initialize <<<
export CUDA_HOME=/usr/local/cuda-10.1/
export PATH=$PATH:/usr/local/cuda-10.1/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.1/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda-10.1/lib64
#cuda
export PATH=$PATH:/usr/local/cuda/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda/lib64
Reproduces the problem - code sample
2022-11-27 23:17:22,114 - mmdet - INFO - workflow: [('train', 1)], max: 40 epochs
2022-11-27 23:17:22,114 - mmdet - INFO - Checkpoints will be saved to /home/dell/zhangweili/mmdetection3d-master/tools/work_dirs/dv_mvx-fpn_second_secfpn_adamw_2x8_80e_kitti-3d-3class by HardDiskBackend.
2022-11-27 23:17:22.798964: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/cv2/../../lib64:/usr/local/cuda-9.0/lib64
2022-11-27 23:17:22.799090: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/cv2/../../lib64:/usr/local/cuda-9.0/lib64
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/apis/train.py", line 319, in train_detector
runner.run(data_loaders, cfg.workflow) # 启动
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in run_iter
**kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
losses = self(**data)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/conv.py", line 384, in forward
raise e
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/conv.py", line 375, in forward
self.transposed)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/ops.py", line 162, in get_indice_pairs
stream)
RuntimeError: /io/build/temp.linux-x86_64-cpython-37/spconv/build/core_cc/src/csrc/sparse/all/SpconvOps/SpconvOps_get_indice_pairs.cc(65)
not implemented for CPU ONLY build.
Reproduces the problem - command or script
run directly train.py
configs/mvxnet/dv_mvx-fp_second_secfpn_adamw_2x8_80e_kitti-3d-3class.py
Reproduces the problem - error message
2022-11-27 23:17:22,114 - mmdet - INFO - workflow: [('train', 1)], max: 40 epochs
2022-11-27 23:17:22,114 - mmdet - INFO - Checkpoints will be saved to /home/dell/zhangweili/mmdetection3d-master/tools/work_dirs/dv_mvx-fpn_second_secfpn_adamw_2x8_80e_kitti-3d-3class by HardDiskBackend.
2022-11-27 23:17:22.798964: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/cv2/../../lib64:/usr/local/cuda-9.0/lib64
2022-11-27 23:17:22.799090: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/cv2/../../lib64:/usr/local/cuda-9.0/lib64
2022-11-27 23:17:22.799101: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/home/dell/zhangweili/mmdetection3d-master/mmdet3d/models/fusion_layers/coord_transform.py:35: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
if 'pcd_rotation' in img_meta else torch.eye(
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
[Exception|native_pair]indices=torch.Size([28776, 4]),bs=2,ss=[41, 1600, 1408],algo=ConvAlgo.Native,ksize=[3, 3, 3],stride=[1, 1, 1],padding=[1, 1, 1],dilation=[1, 1, 1],subm=True,transpose=False
Traceback (most recent call last):
File "/home/dell/zhangweili/mmdetection3d-master/tools/train.py", line 262, in
main()
File "/home/dell/zhangweili/mmdetection3d-master/tools/train.py", line 258, in main
meta=meta)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/apis/train.py", line 351, in train_model
meta=meta)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/apis/train.py", line 319, in train_detector
runner.run(data_loaders, cfg.workflow) # 启动
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in run_iter
**kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
losses = self(**data)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
return old_func(*args, **kwargs)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/models/detectors/base.py", line 60, in forward
return self.forward_train(**kwargs)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/models/detectors/mvx_two_stage.py", line 274, in forward_train
points, img=img, img_metas=img_metas)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/models/detectors/mvx_two_stage.py", line 208, in extract_feat
pts_feats = self.extract_pts_feat(points, img_feats, img_metas)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/models/detectors/mvx_faster_rcnn.py", line 57, in extract_pts_feat
x = self.pts_middle_encoder(voxel_features, feature_coors, batch_size)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, inFile "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 116, in new_func
return old_func(*args, **kwargs)
File "/home/dell/zhangweili/mmdetection3d-master/mmdet3d/models/middle_encoders/sparse_encoder.py", line 123, in forward
x = self.conv_input(input_sp_tensor)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/modules.py", line 137, in forward
input = module(input)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/conv.py", line 384, in forward
raise e
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/conv.py", line 375, in forward
self.transposed)
File "/home/dell/anaconda3/envs/mmdet3d/lib/python3.7/site-packages/spconv/pytorch/ops.py", line 162, in get_indice_pairs
stream)
RuntimeError: /io/build/temp.linux-x86_64-cpython-37/spconv/build/core_cc/src/csrc/sparse/all/SpconvOps/SpconvOps_get_indice_pairs.cc(65)
not implemented for CPU ONLY build.
Additional information
Hope it can help us solve or suggest, thank you!
The text was updated successfully, but these errors were encountered: