Question

如何调试Python的segmentation fault错误

Answer 1

错误
一般都是非法内存操作，例如数组越界，例如申请a[5]，却访问到a[5]或者a[6]，这也会有很多情况，可能是循环操作时循环变量控制有问题，可能是字符串拷贝时长度发生溢出

指针指向了非法内存，例如申明一个指针，但是没有对指针进行初始化，直接就引用，或者引用里面的元素或者函数，或者没有分配内存就进行释放等，另外，申请内存还要检查是否申请成功，如果没有申请成功也会出现这种情况
单步调试或者加打印信息，细心一点总可以找到错误的，注意编译成调试版本 如果是linux，可以产生core文件，从core文件查看出错的地方

Answer 2

博主今天下午在工作时，将本来已经在本地运行完好的程序放置到GPU云上进行训练，结果却遇到了意外的错误。
这个脚本的作用是将Pascal VOC的数据集做成符合MXNet格式的训练集，不涉及到GPU的调用，没想到却报错 non-zero exit，具体代码为-11，如下所示：

tbs@ubuntu:~/workspace/mxnet-ssd$ bash tools/prepare_pascal.sh
saving list to disk...
List file /home/tbs/workspace/mxnet-ssd/tools/../data/train.lst generated...
Creating .rec file from /home/tbs/workspace/mxnet-ssd/data/train.lst in /home/tbs/workspace/mxnet-ssd/data
multiprocessing not available, fall back to single threaded encoding
Traceback (most recent call last):
File "/home/tbs/workspace/mxnet-ssd/tools/prepare_dataset.py", line 111, in <module>
"--shuffle", str(int(args.shuffle)), "--pack-label", "1"])
File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '/home/tbs/workspace/mxnet-ssd/tools/../mxnet/tools/im2rec.py', '/home/tbs/workspace/mxnet-ssd/data/train.lst', '/home/tbs/workspace/mxnet-ssd/data/VOCdevkit', '--shuffle', '1', '--pack-label', '1']' returned non-zero exit status -11
saving list to disk...
List file /home/tbs/workspace/mxnet-ssd/tools/../data/val.lst generated...
Creating .rec file from /home/tbs/workspace/mxnet-ssd/data/val.lst in /home/tbs/workspace/mxnet-ssd/data
multiprocessing not available, fall back to single threaded encoding
Traceback (most recent call last):
File "/home/tbs/workspace/mxnet-ssd/tools/prepare_dataset.py", line 111, in <module>
"--shuffle", str(int(args.shuffle)), "--pack-label", "1"])
File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '/home/tbs/workspace/mxnet-ssd/tools/../mxnet/tools/im2rec.py', '/home/tbs/workspace/mxnet-ssd/data/val.lst', '/home/tbs/workspace/mxnet-ssd/data/VOCdevkit', '--shuffle', '1', '--pack-label', '1']' returned non-zero exit status -11

这个non-zero exit，代码为-11具体是什么错误呢？博主在搜索以后也没有给出具体的答案，不过给出了具体的debug的方式。
这个python的subprocess模块在调用失败后，会抛出CalledProcessError异常，并会将错误输出到returncode和output两个变量中。于是博主写了如下图加粗的一段代码，去打印这个异常，希望能在output变量中发现一些什么，结果令人失望，并没有更多的信息，output变量输出结果为None。

try:
subprocess.check_call(["python",
os.path.join(curr_path, "..", "mxnet/tools/im2rec.py"),
os.path.abspath(args.target), os.path.abspath(args.root_path),
"--shuffle", str(int(args.shuffle)), "--pack-label", "1"])
except subprocess.CalledProcessError as e:
raise RuntimeError("command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))

看来这样调试还是找不到具体的错误，那么直接运行subprocess的命令吧，结果是Segmentation fault (core dumped).
tbs@ubuntu:~/workspace/mxnet-ssd$ python /home/tbs/workspace/mxnet-ssd/tools/../mxnet/tools/im2rec.py /home/tbs/workspace/mxnet-ssd/data/train.lst /home/tbs/workspace/mxnet-ssd/data/VOCdevkit --shuffle 1 --pack-label 1
Creating .rec file from /home/tbs/workspace/mxnet-ssd/data/train.lst in /home/tbs/workspace/mxnet-ssd/data
multiprocessing not available, fall back to single threaded encoding
Segmentation fault (core dumped)

继续搜索发现，可以使用gdb调试这个错误，具体的使用方法是，在命令行中输入gdb，然后使用file python指令加载python环境，接下来使用run ***.py arg1 arg2 ...，这样调试器gdb就会在遇到的第一个错误处停下来。我们观察这个调试信息，发现最后问题定位在
0x00007ffff34f4865 in cv::Mat::copyTo(cv::_OutputArray const&) const () from /usr/lib/x86_64-linux-gnu/libopencv_core.so.2.4
而我们可以继续搜索得到，这个问题是因为系统同时装了OpenCV的2.4版本和3.0版本，或者说python的OpenCV版本与apt包管理的OpenCV版本不一致造成的。最终，博主将python的OpenCV 3.0版本卸载，重新装了一个2.4版本的，问题解决，终于可以正常训练了。

tbs@ubuntu:~/workspace/mxnet-ssd$ gdb
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<.
Find the GDB manual and other documentation resources online at:
<>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) file python
Reading symbols from python...(no debugging symbols found)...done.
(gdb) run /home/tbs/workspace/mxnet-ssd/tools/../mxnet/tools/im2rec.py /home/tbs/workspace/mxnet-ssd/data/train.lst /home/tbs/workspace/mxnet-ssd/data/VOCdevkit --shuffle 1 --pack-label 1
Starting program: /usr/bin/python /home/tbs/workspace/mxnet-ssd/tools/../mxnet/tools/im2rec.py /home/tbs/workspace/mxnet-ssd/data/train.lst /home/tbs/workspace/mxnet-ssd/data/VOCdevkit --shuffle 1 --pack-label 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3a29700 (LWP 7814)]
[New Thread 0x7ffff3228700 (LWP 7815)]