Question

使用scrapy抓取网页出现10055错误

本人使用scrapy抓取某网站网页,每次抓取变换user-agent和代理ip,
运行了大约三小时后连续出现异常
进入异常处理函数process_exception
打印出异常为"An error occurred while connecting: 10055: 由于系统缓冲区空间不足或队列已满，不能执行套接字上的操作。."
求大神解答疑问.
下面是process_request和process_response代码
def process_request(self, request, spider):
if not self._is_enabled_for_request(request):
return
if self.len_valid_proxy() > 0:
self.set_proxy(request)
# if 'download_timeout' not in request.meta:
request.meta['download_timeout'] = self.download_timeout
else:
# 没有可用代理，直连
logger.debug('no valid proxy ip, use local ip directly')
if 'proxy' in request.meta:
del request.meta['proxy']
self.extend_proxy()

def process_response(self, request, response, spider):
if not self._is_enabled_for_request(request):
return response

if response.status in self.ban_code:
self.invaild_proxy(request.meta['proxy'])
logger.debug("Proxy[%s] ban because return httpstatuscode:[%s]. ", request.meta['proxy'], str(response.status))
new_request = request.copy()
new_request.dont_filter = True
return new_request

if 'proxy' in request.meta:
p = request.meta['proxy']
self.counter_proxy[p] = self.counter_proxy.setdefault(p, 1) + 1
return response

Answer 1

这些节点可能是异步加载进来的，网页源码里根本没有的，你可以selenium+phantomjs请求获得网页源码，再试试