Question

帮忙分析一下用python爬取一个网站一直不成功的原因

headers = {
"Accept":
headers={"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "zh-CN,zh;q=0.9",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"DNT": "1",
"Host": "ha.chinaunicombidding.cn",
"Pragma": "no-cache",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36"
}
url1='http://ha.chinaunicombidding.cn/jsp/cnceb/web/info1/infoList.jsp?notice=&time1=&time2=&province=&city=&type='
response = requests.get(url1, headers=headers)
text = response.content.decode('utf-8')
html = etree.HTML(text)
biaotis=html.xpath('//*[@id="div1"]/table/tbody/tr/td//text()') #招标公告标题
for biaoti in biaotis:
print(biaoti)

Answer 1

人家内容是通过js生成就这样,而且就算不是js生成的你写的代码也有一堆问题,人家没有对请求头的返爬你加不加毫无意义,text = response.content.decode('utf-8')你这写法是什么鬼,人家网站charset默认utf8没有必要定义,直接text = response.text就可以了,这个网站你应该爬不下来,还是看看基础吧骚年

追问

回答好nb啊！那你给指点一下啊！