Question

python3 爬虫 一定要用beautiful soup吗

Answer 1

BeautifulSoup4的安装
一、使用pip直接安装beautifulsoup4 （如何安装pip请看上一篇文章介绍）

F:\kanbox\pythoncode\zyspider>pip install beautifulsoup4
Collecting beautifulsoup4
Downloading beautifulsoup4-4.4.0-py3-none-any.whl (80kB)
328kB/s
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.4.0

F:\kanbox\pythoncode\zyspider>

或者从官网下载Beautifulsoup的软件包，然后解压，cmd命令行进入解压包目录，输入以下命令安装：python setup.py install

=======================================
网络爬虫实例代码，抓取新浪爱彩双色球开奖数据实例代码：

1 __author__ = 'zdz8207'
2 from bs4 import BeautifulSoup
3
4 import urllib.request
5 import urllib.parse
6 import re
7 import urllib.request, urllib.parse, http.cookiejar
8
9 def getHtml(url):
10 cj = http.cookiejar.CookieJar()
11 opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cj))
12 opener.addheaders = [('User-Agent',
13 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36'),
14 ('Cookie', '4564564564564564565646540')]
15
16 urllib.request.install_opener(opener)
17
18 html_bytes = urllib.request.urlopen(url).read()
19 html_string = html_bytes.decode('utf-8')
20 return html_string
21
22 html_doc = getHtml("http://zst.aicai.com/ssq/openInfo/")
23 soup = BeautifulSoup(html_doc, 'html.parser')
24
25 # print(soup.title)
26 #table = soup.find_all('table', class_='fzTab')
27 #print(table)#<tr onmouseout="this.style.background=''" 这种tr丢失了
28 #soup.strip() 加了strip后经常出现find_all('tr') 只返回第一个tr
29 tr = soup.find('tr',attrs={"onmouseout": "this.style.background=''"}) 30 #print(tr) 31 tds = tr.find_all('td') 32 opennum = tds[0].get_text() 33 #print(opennum) 34 35 reds = [] 36 for i in range(2,8): 37 reds.append(tds[i].get_text()) 38 #print(reds) 39 blue = tds[8].get_text() 40 #print(blue) 41 42 #把list转换为字符串:(',').join(list) 43 #最终输出结果格式如：2015075期开奖号码：6,11,13,19,21,32, 蓝球：4 44 print(opennum+'期开奖号码：'+ (',').join(reds)+", 蓝球："+blue)