Question

怎么把python 用urllib2爬取的内容 存到数据库中

Answer 1

使用mongodb很简单。
首先安装pymongo：
1pip install pymongo
代码实现：用urllib2读取数据，打包成JSON格式插入到mongodb中。
1234567891011121314151617181920212223242526from pymongo import MongoClienttry: from urllib2 importurlopen, Request, HTTPError, URLErrorexcept ImportError: from urllib.request import urlopen, Request, HTTPError, URLErrorresult = []try: f = urlopen('http://www.dynamsoft.com', timeout=3) while 1: tmp = f.read(10240) if len(tmp) == 0: break else: result.append(tmp) f.close()except HTTPError, URLError: print URLError.codecontent = ''.join(result)post = {"author": "yushulx","content": content}client = MongoClient()db = client.test_databaseposts = db.postsposts.insert(post)print posts.find_one({"author": "yushulx"})