爬取糗事百科段子
确定URL并抓取页面代码,首先我们确定好页面的URL是 http://www.qiushibaike.com/8hr/page/4, 其中最后一个数字1代表页数,我们可以传入不同的值来获得某一页的段子内容。
我们初步构建如下的代码来打印页面代码内容试试看,先构造最基本的页面抓取方式,看看会不会成功
在Composer raw 模拟发送数据
GET http://www.qiushibaike.com/8hr/page/2/ HTTP/1.1
Host: www.qiushibaike.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Accept-Language: zh-CN,zh;q=0.8
在删除了User-Agent、Accept-Language报错
应该是headers验证的问题,加上一个headers验证试试看
# -*- coding:utf-8 -*-
import urllib
import requests
import re
import chardet
from lxml import etree
page = 2
url = 'http://www.qiushibaike.com/8hr/page/' + str(page) + "/"
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36',
'Accept-Language': 'zh-CN,zh;q=0.8'}
try:
response = requests.get(url, headers=headers)
resHtml = response.text
html = etree.HTML(resHtml)
result = html.xpath('//div[contains(@id,"qiushi_tag")]')
for site in result:
#print etree.tostring(site,encoding='utf-8')
item = {}
imgUrl = site.xpath('./div/a/img/@src')[0].encode('utf-8')
username = site.xpath('./div/a/@title')[0].encode('utf-8')
#username = site.xpath('.//h2')[0].text
content = site.xpath('.//div[@class="content"]')[0].text.strip().encode('utf-8')
vote = site.xpath('.//i')[0].text
#print site.xpath('.//*[@class="number"]')[0].text
comments = site.xpath('.//i')[1].text
print imgUrl, username, content, vote, comments
except Exception, e:
print e
好啦,大家来测试一下吧,点一下回车会输出一个段子,包括发布人,发布时间,段子内容以及点赞数,是不是感觉爽爆了!