设为首页收藏本站
查看: 86|回复: 0

[PHP] python爬虫beta版之抓取知乎单页面

[复制链接]

论坛元老

Rank: 6Rank: 6

积分
34274
主题
17031
UID
1347
M币
67
贡献
17176

  • 发表于 2017-5-14 02:44:00 | 显示全部楼层 |阅读模式
    鉴于之前用python写爬虫,帮运营人员抓取过京东的商品品牌以及分类,这次也是用python来搞简单的抓取单页面版,后期再补充哈。
    #-*- coding: UTF-8 -*- import requests import sys from bs4 import BeautifulSoup #------知乎答案收集---------- #获取网页body里的内容 def get_content(url , data = None): header={ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, sdch', 'Accept-Language': 'zh-CN,zh;q=0.8', 'Connection': 'keep-alive', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.235' } req = requests.get(url, headers=header) req.encoding = 'utf-8' bs = BeautifulSoup(req.text, "html.parser") # 创建BeautifulSoup对象 body = bs.body # 获取body部分 return body #获取问题标题 def get_title(html_text): data = html_text.find('span', {'class': 'zm-editable-content'}) return data.string.encode('utf-8') #获取问题内容 def get_question_content(html_text): data = html_text.find('p', {'class': 'zm-editable-content'}) if data.string is None: out = ''; for datastring in data.strings: out = out + datastring.encode('utf-8') print '内容:\n' + out else: print '内容:\n' + data.string.encode('utf-8') #获取点赞数 def get_answer_agree(body): agree = body.find('span',{'class': 'count'}) print '点赞数:' + agree.string.encode('utf-8') + '\n' #获取答案 def get_response(html_text): respOnse= html_text.find_all('p', {'class': 'zh-summary summary clearfix'}) for index in range(len(response)): #获取标签 answerhref = response[index].find('a', {'class': 'toggle-expand'}) if not(answerhref['href'].startswith('Javascript')): url = 'http://www.zhihu.com/' + answerhref['href'] print url body = get_content(url) get_answer_agree(body) answer = body.find('p', {'class': 'zm-editable-content clearfix'}) if answer.string is None: out = ''; for datastring in answer.strings: out = out + '\n' + datastring.encode('utf-8') print out else: print answer.string.encode('utf-8') html_text = get_content('https://www.zhihu.com/question/43879769') title = get_title(html_text) print "标题:\n" + title + '\n' questiOndata= get_question_content(html_text) print '\n' data = get_response(html_text)输出结果:
    回复

    使用道具 举报

    您需要登录后才可以回帖 登录 | 立即注册

    本版积分规则

    在我站开通SVIP可同时获得17个站点VIP资源 立即登录 立即注册
    快速回复 返回顶部 返回列表