Scrapy 一些记录

UnicodeEncodeError: 'gbk' codec can't encode character '\xee' in position 22

print(response.text.encode('gbk', 'ignore').decode('gbk'))

https://zhidao.baidu.com/question/1302043991323187059.html

Scrapyd 部署出现错误 builtins.AttributeError: 'int' object has no attribute 'splitlines'

错误原因 twist 版本过高

pip install twisted==18.9.0

https://stackoverflow.com/questions/55658289/builtins-attributeerror-int-object-has-no-attribute-splitlines 

使用 Scrapyd 部署后找不到 spider

错误信息如下

Administrator@xxx MINGW64 /e/wwwroot/python/wa (master)
$ scrapyd-deploy wasp -p wa
Packing version 1557033361
Deploying to project "wa" in http://localhost:6800/addversion.json
Server response (200):
{"node_name": "xxx", "status": "ok", "project": "wa", "version": "1557033361", "spiders": 0}

Administrator@xxx MINGW64 /e/wwwroot/python/wa/wa (master)
$ curl http://localhost:6800/schedule.json -d project=wa -d spider=ctrip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current  Dload  Upload   Total   Spent    Left  Speed
100    82  100    59  100    23    269    105 --:--:-- --:--:-- --:--:--   374{"status": "error", "message": "spider 'ctrip' not found"}

将 Settings.py 中 LOG_STDOUT = True 注释掉

scrapyd deploy shows 0 spiders by scrapyd-client

https://stackoverflow.com/questions/33646722/scrapyd-deploy-shows-0-spiders-by-scrapyd-client

The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.

[root@instance-jewlel2q ~]# google-chrome
[4719:4719:0505/174032.341388:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

说明需要在 chrome 启动时加入 options.add_argument('--no-sandbox')

selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 75

[root@instance-jewlel2q ~]# google-chrome --version
Google Chrome 74.0.3729.131 

说明版本不对，重新下载 74 的 chromedriver

http://npm.taobao.org/mirrors/chromedriver/

pip3 install mysqlclient 报错

_mysql.c(29): fatal error C1083: 无法打开包括文件: “my_config.h”: No such file or directory

解决方法 使用whl安装

在网站https://www.lfd.uci.edu/~gohlke/pythonlibs/#

上下载

mysqlclient-1.3.13-cp37-cp37m-win_amd64.whl

控制台cmd上运行 pip install wheel mysqlclient-1.3.13-cp37-cp37m-win_amd64.whl