修复 FinGPT content 爬虫脚本
-
修改了2个地方,
1、 判断状态 有错误
2、 获取内容因为网页改版,导致数据获取不到,修改如下:while not ok: try: response = requests.get(url = url, headers = headers) print(url,response.status_code) if response.status_code == 200: res = etree.HTML(response.text) res = res.xpath("//script[2]//text()")[0] res = json.loads(res[17:]) res = pd.Series(res).to_frame().T ok = True return res except : pass
-
FinGPT-v1/data_preparations$ vim download_contents.py