JSON

JSON이란?

---

• url : http://www.json.org/json-ko.html

JSON 이쁘게 보기

---

• http://bodurov.com/jsonformatter/
• https://jsonviewer.stack.hu/

JSON데이터 수집

---

• requests.get(url).json() 사용 해서 수집

daum news 제목과 댓글 수 가져오기

import requests
from bs4 import BeautifulSoup

# daum news
url = 'https://v.daum.net/v/20230515180314322'
resp = requests.get(url)
if resp.status_code == 200:
    soup = BeautifulSoup(resp.text)

# title  제목
h3_tit_view  = main.select_one('h3.tit_view')
if h3_tit_view:
    title = h3_tit_view.string
print(title)

# 댓글 수
a = main.select_one('span.alex-count-area')
print(a.string) # 0

Fetch/XHR : AJAX 주소 찾기

• 크롬 개발자 도구 > Network > Fetch/XHR

# Fetch/XHR : AJAX 주소 찾기
url = 'https://comment.daum.net/apis/v1/ui/single/main/@20230515180314322?version=v3.24.0'
resp = requests.get(url)
if resp.status_code == 200:
    soup = BeautifulSoup(resp.text)

# 정상 적인 json 응답이 아님
print(soup)

# request header 값 중  Authorization 값 추가
headers = {'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJmb3J1bV9rZXkiOiJuZXdzIiwiZ3JhbnRfdHlwZSI6ImFsZXhfY3JlZGVudGlhbHMiLCJzY29wZSI6W10sImV4cCI6MTY4NDI2NDIyMywiYXV0aG9yaXRpZXMiOlsiUk9MRV9DTElFTlQiXSwianRpIjoiNjVhYjBkODUtNjUyMC00ZjQxLThiNWYtN2Y5ZGFjYWYwZDZjIiwiZm9ydW1faWQiOi05OSwiY2xpZW50X2lkIjoiMjZCWEF2S255NVdGNVowOWxyNWs3N1k4In0.2rAxU1Ff6-eSIRdKWS4gZrvybh7ZRd1QemgQr5Ps7Q0'}

resp = requests.get(url, headers=headers)
print(resp.text)

# JSON으로 변경
json = resp.json()

post = json['post']
commentCount = post['commentCount']
print(commentCount)

Kweather 생활 날씨 가져오기

---

• url : https://weather.kweather.co.kr/weather/life_weather

SSL 오류 처리

CERTIFICATE_VERIFY_FAILED

(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

• verify=False 값 추가 해서 요청

requests.get(url, verify=False)

H_KEY_TOO_SMALL

(Caused by SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)')))

• import 추가

import requests
import urllib3

requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

데이터 수집

AJAX 요청 주소 수집

• 지역을 변경 하고 검색 시 동적으로 변함 => AJAX 처리 주소 필요
• 개발자도구 > NetWork 에서 화면이나 url 새로 고침
• 개발자도구 > NetWork > Fetch/XHR > Name > 11B00000 선택
• 개발자도구 > NetWork > Fetch/XHR > Headers > General > Request URL 사용
• 지역을 변경하면 11B00000 값이 변경됨
• 카테고리 요청 url : https://weather.kweather.co.kr/weather/life_weather/get_life_region_list
• 데이터 요청 url : https://weather.kweather.co.kr/weather/life_weather/get_life_factor_list

수집 주소 목록 만들기

url = 'https://weather.kweather.co.kr/weather/life_weather/get_life_region_list'
resp = requests.get(url, verify=False)

if resp.status_code == 200:
    json = resp.json()

category = json
print(category)

AJAX JSON 1개 처리 하기

방법 1

path = 'https://weather.kweather.co.kr/weather/life_weather/get_life_factor_list/'
url = path + category[0]['areacode']
resp = requests.get(url, verify=False)
if resp.status_code == 200:
    json = resp.json()
areacode = json['areacode']
picnicName = json['picnicName']
picnicFactor = json['picnicFactor']
picnicDiscription = json['picnicDiscription']
print(areacode, picnicName, picnicFactor,picnicDiscription)

방법 2

factor = [value for key, value in json.items() if key.endswith('Factor')]
name = [value for key, value in json.items() if key.endswith('Name')]
disc = [value for key, value in json.items() if key.endswith('Discription')]

방법3

Name = [value for key, value in json.items() if 'Name' in key]
Factor = [value for key, value in json.items() if 'Factor' in key]
Discription = [value for key, value in json.items() if 'Discription' in key or 'Discrption' in key]

전체 AJAX JSON 데이터 처리 하기

전체 JSON 데이터 수집

# 반복 처리
path = 'https://weather.kweather.co.kr/weather/life_weather/get_life_factor_list/'
for c in category:
    url = path + c['areacode']
    resp = requests.get(url, verify=False)
    if resp.status_code == 200:
        json = resp.json()

    factor1 = [value for key, value in json.items() if key.endswith('Factor')]
    name1 = [value for key, value in json.items() if key.endswith('Name')]
    disc1 = [value for key, value in json.items() if key.endswith('Discription') or key.endswith('Discrption') ]

    factor2 = [value for key, value in json.items() if 'Factor' in key]
    name2 = [value for key, value in json.items() if 'Name' in key]
    disc2 = [value for key, value in json.items() if 'Discription' in key or 'Discrption' in key]