代码之家 › 专栏 › 技术社区 › Mr.D

将json源中的数据打包到csv中

request web-scraping python-2.7 json python

Mr.D · 技术社区 · 7 年前

我试过这个:

my_url = requests.get(https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL)

data = my_url.json()

name = data['MainContent'][0]['contents'][0]['productList']['products'][0]['productModel']["displayName"]
price = data['MainContent'][0]['contents'][0]['productList']['products'][0]['priceInfo']['priceItemSale']["gross"]
url= data['MainContent'][0]['contents'][0]['productList']['products'][0]['productModel']["url"]

for mc in data['MainContent']:
    for co in mc:
        for prod in co['productList']['products']:
            name = prod['productModel']['displayName']
            price = prod['priceItemSale']['gross']
            url = prod['productModel']['url']

filename = "test.csv"
csv_writer = csv.writer(open(filename, 'w'))
headers = "Name, Price, Link\n"
f.write(headers)
f.close()

在这个web站点中有很多具有这些属性“productModel”的产品,但是我怎样才能得到这些并写入csv呢?我希望web取消这个页面在不同单元格中的名称、价格和url链接,但它不起作用。

def proba():

    my_url = requests.get('https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL')
    data = my_url.json()
    results = []
    products = data['MainContent'][0]['contents'][0]['productList']['products']
    for product in products:
        name = product['productModel']['displayName']
        try:
            priceGross = product['priceInfo']['priceItemSale']['gross']
        except:
            priceGross = product['priceInfo']['priceItemToBase']['gross']
        url = product['productModel']['url']
        results.append([name, priceGross, url])
    df = pd.DataFrame(results, columns = ['Name', 'Price', 'Url'])    
# print(df)  ## print df
    df.to_csv(r'/usr/src/Python-2.7.13/test.csv', sep=',', encoding='utf-8-sig',index = False )

while True:
    mytime=datetime.now().strftime("%H:%M:%S")
    while mytime < "23:59:59":
    print mytime
    proba()
    mytime=datetime.now().strftime("%H:%M:%S")

0 回复 | 直到 7 年前

QHarr 7 年前

并非所有项目的密钥数都相同,因此您无法使用 prod['priceItemSale']['gross'] 从始至终。你需要决定在不存在的情况下从何处获得价值。示例如下:

import requests
import pandas as pd

my_url = requests.get('https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL')
data = my_url.json()
results = []
products = data['MainContent'][0]['contents'][0]['productList']['products']
for product in products:
    name = product['productModel']['displayName']
    try:
        priceGross = product['priceInfo']['priceItemSale']['gross']
    except:
        priceGross = product['priceInfo']['priceItemToBase']['gross']
    url = product['productModel']['url']
    results.append([name, priceGross, url])
df = pd.DataFrame(results, columns = ['Name', 'Price', 'Url'])    
# print(df)  ## print df
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False )

输出:

ravishankar chavare 7 年前

import requests
my_url = requests.get("https://www.telekom.hu/shop/categoryresults/?N=10994&contractType=list_price&instock_products=1&Ns=sku.sortingPrice%7C0%7C%7Cproduct.displayName%7C0&No=0&Nrpp=9&paymentType=FULL")

data = my_url.json()
datas=data["MainContent"][0]["contents"]
finaldata=[]
for mc in datas:
  plist=mc["productList"]["products"]
  for p in plist:
    name =p['productModel']['displayName']
    try:
      price = p['priceInfo']['priceItemToBase']['gross']
    except:
      price=p['priceInfo']['priceItemSale']['gross']
    url = p['productModel']['url']
    finaldata.append([name,price,url])
files=open("data.csv",'w+')
columns = ['Name', 'Price', 'Url']
files.write(",".join(columns))
files.write("\n")
for f in finaldata:
 files.write('{},{},{}\n'.format(f[0],f[1],f[2]))