我需要读取多个大型的.csv(20k行x 6k列),并将它们存储在一个数据帧中。
这个
thread
有很好的例子,在过去为我工作与较小的文件。
例如:
pd.concat((pd.read_csv(f,index_col='Unnamed: 0') for f in file_list))
我尝试过的其他更直接的方法是:
frame = pd.DataFrame()
list_ = []
for file_ in file_list:
print(file_)
df = pd.read_csv(file_,index_col=0)
list_.append(df)
df = pd.concat(list_)
然而,所有的解决方案都围绕着创建一个所有csv文件的列表作为单独的df,然后使用
pd.concat()
在所有DF的最后。
据我所知,这是一种方法,当连接到20个df时会导致内存错误。
-
我怎样才能越过这一步,也许在我走的时候加上每个df?
文件列表示例:
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_26.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_30.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_25.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_19.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_27.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_18.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_28.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_23.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_03.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_24.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_29.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_04.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_20.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_22.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_06.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_05.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_01.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_02.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_31.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_21.csv
