full_join()在r中分段/批处理两个数据帧

full_join 和 merge 从 data.table 在64位Windows系统中(Intel ~ 3.5 Ghz,RAM 120GB)。希望它至少能为您的案例提供参考。

library(data.table)
df1 <- data.table(KEY=sample(1:800,20015,replace = TRUE), 
                  matrix(rnorm(20015*7),20015,7))#1.1MB
df2 <- data.table(KEY=sample(1:800,8534664,replace = TRUE), 
                  matrix(rnorm(8534664*29),8534664,29))#1.9GB
library(dplyr)
tick <- Sys.time()
df_join <- full_join(df1, df2, by = "KEY") #~58.1 GB in memory
tock <- Sys.time()- tick #~1.85min
#With data.table merge.
tick <- Sys.time()
df_join<- merge(df1, df2, by = "KEY", allow.cartesian = TRUE)#~58.1 GB in memory
tock <- Sys.time() - tick #~5.75 mins