代码之家  ›  专栏  ›  技术社区  ›  sweetmusicality

对在R中查找引用的循环进行矢量化

grepl vectorization for-loop string r
  •  0
  • sweetmusicality  · 技术社区  · 7 年前

    我在数据集中有一个变量,其中包含我想对其进行字符串搜索的短语( female$Var2 female_df$MH2 ). 例如, 女性$Var2 看起来像: 

    myocardial infarction drug therapy
imipramine poisoning
oximetry
thrombosis drug therapy
angioedema chemically induced

    我想找出数据框中包含上述每个短语的行数 看起来像这样 

    oximetry, hydrogen peroxide adverse effects, epoprostenol adverse effects
angioedema chemically induced, angioedema chemically induced, oximetry
abo blood group system, imipramine poisoning, adverse effects
isoenzymes, myocardial infarction drug therapy, thrombosis drug therapy
thrombosis drug therapy

    因此,我的结果输出应该如下所示: 

    myocardial infarction drug therapy          1
imipramine poisoning                        1
oximetry                                    2
thrombosis drug therapy                     2
angioedema chemically induced               1

    注意,这不是总发生次数(见血管性水肿…)。它是包含该短语的行数。我目前正在运行一个for循环,该循环耗时太长,因为它在428000多行上搜索5000多个术语。当我尝试使用 occurrences_female(female$Var2) ,我得到 In grepl(word, female_df$MH2, ignore.case = TRUE) : argument 'pattern' has length > 1 and only the first element will be used 错误,仅返回第一个变量 女性$Var2

    这是我正在运行的for循环 

    for (i in 1:nrow(female))
{
  word <- female$Var2[i]
  df_female <- data.frame(word, occurrences_female(word))
  df_female2 <- rbind(df_female2, df_female)
}

    基于此功能 

    occurrences_female <- function(word)
{
  # inserts \\b in the beginning
  word <- paste0("\\b", word)

  # inserts \\b at the end
  n <- nchar(word)
  word <- paste(substr(word, 1, n), "\\b", sep = "")

  occurrences <- sum(grepl(word, female_df$MH2, ignore.case = TRUE))

  return (occurrences)
}

    当我手动执行时,该函数可以工作,但我需要在5000多个条件下完成,并且for循环太慢(已经运行了2个多小时)。我不知道如何在不同数据帧的变量上搜索数据帧的一个变量。

    3 回复  |  直到 7 年前
        1
  •  4
  •   www    7 年前

    总结 

    library(purrr)
library(stringr)

female$Count <- map_int(female$Var2, 
                    function(x){sum(str_detect(female_df$MH2, pattern = x))})

    介绍

    有多种方法可以计算每个单词或短语包含多少行。但根据目前为止这条线索中的答案和讨论,实现这一目标的总体策略是。

    1. lapply 和 sapply map 来自的函数 purrr 包裹
    2. 使用函数来计数或检测特定模式(单词或短语)是否在字符串中。这些函数类似于 grep , grepl 从R基开始,或 str_detect str_which 从 stringr

    由于OP有大量数据需要处理,我进行了分析,比较了R基函数的哪些组合, 和 斯特林格

    我总共调查了八种组合。在使用 map_int , 格雷普 或 str_哪个 和 格雷普 或 str_检测 .

    数据准备

    在这里,我创建了两个数据帧, female female_df ,基于OP的示例。注意,我设置了 stringsAsFactors 确保每一列都是字符格式。 

    # Create the example data frame: female
female <- data.frame(Var2 = c("myocardial infarction drug therapy", 
                              "imipramine poisoning",
                              "oximetry",
                              "thrombosis drug therapy",
                              "angioedema chemically induced"),
                     stringsAsFactors = FALSE)

# Create the example data frame: female_df
female_df <- data.frame(MH2 = c("oximetry, hydrogen peroxide adverse effects, epoprostenol adverse effects",
                                "angioedema chemically induced, angioedema chemically induced, oximetry",
                                "abo blood group system, imipramine poisoning, adverse effects",
                                "isoenzymes, myocardial infarction drug therapy, thrombosis drug therapy",
                                "thrombosis drug therapy"),
                        stringsAsFactors = FALSE)

    microbenchmark 是用于评估代码性能的包。 

    # Load packages
library(purrr)
library(stringr)
library(microbenchmark)

    功能组合

    组合1

    这是卢斯·泰尔斯的回答。它使用 赛普利 和 . 

    sapply(female$Var2, function(x){sum(grepl(pattern = x, female_df$MH2))})

myocardial infarction drug therapy               imipramine poisoning 
                                 1                                  1 
                          oximetry            thrombosis drug therapy 
                                 2                                  2 
     angioedema chemically induced 
                                 1

    组合2

    这是Dave2e的回答。它使用 和 格雷普 

    sapply(female$Var2, function(x){length(grep(x, female_df$MH2))})

myocardial infarction drug therapy               imipramine poisoning 
                                 1                                  1 
                          oximetry            thrombosis drug therapy 
                                 2                                  2 
     angioedema chemically induced 
                                 1

    组合3

    这使用 和 . 

    map_int(female$Var2, function(x){sum(str_detect(female_df$MH2, pattern = x))})
[1] 1 1 2 2 1

    组合4

    这使用 map_int . 

    map_int(female$Var2, function(x){length(str_which(female_df$MH2, pattern = x))})
[1] 1 1 2 2 1

    这使用 和 . 

    map_int(female$Var2, function(x){sum(grepl(pattern = x, female_df$MH2))})
[1] 1 1 2 2 1

    map_int 和 格雷普 

    map_int(female$Var2, function(x){length(grep(x, female_df$MH2))})
[1] 1 1 2 2 1

    这使用 和 str_检测 

    sapply(female$Var2, function(x){sum(str_detect(female_df$MH2, pattern = x))})
myocardial infarction drug therapy               imipramine poisoning 
                                 1                                  1 
                          oximetry            thrombosis drug therapy 
                                 2                                  2 
     angioedema chemically induced 
                                 1

    组合8

    和 . 

    sapply(female$Var2, function(x){length(str_which(female_df$MH2, pattern = x))})
myocardial infarction drug therapy               imipramine poisoning 
                                 1                                  1 
                          oximetry            thrombosis drug therapy 
                                 2                                  2 
     angioedema chemically induced 
                                 1

    所有这些组合都是有效答案。例如,我们可以 female$Count <

    微基准

    在这里,我用30000次抽样对这八种组合进行了基准测试。 

    m <- microbenchmark(
  C1 = {sapply(female$Var2, function(x){sum(grepl(pattern = x, female_df$MH2))})},
  C2 = {sapply(female$Var2, function(x){length(grep(x, female_df$MH2))})},
  C3 = {map_int(female$Var2, function(x){sum(str_detect(female_df$MH2, pattern = x))})},
  C4 = {map_int(female$Var2, function(x){length(str_which(female_df$MH2, pattern = x))})},
  C5 = {map_int(female$Var2, function(x){sum(grepl(pattern = x, female_df$MH2))})},
  C6 = {map_int(female$Var2, function(x){length(grep(x, female_df$MH2))})},
  C7 = {sapply(female$Var2, function(x){sum(str_detect(female_df$MH2, pattern = x))})},
  C8 = {sapply(female$Var2, function(x){length(str_which(female_df$MH2, pattern = x))})},
  times = 30000L
)

print(m)

Unit: microseconds
 expr     min      lq     mean   median       uq       max neval
   C1 166.144 200.784 1503.780 2192.261 2401.063 184228.81 30000
   C2 163.578 198.860 1420.937 1460.653 2280.465 144553.22 30000
   C3 189.238 231.575 1502.319  790.305 2386.309 146455.85 30000
   C4 200.784 246.329 1461.714 1224.909 2306.125 184189.04 30000
   C5 150.107 185.388 1452.586 1970.630 2376.687  32124.08 30000
   C6 148.824 184.105 1398.312 1921.556 2259.937 145843.88 30000
   C7 205.916 251.461 1516.979  851.246 2408.119 146305.10 30000
   C8 215.538 264.932 1481.538 1508.764 2324.727 229709.16 30000

    map_int 和 str_检测

        2
  •  2
  •   Luís Telles    7 年前

    仅以R为基数的解决方案(假设 female$VAR2 只有唯一字符串): 

    counts <- sapply(female$VAR2, function(x){ z <- sum(grepl(pattern = x,
                                                    x = female_df$MH2,
                                                    ignore.case = TRUE))
                                      z
                                     })
word_counts <- cbind(female$VAR2, counts)
        3
  •  2
  •   Dave2e    7 年前

    rbind 在处理时间方面,将每一行添加到数据帧是非常昂贵的。 

    #Data set up
var2<-c("myocardial infarction drug therapy", "imipramine poisoning", "oximetry",
             "thrombosis drug therapy", "angioedema chemically induced")
female<-data.frame(var2, stringsAsFactors = FALSE)

MH2<-c("oximetry, hydrogen peroxide adverse effects, epoprostenol adverse effects",
"angioedema chemically induced, angioedema chemically induced, oximetry",
                "abo blood group system, imipramine poisoning, adverse effects",
                "isoenzymes, myocardial infarction drug therapy, thrombosis drug therapy",
                "thrombosis drug therapy")
female_df<-data.frame(MH2, stringsAsFactors = FALSE)

library(stringr)
#create a matrix where columns is the terms
# and the rows are the lines checked.
termmatrix<-sapply(female$var2, function(x){str_count(female_df$MH2, x)})
#find the sums of the columns to determine the number of times each term is used
ans<-colSums(termmatrix)

    决赛 ans 是具有项和总计数的命名向量。

    附加
    要避免创建庞大的术语矩阵,请尝试: 

    ans<-sapply(female$var2, function(x){length(grep(x, female_df$MH2))})

    推荐文章
    user3152377  ·  Python:矢量化/广播会提高速度吗?
    6 年前
    Remy  ·  如何在MATLAB中有效地求出阵列所选指标的最大值?
    6 年前
    Omry Atia  ·  在R中,对数据帧的行应用函数并返回数据帧
    6 年前
    Kapa11  ·  MATLAB:复制数组中的零项
    6 年前
    bumblebee  ·  回归“累积”期望值的矢量化
    6 年前
    Niklas M.  ·  如何在numpy中矢量化字符串数组的和?[副本]
    6 年前
    fales  ·  Python中的许多对的相同操作
    6 年前
    user5054  ·  有没有办法从矩阵中矢量化列的选择(重复)?
    6 年前
    Fnord  ·  具有屏蔽索引的numpy数组上的向量化和运算
    6 年前
    Rochak Agrawal  ·  以下八度音阶代码段的矢量化代码是什么
    6 年前
    关于   移动版
    代码之家 - 一站式码农服务社区
    沪ICP备11025650号