在Ruby/Rails中处理大型数据集导入

  class DeleteImportStrategy
def pre_process(merchant_prefix, channel_import)
  # channel needed to identify invoices so an import from another channel cannot collude if they had same merchant_prefix
  Jzbackend::Invoice.where(merchant_prefix: merchant_prefix, channel: channel_import.channel).delete_all
  # get rid of all previous import patches which becomes empty after delete_import_strategy
  Jzbackend::Import.where.not(id: channel_import.id).where(channel: channel_import.channel).destroy_all
end

def process_row(row, channel_import)
  debt_claim = Jzbackend::Invoice.new
  debt_claim.import = channel_import
  debt_claim.status = 'pending'
  debt_claim.channel = channel_import.channel
  debt_claim.merchant_prefix = row[0]
  debt_claim.debt_claim_number = row[1]
  debt_claim.amount = Monetize.parse(row[2])
  debt_claim.print_date = row[3]
  debt_claim.first_name = row.try(:[], 4)
  debt_claim.last_name = row.try(:[], 5)
  debt_claim.address = row.try(:[], 6)
  debt_claim.postal_code = row.try(:[], 7)
  debt_claim.city = row.try(:[], 8)
  debt_claim.save
end

终止

////

因此,对于作为CSV输入的每个导入批次,我去掉了以前的批次,开始导入新批次,方法是读取每一行并将其作为发票记录插入到新的导入中。然而,对于10万个条目来说,2.5-3个小时似乎有点过头了。我怎么才能优化这个过程,因为我确信这样肯定效率不高。

已编辑: