代码之家  ›  专栏  ›  技术社区  ›  Giorgio Spedicato

sqlalchemy orm从数据帧大容量插入np.nan公司

  •  0
  • Giorgio Spedicato  · 技术社区  · 5 年前

    my_engine = create_engine(url.URL(**my_db_url))
    Session = sessionmaker(bind=my_engine )
    my_session = Session()
    start = time.time()
    my_session.bulk_insert_mappings(TableObject, mysample)
    my_session.commit()
    durata = time.time() -start
    my_session.close()
    

    mysample 创建的dict列表为:

    mysample=myDataFrame.to_dict(orient='records')
    

    符合TableObject的,声明如下:

    from sqlalchemy import Column, BigInteger, String, Integer, Sequence, DateTime,Date, Float, ForeignKey, Boolean, VARCHAR, MetaData
    from sqlalchemy.ext.declarative import declarative_base
    from sqlalchemy.schema import PrimaryKeyConstraint
    from sqlalchemy.orm import sessionmaker
    from sqlalchemy.sql import null
    
    Base = declarative_base()
    class TableObject(Base):
    
        __tablename__ = 'mytable'
        __table_args__ = {"schema": "dbo"}
    
        Key1= Column('Key1',String(1), nullable=False)
        Key2= Column('Key2',Integer, nullable=False)
        Key3= Column('Key3',Integer, nullable=False)
        Key4= Column('Key4',BigInteger, nullable=False)
        SCORE_DATE= Column('SCORE_DATE',DateTime)
        ScoreVal= Column("ScoreVal",Float)
    
        __table_args__ = (
            PrimaryKeyConstraint(
            Key1, Key2,Key3,Key4
            ), {}
        )
    

    0 回复  |  直到 5 年前
        1
  •  1
  •   Giorgio Spedicato    4 年前

    在使用大容量插入之前,可能需要替换 np.nan 具有 None :

    mysample= mysample.replace({np.nan: None})
    

    这适用于MSSQL SERVER和ORACLE。