代码之家 › 专栏 › 技术社区 › OverflowingTheGlass

Gensim Doc2Vec访问向量(按文档作者)

doc2vec gensim python

1

OverflowingTheGlass · 技术社区 · 6 年前

我在df中有三个文档:

id    author    document
12X   john      the cat sat
12Y   jane      the dog ran
12Z   jane      the hippo ate

这些文档被转换为 TaggedDocuments 标记是语义上无意义的int的典型实践:

def read_corpus(documents):
    for i, plot in enumerate(documents):
        yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(plot, max_len=30), [i])

train_corpus = list(read_corpus(df.document))

然后使用该语料库来训练我的 Doc2Vec 型号:

model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
model.build_vocab(train_corpus)
model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)

模型的结果向量如下所示:

model.docvecs.vectors_docs

如何将原始df与结果向量联系起来?现在,所有文档都经过了训练,每个文档都有了向量,我想按作者查询向量集。例如,如果我只想为Jane返回一组向量,我该怎么做?

我认为基本的想法是识别与Jane对应的int标记,然后执行类似的操作来访问它们:

from operator import itemgetter 
a = model.docvecs.vectors_docs
b = [1, 2]
itemgetter(*b)(a)

我该如何识别标签呢?它们只对模型和标记的文档有意义,所以它们不会连接回我的原始df。

1 回复 | 直到 6 年前

1

3

vumaasha 6 年前

我尝试了一个使用Gensim的简单示例。我认为这里的方法应该适合你

import gensim
training_sentences = ['This is some document from Author {}'.format(i) for i in range(1,10)]
def read_corpus():
    for i,line in enumerate(training_sentences):
        # lets use the tag to identify the document and author
        yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), ['Doc{}_Author{}'.format(i,i)])

您也可以直接从pandas\u df中准备训练语料库,如下所示

data_df = pd.DataFrame({'doc':training_sentences,'doc_id':[i for i in range (1,10)],'author_id':[10+i for i in range (1,10)]})
data_df.head()
tagged_docs = data_df.apply(lambda x:gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(x.doc),['doc{}_auth{}'.format(x.doc_id,x.author_id)]),axis=1)
training_corpus = tagged_docs.values

>> array([ TaggedDocument(words=['this', 'is', 'some', 'document', 'from', 'author'], tags=['doc1_auth11']),
       TaggedDocument(words=['this', 'is', 'some', 'document', 'from', 'author'], tags=['doc2_auth12']),

# training
model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
train_corpus = list(read_corpus())
model.build_vocab(train_corpus)
model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)
# indexing
model.docvecs.index2entity

>>
['Doc0_Author0',
 'Doc1_Author1',
 'Doc2_Author2',
 'Doc3_Author3',
 'Doc4_Author4',
 'Doc5_Author5',
 'Doc6_Author6',
 'Doc7_Author7',
 'Doc8_Author8']

现在,要访问author1的document1对应的向量,可以执行以下操作

model.docvecs[model.docvecs.index2entity.index('Doc1_Author1')]

array([  8.08026362e-03,   4.27437993e-03,  -7.73820514e-03,
        -7.40669528e-03,   6.36066869e-03,   4.03292105e-03,
         9.60215740e-03,  -4.26750770e-03,  -1.34797185e-03,
        -9.02472902e-03,   6.25275355e-03,  -2.49505695e-03,
         3.18572600e-03,   2.56929174e-03,  -4.17032139e-03,
        -2.33384431e-03,  -5.10744564e-03,  -5.29057207e-03,
         5.41675789e-03,   5.83767192e-03,  -5.91145828e-03,
         5.91885624e-03,  -1.00465110e-02,   8.32535885e-03,
         9.72494949e-03,  -7.35746371e-03,  -1.86231872e-03,
         8.94813929e-05,  -4.11528209e-03,  -9.72509012e-03,
        -6.52212929e-03,  -8.83922912e-03,   9.46981460e-03,
        -3.90578934e-04,   6.74136635e-03,  -5.24599617e-03,
         9.73031297e-03,  -8.77021812e-03,  -5.55411633e-03,
        -7.21857697e-03,  -4.50362219e-03,  -4.06361837e-03,
         2.57276138e-03,   1.76626759e-06,  -8.08755495e-03,
        -1.48400548e-03,  -5.26673114e-03,  -7.78301107e-03,
        -4.24248137e-04,  -7.99000356e-03], dtype=float32)

是的,这使用doc-author对排序,您可以只使用doc\u id并维护一个单独的索引,如 {doc_id:author_id} 在python dict中,如果要按作者筛选,请使用 {author_id : [docids,...]}