NLTK标记化但不拆分命名实体

识别命名实体,然后遍历结果并将分块的令牌连接在一起:

>>> from nltk import ne_chunk, pos_tag, word_tokenize
>>> toks = word_tokenize('What are we going to do in San Francisco?')
>>> chunks = ne_chunk(pos_tag(toks))
>>> [ w[0] if isinstance(w, tuple) else " ".join(t[0] for t in w) for w in chunks ]
['What', 'are', 'we', 'going', 'to', 'do', 'in', 'San Francisco', '?']

每个元素 chunks (word, pos) 元组或a Tree() 包含块的各个部分。