turicreate.text_analytics.tf_idf

turicreate.text_analytics.tf_idf(text)

Compute the TF-IDF scores for each word in each document. The collection of documents must be in bag-of-words format.

\[\mbox{TF-IDF}(w, d) = tf(w, d) * log(N / f(w))\]

where \(tf(w, d)\) is the number of times word \(w\) appeared in document \(d\), \(f(w)\) is the number of documents word \(w\) appeared in, \(N\) is the number of documents, and we use the natural logarithm.

Parameters:

text : SArray[str | dict | list]

Input text data.

Returns:

out : SArray[dict]

The same document corpus where each score has been replaced by the TF-IDF transformation.

See also

count_words, count_ngrams, tokenize

References

• Wikipedia - TF-IDF

Examples

>>> import turicreate

>>> docs = turicreate.SArray('https://static.turi.com/datasets/nips-text')
>>> docs_tfidf = turicreate.text_analytics.tf_idf(docs)