Comments on: Similarity of texts: The Vector Space Model with Python http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/ Where stuff from my brain lands Wed, 20 Aug 2008 03:36:15 +0000 http://wordpress.org/?v=2.6 By: Dennis http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-509 Dennis Mon, 12 May 2008 19:56:13 +0000 http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-509 The vector represents a map of words included in the text as compared to words included in the domain of all words included in all texts. I suppose you'd get the same result if the vectors were only long enough to include the words in each text, but you'd have to do extra programming to figure out which word was at the maximum index length and then create the vector that size. It seems much easier to just create a vector the maximum size. IIRC, there isn't any overhead with creating a larger vector with the NumPy library. The vector represents a map of words included in the text as compared to words included in the domain of all words included in all texts. I suppose you’d get the same result if the vectors were only long enough to include the words in each text, but you’d have to do extra programming to figure out which word was at the maximum index length and then create the vector that size. It seems much easier to just create a vector the maximum size. IIRC, there isn’t any overhead with creating a larger vector with the NumPy library.

]]> By: Jeremy http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-508 Jeremy Mon, 12 May 2008 01:26:01 +0000 http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-508 This is a neat tutorial, but out of curiosity, why do you need to set the vectors to the same length? The cosine measurement normalizes vectors, IIRC (else why would we divide by the product of the magnitudes of v1 and v2?). In fact, cosine similarity is popular precisely because it can make sense of vectors of various magnitudes. This is a neat tutorial, but out of curiosity, why do you need to set the vectors to the same length? The cosine measurement normalizes vectors, IIRC (else why would we divide by the product of the magnitudes of v1 and v2?). In fact, cosine similarity is popular precisely because it can make sense of vectors of various magnitudes.

]]> By: Dennis http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-172 Dennis Wed, 12 Dec 2007 13:05:39 +0000 http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-172 Thanks Tyler. I've updated the link. Thanks Tyler. I’ve updated the link.

]]> By: Tyler http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-170 Tyler Wed, 12 Dec 2007 12:02:24 +0000 http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-170 Thanks for linking to my calculator. I am moving the cosine similarity calculator over to www.appliedsoftwaredesign.com very soon. Sorry for the inconvenience. Thanks for linking to my calculator. I am moving the cosine similarity calculator over to http://www.appliedsoftwaredesign.com very soon. Sorry for the inconvenience.

]]> By: Dennis http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-27 Dennis Mon, 22 Oct 2007 03:46:18 +0000 http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-27 The Vector Space Model and diff actually don't have very much in common. diff outputs the differences between texts based on each line of text. Anything not output by diff is the same in the texts. This program does not output the similarities in the texts. Instead, it outputs a number that represents how similar the texts are. If the output is 0, there is no similarity. An output of 1 means they are pretty much the same documents. With the Vector Space Model, you could have an output of 1 even if the words were in a completely different order. The Vector Space Model and diff actually don’t have very much in common. diff outputs the differences between texts based on each line of text. Anything not output by diff is the same in the texts. This program does not output the similarities in the texts. Instead, it outputs a number that represents how similar the texts are. If the output is 0, there is no similarity. An output of 1 means they are pretty much the same documents. With the Vector Space Model, you could have an output of 1 even if the words were in a completely different order.

]]> By: Tmpvar http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-26 Tmpvar Mon, 22 Oct 2007 02:17:27 +0000 http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/#comment-26 Great post, seems very simular to diff. Cheers Great post, seems very simular to diff.

Cheers

]]>