Introduction
Text analysis is a powerful tool that allows us to pull out trends and do distant reading of large corpora of text. In this part of my project, I have used Voyant Tools to run a number of text analysis tasks, and have extracted the particularly relevant ones to showcase here. I began this process by stripping out the prepended and appended text that Project Gutenberg adds, in order to just get the relevant text data. From here, I analyzed the different visualizations provided by Voyant Tools, and interpreted them below.
Corpus Summary
We can begin by analyzing the summary of the corpus that Voyant Tools provides for us, to get a general sense of what we are looking at.
This corpus has 1 document with 667,643 total words and 19,757 unique word forms.
Vocabulary Density: 0.030
Readability Index: 6.738
Average Words Per Sentence: 67.4
Most frequent words in the corpus:
river (4488); side (2039); small (1893); men (1829); miles (1678)
This summary gives us some basic metrics such as total words, unique words, vocabulary density, and readability, as well as a list of the five most common words and their occurances. Interestingly, “river” appears as number one, since Lewis and Clark heavily relied on rivers to travel, and frequently journaled about the different features of the rivers.
Examining Word Frequency
This plot divides up the corpus into ten arbitrary sections - these do not correspond to any thematic sections within the text, but rather exist to plot data. I have selected the frequency of the words lewis* and clark* to be plotted (with the asterisk serving as a wildcard character). Interestingly, we can we that the frequency of lewis* is generally much higher throughout the text, indicating that he was the primary author of the journals, except for sections eight and nine towards the end. Additionally, the freqency of both names trends downward throughout the corpus, possibly as Lewis and Clark begin writing more about events of their jouney and less about themselves. Feel free to interact with the visualization above, plotting different words to see how their frequency changes over the length of the text.
Word Frequency and Collocation
This tool, known as a TermsBerry, provides two very important pieces of information. It represents the most frequent words in the corpus as small circles, with size proportional to the frequency of the word. Additionally, as you hover over the words, the tool highlights other words that appear within two positions (by default, this can be increased) of the target word, and gives the counts of these occurances. This helps us to pick out common trends in the corpus; for example, hovering over the word “killed,” we can see that the two most common collocated words are “deer” (256) and “elk” (141). This tells us that Lewis and Clark most likely subsisted on a diet including deer and elk, as these were common game animals in the American West during the time of their expedition.
WordTree Visualization
Finally, the WordTree tool provided by Voyant allows us to see some of the collocation trends that we saw at a high level in the previous section in finer detail. This tool provides a way to not just see collocation trends, but to also see the relative ordering of commonly collocated words. Branches to the right of the tree represent words that appear after the target word in the corpus, while branches to the left of the tree represent words that precede the target word. One interesting result that I noticed, when using “river” as the target word, is that when the preceding “the” is clicked, nearly all the words somehow relate to spacial positioning. This highlights the fact that Lewis and Clark solely relied on rivers not only for transportation, but also for navigation. As they were traveling through unmapped territory, landmarks such as rivers and other natural features were their main methods of keeping track of their location.