24.1 One-token-per-row (unnest_tokens)

• String: Text can, of course, be stored as strings, i.e., character vectors, within R, and often text data is first read into memory in this form.
• Corpus: These types of objects typically contain raw strings annotated with additional metadata and details.
• Document-term matrix: This is a sparse matrix describing a collection (i.e., a corpus) of documents with one row for each document and one column for each term. The value in the matrix is typically word count or tf-idf (see Chapter 3).

library(tibble)

text <- c("I'm your basic average girl",
          "And I'm here to save the world",
          "You can't stop me",
          "Cause I'm Kim Pos-si-ble")

text
## [1] "I'm your basic average girl"    "And I'm here to save the world"
## [3] "You can't stop me"              "Cause I'm Kim Pos-si-ble"

text_df <- data_frame(line = 1:4, text = text)
text_df
## # A tibble: 4 x 2
##    line text                          
##   <int> <chr>                         
## 1     1 I'm your basic average girl   
## 2     2 And I'm here to save the world
## 3     3 You can't stop me             
## 4     4 Cause I'm Kim Pos-si-ble

library(tidytext)
library(magrittr)

text_df %>%
  unnest_tokens(word, text)
## # A tibble: 22 x 2
##     line word   
##    <int> <chr>  
##  1     1 i'm    
##  2     1 your   
##  3     1 basic  
##  4     1 average
##  5     1 girl   
##  6     2 and    
##  7     2 i'm    
##  8     2 here   
##  9     2 to     
## 10     2 save   
## # ... with 12 more rows