Session 7

Tidy Data

Question: What are the three rules of tidy data?

Question: Which function from tidyr takes data from wide to long format?

1. gather()
2. spread()

Working with Dates

Question: Which of the following functions is used to:

1. extract the month from a date;
2. add a number of months to a date;
3. find the first day of the month to which a date belongs?

1. months()
2. floor_date()
3. month()

Question: Given the following dates

• "15 March 2018 02pm"
• "04/29/18 02:49"
• "Fri Feb 9 2018 14:49:29"
• "2018-09-16"

which functions can be used to convert the dates from strings to dates or date-times?

1. mdy_hm()
2. dmy_h()
3. ymd()
4. mdy_hms()

Clean names, remove empties, find duplicates

Three functions are useful on a daily basis:

1. clean_names(): Makes nice, easy to use names

2. remove_empty(): Removes rows or columns where everything is missing

3. get_dupes(df, col1, col2): Get rows in df that have duplicate values in col1 and col2.

library(janitor)
library(readr)
library(lubridate)

cr <- read_csv("CancerRegistryData.csv") %>% 
  clean_names() %>% 
  remove_empty() %>% 
  mutate(
    date_of_diagnosis = mdy(date_of_diagnosis),
    date_last_patient_contact_dead = mdy(date_last_patient_contact_dead)
  )

Parsed with column specification:
cols(
  .default = col_character(),
  `Medical Record Number` = col_integer(),
  `Sequence Number` = col_integer(),
  `Age at Diagnosis` = col_integer(),
  `Age, Current` = col_integer(),
  `Survival Time (months)` = col_integer(),
  `Clinical Tumor Size` = col_double()
)

See spec(...) for full column specifications.

names(cr)

 [1] "medical_record_number"          "sequence_number"               
 [3] "site_primary_icd_o_3"           "histology_behavior_icd_o_3"    
 [5] "date_of_diagnosis"              "age_at_diagnosis"              
 [7] "age_current"                    "sex"                           
 [9] "spanish_hispanic_origin"        "race"                          
[11] "vital_status"                   "date_last_patient_contact_dead"
[13] "survival_time_months"           "dc_cause_of_death"             
[15] "clinical_tumor_size"            "clinical_size_unit"            
[17] "tobacco_use_fcds"               "summary_of_rx_1st_course"      
[19] "payer_source_1_at_dx"           "patient_history_alcohol"       
[21] "marital_status_at_dx"          

Learn more at the janitor package website.

Quick String Review

Which of the following are valid R strings:

1. "I am OZ, the Great and Terrible"

2. "You're a humbug!"

3. "There's no place like 'home'"

4. "\"Can't you give me brains?,\" asked Scarecrow"

Load Packages and Data

This script loads the tidyverse, lubridate, and janitor and then loads the example Cancer Registry Data.

source("https://git.io/fAOVu")

Work Directly with Strings

All of the stringr functions start with str_, like str_c(). This function takes parts of a string and concatenates them, just like c() takes parts of a vector and puts them together.

str_c("Something", "interesting")

[1] "Somethinginteresting"

str_c("Something", "interesting", sep = " ")

[1] "Something interesting"

str_c("Something", c("funny", "odd"), sep = " ")

[1] "Something funny" "Something odd"  

str_c("Something", c("funny", "odd"), sep = " ", collapse = " or ")

[1] "Something funny or Something odd"

str_detect("apple", "ap")

[1] TRUE

str_detect(c("apple", "grape", "banana"), "ap")

[1]  TRUE  TRUE FALSE

cr %>% 
  filter(str_detect(histology_behavior_icd_o_3, "CARCINOMA"))

# A tibble: 151 x 21
   medical_record_… sequence_number site_primary_ic… histology_behav…
              <int>           <int> <chr>            <chr>           
 1           687391               1 C509 (BREAST NO… 85202 (LOBULAR …
 2           906008               0 C509 (BREAST NO… 85003 (INFILTRA…
 3           162232               0 C503 (BREAST LI… 85003 (INFILTRA…
 4           710246               0 C503 (BREAST LI… 85003 (INFILTRA…
 5           317251               0 C509 (BREAST NO… 85003 (INFILTRA…
 6           175897               0 C504 (BREAST UO… 85003 (INFILTRA…
 7           197932               0 C504 (BREAST UO… 85003 (INFILTRA…
 8           446531               1 C504 (BREAST UO… 85223 (INFILTRA…
 9           148456               0 C509 (BREAST NO… 85003 (INFILTRA…
10           334557               1 C504 (BREAST UO… 85003 (INFILTRA…
# ... with 141 more rows, and 17 more variables: date_of_diagnosis <date>,
#   age_at_diagnosis <int>, age_current <int>, sex <chr>,
#   spanish_hispanic_origin <chr>, race <chr>, vital_status <chr>,
#   date_last_patient_contact_dead <date>, survival_time_months <int>,
#   dc_cause_of_death <chr>, clinical_tumor_size <dbl>,
#   clinical_size_unit <chr>, tobacco_use_fcds <chr>,
#   summary_of_rx_1st_course <chr>, payer_source_1_at_dx <chr>,
#   patient_history_alcohol <chr>, marital_status_at_dx <chr>

cr %>% filter(
  str_detect(histology_behavior_icd_o_3, "CARCINOMA"),
  !str_detect(histology_behavior_icd_o_3, "INFILTRATING")
)

# A tibble: 37 x 21
   medical_record_… sequence_number site_primary_ic… histology_behav…
              <int>           <int> <chr>            <chr>           
 1           687391               1 C509 (BREAST NO… 85202 (LOBULAR …
 2           341050               1 C569 (OVARY)     84603 (PAPILLAR…
 3           896356               2 C182 (COLON ASC… 81403 (ADENOCAR…
 4           896356               1 C509 (BREAST NO… 85203 (LOBULAR …
 5           644232               0 C508 (BREAST OV… 82113 (TUBULAR …
 6           218883               0 C502 (BREAST UI… 85202 (LOBULAR …
 7           558013               1 C649 (KIDNEY NO… 83123 (RENAL CE…
 8           563418               0 C509 (BREAST NO… 85203 (LOBULAR …
 9           883521               0 C509 (BREAST NO… 81403 (ADENOCAR…
10           289034               1 C504 (BREAST UO… 82113 (TUBULAR …
# ... with 27 more rows, and 17 more variables: date_of_diagnosis <date>,
#   age_at_diagnosis <int>, age_current <int>, sex <chr>,
#   spanish_hispanic_origin <chr>, race <chr>, vital_status <chr>,
#   date_last_patient_contact_dead <date>, survival_time_months <int>,
#   dc_cause_of_death <chr>, clinical_tumor_size <dbl>,
#   clinical_size_unit <chr>, tobacco_use_fcds <chr>,
#   summary_of_rx_1st_course <chr>, payer_source_1_at_dx <chr>,
#   patient_history_alcohol <chr>, marital_status_at_dx <chr>

str_count("apple", "p")

[1] 2

str_count(c("apple", "grape", "banana"), "p")

[1] 2 1 0

str_extract(c("apple", "grape", "banana"), "p")

[1] "p" "p" NA 

str_remove(c("apple fruit", "grapefruit", "fruit banana"), "fruit")

[1] "apple "  "grape"   " banana"

str_replace(c("apple fruit", "grapefruit", "fruit banana"), "fruit", "FRUIT")

[1] "apple FRUIT"  "grapeFRUIT"   "FRUIT banana"

str_remove("apple fruit and banana fruit", "fruit")

[1] "apple  and banana fruit"

str_remove_all("apple fruit and banana fruit", "fruit")

[1] "apple  and banana "

str_replace_all("apple fruit and banana fruit", "fruit", "FRUIT")

[1] "apple FRUIT and banana FRUIT"

str_to_lower("STOP SHOUTING")

[1] "stop shouting"

str_to_upper("talk loudly")

[1] "TALK LOUDLY"

str_to_title("be a little more professional")

[1] "Be A Little More Professional"

What is a regular expression?

Our diagnosis records include an ICD code for the location of the cancer, but these values include additional text:

select(cr, site_primary_icd_o_3)

# A tibble: 167 x 1
   site_primary_icd_o_3
   <chr>               
 1 C509 (BREAST NOS)   
 2 C509 (BREAST NOS)   
 3 C503 (BREAST LIQ)   
 4 C503 (BREAST LIQ)   
 5 C509 (BREAST NOS)   
 6 C504 (BREAST UOQ)   
 7 C504 (BREAST UOQ)   
 8 C504 (BREAST UOQ)   
 9 C509 (BREAST NOS)   
10 C509 (BREAST NOS)   
# ... with 157 more rows

How would you describe in words the structure of a breast cancer related IDC code?

Pretend you’re writing instructions for a robot that can only look at one letter of input at a time. How would you instruct the robot to extract just the IDC code "C504" from the full string "C504 (BREAST UOQ)"?

Here are the ICD codes found in this example.

 [1] "C182" "C383" "C421" "C449" "C501" "C502" "C503" "C504" "C505" "C506"
[11] "C508" "C509" "C529" "C541" "C569" "C649" "C700" "C739" "C778" "C809"

Regular expressions are a language for describing patterns in strings.

An RStudio Addin to help you

RStudio Addin - RegExplain.

Basic Regular Expressions

"abc123"

Backslashes!

As we saw with the examples above, there are characters that have special meaning in strings in R. For example, a string starts and ends with " or '.

writeLines("There is no place like home")

There is no place like home

If we used " to open or close the string but we want to have another " inside the string, we need to escape the quote.

writeLines("There is no place like \"home\"")

There is no place like "home"

This is called escaping because adding the backslash lets you escape the special semantics of the character. It indicates that you don’t want a special ", you want a regular literal ".

What if you want to add a backslash? The backslash is also special (it does the escaping), so to write a literal backslash you have to escape the backslash, too.

writeLines("\\")

\

Some of the regular expressions above include backslashes. To write this in a string in R, you need to escape the backslash.

str_extract("abc123", "\w")

Error: '\w' is an unrecognized escape in character string starting ""\w"

str_extract("abc123", "\\w")

[1] "a"

str_extract("abc123", "\\w\\w\\d")

[1] "bc1"

writeLines("\\w\\w\\d")

\w\w\d

Escaping for R or for the regular expression When writing regular expressions, you’re operating at two levels. First, you are writing a code into R to store the strings. Second, when R is done processing that code we want the output to be a regular expression.

So above, we write "\\w" into R so that we get a regular expression like "\w".

But regular expressions also have special characters, like . and +. If we want to write literal versions of these in our regular expressions – for example to match the dot in 10.123 – we need to escape the dot at the regular expression level.

Adding one backslash escapes at the R level:

str_extract("10.123", "0\.1")

Error: '\.' is an unrecognized escape in character string starting ""0\."

But this doesn’t pass down to the regular expression level.

writeLines("0\.1")

Error: '\.' is an unrecognized escape in character string starting ""0\."

To get there we need yet another backslash.

str_extract("10.123", "0\\.1")

[1] "0.1"

writeLines("0\\.1")

0\.1

How can you write an actual backslash at the regular expression level?

writeLines("\\\\")

\\