--- title: "Wrangling Practice (Notes)" format: html: toc: true toc-depth: 2 embed-resources: true --- ```{r include = FALSE} # This chunk just sets up some styling (eg: default size of our images) knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, fig.height = 2.75, fig.width = 4.25, fig.align = 'center') ``` ::: {.callout-note title="Learning goals"} - Practice using the following wrangling verbs appropriately: `select`, `mutate`, `filter`, `arrange`, `summarize`, `group_by` - Start to develop an understanding what code will do conceptually without running it - Start to develop a knowledge of working with dates and `lubridate` functions ::: ::: {.callout-note title="Additional resources"} Read: - [Wickham, Çetinkaya-Rundel, & Grolemund, Date and Times with `lubridate`](https://r4ds.hadley.nz/datetimes) ::: \ \ \ \ **Where are we? Data preparation** ![](https://mac-stat.github.io/images/112/legos.png) \ \ \ \ **Reminder** You will make mistakes. Mistakes are important to learning. AND You will *always* make mistakes -- you will just get better at *fixing* mistakes and avoiding the most *common* mistakes. \ \ \ \ ## Warm-up \ **RECALL** Wrangling is important :). It's much of what we spend our efforts on in Data Science. There are lots of steps, hence R functions, that can go into data wrangling. But we can get far with the following 6 *wrangling verbs*: | verb | action | |:------------|:--------------------------------------------------| | `arrange` | **arrange** the *rows* according to some *column* | | `filter` | **filter** out or obtain a subset of the *rows* | | `select` | **select** a subset of *columns* | | `mutate` | **mutate** or create a *column* | | `summarize` | calculate a numerical **summary** of a *column* | | `group_by` | **group** the *rows* by a specified *column* | \ \ ::: {#fig-examples layout-nrow=3} ![Visual of arrange](https://rstudio-education.github.io/tidyverse-cookbook/images/dplyr-arrange.png){#fig-arrange} ![Visual of filter](https://rstudio-education.github.io/tidyverse-cookbook/images/dplyr-filter.png){#fig-filter} ![Visual of select](https://rstudio-education.github.io/tidyverse-cookbook/images/dplyr-select.png){#fig-select} ![Visual of mutate](https://rstudio-education.github.io/tidyverse-cookbook/images/dplyr-mutate.png){#fig-mutate} ![Visual of summarize](https://rstudio-education.github.io/tidyverse-cookbook/images/dplyr-summarise.png){#fig-summarize} ![Visual of group_by](https://rstudio-education.github.io/tidyverse-cookbook/images/dplyr-groups.png){#fig-group_by} Visuals of the six main verbs ::: \ \ \ \ **EXAMPLE 1: Viz practice** Let's start by working with some TidyTuesday data on penguins. This data includes information about penguins' flippers ("arms") and bills ("mouths" or "beaks"). [Image source.](https://media.giphy.com/media/XZn9yRAjnVEQ0/giphy.gif?cid=790b7611u5bjjkvt54giyxvi6j956yet0m4e6v9dtqwkv87f&ep=v1_gifs_search&rid=giphy.gif&ct=g) ![](https://media.giphy.com/media/XZn9yRAjnVEQ0/giphy.gif?cid=790b7611u5bjjkvt54giyxvi6j956yet0m4e6v9dtqwkv87f&ep=v1_gifs_search&rid=giphy.gif&ct=g) Let's import this using `read_csv()`, a function in the `tidyverse` package. For the most part, this is similar to `read.csv()`, though `read_csv()` can be more efficient at importing large datasets. ```{r} library(tidyverse) penguins <- read_csv('https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-07-28/penguins.csv') # Check it out head(penguins) ``` Construct a plot that allows us to examine how the relationship between body mass and bill length varies by species and sex. ```{r} ``` \ \ \ \ **EXAMPLE 2: verb review** Use the 6 wrangling verbs to address each task below. You can tack on `%>% head()` to print out just 6 rows to keep your knit document manageable. Most of these require just 1 verb. ```{r} # Get data on only Adelie penguins that weigh more than 4700g # Get data on penguin body mass only # Show just the first 6 rows # Sort the penguins from smallest to largest body mass # Show just the first 6 rows # Calculate the average body mass across all penguins # Note: na.rm = TRUE removes the NAs from the calculation # Calculate the average body mass by species # Create a new column that records body mass in kilograms, not grams # NOTE: there are 1000 g in 1 kg # Show just the first 6 rows ``` \ \ \ \ **EXAMPLE 3: Counting** How many penguins of each species do we have? Create a viz that addresses this question. ```{r} ggplot(penguins, aes(x = species)) ``` To be more precise, we can calculate the number of penguins of each species using our 6 verbs. HINT: `n()` calculates group size. ```{r} ``` The `count()` verb provides a handy shortcut! ```{r} penguins %>% count(species) ``` \ \ \ \ **EXAMPLE 4: Multiple verbs** Let's practice *combining* some verbs. For each task: - Translate the prompt into our 6 verbs. That is, think before you type. - Build your code line by line. It's important to understand what's being piped into each function! - Ask what you can rearrange and still get the same result. - Read your final code like a paragraph / a conversation. Would another person be able to follow your logic? ```{r} # Sort Gentoo penguins from biggest to smallest with respect to their # bill length in cm (there are 10 mm in a cm) ``` ```{r} # Sort the species from smallest to biggest with respect to their # average bill length in cm ``` \ \ \ \ **EXAMPLE 5: Interpret this code** Let's practice reading and making sense of somebody *else*'s code. What do you think this produces? - How many columns? Rows? - What are the column names? - What's represented in each row? Once you've thought about it, put the code inside a chunk and run it! penguins %\>% filter(species == "Chinstrap") %\>% group_by(sex) %\>% summarize(min = min(body_mass_g), max = max(body_mass_g)) %\>% mutate(range = max - min) \ \ \ \ ## Exercises Part 1: Same verbs, new tricks **Goals** - Part 1 - Learn some new ways to use our 6 verbs, using the penguins. - Explore how to work *dates* (eg: "2024-02-20"). - Part 2: You will practice wrangling using the birthday data you explored visually in Homework 1. These are *similar* to exercises that will be on Homework 4. **Directions** - Work together! - Stay on track / focused on this activity. This is helpful to you, *and* the students around you :) \ \ \ \ ### Exercise 1: More filtering {.unnumbered} Recall the "logical comparison operators" we can use to `filter()` our data: | symbol | meaning | |:-----------------|:--------------------------| | == | equal to | | != | not equal to | | > | greater than | | >= | greater than or equal to | | < | less than | | <= | less than or equal to | | %in% c(***,*** ) | a list of multiple values | #### Part a {.unnumbered} ```{r} # Create a dataset with just Adelie and Chinstrap using %in% # Pipe this into `count(species)` to confirm that you only have these 2 species # ___ %>% # filter(___) %>% # count(species) ``` ```{r} # Create a dataset with just Adelie and Chinstrap using != # Pipe this into `count(species)` to confirm that you only have these 2 species # ___ %>% # filter(___) %>% # count(species) ``` #### Part b {.unnumbered} Notice that some of our penguins have missing (`NA`) data on some values: ```{r} head(penguins) ``` There are many ways to handle this. The right approach depends upon your research goals. A general rule is: Only get rid of observations with missing data if they're missing data on variables you *need* for the specific task at hand! **Example 1** Suppose our research focus is *just* on `body_mass_g`. 2 penguins are missing this info: ```{r} # NOTE the use of is.na() penguins %>% summarize(sum(is.na(body_mass_g))) ``` Let's define a new dataset that removes these penguins: ```{r} # NOTE the use of is.na() penguins_w_body_mass <- penguins %>% filter(!is.na(body_mass_g)) # Compare the number of penguins in this vs the original data nrow(penguins_w_body_mass) nrow(penguins) ``` Note that some penguins in `penguins_w_body_mass` are missing info on `sex`, but we don't care since that's not related to our research question: ```{r} penguins_w_body_mass %>% summarize(sum(is.na(sex))) ``` **Example 2** In the *very rare case* that we need complete information on every variable for the specific task at hand, we can use `na.omit()` to get rid of *any* penguin that's missing info on *any* variable: ```{r} penguins_complete <- penguins %>% na.omit() ``` How many penguins did this eliminate? ```{r} nrow(penguins_complete) nrow(penguins) ``` #### Part c {.unnumbered} Explain why we should only use `na.omit()` in extreme circumstances. \ \ \ \ ### Exercise 2: More selecting {.unnumbered} Being able to `select()` only certain columns can help simplify our data. This is especially important when we're working with *lots* of columns (which we haven't done yet). It can also get tedious to type out every column of interest. Here are some shortcuts: - `-` *removes* a given variable and keeps all others (e.g. `select(-island)`) - `starts_with("___")`, `ends_with("___")`, or `contains("___")` selects only the columns that either start with, end with, or simply contain the given string of characters Use these *shortcuts* to create the following datasets. ```{r} # First: recall the variable names names(penguins) ``` ```{r} # Use a shortcut to keep everything but the year and island variables ``` ```{r} # Use a shortcut to keep only species and the penguin characteristics measured in mm ``` ```{r} # Use a shortcut to keep only species and bill-related measurements ``` ```{r} # Use a shortcut to keep only species and the length-related characteristics ``` \ \ \ \ ### Exercise 3: Arranging, counting, & grouping by multiple variables {.unnumbered} We've done examples where we need to `filter()` by more than one variable, or `select()` more than one variable. Use your intuition for how we can `arrange()`, `count()`, and `group_by()` more than one variable. ```{r} # Change this code to sort the penguins by species, and then island name # NOTE: The first row should be an Adelie penguin living on Biscoe island penguins %>% arrange(species) ``` ```{r} # Change this code to count the number of male/female penguins observed for each species penguins %>% count(species) ``` ```{r} # Change this code to calculate the average body mass by species and sex penguins %>% group_by(species) %>% summarize(mean = mean(body_mass_g, na.rm = TRUE)) ``` \ \ \ \ ### Exercise 4: Dates {.unnumbered} Before some wrangling practice, let's explore another important concept: working with or mutating *date* variables. Dates are a whole special object type or class in RStudio that automatically respect the order of time. ```{r} # Get today's date as.Date(today()) # Let's store this as "today" so we can work with it below today <- as.Date(today()) # Check out the class of this object class(today) ``` The `lubridate` package inside `tidyverse` contains functions that can extract various information from dates. Let's learn about some of the most common functions by applying them to `today`. For each, make a comment on what the function does ```{r} year(today) ``` ```{r} # What do these lines produce / what's their difference? month(today) month(today, label = TRUE) ``` ```{r} # What does this number mean? week(today) ``` ```{r} # What do these lines produce / what's their difference? mday(today) yday(today) # This is often called the "Julian day" ``` ```{r} # What do these lines produce / what's their difference? wday(today) wday(today, label = TRUE) ``` ```{r} # What do the results of these 2 lines tell us? today >= ymd("2024-02-14") today < ymd("2024-02-14") ``` \ \ \ \ ## Exercises Part 2: Application RECALL: The remaining exercises are *similar* to some on Homework 4, thus solutions aren't provided. Let's apply these ideas to the daily `Birthdays` dataset in the `mosaic` package that we explored in Homework 2: ```{r} library(mosaic) data("Birthdays") head(Birthdays) ``` `Birthdays` gives the number of births recorded on each day of the year in each state from 1969 to 1988.[^09-wrangling-practice-notes-1] We can use our wrangling skills to understand some drivers of daily births. Putting these all together can be challenging! Remember the following ways to make tasks more manageable: [^09-wrangling-practice-notes-1]: The `fivethirtyeight` package has more recent data. - Translate the prompt into our 6 verbs (and `count()`). That is, think before you type. - Build your code line by line. It's important to understand what's being piped into each function! \ \ \ \ ### Exercise 5: Warming up {.unnumbered} ```{r} # How many days of data do we have for each state? # How many total births were there in this time period? # How many total births were there per state in this time period, sorted from low to high? ``` \ \ \ \ ### Exercise 6: Homework 2 reprise {.unnumbered} Create a new dataset named `daily_births` that includes the total number of births per day (across all states) and the corresponding day of the week (eg: Mon). NOTE: Name the column with total births so that it's easier to wrangle and plot. ```{r} ``` Using this data, construct a plot of `births` over time, indicating the day of week. ```{r} ``` \ \ \ \ ### Exercise 7: Wrangle & plot {.unnumbered} For each prompt below, you can decide whether you want to: (1) wrangle and store data, then plot; or (2) wrangle data and pipe directly into ggplot. For example: ```{r} penguins %>% filter(species != "Gentoo") %>% ggplot(aes(y = bill_length_mm, x = bill_depth_mm, color = species)) + geom_point() ``` #### Part a {.unnumbered} Calculate the total number of births in each month and year (eg: Jan 1969, Feb 1969, ...). Label month by names not numbers (Jan not 1). Then plot the births by month and comment on what you learn. ```{r} ``` #### Part b {.unnumbered} In 1988, calculate the total number of births per week in each state. (Get rid of week "53", which isn't a complete week!) Then make a line plot of births by week for each state, and comment on what you learn. For example, do you notice any seasonal trends? Are these the same in every state? Any outliers? ```{r} ``` #### Part c {.unnumbered} Repeat the above for just Minnesota (MN) and Louisiana (LA). MN has one of the coldest climates, and LA has one of the warmest. How do their seasonal trends compare? (Do you think these trends are similar in other colder and warmer states? Try it!) ```{r} ``` \ \ \ \ ### Exercise 8: More practice {.unnumbered} #### Part a {.unnumbered} Create a dataset with only births in Massachusetts (MA) in 1979, and sort the days from those with the most births to those with the fewest. ```{r} ``` #### Part b {.unnumbered} Make a table showing the five states with the most births between September 9, 1979 and September 12, 1979, including the 9th and 12th. Arrange the table in descending order of births. \ \ \ \