--- title: "Multivariate Viz (Notes)" format: html: toc: true toc-depth: 2 embed-resources: true --- ```{r include = FALSE} # This chunk just sets up some styling (eg: default size of our images) knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, fig.height = 2.75, fig.width = 4.25, fig.align = 'center') ``` ::: {.callout-note title="Learning goals"} - Review univariate & bivariate viz ideas and code. - Explore how to visualize *relationships* between *more than* 2 variables. ::: \ \ \ \ ::: {.callout-note title="Additional resources"} Watch: [more ggplot](https://www.youtube.com/watch?v=bkAJ3FAAGqA) (Lisa Lendway) ::: \ \ \ \ ## Review We have covered **a lot** of new code rapidly. This is like rapidly learning a foreign language. **Goal for data viz and its associated code:** Based on a research question, be able to draw what you want a plot to look like (a rough draft), and write `ggplot` code to make that plot from scratch. - We have only been working at visualization for 1 week! - This will take time. Let's take time now to think about what strategies will best serve us in learning this material. RStudio maintains a [collection of cheatsheets](https://rstudio.github.io/cheatsheets/) for its core packages. - Open up a PDF of the [ggplot2 cheatsheet](https://rstudio.github.io/cheatsheets/data-visualization.pdf). (There is also an [HTML version](https://rstudio.github.io/cheatsheets/html/data-visualization.html).) - Take a few minutes to look through it on your own. Think about the following: - What do you like and dislike about it? - What ideas does it give you for structuring your own notes document for `ggplot` code? - Take a few minutes to brainstorm/share with your peers some strategies you would like to try to move towards our goal above. - **Write down a plan and what you will do to stick to that plan.** \ \ \ \ Let's review some *univariate* and *bivariate* plotting concepts using some daily weather data from Australia. This is a subset of the data from the `weatherAUS` data in the `rattle` package. ```{r} library(tidyverse) # Import data weather <- read.csv("https://mac-stat.github.io/data/weather_3_locations.csv") %>% mutate(date = as.Date(date)) # Check out the first 6 rows # What are the units of observation? # How many data points do we have? # What type of variables do we have? ``` \ \ \ \ **EXAMPLE 1** Construct a plot that allows us to examine how `temp3pm` varies. ```{r} ``` \ \ \ \ **EXAMPLE 2** Construct 3 plots that address the following research question: How do afternoon temperatures (`temp3pm`) differ by `location`? ```{r} # Plot 1 (no facets & starting from a density plot of temp3pm) ggplot(weather, aes(x = temp3pm)) + geom_density() ``` ```{r} # Plot 2 (no facets or densities) ``` ```{r} # Plot 3 (facets) ``` \ \ \ \ **REFLECTION** - Temperatures tend to be highest, and most variable, in Uluru. There, they range from \~10 to \~45 with a typical temp around \~30 degrees. - Temperatures tend to be lowest in Hobart. There, they range from \~5 to \~45 with a typical temp around \~15 degrees. - Wollongong temps are in between and are the least variable from day to day. \ \ \ \ **SUBTLETIES: Defining `fill` or `color` by a variable** How we define the `fill` or `color` depends upon whether we're defining it by a named color or by some variable in our dataset. For example: - `geom___(fill = "blue")` *named* colors are defined outside the `aes`thetics and put in quotes - `geom___(aes(fill = variable))` or `ggplot(___, aes(fill = variable))` colors/fills defined by a *variable* are defined inside the `aes`thetics \ \ \ \ **EXAMPLE 3** Let's consider Wollongong alone: ```{r} # Don't worry about the syntax (we'll learn it soon) woll <- weather %>% filter(location == "Wollongong") %>% mutate(date = as.Date(date)) ``` ```{r} # How often does it raintoday? # Fill your geometric layer with the color blue. ggplot(woll, aes(x = raintoday)) ``` ```{r} # If it does raintoday, what does this tell us about raintomorrow? # Use your intuition first ggplot(woll, aes(x = raintoday)) ``` \ \ \ \ ```{r} # Now compare different approaches # Default: stacked bars ggplot(woll, aes(x = raintoday, fill = raintomorrow)) + geom_bar() ``` ```{r} # Side-by-side bars ggplot(woll, aes(x = raintoday, fill = raintomorrow)) + geom_bar(position = "dodge") ``` ```{r} # Proportional bars # position = "fill" refers to filling the frame, nothing to do with the color-related fill ggplot(woll, aes(x = raintoday, fill = raintomorrow)) + geom_bar(position = "fill") ``` \ \ \ \ **REFLECTION** There's often not one "best plot", but a *combination* of plots that provide a complete picture: - The stacked and side-by-side bars reflect that on most days, it does *not* rain. - The proportional / filled bars *lose* that information, but make it easier to compare proportions: it's more likely to rain tomorrow if it also rains today. \ \ \ \ **EXAMPLE 4** Construct a plot that illustrates how 3pm temperatures (temp3pm) vary by `date` in Wollongong. Represent each day on the plot and use a curve/line to help highlight the trends. ```{r} # THINK: What variable goes on the y-axis? # For the curve, try adding span = 0.5 to tweak the curvature ``` ```{r} # Instead of a curve that captures the general TREND, # draw a line that illustrates the movement of RAW temperatures from day to day # NOTE: We haven't learned this geom yet! Guess. ggplot(woll, aes(y = temp3pm, x = date)) ``` **NOTE:** A line plot isn't always appropriate! It can be useful in situations like this, when our data are chronological. \ \ \ \ **REFLECTION** There's a seasonal / cyclic behavior in temperatures -- they're highest in January (around 23 degrees) and lowest in July (around 16 degrees). There are also some outliers -- some abnormally hot and cold days. \ \ \ \ ## New stuff Next, let's consider the entire `weather` data for all 3 locations. The addition of `location` adds a 3rd variable into our research questions: - How does the relationship between `raintoday` and `raintomorrow` vary by `location`? - How does the behavior of `temp3pm` over `date` vary by `location`? - And so on. Thus far, we've focused on the following components of a plot: - setting up a **frame** - adding **layers** / geometric elements - splitting the plot into **facets** for different groups / categories - change the **theme**, e.g. axis labels, color, fill We'll have to think about all of this, along with **scales**. Scales change the color, fill, size, shape, or other properties according to the levels of a new *variable*. This is different than just assigning scale by, for example, `color = "blue"`. Work on the examples below in your groups. Check in with your intuition! We'll then discuss as a group as relevant. \ \ \ \ **EXAMPLE 5** ```{r} # Plot temp3pm vs temp9am # Change the code in order to indicate the location to which each data point corresponds ggplot(weather, aes(y = temp3pm, x = temp9am)) + geom_point() ``` ```{r} # Change the code in order to indicate the location to which each data point corresponds # AND identify the days on which it rained / didn't raintoday ggplot(weather, aes(y = temp3pm, x = temp9am)) + geom_point() ``` ```{r} # How many ways can you think to make that plot of temp3pm vs temp9am with info about location and rain? # Play around! ``` \ \ \ \ **EXAMPLE 6** ```{r} # Change the code in order to construct a line plot of temp3pm vs date for each separate location (no points!) ggplot(weather, aes(y = temp3pm, x = date)) + geom_line() ``` \ \ \ \ **EXAMPLE 7** ```{r} # Plot the relationship of raintomorrow & raintoday # Change the code in order to indicate this relationship by location ggplot(weather, aes(x = raintoday, fill = raintomorrow)) + geom_bar(position = "fill") ``` \ \ \ \ **HOW TO NOT GET OVERWHELMED** There's no end to the number and type of visualizations you *could* make. And it's important to not just throw spaghetti at the wall until something sticks. [FlowingData](http://flowingdata.com/2017/01/24/one-dataset-visualized-25-ways/) shows that one dataset can be visualized *many* ways, and makes good recommendations for data viz workflow, which we modify and build upon here: - **Identify simple research questions.**\ What do you want to understand about the variables or the relationships among them? - **Start with the basics and work incrementally.** - Identify what variables you want to include in your plot and what structure these have (eg: categorical, quantitative, dates) - Start simply. Build a plot of just 1 of these variables, or the relationship between 2 of these variables. - Set up a plotting frame and add just **one geometric layer at a time**. - Start tweaking: add whatever new variables you want to examine, - **Ask your plot questions.** - What questions *does* your plot answer? What questions are left *unanswered* by your plot? - What *new* questions does your plot spark / inspire? - Do you have the viz tools to answer these questions, or might you learn more? - **Focus.**\ Reporting a large number of visualizations can overwhelm the audience and obscure your conclusions. Instead, pick out a focused yet comprehensive set of visualizations. \ \ \ \ ## Exercises (required) **Directions** - These exercises include univariate viz review, bivariate viz review, and some new trivariate plots. - There are both "required" and "optional" exercises. The goal is for every student to finish the required exercises before we meet again. After that, choose your own adventure! - If you're feeling like the required exercises gave you plenty to chew on, simply stop the activity. Go back and review the material you learned. *You never need to think about the optional exercises.* - If you're feeling like your brain still has some space after completing the required exercises, try the optional ones. These exercises are either "bonus" visualizations or intuition checks about some material coming up. \ \ \ \ **The story** Though far from a perfect assessment of academic preparedness, SAT scores have historically been used as one measurement of a state's education system. The `education` dataset contains various education variables for each state: ```{r} # Import and check out data education <- read.csv("https://mac-stat.github.io/data/sat.csv") head(education) ``` A codebook is provided by Danny Kaplan who also made these data accessible: ![](https://mac-stat.github.io/images/112/SATcodebook.png) \ \ \ \ ### Exercise 1: SAT scores {.unnumbered} ### Part a {.unnumbered} Construct a plot of how the average `sat` scores vary from state to state. (Just use 1 variable -- `sat` not `state`!) ```{r} ``` ### Part b {.unnumbered} Summarize your observations from the plot. Comment on the basics: range, typical outcomes, shape. (Any theories about what might explain this non-normal shape?) \ \ \ \ ### Exercise 2: SAT scores vs per pupil spending & SAT scores vs salaries {.unnumbered} The first question we'd like to answer is: Can the variability in `sat` scores from state to state be partially explained by how much a state spends on education, specifically its per pupil spending (`expend`) and typical teacher `salary`? #### Part a {.unnumbered} ```{r} # Construct a plot of sat vs expend # Include a "best fit linear regression model" (HINT: method = "lm") ``` ```{r} # Construct a plot of sat vs salary # Include a "best fit linear regression model" (HINT: method = "lm") ``` #### Part b {.unnumbered} What are the relationship trends between SAT scores and spending? Is there anything that surprises you? \ \ \ \ ### Exercise 3: SAT scores vs per pupil spending *and* teacher salaries {.unnumbered} Construct *one* visualization of the relationship of `sat` with `salary` *and* `expend`. HINT: Start with just 2 variables and tweak that code to add the third variable. Try out a few things! ```{r} ``` \ \ \ \ ### Exercise 4: Another way to incorporate scale {.unnumbered} It can be tough to distinguish color scales and size scales for quantitative variables. Another option is to *discretize* a quantitative variable, or basically cut it up into *categories*. Construct the plot below. Check out the code and think about what's happening here. What happens if you change "2" to "3"? ```{r eval = FALSE} ggplot(education, aes(y = sat, x = salary, color = cut(expend, 2))) + geom_point() + geom_smooth(se = FALSE, method = "lm") ``` Describe the trivariate relationship between `sat`, `salary`, and `expend`. \ \ \ \ ### Exercise 5: Finally an explanation {.unnumbered} It's strange that SAT scores *seem* to decrease with spending. But we're leaving out an important variable from our analysis: the fraction of a state's students that actually take the SAT. The `fracCat` variable indicates this fraction: `low` (under 15% take the SAT), `medium` (15-45% take the SAT), and `high` (at least 45% take the SAT). #### Part a {.unnumbered} Build a univariate viz of `fracCat` to better understand how many states fall into each category. ```{r} ``` #### Part b {.unnumbered} Build 2 bivariate visualizations that demonstrate the relationship between `sat` and `fracCat`. What story does your graphic tell and why does this make contextual sense? ```{r} ``` #### Part c {.unnumbered} Make a trivariate visualization that demonstrates the relationship of `sat` with `expend` AND `fracCat`. Highlight the differences in `fracCat` groups through color AND unique trend lines. What story does your graphic tell?\ Does it still seem that SAT scores decrease as spending increases? ```{r} ``` #### Part d {.unnumbered} Putting all of this together, explain this example of **Simpson’s Paradox**. That is, why did it appear that SAT scores decrease as spending increases even though the *opposite* is true? \ \ \ \ ## Exercises (optional) ### Exercise 6: Heat maps {.unnumbered} As usual, we've only just scratched the surface! There are lots of other data viz techniques for exploring multivariate relationships. Let's start with a **heat map**. #### Part a {.unnumbered} Run the chunks below. Check out the code, but don't worry about every little detail! NOTES: - This is *not* part of the `ggplot()` grammar, making it a bit complicated. - If you're curious about what a line in the plot does, comment it out (`#`) and check out what happens! - In the plot, for each state (row), each variable (column) is scaled to indicate whether the state has a relative high value (yellow), a relatively low value (purple), or something in between (blues/greens). - You can also play with the color scheme. Type `?cm.colors` in the *console* to learn about various options. - We'll improve the plot later, so don't spend too much time trying to learn something from this plot. ```{r eval = FALSE, fig.width = 8, fig.height = 15} # Remove the "State" column and use it to label the rows # Then scale the variables plot_data <- education %>% column_to_rownames("State") %>% data.matrix() %>% scale() # Load the gplots package needed for heatmaps library(gplots) # Construct heatmap 1 heatmap.2(plot_data, dendrogram = "none", Rowv = NA, scale = "column", keysize = 0.7, density.info = "none", col = hcl.colors(256), margins = c(10, 20), colsep = c(1:7), rowsep = (1:50), sepwidth = c(0.05, 0.05), sepcolor = "white", trace = "none" ) ``` ```{r eval = FALSE, fig.width = 8, fig.height = 15} # Construct heatmap 2 heatmap.2(plot_data, dendrogram = "none", Rowv = TRUE, ### WE CHANGED THIS FROM NA TO TRUE scale = "column", keysize = 0.7, density.info = "none", col = hcl.colors(256), margins = c(10, 20), colsep = c(1:7), rowsep = (1:50), sepwidth = c(0.05, 0.05), sepcolor = "white", trace = "none" ) ``` ```{r eval = FALSE, fig.width = 8, fig.height = 15} # Construct heatmap 3 heatmap.2(plot_data, dendrogram = "row", ### WE CHANGED THIS FROM "none" TO "row" Rowv = TRUE, scale = "column", keysize = 0.7, density.info = "none", col = hcl.colors(256), margins = c(10, 20), colsep = c(1:7), rowsep = (1:50), sepwidth = c(0.05, 0.05), sepcolor = "white", trace = "none" ) ``` #### Part b {.unnumbered} In the final two plots, the states (rows) are rearranged by similarity with respect to these education metrics. The *final* plot includes a **dendrogram** which further indicates *clusters* of similar states. In short, states that have a shorter path to connection are more similar than others. Putting this all together, what insight do you gain about the education trends across U.S. states? Which states are similar? In what ways are they similar? Are there any outliers with respect to 1 or more of the education metrics? \ \ \ \ ### Exercise 7: Star plots {.unnumbered} Like heat maps, star plots indicate the relative scale of each variable for each state. Thus, we can use star maps to identify similar groups of states, and unusual states! #### Part a {.unnumbered} Construct and check out the star plot below. Note that each state has a "pie", with each segment corresponding to a different variable. The larger a segment, the larger that variable's value is in that state. For example: - Check out Minnesota. How does Minnesota's education metrics compare to those in other states? What metrics are relatively high? Relatively low? - What states appear to be similar? Do these observations agree with those that you gained from the heat map? ```{r eval = FALSE, fig.width = 10, fig.height = 20} stars(plot_data, flip.labels = FALSE, key.loc = c(10, 1.5), cex = 1, draw.segments = TRUE ) ``` #### Part b {.unnumbered} Finally, let's plot the state stars by *geographic* location! What new insight do you gain here?! ```{r eval = FALSE, fig.width = 10, fig.height = 7} stars(plot_data, flip.labels = FALSE, locations = data.matrix(as.data.frame(state.center)), # added external data to arrange by geo location key.loc = c(-110, 28), cex = 1, draw.segments = TRUE ) ``` \ \ \ \