--- title: "Homework 7: Exploratory data analysis" author: "PUT YOUR NAME HERE" date: now date-format: "YYYY-MM-DDTHH:mm:ssZ" format: html: toc: true toc-depth: 2 embed-resources: true code-tools: true df-print: paged --- ```{r include = FALSE} # This chunk just sets up some styling (eg: default size of our images) knitr::opts_chunk$set( collapse = TRUE, warning = FALSE, message = FALSE, fig.height = 3, fig.width = 5, fig.align = 'center') ``` \ **DIRECTIONS** - Save this file as `homework_7.qmd` in your `DS112 > homework` folder. - Type your name in line 3 above (where it says "author"). - Type your responses in this template. - Do not modify the structure of this document (e.g. don't change section headers, spacing, etc). - There are lots of ways to do things in R. In these exercises, be sure to use the `tidyverse` code and style / structure we've learned in this class. > **Important:** If you find and use code that we haven't covered in class (through internet searches, GenAI, human resources), **you need to cite your sources and explain what the new code does**. You can add this text in a paragraph below your main response for an exercise part. \ \ \ \ # Context The `running_112.csv` data file posted in the Homework 7 link on Moodle contains data on people that have run in the annual 10-mile "Cherry Blossom" race in Washington, D.C.. Our general goal will be to use these data to explore the broad research question: **What's the relationship between running time and age?** You will address this questions through *guided* EDA. That is, I'll tell you what steps to take in this process. The steps provided are merely *one* approach, not the *only* approach. With that, pay special attention to: - What follow-up questions *you* have along the way / how your EDA might deviate from that here. - The general vibe behind the EDA process. \ \ \ \ # Exercise 1: Import and get to know the data The first step in our EDA is to import and get to know some basics about the data structure. ## Part a Import the `running_112.csv` data into RStudio--this dataset is posted on Moodle under the Homework 7 assignment. NOTE: This data is a modified version of the `Cherry` data in the `mdsr` package, and contains different info than the running data we've used elsewhere. You must use a **local copy** of the provided `running_112.csv` dataset for these exercises. You may not import this dataset from the internet, an R package, etc. ```{r} ``` Check out and familiarize yourself with the data in a `View()` window. Included in each row is data on: - `name.yob` = name and year of birth (yob) for the runner - `sex` = binary sex - `net` = net running time to complete the 10-mile race, in minutes - `year` = year of the race - `previous` = how many times the runner participated in the years from 1999 to 2008, *previous* to this race - `nruns` = number of times the runner participated in the years from 1999 to 2008 ## Part b Run some code below that helps you familiarize yourself with the basics of this data. Mainly, after this exercise you should have a sense of: - how many data points you have - what the first several rows of the dataset look like - a quick `summary()` of each variable in the dataset. (Each quantitative variable should be summarized with some simple summary statistics (eg: mean), and each categorical variable should be summarized by its categories and the number of observations per category.) ```{r} ``` ## Part c Reflect on what you've noticed about the data thus far. For example: Are there any interesting patterns? Anything that looks like it will need to be cleaned before addressing our research questions? Any obvious outliers? (The only wrong answers here are answers that lack reflection.) **YOUR REFLECTION:** \ \ \ \ # Exercise 2: Prep the variables we'll need Recall that our general research question is about the relationship between running time and age. In this exercise, prep any variables you'll need for this analysis and show the `head()` of your new dataset to illustrate your results. NOTE: After completing this exercise, you should have the same number of rows that you started with, but you might have a different number of columns. ```{r} ``` \ \ \ \ # Exercise 3: Check out the most important variables, alone...& do more cleaning Before examining the *relationship* between running time and age, we should check these 2 variables out on their own in order to understand their scale, range, typical outcomes, etc. ## Part a Construct a plot of age. You don't need to write anything, but take note of the range, shape, and typical outcome of the age distribution among runners. ```{r} ``` ## Part b Similarly, construct and think about the plot of running time. ```{r} ``` ## Part c Follow-up observation: The plot in part b revealed a data point that needs to be removed from our analysis. (If this isn't clear yet, look back at the numerical summaries in exercise 1 and/or move on to the next exercise and then come back.) Do 3 things here: - Show all data points for the runner associated with this odd data point. - Explain what you think the odd data point is *supposed* to be. - Explain why removing this particular data point isn't a deceptive practice (i.e. we're not being dishonest). ```{r} # Show all data points on the runner of interest # (There should be 5 data points!) ``` **YOUR EXPLANATION:** ## Part d Permanently remove the data point in question from your dataset. You shouldn't use it in any of the remaining exercises! Be careful in this process to not remove more than 1 data point. Confirm that your new dataset has 41127 data points, only 1 fewer than your original. ```{r} # Remove the data point # HINT: "a | b" indicates condition "a" OR "b" (or look to Google for other approaches) # Confirm that your dataset now has 41127 data points ``` \ \ \ \ # Exercise 4: Simple calculations & aggregate explorations In studying the relationship between running time and age, let's start by obtaining and plotting some simple calculations. ## Part a In a single plot, plot the fastest running time by age, the average running time by age, and the slowest running time by age. ```{r} ``` ## Part b Provide a list of 3 things that your plot from Part a indicates about the relationship between running time and age. **YOUR ANSWER:** - put thing 1 here - put thing 2 here - put thing 3 here ## Part c Follow-up observation: I noticed that there's more volatility in the fastest, average, and slowest running times across *older* runners. This is probably a feature of our *data*, not because of actual volatility across older runners. Explain *why* this is happening. AND support your answer with numerical evidence from the data. ```{r} # Numerical evidence ``` **YOUR EXPLANATION:** ## Part d This isn't the end of the EDA! Reflecting on what you did and didn't learn from the plot above, identify some follow-up questions that you have about the relationship between running time and age. (The only wrong answers here are answers that lack reflection.) **YOUR REFLECTION:** \ \ \ \ # Exercise 5: Follow-up plots at a different level of information In Exercise 4, we used *aggregate* information (i.e. summaries by groups, not individual observations) to explore the relationship between running time and age. This group-level information provides a nice, *first* glimpse into the relationship. BUT it oversimplifies the information in our dataset, and so we don't want to *stop* with Exercise 4. Let's now consider the relationship between running time and age using the *individual* observations. ## Part a Construct a plot of running time vs age that reflects the outcome for each data point in our dataset. ```{r} ``` ## Part b Follow-up observation: The plot in part a puts all runners into the same bucket, thus the relationship between running time and age doesn't seem very strong. Perhaps this is because we ignored information about prior running experience. Do the following: - Create a new version of the plot from part a to include information related to a person's prior running experience. - Provide a one sentence caveat. Specifically, explain why the additional variable you used is a *proxy* for prior experience but not a perfect measure. (THINK: Can you imagine a runner for whom your proxy measure is high even though prior experience is actually low? Or a runner for whom your proxy measure is low even though prior experience is actually high?) ```{r} ``` **YOUR CAVEAT:** ## Part c In one sentence, explain what new information you've gained about the relationship between running time and age through the plots in this exercise. **YOUR ANSWER:** ## Part d This isn't the end of the EDA! Each new step in the process is helping us *refine* our research questions and introducing even *more* research questions. Pick the option below which best reflects the refined research questions we *do* have insight into at this point in our analysis: a. How do older runners compare to younger runners? b. How does a person's running time change with age? c. Both a and b. \ \ \ \ \ # Exercise 6: Yet more follow-up plots at a yet a different level of information At this point, we've "sliced" the data in two different ways or at different levels to examine the relationship between running time and age: 1. at the aggregate level at each age 2. at the individual race outcome level at each age Let's slice the data yet another way by exploring running time and age dynamics among individual *runners*. ## Part a Some people appear in our dataset more than once. To that end, address some related properties of our data. ```{r} # How many unique runners are there in the dataset? # Address this with code and provide just one number (e.g. do not print out a long table) # What is the fewest number of times that somebody appears in the dataset? # Again, address this with code and provide just one number # What is the most number of times that somebody appears in the dataset? # Again, address this with code and provide just one number ``` ## Part b Let's focus *only* on the runners that participated in the Cherry Blossom Race *at least* 7 times (i.e. 7 or more times) in the years from 1999 to 2008. Construct a plot that represents the relationship between running time and age for each of these individual runners. - Show a smooth trend *line* (not curve) for each runner. - Put these trend lines all on the same frame. - Show *only* the trend lines for each runner, otherwise it would be very messy. - Use `group` not `color`, `fill`, or facets to represent the individual runners. Otherwise, we'd get messy legends. Since we don't necessarily care about these particular runners, just runners in general, we don't need a legend. ```{r} ``` ## Part c Our *intention* in restricting our data to runners with at least 7 `nruns` was to focus on people for whom we had at least 7 observed data points. But our plot makes clear that this isn't always the case -- some lines span only 2 or 3 years. Explain *how* this happened. That is, how can somebody have fewer data points than suggested by their `nruns`? **YOUR ANSWER:** \ \ \ \ # Exercise 7: A "final" follow-up plot Let's improve upon our analysis from Exercise 6. ## Part a Obtain data on *only* the runners for whom we *observed* at least 7 race outcomes, and *only* on their races for which we have complete information. Again, since we're missing some data, this is different than the runners that *completed* at least 7 races. ```{r} # Create new dataset # Confirm that we have 511 data points ``` ## Part b Ignoring individual *runners* and focusing on the individual *race outcomes*, construct a scatterplot of running time vs age using your new dataset. ```{r} ``` ## Part c Now focusing on individual *runners*, recreate the plot from Exercise 6 Part b using your new dataset. ```{r} ``` ## Part d Summarize your combined observations from Parts b and c. Your summary should address: - What you learned about the relationship between running time and age in Part b. - What you learned about the relationship between running time and age in Part c. - What explains any conflicting messages between these 2 plots. **YOUR ANSWER** \ \ \ \ # Reflection The exercises above represent *one* possible EDA of the relationship between running time and age. As we did here, any EDA has the following steps: - import the data - do some quick data checks - **iterate** between the following: - clean the data - construct some summaries and plots - ask yourself what questions these summaries and plots *do* answer, what questions *don't* they answer, and what follow-up questions they provoke - repeat as necessary, letting your reflections on the previous questions inspire your next steps Further, this is the beginning, not the *end* of our analysis! \ \ \ \ # Resource reflection List the resources you used to complete this assignment (e.g. office hours, friends, course notes, Gen AI (including prompts), internet searches, R help pages (by entering `?function` in the console), etc.). - - - Write a few sentences about which resources were the most useful in helping you complete the assignment. \ \ \ \ # Finalize your homework - Render your qmd one more time and check out the rendered html. - Confirm that the html appears as you expect it and that it's correctly formatted. - Confirm that you haven't accidentally printed out long datasets. - Review your answers and make sure you addressed each question. For example, several questions ask for *both* some code / plot and a *discussion* or *summary* in words. - Submit your **HTML** file to the Homework 7 assignment on Moodle. - You're done with Homework 7. Congrats!!