--- title: "Homework 6: Wrangling + More TidyTuesday" author: "PUT YOUR NAME HERE" date: now date-format: "YYYY-MM-DDTHH:mm:ssZ" format: html: toc: true toc-depth: 2 embed-resources: true code-tools: true df-print: paged --- ```{r setup, echo = FALSE, warning = FALSE} # Don't modify this! knitr::opts_chunk$set(collapse = TRUE, warning = FALSE, message = FALSE, fig.height = 2.75, fig.width = 4.25, fig.env='figure', fig.pos = 'h', fig.align = 'center') ``` \ **DIRECTIONS** - Save this file as `homework_6.qmd` in your `DS112 > homework` folder. - Type your name in line 3 above (where it says "author"). - Type your responses in this template. - Do not modify the structure of this document (e.g. don't change section headers, spacing, etc). - There are lots of ways to do things in R. In these exercises, be sure to use the `tidyverse` code and style / structure we've learned in this class. > **Important:** If you find and use code that we haven't covered in class (through internet searches, GenAI, human resources), **you need to cite your sources and explain what the new code does**. You can add this text in a paragraph below your main response for an exercise part. \ \ # Kiva ## Exercise 1: Kiva partners [Kiva](https://www.kiva.org/) is a non-profit that allows people from around the world to lend small amounts to others to start or grow a business, go to school, access clean energy, etc. Since its founding in 2005, more than \$1.2 billion in loans to over 3 million borrowers have been funded. In the remaining exercises, we'll examine some Kiva data from 2005-2012. To begin, let's explore data on Kiva's *field partners*. These partners act as intermediaries between Kiva (the lenders) and borrowers. They evaluate borrower risk, post loan requests on Kiva, and process payments. Load data on the field partners below. A codebook with variable descriptions is [here](https://mac-stat.github.io/data/kiva_partners2_codebook.xlsx) ```{r warning = FALSE, message = FALSE} # Load the tidyverse library(tidyverse) # Load the data partners <- read_csv("https://mac-stat.github.io/data/kiva_partners2.csv") ``` ### Part a Let's get to know the data. ```{r} # Calculate the lowest, median, and highest total amount raised by any partner # Identify the 6 partners that have raised the highest total amount # Show just the partner names, countries, and total amount raised # Show the names of the partners in Bolivia # (Don't include any other variables, just the partner names) ``` ### Part b Create a new table with only five columns: - `countries.region` - `total_partners` = total number of partners per region - `total_loans` = total number of loans posted per region - `total_amount` = total amount raised per region - `average_loan` = average loan size per loan posted in each region (calculated as total amount raised per region / total number of loans posted per region) Print the entire table, sorted from high to low with respect to `total_amount` raised. NOTE: - Your table should have 7 rows and 5 columns. - Your first row should have `countries.region = Asia`, `total_partners = 40`, `total_loans = 133060`, `total_amount = 84816225`, `average_loan = 637`. ```{r} ``` ### Part c Identify two things that you learned from the table in Part b. Just pick whatever you found most interesting. ### Part d Draw a map that includes a dot for each of Kiva's `partners`. Color the dot corresponding to the total amount raised by the partner. NOTE: It's easier to do this on a static map, than with `leaflet`. Refer to Activity 6 (Spatial Viz) - Exercise 5 for a code example. ```{r warning = FALSE, message = FALSE, fig.width = 8} library(rnaturalearth) library(ggthemes) # Get a background map of the entire world # world_boundaries <- ___ # Plot the partner locations on the background map # ggplot(___) + # geom___() + # geom_point( # data = ___, # aes(x = ___, y = ___, color = ___) # ) + # theme_map() ``` \ \ \ \ \ ## Exercise 2: Kiva loans (Part 1) The `loans` data contains information on a sample of 10,000 individual loans to borrowers: ```{r} # a random sample of 10,000 loans loans <- read_csv("https://mac-stat.github.io/data/kiva_loans_small.csv") ``` View the `loans` table, browse through some of the data, and check out the [codebook](https://mac-stat.github.io/data/kiva_loans_codebook.xlsx). Before working with this data, we have to do some pre-processing / wrangling. Take the following steps and store the results as `loans_2`. 1. Only keep the loans that have a positive `funded_amount` (greater than zero dollars). 2. The information about *when* a loan request was posted is separated out into different fields for year, month, day, hour, etc. Combine some of this information into a single variable that records the exact posting time. Do this in three steps. NOTE: You'll get some warning messages, but these are not errors. - Define a new variable which records the exact date a loan request was posted by *pasting* together the year, month, and date of the request, separated by hyphens: `post_dt = paste(posted_yr, posted_mo, posted_day, sep = '-')` - Define a new variable which records the exact time of day a loan request was posted by *pasting* together the hour and minute of the request, separated by a colon:\ `post_time = paste(posted_hr, posted_min, posted_sec, sep = ':')`. - Define a new variable which combines the exact date and time of day a loan request was posted:\ `post_date = ymd_hms(paste(post_dt, post_time, sep = ' '))` 3. Similar to `post_date`, define a new variable called `fund_date` that reports the exact date and time at which each loan was *funded* (not *posted*). 4. Define a new variable called `days_to_fund = difftime(fund_date, post_date, units = "days")` which records the number of days between the time a loan was posted and the time it was funded. 5. Get information about the `countries.region` for the partner of each loan from the `partners` dataset. - NOTE: `partners` and `loans` have many variable names in common. To see these, enter `intersect(colnames(partners), colnames(loans))` in the Console. - If we join by **all** of those variables, we won't get the intended result. (Convince yourself of this by looking at what each of these common variables means in the `partners` and `loans` datasets.) - To have the join work as intended, simplify the `partners` dataset to just 2 columns before joining it to `loans`. The 2 columns should be (1) the key that truly links the rows of `partners` and `loans` and (2) the new information that we are bringing in from `partners` to `loans`. 6. Keep only the following columns: `loan_id, status, funded_amount, paid_amount, sector, countries.region, location.country, lat, lon, partner_id, post_date, fund_date, days_to_fund` ```{r} # Define loans_2 # Confirm that loans_2 has 9884 rows and 13 columns ``` \ \ \ \ ## Exercise 3: Kiva loans (Part 2) ### Part a ```{r} # Show the top 5 countries by number of loans # Show the top 5 countries by total funded loan amount ``` ### Part b Plot the mean loan size in each sector (y-axis) vs the number of loans in each sector (x-axis). Represent each sector by its name (text), not a point. HINT: You'll have to calculate the mean loan size and number of loans in each sector before you can plot them. \ \ \ \ ## OPTIONAL exercise: funding time NOTE: This exercise won't be graded, but it is strongly recommended as additional practice for the quiz. How many days does it take borrowers to get a loan funded? Let's explore the `days_to_fund` variable in the `loans_2`. ### Part a Construct a univariate visualization of `days_to_fund`. ```{r} ``` ### Part b Construct a plot of `days_to_fund` (y-axis) vs `funded_amount` (x-axis) for each loan in `loans_2`. ```{r} ``` ### Part c Construct a plot of `days_to_fund` (y-axis) vs `funded_amount` (x-axis) in each `countries.region`. Don't represent each loan on this plot. Rather, include curves to represent the trends in each region. ```{r} ``` ### Part d Summarize, in words, some takeaway messages from parts a-c about how long it takes to get a loan funded. \ \ \ \ ## OPTIONAL exercise: loan status NOTE: This exercise won't be graded, but it is strongly recommended as additional practice for the quiz. The `status` variable in `loans_2` indicates the status of a loan: ```{r} # Check out a table summary after you've defined loans_2 # loans_2 %>% # count(status) ``` Let's focus here on the patterns in loans that were paid back and those that were defaulted, i.e. not paid back. ### Part a Define a new dataset, `defaults`, that only includes loans that were either defaulted or paid. ```{r} # Define defaults # Confirm that defaults has 7010 rows and 13 columns ``` ### Part b Using `defaults`, construct a visualization of the relationship between the funded amount and status of a loan. ```{r} ``` ### Part c Define a new dataset with four columns: - `partner_id` - number of defaulted loans through that partner - number of loans completely paid back through that partner - percentage of loans defaulted Sort your table from highest default percentage to lowest, and print out only those with at least a 50% default percentage. HINT: You'll have to reshape the data in this process. ```{r} ``` ### Part d Provide some take-away messages about loan defaults using your results in parts a-d. # TidyTuesday As with Homework 3, you will pick a TidyTuesday dataset and do a quick analysis. There are several goals: 1. Practice generating questions. You have to decide what to ask and how to answer it with a graphic. 2. Practice identifying what viz and wrangling tools are useful for addressing your questions. 3. Hone your visualization and wrangling skills. I encourage you to be creative while also maintaining the integrity of the graph. 4. Get a sense of the broader data science community. Check out what people share out on X / Twitter using the #TidyTuesday hashtag. Maybe even share your own #TidyTuesday work on social media. Recent Mac alum Erin Franke (@efranke7282) has an inspiring account! Scrolling through, you'll notice the trajectory of her work, starting from COMP/STAT 112 to today. Very cool. \ \ **Tips** - You should plan to spend 2-3 hours on the TidyTuesday part, *including* in-class work time. - Don't spend more than 30 minutes (at a very maximum) picking a dataset. Try to pick from just the past few weeks of TidyTuesday options, even if it's less interesting to you than another dataset might be. You won't be tied to this data all semester. - Try to complete what you start and turn it in, rather than getting half way and deciding you want to use a different dataset. - Think about a specific skill you'd like to work on, and let this influence what questions you ask of the data. - This is an opportunity for practice and exploration, not perfection! **IF** you have some extra time and really want to dig in and do something really creative, go ahead. But again, the expectation is that you don't spend more than 2-3 hours on this assignment. - Work together! I encourage you to work with others on the same dataset, but **your graphics should be distinct. Though all code and writing must be your own, it helps to bounce around ideas.** ## Part a Pick a dataset from [TidyTuesday](https://github.com/rfordatascience/tidytuesday). Tabs for data from multiple years are provided in the "DataSets" section (down towards the bottom of the page). (Pick a different dataset than the one you used in HW3.) Write a short (\~2 sentence) description of your data at the top of the document, before doing any analysis. This should include: - the original data source (where did TidyTuesday get the data from?), - units of observation (what are you analyzing?), and - the data size (how many data points do you have? how many variables are measured on each data point?). > Put your data description here ## Part b - Construct 3 separate graphs that tell one *connected* story about this data. - Each graph and its discussion will go in its own section (`### Viz 1`, etc.) - Tips: - Start with some questions in mind of what you want to learn. - Start with a simple viz (viz 1), and build this up into something multivariate (viz 3). - Reflect on each viz -- what new questions do you have after checking out the viz? Let these questions guide your viz process. - For an example, recall how we worked through the MacNaturalGas data at the start of the Spatial Viz activity. - Remember that code is communication. Use appropriate commenting, spacing, formatting, etc in the same "tidyverse" style we've been using in class. - Above each viz, write: - A simple but specific research question you're trying to address with the viz. - A brief (2-4 sentence) summary of what you learn from the viz. This should connect back to your research question! - Make sure each viz: - has meaningful axis labels and legend titles - has a figure caption (fig.cap) - uses alt text (fig.alt) - uses a more color-blind friendly color palette (if using color in your viz) - **Expectation**: You should be wrangling data throughout your visual analysis. For example, perhaps you'll need to wrangle the data before plotting it. Perhaps wrangling / summaries will help address some follow-up questions you have from your viz. At least some of this wrangling should demonstrate the combination of multiple verbs, i.e. span multiple lines. ### Visualization 1 > Research Question: ```{r} ``` > Discussion: ### Visualization 2 > Research Question: ```{r} ``` > Discussion: ### Visualization 3 > Research Question: ```{r} ``` > Discussion: \ \ \ \ # Resource reflection List the resources you used to complete this assignment (e.g. office hours, friends, course notes, Gen AI (including prompts), internet searches, R help pages (by entering `?function` in the console), etc.). - - - Write a few sentences about which resources were the most useful in helping you complete the assignment. \ \ \ \ # Finalize your homework - Render your qmd one more time and check out the rendered html. - Confirm that the html appears as you expect it and that it's correctly formatted. - Confirm that you haven't accidentally printed out long datasets. - Review your answers and make sure you addressed each question. For example, several questions ask for *both* some code / plot and a *discussion* or *summary* in words. - Submit your **HTML** file to the Homework 6 assignment on Moodle. - You're done with Homework 6. Congrats!!