```{r setup, echo=FALSE, message=FALSE}
library(tidyverse)
library(httr2)
library(xml2)
```

# Learning goals

After this lesson, you should be able to:

- Explain what an API is
- Set up an API key for a public API
- Develop comfort in using a wrapper package or URL-method of calling a web API
- Recognize the structure in a URL for a web API and adjust for your purposes
- Explore and subset complex nested lists


<br><br><br><br>


# APIs

When we interact with sites like The New York Times, Zillow, and Google, we are accessing their data via a graphical layout (e.g., images, colors, columns) that is easy for humans to read but hard for computers.

An **API** stands for **Application Programming Interface**, and this term describes a general class of tool that allows computers, rather than humans, to interact with an organization's data. How does this work?

- When we use web browsers to navigate the web, our browsers communicate with web servers using a technology called [HTTP](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) or Hypertext Transfer Protocol to get information that is formatted into the display of a web page.
- Programming languages such as R can also use HTTP to communicate with web servers. The easiest way to do this is via [Web APIs](https://en.wikipedia.org/wiki/Web_API), or Web Application Programming Interfaces, which focus on transmitting raw data, rather than images, colors, or other appearance-related information that humans interact with when viewing a web page.

A large variety of web APIs provide data accessible to programs written in R (and almost any other programming language!). Almost all reasonably large commercial websites offer APIs. Todd Motto has compiled an expansive list of [Public Web APIs](https://github.com/toddmotto/public-apis) on GitHub. Let's browse this list to see what data sources are available.


# Wrapper packages

In R, it is easiest to use Web APIs through a **wrapper package**, an R package written specifically for a particular Web API.

- The R development community has already contributed wrapper packages for most large Web APIs.
- To find a wrapper package, search the web for "R package" and the name of the website. For example:
    - Searching for "R Reddit package" returns [RedditExtractor](https://cran.r-project.org/web/packages/RedditExtractoR/index.html)
    - Searching for "R Weather.com package" returns [weatherData](https://ram-n.github.io/weatherData/)
- [rOpenSci](https://ropensci.org/packages/) also has a good collection of wrapper packages.

In our work with maps, we've used the `tidycensus` package to obtain census data to display on maps. `tidycensus` is a wrapper package that makes it easy to obtain desired census information:

```{r tidycensus, eval=FALSE}
tidycensus::get_acs(
    year = 2020,
    state = "MN",
    geography = "tract",
    variables = c("B01003_001", "B19013_001"),
    output = "wide",
    geometry = TRUE
)
```

Extra resources:

- `tidycensus`: wrapper package that provides an interface to a few census datasets with map geometry included
    - Full documentation is available at <https://walker-data.com/tidycensus/>
- `censusapi`: wrapper package that offers an interface to all census datasets
    - Full documentation is available at <https://www.hrecht.com/censusapi/>

What is going on behind the scenes with `get_acs()`?


<br><br><br><br>


# Accessing web APIs directly

## Getting a Census API key

Many APIs require users to obtain a **key** to use their services.

- This lets organizations keep track of what data is being used.
- It also **rate limits** their API and ensures programs don't make too many requests per day/minute/hour. Be aware that most APIs do have rate limits --- especially for their free tiers.

Navigate to <https://api.census.gov/data/key_signup.html> to obtain a Census API key:

- Organization: Macalester College
- Email: Your Mac email address

You will get the message:

> Your request for a new API key has been successfully submitted. Please check your email. In a few minutes you should receive a message with instructions on how to activate your new key.

Check your email. Copy and paste your key into a new text file:

- File > New File > Text File (towards the bottom of the menu)
- Save as `census_api_key.txt` in the same folder as this `.qmd`.

Read in the key with the following code:

```{r read_census_api_key}
census_api_key <- readLines("census_api_key.txt")
```

## Navigating API documentation

Navigate to the [Census API user guide](https://www.census.gov/data/developers/guidance/api-user-guide.html) and click on the "Example API Queries" tab.

Let's look at the Population Estimates Example and the American Community Survey (ACS) Example. These examples walk us through the steps to incrementally build up a URL to obtain desired data. This URL is known as a web API **request**. 

https://api.census.gov/data/2019/acs/acs1?get=NAME,B02015_009E,B02015_009M&for=state:*

- `https://api.census.gov`: This is the **base URL**.
    - `http://`: The **scheme**, which tells your browser or program how to communicate with the web server. This will typically be either `http:` or `https:`.
    - `api.census.gov`: The **hostname**, which is a name that identifies the web server that will process the request.
- `data/2019/acs/acs1`: The **path**, which tells the web server how to get to the desired resource.
    - In the case of the Census API, this locates a desired dataset in a particular year.
    - Other APIs allow search functionality. (e.g., News organizations have article searches.) In these cases, the path locates the search function we would like to call.
- `?get=NAME,B02015_009E,B02015_009M&for=state:*`: The **query parameters**, which provide the parameters for the function you would like to call.
    - We can view this as a string of key-value pairs separated by `&`. That is, the general structure of this part is `key1=value1&key2=value2`.

key      value
----     ------
get      NAME,B02015_009E,B02015_009M
for      state:*

Typically, each of these URL components will be specified in the API documentation. Sometimes, the scheme, hostname, and path (`https://api.census.gov/data/2019/acs/acs1`) will be referred to as the **[endpoint](https://en.wikipedia.org/wiki/Web_API#Endpoints)** for the API call.

We will use the `httr2` package to build up a full URL from its parts.

- `request()` creates an API request object using the **base URL**
- `req_url_path_append()` builds up the URL by adding path components separated by `/`
- `req_url_query()` adds the `?` separating the endpoint from the query and sets the key-value pairs in the query
    - The `.multi` argument controls how multiple values for a given key are combined.
    - The `I()` function around `"state:*"` inhibits parsing of special characters like `:` and `*`. (It's known as the "as-is" function.)
    - The backticks around `for` are needed because `for` is a reserved word in R (for for-loops). You'll need backticks whenever the key name has special characters (like spaces, dashes).
    - We can see from [here](https://www.census.gov/data/developers/guidance/api-user-guide.Help_&_Contact_Us.html) that providing an API key is achieved with `key=YOUR_API_KEY`.

```{r httr2_census}
req <- request("https://api.census.gov") %>% 
    req_url_path_append("data") %>% 
    req_url_path_append("2019") %>% 
    req_url_path_append("acs") %>% 
    req_url_path_append("acs1") %>% 
    req_url_query(get = c("NAME", "B02015_009E", "B02015_009M"), `for` = I("state:*"), key = census_api_key, .multi = "comma")
```

**Why would we ever use these steps instead of just using the full URL as a string?**

- To generalize this code with functions! (This is exactly what wrapper packages do.)
- To handle special characters
    - e.g., query parameters might have spaces, which need to be represented in a particular way in a URL (URLs can't contain spaces)

Once we've fully constructed our request, we can use `req_perform()` to send out the API request and get a **response**.

```{r census_response}
resp <- req_perform(req)
resp
```

We see from `Content-Type` that the format of the response is something called JSON. We can navigate to the request URL to see the structure of this output.

- JSON (Javascript Object Notation) is a nested structure of key-value pairs.
- We can use `resp_body_json()` to parse the JSON into a nicer format.
    - Without `simplifyVector = TRUE`, the JSON is read in as a list. 

```{r census_json}
resp_json_list <- resp %>% resp_body_json()
head(resp_json_list, 2)
resp_json_df <- resp %>% resp_body_json(simplifyVector = TRUE)
head(resp_json_df)
resp_json_df <- janitor::row_to_names(resp_json_df, 1)
head(resp_json_df)
```

**Exercises: (Homework 7)**

1. Write a function called `get_acs1()` that gets 1-year ACS data for a given year for a set of user specified variables for all states. Function arguments should include:
    - `year`: Within your function check that this is between 2005 and 2022 and not 2020 (data only available from 2005-2022 except for 2020). If the year is outside this range, include `stop("Helpful error message here")` to display a helpful error message to the user.
    - `variables`: A character vector of variable names to obtain. e.g., `c("B02015_009", "B01003_001")`. Note that these don't have "E" or "M" at the end. You'll always return the "E" version (for the estimate). See next argument for the "M" version.
    - `include_moe`: TRUE/FALSE indicator of whether to additionally include the "M" version (for the margin of error on the estimate). Set the default to TRUE.
    - `key`: Census API key
    Your function should include a `Sys.sleep(0.5)` at the beginning to pause code execution for 0.5 seconds. While the Census API does not seem to have a [rate limit](https://www.census.gov/data/developers/guidance/api-user-guide.Help_&_Contact_Us.html) when you provide an API key, include this pause just to be safe.

2. Come up with a set of test cases that will **fully** test your function. Update your function as needed as you run your tests. Show output from successful tests on all test cases.

3. Obtain 1-year ACS data for all states from 2005-2022 (except 2020) for the following two variables: `B01003_001` (total population) and `B06011_001` (Median Income in the Past 12 Months in 2022 Inflation-Adjusted Dollars).
    - Get both the estimate and the margin of error for these two variables.
    - Put all of this information into a single data frame.
    - Do this with both a `for` loop and with an appropriate `purrr` function. (See our [iteration day 2 activity](https://lmyint.github.io/212_spring_2024/12-iteration-2.html) for reference.)
    - Verify using the `identical()` function that you get identical data frames from both approaches.

4. Create visualizations that show how the two variables have changed over 2005-2022 (except 2020).
    - Show both the estimate and margin of error in an effective way.
    - Make sure that your plots have clean labels and a title.


<br><br><br><br>


# Board Game Geek

The [Board Game Geek API](https://boardgamegeek.com/wiki/page/BGG_XML_API2) is referenced in the [Games & Comics](https://github.com/toddmotto/public-apis?tab=readme-ov-file#games--comics) section of [toddmotto's public API list](https://github.com/toddmotto/public-apis).

Our goal is to use the search API at the bottom of the page. Let's start at the top of the API documentation page to see how to navigate this reference.

- We can see from the XML references at the top that we will be expecting a new output format: XML stands for Extensible Markup Language
- The "Root Path" section tells us the base URL for the Board Game Geeks API endpoints and related APIs: <https://boardgamegeek.com/xmlapi2/>
- The "Search" section at the bottom of the page tells us:
    - the path for the search endpoint (`/search`)
    - what query parameters are possible
    - particular formatting instructions for query parameter values

The following request searches for board games, board game accessories, and board game expansions with the words "mystery" and "curse" in the title:

```{r httr2_bgg}
req <- request("https://boardgamegeek.com/xmlapi2") %>% 
    req_url_path_append("search") %>% 
    req_url_query(query = I("mystery+curse"), type = I("boardgame,boardgameaccessory,boardgameexpansion"))
```

When we use `req_perform()`, we see from `Content-Type` that the format of the response is something called XML. We can navigate to the request URL to see the structure of this output.

- XML (Extensible Markup Language) is a tree structure of named nodes and attributes.
- We can use `resp_body_xml()` to read in the XML as an R object.

```{r bgg_response}
resp <- req_perform(req)
resp
resp <- resp_body_xml(resp)
resp
```

The XML output is not packaged in a nice way. (We'd love to have a data frame.) We can use the `xml2` package to explore and navigate the XML structure to extract the information we need.

Let's first use the `xml_structure()` function to see how information is organized:

```{r xml_structure}
xml_structure(resp)
```

The key navigation and extraction functions in `xml2` are:

- `xml_children()`: Get nodes that are nested inside 
    - Like getting the first level bullet points inside a given bullet point
- `xml_find_all()`: Finds nodes matching an XPath expression (XPath stands for XML path)
    - XPath expressions are like string regular expressions for XML trees
    - See [here](https://www.w3schools.com/xml/xpath_intro.asp) for a deeper dive into XPath
- `xml_attr()`: Selects the value of an attribute (the information to the right of the `=` in quotes)
    - `<node_name attribute_name1="attribute_value1" attribute_name2="attribute_value2">`

```{r xml_nav_extract}
# Get the item nodes in 2 different ways
resp %>% xml_find_all("item")
resp %>% xml_children()

# Get the item "type"
resp %>% xml_find_all("item") %>% xml_attr("type")

# The <name> and <yearpublished> nodes are nested within each <item>
resp %>% xml_find_all("item/name")
resp %>% xml_find_all("item/yearpublished") # Notice that this is length 8 instead of 9!

# Get the "primary" or "alternate" designation for each name
resp %>% xml_find_all("item/name") %>% xml_attr("type")
```

**Exercises: (not part of Homework 7)**

1. Get the board game name (e.g., "Murder Mystery..." and  "Robinson Crusoe")
2. Get the board game ID number (e.g., 63495, 40175)

