---
title: "Chapter 02 Template"
author: "<insert-name>"
format: html
---

## Introduction

Reopen the project you created in Chapter 1 and save this `.qmd` file in the top-level folder. When working through the materials from this chapter run each code chunk separately by clicking the small green arrow beside each code chunk. By default `eval: false` on most code chunks, so when you knit the code chunk will not execute. We are working with a reasonably large dataset so this is to avoid the tedium of waiting long periods for this document to render.

The technical element to this chapter works with data from a large bikeshare scheme -- New York's [Citibike](https://www.citibikenyc.com/) scheme.

## Setup

The code block below points to the packages you need to do the work discussed in the chapter. If you have been following the instructions from the last chapter you will have already installed the key packages -- `tidyverse` , `fst`, `lubridate`, `sf`, `here` -- and so simply need to make them available in session with `library(<package-name>)`. If you have not yet downloaded these packages you will need to make separate calls to `install.packages("<package-name>")` as described in the previous chapter.

```{r}
#| echo: true
#| eval: false
# Bundle of packages containing functions for most data analysis.
library(tidyverse) 
# For fast/efficient working with tibbles.
library(fst)
# For working with dates.
library(lubridate) 
# For navigating a project's directory.
library(here) 
# For performing trigonometry operations on spatial data.
install.packages("geosphere") 
```

Run the code block below to download the pre-processed and extracted data to your project's data folder. Note that you should only run the import code below once. When returning to this work in a new R session, you need only then read in the `ny_trips.fst` and `ny_stations.csv` file that is held locally on your machines. Unfortunately due to storage limits I had to make `ny_trips.fst` just a 500k random sample of the c.1.9m trips made in Citibike in June 2020.

```{r}
#| echo: true
#| eval: false
# Create subdirectory 'data' if none exists.
if(!dir.exists(here("data"))) dir.create(here("data"))

# Download .csv file of stations data from book's data repo.
# Save to the new data folder.
url <- "https://vis4sds.github.io/data/ch2/ny_stations.csv"
download.file(url, here("data", "ny_stations.csv"))
ny_stations <- read_csv(here("data", "ny_stations.csv"))

# Read in .fst file of trips data from book's data repo.
# Save to the new data folder.
url <- "https://vis4sds.github.io/data/ch2/ny_trips.fst"
download.file(url, here("data", "ny_trips.fst"))
ny_trips <- read_fst(here("data", "ny_trips.fst"))

# The two untidy datasets.
url <- "https://vis4sds.github.io/data/ch2/ny_spread_rows.csv"
download.file(url, here("data", "ny_spread_rows.csv"))
url <- "https://vis4sds.github.io/data/ch2/ny_spread_columns.csv"
download.file(url, here("data", "ny_spread_columns.csv"))
```

## Describe

The code block below performs some useful recoding operations: convert `chr` fields to `int` for efficient storage, temporal fields to `dttm` objects, and longitude and latitude to `dbl` so that we can do calculations.

```{r}
#| echo: true
#| eval: false
ny_trips <- ny_trips |> select(-city) |>
  mutate(
    across(
      .cols=c(start_station_id, end_station_id),
      .fns=~as.integer(str_remove(., "ny"))
    ),
    across(
      .cols=c(start_time, stop_time), 
      .fns=~as.POSIXct(., format="%Y-%m-%d %H:%M:%S")
      ),
    bike_id=as.integer(bike_id),
    gender=case_when(
      gender == 0 ~ "unknown",
      gender == 1 ~ "male",
      gender == 2 ~ "female")
  )

ny_stations <- ny_stations |>
  select(-city) |>
  mutate(
    stn_id=as.integer(str_remove(stn_id, "ny")),
    across(.cols=c(longitude, latitude),.fns=~as.double(.))
  )
```

## Transform

An example application of `dplyr` for counting:

```{r}
#| echo: true
#| eval: false
ny_trips |>
  group_by(user_type) |>
  summarise(count=n()) |>
  arrange(desc(count))
```

An example application of `dplyr` for computing over aggregates:

```{r}
#| echo: true
#| eval: false
ny_trips |>
  group_by(user_type) |>
  summarise(
    count=n(),
    avg_duration=mean(trip_duration/60),
    median_duration=median(trip_duration/60),
    sd_duration=sd(trip_duration/60),
    min_duration=min(trip_duration/60),
    max_duration=max(trip_duration/60)
    ) |>
  arrange(desc(count))
```

Example code for generating a temporal summary by user type:

```{r}
#| echo: true
#| eval: false
ny_temporal <- ny_trips |>
  mutate(
    day=wday(start_time, label=TRUE),
    hour=hour(start_time)) |>
  group_by(user_type, day, hour) |>
  summarise(count=n()) |>
  ungroup()

ny_temporal |>
  group_by(user_type) |>
  mutate(user_count=sum(count)) |>
  group_by(user_type, day) |>
  summarise(count=sum(count), prop=count/first(user_count)) |>
  select(user_type, day, prop) |>
  pivot_wider(names_from=user_type, values_from=prop)
```

`dplyr` code with `ggplot2` for exploring temporal travel behaviours by user type visually:

```{r}
#| echo: true
#| eval: false
ny_temporal |>
  ggplot(aes(x=hour, y=count, group=user_type)) +
  geom_line(aes(colour=user_type), linewidth=1.1) +
  scale_colour_manual(values=c("#e31a1c", "#1f78b4")) +
  facet_wrap(~day, nrow=1)+
  labs(x="", y="trip counts")
```

Code for calculating approximate distance travelled by od_pair.

First we generate a table of distinct ODs that have been cycled and from here calculate approximate distance travelled:

```{r}
#| echo: true
#| eval: false
ny_trips <- ny_trips |>
  mutate(
    duration_minutes=
      as.numeric(as.duration(stop_time-start_time),"minutes")
    )

od_pairs <- ny_trips |> select(start_station_id, end_station_id) |> unique() |>
  left_join(ny_stations |> select(stn_id, longitude, latitude), by=c("start_station_id"="stn_id")) |>
  rename(o_lon=longitude, o_lat=latitude) |>
  left_join(ny_stations |> select(stn_id, longitude, latitude), by=c("end_station_id"="stn_id")) |>
  rename(d_lon=longitude, d_lat=latitude) |>
  rowwise() |>
  mutate(
    dist=geosphere::distHaversine(c(o_lat, o_lon), c(d_lat, d_lon))/1000
    ) |>
  ungroup()
```

Then `left_join()` `ny_trips` on this `od_pairs` dataset:

```{r}
#| echo: true
#| eval: false
ny_trips <- ny_trips |>
  mutate(od_pair=paste0(start_station_id,"-",end_station_id)) |>
  left_join(od_pairs |>
              mutate(od_pair=paste0(start_station_id,"-",end_station_id)) |>
              select(od_pair, dist)
            )
```

The function to calculate age from year of birth:

```{r}
#| echo: true
#| eval: false
# Function for calculating time elapsed between two dates in years (age).
get_age <- function(yob, yref) {
    period <- lubridate::as.period(lubridate::interval(yob, yref),unit = "year")
    return(period$year)
}

# Derive a new age variable from birth_year.
ny_trips <- ny_trips |> 
  mutate(
    age=get_age(as.POSIXct(birth_year, format="%Y"), as.POSIXct("2020", format="%Y"))
    )
```

The code for plotting trip speeds (approximate) by age, gender and distance travelled. 

```{r}
#| echo: true
#| eval: false
temp_plot_data <- ny_trips |>
  mutate(day=wday(start_time, label=TRUE), is_weekday=!day %in% c("Sat", "Sun")) |>
  filter(
    is_weekday,
    start_station_id!=end_station_id,
    duration_minutes<=60,
    user_type=="Subscriber",
    between(age, 16, 74),
    gender!="unknown") |>
  mutate(
    dist_bands=case_when(
      dist < 1.5 ~ "<1.5km",
      dist < 3 ~ ">1.5-3km",
      dist < 4.5 ~ ">3-4.5km",
      TRUE ~ ">4.5km"),
    age_band=if_else(age %% 10 > 4, ceiling(age/5)*5, floor(age/5)*5),
    speed=dist/(duration_minutes/60)
  ) |>
  group_by(gender, age_band, dist_bands) |>
  summarise(speed=mean(speed), n=n())

temp_plot_data |>
  ggplot(aes(x=age_band, y=speed))+
  geom_line(aes(colour=gender))+
  facet_wrap(~dist_bands, nrow=1)
```

