2.1.1 R packages

• New: pbapply
• Already used: tidyverse, data.table

We’ll be sticking mostly with base R functions in this chapter. But we’ll also show you a few extra features and considerations for the main data wrangling packages. As per usual, run the following code chunk to install (if necessary) and load everything.

if (!require("pacman")) install.packages("pacman")
pacman::p_load(pbapply, data.table, tidyverse)

2.3.1 Specifying return values

Notice that we didn’t specify a return value for our function. This will work in many cases because R’s default behaviour is to automatically return the final object that you created within the function. However, this won’t always be the case. Opinions on this differ, but our own recommendation is that you get into the habit of assigning the return object(s) explicitly with return(). Let’s modify our function to do exactly that.

square = 
  function(x) { 
    ## Create an intermediary object (that will be returned)
    x_sq = x^2  
    
    ## The value(s) or object(s) that we want returned.
    return(x_sq)  
  }

Again, test that it works.

square(5)
#> [1] 25

Specifying an explicit return value is also helpful when we want to return more than one object. For example, let’s say that we want to remind our user what variable they used as an argument in our function:

square = 
  function(x) { 
    x_sq = x^2 
    
    ## The list of object(s) that we want returned.
    return(list(value = x, value_squared = x_sq))
  }

square(3)
#> $value
#> [1] 3
#> 
#> $value_squared
#> [1] 9

Note that multiple return objects have to be combined in a list. We didn’t have to name these separate list elements — i.e. “value” and “value_squared” — but it will be helpful for users of our function. Nevertheless, remember that many objects in R contain multiple elements (vectors, data frames, and lists are all good examples of this). So we can also specify one of these “array”-type objects within the function itself if that provides a more convenient form of output. For example, we could combine the input and output values into a data frame:

square = 
  function(x) { 
    x_sq = x^2 
    
    ## Bundle up our input and output values into a convenient dataframe.
    d = data.frame(value=x, value_squared=x_sq) 
    
    return(d)
  }

Test.

square(12)
#>   value value_squared
#> 1    12           144

2.3.2 Specifying default argument values

Another thing worth noting about R functions is that you can assign default argument values. You have already encountered some examples of this in action.⁷ We can add a default option to our own function pretty easily.

square = 
  function(x = 1) { ## Setting the default argument value 
    x_sq = x^2 
    d = data.frame(value = x, value_squared = x_sq)
    
    return(d)
  }

square()  ## Will take the default value of 1.
#>   value value_squared
#> 1     1             1
square(2) ## Now takes the explicit value that we give it.
#>   value value_squared
#> 1     2             4

We’ll return to the issues of specifying default values, handling invalid inputs, and general debugging in Section 3.2.

2.3.3 Aside: Environments and lexical scoping

Before continuing, take a second to note that none of the intermediate objects that we created inside the above functions (x_sq, df, etc.) have made their way into your global environment.⁸ R has a set of so-called lexical scoping rules, which govern where it stores and evaluates the values of different objects. Without going into too much depth, the practical implication of lexical scoping is that functions operate in a quasi-sandboxed environment. They don’t return or use objects in the global environment unless they are forced to (e.g. with a return() command). Similarly, a function will only look to outside environments (e.g. a level “up”) to find an object if it doesn’t see the object named within itself.

We’ll revisit the ideas of distinct object environments and lexical scoping when we get to functional programming in Section 2.5.3 below, and then again in Section 3.2.

2.4.1 if and ifelse

We’ve already encountered conditional statements like if() and ifelse() numerous times in the book thus far.⁹ However, let’s see how they can work in our own bespoke functions by slightly modifying our square function. This time, instead of specifying a default input value of 1 in the function argument itself, we’ll specify a value of NULL. Then we’ll use an if() statement to reassign this default to one.

square = 
  function(x = NULL) {  ## Default value of NULL
    if (is.null(x)) x = 1 ## Re-assign default to 1
    
    x_sq = x^2 
    d = data.frame(value = x, value_squared = x_sq)
    
    return(d)
  }
square()
#>   value value_squared
#> 1     1             1

Why go through the rigmarole of specifying a NULL default inpute if we’re going to change it to 1 anyway? Admittedly, this is a pretty silly thing to do in the above example. However, consider what it buys us in the next code chunk:

square = 
  function(x = NULL) {
    
    if (is.null(x)) { ## Start multi-line IF statement with `{`
      x = 1
      ## Message to users:
      message("No input value provided. Using default value of 1.")
      }               ## Close multi-line if statement with `}`
    
    x_sq = x^2 
    d = data.frame(value = x, value_squared = x_sq)
    
    return(d)
  }
square()
#> No input value provided. Using default value of 1.
#>   value value_squared
#> 1     1             1

This time, by specifying NULL in the argument — alongside the expanded if() statement — our function now both takes a default value and generates a helpful message.¹⁰ Note too the use of curly brackets for conditional operations that span multiple lines after an if() statement. This provides a nice segue to ifelse() statements. As we’ve already seen , these be written as a single conditional call where the format is:

ifelse(CONDITION, DO IF TRUE, DO IF FALSE)

Within our own functions, though we’re more likely to write them over several lines. Consider, for example a new function that evaluates whether our square() function is doing its job properly.

eval_square =
  function(x) {
    
    if (square(x)$value_squared == x*x) {
      ## What to do if the condition is TRUE 
      message("Nailed it.")
    } else {
      ## What to do if the condition is FALSE
      message("Dude, your function sucks.")
    }
    
  }
eval_square(64)
#> Nailed it.

today = function(...) ifelse(..., Sys.Date(), Sys.Date()-1)
today(TRUE)
#> [1] 18842

today2 = function(...) dplyr::if_else(..., Sys.Date(), Sys.Date()-1)
today2(TRUE)
#> [1] "2021-08-03"

today3 = function(...) data.table::fifelse(..., Sys.Date(), Sys.Date()-1)
today3(TRUE)
#> [1] "2021-08-03"

2.4.2 case when (nested ifelse)

As you may have guessed, it’s certainly possible to write nested ifelse() statements. For example,

ifelse(CONDITION1, DO IF TRUE, ifelse(CONDITION2, DO IF TRUE, ifelse(...)))

However, these nested statements quickly become difficult to read and troubleshoot. A better solution was originally developed in SQL with the CASE WHEN statement. Both dplyr with case_when() and data.table with fcase() provide implementations R. Here is a simple illustration of both implementations.

x = 1:10

## dplyr::case_when()
case_when(
  x <= 3 ~ "small",
  x <= 7 ~ "medium",
  TRUE ~ "big" ## Default value. Could also write `x > 7 ~ "big"` here.
  )
#>  [1] "small"  "small"  "small"  "medium" "medium" "medium" "medium" "big"   
#>  [9] "big"    "big"

## data.table::fcase()
fcase(
    x <= 3, "small",
    x <= 7, "medium",
    default = "big" ## Default value. Could also write `x > 7, "big"` here.
    )
#>  [1] "small"  "small"  "small"  "medium" "medium" "medium" "medium" "big"   
#>  [9] "big"    "big"

Not to belabour the point, but you can easily use these case when implementations inside of data frames/tables too.

## dplyr::case_when()
data.frame(x = 1:10) %>%
    mutate(grp = case_when(x <= 3 ~ "small",
                           x <= 7 ~ "medium",
                           TRUE ~ "big"))
#>     x    grp
#> 1   1  small
#> 2   2  small
#> 3   3  small
#> 4   4 medium
#> 5   5 medium
#> 6   6 medium
#> 7   7 medium
#> 8   8    big
#> 9   9    big
#> 10 10    big
## data.table::fcase()
data.table(x = 1:10)[, grp := fcase(x <= 3, "small",
                                    x <= 7, "medium",
                                    default = "big")][]
#>      x    grp
#>  1:  1  small
#>  2:  2  small
#>  3:  3  small
#>  4:  4 medium
#>  5:  5 medium
#>  6:  6 medium
#>  7:  7 medium
#>  8:  8    big
#>  9:  9    big
#> 10: 10    big

2.5.1 Vectorisation

The first question you should be asking yourself is: “Do I need to iterate at all?”

You may remember from earlier chapters that we spoke about R being vectorised. This means that we can apply a function over an entire vector all at once, rather than having to iterate explicitly over each element. Vectorisation is a common feature of high-level programming languages like R and Python. What’s really happening with vectorisation is that a loop is being called behind the scenes. But that loop is implemented in a low-level compiled language like C or Fortran.¹² Let’s demonstrate this property with our square function:

square(1:5)
#>   value value_squared
#> 1     1             1
#> 2     2             4
#> 3     3             9
#> 4     4            16
#> 5     5            25
square(c(2, 4))
#>   value value_squared
#> 1     2             4
#> 2     4            16

So, again, you may not need to think about explicit iteration at all.

That being said, there are certainly cases where you will need to think about iteration. For the remainder of this section, we’ll explore with some simple examples — some of which are already vectorised! — that provide a mental springboard for thinking about more complex cases.

2.5.2 for loops

In R, standard for loops take a pretty intuitive form. For example:

for(i in 1:10) print(LETTERS[i])
#> [1] "A"
#> [1] "B"
#> [1] "C"
#> [1] "D"
#> [1] "E"
#> [1] "F"
#> [1] "G"
#> [1] "H"
#> [1] "I"
#> [1] "J"

Note that in cases where we want to “grow” an object via a for loop, we first have to create an empty (or NULL) object.

kelvin = 300:305
fahrenheit = NULL
# fahrenheit = vector("double", length(kelvin)) ## Better than the above. Why?
for(k in 1:length(kelvin)) {
  fahrenheit[k] = kelvin[k] * 9/5 - 459.67
}
fahrenheit
#> [1] 80.33 82.13 83.93 85.73 87.53 89.33

Unfortunately, basic for loops in R also come with some downsides. Historically, they used to be significantly slower and memory consumptive than alternative methods (see below). This has largely been resolved, but we’ve still run into cases where an inconspicuous for loop has brought an entire analysis crashing to its knees.¹³ The bigger problem with for loops, however, is that they deviate from the norms and best practices of functional programming.

2.5.3 Functional programming

The concept of functional programming (FP) is arguably the most important thing that you can take away from this chapter. In his excellent book, Advanced R, Hadley Wickham explains the core idea as follows.

R, at its heart, is a functional programming (FP) language. This means that it provides many tools for the creation and manipulation of functions. In particular, R has what’s known as first class functions. You can do anything with functions that you can do with vectors: you can assign them to variables, store them in lists, pass them as arguments to other functions, create them inside functions, and even return them as the result of a function.

That may seem a little abstract, so here is video of Hadley giving a much more intuitive explanation through a series of examples.

Summary: for loops tend to emphasise the objects that we’re working with (say, a vector of numbers) rather than the operations that we want to apply to them (say, get the mean or median or whatever). This is inefficient because it requires us to continually write out the for loops by hand rather than getting an R function to create the for-loop for us.

As a corollary, for loops also pollute our global environment with the variables that are used as counting variables. Take a look at your “Environment” pane in RStudio. What do you see? In addition to the kelvin and fahrenheit vectors that we created, we also see two variables i and k (equal to the last value of their respective loops). Creating these auxiliary variables is almost certainly not an intended outcome when your write a for-loop.¹⁴ More worryingly, they can cause programming errors when we inadvertently refer to a similarly-named variable elsewhere in our script. So we best remove them manually as soon as we’re finished with a loop.

rm(i, k)

Another annoyance arrived in cases where we want to “grow” an object as we iterate over it (e.g. the fahrenheit object in our second example). In order to do this with a for loop, we had to go through the rigmarole of creating an empty object first.

FP allows to avoid the explicit use of loop constructs and its associated downsides. In practice, there are two ways to implement FP in R:

1. The *apply family of functions in base R.
2. The map*() family of functions from the purrr.

Let’s explore these in more depth.

# for(i in 1:10) print(LETTERS[i]) ## Our original for loop (for comparison)
lapply(1:10, function(i) LETTERS[i])
#> [[1]]
#> [1] "A"
#> 
#> [[2]]
#> [1] "B"
#> 
#> [[3]]
#> [1] "C"
#> 
#> [[4]]
#> [1] "D"
#> 
#> [[5]]
#> [1] "E"
#> 
#> [[6]]
#> [1] "F"
#> 
#> [[7]]
#> [1] "G"
#> 
#> [[8]]
#> [1] "H"
#> 
#> [[9]]
#> [1] "I"
#> 
#> [[10]]
#> [1] "J"

lapply(1:10, function(i) data.frame(num = i, let = LETTERS[i])) %>%
  # do.call("rbind", .)      ## Also works
  # data.table::rbindlist()  ## Ditto
  dplyr::bind_rows()
#>    num let
#> 1    1   A
#> 2    2   B
#> 3    3   C
#> 4    4   D
#> 5    5   E
#> 6    6   F
#> 7    7   G
#> 8    8   H
#> 9    9   I
#> 10  10   J

sapply(1:10, function(i) LETTERS[i]) 
#>  [1] "A" "B" "C" "D" "E" "F" "G" "H" "I" "J"

# library(pbapply) ## Already loaded

pblapply(1:10, function(i) {
  d = data.frame(num = i, let = LETTERS[i])
  Sys.sleep(1)
  return(d)
  }) %>%
  bind_rows()
#>    num let
#> 1    1   A
#> 2    2   B
#> 3    3   C
#> 4    4   D
#> 5    5   E
#> 6    6   F
#> 7    7   G
#> 8    8   H
#> 9    9   I
#> 10  10   J

map(1:10, function(i) { ## only need to swap `lapply` for `map`
  d = data.frame(num = i, let = LETTERS[i])
  return(d)
  })
#> [[1]]
#>   num let
#> 1   1   A
#> 
#> [[2]]
#>   num let
#> 1   2   B
#> 
#> [[3]]
#>   num let
#> 1   3   C
#> 
#> [[4]]
#>   num let
#> 1   4   D
#> 
#> [[5]]
#>   num let
#> 1   5   E
#> 
#> [[6]]
#>   num let
#> 1   6   F
#> 
#> [[7]]
#>   num let
#> 1   7   G
#> 
#> [[8]]
#>   num let
#> 1   8   H
#> 
#> [[9]]
#>   num let
#> 1   9   I
#> 
#> [[10]]
#>   num let
#> 1  10   J

map_df(1:10, function(i) { ## don't need bind_rows with `map_df`
  d = data.frame(num = i, let = LETTERS[i])
  return(d)
  })
#>    num let
#> 1    1   A
#> 2    2   B
#> 3    3   C
#> 4    4   D
#> 5    5   E
#> 6    6   F
#> 7    7   G
#> 8    8   H
#> 9    9   I
#> 10  10   J

2.5.4 Create and iterate over named functions

As you may have guessed already, we can split the function and the iteration (and binding) into separate steps. This is generally a good idea, since you typically create (named) functions with the goal of reusing them.

## Create a named function
num_to_alpha = 
  function(i) {
  d = data.frame(num = i, let = LETTERS[i])
  return(d)
  }

Now, we can easily iterate over our function using different input values. For example,

lapply(1:10, num_to_alpha) %>% bind_rows()
#>    num let
#> 1    1   A
#> 2    2   B
#> 3    3   C
#> 4    4   D
#> 5    5   E
#> 6    6   F
#> 7    7   G
#> 8    8   H
#> 9    9   I
#> 10  10   J

Or, say

map_df(c(1, 5, 26, 3), num_to_alpha)
#>   num let
#> 1   1   A
#> 2   5   E
#> 3  26   Z
#> 4   3   C

2.5.5 Iterate over multiple inputs

Thus far, we have only been working with functions that take a single input when iterating. For example, we feed them a single vector (even though that vector contains many elements that drive the iteration process). But what if we want to iterate over multiple inputs? Consider the following function, which takes two separate variables x and y as inputs, combines them in a data frame, and then uses them to create a third variable z.

Note: Again, this is a rather silly function that we could easily improve upon using standard (vectorised) tools. But our goal here is to demonstrate programming principles with simple examples that carry over to more complicated cases where vectorisation is not possible.

## Create a named function
multi_func = 
  function(x, y) {
    z = (x + y) / sqrt(x)
    data.frame(x, y, z)
  }

Before continuing, quickly test that it works using non-iterated inputs.

multi_func(1, 6)
#>   x y z
#> 1 1 6 7

Great, it works. Now let’s imagine that we want to iterate over various levels of both x and y. There are two basics approaches that we can follow to achieve this:

1. Use base::mapply()/base::Map() or purrr::pmap().
2. Use a data frame of input combinations.

We’ll quickly review both approaches, continuing with the multi_func() function that we just created above.

## Note that the inputs are now moved to the *end* of the call. Also, mapply() 
## is based on sapply(), so we also have to tell it not to simplify if we want 
## to keep the list structure.
mapply(
  multi_func,
  x = 1:5,         ## Our "x" vector input
  y = 6:10,        ## Our "y" vector input
  SIMPLIFY = FALSE ## Tell it not to simplify to keep the list structure
  ) %>%
  do.call("rbind", .)
#>   x  y        z
#> 1 1  6 7.000000
#> 2 2  7 6.363961
#> 3 3  8 6.350853
#> 4 4  9 6.500000
#> 5 5 10 6.708204

## Note that the function inputs are combined in a list and come first.
pmap_df(
  list(x = 1:5, y = 6:10), 
  multi_func
  )
#>   x  y        z
#> 1 1  6 7.000000
#> 2 2  7 6.363961
#> 3 3  8 6.350853
#> 4 4  9 6.500000
#> 5 5 10 6.708204

parent_func =
  ## Main function: Takes a single data frame as an input
  function(input_df) {
    d =
      ## Nested iteration function
      map_df(
      1:nrow(input_df), ## i.e. Iterate (map) over each row of the input data frame
      function(n) {
        ## Extract the `x` and `y` values from row "n" of the data frame
        x = input_df$x[n]
        y = input_df$y[n]
        ## Apply our function on the the extracted values
        multi_func(x, y)
      })
    return(d)
    }

## Case 1: Iterate over x=1:5 and y=6:10
input_df1 = data.frame(x = 1:5, y = 6:10)
parent_func(input_df1)
#>   x  y        z
#> 1 1  6 7.000000
#> 2 2  7 6.363961
#> 3 3  8 6.350853
#> 4 4  9 6.500000
#> 5 5 10 6.708204

## Case 2: Iterate over *all possible combinations* of x=1:5 and y=6:10
input_df2 = expand.grid(x = 1:5, y = 6:10)
parent_func(input_df2)
#>    x  y         z
#> 1  1  6  7.000000
#> 2  2  6  5.656854
#> 3  3  6  5.196152
#> 4  4  6  5.000000
#> 5  5  6  4.919350
#> 6  1  7  8.000000
#> 7  2  7  6.363961
#> 8  3  7  5.773503
#> 9  4  7  5.500000
#> 10 5  7  5.366563
#> 11 1  8  9.000000
#> 12 2  8  7.071068
#> 13 3  8  6.350853
#> 14 4  8  6.000000
#> 15 5  8  5.813777
#> 16 1  9 10.000000
#> 17 2  9  7.778175
#> 18 3  9  6.928203
#> 19 4  9  6.500000
#> 20 5  9  6.260990
#> 21 1 10 11.000000
#> 22 2 10  8.485281
#> 23 3 10  7.505553
#> 24 4 10  7.000000
#> 25 5 10  6.708204