Most of the data available through tidycensus and the Census API is
aggregated to certain geographic levels (tract, county, state, etc.). In
other words, the data we get by executing get_acs() has
been summarized by the Census Bureau so that we are able to learn how
many people live in a particular county or what the median household
income of a state is. There are thousands of individual variables that
the Census aggregates and publishes in tabular form.
For many purposes, these pre-aggregated tables have enough information to work with. But, the Census Bureau also releases microdata from the American Community Survey. Microdata is the individual-level responses to the ACS that is used to create the summary and detail tables the Census publishes. Instead of a getting one row per state from a table, we can get one row per respondent. For the American Community Survey, this data is called the Public Use Microdata Sample (PUMS).
Using PUMS data instead of the published tables can be a very powerful tool. It can, for instance, allow you to create custom estimates that aren’t available in pre-aggregated tables. You can also use microdata to fit models on individual-level data instead of modeling block groups or census tracts.
Until recently, PUMS data was only available from the Census Bureau FTP site and was somewhat cumbersome to use. Now, PUMS data is available via web API, which means you can easily access it in R by using tidycensus.
PUMAs
One trade-off with using PUMS data as compared to aggregated data is that you only get the state and public use microdata area (PUMA) of each individual in the microdata. PUMAs are Census geographies that contain at least 100,000 people and are entirely within a single state. They are built from census tracts and counties and may or may not be similar to other recognized geographic boundaries. In New York City, for instance, PUMAs are closely aligned to Community Districts. So, if you are interested in pulling data about block groups, census tracts, or other small areas, you can’t use PUMS data.
PUMS data dictionaries
The first place to start is to identify the variables you want to
download. There are hundreds of PUMS variables and each describes
characteristics of a person or a housing unit.
pums_variables is a dataset built into tidycensus that
contains names, descriptions, codes, and other metadata about the PUMS
variables.
Because PUMS variables and codes can change slightly by year, you
should select which year’s data you will be working with and filter
pums_variables for only that year and survey. In the
following example, we will be working with the 2018 1-year American
Community Survey.
Later in this vignette, we use the survey and srvyr to calculate PUMS estimates, so if you want to follow along and you don’t already have them installed, you should install these two packages from CRAN.
install.packages(c("survey", "srvyr"))
library(tidyverse)
library(tidycensus)
pums_vars_2018 <- pums_variables %>%
filter(year == 2018, survey == "acs1")pums_variables contains both the variables as well as
their possible values. So let’s just look at the unique variables.
## # A tibble: 513 × 4
## var_code var_label data_type level
## <chr> <chr> <chr> <chr>
## 1 SERIALNO Housing unit/GQ person serial number chr NA
## 2 DIVISION Division code based on 2010 Census definitions chr NA
## 3 PUMA Public use microdata area code (PUMA) based on 2010 Census definition (areas … chr NA
## 4 REGION Region code based on 2010 Census definitions chr NA
## 5 ST State Code based on 2010 Census definitions chr NA
## # ℹ 508 more rowsIf you’re new to PUMS data, this is a good dataset to browse to get a feel for what variables are available.
Person vs. housing unit
Some PUMS variables relate to individuals (e.g. age, race, educational attainment) while others relate to households/housing units (e.g. number of bedrooms, electricity cost, property value).
Individuals in PUMS data are always nested within a housing unit (The
ACS is sent to an housing unit address and all people living in that
unit complete the survey.) Housing units are uniquely identified by the
SERIALNO variable and persons are uniquely identified by
the combination of SERIALNO and SPORDER. In
the data dictionary included in tidycensus, variables are identified as
either “housing” or “person” variables in the level column. Here, for
example, are the first five person-level variables in the data
dictionary.
## # A tibble: 279 × 4
## var_code var_label data_type level
## <chr> <chr> <chr> <chr>
## 1 SPORDER Person number num person
## 2 PWGTP Person's weight num person
## 3 AGEP Age num person
## 4 CIT Citizenship status chr person
## 5 CITWP Year of naturalization write-in num person
## # ℹ 274 more rowsIt is important to be mindful of whether the variables you choose to analyze are person- or household-level variables.
Using get_pums() to download PUMS data
To download PUMS data from the Census API, use the
get_pums() function. If you’ve used other
get_*() functions in tidycensus, it should be familiar to
you. The key arguments to specify are variables,
state, survey, and year. Here we
get PUMA, SEX, AGEP, and
SCHL variables for Vermont from the 2018 1-year ACS.
vt_pums <- get_pums(
variables = c("PUMA", "SEX", "AGEP", "SCHL"),
state = "VT",
survey = "acs1",
year = 2018
)
vt_pums## # A tibble: 6,436 × 9
## SERIALNO SPORDER WGTP PWGTP AGEP PUMA ST SCHL SEX
## <chr> <dbl> <dbl> <dbl> <dbl> <chr> <chr> <chr> <chr>
## 1 2018GQ0000859 1 0 61 19 00200 50 19 1
## 2 2018GQ0001119 1 0 67 80 00200 50 11 2
## 3 2018GQ0001888 1 0 177 82 00400 50 16 2
## 4 2018GQ0002438 1 0 17 17 00100 50 16 2
## 5 2018GQ0003293 1 0 68 20 00400 50 19 2
## # ℹ 6,431 more rowsWe get 6436 rows and 9 columns. In addition to the variables we
specified, get_pums() also always returns
SERIALNO, SPORDER, WGTP,
PWGTP, and ST. SERIALNO and
SPORDER are the variables that uniquely identify
observations, WGTP and PWGTP are the
housing-unit and person weights, and ST is the state
code.
Notice that ST, SEX and SCHL
return coded character variables. You can look up what these codes
represent in the pums_variables dataset provided or set
recode = TRUE in get_pums() to return
additional columns with the values of these variables recoded.
vt_pums_recoded <- get_pums(
variables = c("PUMA", "SEX", "AGEP", "SCHL"),
state = "VT",
survey = "acs1",
year = 2018,
recode = TRUE
)
vt_pums_recoded## # A tibble: 6,436 × 12
## SERIALNO SPORDER WGTP PWGTP AGEP PUMA ST SCHL SEX ST_label SCHL_label SEX_label
## <chr> <dbl> <dbl> <dbl> <dbl> <chr> <chr> <chr> <chr> <ord> <ord> <ord>
## 1 2018GQ0001230 1 0 3 26 00300 50 20 1 Vermont/VT Associate's degree Male
## 2 2018GQ0002135 1 0 64 18 00100 50 19 2 Vermont/VT 1 or more years o… Female
## 3 2018GQ0002999 1 0 62 20 00400 50 19 2 Vermont/VT 1 or more years o… Female
## 4 2018GQ0004077 1 0 16 94 00200 50 21 2 Vermont/VT Bachelor's degree Female
## 5 2018GQ0006486 1 0 26 20 00400 50 19 1 Vermont/VT 1 or more years o… Male
## # ℹ 6,431 more rowsAnalyzing PUMS data
Remember that PUMS data is a sample of about 1% of the US population. When we want to estimate a variable that represents the entire population rather than a sample, we have to apply a weighting adjustment. A simple way to think about PUMS weights is that for each observation in the sample, the weight variable tells us the number of people in the population that the observation represents. So, if we simply add up all the person weights in our VT PUMS data, we get the (estimated) population of Vermont.
sum(vt_pums_recoded$PWGTP)## [1] 626299Another convenient approach to weighting PUMS data is to use the
wt argument in dplyr::count(). Here, we
calculate the population by sex for each PUMA in Vermont (there are only
four in the whole state!).
## # A tibble: 8 × 3
## PUMA SEX_label n
## <chr> <ord> <dbl>
## 1 00100 Male 108732
## 2 00100 Female 111564
## 3 00200 Male 73768
## 4 00200 Female 73416
## 5 00300 Male 61601
## 6 00300 Female 64806
## 7 00400 Male 65675
## 8 00400 Female 66737Many of the variables included in the PUMS data are categorical and we might want to group some categories together and estimate the proportion of the population with these characteristics. In this example, we first create a new variable that is whether or not the person has a Bachelor’s degree or above, group by PUMA and sex, then calculate the total population, average age, total with BA or above (only for people 25 and older), and percent with BA or above.
vt_pums_recoded %>%
mutate(ba_above = SCHL %in% c("21", "22", "23", "24")) %>%
group_by(PUMA, SEX_label) %>%
summarize(
total_pop = sum(PWGTP),
mean_age = weighted.mean(AGEP, PWGTP),
ba_above = sum(PWGTP[ba_above == TRUE & AGEP >= 25]),
ba_above_pct = ba_above / sum(PWGTP[AGEP >= 25])
)## # A tibble: 8 × 6
## # Groups: PUMA [4]
## PUMA SEX_label total_pop mean_age ba_above ba_above_pct
## <chr> <ord> <dbl> <dbl> <dbl> <dbl>
## 1 00100 Male 108732 38.2 34113 0.469
## 2 00100 Female 111564 40.4 36873 0.473
## 3 00200 Male 73768 41.4 15831 0.303
## 4 00200 Female 73416 43.6 20248 0.367
## 5 00300 Male 61601 43.7 14869 0.326
## 6 00300 Female 64806 45.0 21527 0.434
## 7 00400 Male 65675 41.9 12788 0.278
## 8 00400 Female 66737 44.6 18980 0.391Calculating standard errors
In the previous examples, you might have noticed that we calculate a point estimate, but do not get a standard error or margin of error as is included in aggregated Census tables. It is important to calculate standard errors when making custom estimates using PUMS data, because it can be easy to create a very small sub-group where there are not enough observations in the sample to make a reliable estimate.
The standard package for calculating estimates and fitting models
with complex survey objects is the survey
package. The srvyr
package is an alternative and wraps some survey functions to allow
for analyzing surveys using dplyr-style syntax.
tidycensus provides a function, to_survey(), that converts
data frames returned by get_pums() into either a survey or
srvyr object.
In order to generate reliable standard errors, the Census Bureau provides a set of replicate weights for each observation in the PUMS dataset. These replicate weights are used to simulate multiple samples from the single PUMS sample and can be used to calculate more precise standard errors. PUMS data contains both person- and housing-unit-level replicate weights.
To download replicate weights along with PUMS data, set the
rep_weights argument in get_pums() to
"person", "housing", or "both".
Here we get the same Vermont PUMS data as above in addition to the 80
person replicate weight columns.
vt_pums_rep_weights <- get_pums(
variables = c("PUMA", "SEX", "AGEP", "SCHL"),
state = "VT",
survey = "acs1",
year = 2018,
recode = TRUE,
rep_weights = "person"
)To easily convert this data frame to a survey or srvyr object, we can
use the to_survey() function.
vt_survey_design <- to_survey(vt_pums_rep_weights)By default, to_survey() converts a data frame to a
tbl_svy object by using person replicate weights. You can
change the arguments in to_survey if you are analyzing
housing-unit data (type = "housing") or if you want a
survey object instead of a srvyr object. (Although, tbl_svy
objects also have a svyrep.design class so you can use
survey functions on srvyr objects.)
Now, with our srvyr design object defined, we can calculate the same estimates as above, but we now get column with standard errors as well.
library(srvyr, warn.conflicts = FALSE)
vt_survey_design %>%
survey_count(PUMA, SEX_label)## # A tibble: 8 × 4
## PUMA SEX_label n n_se
## <chr> <ord> <dbl> <dbl>
## 1 00100 Male 108732 1123.
## 2 00100 Female 111564 1360.
## 3 00200 Male 73768 754.
## 4 00200 Female 73416 809.
## 5 00300 Male 61601 491.
## 6 00300 Female 64806 578.
## 7 00400 Male 65675 702.
## 8 00400 Female 66737 776.The srvyr syntax is very similar to standard dplyr syntax, so this
should look familiar; we’ve swapped out count() for
survey_count() and we don’t need a wt argument
because we defined the weights when we set up the srvyr object.
The equivalent estimate using survey syntax looks like this:
## PUMA SEX_labelMale SEX_labelFemale se1 se2
## 00100 00100 108732 111564 1122.9765 1360.3476
## 00200 00200 73768 73416 754.0906 808.7364
## 00300 00300 61601 64806 490.5432 578.3529
## 00400 00400 65675 66737 702.0659 775.9812We can also repeat the estimate we did above and calculate the percentage of people that are 25 and up with a bachelor’s degree, while this time returning the upper and lower bounds of the confidence interval for these estimates. This time though, we have to subset our data frame to only those 25 and older before we summarize.
vt_survey_design %>%
mutate(ba_above = SCHL %in% c("21", "22", "23", "24")) %>%
filter(AGEP >= 25) %>%
group_by(PUMA, SEX_label) %>%
summarize(
age_25_up = survey_total(vartype = "ci"),
ba_above_n = survey_total(ba_above, vartype = "ci"),
ba_above_pct = survey_mean(ba_above, vartype = "ci")
)## # A tibble: 8 × 11
## # Groups: PUMA [4]
## PUMA SEX_label age_25_up age_25_up_low age_25_up_upp ba_above_n ba_above_n_low ba_above_n_upp
## <chr> <ord> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 00100 Male 72680 70216. 75144. 34113 29913. 38313.
## 2 00100 Female 77966 75671. 80261. 36873 32202. 41544.
## 3 00200 Male 52278 50826. 53730. 15831 13327. 18335.
## 4 00200 Female 55162 53643. 56681. 20248 17679. 22817.
## 5 00300 Male 45634 44743. 46525. 14869 12638. 17100.
## 6 00300 Female 49546 48576. 50516. 21527 19010. 24044.
## 7 00400 Male 45960 45067. 46853. 12788 10699. 14877.
## 8 00400 Female 48601 47783. 49419. 18980 16540. 21420.
## # ℹ 3 more variables: ba_above_pct <dbl>, ba_above_pct_low <dbl>, ba_above_pct_upp <dbl>Modeling with PUMS data
An important use case for PUMS data is doing regression analysis or
other modeling. Let’s try to predict the length of an individual’s
commute to work (JWMNP) based on their wages, employer type
(private, public, self), and which PUMA they live in. We’ll use the 2018
5-year estimates so we have a bigger sample to work with.
vt_pums_to_model <- get_pums(
variables = c("PUMA", "WAGP", "JWMNP", "JWTR", "COW", "ESR"),
state = "VT",
survey = "acs5",
year = 2018,
rep_weights = "person"
)Now, we filter out observations that aren’t relevant, do a little recoding of the class of worker variable, and finally convert the data frame to a survey design object.
vt_model_sd <- vt_pums_to_model %>%
filter(
ESR == 1, # civilian employed
JWTR != 11, # does not work at home
WAGP > 0, # earned wages last year
JWMNP > 0 # commute more than zero min
) %>%
mutate(
emp_type = case_when(
COW %in% c("1", "2") ~ "private",
COW %in% c("3", "4", "5") ~ "public",
TRUE ~ "self"
)
) %>%
to_survey()Let’s quickly check out some summary stats using
srvyr.
vt_model_sd %>%
summarize(
n = survey_total(1),
mean_wage = survey_mean(WAGP),
median_wage = survey_median(WAGP),
mean_commute = survey_mean(JWMNP),
median_commute = survey_median(JWMNP)
)## # A tibble: 1 × 10
## n n_se mean_wage mean_wage_se median_wage median_wage_se mean_commute mean_commute_se
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 282733 1933. 44601. 437. 35000 251. 23.3 0.233
## # ℹ 2 more variables: median_commute <dbl>, median_commute_se <dbl>
vt_model_sd %>%
survey_count(emp_type)## # A tibble: 3 × 3
## emp_type n n_se
## <chr> <dbl> <dbl>
## 1 private 224760 2001.
## 2 public 42966 1374.
## 3 self 15007 756.And now we’re ready to fit a simple linear regression model.
model <- survey::svyglm(log(JWMNP) ~ log(WAGP) + emp_type + PUMA, design = vt_model_sd)
summary(model)##
## Call:
## survey::svyglm(formula = log(JWMNP) ~ log(WAGP) + emp_type +
## PUMA, design = vt_model_sd)
##
## Survey design:
## Called via srvyr
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 1.699601 0.102181 16.633 < 2e-16 ***
## log(WAGP) 0.113913 0.009548 11.931 < 2e-16 ***
## emp_typepublic -0.052845 0.027420 -1.927 0.0578 .
## emp_typeself -0.306227 0.050016 -6.123 4.20e-08 ***
## PUMA00200 0.007800 0.023943 0.326 0.7455
## PUMA00300 0.023629 0.025111 0.941 0.3498
## PUMA00400 -0.117300 0.026028 -4.507 2.46e-05 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for gaussian family taken to be 0.6816895)
##
## Number of Fisher Scoring iterations: 2Mapping PUMS data
If you’ve used the spatial functionality in tidycensus before, you
might also want to put your custom PUMS estimates on a map. To do this,
let’s use tigris to download PUMA boundaries for the states
in New England as an sf object.
ne_states <- c("VT", "NH", "ME", "MA", "CT", "RI")
ne_pumas <- map(ne_states, tigris::pumas, class = "sf", cb = TRUE, year = 2018) %>%
reduce(rbind)Next we download the income-to-poverty ratio from the PUMS dataset and calculate the percentage of population below 200% of the poverty line for each PUMA.
ne_pums <- get_pums(
variables = c("PUMA", "POVPIP"),
state = ne_states,
survey = "acs1",
year = 2018
)
ne_pov <- ne_pums %>%
group_by(ST, PUMA) %>%
summarize(
total_pop = sum(PWGTP),
pct_in_pov = sum(PWGTP[POVPIP < 200]) / total_pop
)And now we can make a choropleth map by joining the PUMA boundaries with the PUMS data.
ne_pumas %>%
left_join(ne_pov, by = c("STATEFP10" = "ST", "PUMACE10" = "PUMA")) %>%
ggplot(aes(fill = pct_in_pov)) +
geom_sf() +
scale_fill_viridis_b(
name = NULL,
option = "magma",
labels = scales::label_percent(1)
) +
labs(title = "Percentage of population below 200% of the poverty line") +
theme_void()
Verification of PUMS estimates
The Census provides calculated estimates and standard errors for a number of states and variables to ensure that users’ analysis of PUMS data matches the expected values.
In the following example, we generate a few estimates using the methodology described above. You may want to perform a similar verification on your system to ensure your PUMS estimates will be accurate.
Here are a few estimates we’ll verify from the 2018 ACS 1-year PUMS estimates for user verification file:
| State | Variable | Estimate | SE |
|---|---|---|---|
| Wyoming | GQ institutional population (RELP=16) | 7018 | 3 |
| Utah | Owned free and clear (TEN=2) | 209632 | 3972 |
| Hawaii | Age 0-4 | 86452 | 944 |
wy_relp <- get_pums(
variables = "RELP",
state = "Wyoming",
survey = "acs1",
year = 2018,
rep_weights = "person"
)
ut_ten <- get_pums(
variables = "TEN",
state = "Utah",
survey = "acs1",
year = 2018,
rep_weights = "housing"
)
hi_age <- get_pums(
variables = "AGEP",
state = "Hawaii",
survey = "acs1",
year = 2018,
rep_weights = "person"
)## # A tibble: 1 × 3
## RELP n n_se
## <chr> <dbl> <dbl>
## 1 16 7018 2.64
ut_ten %>%
distinct(SERIALNO, .keep_all = TRUE) %>%
to_survey(type = "housing") %>%
survey_count(TEN) %>%
filter(TEN == 2)## # A tibble: 1 × 3
## TEN n n_se
## <chr> <dbl> <dbl>
## 1 2 209632 3972.## # A tibble: 1 × 2
## age_0_4 age_0_4_se
## <dbl> <dbl>
## 1 86452 944.3 for 3 – yay!