The American Voices Project (AVP) is a nationally representative mixed-methods research project that characterizes everyday life in the United States. We briefly describe here the methodology for sampling, recruitment, and interviewing.
Sampling methodology for first national sample
The AVP is based on three-stage cluster sampling. In the first stage, Census tracts were selected by stratified sampling, with the stratification ensuring that key geographic areas are covered. In the second stage, a single block group was sampled within each tract, allowing us to focus on well-defined communities. The tracts and block groups chosen in these first two stages were selected with a probability proportional to the poverty population. Because one of the goals of the AVP is to better understand the everyday life of the low-income population, the third stage of the cluster sampling oversampled the bottom half of the income distribution, with addresses stratified on a modelled estimate of income.
We randomly selected 192 Census block groups and thirteen Native Nation tribal areas.1 Because of the pandemic, we have yet to conduct interviews in the tribal areas and in four of the 192 block groups that were slated for in-person enumeration in the spring and summer of 2020. Below we describe the sampling strategy for the 192 selected Census block groups.
First stage site selection: census tracts
The primary sampling unit in the first stage is the Census tract. Census tracts are sampled proportional to their poverty population as indexed by the official poverty measure using 2016 five-year estimates of the American Community Survey (2012–2016).
The first step in implementing the sampling design was to divide the United States into a set of 82 areas defined by (a) the fifty states, (b) twenty-two high-poverty cities, (c) six high-poverty rural areas in Appalachia, (d) two high-poverty rural areas in the Mississippi Delta, (e) one high-poverty area in Texas bordering Mexico, and (f) Puerto Rico. These areas are mutually exclusive. The area defined by any given state, for example, is the residual portion of that state that remains after subtracting out areas pertaining to the cities or other areas in our design. This subdivision of the United States into 82 mutually exclusive and exhaustive areas ensures that our sites are well distributed across the very heterogeneous U.S. geography and includes, in particular, areas (e.g., the Mississippi Delta) that are especially important in understanding U.S. poverty. We then selected one or more tracts within each of these 82 areas proportional to the poverty population.
Key cities: We selected at least one site in 22 high-poverty cities in the United States. The cities were selected to represent large, high-poverty cities in each Census region. Although only one site was selected in most of these cities, more than one site was allocated to cities representing more than 1 percent of the U.S. poverty population. The cities (and the number of sites within them if more than one) are Atlanta, Baltimore, Boston, Chicago (2), Cleveland, Dallas, Detroit, Fresno, Houston (2), Kansas City, Los Angeles (3), Memphis, Miami, Milwaukee, New Orleans, New York (7), Philadelphia (2), and Phoenix, St. Louis, Tucson, Minneapolis-St. Paul, and Washington, D.C. It follows that 33 sites in total were selected from these 22 cities.
Appalachian high-poverty rural areas: The Appalachian Regional Commission (ARC) is a regional development agency that provides funding to projects in Appalachia. From the ARC’s service area, we identified Census tracts that have at least a 25 percent poverty rate and are at least 75 percent rural using the Census Bureau definition.2 Under this definition, eight states contain Appalachian high-poverty rural populations: Alabama, Kentucky, Mississippi, North Carolina, Ohio, Pennsylvania,Tennessee, and West Virginia. We selected six sites from these areas, one from each state except for Mississippi and West Virginia, whose “Appalachian” area contained less than 0.05 percent of the U.S. poverty population and was thus combined with the rest of the counties in the state.
Mississippi Delta high-poverty rural areas: The Delta Regional Authority (DRA) is an agency that supports economic development and job creation in and around the Mississippi Delta. We identified the high-poverty rural areas within the DRA’s jurisdiction using the “25–75” definition given above with the following exceptions: (1) When the poverty population in a state’s DRA-served area comprised less than 0.05 percent of the total U.S. poverty population, we combined that area with the rest of the state; (2) We also combined Alabama’s DRA-served area with the non-Appalachian part of Alabama, as the state does not border the Mississippi River. Under this definition, Mississippi Delta rural poverty is concentrated primarily in two states, Louisiana and Mississippi. We selected two sites, one from each state.
Texas-Mexico border area: The Texas-Mexico border includes large population centers with high poverty rates. Although the poverty population in border counties within Arizona and New Mexico is not large enough to justify a special stratum, the poverty population in border counties in Texas is, by contrast, very large, containing 1.5 percent of the U.S. poverty population. We therefore selected three sites from the Texas counties that border Mexico. We do not define a border region in California because San Diego and Imperial counties are geographically very large and contain many places that are not representative of border communities.
Puerto Rico: We selected five census tracts in Puerto Rico because of the variety of poverty conditions in Puerto Rico. We therefore sought to ensure that Puerto Rico was well represented in the AVP sample.
50 states: The remaining 143 sites were distributed under the constraint that each state would have at least one site. If a state contained one or more of our 49 special areas described above (e.g., the two Chicago sites within Illinois), we always selected at least one site in the balance of the state that fell outside the relevant predesignated area (i.e., Chicago). The number of sites per state depends on the size of its poverty population. Of the 143 sites drawn outside the cities and special areas described above, there are 18 states with just one such site, 13 states with two, 8 states with three, 7 states with four, 1 state (i.e., Georgia) with six, 2 states (i.e., Florida and Texas) with eleven, and 1 state (i.e., California) with eighteen such sites. Table 1 lists how all sites are distributed by state and special area.
Second stage selection: block groups
The goal of the AVP is to understand the diversity of neighborhood-level experiences across the country. By focusing on well-defined neighborhoods, we hope to gain a better understanding of each neighborhood and how people live within it. We randomly select (proportional to its poverty population) one Census block group within each site before sampling addresses within the block group. Block groups contain between 600 and 3,000 people (approximately 250–1,300 households).
Third stage: address-based sample
As detailed below, we oversampled addresses that are likely to be low-income in order to ensure that our sample includes enough low-income households to reliably characterize them. The structure of the sample also allows us to generalize to two populations. The primary unit of interest is the household. Second, with proper weighting and under certain plausible assumptions, it will be possible to generalize to the population of adult individuals in the United States. We achieved a representative sample of households by sampling addresses and conducting interviews of households heads. These interviews include household-level data and individual-level data on household members.
We acquired an address sample frame from a commercial address vendor. The sample frame contained an estimate of the household income by address. We also built a model (based on American Community Survey data) to construct an additional imputed estimate of household income.3 The imputed income and the estimate of income provided by the vendor were combined and used to stratify the addresses into four groups based on three cut points at 50 percent, 100 percent, and 200 percent of the supplemental poverty threshold.
Within each block group, a total of 65 addresses were sampled: 17 from the lowest-income stratum (i.e., less than 50% of the supplemental poverty measure threshold), and 16 from each of the remaining strata. This sample was then ordered randomly within strata and split into two samples: a “main sample” and a “refresher sample.” The refresher sample for each income stratum served as a real-time, in-field substitution when interviewers did not secure enough interviews from the main sample for that stratum. This approach allowed us to adjust for differential non-response or vacancies across the income strata (and thereby ensure a balanced number of interviews across income strata).
The main sample for each block group consisted of 25 addresses. The “refresher sample” contained the remaining 40 addresses.
Sampling methodology for subsequent national samples
Starting in September 2020, the Gates Foundation asked us to draw a second national sample, with the goal being to understand how the health and economic crises were playing out. This second sample, like the first one, rests on (a) a representative draw of Census tracts and block groups within them (again with over-sampling on low-income and middle-income sites), and (b) a representative address-based sample of households within each block group. We simplified the sampling design, however, by no longer ensuring representation within each of the 82 geographical areas (defined by states, high-poverty cities, and other high-poverty areas). Between September 2020 and August 2021, we interviewed another 706 households for this second national sample.
The sampling for this second national sample allowed us to generalize to three successive national subsamples, the first pertaining to the fall of 2020 (i.e., the “election and infection take-off period”), the second pertaining to early 2021 (i.e., the “storming of the Capitol” period), and the third pertaining to spring and summer of 2021 (i.e., the “vaccination period”).
Within-address selection of respondent
The interview protocol includes many questions and prompts pertaining to the household (rather than the individual). We defined a “household” as comprising all residents of an address that share resources. This definition was simple enough to allow interviewers in the field to readily implement it. Under this definition, there may be multiple households in an address, but a household cannot span multiple addresses.
We define a household head to be (a) the person (or persons) who make financial contributions to the household’s budget (by covering the expenses of other household members), or (b) household members who make important decisions or help with daily operations. If a resident at the address doesn’t qualify as a household head, they are a member of someone else’s household. If a resident covers their own expenses, they are their own household. We defined “covering one’s own expenses” as being responsible for at least two of the following three categories of expenditures: housing expenses, food expenses, and other living expenses.
Most addresses have only one household with one or multiple household heads. However, there are cases where an address has multiple households, such as roommates or multiple generations or extended families living together. When the first household head is interviewed, the interviewer assesses whether any of the other residents constitute a separate household. If there is one or more separate households that are related to the respondent, the interviewer asks to conduct an interview with one household head from each related household. If there is one or more separate households that are not related to the respondent, the interviewer randomly selects one of the related households to interview.
Respondent recruitment
The sampled households were recruited via a variety of methods that varied as the AVP shifted from a face-to-face to remote interview mode. In all cases, respondents were compensated, with the amount of compensation ranging from $60 to $145. The incentives were increased when the AVP shifted to a remote-interviewing mode. The following is a detailed description of the recruitment process within each of the key phases of the AVP.
In-person recruitment in first national sample (July 2019 to March 2020)
The AVP was initially designed for in-person recruitment and interviews. In-person fieldwork commenced in July 2019. In March 2020, all in-person fieldwork was halted, and any interviews that were previously scheduled were completed by telephone. No new addresses were recruited during this time. During the in-person recruitment phase, an invitation letter was sent to each sampled address, after which a pair (or in some cases a trio) of researchers visited the address. Two additional mailings were sent when we were unable to contact someone at the address. During this phase, 4,208 addresses were recruited, and 1,564 interviews were completed, with 163 of these interviews completed by telephone after in-person fieldwork ceased.
All-remote and previously visited sites in first national sample (April 2020 to August 2020)
Beginning in April 2020, the AVP began recruiting addresses in the sample by mail and by telephone (using the telephone number associated with the sampled address in the sampling frame). Because in-person recruitment was not possible, a lower response rate was assumed, and the full sample of 65 addresses were immediately released in 66 sampled sites that had not yet been recruited in person. Addresses in these sites received between 6 and 10 mailings over 41 to 67 days. Additionally, for the 60 percent of addresses with a telephone number listed on the sample frame, we attempted to reach them by telephone (with 4 calls, on average, per address). If there was no answer, a voicemail was left when possible.
During this period, we also sent 3 mailings over 18 days to addresses in 17 sites that had only been partially recruited during the in-person phase. Of these addresses, 39 percent had been sent one letter prior to March 2020. The remaining 61 percent had not been contacted previously. In total, 449 interviews were completed from April to August 2020.
All-remote recruitment in second national sample (September 2020 to August 2021)
The second national sample, as described above, relied entirely on mail recruitment in which addresses received 3–5 mailers. This second national sample occurred over three quarters (i.e., fall 2020; winter 2021; spring 2021), with the sample drawn for each quarter representing the U.S. population of households. The number of respondents making up these three subsamples are 296 (9/2020-12/2020), 262 (01/2021-03/2021), and 148 (04/2021-08/2021).
Interview protocol
The American Voices Project protocol blends qualitative, survey, and experimental methods. The qualitative protocol consists of open-ended questions about everyday experiences. It covers (a) a short life history and defining life course events; (b) the rhythm of everyday life; (c) the pattern of relationships with family, friends, coworkers, ad relatives; (d) the extent of economic hardship and characteristic reactions to such hardship; (e) the main sources of income and expenses; (f) the formal and informal jobs held by household members; (g) safety net program usage; (h) neighborhood and living situations; (i) health and health care; (j) mental health, drug use, anxiety, and stress; (k) political beliefs, civic engagement, and participation; and (l) key identities and sources of meaning.
After the qualitative portion of the interview, the respondent completes a dignity exercise and a quantitative survey to assess subjective social standing, perceived discrimination, perceived control, perceived opportunity, and perceived health and wellbeing.
Each interview is audio-recorded, and the economic, demographic, and survey data are recorded by hand and digitally entered after the interview. The average interview recording is 2.2 hours. Interviews lasted about 15 minutes longer, on average, when the interview was conducted over the telephone. When foreign-language interpreters are required, or when time is constrained, interviewers are trained to administer a shortened protocol.
As the pandemic unfolded, new questions were added on recent changes in health and health care (including questions on Covid-19), political views, employment and earnings, safety net usage (including new stimulus programs), schooling and child care arrangements, views on vaccination, critical junctures in the life course, and racism and policing.
Interviewers
The interviewers, all of whom received intensive training in interviewing and qualitative methods, were a mix of advanced degree-holders (e.g., PhDs), graduate students, college graduates, and undergraduates selected through a highly competitive process. For the first national sample, an open call for interviewers was disseminated to the CPI’s national mailing list and an extensive list of colleges and universities, including HBCUs, HSIs, and TCUs. Because the response was overwhelming, only a very small proportion of applicants could be selected. These applicants underwent several weeks of intensive training in qualitative interviewing in Baltimore in the summer of 2019 and were then overseen by regional managers with extensive experience in qualitative methods.
The interviewers for the second national sample were drawn from Stanford undergraduates, Princeton undergraduates, and community-college undergraduates participating in Stanford’s Research Experience Program (REP). These interviewers also received intensive training in qualitative interviewing, either as part of a full-time summer fellowship or during a quarter-long class (“Monitoring the Crisis”). Interviewers were overseen by professors, TAs, and research managers.
Notes
1. We selected one tribal area from the twelve Bureau of Indian Affair regions, and we selected one Hawaiian Home Land.
2. https://www.census.gov/programs-surveys/geography/guidance/geo-areas/urban-rural.html
3. The model for imputing income was constructed using American Community Survey microdata at the Public Use Microdata Area (PUMA) level. Household income was regressed on home value (or rent), a nonlinear function of age, marital status, and education level. These PUMA-specific models were estimated for homeowners and renters separately. The calibrated model parameters were applied to the address database (which contained demographic data provided by the vendor and an appended estimate of home value).