"V体育平台登录" The Effects of Failing to Include Hard-to-Reach Respondents in Longitudinal Surveys

Data Collection Procedures

In the summer of 2001, WCLS researchers approached 1786 male and female applicants for cash aid and successfully interviewed a representative sample of adults (n = 1510), yielding a baseline response rate of 85%. For the main survey cohort, the researchers randomly selected applicants who received aid, intentionally oversampling problem drinkers and frequent drug users. They reinterviewed respondents at the 1-year anniversary of the baseline survey. Following Lown et al., we focused on female WCLS respondents. The women in our sample (N = 498) included 108 General Assistance recipients and 390 TANF recipients. One woman died before the second wave of data collection was complete. She was dropped from the analysis.

WCLS contracted a fieldwork agency to interview respondents, with the expectation that all cases would remain open until the respondents were found. This agency returned cases with no remaining viable leads to WCLS scientific staff for further tracking. A survey tracker with private investigator training was hired to locate the most difficult cases, a strategy that is costly and unusual in public health research.¹⁹ Tracking consisted of multiple mailings, telephone calls to respondents and contacts, searches of public databases, home visits, and visits to parks, street corners, shelters, check cashing stores, soup kitchens, jails, drug treatment programs, and other promising sites. Interviewers shared files; at times, more than 1 interviewer simultaneously tracked a respondent. We did not place any predetermined limits on the number of contact attempts. As a result, respondents received up to 12 letters, 57 calls, and 28 visits before either we found them or we ended the search. Cases remained open for an average of 55 days. Continuing respondents received cash incentives of $40 to $50 at each wave of data collection.

Measures

To empirically identify individuals who were hard to reach, we coded and analyzed field note files maintained by the study team throughout tracking for the 1-year follow-up. These files contained complete logs of all contact attempts for every respondent in the sample. Multiple raters noted the details of each contact attempt. We entered the data into a spreadsheet program (Microsoft Excel 2000; Microsoft Corp, Redmond, WA), cleaned the data, and concatenated them with the survey data files, matching by respondent identification number. To assess interrater reliability, a subsample of 94 files was test-rated by 4 different raters. The raters agreed on the final classification of hard-to-reach status in 93 of the 94 cases.

Interviews with fieldwork experts in US survey research centers²⁰ and descriptions of extended effort in the literature^1,3,16,19,37 informed our definition of “hard to reach.” Because interviewers shared files and there was no strict order for the methods used to track respondents, we did not attempt to evaluate each tracking method separately. Instead, we used a summary measure. Respondents were classified as hard to reach if extended effort was needed to contact them—that is, if any of the following conditions were met: (1) more than 14 telephone calls were made to the respondent or her designated contacts, (2) more than 3 residential visits or 1 or more nonresidential visit was made, (3) more than 5 letters were mailed to the respondent and alternative contacts, (4) the duration of the search lasted over 60 days, or (5) the fieldwork agency turned the case back to the WCLS scientific staff. We classified respondents who were contacted without extended effort as easily found.

The outcome variable was participation status 12 months postbaseline (i.e., easily found, hard to reach, or lost). Of the 498 respondents in this analysis, we classified 339 as easily found and 100 as hard to reach. We classified the 59 respondents the interviewers were unable to reach as lost. Because we were interested in researchers' ability to find respondents rather than in respondents' willingness to be interviewed once found, we considered 4 respondents who were contacted but explicitly refused to be interviewed to be successfully located for the purposes of our study. Of these, 3 were easily found, and 1 was hard to reach.

Independent variables included race/ethnicity, age, education, and marital or cohabitation status, social isolation (no close friends or family members), current disability, episodes of homelessness in the past year, long-term ( > 1 year) unemployment, and family income under $10 000. Other measures included program type (General Assistance or TANF), self-reported health status (4 categories collapsed into 2: poor or fair, and good or excellent), and previously used self-reported measures^35,38–40 of past-year heavy drug use, problem drinking, and substance dependence. In addition, the nonresponse simulation comprised 8 summary measures of violent victimization identical to those developed by Lown et al. for their study.³¹

Data Analysis

We began by using the χ² test of association to compare the baseline characteristics of hard-to-reach and easily found respondents. Next, we examined crude response rates in the full sample and various subgroups, including and excluding the hard-to-reach respondents. We then performed a simulation based on the study on women and violence of Lown et al. In this simulation, we sequentially examined the effect on findings from the baseline responses of a cohort when excluding respondents who were lost at 12 months and also those who were hard to reach at 12 months from the analysis.

In the simulation, we analyzed exposure to violence among General Assistance and TANF recipients through nested comparisons of (1) the full baseline cohort sample of all aid recipients, (2) the baseline cohort sample excluding those lost at 12 months, and (3) the baseline cohort sample excluding those lost and those defined as hard to reach at 12 months. In keeping with Lown et al., we used the χ² test to compare General Assistance and TANF recipients in terms of baseline variables. The final steps in the simulation replicated the logistic regression analyses of Lown et al. in the 3 nested comparisons.

Lown et al. examined associations among variables in the WCLS applicant sample. Because we made no attempt to track applicants, our simulation was limited to those applicants who became aid recipients, participated in the longitudinal study, and were tracked over time. We analyzed baseline survey responses of aid recipients in the cohort study for 2 reasons. First, we wanted to include data from lost respondents in our simulation; we interviewed these respondents at baseline but not at 12 months. Second, Lown et al. used baseline data, and we wanted our simulation to follow their methods as closely as possible to approximate the effect of an extended follow-up effort on actual study results.

We applied poststratification weights to correct for sampling design (i.e., oversampling of problem drinkers and drug users in TANF and time spent recruiting participants in different programs, locations, and offices) and baseline nonresponse. Because our analysis focuses on response status at wave 2, we did not weight for nonresponse at 12 months. We considered results significant if P was less than .05. We conducted statistical analyses with the survey commands in Stata versions 8 and 9 (StataCorp LP, College Station, TX). These commands implement a design-based F statistic to assess associations between categorical variables.

"VSports" Descriptive Analysis of Effects of Extended Tracking

We expected to find that retaining hard-to-reach respondents would substantially improve response rates. Table 1 shows that excluding the hard-to-reach group would have lowered the overall 1-year follow-up response rate from 89% to 71% and decreased response rates in some subgroups to well under 70%. Without the extended tracking, it appears that the WCLS would have failed to retain substantial numbers of respondents who were African American, in poor or fair health, homeless, or heavy substance users or who had very low incomes or were victims of violence.

Comparison of Hard-to-Reach and Easily Found Respondents

We hypothesized that bivariate comparisons would reveal many significant differences between respondents who were easily found and those who were found but hard to reach at 12 months. However, as shown in Table 2, the only characteristic that was significantly associated with being hard to reach was substance dependence. There were no significant differences in race/ethnicity, age, education, marital status, parental status, income, employment, disability, housing stability, program type, health status, problem drinking, frequent drug use, or violent victimization.

Nonresponse Simulation Analysis

To simulate the effects of excluding hard-to-reach respondents on the findings of Lown et al. (i.e., women who applied for General Assistance had more significant risk factors for violent victimization—problem drinking, divorce or separation, poverty, or homelessness—than did women who applied for TANF),³¹ we compared sociodemographic, health, and behavioral characteristics of the General Assistance and TANF subsamples with and without the hard-to-reach respondents. Because, as in the study by Lown et al., significant differences between the General Assistance and TANF groups remained even when the hard-to-reach women were excluded from the analysis (data not shown), this initial bivariate comparison suggested that excluding the hard to reach would not have a measurable impact on overall results. However, as Table 3 illustrates, biasing effects emerged when we compared violent victimization among General Assistance and TANF recipients.

Odds ratios and prevalence estimates in all 8 categories of violence in the General Assistance and TANF groups were similar for the full baseline cohort and for all recipients who were found at the 12-month follow-up. However, excluding the hard-to-reach women from the analysis produced substantially different point estimates; odds ratios were biased downward in 7 of 8 categories. In 2 of the categories, confidence intervals widened, but differences remained significant; in 6 of the categories, between-group differences decreased below the point of statistical significance; and in 5 of the categories, confidence intervals narrowed. This suggests that, in the absence of the extended tracking efforts, we would have seen associations between violent victimization and program type in only 2, rather than all 8, categories of violence and in no category of partner violence.

"VSports手机版" Strengths and Limitations

Our access to fieldwork files allowed us to systematically document the extent of tracking efforts for all study participants. Respondents are usually defined a priori as “hard to reach” by virtue of their sociodemographic and lifestyle characteristics. In this study, we used interviewers' field notes to empirically examine a posteriori the characteristics of demonstrably hard-to-reach respondents. However, the files were working documents that WCLS scientific staff and several interviewers used, sometimes simultaneously, to track respondents. The resulting complexity puts meaningful cost estimations beyond the scope of this study. Future studies of tracking procedures and level of effort could be designed to provide data that support cost effectiveness analyses.

Other limitations apply. The relatively narrow socioeconomic distribution of the WCLS sample provided us with an opportunity for detailed examination of a population that includes a large proportion of hard-to-reach study participants. Yet findings for the exclusively female welfare sample are not necessarily generalizable to other poor populations in other locales or to men living in poverty. We included only baseline survey responses in our simulation analysis, which sorted respondents by their participation status in the second wave of data collection 12 months later. Differences that exist at baseline may increase or decrease over time. In addition, because we used only self-reported responses and did not compare them with medical or administrative records, we are unable to address concerns about the quality and validity of data provided by respondents who were hard to reach.²⁶

Future research should examine the effects of retaining hard-to-reach respondents in general population studies and in changes across multiple waves of longitudinal data collection. Finally, we looked at only the likely effects of failure to retain hard-to-reach respondents on results for violent victimization. Estimates for some variables may be more vulnerable to nonresponse bias than others in the same survey.^26,27

"V体育官网" Implications

Attrition of hard-to-reach respondents can be described as a form of exclusion, a separation of study respondents from research itself, from the attention of service providers and policymakers who use the results of research to inform their decision-making processes, and from social benefits that may arise as research is translated into policy and practice.

Our findings underscore the importance of retaining hard-to-reach respondents for accuracy and reliability in public health research. Some investigators suggest the risk of attrition bias may be reduced by oversampling respondents with characteristics linked to being hard to reach,⁴ by making postsurvey adjustments based on carefully chosen auxiliary variables,²⁶ or by including survey questions that make it possible to identify respondents who are likely to be hard to reach and then plan for the extended tracking of these respondents.² However, these techniques may be of questionable value in studies in which few measurable characteristics differentiate hard-to-reach respondents from others in the population.

The findings of this study suggest that even when demographic and health-related differences appear to be small, the failure to retain hard-to-reach respondents can substantially affect study findings. Therefore, at the outset of data collection, it may be difficult, perhaps impossible, to predict if such failure would introduce bias and affect the accuracy of the final results. We have also shown that extended follow-up efforts and protocols tailored for the study population as a whole can be an important factor in increasing response rates and limiting bias in studies of poor populations such as this. Groves warns that the risk of producing biased and inaccurate results increases as the differences between responders and non-responders become larger⁶; in studies of more economically and socially diverse populations, the effects of retaining hard-to-reach respondents may prove to be even more pronounced than those reported here.

The results of this study suggest that attaining high response rates should be a high priority for public health researchers and their funding agencies. Moreover, the findings provide justification for developing standard procedures for extended follow-up efforts to retain hard-to-reach respondents in low-income and marginalized populations. To the extent that future research can empirically identify hard-to-reach groups in a wider range of study populations, researchers will be able to cultivate more-efficient strategies for tracking these groups and for studying the extent to which extended tracking effort affects response rates and bias in results. Ultimately, such studies will allow researchers to develop realistic standards for careful, extended follow-up in key target populations and in general population studies. Only then will we be in a position to advocate sufficient resources for including and retaining members of hard-to-reach groups in all public health research.