Sampling Telephone Numbers and Adults, Interview Length, and Weighting in The California Health Interview Survey Cell Phone Pilot Study
e-mail: MikeBrick{at}Westat.com.
 |
Abstract |
|---|
 TOP Abstract california health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
This article describes several features included in a California Health Interview Survey cell phone pilot study that differ from earlier cell phone surveys conducted in the United States. One difference is that the study used a screening design and only adults living in cell-only households were interviewed. Most of the previous studies used dual frame designs. Another difference was the development and implementation of a within-household adult sampling procedure to cover adults when cell phones were shared in the household. The study was also intended to determine if conducting a cell phone interview of the same scope and length as a regular telephone interview was feasible. Most previous surveys were focused on cell phone topics and were shorter than the comparable landline surveys. We then explore some interesting problems that arise in weighting a cell phone survey using a screening design. We conclude with a discussion of planning a new cell phone survey in 2007 that incorporates findings from this study.
A pilot study of interviewing adults on cell phones was conducted in conjunction with the 2005 California Health Interview Survey (CHIS 2005). The goal of the study was to evaluate the feasibility of sampling and interviewing adults in households with only cell phones in future administrations of the CHIS. These adults are not included in most random digit dial (RDD) surveys in the United States. The study was intended primarily to address two issues that have not been fully investigated in previous cell phone surveys. The first is the feasibility of conducting a lengthy health survey interview on a cell phone, since most cell phone surveys have limited the length of the interview to make it more acceptable to the respondent. The second concern is the ability to randomly sample adults within a household for the interview using a procedure similar to those used in many RDD telephone surveys in the United States. Previous cell phone surveys interviewed any adult who answered the phone, irrespective of household composition. As discussed later, this practice may result in undercoverage. Understanding the feasibility of conducting full interviews and the practicality of within-household sampling of adults are essential if cell phone households are to be included in telephone surveys. To accomplish these goals, the study was designed with the target of completing 100 interviews with adults who lived in households with only cell phones.
The earliest attempts to interview adults by cell phone in the United States differed in some ways from the typical landline RDD survey. These cell phone surveys kept the interview relatively short, and the survey topic was often related specifically to cell phones or communications technology. Steeh (2004)
and Brick et al. (2007) describe two of the early surveys that included interviews of random samples of cell phone numbers. In both of these surveys, samples of both cell phone numbers and landline numbers were selected. Steeh reported the interview averaged less than 18 minutes, while Brick et al. reported an average of less than 10 minutes. In both surveys, the content and introduction were related to cell phones and technology, although some other items were included. More recent cell phone surveys are similar in restricting the length of the interview, but the content of the survey has become a bit broader. Keeter and Kennedy (2006) describe a 10-minute cell phone survey conducted by Pew Research in conjunction with the Associated Press. In that survey, topics other than cell phone use were included, but the first questions were about cell phones. The Behavioral Risk Factor Surveillance Survey (BRFSS) tested conducting cell phone surveys in six states in 2006 (Link et al. 2007). The average interview lasted about 10 minutes, and was an abbreviated version of the 20-minute landline BRFSS adult interview.
Fleeman (2007) reported on three cell phone tests conducted by Arbitron in which the goal of the telephone contact was to obtain enough information to send the household a mail (diary) survey. In this application, the phone interview was very short because the key data were obtained from responses to the mail questionnaire/diary. Because the telephone survey was short, no attempt was made to further shorten the cell phone interview. The last test reported by Fleeman (2007) differed from the other two in that households were screened and interviews were only attempted for cell-only households; households with landlines were screened out.
None of these surveys sampled adults within the household. The surveys described by Fleeman (2007) and Brick et al. (2007) were household surveys in which one adult responded for the entire household. However, most RDD surveys are surveys of adults rather than of households, and one or more adults are sampled from within the household and then interviewed. None of the other cell phone studies mentioned addressed within-household sampling of adults. For example, while the BRFSS survey randomly samples one adult per household in its RDD samples, its cell phone test (Link et al. 2007) did not sample adults within a household. They considered the cell phone a personal device that was linked to the adult who answered it.
Below, we address both the feasibility of conducting cell phone interviews of length comparable to the length of RDD surveys and the sampling of adults within a household. We present the sampling approach used in the CHIS cell phone study and the outcomes of the data collection. Some comparisons are made to results from the CHIS 2005 to provide context. We discuss the advantages and disadvantages of the within-household sampling approach used in the study. We then examine weighting a cell phone survey that screens and interviews only adults who live in households without landlines. The conclusion includes a description of some plans for continuing to survey cell phone households in the CHIS 2007.
|
CALIFORNIA HEALTH INTERVIEW SURVEY (CHIS) |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
The CHIS is a biennial, RDD telephone survey of California's population first administered in 2001. It is a collaborative project of the UCLA Center for Health Policy Research, the California Department of Health Services, and the Public Health Institute; Westat is the data-collection contractor. In the CHIS, one adult is sampled and interviewed in each screened household. The adult interview obtains extensive information on health status, health conditions, health-related behaviors, health insurance coverage, access to health care services, and other health and developmental issues. More information about the context of the CHIS interview is available at the web site (www.chis.ucla.edu). The CHIS 2005 adult interview took an average of 35 minutes to complete. The content of the adult interview and the data collection procedures used in the cell phone study mirrored those used in CHIS 2005 as closely as possible. The adult interview collected all the substantive data included in the CHIS 2005 interview, without shortening the interview. The data were collected for the cell phone study in 2006 immediately after the CHIS 2005 was completed.
|
Sampling |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
The sample for the cell phone study was selected in three stages: (1) sampling telephone numbers, (2) sampling households, and (3) sampling one adult within a household. All three stages are described below, but the details for the approaches that are the same as those used in other cell phone surveys are omitted.
SAMPLING TELEPHONE NUMBERS
In the first stage, a sample of telephone numbers was selected from two sources. The main sample was selected using essentially the same procedures described in Brick et al. (2007). The source for this component was the set of telephone numbers in the dedicated 1000-series cellular banks in the Telcordia database in March 2006 – numbers typically excluded in landline RDD surveys in the United States. A sample of 5,200 telephone numbers was selected by the sampling vendor, Marketing Systems Group (MSG). The sampling frame was telephone numbers with California area codes dedicated to cell phones (NXXTYPEs 04, 55, 60, 65, or 68).
Fleeman (2007) reported that about 5 percent of respondents in a national sample of cell phone numbers reported a state that was not the same as the state used to select the sample. In a local or state survey such as the CHIS, this implies that sampling from the state area codes will exclude a proportion of all households from the sample, as their cell numbers are linked to an area code in another state. Additionally, not all cell phone numbers fall in the dedicated series because people may transfer or port their landline number to a cell number, or local telephone companies may not assign the numbers consistently. To address the ported and misclassified numbers, we utilized a second source of cell phone numbers obtained from the regular "landline" RDD sample for CHIS 2005. In the normal processing of the RDD sample, MSG compares the sampled numbers to a proprietary database of cell phone numbers. This process identified 671 cell phone numbers (0.1 percent) in the entire CHIS 2005 RDD sample. These cell phone numbers were excluded from the CHIS 2005 sample, but were added to the cell phone study. As a result, the total sample size for the study was 5,871 numbers.
The sampled numbers were randomly ordered and prepared for fielding. As the data collection progressed, it became clear that a smaller sample would yield the target of approximately 100 completed adult interviews in cell-only households. To reduce costs, the last 613 sampled cell phone numbers were eliminated from any field work. The total sample of 5,258 numbers that were dialed at least once included 4,653 numbers from the cell-phone-dedicated series sample and 605 numbers from those identified by pre-processing the RDD sample. Because the yield was greater than expected, the level of effort expended for the numbers that were worked was also reduced. The implications of this reduced level of effort on outcomes such as response rates are discussed below.
SAMPLING HOUSEHOLDS
In the second stage, the residential status of the sampled telephone numbers was determined. If the number was for a household, then the household was classified by telephone status ("cell-only" if they lived in a household with no landline telephone or "both" if they also had a landline; see the appendix for question wording on these items). If the household was classified as being cell-only, then it was considered eligible and one adult was sampled. This approach of only interviewing in cell-only households was described in Fleeman (2007). All of the other cell phone surveys discussed above interviewed adults in both cell-only households and households with both types of service.
The screening approach used in this study does introduce a potential for bias due to nonsampling errors. One error could occur if the respondent incorrectly classified the telephone status of the household because he or she did not understand terms such as "regular telephone," or if the respondent deliberately tried to avoid being interviewed by misclassifying the household (Kalton 2003). This response error or measurement error results in the exclusion of the household from the study when it should have been included (i.e., a false negative). Another way of thinking about this is that the procedure introduces a coverage error because some households that should be surveyed are excluded. A second potential nonsampling error is due to nonresponse. Nonresponse bias could occur if households with certain characteristics responded at different rates. Despite the potential for bias, the results of the data collection presented below do not indicate any obvious signs of nonsampling errors in responding to the screening items or differential nonresponse within cell-only households.
An alternative approach to sampling households is a dual frame design such as that described in Brick et al. (2006). However, that study found compelling evidence that the dual frame design was subject to highly differential nonresponse, with cell-only households responding at a higher rate than households with both types of telephone service. The screening approach avoided these problems by only interviewing in cell-only households. It is also worth noting that if adults are not sampled within household in the dual frame approach, then the cell phone sample is potentially subject to a coverage bias in addition to a nonresponse bias.
SAMPLING ADULTS
As discussed above, previous cell phone surveys interviewed the adult who answered the cell phone. No attempt was made to sample from among eligible adults within the household. This approach contrasts with most landline RDD surveys in which one adult is sampled from among all adults in the household. The sampling of adults is necessary in RDD surveys to avoid the potential for bias, because the set of persons answering the telephone does not constitute a random sample of all adults (e.g., they are disproportionately female).
Secondary analysis of 2004 CPS cell phone supplement. In cell phone surveys, interviewing the adult who answers the telephone would yield a probability sample of adults provided there is a one-to-one link between cell phones and adults in the household. However, Tucker, Brick and Meekins (2007) show estimates from the cell phone supplement to the 2004 Current Population Survey (CPS) that suggest that the linkage is not unique. They reported a high proportion of sharing of cell phones in households with more than one adult (the CPS estimated 66 percent of cell-only households with more than one adult shared cell phones).
Because of the high percentage of households sharing cell phones reported in the CPS, we conducted secondary analyses of that dataset to investigate whether there were any factors that might be highly correlated with sharing cell phones. The analysis was restricted to households with a cell phone and more than one person. One hypothesis we considered was that households that share cell phones might be of lower economic status than those that do not share. We found little or no relationship to cell phone sharing by family income, home ownership, race (black or not), or education. Only marital status, Hispanic origin, and size of household had differences that were greater than three percentage points and statistically significant. Households including married persons, Hispanics, or five or more persons were more likely to share cell phones. These findings do not support the lower economic status hypotheses, but suggest other factors may play a role. Unfortunately, the 2004 CPS cell phone supplement data are not ideal for this analysis because they were collected when cell-only households were a much smaller proportion of the total U.S. population and the CPS estimates of sharing behavior may no longer generalize.
Within household selection in CHIS cell phone study. Procedures for the sampling of adults within the cell-only household were developed and implemented for this study based on principles similar to those used in landline RDD surveys. The sampling procedures were done in the screening interview, and the specific items that were used to screen for cell-only households and to sample adults are given in the appendix. In households with only one adult, no sampling was required. In cell-only households with more than one adult, sampling adults depended on whether other household members shared the cell phone. When adults shared the cell phone, we used the same within-household sampling method used in CHIS 2005 (Rizzo et al. 2004). In this scheme, the screener respondent (SR) is randomly selected for the adult interview with a probability equal to the inverse of the number of adults in the household. If the SR is not selected, then one adult other than the SR is selected for the adult interview using the next birthday method.
A total of 176 of the 837 screening interviews were completed with cell-only households. In 53 cell-only households there was only one adult, so the adult answering the screening items (the SR) was selected for the adult interview. The remaining 123 cell-only households had more than one adult, and the SR was asked if this cell phone number was shared with other adults. If the cell phone was not shared, then the SR was sampled. This situation occurred in 89 percent of the households with more than one adult (109 of 123 cases). In the 14 households with a shared cell phone, the SR was randomly selected seven times and an adult other than the SR was sampled seven times. Consequently, the SR was sampled in 169 of the 176 households. (It is unclear why the proportion of households with more than one adult whose members share a cell phone is so much lower than what was found in the 2004 CPS survey.)
This sampling scheme assumes that, in cell-only households with more than one adult, each adult has a cell phone if the sampled cell phone is not shared. However, this assumption may not be true; about 9 percent of SRs in the study said that they did not share this cell phone but reported more adults than cell phones in the household. This difference could be due to response error, but it may also indicate that the assumption of all adults having their own cell does not hold universally even in this situation. When we were developing the sampling scheme we recognized this weakness, but decided that asking the full battery of items required to ascertain the cell phone status of each adult in the household was a heavy burden and could distract from gaining cooperation. Thus, while our sampling scheme did not address all possible forms of within-household undercoverage, we believed it was a good compromise between reducing the potential for increased nonresponse and coverage errors. However, findings from the survey presented below raise questions about this compromise.
|
Data Collection |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
The data for the study were collected from April 1 to May 14, 2006, by Westat. The telephone interviewers dialed all calls to avoid any issues with the restrictions in the U.S. Telephone Consumer Protection Act regarding automated dialings to cell phone numbers. Apart from this, essentially the same computer-assisted telephone interviewing system used in the CHIS 2005 was used for the cell phone study. Some revisions to the interview were required, but the substantive items were retained intact. One revision was that the cell phone study only sampled adults, whereas a child or adolescent could also be sampled in the CHIS 2005. Another important difference was that the cell phone study was done only in English, whereas the CHIS 2005 interviews were conducted in five languages: English, Spanish, Chinese (Mandarin and Cantonese dialects), Vietnamese, and Korean.
The primary purpose of the screening interview was to determine if the respondent was an adult (18 years or older), used the cell phone for non-business purposes, and lived in a household without a landline. The introduction is shown in the appendix. It included text that asked if the respondent was driving or doing something that required full attention, in which case the interviewer immediately ended the call. Although there was not a specific question about the frequency of reaching a person while driving, interviewer messages show about 8 percent of the contacted households were driving when reached This percentage is likely to be an underestimate because some respondents refused without a comment and not all interviewers left complete messages on this outcome. (Concerns about safety and other related topics when calling cell phones are discussed in Lavrakas and Shuttles (2005).)
When the screener respondent was contacted, the interviewer indicated that a $5 payment would be sent to thank them for answering the screener interview. Of the 661 completed screeners where the SR said they had a landline, only 44 percent of respondents provided the name and address needed to mail the payment. One reason for the relatively low rate of asking for the payment is that the screening interview only took an average of 2.1 minutes to complete. If the household was identified in the screener as being cell-only, the name and address were not asked until the end of the full interview (similar to where the address information is requested in the CHIS 2005 adult interview).
The adult interview began by explaining the purpose of the call and its length and offered to call the sampled adult back on a landline if the adult was not comfortable talking on the cell phone. Of the 176 sampled adults, only seven asked to switch to a landline, and five of these completed the interview on a subsequent call. The respondents were informed about the survey's certificate of confidentiality and given information about the study and how to contact the UCLA Office for the Protection of Research Subjects. They also were offered $25 for participating in the interview. Nearly all of those who responded (96 percent) provided their name and address so the money could be mailed to them. The interview took on average 31.9 minutes to complete, which is shorter than the 34-minute average for CHIS 2005 adult interviews done in English. The shorter average length of interview likely was due to the different characteristics of the sampled adults (e.g., the cell-only respondents were younger than the RDD respondents).
CALLING OUTCOMES
The outcomes from dialing the 5,258 sampled numbers are shown in table 1. The calling protocol for the cell phone study was modeled after that used in CHIS 2005, and involved calling at various times of the day and days of the week. However, the level of effort was reduced since maximizing the response rate was not a priority of the cell phone study. For example, telephone numbers with unknown residential status outcomes (i.e., numbers never answered or answered only by a machine) in the cell phone study were attempted no more than five times, and half of these were dialed only once. In CHIS 2005, all numbers with this status were attempted at least 14 times. As a result, the 22 percent of numbers with unknown residential status in the study is about twice as large as in that category in CHIS 2005 and in a previous cell phone survey (Brick et al. 2007).
|
Table 1 shows that about 43 percent of all the numbers were identified as residential, 34 percent were not residential (all but 99 of these were not working numbers), and 23 percent could not be resolved. The distribution of the telephone numbers by residential status where that status was determined is similar to the distribution reported in Brick et al. (2007)
. The percentage of language problem cases is relatively large because the cell phone interview was only conducted in English and a substantial proportion of adults in California speak a language other than English. (As noted above, the inability to interview these adults is a consequence of the reduced level of effort in the cell phone study.) The number of cases classified as "respondent not available" is also probably higher than otherwise would be expected due to the limited calling protocol.
REFUSAL CONVERSION
An important part of any telephone data collection protocol is developing procedures to avoid refusals and trying to convert those who initially refuse to participate (refusal conversion). Interviewers in CHIS were trained in refusal avoidance techniques both before they began any interviews and then again after they had some interviewing experience. The training was intended to support interviewers in their use of specific techniques such as pace and clarity as well as study-specific responses to objections frequently raised by the households. For the main study, interviewers with higher than average cooperation rates were selected to work for refusal conversion cases. The refusal conversion interviewers reviewed the messages written by the interviewer from the previous refusal to help prepare before recontacting the case. They were allowed more flexibility in addressing respondent's concerns about the survey in the introduction, but were still required to ask the questions (including the screener) as they were written.
At the household screening level, refusal conversion attempts were made to households that refused. If they refused again, a second set of refusal conversion attempts were made. If any respondent gave what is termed a "hostile" response, then no further refusal conversion attempt was made.
The refusal conversion procedures used in the main study were used for the cell phone study with only minor modifications. The changes were the addition of training items such as how the study got the respondent's cell phone number and why we were calling on his or her cell phone. The other difference was that not all the planned conversion attempts were made due to the more limited calling protocol.
The results of the refusal conversion effort for the screening of households are given in table 2. Of the households that initially refused to participate 21.1 percent were finalized in some category other than refusal on a subsequent call. This refusal conversion effort was much more effective than the 7 percent cell phone survey conversion rate reported by Brick et al. (2007). Refusal conversion cases accounted for 21 percent of all the completed screener interview cases and 23 percent of the cell-only completed interviews.
|
Of note, a second refusal conversion attempt was made to nearly 700 telephone numbers that refused both the initial and the first refusal conversion attempts. None of the second conversion attempts resulted in a completed interview. To highlight the futility of this effort, it is worth noting that interviewers classified 40 households as hostile refusals and 38 of these were on second refusal conversion attempts. These results show fairly clearly that first refusal conversions may contribute substantially to the yield and the response rate, but second refusals are not effective in this type of cell phone survey.
SCREENING FOR CELL-ONLY HOUSEHOLDS
Of all 1,013 of the telephone numbers that were identified as residential and completed the screener, table 3 shows that 21 percent (n = 176) were classified as cell-only households. Adults were sampled only from these households. While there is no precise estimate of the percentage of California households with cell phones that were cell-only in the spring of 2006, we speculate that it was about 15 percent. This assumes 10 percent of all households are cell-only and 65 percent of all households have at least one cell phone number. Observing a higher rate of completing an interview with cell-only phone households is consistent with previous cell phone surveys. Brick et al. (2006) suggest this may occur because cell-only users are more accessible and likely to respond to calls to their cell phones than those with both types of telephone service. The higher response of cell-only households was expected, and the sample size requirements were estimated assuming that cell-only households would be over-represented in the completed interviews.
|
Once the adult was sampled, additional calls were made to complete the adult interview. Some adults were called up to 15 times, with a median of 10 call attempts. While this is a reasonable number of call attempts, it is still lower than the 23 call minimum in CHIS 2005. Refusal conversions attempted for the sampled adults were less successful than those for the screening interview. Only 2 of the 29 refusal conversion attempts resulted in a completed adult interview.
RESPONSE RATES
To compute the response rate for the screening interview, the unknown residential status numbers or e were assigned using the proportional allocation method, consistent with the AAPOR (2006) RR3 definition. The RR3 response rate for the screening interview was 28.6 percent. Since there were two sources for the sampled numbers, those cell numbers that had been sampled in the "landline" RDD sample but identified as cell prior to dialing them and those sampled from the cellular banks, the response rates for these samples were computed separately. The numbers from the RDD were more likely to be residential (69.9 percent) than the numbers sampled from the cellular banks (53.7 percent). They also had a higher screener response rate: 34.1 percent for numbers from the RDD and 27.7 percent for those from the cell phone banks1.
Because the level of effort for the cell phone sample was significantly lower than for the RDD sample, it is likely that the response rate in the study was depressed relative to CHIS 2005. The CHIS 2005 RR3 response rate for the screening interview was about 20 percentage points higher than the rate for the cell phone study. Another measure of the willingness of the sample to participate in the survey that is less sensitive to the level of effort is the cooperation rate. The cooperation rate is the number of sampled units that completed divided by the number of eligible households that were contacted, following AAPOR COOP3. For the cell phone study, the cooperation rate at the screener was 36.9 percent, which is lower than the 50.3 percent rate in CHIS 2005. We note that CHIS 2005 included a $2 prepaid incentive for households where the sampled telephone number was matched to an address, while the cell phone study offered a $5 promised incentive for the screening interview. There is consistent evidence from telephone surveys that promised incentives (such as that used in the cell phone study) have little or no effect on response rates, whereas the $2 prepaid incentive in CHIS 2005 resulted in a three percentage point increase in response rates (Edwards et al. 2006). Prepaid incentives could not be used in the cell phone study because no addresses are available for cell phone numbers in the United States.
Table 3 shows the results of the attempts to interview the sampled 176 adults in cell-only households, with one partial complete counted as a complete. The adult interview response rate for the cell phone study was 56.9 percent, which was slightly higher than the 54.0 percent adult interview response rate in the CHIS 2005. The adult interview cooperation rate in the cell phone study was 78.0 percent, compared to 72.4 percent, the rate in CHIS 2005. In CHIS 2005, there was no financial incentive for the adult interview beyond the $2 in the advance letter, while the cell phone respondents were offered a $25 promised payment. As noted earlier, promised incentives generally have little effect on response rates (see Brick et al. 2007 for an experiment with different levels in a cell phone survey) and the primary intent of the $25 payment in the study was to treat the cell respondents fairly by compensating them for their cell phone use rather than to improve the response rate.
The overall response rate is the product of the screening and adult interview rates. The overall response rate for the cell phone study was 16.3 percent, which is about 10 percentage points lower than the 26.9 percent adult overall adult response rate in CHIS 2005. The difference is due to the lower response rate in the screening interview. A 12 percentage point difference in response rates between comparable cell phone and landline samples was reported by Brick et al. (2007) and Steeh (2004). However, in those surveys all cell phone households contacted were interviewed, and adults were not sampled within the household.
LENGTH OF INTERVIEW
To understand the feasibility of conducting lengthy cell phone interviews we examined how far sampled adults, who began the interview but never completed it, progressed through the interview before stopping. As the text in the appendix shows, the sampled adults were informed that the interview was on average of about 30 minutes at the very beginning of the adult interview. Almost all of the sampled adults who began the interview but never completed the interview dropped out early, and never finished the first set of demographic items (96 percent of the nonrespondents who started did not complete this first section). This is very similar to the CHIS 2005 and other RDD surveys in that once a respondent has invested in responding to the survey, they continue to do so to completion.
In general, the cell phone study did not reveal any major differences in conducting a 30-minute interview with respondents on their cell phone than doing shorter interviews. Since only adults in cell-only households were eligible for the interview, it is possible that these findings might not be replicated in cell phone surveys that include adults in households with both cell and landline phones. The CHIS cell phone study also did not encounter operational problems even though it was introduced as a health survey, rather than a survey about cell phones or technology. The interviewers did run into standard respondent queries such as asking how we obtained their cell phone number and objections that their cell phone was private and should not have been called. While these questions did arise in the study, they were roughly of the same proportion, type, and intensity that normally are found in RDD surveys for respondents with unlisted telephone numbers.
|
Efficacy of Sampling Adults |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
In this section, we evaluate the procedures for sampling adults in terms of completing interviews. As described earlier, of the 176 households there were 53 with only one adult and 109 where the SR did not share the cell phone. The SR was sampled in all of these households. There were only 14 cases in which adults shared the cell phone and the SR was sampled in seven of these cases. In all, the SR was sampled in 169 of the 176 households. Two of these sampled adults were found to be ineligible.
Among the sampled and eligible SRs, 99 responded for a 58.9 percent adult interview response rate. For the set of sampled SRs, the response rate did not differ substantially by the number of adults in the household or whether the cell phone was shared. For the six eligible adults who were not the SR, no adult interviews were completed. This outcome raises the question about the utility of sampling adults rather than simply interviewing the SR.
While the lack of any completed interviews at first appears compelling, it is too early to conclude that sampling adults in cell-only households is unwarranted. First, with only six sampled adults, the sample size is far too small to draw definitive conclusions. Second, four of the six sampled adults who were not SRs were not completed because the sampled adult did not speak English. In the RDD sample, these non-English speaking adults may have cooperated and completed the interview. Third, even in RDD samples there is a large difference in the response rate when the SR is not the sampled adult. For example, in CHIS 2005 there is a 30 percentage point difference in the adult response rate by this characteristic (69 percent for SRs and 39 percent for those who were not SRs). This differential holds in other surveys we have conducted (e.g., see Roth, Montaquila and Chapman 2006). Therefore, a lower response rate should be expected, and should not be sufficient reason to eliminate within-household sampling.
A final reason for considering within-household sampling of adults is that it provides the opportunity to reduce bias by converting a coverage problem to a nonresponse problem. If data about the adults in the household are not obtained during the interview, then developing weighting adjustments to account for these adults who are not covered is very difficult. If a sample of adults is selected, then some characteristics of the sampled adults can be captured in the screening interview and used to adjust the weights if the adult does not respond. For example, in the study, we know that four of the six sampled adults who were not the SR, did not speak English, and this information could be used in nonresponse adjustment to reduce the bias of the estimates.
|
Weighting |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
In an earlier dual frame survey design, Brick et al. (2006)
found evidence of differential nonresponse by telephone status and usage that was not ameliorated even though several different estimation schemes were applied. They concluded that more frequent cell phone users were more likely to respond to a cell phone survey. There was also some indication that those who had landlines but rarely used them might respond at a lower rate when called on their landline. They demonstrated that these differential nonresponse rates caused severe nonresponse bias in estimates of telephone status, and showed that estimates correlated with telephone usage might also be biased. Based on the findings of Brick et al. (2006), the cell phone study for CHIS used a screening approach to avoid potential biases correlated with cell phone usage. As noted earlier, the disadvantage of this design is the additional cost associated with screening out households with a landline. Effective methods of weighting cell phone samples are in their infancy, and thus far no weighting procedures have been described for the screening design used in the CHIS cell phone study. We present some simple procedures for weighting the sample data from this type of design, but with only 99 completed cell-only interviews they cannot be adequately evaluated. In general, we consider weighting for a design in which adults in cell-only households are interviewed as a supplement to an equivalent landline RDD survey, and the primary goal is to produce estimates of the entire adult population rather than estimates for separate domains by telephone status (landline only, cell-only, and both). This seems to be the most common situation since telephone status is not likely to be a scientifically germane domain for most social and health surveys. For the screening design, we essentially weight the RDD and cell samples separately until one of the final weighting steps when the two samples are combined.
The first step in weighting is creating the base weights. For the cell sample, these can be directly computed as the ratio of the number of telephone numbers in the frame to the number of sampled telephone numbers. Since the sample was selected from two frames, the base weights differ by frame. For the sample from the Telcordia series of dedicated cell phone numbers, both the numerator and the denominator are directly available from the sampling vendor, just as they are in RDD samples. For those from the pre-processed RDD sample, the base weights from the RDD sample are brought forward. Having sample from two distinct frames introduces some variability in the weights within the overall sample that is unavoidable if the recently ported numbers are to be covered in the survey. The effect of differential weighting is discussed in more detail below.
The next step in most telephone surveys is to adjust the base weights to account for nonresponse. In the RDD sample, the nonresponse adjustment cells are determined by linking the telephone number to the geographic area assigned to those numbers, and then using census data from the areas to form cells. The specifics of the methodology as applied in CHIS are described in the weighting methodology reports for each year (www.chis.ucla.edu/methods.html) and were unaltered for this study. Johnson et al. (2006) describe a similar approach that uses these ecological data for nonresponse adjustment. With cell phones, this approach is not likely to be as effective because of the weaker association between the cell phone number and the geographic location of the owner of the cell phone (see Fleeman 2007). For this study, we adjusted by the inverse of the response rate and did not use the ecological data to form nonresponse adjustment cells.
The next step in weighting is an adjustment to account for multiple chances of selecting the person on different cell phone numbers. The adjustment is the number of cell phones from which the adult could be sampled. Given the design of this cell phone study, the multiplicity adjustment is only applied when the cell phone is shared. When the cell phone is not shared, there is a one-to-one link between the cell phone and the adult and no multiplicity adjustment is needed. This procedure also assumes that the sampled adults who do not share have only one non-business cell phone.
The adjusted base weights for the cell-only households resulting from these steps are essentially domain estimates of totals based on the inverse probability of selection weights. Estimates of the total based on this estimation scheme are known to have high variances (e.g., Kish 1965, pp. 434–38). Ratio- or poststratification-type adjustments are usually employed to reduce the variance of the estimates. For this study we considered two approaches to combining the cell phone and the RDD samples, and raking the weights to reduce the variances of the estimates. Some of the advantages and disadvantages of the two approaches are discussed below.
One method is to adjust the RDD sample and the cell-only sample to separate control totals and then combine the two. The RDD sample would be adjusted to the total for all adults in California who live in households that are not cell-only; the cell-only sample would be adjusted to the total for all adults in California who live in cell-only households. The two samples would then be merged together and raked to detailed demographic control totals for all adults in the state. The major disadvantage of this approach is that no estimates exist for the number of adults in California by telephone status. The National Health Interview Survey (NHIS) produces national estimates by telephone status, but the California sample supports only relatively imprecise state estimates by telephone status. Even nationally, the NHIS estimates by telephone status may be a bit problematic for this kind of use because they may only be available for an earlier time period, and likely would be outdated at their time of use.
For this study, we used an alternative approach. First, the data from the RDD sample and the cell-only sample were combined, each bringing its own nonresponse adjusted weights. Second, the weights from the combined dataset were raked to make the sum of the weights consistent with the adult population control totals for California. The raking dimensions were the same ones used for the regular CHIS 2005 sample. This procedure should reduce the variances of the estimates as compared to those derived from the inverse probability of selection weights. One disadvantage of the method is that estimates of adults in cell-only households are likely to be imprecise, but as noted above these types of estimates are not of primary interest in this health survey. Using these weights, the cell-only sample accounted for 8.7 percent of all adults, even though it is likely that 10 percent or more of adults were cell-only.
For national surveys, the alternative of using the NHIS estimates by telephone status as population control totals may be more attractive because the NHIS also has an item on telephone usage. This variable might be useful in reducing nonresponse bias, especially in dual frame surveys that do not screen for cell-only households. An issue that requires further research with this approach is the effect of the relatively large sampling errors for the NHIS estimated totals by telephone status.
When the weights for the cell-only adults in the combined data set were examined, they were on average 40 times larger than those from the RDD sample. The effect of this huge differential in average weights is an increase in the variance of the estimates by a factor of nearly two. This differential is due to the disproportional allocation of the sample to the RDD and cell samples. Although in this study, the cell sample was intended to test feasibility and not to produce estimates, the difference points out the importance of sample allocation.
An examination of the optimal allocation of the sample using a simple linear cost function can help avoid large penalties in the variances of the estimates due to differential weights. For example, assume the cost of conducting the survey is clnl + ccnc, where cl is the cost per completed landline interview, nl is the number of completed landline interviews, cc is the cost per completed cell-only interview, and nc is the number of completed cell-only interviews. If p is the proportion of cell-only adults in the population and the interview cost ratio is r = cc/cl, then the optimum is to allocate p(p + (1–p)r)–1 of the interviews to the cell-only sample. For example, if p =.1 and a cell-only complete cost five times as much as a landline complete, then 4.7 percent of the interviews should be with cell-only adults. Since the RDD sample had over 43,000 completed adult interviews, the allocation of 99 cell-only interviews was very far from the optimum.
The CHIS application raises another sampling and weighting issue that should be considered when designing a survey to produce estimates for different levels of geography. In CHIS, the large sample size is essential to produce estimates of sufficient precision for counties and for smaller ethnic groups. In these situations, it is generally impossible to allocate the cell-only sample proportionately to each county and ethnic group. In this case, an option is to implement weighting techniques to protect against large biases when the cell-only sample is small or zero for a particular subgroup. In CHIS, this was done by using a raking dimension specifically designed to reduce the bias due to exclusion of nontelephone and cell-only households. The dimension contained cells based on the cross-classification of home ownership, age, education, and number of adults in the household.
|
Conclusions |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
This article describes a cell phone study with several features that distinguish it from earlier attempts to survey this population. One of the differences is that the cell phone sample was screened and only adults living in cell-only households were administered the full CHIS interview. This procedure was used to avoid the nonresponse biases that might be associated with telephone usage as discussed in Brick et al. (2006)
. The screening procedure was relatively easy to implement and was successful, although there are costs associated with screening. This study also attempted to sample adults from within the household. A sampling method was developed to cover adults who live in households in which every adult does not have his or her own cell phone. Other adults in the household were sampled only if the sampled cell phone was shared. This method is a compromise in the sense that it assumes that when the cell phone is not shared, then other adults in the household could be sampled on their own cell phone. While this is not a perfect solution, it does address most of the within-household coverage problems. The study was not definitive about the ability to interview a different sampled adult when the cell phone is shared.
The study did provide evidence that conducting a cell phone interview of the same scope and length as a landline RDD interview is feasible in the United States. One of the main issues in cell phone surveys is the lower response rate to the screening interview. Once cell-only households were identified in the study, the adults participated in the interview at the same rate as in CHIS 2005 even though the effort to achieve high response rates was not as rigorous as in CHIS 2005. At least part of the reason for the relatively high level of participation by the adults in the cell phone sample is that the SR is sampled more often in a cell survey than in an RDD survey. Having a higher proportion of SRs sampled has the effect of increasing the adult response rate in the cell sample.
Weighting the cell phone sample with the RDD sample posed significant challenges. The major steps of weighting in a screening design were described. Two approaches to combining and weighting the cell-only and landline samples were outlined, along with some of the advantages and disadvantages of each method. Optimal allocation was also discussed because allocating too small a sample to the cell-only adults might inflate the variances of the estimates greatly.
While the results of this cell phone study are encouraging, there are still unresolved issues. A new cell phone study will be conducted as a part of the CHIS 2007. We hope that study provides more insight into the unresolved issues because it will be larger, have a more intensive calling protocol, and will be conducted in multiple languages. The goal of the 2007 study is to supplement the CHIS landline sample with completed interviews from a survey of cell-only households and to produce estimates that avoid the coverage bias due to the ever-increasing proportion of cell-only households. Essentially, the same telephone sampling and within-household sampling procedures described here will be implemented in the 2007 study. We have added a few more interview items when other adults in the household are sampled to support nonresponse adjustments. The sample also should be large enough to gather more definitive information about the utility of sampling adults who do not answer the cell phone.
The goal in CHIS 2007 is to complete about 800 interviews with adults living in cell-only households. Even with this relatively large sample, it is very likely that some counties or ethnic groups will be under-represented or will be absent in the cell-only sample. Two steps are being planned to address this problem. First, the sample will be allocated to each of the counties to the extent possible without introducing large differential sampling rates. Second, a raking dimension to address nontelephone and cell-only households will still be included.
Finally, in planning for the CHIS 2007, the allocation of the interviews by the type of sample was considered. The 800 cell-only interviews are expected to account for about 2.3 percent of the total number of adult interviews in the survey. This allocation might appear to be far from the optimal allocation as discussed in the previous section. However, the landline sample is allocated disproportionately to the counties, and results in a design effect of about 2.0 for state-level estimates. When effective sample sizes are substituted for nominal sample sizes in the optimal allocation formula, then the cell-only sample accounts for over 4 percent of the total. This is still less than the optimal allocation for cell-only adults, but it is close enough so that it should not increase the variability of the weights greatly.
|
Appendix |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
Screener for the Chis 2005 Cell Phone Pilot
Hello, my name is INTERVIEWER NAME. I am calling for the University of California. We are doing a scientific study about health in California. If you are currently driving a car or doing any activity that requires your full attention, I need to call you back at a later time.
It will take less than 2 minutes to see if you qualify for the study. We will send you $5 to thank you for answering these questions.
Are you at least 18 years old?
- Does an adult, 18 years or older, ever use this phone?
- Can we speak to the adult now?
- Can we speak to the adult now?
Is this cell phone your only phone or do you also have a regular telephone at home?
Compared to your regular home phone, do you...
- receive many more calls on your cell phone,
- receive somewhat more calls on your cell phone,
- receive somewhat fewer calls on your cell phone, or
- receive many fewer calls on your cell phone?
- ABOUT THE SAME IF VOLUNTEERED
- receive somewhat more calls on your cell phone,
Is this cell phone used for personal use, personal and business use, business use only?
Including yourself, how many adults (AGE 18 YEARS AND OLDER) currently live in your household?
Do any of these adults share this cell phone number?
Of the other adults in your household, how many have their/have own cell phone?
[IF SCREENER RESPONDENT NOT SELECTED]
Please tell me just the first name of the adult in this household, OTHER THAN YOURSELF, who will have the next birthday.
Extended Interview Consent Script
[FOR EXTENDED INTERVIEWS IF NOT CONTINUED FROM SCREENER]
We are doing a scientific study about health and healthcare that may help improve services in your community. This part of the study has to do with people who only use cell phones. If you are currently driving a car or doing any activity that requires your full attention, I need to call you back at a later time. Some of the numbers we are calling are for cell phones. Some people are concerned about the privacy of conversations on cell phones. If you would prefer, I would be happy to call you back on a landline phone to conduct this interview at a time that is convenient for you.
[FOR ALL EXTENDED INTERVIEWS]
The interview takes about 30 minutes on average, but may be as short as 20 minutes. There are questions about your health, your health and sexual behaviors, your healthcare, and your health insurance. I will also ask about your race, occupation, and income, to help plan and improve health services in your community. This interview is voluntary and confidential. Your answers will be used only for this survey. You can skip any question and you can stop at any time. We will send you $25 to thank you for your help with this survey.
Do you have any questions about this?
|
Footnotes |
|---|
J. MICHAEL BRICK is with Westat, 1650 Research Boulevard, Rockville, MD 20850, USA.
W. SHERMAN EDWARDS is with Westat, 1650 Research Boulevard, Rockville, MD 20850, USA.
SUNGHEE LEE is with UCLA Center for Health Policy Research, 10960 Wilshire Boulevard, Suite 1550 Los Angeles, CA 90024, USA. We would like to thank David Grant of UCLA Center for Health Policy Research, and Ismael Cervantes-Flores and Michael Jones of Westat for their contributions to conducting and analyzing this study.
1 The 16.2 percent difference (69.9–53.7) has a test statistic of t = 7.1. The 6.3 percent difference (34.1–27.7) has a test statistic of t = 3.1. Both differences are statistically significant with p values of <.01. 
|
References |
|---|
|
TOPAbstractcalifornia health interview...SamplingData CollectionEfficacy of Sampling AdultsWeightingConclusionsAppendixReferences |
|---|
American Association for Public Opinion Research (AAPOR). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys (2006) Lenexa, KS: AAPOR.
Brick J. Michael, Brick Pat Dean, Dipko Sarah, Presser Stanley, Tucker Clyde, Yuan Yangyang. "Cell Phone Survey Feasibility in the U.S.: Sampling and Calling Cell Numbers Versus Landline Numbers," Public Opinion Quarterly (2007) 71:29–33.
Brick J. Michael, Dipko Sarah, Presser Stanley, Tucker Clyde, Yuan Yangyang. "Nonresponse Bias in a Dual Frame Sample of Cell and Landline Numbers." Public Opinion Quarterly (2006) 70:780–93.
Brock Roth, Montaquila Shelley, Jill, Chapman Chris. "Nonresponse Bias in the 2005 National Household Education Surveys Program." (2006) U.S. Department of Education, Institute for Education Sciences, NCES 2007-016 (nces.ed.gov/pubs2007/2007016.pdf).
Edwards W. Sherman, Brick J. Michael, Kurata John, Grant David. "Effects on Response Rates of Multiple Sponsors on Advance Letters for an RDD Survey." (2006) Paper presented at the Annual Conference of the American Association for Public Opinion Research, Montreal, Canada.
Fleeman Anna. "Survey Research Using Cell Phone Sample: Important Operational and Methodological Considerations." (2007) Paper presented at the Annual Conference of the American Association for Public Opinion Research, Anaheim, CA.
Johnson Timothy, Holbrook Allyson, Cho Young, Bossarte Robert. "Nonresponse Error in Injury-Risk Surveys." American Journal of Preventive Medicine (2006) 31:427–36.[CrossRef][Web of Science][Medline]
Kalton Graham. "Practical Methods for Sampling Rare and Mobile Populations." Statistics in Transition (2003) 6:491–501.
Kish Leslie. Survey Sampling (1965) New York: Wiley.
Keeter Scott, Kennedy Courtney. "The Cell Phone Challenge to Survey Research." (2006) The Pew Research Center for the People and the Press. Available at http://people-press.org/reports/pdf/276.pdf.
Lavrakas Paul, Shuttles Charles. "Cell Phone Sampling Summit II Statements on Accounting for Cell Phones in Telephone Survey Research in the U.S." (2005) Available at http://www.nielsenmedia.com/cellphonesummit/statements.html.
Link Michael, Battaglia Michael, Frankel Martin, Osborn Larry, Mokdad Ali. "Conducting Public Health Surveys over Cell Phones: The Behavioral Risk Factor Surveillance System Experience." (2007) Paper presented at the Annual Conference of the American Association for Public Opinion Research, Anaheim, CA.
Steeh Charlotte. "A New Era for Telephone Surveys." (2004) Paper presented at the Annual Conference of the American Association for Public Opinion Research, Phoenix, AZ.
Tucker Clyde, Brick J. Michael, Meekins Brian. "Household Telephone Service and Usage Patterns in the U.S. in 2004: Implications for Telephone Samples." Public Opinion Quarterly (2007) 71:3–22.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||