The Promises and Perils of Pushing Respondents to the Web in Mixed-Mode Surveys

(1)

The Promises and Perils of Pushing Respondents to the Web in Mixed-Mode Surveys

For

9^th Colloque Francophone sur les sondages Gatineau, Quebec, Canada

11 au 14 Octobre 2016

By

Don A. Dillman, Regents Professor Department of Sociology and the Social and

Economic Sciences Research Center

Washington State University, Pullman, WA 99164-4014 [email protected]

(2)

A brief look to the past

• In 1989 an iconic American movie, Field of Dreams, was created around the idea that if a baseball field were to be built in a rural Iowa corn field, “They (the players and

people) will come.”

• In the late 1990’s a similar idea developed about

conducting surveys by email with responses sent over the Internet.

• Hopefully, thepeople would not only come, but the cost of surveying would decline dramatically, making it a

surveyor’s field of dreams.

(3)

Three parts to this presentation

• Part I. The decline of telephone and difficulty with finding alternatives.

• Part II. A decade of research aimed at developing an effective web-push

methodology.

• Part III. Specific challenges associated with the promises and perils of web-push surveying.

c Don A. Dillman, September 1, 2016 3

(4)

My specific focus

• My focus in this talk is on probability sampling of the general public.

• In this context I am also focused on simultaneously reducing four types of error: coverage, sampling, nonresponse and measurement.

• By web-push I am referring to initially asking people to respond only over the web, while reserving alternative response modes until later in the data collection process.

• Mixed-mode refers to using more than one survey mode

as contact and/or response mode.

(5)

Part I.

The Decline of Telephone and difficulty with finding

alternatives

(6)

Around 2005 it became clear that the voice telephone era of RDD household surveys

was coming to an end

• The presenting problems for RDD surveys were:

– Low response rates.

– Need to combine individual (cell) and household (land) lines.

– Need for brevity.

– Need to devote questions to correcting for area code transportability, ownership of cell/landline access, not driving a vehicle, age of

respondent.

– People answered phone only once, so that it is becoming a “one chance to persuade” (in 5 seconds or less) methodology

• However, cultural change is the biggest problem; the telephone is no longer used for ordinary conversations.

• Most telephone communication is asynchronous; texts and emails.

(7)

The hoped for Email contact/web

response replacement, is not yet working well.

• Coverage for household populations remains limited (<80%), some on slow connections.

• Individual internet skills were limited among some adults, especially the less educated, older and lower income individuals.

• No email sample frame for household selection comparable to telephone RDD.

• Email only contact produces response rates similar to phone.

(8)

The hoped for Email contact web response replacement, is not yet working well (2).

• Lack of TRUST on multiple levels; sponsorship,

misrepresentation of purpose, threat of malware, and consequences of making a mistake.

• We are moving from age of desktops, to laptops, (and now) to devices, which are not optimally designed for answering surveys—screen size affects when and how they get used.

• The “purse and pocket” problem combined with the

smartphone response situation of people being “on the

move” from one task to another makes getting lengthy and

thoughtful survey responses very difficult.

(9)

I am uncomfortable with some of the assessments and proposed solutions

• “Response rates no longer matter”

(Higher the response, the better our chance of going after non-responders)

• “All response modes are facing declining response rates”

(less true for mail than telephone or the Internet)

• “Repair the telephone”

(Too many problems: coverage, brevity, extra questions, minimal opportunity to persuade, CULTURE moving away from voice exchanges)

• “Give households the Internet and rely on web-only panels”

(Too many people don’t trust or are uncomfortable with

computers and internet; results are biased)

• “Give people a choice of modes”

(Most choose mail)

(10)

Postal addresses have emerged as sample frame with most complete household

coverage

• Early focus (2007) was on drawing area-probability samples for personal interviewing.

• I am struck by the high coverage rates for households—98 %—in the U.S. Postal Service “Delivery Sequence File” available as addresses without names.

• If this could work for personal interview samples, why not mail surveys.

• Reason for my interest in mail surveys was:

– I could deliver incentives with the survey request (much more effective that post payments) – I could deliver follow-up contacts in different forms to offer additional arguments for

responding.

– Research done in the 90’s at the Census Bureau convinced me that we didn’t need names for mail methods to be effective.

– I had not seen evidence of a precipitous decline in mail response rates as happened for telephone.

(11)

In order to replace telephone with a web- push methodology…

• We needed a better sample frame than “random email

addresses”. Postal mail appeared to be the only realistic choice.

• Using postal addresses would require mixing survey modes:

– a different contact mode than response mode.

– And might involve obtaining responses from more than one mode (e.g.

mail or telephone)

– Mixing modes could introduce measurement issues

• Potential cost savings provided a powerful motivation for finding ways to make this work.

• Acceptance by the survey methodology community could be challenge because of losing “interviewer assistance”, our

standard way of eliciting answers to survey questions.

(12)

Part II.

A decade of research aimed at developing an effective web-

push methodology

(13)

The Prototype (Lewiston-Clarkston) Experiment

• Address-based sample.

• Push-to-web with mail contact (send mail response option in third contact).

• Undo old mail strategies, e.g. name personalization, envelopes, use of postcard.

• Token cash incentives with response request

• Unified mode construction to reduce measurement differences.

• Tailor to population with graphic design and color.

• Reasonable length: At least 20 minutes of questions

• My background model was, “What I thought could be applied to the American Community Survey”

(14)

Conceptually, I was trying to undo earlier mail procedures (1978- 2000) and use the visual design and mixed-mode concepts

introduced in (2000 and 2009) versions of book. Each revision was substantial (see% below). New concepts are coming at us faster and

faster

1978 2009

~75% 2014

2000 ~65%

~95%

(15)

c Don A. Dillman, September 1, 2016

2007 Lewiston-Clarkston Study

• 51 numbered questions, 90 responses, 10 pages (paper)

—a 20 minute (if it were telephone) conversation

• Four contacts.

1. Pre-notice letter.

2. Questionnaire (or web request).

3. Thank-you post card.

4. Replacement questionnaire (adjusted by treatment).

• $5 token cash incentive included with mail questionnaire or web contact

(Later studies would change number and nature of contacts)

15

(16)

We compared four treatments

1.

Mail preference with web mention: Send mail questionnaire and mention web with initial request

2.

Push-to-mail: Send mail questionnaire but withhold mention of web for about two weeks

3.

Push-to-web: Web invitation with no mail questionnaire, but explain that mail questionnaire will be sent in about two

weeks

4.

Equal preference: It is your choice!

(17)

Larger plain envelopes (even for web-only request) to get them opened

• Normal business stationery more likely to be ignored.

• Used a return label showing the photo from survey cover and the survey title to increase familiarity.

17 c Don A. Dillman, September 1, 2016

(18)

Correspondence tailored to city address

• All letters used WSU

stationary (legitimation).

• $5 affixed to letter (to get it read)

• Photo of questionnaire cover used to tie

different elements

together (interest).

(19)

A picture of the Lewiston and Clarkston region identified the survey area (confluence of the

Snake and Clearwater Rivers)

19

(20)

Additional pictures identified the survey area

• Photos taken of local landmarks, artwork, and symbols to make survey recognizable and

visually attractive.

(21)

Mail Questionnaire tailored with broad survey topic, who should respond, and back-page pictures

Use of tailored images to help connect respondents to survey and to place an emphasis on study area instead of on survey source. 90 responses requested

21

(22)

Web questionnaire was similarly tailored

• Used an entry page similar to front cover of paper survey, still focusing on making the survey recognizable through familiar images.

(23)

Web survey followed unified mode construction principles to link modes

• Question 2

– Similar design format to paper survey, and use of familiar image in upper left-hand corner of the screen. Emphasized respondents over sponsors (using other communications to do that).

23

(24)

Second example of unifying web presentation with mail presentation

 Question 13

 The small image would change every ten questions. Trying to show all of the same images as the paper survey, as well as keep interest and familiarity.

(25)

Mail (on left) and web (on right) unified to control measurement

(26)

Initial withholding of mail drove 41% to the web; paper follow-up added 14%. Offering

choice drove ~80% to mail  .

Treatments Web (%) Paper(%) Total (%)

Mail preference with web

mention 4 58 62

Push-to-Mail

(web in third contact) 1 70 71

Push-to-web

Mail questionnaire sent in 3

^rd

of 4 contacts

41 14 55

Equal preference (choice) 13 50 63

(27)

High response rates are desirable but we need also to focus on nonresponse error

• It does not help much to improve response rates if our respondents are different from non-respondents on variables important to the study objectives

• Thus, we need to compare respondent characteristics on web vs. mail within the different treatment groups.

• If respondents are different from non-respondents in

ways that affect results of the data analysis, then we have a significant problem

(28)

In the push-to-web treatment, web and mail respondents

were quite different

0 10 20 30 40 50 60 70 80 90 100

* * * *

*

(29)

But the complete push-to-web group was quite similar to the complete

push-to-mail treatment

*

* p ≤ .05

29 0

10 20 30 40 50 60 70 80 90 100

Push-to-web Push-to-mail

(30)

Initial success encouraged us to do four more large-scale tests to evaluate the web- push methodology

0 Lewiston, ID-Clarkston, WA Mail Test 2005 1. Lewiston, ID-Clarkston, WA Survey 2007 2. Washington Community Survey 2008

3. Washington Economic Survey 2009

4. WA, PA, AL Tri-state Electricity Survey2011 5. WA and NE Water Management Survey2012

• Needed multiple topics and populations

• Research goal was to refine through subtraction and addition experiments, e.g. token cash incentives, other state populations, questionnaire design, respondent selection, location of sponsor, etc.)

• ~ 28 additional experiment treatments, with successful panels being carried forward.

For summarized detail, see Chapter 11 of Dillman, D., Smyth, J.D., Christian, L.M.

(31)

Tests 2-5; Changes in burden, topic, and procedures (e.g. no pre-notice, no postcard reminder)

• Questionnaire— 20-30 minutes to complete.

(equivalent to ten 8 ½ x 11” pages of paper. Up to 140 individual questions

• General Implementation Protocol : specifics varied somewhat across studies.

Week 1: Postal request that includes $4- $5 token cash incentive with request to respond by web.

Week 2: A thank you/reminder letter (not postcard).

Week 5: Another letter request with a mail questionnaire and, in some cases, an additional smaller token cash incentive

Week 6: A final thankyou/reminder.

(32)

2008 Washington Community Survey (WCS): first statewide test

• Example of the mail version:

(33)

A $5 cash incentive (WCS) more effective for web-push than paper-only

$5 Mail Mail w/o $5

0 10 20 30 40 50 60

39,2

Mail

$5 Internet Internet w/o $5 0

10 20 30 40 50 60

31,3

13,4

(+17.9*)

(+13.3*)

* p ≤ .05 c Don A. Dillman, September 1, 2016 33

(34)

Push-to-web had lower total response, but 2/3 of responses came over the web. For push-to-mail, a

late web request had very small effect.

10 20 30 40 50 60

31,3

51,9 15,0

3,1

56,7 46%

55% 57%

(35)

Introducing a paper questionnaire in third

contact brought in different kinds of respondents than did the initial push-to-web effort. (Messer

and Dillman, 2011)

Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)

0 10 20 30 40 50 60 70 80 90

Web Mail follow-up

* *

* * *

*

* p ≤ .05

(36)

The combined push-to-web group (web plus mail respondents) was demographically

similar to the mail-only treatment

Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)

0 10 20 30 40 50 60 70

(37)

Complete push-to-web group was more representative than the web-only

respondents(comparison to U.S. Census Bureau’s American Community Survey)

Education (HS or less)Children in HH # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)

5 15 25 35 45 55 65 75

Web-only Push-to-web ACS

(38)

2009 statewide economic survey, tailored to state with map and pictures

• 11” X 8.5” booklet with ‘personalized’ images to help respondents

feel connected to survey and to place an emphasis on study area

instead of on survey source. Also used similar color and design to

connect to web survey. (Messer and Dillman, 2012)

(39)

2011 Three state Electricity Survey, with tailored covers

• Examples of the mail covers:

(40)

2012 water management survey:

tailored to state by pictures and sponsorship

• Examples of the mail covers:

(41)

Response rates for 10 statewide web-push (mean, 43%, with 62% over the web) vs. mail out/mail back (mean, 53%) surveys,

2007-2012

(Dillman, Smyth and Christian, 2014, Chapter 11).

(42)

Does mail item non-response negate gains in response rates?

• If mail higher item non-response rates then perhaps that cancels out the benefits of

gaining the additional responses in web-push designs.

• Thus, it was important to evaluate this ^{. (See}

Messer, Edwards and Dillman, Survey Practice, 2012).

(43)

Web vs. mail item non-response in the push- to-web groups for LCS, WCS, and WES

• Item nonresponse rates lower for web

2007 LCS 2008 WCS 2009 WES

0 2 4 6 8 10 12 14

2,7 2,7

6,2 6,9 6,1

11,6

(44)

BUT, push-to-web and mail-only groups have similar overall item nonresponse rates; Late mail

respondents in push-to-web groups are probably less capable respondents(older with less education).

0 2 4 6 8 10 12 14

3,6 4,2

8,0

5,0 4,2

8,1

Push-to- web

(45)

Summary of Major findings

1. 10 Postal-only response = 53% (38% to 71%).

2. 10 Push-to-web response = 43%(31% to 55%).

3. Offering initial choice pushes most people (~80%) to mail.

4. Web-push produces about 60% of responses over the web.

5. Token cash incentives with request significantly improves web and overall response rates.

(46)

Summary of Major findings (2)

1. Significant differences exist between web and mail responders (education, age, income, marital status).

2. Web+mail treatment respondents are quite similar to all mail-only treatment respondents.

3. Item non-response to mail is a less of a problem than we had anticipated when visual design principles are applied.

4. Response decreases when surveying in distant states.

5. Web-push costs slightly more per respondent than mail-only

because of lower response rates.

(47)

Web-push methods being

used for many national surveys

• 2015 Japanese Census.

• American Community Survey (3.5million households per year)

• Screen and collect data for U.S. child education survey.

• Screen and collect household data for U.S. child health survey.

• 2016 Australian Census

• 2020 U.S. Census

(48)

Part III.

Additional challenges

associated with the promises

and perils of web-push surveys

(49)

Promises and Perils of Web-Push

• In the long run, we need to push respondents to the web, but we need to do it with eyes

wide-open.

• There are significant promises to be realized.

• There are also some perils in going this direction that we need to contemplate

(50)

12 ways Web-push exhibits promise

1 Address-based sampling provides superior coverage.

2 Response rates to web-push are dramatically higher than RDD telephone.

3 Cost reductions seem likely.

4 Quicker responses from “most” people seems likely.

5 A better fit with society’s “do-it-myself” culture.

(51)

Web-push exhibits promise (2)

6 Reduction in socially desirable answers (vs. interviews) 7 Reduction in extremeness of scalar responses.

8 More effective branching (vs. mail)

9 Initial mail contact increases feelings of trust.

10 Pre-incentives improve likelihood of response.

11 Two-phase data collection may be collapsed into one phase (screen, plus longer questionnaire).

12 Better fit with organizational preferences and pressures.

(52)

Web-push perils calling for research and innovation

1 How to overcome “irritation” of not providing immediate choice.

2 Pre-incentives needed for getting people to transfer across modes (i.e. mail contact to web response).

3 Decisions on communication access have switched from organizational control to individual preference and behavior.

4 Pocket/Purse devices are dominant (and perhaps only) interface some people have with computers.

5 Web response is not sufficient for most surveys—bias towards higher education, income and younger households with kids.

6 Mail follow-up to get balance faces constraints—branching plus

need for unified mode construction.

(53)

Additional web-push perils calling for attention (2)

7 Mixing of response modes are likely to create measurement differences, if allowed to ask questions in different ways.

8 Switching to mail in 3rd or 4

^th

contact is not

being handled well, e.g. still emphasizing web, minimal incentive and/or follow-up.

9 Not developing coordinated or mutually

supportive communications focused on the process of responding

(54)

Additional web-push perils calling for attention (3)

10 Not treating mail and web as help-mates. Respondents can use mail to think through answers before entering them on web, but different construction makes that difficult.

11 Are we on a slippery slope because of do it yourself software?

Let’s use only people contactable by email Let’s only obtain web responses

Increase sample size because of low response

Treating large sample sizes as if they have sampling error.

Use of weighting to create data for underrepresented categories.

A catch, release, and catch-again mentality develops.

(55)

Some additional steps

• Introducing more effective mail follow-up to the initial web- push strategy, e.g. multiple contacts with additional

incentives to get those communications read.

• Dealing with unified design between mail follow-up and smartphones that are requiring redesign of laptop/desktop surveys.

• Dealing with branching issues in visual design of paper surveys.

• Is there a role for telephone follow-up despite social desirability and aural vs. visual design effects?

• There is a lot more that can and needs to be done.

(56)

Selected references

1. Smyth, J.D., Dillman, D.A., Christian, L.M., & O’Neill, A. (2010). “Using the Internet to survey small towns and communities: Limitations and

possibilities in the early 21st century.” American Behavioral Scientist 53:

1423-1448.

2. Dillman, D.A., Smyth, J.D., Christian, L.M. 2014. Internet, Phone, Mail and Mixed-Mode Surveys; The Tailored Design Method 4th edition. (John

Wiley Co Hoboken, NJ)

3. Messer, Benjamin L. and Don A. Dillman. 2011. “Surveying the General Public Over the Internet Using Address-Based Sampling and Mail Contact Procedures.” Public Opinion Quarterly 75(3):429-57.

4. Millar, Morgan M. and Don A. Dillman. 2011. Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly 75 (2): 249-269 5. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth. 2014. An

Experimental Test of the Effects of Survey Sponsorship on Internet and

(57)

Selected references, page 2

6. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth. 2014. An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3): 734-750.

7. Messer, Benjamin L., Michelle L. Edwards, & Don A. Dillman. (2012).

“Determinants of Web & Mail Item Nonresponse in Address-Based Samples of the General Public.” Survey Practice, April:.

http://wwww.surveypractice.org

8. Messer, Benjamin L. 2012. “Pushing households to the web: Results from Web+Mail experiments using address based samples of the general public and mail contact procedures.” Ph.D. Dissertation. Washington State

University, Pullman.

8. Edwards, Michelle L. 2013. “Measuring Public Perceptions of Water

Governance in Nebraska and Washington.” Ph.D. Dissertation. Washington State University, Pullman.

(58)

Selected references page 3

• Dillman, Don A. and Michelle L. Edwards. In Press. Chapter 17. Designing a Mixed-Mode Survey. Sage Handbook of Survey Methodology. Sage Publications Wolf, Joye, Smith and Fu. Thousand Oaks. CA

• Dillman, Don A., Feng Hao, Morgan M. Millar. In Press. Chapter 15. Improving the Effectiveness of Online Data Collection by Mixing Survey Modes. In Fielding, Nigel,

Raymond M. Lee and Grant Blank. The Sage handbook of Online Research Methods, 2^nd edition. Sage Publications, London.

• Harter, Rachel, Battaglia, Michael P., Buskirk, Trent D., Dillman, Don A., English, Ned, Mansour Fahimi, Frankel, Martin R., Kennel, Timothy, McMichael, Joseph, McPhee, Cameron Brook, Montaquila, Jill, Yancey, Tracie, Zukerberg, Andrew L. 2016. Address- base Sampling. American Association for Public Opinion Research Task Force Report 133 pp.

http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Repor

(59)

Thank you!

For additional information contact:

[email protected] 133 Wilson-Short Hall Washington State University

Pullman, WA 99164-4014 [email protected]

509-335-4150