The Canadian Journal of Higher Education, Vol. XI-2, 1981 La revue canadienne d'enseignement supérieur, Vol. XI-2, 1981 Factors Affecting Response Rate and Response Speed in a Mail Survey of Part-Time University Students GWENNA M. MOSS* ABSTRACT This paper reports the results of an experiment to determine the effects of questionnaire format, reminder format, and followup format on both response rate and response speed in a mail survey. Complete responses were received from 2212 of a sample of 2638 parttime university students, a rate of 83.8%. Mean response time was 16.09 days. Results indicated that: (1) typeset questionnaires were more effective than photocopied questionnaires in terms of both response rate and speed; (2) the use of reminder postcards significantly increased both rate and speed; (3) there was no advantage in hand-addressed reminders over computer-produced labels; and (4) sending a replacement questionnaire as opposed to only a followup letter did not significantly increase response rate, but followup format interacted with questionnaire format in influencing response speed. RESUME Cet article expose les résultats d'une expérience menée pour déterminer les effets que peuvent avoir le format des questionnaires, le format des lettres de rappel et le format des lettres complémentaires sur le pourcentage et la vitesse de réponse à une enquête faite par voie postale. Sur un échantillon de 2638 étudiants universitaires à temps partiel, 2212 réponses complètes furent reçues, soit un taux de 83.8 pour cent. Le temps moyen des réponses fut de 16.09 jours. Les résultats indiquaient que (1) les questionnaires composés chez l'imprimeur furent plus efficaces que les questionnaires dactylographiés puis photocopiés, tant pour le pourcentage que pour la vitesse des réponses; (2) l'emploi des rappels sous forme de cartes postales augmenta de manière significative le pourcentage et la vitesse des réponses; (3) les rappels adressés à la main ne furent d'aucun avantage sur les étiquettes faites par des ordinateurs; et (4) le fait d'envoyer un questionnaire de remplacement plutôt qu'une simple lettre complémentaire n'augmenta pas de manière significative le taux des réponses, alors que le format de la correspondance complémentaire et de celui du questionnaire exercèrent une influence sur la vitesse des réponses. * Department of Continuing Education University of Saskatchewan. 8 0 Gwenna M. Moss Universities are frequently faced with the need to obtain information from their students. Because of time and money constraints the mail survey is often used. With a mail survey the university's concern lies in stimulating a high enough response rate for the results to be representative of the student population surveyed. Response speed, as distinct from response rate, should also be of interest in university surveys. Houston and Ford (1976) mentioned three values of reducing the time lag between receiving and returning a questionnaire: (1) urgently-needed results will be obtained earlier; (2) there will be less chance of external events influencing responses; (3) followup costs will be reduced if responses come in faster. This study reports on a mail survey of the University of Saskatchewan's part-time students. It was designed to test the effects of the following factors on both response rate and response speed: questionnaire format (two typeset versions versus photocopying); reminder format (no reminder versus hand-addressed reminders versus computer-produced labels); and followup format (letter only versus letter plus replacement questionnaire). These factors were selected for investigation primarily because the evidence on their effects is inconclusive in regard to either response rate or response speed or both. Additionally, they all have implications for survey budgets. Typesetting, reminder postcards, and replacement questionnaires cost money; handwritten address labels take additional time and/or money. If these factors significantly increase either response rate or response speed or both, their use may be justified; otherwise researchers on limited budgets may prefer to spend their money elsewhere. PREVIOUS RESEARCH Questionnaire Format Despite the fact that the investigation of questionnaire format goes back a number of years, there are surprisingly few studies reporting systematic investigations of that factor. As early as 1940 Sletto was experimenting with different formats but his report did not provide complete information on his results (Sletto, 1940). Scott (1961) reported a survey of British motorcyclists which compared three letterpress versions of a questionnaire to three duplicated versions. Very high response rates were obtained with all versions and the differences were not significant. Brown (1965) reported a study of physicians which compared a two-page questionnaire with a postcard questionnaire. The postcard version drew 15% more responses initially, but after a followup mailing this advantage disappeared. In addition, data from the postcard format were less complete. This was the only study to refer to response speed and format. Ford(1968),inaconsumershoppingsurvey,compared a printed, folded questionnaire to a stapled, mimeographed questionnaire. Response rates were 22% and 20% respectively, a non-significant difference. Horowitz and Sedlacek (1974) used three versions of a one-page combination cover letter and questionnaire in a survey of college professors. Response rates were not significantly different for typed (73.3%), mimeographed (63.3%), and photocopied versions (71.6%). The authors noted that "the generalizability of these results to other populations and other content and length of questionnaires remains open to further empirical study" (p. 364). Reminder Format Reminder postcards, sent to all respondents shortly after the initial mailing, serve both as a thank-you to those who have completed the questionnaire and as a followup reminder 81 Factors Affecting Response Rate & Response Speed in a Mail Survey to those who have not yet done so. Nichols and Meyer (1966) demonstrated that reminder postcards mailed 3 days after the questionnaire significantly increased both response rate and response speed with college students. Wiseman (1973) found that reminder postcards increased response rates by 8%. Cox, Anderson, and Fulcher (1974) obtained contradictory results; the use of postcards had no significant impact on either response rate or response speed. The reminder postcard in this study was also designed to test the effects of a personalized versus an impersonal reminder. Personalization, as it is operationalized in research on mail questionnaires, may include one or more of a variety of techniques including handwritten versus facsimile signature, personal versus impersonal salutation, titled versus untitled signature, handwritten postscripts urging reply versus no postscript, individually typed letters versus form letters, identified questionnaire versus anonymous questionnaire, use of telephone calls versus no calls. The effects on response rates are usually positive (Carpenter, 1974; Cox et al., 1974; Dillman & Frey, 1974;Kerin, 1974; Kerin & Harvey, 1976). The few exceptions appear to reflect more a fear of loss of anonymity and confidentiality than the effects of personalization per se (Houston & Jefferson, 1975; Simon, 1967). Two studies used definitions of personalization which were similar to the present study. Anderson and Berdie (1975) tested the effect of handwritten addresses versus typed address labels on reminder postcards in a survey of university faculty, administrators, and students. While the total group did not respond differently, undergraduate response was higher for the hand-addressed followup than for the typed label version. Kahle and Sales (1978) reported a study of psychologists and clinical psychologists in which preprinted address labels decreased the response rate (by approximately 11%) as compared to individually typed addresses. The authors suggested that the respondent sees preprinted address labels as indicating the letter is from a computer, not a person, and speculated that fewer letters may be opened. Fewer investigators have studied the impact of personalization on response speed. Dillman and Frey (1974) found that personalization increased response speed, but only slightly. Cox et al. (1974) found that "the treatments involving personalization accumulate replies at a much quicker rate than do the treatments without personalization" (p. 416). Followup Format There is ample evidence to show that followups have a great impact on response rate. One decision the researcher faces is whether to send a duplicate questionnaire with the followup letter. Linsky noted in 1975 that "the advantage of including a new questionnaire with the followup letter does not seem to have been clearly tested in any of the studies reviewed" (p. 87). However, Dillman (1972) presented data from two independent surveys of the general public which suggested that including a duplicate questionnaire increased response speed. Etzel and Walker (1974) found no significant increase in response rate through enclosing a duplicate questionnaire. METHOD Sample The present study was conducted in the spring of 1979 using a random sample of students registered in the six part-time degree programs at the University of Saskatchewan during the 1978-79 academic year. The six programs included part-time day, evening, off-campus, 8 2 Gwenna M. Moss correspondence, intersession, and summer session. Duplicate registrants were removed and a final sample of 2688 students was included in the study. Mailing The questionnaire was designed to obtain students' assessments of the University's services for students in part-time degree programs. Questions on the students' educational background, experience, and plans for the future, and demographic questions were also included. All questionnaires were mailed in white envelopes using computer-printed address labels. The cover letter with black, preprinted signature was typed and photocopied on University letterhead. A stamped return envelope was enclosed. Experimental Treatments The experimental treatments were as follows: Questionnaire format. Three versions of the questionnaire were prepared: 1. Photocopied. The questionnaire was typed on an IBM Selectric typewriter, photoreduced to 74%, and photocopied on 8V2" by 11" paper. Two pages with questions on both sides were stapled in the upper léft-hand corner. 2. Typeset stapled. This version of the questionnaire was identical to the first version except that it was typeset. 3. Typeset folded. This version of the questionnaire was the most "professional-looking." It was produced using typeset on a single sheet of 17" by 11" paper which was then folded in the centre to produce a one-fold 8V2" by 11" booklet. All versions were on white bond, were precoded, and used identical wording. Reminder format. Half the respondents received a reminder postcard mailed 1 week after the initial mailing of the questionnaire. The postcard thanked those who may have already returned the questionnaire and encouraged the others to respond. It included a telephone number to be called to request a replacement questionnaire. One-third of the reminder postcards were addressed by hand; the computer-printed address labels were used on the remainder. The labels gave the student's name, with surname first, and the student's number. Followup format. Three weeks after the initial mailing, students who had not responded (N = 1049) were sent a followup letter urging them to complete the questionnaire. Half received the letter only and half received the letter with a replacement questionnaire. The letter was, as before, photocopied on letterhead with a preprinted black signature. The computer-produced address labels were used on the envelopes. Students received the same version of the questionnaire they had initially received. A final followup letter was sent to all non-respondents (N = 656) 2 weeks after the first followup letter. Response Rate Removing ineligible respondents and non-deliverable questionnaires reduced the number of potential respondents to 2638. Responses were received from 2235 students, a rate of 84.7%. However, 23 of these responses were incomplete. The data reported in this article are based on the 2212 students (83.8%) who returned complete questionnaires. As the questionnaires came in they were stamped with the date of receipt. The number of days which elapsed from initial mailing to receipt of the completed questionnaire was used to calculate response speed. 83 Factors Affecting Response Rate & Response Speed in a Mail Survey Table 1 Number and Percentage o f Respondents By Questionnaire Format and Reminder Fbrmat Questionnaire format3 Reminder format13 Photocopied Typeset stapled Typeset folded Total No reminder 81.0% (360) 84.2% (373) 77.3% (344) 80.8% (1077) Computer label 84.8 (247) 91.0 (263) 86.1 (249) 87.3 (759) Handwritten 86.9 (127) 84.8 (123) 86.3 (126) 86.0 (376) Total 83.3 (734) 86.5 (759) 81.7 (719) 83.8 (2212) NOTE: In Tables 1 and 2 the top number in each cell is the percentage of questionnaires returned and the number in parentheses is the number returned. ®X2 = 7.879, £ <.025 b X 2 = 18.190, £ <.001 Analysis To determine the effect of the factors on response rate a partitioned chi-square analysis was used as described by Winer (1971, pp. 855-9). To analyze the effects on response speed, two-way and three-way analysis of variance were used. Two sets of analyses are presented: the effects of questionnaire and reminder formats on response rate and response speed for the total sample, and the effects of questionnaire, reminder, and followup formats on response rate and speed for that portion of the sample which received the first followup (N = 1049). The .05 level of significance was used. 1 RESULTS Response Rate Both questionnaire format and reminder format had an impact on response rate, as shown in Table 1. For questionnaire format, the highest return rate was achieved by the typeset stapled version (86.5%), followed by the photocopied format at 83.3%. The lowest rate was achieved by the more professional-appearing typeset folded version (81.7%). These results, which are significant, differ from other studies which generally found no differences due to format. The use of reminder postcards increased the response rate as expected, but there was no advantage to handwritten addresses over computer-produced address Table 2 Nimber and Percentage of Respondents by Questionnaire Format, Reminder Format, and Followup Format Questionnaire formata Reminder format13 Photocopied Lettere Letter & Quest Typeset stapled Letter & Quest Letter Typeset folded Total Letter Letter & Quest Letter Grand Total Letter & Quest No reminder 56.9% (62) 61.5% (56) 63.1% (60) 64.1% (59) 53.7% (58) 60.5% (69) 57.7% (180) 61.9% (184) 59.8% (364) Computer label 65.1 (28) 57.4 (31) 66.7 (32) 83.7 (41) 61.7 (29) 70.6 (36) 64.0 (89) 70.1 (108) 67.5 (197) Handwritten 66.7 (16) 58.6 (17) 58.3 (14) 64.0 (16) 66.7 (18) 68.4 (13) 64.0 (48) 63.0 (46) 63.5 (94) Total 60.2 (106) 59.8 (104) 63.5 (106) 69.8 (116) 57.7 (105) 64.1 (118) 60.4 (317) 64.5 (338) Grand Total 60.0 (210) «P = 3.782, £ >.10 ^ =5.066, £ >.05 °F = 1.905, £ >.10 66.6 (222) 60.9 (223) 62.4 (655) 85 Factors Affecting Response Rate & Response Speed in a Mail Survey •fable 3 Mean Response Time i n Days By Questionnaire Fbrmat and Reminder Format Questionnaire format3 Reminder format*3 Photocopied Typeset stapled Typeset folded Total No reminder 19.35 15.63 17.09 17.34 Computer label 16.24 14.40 13.80 14.80 Handwritten 16.70 13.69 14.90 15.11 Total 17.85 14.89 15.57 16.09 NOTE: The numbers in each cell are the same as in Table 1. ®F = 8.248, £ <.001 b F = 7.649, £ <.001 labels. The rates were 80.8% for no reminder, 87.3% for computer labels, and 86% for handwritten addresses — differences which are significant at the .001 level. There was no interaction effect. Table 2 shows the effects of questionnaire, reminder, and followup format on those students who had not responded 3 weeks after the initial mailing (N = 1049). Half of this group received a followup letter and the other half received a letter plus a replacement questionnaire; 62.4% returned questionnaires. Neither questionnaire format nor reminder format made a difference to response rate, although both had an effect on the response rate of the total sample as shown in Table 1. Differences in response rate were in a similar direction in the subsample but were not large enough to be significant. The third factor — followup format — did not significantly affect response rate; the rate for letter only was 60.4% as compared to 64.5% for letter plus questionnaire. Thus the additional expense of printing and mailing a replacement questionnaire did not pay off in a significantly higher response rate. None of the interactions was significant. It should be noted that the followup letter encouraged respondents to phone and request a replacement questionnaire if they had discarded or mislaid the original. A number of students did request a replacement and this may have increased the response rate from the letter only group. Response Speed The mean response time for the total group (N = 2212) was 16.09 days, as measured by the number of days from initial mailing to receipt of the completed questionnaire. Both questionnaire format and reminder format had significant effects on response speed, as Table 4 Mean Response Time i n Days By Questionnaire Format, Reminder Format, and Follcwup Format Questionnaire formata Photocopied 3 Reminder format' Letterc No reminder 44.55 Computer label Typeset folded Grand Total Total Letter Letter & Quest Letter Letter & Quest Letter Letter & Quest 33.36 35.68 31.27 34.60 32.39 38. .39 32.33 35.32 37.96 33.65 32.34 26.15 27.03 27.39 32. .38 28.71 30.37 Handwritten 35.69 29.53 32.00 33.00 31.11 31.15 32. .90 31.20 32.06 Total 41.47 32.82 34.19 29.70 31.91 30.73 35. ,87 31.02 Grand Total NOTE: 37.19 Letter & Quest Typeset stapled 31. .84 The numbers in each cell are the same as in Table 2. ^ = 9.866, £ <.001 = 7.066, g =.001 °F = 15.956, 2 <-001 31.29 33.37 87 Factors Affecting Response Rate & Response Speed in a Mail Survey shown in Table 3. The two typeset versions were returned faster than the photocopied version - means of 14.89 days for the typeset stapled version, 15.57 days for the typeset folded format, and 17.85 days for the photocopied version. The use of a reminder postcard increased speed of response but it made little difference whether the reminders were addressed by hand (15.11 days) or with a computer label (14.80 days). Both were more than 2 days faster than the group which did not receive a reminder (17.34 days). The interaction between questionnaire format and reminder format was not significant, meaning that the effects were additive. Thus, the use of both typesetting and a reminder increased speed of response by 5 to 6 days. The additional effects of followup format are shown in Table 4. The grand mean for respondents who were sent a followup was 33.37 days, reflecting the fact that the followup was mailed if no response had been received 3 weeks after the initial mailing. The effects of questionnaire format and reminder format were similar for the subsample as for the total sample. The photocopied version had a longer response time than both typeset versions. Response time was also longer for those who did not receive a reminder postcard; hand addressed reminders were just slightly slower than computer addressed reminders. A faster response speed was obtained by sending a replacement questionnaire with the followup letter — 31.02 days versus 35.87 days for the letter alone. All these differences were statistically significant. Only one of the four interaction terms was significant. This interaction between questionnaire format and followup format indicated that a replacement questionnaire increased response speed by 9 days for photocopied questionnaires, but did not make as much difference for the two typeset versions. One possible explanation is that photocopied questionnaires are more apt to be discarded than typeset questionnaires, thus requiring additional time for the respondent to request a replacement. With this exception the effects of the three factors on response speed are roughly additive, ranging from a high of 44.55 days (for the photocopied, no reminder, no replacement questionnaire condition) to lows of 26.15 days (for the typeset stapled, computer label, replacement questionnaire condition), and 27.39 days (for the typeset folded, computer label, replacement questionnaire condition). DISCUSSION Based on these results what recommendations can be made in regard to questionnaire format, reminder format, and followup format? Assuming it is desirable to maximize both response rate and response speed then the following recommendations are made: Questionnaire Format Typesetting is more effective than photocopying in terms of both response rate and response speed. With this group of respondents the most professional-looking questionnaire (typeset folded) did not have as high a response rate as the typeset stapled version, although there was little difference between the two versions in terms of response speed. Based on these results the typeset stapled version is recommended. Reminder Format There is no doubt that the use of a reminder postcard sent to all respondents has a payoff in terms of a higher response rate and a shorter response time. Its use is recommended. 88 Gwenna M. Moss However, it appears that there is no advantage to be gained by hand-addressing such reminders. Neither response rate nor response speed is improved by using hand-addressed cards over computer-produced labels, at least on the one reminder. In fact, with both typeset versions, computer labels yielded as fast or faster responses than hand-addressed reminders. Whether similar results would be obtained with a population not so accustomed to receiving mail using computer-produced labels is an open question. Followup Format A recommendation in regard to using a followup letter only or a letter plus a replacement questionnaire is less clearcut. The response rate was not significantly increased by sending a replacement questionnaire (60% versus 64%), but response speed was significantly faster (31 days versus nearly 36 days). Based on that alone one might recommend the use of a replacement questionnaire. However, the interaction between questionnaire format and followup format clouds the issue. With typeset versions there is less to be gained by sending a replacement questionnaire. With a photocopied questionnaire the use of a replacement questionnaire is certainly merited, since in this study it resulted in a mean saving of over 8 days. On balance, unless money is a major limiting factor, it may well be desirable to send a replacement questionnaire. SUMMARY Perhaps the most significant implication of these findings for universities is that it is possible to obtain high response rates from students without the added expense of personalized approaches. The overall response rate of 83.8% was certainly respectable. The highest rate — 91% — was achieved at a moderate cost with a typeset stapled questionnaire and a computer-addressed reminder postcard. This combination also results in an acceptable mean response time of about 2 weeks. The added expense of a replacement questionnaire is an optional courtesy, with no significant impact on response rate, although it does shorten response time. FOOTNOTE 1. The tables presented in this article are abbreviated. Complete results can be obtained from the author. REFERENCES Anderson, J.F., & Berdie, D.R. Effects on response rates of formal and informal questionnaire followup techniques. Journal of Applied Psychology, 1975, 60, 2 5 5 - 2 5 7 . Brown, M.L. Use of a postcard query in mail surveys. Public Opinion Quarterly, 1965, 29, Carpenter, E.H. Personalizing mail surveys: a replication and reassessment. Public Opinion 1 9 7 4 , 3 8 , 614-620. 635-637. Quarterly, Cox, E.P., III, Anderson, W.T., Jr., & Fulcher, D.G. Reappraising mail survey response rates. Journal of Marketing Research, 1 9 7 4 , 1 1 , 4 1 3 - 4 1 7 . Dillman, D.A. Increasing mail questionnaire response in large samples of the general public. Public Opinion Quarterly, 1 9 7 2 , 3 6 , 254-257. Dillman, D.A., & Frey, J.H. Contribution of personalization to mail questionnaire response as an element of a previously tested method. Journal of Applied Psychology, 1974, 59, 2 9 7 - 3 0 1 . Etzel, M.J., & Walker, B.J. Effects of alternative follow-up procedures on mail survey response rates. Journal of Applied Psychology, 1 9 7 4 , 5 9 , 219-221. 89 Factors Affecting Response Rate & Response Speed in a Mail Survey Ford, N.M. Questionnaire appearance and response rates in mail surveys. Journal Research, 1968,8, 4345. of Advertising Horowitz, J.L., & Sedlacek, W.E. Initial returns on mail questionnaires: a literature review and research note. Research in Higher Education, 1974, 2, 361-367. Houston, M.J., & Ford, N.M. Broadening the scope of methodological research on mail surveys. Journal of Marketing Research, 1 9 7 6 , 1 3 , 3 9 7 4 0 3 . Houston, M.J., & Jefferson, R.W. The negative effects of personalization on response patterns in mail surveys. Journal of Marketing Research, 1 9 7 5 , 7 2 , 114-117. Kahìe, L.R., & Sales, B^p. Personalization of the outside envelope in mail surveys. Public Quarterly, 1 9 7 8 , 4 2 , 5 4 7 - 5 5 0 . Opinion Kerin, R.A. Personalization strategies, response rate and response quality in a mail survey. Science Quarterly, 1 9 7 4 , 5 5 , 175-181. Social Kerin, R.A., & Harvey, M.G. Methodological considerations in corporate mail surveys: a research note. Journal of Business Research, 1916,4, 277-281. Linsky, A.S. Stimulating responses to mailed questionnaires: a review. Public Opinion 39, 82-101. Quarterly, Nichols, R.C., & Meyer, M.A. Timing postcard follow-ups in mail-questionnaire surveys.Public Quarterly, 1 9 6 6 , 3 0 , 3 0 6 - 3 0 7 . Scott, C. Research on mail surveys. Royal Statistical Society Sletto, R.F. Pretesting of questionnaires. American Principles in Experimental Opinion Journal Series A, 1 9 6 1 , 1 2 4 , 143-205. Simon, R. Responses to personal and form letters in mail surveys. Journal 1967, 7, 28-30. Winer, B.J. Statistical 1975, Sociological Review, of Advertising Research, 1 9 4 0 , 5 , 193-200. Design (2nd ed.). N e w York: McGraw-Hill, 1971. Wiseman, F. Factor interaction effects in mail survey response rates. Journal of Marketing 1 9 7 3 , 1 0 , 330-333. Research,