The Economics Network

Improving economics teaching and learning for over 20 years

2.1 Designing questionnaires

This section contains extensive guidelines on how to design a questionnaire. They are developed in simple headers and bullet points, which, I hope, will make this material more accessible and of practical benefit to potential users. There are many useful texts and guides to designing questionnaires, such as Newell (1993), Burns (2000), Bloom and Fischer (1982) and Kidder and Judd (1986).

Getting started

Before you start to design a questionnaire, identify its objectives. More specifically, identify what kind of information you want to obtain. Then brainstorm – write down all possible questions for incorporating in the questionnaire.

Constructing questions

This is the most difficult part of developing a questionnaire. Here are some useful rules of thumb to follow:

  • Keep questions simple. Avoid ambiguous, leading, double-barrelled and hypothetical questions, double-barrelled questions being ones that ask two questions in one.
  • Avoid words of more than three or four syllables and over-long sentences.
  • In closed questions, allow the respondent the option of answering with ‘not appropriate’, ‘don’t know’ or ‘have no strong feelings’. This helps the respondent and avoids difficulties later in interpreting questions that have no responses.
  • Avoid overly sensitive questions – you are unlikely to get a ‘true’ response.

Use of open and closed questions

Most questionnaires contain both types of question and this is advisable. Closed and open questions are appropriate in different contexts and provide different kinds of information.

Closed questions

Closed questions are questions in which all possible answers are identified and the respondent is asked to choose one of the answers. In the following example, students were asked to evaluate the quality of programme materials (handouts, etc.) by a series of five closed questions. (The questionnaire is not well designed but illustrates clearly the nature of closed questions.)

Example 1

Help us measure the success of the programme. Please tick one box for each of the questions.

Programme materials Excellent Good Fair Poor Unable to judge
1) the availability of the materials          
2) the quality of the materials          
3) the durability of the materials          
4) the quantity of the materials          
5) the suitability of the materials for students          

Source: Fitz-Gibbon and Morris (1987), p. 62.

Advantages of closed questions

  • Closed questions are an appropriate means of asking questions that have a finite set of answers of a clear-cut nature. Sometimes this is factual information but closed questions are also used for obtaining data on attitudes and opinions (see ranked closed questions below).
  • They oblige the respondent to answer particular questions, providing a high level of control to the questioner.
  • They involve minimal effort on the part of the respondent.
  • They provide uniformity of questions and student responses, so they are potentially easier for evaluating the opinion of the sample group as a whole.
  • They save time. Closed questions are less time consuming for respondents to complete, and this allows the questionnaire to ask more questions.
  • They avoid problems of interpreting respondents’ handwriting.
  • They can provide better information than open-ended questions, particularly where respondents are not highly motivated.

Disadvantages of closed questions

  • Closed questions are appropriate only when the set of possible answers are known and clear-cut.
  • If poorly designed, closed questions may be misleading and frustrate respondents. Typical problems are poorly designed questions, inappropriate questions and questions that have answers other than those listed.

Closed questions with ranked answers

  • These are closed questions for which answers are located on a scale of alternatives. This type of question is often used in evaluation to uncover respondents’ attitudes and opinions. The scale often represents degrees of satisfaction with a particular service or degrees of agreement with a statement.
  • Always balance scales around a mid-point in the response answer. For example, respondent may choose from the following alternatives: strongly agree, agree, have no strong feelings, disagree, strongly disagree.

Advantages of ranked questions

  • The advantages of using closed questions apply – see above.
  • Answers can be pooled across students to derive summary statistics that potentially measure an overall degree of satisfaction/agreement.
  • Summary scores can also be obtained from pooling across different questions related to some overriding issue.

Disadvantages of ranked questions

  • Attitudes and opinions are complex and not readily summarised in a scale.
  • Ranked questions do not provide means for students to elaborate on or explain reasons behind the stated degree of satisfaction.
  • Summary statistics are powerful and if based on poorly designed questionnaires can be damaging.

The following are examples of ranked closed questions drawn from questionnaires used to evaluate teaching in anonymous economics departments.

Example 2

Fill in one response for each question.

5 = Excellent, 4 = Very Good, 3 = Satisfactory, 2 = Fair, 1 = Poor

Skill of the instructor

1) Instructor’s effectiveness as a lecturer 1 2 3 4 5
2) Clarity of instructor’s presentations 1 2 3 4 5
3) Instructor’s ability to stimulate interest in the subject 1 2 3 4 5

Example 3

For each of the following questions, please ring your answer.

The module as a whole

1. The module stimulated my interest
Disagree   1   2   3   4   5   Agree
2. The module was
Too easy  1   2   3   4   5   Too hard
3. The module objectives were fulfilled
Disagree   1   2   3   4   5   Agree

Example 4

This is an example of how ranked questions may be pooled to generate an overall index (from Henerson et al., 1987):

Teachers in a new experienced-based science programme filled out a questionnaire about each of several children in their classes. Here is a portion of the questionnaire:

The scores for questions 2, 3, 4 and 5 were summed to obtain ‘an enthusiasm index’ for each child, a point on a scale of 4–20. There are difficulties designing and interpreting these results, of course. We have to be sure that every question used in computing the index indeed reveals information about a student’s level of enthusiasm, and that the scales of the questions are consistent, i.e. that high enthusiasm is always indicated by scores close to or equal to 5. The greatest difficulty lies in interpretation of the final scores – usually researchers consider scores above or beneath threshold levels as revealing something definite about behaviour and attitudes, but it is difficult to know where to fix the thresholds. The alternative approach here would be to ask teachers to rate the enthusiasm of students.

Open questions

Open questions are questions that allow the respondent to answer in any way they wish. For example, students might be asked to respond to the following question: ‘What do you feel is the best thing(s) about the course?’

Advantages of open questions

  • Flexibility. The respondent can answer in any way he/she wishes.
  • They may be better means of eliciting true opinions/attitudes and identifying how strongly attitudes are held or not.

Disadvantages of open questions

  • They require more thought and time on the part of respondent and analyst. This dramatically reduces the number of questions that the questionnaire can realistically ask.
  • It is more difficult to pool opinion across the sample when questionnaires use open questions.
  • Respondents may answer in unhelpful ways.

Open versus closed questions

‘. . . closed questions should be used where alternative replies are known, are limited in number and are clear-cut. Open-ended questions are used where the issue is complex, where relevant dimensions are not known, and where a process is being explored’ (Stacey, 1969).

Most questionnaires are ‘mixed’, containing both open and closed questions. This is often the best approach, avoiding an overly restrictive questionnaire and one that is too open and difficult to analyse. Open-ended questions can be used by students to elaborate on the reasons underlying their answers to the closed-form questions.

Supporting text

All questionnaires must be supported with some text. This should contain the following features:

  • The purpose of the questionnaire should be communicated clearly to potential respondents.
  • Where deemed appropriate, the confidentiality of responses should be assured.
  • The supporting text should contain simple instructions on how to complete.
  • At the end of the questionnaire it is a nice touch to thank the respondent for his/her time and consideration.

Questionnaires should be attractive

  • Warm-up questions are recommended. These are questions that are simple to answer, such as questions on the age of student, year of study, degree programme, etc. Use of such questions makes it less likely that the respondent will disengage from the questionnaire.
  • A good questionnaire has a coherent structure. Where possible, collect questions under definable subject areas and develop a logical order of questionnaires.
  • Do not leave important questions to the end of questionnaire.
  • Do not split questions over pages, or ask questions that require answers to be completed on subsequent pages.
  • Do not overcrowd the questionnaire with questions and text.

Length of the questionnaire

  • It is advised that questionnaires should not be too long (for obvious reasons). However, the appropriate length does depend upon the purpose of the questionnaire, the type of respondents targeted and the type of questions.
    • Appropriately chosen and designed closed questions are easy to answer, so you can have more of them.
    • For the validity of the questionnaire, it is often appropriate to include a number of questions relating to one broad issue.

Testing questionnaires

It is essential that questionnaires are thoroughly tested prior to use. Bloom and Fischer (1982) identify five key criteria that may be used in evaluating the quality of a questionnaire – these are listed and discussed below. To evaluate a questionnaire effectively, it should be tested on an appropriate sample, which, in our case, is a sample of students. Test results are analysed and any changes to the questionnaire made. After initial implementation, questionnaires should continue to be evaluated as an ongoing process.

The criteria to use in evaluating a questionnaire are:

  • Purpose. In evaluating a questionnaire, one has to be absolutely clear about the purpose.
    • Often, insufficient thought is given to the purpose of a questionnaire. Designers need to identify at the outset what kinds of knowledge they are trying to obtain from the questionnaire.
  • Directness. Questionnaires should be as direct as possible, i.e. they should ask questions that address as directly as possible the issues you want to evaluate.
  • Utility. This criterion relates to the practicalities of implementing and using a questionnaire. Questions to consider include:
    • Is the questionnaire easy to administer, score and interpret?
    • What resources are involved in implementing the questionnaire?
  • Reliability. A study is reliable if similar results would be obtained by others using the same questions and using the same sampling criteria.
    • Where questionnaires are administered at the beginning of the lecture, the sample is biased towards those students who attend lectures – of course, this bias may raise the quality of responses!
  • Validity: A study is valid if it actually measures what it sets out to measure.
    • Here, much depends on the quality of the questioning.