2.1 Designing questionnaires

This section contains extensive guidelines on how to design a questionnaire. They are developed in simple headers and bullet points, which, I hope, will make this material more accessible and of practical benefit to potential users. There are many useful texts and guides to designing questionnaires, such as Newell (1993), Burns (2000), Bloom and Fischer (1982) and Kidder and Judd (1986).

Getting started

Before you start to design a questionnaire, identify its objectives. More specifically, identify what kind of information you want to obtain. Then brainstorm – write down all possible questions for incorporating in the questionnaire.

Constructing questions

This is the most difficult part of developing a questionnaire. Here are some useful rules of thumb to follow:

• Keep questions simple. Avoid ambiguous, leading, double-barrelled and hypothetical questions, double-barrelled questions being ones that ask two questions in one.
• Avoid words of more than three or four syllables and over-long sentences.
• In closed questions, allow the respondent the option of answering with ‘not appropriate’, ‘don’t know’ or ‘have no strong feelings’. This helps the respondent and avoids difficulties later in interpreting questions that have no responses.
• Avoid overly sensitive questions – you are unlikely to get a ‘true’ response.

Use of open and closed questions

Most questionnaires contain both types of question and this is advisable. Closed and open questions are appropriate in different contexts and provide different kinds of information.

Closed questions

Closed questions are questions in which all possible answers are identified and the respondent is asked to choose one of the answers. In the following example, students were asked to evaluate the quality of programme materials (handouts, etc.) by a series of five closed questions. (The questionnaire is not well designed but illustrates clearly the nature of closed questions.)

Example 1

Help us measure the success of the programme. Please tick one box for each of the questions.

Programme materials	Excellent	Good	Fair	Poor	Unable to judge
1) the availability of the materials
2) the quality of the materials
3) the durability of the materials
4) the quantity of the materials
5) the suitability of the materials for students

Source: Fitz-Gibbon and Morris (1987), p. 62.

Advantages of closed questions

• Closed questions are an appropriate means of asking questions that have a finite set of answers of a clear-cut nature. Sometimes this is factual information but closed questions are also used for obtaining data on attitudes and opinions (see ranked closed questions below).
• They oblige the respondent to answer particular questions, providing a high level of control to the questioner.
• They involve minimal effort on the part of the respondent.
• They provide uniformity of questions and student responses, so they are potentially easier for evaluating the opinion of the sample group as a whole.
• They save time. Closed questions are less time consuming for respondents to complete, and this allows the questionnaire to ask more questions.
• They avoid problems of interpreting respondents’ handwriting.
• They can provide better information than open-ended questions, particularly where respondents are not highly motivated.

Disadvantages of closed questions

• Closed questions are appropriate only when the set of possible answers are known and clear-cut.
• If poorly designed, closed questions may be misleading and frustrate respondents. Typical problems are poorly designed questions, inappropriate questions and questions that have answers other than those listed.

Closed questions with ranked answers

• These are closed questions for which answers are located on a scale of alternatives. This type of question is often used in evaluation to uncover respondents’ attitudes and opinions. The scale often represents degrees of satisfaction with a particular service or degrees of agreement with a statement.
• Always balance scales around a mid-point in the response answer. For example, respondent may choose from the following alternatives: strongly agree, agree, have no strong feelings, disagree, strongly disagree.

Advantages of ranked questions

• The advantages of using closed questions apply – see above.
• Answers can be pooled across students to derive summary statistics that potentially measure an overall degree of satisfaction/agreement.
• Summary scores can also be obtained from pooling across different questions related to some overriding issue.

Disadvantages of ranked questions

• Attitudes and opinions are complex and not readily summarised in a scale.
• Ranked questions do not provide means for students to elaborate on or explain reasons behind the stated degree of satisfaction.
• Summary statistics are powerful and if based on poorly designed questionnaires can be damaging.

The following are examples of ranked closed questions drawn from questionnaires used to evaluate teaching in anonymous economics departments.

Example 2

Fill in one response for each question.

5 = Excellent, 4 = Very Good, 3 = Satisfactory, 2 = Fair, 1 = Poor

Skill of the instructor

1) Instructor’s effectiveness as a lecturer	1	2	3	4	5
2) Clarity of instructor’s presentations	1	2	3	4	5
3) Instructor’s ability to stimulate interest in the subject	1	2	3	4	5

Example 3

For each of the following questions, please ring your answer.

The module as a whole

1.	The module stimulated my interest
	Disagree   1   2   3   4   5   Agree
2.	The module was
	Too easy  1   2   3   4   5   Too hard
3.	The module objectives were fulfilled
	Disagree   1   2   3   4   5   Agree

Example 4

This is an example of how ranked questions may be pooled to generate an overall index (from Henerson et al., 1987):

Teachers in a new experienced-based science programme filled out a questionnaire about each of several children in their classes. Here is a portion of the questionnaire:

The scores for questions 2, 3, 4 and 5 were summed to obtain ‘an enthusiasm index’ for each child, a point on a scale of 4–20. There are difficulties designing and interpreting these results, of course. We have to be sure that every question used in computing the index indeed reveals information about a student’s level of enthusiasm, and that the scales of the questions are consistent, i.e. that high enthusiasm is always indicated by scores close to or equal to 5. The greatest difficulty lies in interpretation of the final scores – usually researchers consider scores above or beneath threshold levels as revealing something definite about behaviour and attitudes, but it is difficult to know where to fix the thresholds. The alternative approach here would be to ask teachers to rate the enthusiasm of students.

Open questions

Open questions are questions that allow the respondent to answer in any way they wish. For example, students might be asked to respond to the following question: ‘What do you feel is the best thing(s) about the course?’

Advantages of open questions

• Flexibility. The respondent can answer in any way he/she wishes.
• They may be better means of eliciting true opinions/attitudes and identifying how strongly attitudes are held or not.

Disadvantages of open questions

• They require more thought and time on the part of respondent and analyst. This dramatically reduces the number of questions that the questionnaire can realistically ask.
• It is more difficult to pool opinion across the sample when questionnaires use open questions.
• Respondents may answer in unhelpful ways.

Open versus closed questions

‘. . . closed questions should be used where alternative replies are known, are limited in number and are clear-cut. Open-ended questions are used where the issue is complex, where relevant dimensions are not known, and where a process is being explored’ (Stacey, 1969).

Most questionnaires are ‘mixed’, containing both open and closed questions. This is often the best approach, avoiding an overly restrictive questionnaire and one that is too open and difficult to analyse. Open-ended questions can be used by students to elaborate on the reasons underlying their answers to the closed-form questions.

Supporting text

All questionnaires must be supported with some text. This should contain the following features:

• The purpose of the questionnaire should be communicated clearly to potential respondents.
• Where deemed appropriate, the confidentiality of responses should be assured.
• The supporting text should contain simple instructions on how to complete.
• At the end of the questionnaire it is a nice touch to thank the respondent for his/her time and consideration.

Questionnaires should be attractive

• Warm-up questions are recommended. These are questions that are simple to answer, such as questions on the age of student, year of study, degree programme, etc. Use of such questions makes it less likely that the respondent will disengage from the questionnaire.
• A good questionnaire has a coherent structure. Where possible, collect questions under definable subject areas and develop a logical order of questionnaires.
• Do not leave important questions to the end of questionnaire.
• Do not split questions over pages, or ask questions that require answers to be completed on subsequent pages.
• Do not overcrowd the questionnaire with questions and text.

Length of the questionnaire

• It is advised that questionnaires should not be too long (for obvious reasons). However, the appropriate length does depend upon the purpose of the questionnaire, the type of respondents targeted and the type of questions.
  • Appropriately chosen and designed closed questions are easy to answer, so you can have more of them.
  • For the validity of the questionnaire, it is often appropriate to include a number of questions relating to one broad issue.

Testing questionnaires

It is essential that questionnaires are thoroughly tested prior to use. Bloom and Fischer (1982) identify five key criteria that may be used in evaluating the quality of a questionnaire – these are listed and discussed below. To evaluate a questionnaire effectively, it should be tested on an appropriate sample, which, in our case, is a sample of students. Test results are analysed and any changes to the questionnaire made. After initial implementation, questionnaires should continue to be evaluated as an ongoing process.

The criteria to use in evaluating a questionnaire are:

• Purpose. In evaluating a questionnaire, one has to be absolutely clear about the purpose.
  • Often, insufficient thought is given to the purpose of a questionnaire. Designers need to identify at the outset what kinds of knowledge they are trying to obtain from the questionnaire.
• Directness. Questionnaires should be as direct as possible, i.e. they should ask questions that address as directly as possible the issues you want to evaluate.
• Utility. This criterion relates to the practicalities of implementing and using a questionnaire. Questions to consider include:
  • Is the questionnaire easy to administer, score and interpret?
  • What resources are involved in implementing the questionnaire?
• Reliability. A study is reliable if similar results would be obtained by others using the same questions and using the same sampling criteria.
  • Where questionnaires are administered at the beginning of the lecture, the sample is biased towards those students who attend lectures – of course, this bias may raise the quality of responses!
• Validity: A study is valid if it actually measures what it sets out to measure.
  • Here, much depends on the quality of the questioning.

2.2 Administering questionnaires

The key elements of the process of implementing and making successful use of questionnaires in teaching can be summarised as follows:

• Agree schedule of courses and modules to receive questionnaires.
• Prepare students.
• Administer questionnaire.
• Analyse questionnaires.
• Write summary report of questionnaire and determine plan for course improvements.
• Report to stakeholders, including students.
• Implement action plan.
• Review changes made to the course in light of questionnaire.

In this section, I discuss the administration of questionnaires, i.e. the process by which students receive and submit their questionnaires. In the subsequent sections, 2.3 and 2.4, I shall discuss the analysis of questionnaires, how results are used to improve teaching and the feedback of results to students and other stakeholders. Successful implementation of all stages of the process of evaluation requires active involvement of various individuals or groups; this is summarised in Figure 1. Lecturers are primarily responsible for administering, evaluating and acting upon the questionnaire. Students are responsible for answering the questionnaire and, together with the responsible authority within the department, for ensuring that their views are heard and acted upon.

Figure 1 Questionnaires: the process of evaluation

Administering questionnaires

A criterion for successful questionnaires is maximisation of the student response rate. There are various ways of administering questionnaires that can help in achieving this:

• Hand out paper forms at the beginning of a lecture/tutorial and allow students time to complete and collect.
  • This should take place towards the end of the course but not at times when student attendance may be relatively low, such as in the final lecture.
  • Departments may insist on a specific time that lecturers hand out questionnaires to reduce scope for lecturers to bias the response rate downwards.
• It may be useful for the department/university to make a declaration that the lecturer is responsible for administering the questionnaire, and to suggest a target response rate. This should be around two-thirds of all students registered for the course or half of all students for larger courses.
• Take time to prepare students and impress upon them that the questionnaire and the process of evaluation are important. Key things to communicate to students before they complete the forms are the purpose of questionnaires and, where appropriate, the confidentiality of all responses. The administrator should explain verbally to the class (even if they have completed similar forms before).
  • Students may be influenced to take the questionnaire more seriously if they are requested to do so by the head of department.
  • Illustrate ways in which questionnaires have been used to improve previous courses.
  • Where questionnaires are used to develop summary scores, it may be useful to demonstrate these scores and how they are derived, and to show examples of scores developed from previous questionnaires.
  • All questionnaires should contain at least a paragraph at the top of the form stressing the value of questionnaires, confidentiality of responses and a courteous request for full student co-operation.
  • Separate guidance notes to students are a useful device, perhaps in their student handbooks.
• Organise and provide the means for collection and return of questionnaires to the department.
  • Students may take responsibility for return of forms.
  • Envelopes should be provided by the department together with information on where to return the questionnaires.

2.3 Analysing the results of questionnaires

I shall assume that the questionnaires were completed and submitted for analysis in paper form. Online questionnaires are discussed in section 4.1. Here is a summary of the key stages in the process of analysing the data with useful tips – more extensive discussion follows:

• Prepare a simple grid to collate the data provided in the questionnaires.
• Design a simple coding system – careful design of questions and the form that answers take can simplify this process considerably.
  • It is relatively straightforward to code closed questions. For example, if answers are ranked according to a numerical scale, you will probably use the same scale as code.
  • To evaluate open questions, review responses and try to categorise them into a sufficiently small set of broad categories, which may then be coded. (There is an example of this below.)
• Enter data on to the grid.
• Calculate the proportion of respondents answering for each category of each question.
  • Many institutions calculate averages and standard deviations for ranked questions. Statistically, this is not necessarily a very sound approach (see the discussion on ‘evaluating data’ below).
• If your data allow you to explore relationships in the data – for example, between the perceived difficulties that students experience with the course and the degree programme to which they are attached – a simple Chi-squared test may be appropriate.
  • For a review of this test and an example, see Munn and Drever (1999) and Burns (2000) – the page references are indexed.
• You may wish to pool responses to a number of related questions. In this case, answers must conform to a consistent numerical code, and it is often best simply to sum the scores over questions, rather than compute an average score.

Preparing a grid

You will have a large number of paper questionnaires. To make it easier to interpret and store the responses, it is best to transfer data on to a single grid, which should comprise of no more than two or three sheets depending on the number of questions and student respondents. A typical grid looks like this:

	Questions
	1	2	3	4	5	6	7	8	9	10
Respondent 1
Respondent 2
Respondent 3
Respondent 4
Respondent 5

Coding data

If the answers to a question are represented on the questionnaire as points on a scale from 1 to 5, usually you will enter these numbers directly into the grid. If the answers take a different form, you may wish to translate them into a numerical scale. For example, if students are asked to note their gender as male/female, you may ascribe a value of 1 to every male response and 0 to female responses – this will be helpful when it comes to computing summary statistics and necessary if you are interested in exploring correlations in the data. It will make it much easier to analyse the data if there is an entry for all questions. To do this, you will need to construct code to describe ‘missing data’, ‘don’t know’ answers or answers that do not follow instructions – for example, if some respondents select more than one category.

Coding open questions is not straightforward. You must first read through all of the comments made in response to the open questions and try to group them into meaningful categories. For example, if students are asked to ‘state what they least like about the course’, there are likely to be some very broad themes. A number may not find the subject matter interesting; others will have difficulties accessing reading material. It may be useful to have an ‘other’ category for those responses that you are unable to categorise meaningfully.

Evaluating data

Often, it is sufficient and best simply to calculate the proportions of all respondents answering in each category. (An Excel spreadsheet is much quicker than using a calculator!) It is clear that having a category for all respondents who either don’t know or didn’t answer is very important, as it provides useful information on the strength of feeling over a particular question.

Questionnaire results are often used to compute mean scores for individual questions or groups of questions. For example, the questionnaire may ask students to rate their lecturer on a five-point scale, with 5 denoting excellent, 4 good, 3 average, 2 poor and 1 very poor. The mean score is then used as an index of the overall quality of a lecturer with high scores indicating good quality. This is not a particularly useful or legitimate approach as it assumes that you are working on an evenly spaced scale, so that, for example, ‘very poor’ is twice as bad as ‘poor’, and ‘excellent’ twice as good as ‘good’.

Often analysts add up scores over a number of related questions. For example, you may ask students ten questions related to a lecturer’s skills, all ranked from 1 to 5 with 5 indicating a positive response, and add up the scores to derive some index of the overall ability of the lecturer. Again, except in carefully designed questionnaires, this approach is inappropriate. It assumes that each question is relevant and of equal importance. Comparing scores across different lecturers and modules, this assumption is unlikely to hold. If you are interested in summative indices of quality, it may be best simply to ask the students to rate the lecturer themselves on a ranked scale.

2.4 So what? Using the results of questionnaires to improve teaching and learning

It is primarily the responsibility of the lecturer to review the responses and results of the questionnaires and these should be summarised in a summary report, which is presented to the department and to a representative student body. The key feature of the report is an ‘action plan’ indicating how the lecturer intends to act upon the findings of the questionnaire to improve the learning experience in future courses. Where no changes are envisaged, the reasons for these must be clearly stated. It is important that teachers receive some form of training in how to go about interpreting and using questionnaire results – as stated earlier, reading questionnaire responses can be a difficult process for inexperienced teachers and support should be available.

It is good practice to ensure that lecturers and tutors do not see questionnaires relating to themselves and to the modules for which they have responsibility until assessment of the module is completed. Analysis and report writing should then be done as soon as possible.

Mechanical data processing

It is possible that your questionnaire, if formatted appropriately, may be read and scored by machine, or that you can use a machine-scorable answer sheet. This can significantly reduce time involved in analysing questionnaires.