1 Introduction

1.1 Summary of the chapter: objectives and key results

The aim of this chapter is to provide some practical advice on the design and implementation of questionnaires to evaluate teaching and learning in economics. The structure of the chapter is as follows:

  • The next section, 1.2, offers some Top Tips: key ideas as to good practice in the administration and analysis of questionnaires and their use in evaluation.
  • Section 1.3 discusses in a general way the role and purpose of questionnaires, identifying the particular strengths of questionnaires in comparison to other means of evaluation.
  • Section 2 identifies the key stages in implementation of questionnaires from the initial design stage to the process of using questionnaire results to improve teaching practice. Each stage is discussed in turn. This section draws largely although not entirely from generic literature.
  • Section 3 reviews the practice of questionnaire implementation in a random sample of (anonymous) economics departments in the UK, in light of the good practice guidelines discussed in section 2. As stated, the aim of the chapter is to provide practical advice that can support departments in the design and refinement of their evaluation procedures. Where there are potentially useful questions or procedures, these are identified and reproduced in detail.
  • Section 4 discusses the use of electronic questionnaires, the frequency of questionnaires and issues related to confidentiality of questionnaire responses.
  • Section 5 reproduces substantial parts of three questionnaires that have interesting features.

Key ideas and tips on good practice are concisely summarised, sometimes in note form, using bullet points.

Summary results – some thoughts on questionnaires and staff morale

Questionnaires and their use in academic departments are a controversial issue. Questionnaires typically contain ranked questions that are used to measure the perceived quality of specific aspects of a module and its teaching staff. Where the scores are low, this has potential to be extremely damaging to the morale (and possibly to the careers) of staff. In addition, most questionnaires contain ‘open’ questions that allow students some freedom to express their opinions about a module or tutorial programme. In a minority of cases, this is used irresponsibly and lecturers have been subjected to personal abuse. More generally, in their comments, students tend to focus on negative aspects of a module or its staff and do not necessarily evaluate the module according to the appropriate criteria, i.e. the extent to which it supports and facilitates learning.

In the way that we design and particularly in the ways that we use questionnaire results, we need to be aware of these issues. This is discussed fully in the subsequent sections, but a number of key points emerge. First, staff and students need to be clear as to the purpose of questionnaires – questionnaires comprise part of a multifaceted process whose goal is constructively to support teachers in making improvements in teaching and learning, where appropriate. They are not a mechanism for assessing the performance of members of staff, and should not be used in that way.

The practice of comparing scores across staff is totally inappropriate, and it should be made clear to staff that questionnaire results will not be used in this way. As suggested, scores are sensitive to non-appropriate criteria, and have been shown to be highly correlated to factors outside of the control of the teaching staff, such as the type of module, the background, level and year of the students, whether the module is optional or core, and exactly when in the module the questionnaire is implemented.

It is standard practice for students to submit their responses to questionnaires anonymously. It is argued that this approach increases the rate and quality of response. In this chapter, it is suggested that departments might consider relaxing the confidentiality of questionnaires, and oblige or request students to put their name to at least some of their responses. It is argued that anonymity may induce disingenuous responses that ultimately threaten the whole process and the objective of improving the teaching and learning experience. Positive effects of removing anonymity are that students are encouraged to articulate their concerns and ideas in a constructive and open manner, and there is a basis for dialogue and feedback after the questionnaire is submitted.

As stated, the purpose of questionnaires is to improve teaching and learning. To achieve this, teachers should receive some possibly informal training in how to read, interpret and respond to questionnaire responses. This is particularly relevant to inexperienced staff.

1.2 Top Tips: key ideas as to good practice

  • The purpose of questionnaires is to support teachers in making improvements in teaching and learning.
    • Questionnaires are not a mechanism for assessing the performance of members of staff.
    • Departments should not compare scores across staff.
  • Teachers should receive some kind of instruction in how to interpret and respond to questionnaire responses.
  • Open and closed questions elicit different kinds of information and most questionnaires should contain both.
    • Closed questions are efficient mechanisms for gleaning information about a range of specific issues.
    • Open questions allow students the freedom to discuss what matters most to them and to elaborate on answers provided to closed questions.
    • It may be useful to comprise questionnaires of two separate and detachable sections – for example, two A4 sheets. The first contains closed and ranked questions and is submitted anonymously; the second contains open questions and students are requested to identify themselves with these responses.
  • There is nothing wrong with ranked questions that elicit responses on an ordered scale – for example, from 1 to 5 – although they have to be used appropriately.
    • In analysing them the useful statistics are the proportion of respondents responding in each category.
    • Constructing average scores (i.e. averaging the scores for each question across all respondents) is not a sound statistical approach.
    • Computing single scores from a pool of questions is fraught with difficulties and can only work in an extremely well-designed questionnaire with a precise objective used in the right way.
  • Before designing the questionnaire, think carefully about what kinds of information might be useful – too many questionnaires contain questions that are inappropriate and this is frustrating to the respondents.
  • On the questionnaire, group questions into themes – this makes it more comprehensible and attractive to respondents. Questions can be grouped under the following key themes (these are discussed in more depth in section 3.1):
    • overall quality indicators;
    • open questions;
    • student behaviour and status;
    • the module;
    • skills of the lecturer;
    • reading and facilities;
    • contribution to learning.
  • Make the questionnaire attractive – this will increase the rate and quality of response.
  • When analysing questionnaire responses, do not read too much into the results.

1.3 What is the purpose of questionnaires?

As a mechanism for obtaining information and opinion, questionnaires have a number of advantages and disadvantages when compared with other evaluation tools. The key strengths and weaknesses of questionnaires are summarised in bullet points below. In general, questionnaires are effective mechanisms for efficient collection of certain kinds of information. They are not, however, a comprehensive means of evaluation and should be used to support and supplement other procedures for evaluating and improving teaching.

Advantages of questionnaires

  • They permit respondents time to consider their responses carefully without interference from, for example, an interviewer.
  • Cost. It is possible to provide questionnaires to large numbers of people simultaneously.
  • Uniformity. Each respondent receives the identical set of questions. With closed-form questions, responses are standardised, which can assist in interpreting from large numbers of respondents.
  • Can address a large number of issues and questions of concern in a relatively efficient way, with the possibility of a high response rate.
  • Often, questionnaires are designed so that answers to questions are scored and scores summed to obtain an overall measure of the attitudes and opinions of the respondent.
  • They may be mailed to respondents (although this approach may lower the response rate).
  • They permit anonymity. It is usually argued that anonymity increases the rate of response and may increase the likelihood that responses reflect genuinely held opinions.

Disadvantages of questionnaires

  • It may be difficult to obtain a good response rate. Often there is no strong motivation for respondents to respond.
  • They are complex instruments and, if badly designed, can be misleading.
  • They are an unsuitable method of evaluation if probing is required – there is usually no real possibility for follow-up on answers.
  • Quality of data is probably not as high as with alternative methods of data collection, such as personal interviewing.
  • They can be misused – a mistake is to try to read too much into questionnaire results.