3 Questionnaires in evaluating teaching and learning in economics

3.1 A review of questionnaires and their use in economics

I have sampled a number of questionnaires in use in economics departments in the UK and have grouped questions into the following broad categories:

  • overall quality indicators;

  • open questions;

  • student characteristics, behaviour and status;

  • the module;

  • the skills of the lecturer;

  • reading and facilities;

  • contribution to learning.

These are discussed in turn. I try to draw out the key features, illustrating with examples of questions in use. In the subsequent section, there is a broader discussion of questionnaires in economics, containing some ideas and tips regarding best practice.

Overall quality indicators

Less than half of the questionnaires sampled include questions or statements that invite students to rate the overall quality of modules and lecturers. Asking students to rate the overall quality of the lecturer is rare. The following are examples of these kinds of question and statements drawn from the sample of questionnaires reviewed:

  • Overall quality of module (excellent, …, satisfactory, …, poor).
  • Overall I found this a valuable course (strongly agree, …, no strong feelings, …, strongly disagree).
  • My overall rating of the lecturer (excellent, …, acceptable, …, very poor).
  • Give your overall assessment of each lecturer’s contribution to the course (very good, …, satisfactory, …, very poor).
  • Overall this was an excellent module (agree, …, disagree).
  • Overall how satisfied were you with this module? (on a scale of seven).
  • In general, how helpful were the lectures? (not at all, …, very).
  • To what extent were your expectations of the module fulfilled?
  • In general, how useful were the seminars?

Open questions

All questionnaires contain at least one open question, although they vary significantly in the number of open questions and the proportion of open to closed questions – the largest number of open questions used is 13. I have detailed the most common questions asked – the percentage figures refer to the proportion of sampled questionnaires containing this question or a closely related question:

  • What did you like most about the module? (47%)
  • What did you like least about the module? (40%)
  • What improvements would you suggest? (40%)
  • General comments. (67%)

Other open questions used

Here is a selection of other open questions used in economics questionnaires. Some of these are probably better dealt with as closed questions (for example, the question on the technical level of the course). One questionnaire asks what textbook(s) students have bought. In the light of increasing numbers of students and difficulties accessing library resources, this is an interesting question:

  • Explain in more detail your responses to ‘the closed questions’.
  • Would you recommend this module to another student?
  • What advice would you give a student about to embark on this module?
  • What topics do you find most/least interesting?
  • Did you find the technical level of the unit too high, too low or about right?
  • Was the unit content roughly the same as you had expected?
  • Which textbooks (if any) did you buy?

Student behaviour and status

A small proportion of questionnaires ask questions about the students’ characteristics and behaviour. The most common question of this sort concerns student attendance at lectures and tutorials. Typically students are asked to rank their level of attendance from excellent to poor.

In some cases, students are asked whether they agree or not with the following statement:

  • I attended the lectures regularly. (27%)

Students may not wish to admit a level of delinquency, so responses may be biased upwards. It might help to be more precise in the question – one questionnaire asked students:

  • What proportion of lectures did you attend? (25%, 50%, 75% 100%)

Other questions/statements that measure characteristics and status of students include:

  • Other than time involved attending lectures and tutorials, how many hours per week did you work on the course?
  • Circle your degree programme and year of study. (27%)
    • Identifying a student’s degree course may, in small courses, appear to compromise confidentiality. It may be better to make the question optional.
  • Have you studied mathematics A-level?
  • What is your age?
  • What grade do you expect to obtain from this module?
    • Better understanding of students’ expectations may be valuable to teachers, although a module evaluation form may not be the best place to locate questions of this sort.
  • Why did you choose the module? Was it compulsory, optional …?

The module

All questionnaires contain a number of closed questions about the structure, coherence and level of the module as a whole. The key areas of concern are:

  • Interest/stimulation

    • On a number of questionnaires, students are asked to respond to the following statement: ‘The course material stimulated my interest’ (strongly agree, …, strongly disagree).

  • Level

    • Example: ‘The overall level of the course was about right, given my background’ (strongly agree, …, strongly disagree).

  • Design and organisation

    • Example: ‘The course was well organised’ (strongly agree, …, strongly disagree).

  • Clarity of course objectives

    • Example: ‘The course objectives were clearly explained at the outset’ (strongly agree, …, strongly disagree).

  • Workload

    • Difficulty of material (much too difficult, …, much too easy).

    • How did the level of difficulty of the material and quantity of material compare to other courses? (much more difficult, …, much easier).

    • Quantity of work required (much too much, …, much too little).

  • Consistency of content of course with course outline

One questionnaire contained a single question relating to the method of assessment. Students were asked:

  • Are you happy with the means of assessment?

I think this is an important question simply because assessment is such a key and contentious area and may give rise to valuable information that can be used in the design of assessment procedures. The form of this particular question is not ideal, as it is very likely to induce a negative response. It would be more useful to ask students to suggest alternative forms of assessment, possibly in the form of an open question.

Skills of the lecturer

Questionnaires contain relatively few questions that relate directly to the qualities and skills of the lecturer. In many cases, questions relate to aspects of the module and it is open to interpretation whether this implies anything or not about the performance of the lecturer. For example, it is common for questionnaires to ask whether a module is interesting or intellectually stimulating – it is quite a different question to ask whether the lecturer seeks to make the course interesting or stimulating.

Questions relating to the skills of the lecturer cover the following broad areas:

  • Presentation

    • Speed of delivery

    • Audibility

    • Communication skills

  • Interest

    • Instructor’s ability to stimulate interest in the subject

  • Organisation

    • Students are asked: ‘Were lectures well prepared and organised?’

  • Use of and quality of visual aids, overheads and handouts

    • Examples: ‘Did the lecturer use visual aids?’, ‘Were the visual aids helpful?’

  • Accessibility

    • Instructor’s availability and helpfulness to students (excellent, …, very poor)

  • Feedback

    • Were your essays/assignments marked and returned promptly? (always, …, never)

    • Has the lecturer been accessible to answer questions or give advice? (yes, …, no)

Reading and facilities

Most questionnaires ask about reading material. These are typical questions:

  • Did you receive helpful guidance regarding reading material?

  • Was the reading material readily available?

Some questionnaires include questions about facilities. For example, students are asked about the quality of the lecture rooms and access to computing facilities:

  • The computing facilities I needed for this module were adequate (agree, …, disagree).

Contribution to learning

As the objective of the courses is to promote learning, it can be useful to ask students whether they believe the course has promoted learning and the development of key skills. It is very rare for questionnaires to address these issues, but there are some questions of this sort. For example, in one questionnaire, students were asked to rate:

  • Contribution of the module to improving general analytic skills (excellent, …, very poor).

The third case study in section 5 provides the most comprehensive example of a questionnaire that addresses these issues.

3.2 Better practice in questionnaire design and use in economics

This section contains a review of the design and use of questionnaires in a sample of economics departments in the United Kingdom. I have identified various features of these that I believe are worth highlighting and which may be of use to other departments in designing or modifying their questionnaires.

The questionnaires reviewed have some common features. All but one of the questionnaires reviewed contain a number of ‘closed’ questions that require respondents to provide answers on a ranked scale of 1–5. Typically, students are asked to express a degree of agreement/disagreement with a series of statements. In some cases, students are asked to rate specific features of a course on a five-point scale from ‘excellent’ to ‘very poor’. All questionnaires contain some ‘open’ questions (or statements) that invite comments. The most common questions of this type are ‘What do you like least about the module?’ and ‘What do you like best about the module?’ All questionnaires provide space for ‘further comments’, giving students flexibility to say what they wish about the course or lecturer. Otherwise, there is significant heterogeneity in design of questionnaires, especially in the extent to which the attributes of individual lecturers are evaluated and in the use of closed questions.

Here are some observations and ideas that are worth flagging up:

1. Some of the questionnaires are much more attractive than others, in the formatting and layout of the page(s). As stated in section 3, it is always worth making a document as appealing as possible, as this will affect the response rate and the quality of the responses. A few forms use colour as background and in some of the text. Departments are always going to struggle to make students make the effort to complete forms in a useful way and touches like this can help.

2. A small proportion of questionnaires ask for information about the characteristics and behaviour of the respondent. For example, in one questionnaire, students are asked to state their degree programme, age (within specified bands) and year of study. A number of questionnaires contain question(s) about the students’ attendance at lectures. For example, students are asked to respond to the statement ‘I attended lectures regularly.’

Questions of this sort can significantly increase the usefulness of the questionnaire by revealing relationships between the characteristics or background of students and the types of responses and comments they make. For example, information on the degree programme of the respondent can be especially useful in interpreting responses to questionnaires on core modules that attract large numbers of students from a variety of degree programmes and departments. The heterogeneity of students is a growing problem that all departments have to face – a particular problem for economics departments in this regard concerns the issue of mathematics and its use in courses attended by students from other degree programmes who do not have a mathematical background.

The motivation and value of questioning students’ attendance at lectures and tutorials is not clear. The answer may have an impact on how seriously the lecturer and department view the student’s responses – it may subtly influence the student’s approach to answering the questions.

3. All questionnaires should contain a paragraph or two at the top of the form explaining the purpose of the questionnaire. It is important to stress that the forms are confidential and stress to the students the constructive purpose of the questionnaire process and that it helps to improve teaching and learning for students. It is also important to explain to the students how the results of the questionnaires are analysed and disseminated. For example, it is common that a summary report and action plan are presented to a student representative committee – making this clear on the form can help to convince students that their views will be considered seriously and raise the quality of response. Most forms contain a request for students to be honest and candid in their responses.

4. The length and nature of forms vary dramatically, from four questions in one case to 27 in another. However, none exceeds two pages in length, which is relatively short for questionnaires.

5. As discussed in section 2.1, one of the questionnaires relies exclusively on open questions – the form contains the following three questions:

  • What were the best features of the module?

  • Where could improvements be made in the module?

  • Are there any other comments you wish to make?

The questions invite discussion of the positive features of teaching and learning and, unusually, contain no questions with ranked answers. There is a means for students to express their opinions freely and express criticisms but there is no real attempt to evaluate teaching formally or to draw out specific concerns.

6. Questionnaires vary in the extent to which they directly assess the individual qualities and attributes of lecturers and tutors. In one example, students are asked to comment on the ability of the lecturer to communicate, his/her knowledge of the subject, whether the lecturer can be contacted easily and his/her level of preparedness for lectures. This is not usual, however. In most questionnaires, questions relate more to the characteristics of the module. For example, it is common to ask students to respond to the following statements:

  • The lectures were clear and understandable.

  • The lectures increased my understanding of the subject.

  • The lectures were interesting.

It is not straightforward to know how to interpret responses to these kinds of question, and unfavourable responses do not necessarily imply anything about the qualities or efforts of the teacher. For example, if students respond that lectures are not interesting and do not increase their understanding, this may be due to the nature of the topic and the match of the module with their background and interests, or it may be due to the lecturer’s presentation and use of the material. It follows that it is probably best to include at least one question about the lecturer’s qualities.

7. One tutorial evaluation questionnaire was divided into two parts and the first part asks questions about the ‘tutorials’. Students were asked whether they considered the tutorials to be valuable and stimulating, whether the tutorials were relevant and whether students learnt from tutorials. The second part asked questions about the quality of the tutor. Students were asked about the tutor’s command of the subject, ability to communicate, accessibility and so on. Often, weaknesses in tutorials can be attributed to fundamental problems in the structure, content and methodology of the tutorial – issues that often are out of the control of the tutor. This approach can usefully distinguish between such problems and problems related to the tutor him or herself.

8. A number of questionnaires ask respondents to identify ways in which the module may be improved. This is a useful question as it most directly relates to the purpose of the questionnaire.