3.1 A review of questionnaires and their use in economics

I have sampled a number of questionnaires in use in economics departments in the UK and have grouped questions into the following broad categories:

• overall quality indicators;

• open questions;

• student characteristics, behaviour and status;

• the module;

• the skills of the lecturer;

• reading and facilities;

• contribution to learning.

These are discussed in turn. I try to draw out the key features, illustrating with examples of questions in use. In the subsequent section, there is a broader discussion of questionnaires in economics, containing some ideas and tips regarding best practice.

Overall quality indicators

Less than half of the questionnaires sampled include questions or statements that invite students to rate the overall quality of modules and lecturers. Asking students to rate the overall quality of the lecturer is rare. The following are examples of these kinds of question and statements drawn from the sample of questionnaires reviewed:

• Overall quality of module (excellent, …, satisfactory, …, poor).
• Overall I found this a valuable course (strongly agree, …, no strong feelings, …, strongly disagree).
• My overall rating of the lecturer (excellent, …, acceptable, …, very poor).
• Give your overall assessment of each lecturer’s contribution to the course (very good, …, satisfactory, …, very poor).
• Overall this was an excellent module (agree, …, disagree).
• Overall how satisfied were you with this module? (on a scale of seven).
• In general, how helpful were the lectures? (not at all, …, very).
• To what extent were your expectations of the module fulfilled?
• In general, how useful were the seminars?

Open questions

All questionnaires contain at least one open question, although they vary significantly in the number of open questions and the proportion of open to closed questions – the largest number of open questions used is 13. I have detailed the most common questions asked – the percentage figures refer to the proportion of sampled questionnaires containing this question or a closely related question:

• What did you like most about the module? (47%)
• What did you like least about the module? (40%)
• What improvements would you suggest? (40%)
• General comments. (67%)

Other open questions used

Here is a selection of other open questions used in economics questionnaires. Some of these are probably better dealt with as closed questions (for example, the question on the technical level of the course). One questionnaire asks what textbook(s) students have bought. In the light of increasing numbers of students and difficulties accessing library resources, this is an interesting question:

• Explain in more detail your responses to ‘the closed questions’.
• Would you recommend this module to another student?
• What advice would you give a student about to embark on this module?
• What topics do you find most/least interesting?
• Did you find the technical level of the unit too high, too low or about right?
• Was the unit content roughly the same as you had expected?
• Which textbooks (if any) did you buy?

Student behaviour and status

A small proportion of questionnaires ask questions about the students’ characteristics and behaviour. The most common question of this sort concerns student attendance at lectures and tutorials. Typically students are asked to rank their level of attendance from excellent to poor.

In some cases, students are asked whether they agree or not with the following statement:

• I attended the lectures regularly. (27%)

Students may not wish to admit a level of delinquency, so responses may be biased upwards. It might help to be more precise in the question – one questionnaire asked students:

• What proportion of lectures did you attend? (25%, 50%, 75% 100%)

Other questions/statements that measure characteristics and status of students include:

• Other than time involved attending lectures and tutorials, how many hours per week did you work on the course?

• Circle your degree programme and year of study. (27%)
  • Identifying a student’s degree course may, in small courses, appear to compromise confidentiality. It may be better to make the question optional.
• Have you studied mathematics A-level?

• What is your age?

• What grade do you expect to obtain from this module?
  • Better understanding of students’ expectations may be valuable to teachers, although a module evaluation form may not be the best place to locate questions of this sort.
• Why did you choose the module? Was it compulsory, optional …?

The module

All questionnaires contain a number of closed questions about the structure, coherence and level of the module as a whole. The key areas of concern are:

• Interest/stimulation

  • On a number of questionnaires, students are asked to respond to the following statement: ‘The course material stimulated my interest’ (strongly agree, …, strongly disagree).

• Level

  • Example: ‘The overall level of the course was about right, given my background’ (strongly agree, …, strongly disagree).

• Design and organisation

  • Example: ‘The course was well organised’ (strongly agree, …, strongly disagree).

• Clarity of course objectives

  • Example: ‘The course objectives were clearly explained at the outset’ (strongly agree, …, strongly disagree).

• Workload

  • Difficulty of material (much too difficult, …, much too easy).

  • How did the level of difficulty of the material and quantity of material compare to other courses? (much more difficult, …, much easier).

  • Quantity of work required (much too much, …, much too little).

• Consistency of content of course with course outline

One questionnaire contained a single question relating to the method of assessment. Students were asked:

• Are you happy with the means of assessment?

I think this is an important question simply because assessment is such a key and contentious area and may give rise to valuable information that can be used in the design of assessment procedures. The form of this particular question is not ideal, as it is very likely to induce a negative response. It would be more useful to ask students to suggest alternative forms of assessment, possibly in the form of an open question.

Skills of the lecturer

Questionnaires contain relatively few questions that relate directly to the qualities and skills of the lecturer. In many cases, questions relate to aspects of the module and it is open to interpretation whether this implies anything or not about the performance of the lecturer. For example, it is common for questionnaires to ask whether a module is interesting or intellectually stimulating – it is quite a different question to ask whether the lecturer seeks to make the course interesting or stimulating.

Questions relating to the skills of the lecturer cover the following broad areas:

• Presentation

  • Speed of delivery

  • Audibility

  • Communication skills

• Interest

  • Instructor’s ability to stimulate interest in the subject

• Organisation

  • Students are asked: ‘Were lectures well prepared and organised?’

• Use of and quality of visual aids, overheads and handouts

  • Examples: ‘Did the lecturer use visual aids?’, ‘Were the visual aids helpful?’

• Accessibility

  • Instructor’s availability and helpfulness to students (excellent, …, very poor)

• Feedback

  • Were your essays/assignments marked and returned promptly? (always, …, never)

  • Has the lecturer been accessible to answer questions or give advice? (yes, …, no)

Reading and facilities

Most questionnaires ask about reading material. These are typical questions:

• Did you receive helpful guidance regarding reading material?

• Was the reading material readily available?

Some questionnaires include questions about facilities. For example, students are asked about the quality of the lecture rooms and access to computing facilities:

• The computing facilities I needed for this module were adequate (agree, …, disagree).

Contribution to learning

As the objective of the courses is to promote learning, it can be useful to ask students whether they believe the course has promoted learning and the development of key skills. It is very rare for questionnaires to address these issues, but there are some questions of this sort. For example, in one questionnaire, students were asked to rate:

• Contribution of the module to improving general analytic skills (excellent, …, very poor).

The third case study in section 5 provides the most comprehensive example of a questionnaire that addresses these issues.