In this section, the key stages of implementing a questionnaire are discussed. In section 2.1, I discuss best practice in the design of questionnaires – examples are used to illustrate where appropriate. Section 2.2 looks at the administration of questionnaires and how best to obtain a good level of response. Sections 2.3 and 2.4 review issues related to the analysis of questionnaire responses and the use of results to improve teaching. All of the material is entirely relevant to use of questionnaires in economics, but the approach is generic and illustrates with examples drawn from various uses of questionnaires. Section 3 of the chapter is devoted to analysis of questionnaire use and practice in economics.
This section contains extensive guidelines on how to design a questionnaire. They are developed in simple headers and bullet points, which, I hope, will make this material more accessible and of practical benefit to potential users. There are many useful texts and guides to designing questionnaires, such as Newell (1993), Burns (2000), Bloom and Fischer (1982) and Kidder and Judd (1986).
Before you start to design a questionnaire, identify its objectives. More specifically, identify what kind of information you want to obtain. Then brainstorm – write down all possible questions for incorporating in the questionnaire.
This is the most difficult part of developing a questionnaire. Here are some useful rules of thumb to follow:
Most questionnaires contain both types of question and this is advisable. Closed and open questions are appropriate in different contexts and provide different kinds of information.
Closed questions are questions in which all possible answers are identified and the respondent is asked to choose one of the answers. In the following example, students were asked to evaluate the quality of programme materials (handouts, etc.) by a series of five closed questions. (The questionnaire is not well designed but illustrates clearly the nature of closed questions.)
Help us measure the success of the programme. Please tick one box for each of the questions.
|Programme materials||Excellent||Good||Fair||Poor||Unable to judge|
|1) the availability of the materials|
|2) the quality of the materials|
|3) the durability of the materials|
|4) the quantity of the materials|
|5) the suitability of the materials for students|
Source: Fitz-Gibbon and Morris (1987), p. 62.
The following are examples of ranked closed questions drawn from questionnaires used to evaluate teaching in anonymous economics departments.
Fill in one response for each question.
5 = Excellent, 4 = Very Good, 3 = Satisfactory, 2 = Fair, 1 = Poor
Skill of the instructor
|1) Instructor’s effectiveness as a lecturer||1||2||3||4||5|
|2) Clarity of instructor’s presentations||1||2||3||4||5|
|3) Instructor’s ability to stimulate interest in the subject||1||2||3||4||5|
For each of the following questions, please ring your answer.
The module as a whole
|1.||The module stimulated my interest|
|Disagree 1 2 3 4 5 Agree|
|2.||The module was|
|Too easy 1 2 3 4 5 Too hard|
|3.||The module objectives were fulfilled|
|Disagree 1 2 3 4 5 Agree|
This is an example of how ranked questions may be pooled to generate an overall index (from Henerson et al., 1987):
Teachers in a new experienced-based science programme filled out a questionnaire about each of several children in their classes. Here is a portion of the questionnaire:
The scores for questions 2, 3, 4 and 5 were summed to obtain ‘an enthusiasm index’ for each child, a point on a scale of 4–20. There are difficulties designing and interpreting these results, of course. We have to be sure that every question used in computing the index indeed reveals information about a student’s level of enthusiasm, and that the scales of the questions are consistent, i.e. that high enthusiasm is always indicated by scores close to or equal to 5. The greatest difficulty lies in interpretation of the final scores – usually researchers consider scores above or beneath threshold levels as revealing something definite about behaviour and attitudes, but it is difficult to know where to fix the thresholds. The alternative approach here would be to ask teachers to rate the enthusiasm of students.
Open questions are questions that allow the respondent to answer in any way they wish. For example, students might be asked to respond to the following question: ‘What do you feel is the best thing(s) about the course?’
‘. . . closed questions should be used where alternative replies are known, are limited in number and are clear-cut. Open-ended questions are used where the issue is complex, where relevant dimensions are not known, and where a process is being explored’ (Stacey, 1969).
Most questionnaires are ‘mixed’, containing both open and closed questions. This is often the best approach, avoiding an overly restrictive questionnaire and one that is too open and difficult to analyse. Open-ended questions can be used by students to elaborate on the reasons underlying their answers to the closed-form questions.
All questionnaires must be supported with some text. This should contain the following features:
It is essential that questionnaires are thoroughly tested prior to use. Bloom and Fischer (1982) identify five key criteria that may be used in evaluating the quality of a questionnaire – these are listed and discussed below. To evaluate a questionnaire effectively, it should be tested on an appropriate sample, which, in our case, is a sample of students. Test results are analysed and any changes to the questionnaire made. After initial implementation, questionnaires should continue to be evaluated as an ongoing process.
The criteria to use in evaluating a questionnaire are:
The key elements of the process of implementing and making successful use of questionnaires in teaching can be summarised as follows:
In this section, I discuss the administration of questionnaires, i.e. the process by which students receive and submit their questionnaires. In the subsequent sections, 2.3 and 2.4, I shall discuss the analysis of questionnaires, how results are used to improve teaching and the feedback of results to students and other stakeholders. Successful implementation of all stages of the process of evaluation requires active involvement of various individuals or groups; this is summarised in Figure 1. Lecturers are primarily responsible for administering, evaluating and acting upon the questionnaire. Students are responsible for answering the questionnaire and, together with the responsible authority within the department, for ensuring that their views are heard and acted upon.
Figure 1 Questionnaires: the process of evaluation
A criterion for successful questionnaires is maximisation of the student response rate. There are various ways of administering questionnaires that can help in achieving this:
I shall assume that the questionnaires were completed and submitted for analysis in paper form. Online questionnaires are discussed in section 4.1. Here is a summary of the key stages in the process of analysing the data with useful tips – more extensive discussion follows:
You will have a large number of paper questionnaires. To make it easier to interpret and store the responses, it is best to transfer data on to a single grid, which should comprise of no more than two or three sheets depending on the number of questions and student respondents. A typical grid looks like this:
If the answers to a question are represented on the questionnaire as points on a scale from 1 to 5, usually you will enter these numbers directly into the grid. If the answers take a different form, you may wish to translate them into a numerical scale. For example, if students are asked to note their gender as male/female, you may ascribe a value of 1 to every male response and 0 to female responses – this will be helpful when it comes to computing summary statistics and necessary if you are interested in exploring correlations in the data. It will make it much easier to analyse the data if there is an entry for all questions. To do this, you will need to construct code to describe ‘missing data’, ‘don’t know’ answers or answers that do not follow instructions – for example, if some respondents select more than one category.
Coding open questions is not straightforward. You must first read through all of the comments made in response to the open questions and try to group them into meaningful categories. For example, if students are asked to ‘state what they least like about the course’, there are likely to be some very broad themes. A number may not find the subject matter interesting; others will have difficulties accessing reading material. It may be useful to have an ‘other’ category for those responses that you are unable to categorise meaningfully.
Often, it is sufficient and best simply to calculate the proportions of all respondents answering in each category. (An Excel spreadsheet is much quicker than using a calculator!) It is clear that having a category for all respondents who either don’t know or didn’t answer is very important, as it provides useful information on the strength of feeling over a particular question.
Questionnaire results are often used to compute mean scores for individual questions or groups of questions. For example, the questionnaire may ask students to rate their lecturer on a five-point scale, with 5 denoting excellent, 4 good, 3 average, 2 poor and 1 very poor. The mean score is then used as an index of the overall quality of a lecturer with high scores indicating good quality. This is not a particularly useful or legitimate approach as it assumes that you are working on an evenly spaced scale, so that, for example, ‘very poor’ is twice as bad as ‘poor’, and ‘excellent’ twice as good as ‘good’.
Often analysts add up scores over a number of related questions. For example, you may ask students ten questions related to a lecturer’s skills, all ranked from 1 to 5 with 5 indicating a positive response, and add up the scores to derive some index of the overall ability of the lecturer. Again, except in carefully designed questionnaires, this approach is inappropriate. It assumes that each question is relevant and of equal importance. Comparing scores across different lecturers and modules, this assumption is unlikely to hold. If you are interested in summative indices of quality, it may be best simply to ask the students to rate the lecturer themselves on a ranked scale.
It is primarily the responsibility of the lecturer to review the responses and results of the questionnaires and these should be summarised in a summary report, which is presented to the department and to a representative student body. The key feature of the report is an ‘action plan’ indicating how the lecturer intends to act upon the findings of the questionnaire to improve the learning experience in future courses. Where no changes are envisaged, the reasons for these must be clearly stated. It is important that teachers receive some form of training in how to go about interpreting and using questionnaire results – as stated earlier, reading questionnaire responses can be a difficult process for inexperienced teachers and support should be available.
It is good practice to ensure that lecturers and tutors do not see questionnaires relating to themselves and to the modules for which they have responsibility until assessment of the module is completed. Analysis and report writing should then be done as soon as possible.
It is possible that your questionnaire, if formatted appropriately, may be read and scored by machine, or that you can use a machine-scorable answer sheet. This can significantly reduce time involved in analysing questionnaires.