The most common form of evaluation is summative, used to judge performance and provide grades. In addition, formative evaluation can be used to support students’ learning. Most evaluation instruments such as quizzes and examinations are summative whereas active learning techniques such as experiments and cooperative learning tend to be formative. The remainder of this section will highlight some key points to consider in developing criteria and processes for three dimensions of evaluation. It is critical that instructors develop formative and summative evaluation practices at three levels:
The advantage of grounding the research project in a learning taxonomy (such as Hansen’s proficiencies) is that it explicitly identifies the research skills we want our students to develop. Consider, for example, a research project that is designed so that students create new knowledge. This requires that students use their research skills to understand a body of literature, choose a topic and then develop an effective economic research question. The summative part of the evaluation process would include a measure of the students’ success in demonstrating that they have a command of the foundational knowledge that supports such a question and the extent to which they could justify it in terms of the components – problem-oriented, analytical, interesting and significant, amenable to economic analysis and feasible (Greenlaw, 2006: 14–18 ) – required for the question to be effective. But it is also critical at this juncture to provide constructive feedback (formative evaluation) to the student as they simply cannot continue until their research question is viable. Similar arguments can be made for other research skills that would be expected of a student for a project that is intended to have them create new knowledge including: locating key literature relating to this question, properly identifying an area of contribution, developing economic evidence in support of their hypothesis, and contextualising the results of their work.
Developing structured arguments using economic analysis is one of the hardest tasks that students face in the research process. Detailed descriptions of project components can communicate the form and quality of economic analysis that is expected in a research project However, even when students are generally competent in summarising and synthesising literature, striking out on their own to conduct original analysis is completely foreign to them. This is one instance in which the process by which professionals conduct their own research is a natural model for students. Rarely does a professional paper become published that is not subject to review and revision. Often the revision entails strengthening the economic analysis and arguments: exactly what we want for our students. Challenging student arguments, offering alternative perspectives and correcting inaccuracies are critical inputs into the research process. It is important, however, to make sure that a student learns from this process and does not simply wait for feedback and revise the work accordingly. Feedback on multiple stages of a project can begin by providing nurturing comments in early assignments that signal expectations and provide editorial comments designed to show students how to improve their analysis. As the project progresses, however, these comments should simply point out problems and allow the student to develop the solution.
Special consideration needs to be given to the extent to which students are expected to create new knowledge. Undergraduate research projects have great potential to push students beyond the boundaries of their current knowledge in a variety of ways. The use of undergraduate research as a pedagogical tool is all about stretching students’ minds and providing them with skills to act like economists. A word of caution is appropriate, however, since the structure of undergraduate research projects closely mimic professionally developed publications; it is tempting to develop too high expectations for students. To avoid unnecessary frustrations it is important that new knowledge creation be judged in the context of the students’ knowledge base and demonstrate the achievement of the proficiencies rather than using our professional knowledge base as the benchmark.
As suggested above, presentation matters in the case of undergraduate research more than in most other learning exercises. Student learning as a result of the research process is more sophisticated and it requires a complementary level of communication, either oral or written.
In the case of oral skills one can model expectations and provide formative feedback through class discussions regarding specific components of the research process. For example, students can be asked to provide an oral justification of their research question to a small group of classmates. This provides a good opportunity for students to practise formulating their thoughts in a public forum and to receive constructive feedback that will improve their question (and subsequent research project). Oral communication skills can also be evaluated during one-on-one meetings. Since students must explain the status of their project and their future plans, the tutor can observe communication skills in an informal setting and provide guidance on how such skills may be improved. Summative evaluation can be achieved through presentations to peers and academics.
Research (Lehr, 1995) suggests that an understanding of the revision process is a key component for developing and improving writing. Writing skills can be enhanced when students are required to develop sections of their projects through revision and resubmission. This is likely to help students to improve their arguments and it is also likely to help with two key components with which students often struggle: organisation and grammar. As with the process described for economic analysis, feedback on students’ written communication should progress from nurturing and providing examples of solutions to the point at which they simply point out errors and require the student to develop the necessary revisions. Once again, the one-on-one meetings can help this process because they provides a more collaborative forum for discussing recurrent problems and encourages students to see the value in this formative evaluation of their work and the opportunities that the revision process provides.
Other helpful processes that encourage good writing include the use of grading rubrics and model writing. Signaling expectations through the use of grading rubrics that include sections for both content and communication skill evaluation encourage students to take the communication of content more seriously, especially if they are also required to evaluate their own work using the rubric. While presenting models of good student writing from previous semesters might also help students understand expectation levels, it can provide some students with a crutch leading to papers that are mirror images of the models. A more effective way to use sample writing as a model is to provide a single first draft paragraph along with the multiple revisions that demonstrate the development into a much-improved final version.
In summary, it is important to recognise and provide opportunities for summative and formative methods of evaluation that can occur formally via written evaluations of work and informally during class exercises and one-on-one meetings.
How do we get our students to recognise the quality of their own work? My experience suggests that we need to provide guides and models that students can use as indicators of quality. Such guides should run the gamut from general to specific. For example, three handouts I have used to help my students develop a better understanding of what it means to do quality research include:
Students are less likely to have accurate understanding of tutors’ definitions of top-quality work if these definitions are not disclosed to them. The course guide I provide all my research students with is over 100 pages and contains descriptions of expectations, research and writing tips, course assignments and evaluations (to be completed by the student, peers and the instructor). The guide aims to:
One of the very first documents in this guide is a grading scale that specifically identifies what it takes to earn an A, B, C, D or F. Each grade is described in terms of what the written report includes, what it does not include and the way that information should be communicated. There are many such grading scale models available from which one might develop a specific guide for students. See for example:
Since mistakes made by novice researchers tend to be consistent over time, I provide my students with a series of research tips before they even begin their research projects (Figure 7). These tips focus on the research process as well as the expected output. More importantly, the class discussion that accompanies this handout is focused on why these items are important.
While many of these statements are specific to the final research product, now is a good time to get an idea of my expectations and problems to avoid.
Greenlaw, S. (2006) Research Methodology: A Guide for Undergraduate Economic Research, Houghton-Mifflin.
McCloskey, D. (2000) Economical Writing, 2nd edition, Waveland Press, Inc.
Finally, because students (not unlike ourselves) find it easier to be critical of another person’s work rather than see similar flaws in their own work, I implement a peer review process of all final drafts of research projects. The peer review document in Figure 8 has a combination of general and specific questions relating to both the content and presentation of the economic analysis. Not only does this review provide students with a new set of comments on their papers, but often reading another student’s paper provides another model for organising arguments, shows examples of common mistakes that are made, and stresses the importance of clear and concise communication. After students have reviewed a peer’s work, I ask them to re-read their own work (prior to receiving the peer comments). They are asked to identify (in much the same manner as they did in reviewing their peers’ work) the strengths and weaknesses of their own research. Because the process of reviewing is fresh in their minds, they are more likely to be critical of their own research.
Title of Paper: _________________________________________________________
Answer the following questions thoughtfully, clearly and concisely. Use complete sentences and specific examples (or page references) to make your advice as understandable as possible. You will be evaluated on the thoroughness and helpfulness of your responses to your peer partner. To make your review as beneficial as possible, it is important that you follow the review steps in the order given below:
First, read the paper carefully, without pausing to comment, to get the overall effect of the paper.
In the space below, write down your general feelings about the paper. Is it interesting? Did it generally maintain your interest? Is it easy to follow (organisation is clear)? Did you get lost along the way?
Next, look over all the questions below and then reread each section of the paper and answer the related questions. As you reread the paper make margin notes to the author identifying which paragraphs/sections were clear and which ones were not.
The introduction should present the topic and its importance. It should clearly define relevant concepts, provide an overview of relationships, present the thesis and indicate how it will be developed, and generally prepare the reader for the paper to follow.
The review is designed to inform the reader of what other work has been completed on this topic and to motivate the current work. The review presentation should be organised (such as chronologically or by theme) and be directly related to the research project.
In reviewing this section you should mark each sub-section as excellent if it can be clearly identified in the paper, if it accomplishes what is expected, if you are not left with any content questions, and if it is well written (good flow between and within paragraphs, good sentence structure, etc.). Poor sub-sections are those that are not clear because of incoherent content or flow issues, are incomplete leaving you with unanswered questions or issues not sufficiently addressed, or missing altogether.
Please be sure to provide comments for each subsection!
| Contribution Section | Excellent | Good | Poor |
|---|---|---|---|
| Statement of what will be demonstrated in the analysis Comments: |
|||
| Set up of relationships Comments: |
|||
| Hypothesis development Comments: |
|||
| Description of method Comments: |
|||
| Evidence Comments: |
|||
| Relation of evidence to general hypothesis Comments: |
|||
| Comparison of findings to previous research Comments: |
|||
| Effective use of tables, etc. (where applicable) Comments: |
Rating: On a scale of 1 to 5 (5 being excellent) what rating would you give this contribution section? Explain.
A conclusion must go beyond a simple restatement of the overall argument of the paper. It should provide a final push to convince the reader that the economic question proposed in the thesis has been answered.
A research paper can only effectively convey its message if it is well written.