3. Factors to Consider when Developing a Research Intensive Experience for Undergraduates

The guidance that is provided to students is crucial for the effectiveness of an undergraduate research programme. A first step is to agree upon a common strategy across the faculty. Explicit choices must be made regarding:

  • the form of research process to be encouraged,
  • the degree to which students are free to select the research topic,
  • project formats that will be acceptable,
  • level of guidance that will be provided, and
  • skills the project is designed to enhance.

3.1 The form of the research process

Although it may seem elementary to define the research process, it is a logical first step in developing an undergraduate research experience. By defining the process through a series of key components, one can begin to identify important issues that must be resolved when developing projects. For the purpose of discussion, let us assume the definition of the research process in Figure 1.

Figure 1: Steps in the Research Process
  1. Identify economic issues
  2. Develop a research question
  3. Undertake a literature search
  4. Summarise relevant literature
  5. Identify an area of potential contribution
  1. Develop evidence
  2. Draw conclusions
  3. Compare conclusions to identified literature
  4. Apply analysis to current policy issue
  5. Present research to peers and/or faculty

Before proceeding to a discussion of some of the key aspects associated with each of these steps in the research process, a number of general issues should be kept in mind throughout this discussion. Although papers are typically structured in a traditional format starting with an introduction, this does not imply that the process of developing a final product should be conducted in that same order. In fact, students who try to write their introduction before their analysis is conducted often find themselves having to completely rewrite it because the focus of the paper has shifted from their initial conceptualisation. Students also need to understand that there are two audiences for their work. The final product should be of interest to their peers while also demonstrating students’ competencies to the instructor. Traditional instructor feedback might be complemented with peer review. Furthermore, feedback provided throughout the process should be both formative and summative so that students learn how to interpret critical comments and incorporate changes. Using the revise and resubmission method that is standard in our own process of research allows instructors to nurture students’ understanding and skills. These issues of evaluation are developed in greater detail in the ‘Evaluating Student Projects’ section of this chapter.

3.1.1 Identifying economic issues

Students typically bring many interests to the research process. In fact, getting them to narrow their interests to a single economic issue can be quite a challenge. It is important to have students distinguish between choosing a topic and identifying an economic issue. For example, a student might be interested in labour economics as a topic but their research issue must be much more narrow. For example, they might analyse the degree to which gender affects compensating wage differentials.

3.1.2 Developing an effective economic research question

Once students have settled on a viable topic and issue, they must narrow their focus by developing a research question. Students typically have the capacity to generate a lot of interesting questions with respect to a topic, but developing an effective question is a much more difficult task which will determine the success of the research. Effective economic questions have distinct characteristics that can be directly linked to the research process. Greenlaw (2006: 14–18 ) suggests that effective economic questions are problem-oriented, analytical, interesting and significant, amenable to economic analysis and feasible. Figure 2 provides examples of first attempts made by students at developing effective economic questions. Along with each question is an evaluation of the degree to which the criteria for an effective economic question are satisfied.

Figure 2: Developing Effective Economic Questions

Criteria:

  • Problem-oriented: to focus the question in a meaningful way the question is framed in the form of a problem to be addressed rather than a question that can be answered with a simple yes or no.
  • Analytical: the question should drive the researcher to explain rather than describe a problem.
  • Interesting and significant: in order to encourage the researcher to produce exemplary work and to engage the audience.
  • Amenable to economic analysis: what insights can be gained from applying economic analysis to the question at hand?
  • Feasible: can the proposed question be answered in the given time allotment?
Topic and Questions Effective? Evaluation:
criteria not satisfied
Topic: Consolidation in the Public Accounting market, specifically in the area of external audit
  1. Is the public accounting market no longer competitive due to consolidation?
  2. Does the public accounting market need to be competitive?
  1. no
  2. no
  1. Not problem-oriented.
  2. Not analytical, this question is opinion based.
Topic: Economic Impact of Hurricane Katrina
  1. Will the US economy continue to scuffle in the wake of Katrina or will reconstructive efforts provide the projected boost?
  2. How much will the reconstruction effort cost and will that be larger than the benefits given the characteristics of New Orleans?
  1. no
  2. no
  1. Not problem-oriented, analytical, or feasible.
  2. Not problem-oriented or feasible.
Topic: An analysis of the US Film Industry
  1. Do certain critics’ reviews have an influence on the demand for a film in the box office?
  2. An analysis of the American Film Industry during the 1990s.
  1. potentially
  2. no
  1. More detail needs to be provided to define what is meant by ‘certain critics’ and ‘influence’.
  2. Not problem-oriented, analytical, amenable to economic analysis or feasible.
Topic: The economic and environmental implications of agricultural subsidies
  1. What are the economic and environmental implications of California's Central Valley Project?
  2. Do the economic and environmental costs of direct payment agricultural subsidy programmes outweigh the benefits?
  1. no
  2. potentially
  1. Descriptive rather than analytical.
  2. The issue of concern is feasibility; namely, can these costs and benefits be measured? Also, this would need to be rephrased to make it more problem-oriented.
Topic: Tort reform and its effects on California compared to various other states
  1. Will tort reform considerably change the malpractice environment?
  2. Why has tort reform in California changed the malpractice environment?
  1. no
  2. potentially

The question is not problem-oriented and it is not clear what analysis would be used.

It is not clear what is meant by the malpractice environment and thus hard to judge whether it is problem-oriented or analytical.

Topic: The spa industry in relation to the small business world.
  1. Why has the recent boom in spa attendance started to decline?
  2. What are the most significant economic factors in creating a successful spa?
  1. yes
  2. no
  1. Fits all criteria, but care must be taken to make sure this is analytical and not descriptive.
  2. Descriptive rather than analytical.
Topic: AIDS in South Africa
  1. How does the incidence of HIV/AIDS in South Africa affect its GDP?
  2. How does the AIDS epidemic in South Africa affect its human capital and labour supply?
  1. yes
  2. yes
  1. Fits all criteria.
  2. Fits all criteria.
Topic: Whether or not increasing the amount of domestically produced oil, particularly in the Alaskan National Wildlife Preserve, would economically benefit the United States
  1. How has the price of oil affected indicators of economic well being such as the unemployment rate and the GDP in the United States over the past decade?
  2. How would drilling in the Alaskan National Wildlife Preserve (ANWR) affect the United States’ reliance on foreign oil?
  1. potentially
  2. yes
  1. The problem is whether this question is feasible and to what extent a micro change (price) can be used to explain macro conditions.
  2. Fits all criteria.
Topic: How are baseball players valued based on the MRP they bring to their respective teams
  1. How effective is the MLB free-agency market in efficiently valuing players?
  2. Why are some MLB small market teams continually competitive (i.e. Oakland Athletics) while others are not (i.e. Pittsburgh Pirates)?
  1. yes
  2. yes
  1. Fits all criteria.
  2. Fits all criteria, but care must be taken to make sure this is analytical and not descriptive.
Topic: The possibility of an organ market and the effect it would have on both the number of donors and the recipients waiting for organs.
  1. Would an open market for organs be an effective alternative to the present allocation techniques?
  2. What impact would restitution for donation of organs have on the overall supply of organs?
  1. no
  2. yes
  1. Not problem-oriented, analytical, or feasible.
  2. Fits all criteria

The examples provided in Figure 2 underscore the general lack of understanding that students have regarding the development of an effective research question and thus practise with existing questions can help. Requiring students to read papers that model acceptable formats for completed research (such as traditional quantitative analyses, historical perspectives, analysis of contemporary economic issues, comparing alternative viewpoints in economics, and community-based projects) and identify the key question is a valuable first step. Alternatively, students can be provided with a list of economic questions (such as first and final drafts of questions from student projects in previous semesters) and be asked to evaluate each question to demonstrate their comprehension of the characteristics. After a bit of practise in evaluating questions and developing a richer understanding of effective economic research questions, students can be asked to develop and justify their own questions.

3.1.3 Undertaking a literature search, summarising the literature and identifying an area of potential contribution

The completion of a well-developed literature review requires that a student demonstrates skill in locating and summarising publications that are key to their project. Unless guided to do otherwise, students are likely to rely on general internet searches to locate literature. Using academic search engines such as EconLit and Google Scholar, accessing working paper series that are available online such as the National Bureau of Economic Research (www.nber.org) and NetEc (http://netec.wustl.edu/NetEc.html), and searching more general resource sites such as Resources for Economists (http://rfe.org) will greatly enhance the probability of finding appropriate sources. Students readily pick up the skills associated with search techniques specific to the economics literature.

The literature review for an undergraduate research project is, arguably, different from those of professional researchers because it will be addressing a different audience – instructors/tutors and peers:

The literature review will be used by tutors to assess students’ understanding of the arguments and evidence presented in the literature. A literature review in a published article might be expected to be more truncated, demonstrating appreciation of the breadth of the field and locating research questions in the context of previous work. Published authors are generally not expected to demonstrate their understanding of arguments they refer to, only to avoid misrepresentation. Thus, a guide suggesting what details should be included for each article reviewed may include the article’s main question, evidence provided, caveats, conclusions and relevant policy arguments.

Tutors may also wish to assess students’ ability to relate their research findings to public debate and to communicate economic analysis to a range of audiences and the student peer group provides a practicable opportunity to assess this capability. For these reasons it might be helpful for students to refer to some non-academic sources (such as newspapers) in their literature review. Allowing the review to include popular press sources and interdisciplinary perspectives will help develop peer interest in the more narrowly focused analytical section of the paper. Inclusion of these alternative sources suggests, however, that students also develop skills in evaluating the reliability of such sources.

For more advanced undergraduate research projects that expect students to contribute to the knowledge base in economics, the review presents an additional challenge. Despite their skill at synthesising literature, students are more likely to struggle with the second goal of a literature review: providing a launching point for their own analysis and contribution. Students are much less practised at critiquing the process used by professionals, providing alternative interpretations of results generated and finding gaps in the literature that are ripe for exploration. Building off the modelling process in developing an effective economic question described above, students can use the same model articles to practise these critiques in cooperative settings.

3.1.4 Develop evidence

Research projects that require students to perform original work have an added set of hurdles to overcome. As those who practise the art of research on a regular basis, we often take for granted the process of analysis that is used in a research paper. Students, on the other hand, are likely to be facing this process for the first time and thus it is natural for them continually to fall back on what others have done rather than presenting original evidence and arguments. Convincing the audience of the validity of claims depends on the reasoning process employed and the evidence presented. Although most of the research that undergraduates attempt is applied empirical research and thus inductive in nature (using specific cases to reason out a general rule), both deductive (moving from general to specific) and warrant-based arguments (using underlying assumptions to make links between evidence and claims) can be employed as alternative methods of analytical reasoning (Greenlaw, 2006: 57–59 ).

Students also need to develop an understanding of these methods of reasoning and appreciate that audiences must be convinced by the evidence that is presented. Modelling this process using papers that are exemplars of acceptable formats will provide opportunities for instructors to address each of these issues. Students can be asked to break down the analytical section of a paper to identify the key components of the process and describe the evidence used. They can argue why they think the evidence presented is, or is not, sufficient and the degree to which evidence is open to alterative interpretations. The nurturing of students through this process provides them with the necessary understanding to attempt their own analysis. Figure 3 demonstrates a set of model questions that can be used to facilitate this process. Students are asked to read a model article and answer a series of questions that require them to identify the process of answering an effective economic question. A majority of the class discussion that is used as a follow-up activity to this assignment focuses on the types of evidence and the answer to the economic question. Students are placed into groups and asked to compile a comprehensive list of the evidence they have identified in the article. The final, and probably hardest, component is to have students begin to evaluate the evidence in terms of the degree to which it sufficiently answers the question posed and to identify potential caveats to the authors’ argument.

Figure 3: Developing the Evidence

The assignment:

Read KimMarie McGoldrick and Lisa Ford Voeks (2005) " 'We Got Game!' An Analysis of Win/Loss Probability and Efficiency Differences between the NBA and WNBA", Journal of Sports Economics, Vol. 6(1), pp. 5-23.
http://jse.sagepub.com/cgi/content/abstract/6/1/5

Goal of this assignment:

Look at Greenlaw's Table 2.1 (page 13) that describes "The Research Process in Economics" and the corresponding text. The WNBA is an example of an empirical paper and was developed in exactly this fashion. By discussing a research paper so early in the semester you will be more familiar with some of the components that you will need to complete in your own work. Thus, we will use this work to show an overview of the research process. I will also be discussing some of the hurdles that we had to overcome in conducting this research.

Questions:

  1. Provide a general description (3-4 sentences) of the purpose of this paper.
  2. What is the economic question that the authors are attempting to answer? Can you locate this specifically in the text? Where? What sub-questions are used to answer this main question? List these.
  3. What contribution to the literature does this paper make?
  4. What type of evidence does this paper provide in answering the economic question? Is this form of evidence consistent with that found in cited literature?
  5. Is the evidence convincing? Here you want to consider how much evidence is presented, the extent to which proxies are used, whether the data is complete or restricted in any way, etc. What additional evidence would add to the story told by the authors?
  6. What is the answer to the economic question? Can you locate this specifically in the text? Where?
  7. What was most confusing about the paper? Be specific and, if possible, provide page numbers.
  8. What was most clear about the paper? Be specific and, if possible, provide page numbers.
  9. Construct a well-detailed outline of this paper on a separate sheet of paper (typed). Bring two copies to class. Please note that a good outline does not have complete sentences and is detailed enough that you could actually write the paper from it if necessary.

We must allow for the possibility that a student may demonstrate good understanding and practice in the research process without managing to gather and analyse data that support their original hypothesis. Such ‘failures’ may turn out to be excellent learning experiences and students can be encouraged to use these as opportunities to develop a critical analysis of their own work in their conclusions.

3.1.5 Draw conclusions and compare conclusion to identified literature

Many students tend to use the final section of the research project to summarise key points already made in the body of the paper. A conclusion section should challenge students to place their work in the context of the larger body of knowledge. This requires that they first conduct critical reflection regarding the evidence that they have presented. Are proxies used in their analysis satisfactory? Does the evidence support their hypothesis? How does it compare to other conclusions in the literature on this topic? Is there an alternative interpretation? In light of these caveats, students should also be required to identify questions that remain unanswered and to indicate what they regard as priorities for further research. These observations might be linked specifically to what they would do if they had more time.

In order to facilitate this process, a few exercises can be implemented throughout the semester in order to model the critical thinking we wish to see in students’ conclusions. For example, when they are assembling their evidence students can be asked to develop a three-column table. The first column contains a description of each piece of evidence that when aggregated would be sufficient to support their claims. The second column would identify what ideal variables would measure, describe or quantify. The final column would list the actual data they have for each piece of evidence. By comparing the second and third columns of this table, students can begin to develop a sense of the extent to which they must state caveats to their research and suggest alternative explanations that cannot be ruled out given the limitations of their evidence.

3.1.6 Apply analysis to current policy issue

The value of doing research can be demonstrated to students through its connection with policy recommendations. In order to encourage students to begin making policy connections early in their research, one might implement a modelling exercise directly after the literature review section of their paper is complete. Students can be asked to identify all the relevant policy arguments included in the research cited in their review. A summary of these policies and the extent to which they might also apply in their own work can be a follow-up exercise that is used later when developing the conclusion.

3.1.7 Present research to peers and/or faculty

Effectively communicating key components of conducted research in front of peers and faculty may not be a comfortable task for students. Students often fall into the trap of trying to communicate everything they have done in a step-by-step description of their work rather than conveying the meaning of their findings. It is usually helpful to provide guides to the processes of summarising research and developing complementary visual aids. One logical starting point is for students to make a detailed outline of their paper and to use this as a guide for what to include in the presentation. As with professional presentations, however, one cannot hope to include all the work that has been completed for the project. Recommending that each sub-section of the paper be limited to one or two visual representations (such as an overhead or PowerPoint slide) with no more than four points on each is one step toward avoiding the problem of overwhelming presentations. Students should also be reminded of the need to keep their audience focused on key aspects of their work: the economic question, the way in which their question is related to previous work, the analysis of their evidence and their conclusions.

3.2 The degree to which students are free to select the research topic

Allowing students to research topics that capture their interest and which they believe are relevant to them makes sense if we wish to develop optimal motivation for their effort. Because most students are novices in the research process, it is tempting to skip this and provide a list of research ideas from which students can choose. This is especially likely if the project is developed for a course with a prescribed set of topics.

Although providing topics allows students to get a jump start on their research, there is a strong likelihood that the student will not have a passion for the topic. Alternatively, students can be asked to develop their own research topic. While this freedom of choice can occur with projects developed for topic-driven elective courses, it is more likely in the case of an independent study, a senior thesis or a capstone course dedicated to the research process. The advantage of free choice is that it allows students to be creative in the development of topics that may be interdisciplinary, introduce new topics not covered in currently offered courses or serve as extensions of their own past work. The downside of this choice is that it also requires students to overcome a significant hurdle: narrowing their interests to a single topic.

3.3 Project formats

One might broaden the scope of the undergraduate research experience by encouraging students to reflect on what economists actually do. Essentially economists discover, develop and promote the knowledge and study of economics through the varied facets of academic life. This process is aptly captured in the words of Boyer (1990: 24), ‘knowledge is acquired through research, through synthesis, through practice and through teaching’, which he defined as the scholarships of discovery, integration, application and teaching, respectively.

While it is clear that, as professional economists, we actively partake in these forms of knowledge acquisition, Boyer (1997: 79) suggests ‘the redefinition of scholarship might also be appropriate for students. Why not have all incoming students join with the faculty right away as young scholars in the discovery of knowledge, in the integration of knowledge, in the application of knowledge, and in the communication of knowledge? Why not have these four dimensions of scholarship become the four essential goals of undergraduate education?’

Explicitly identifying steps of the research process can stimulate discussion of alternative methods of conducting research. Students are more likely to develop an understanding of the methods of research traditionally used by economists if they are aware of alternatives that could be adopted. Comparison of methods facilitates critical reflection that is usually a stated outcome of degree programmes. Comparison of this kind might be provided through examples taken from heterodox literature.

3.4 Level of guidance that will be provided

Students come into the research process equipped with varying skills. The degree to which students are allowed to develop their projects independently depends on their maturity to tackle such an undertaking. Issues that need to be considered regarding the level of guidance include:

  • determining the degree to which the research process will be demonstrated to students through specific exercises;
  • choosing the number of intermediate points of evaluation at critical junctures (such as the effective economic question, situating current research in past literature, developing and analysing the evidence, and the conclusion); and
  • the degree to which face-to-face meetings will be conducted to guide and push students to do better research.

While it might appear that senior research theses or capstone courses are the time at which more independence can be assumed, this depends entirely on the degree to which students have developed the necessary skills in previous courses.

3.5 Skills the project is designed to enhance

Because the undergraduate research project can be developed at many levels of the curriculum, it is important to identify specifically what skills the project is being designed to enhance. Although one might expect students to be proficient in accessing existing knowledge, displaying command of that knowledge, providing both theoretical and empirical interpretations, applying knowledge, asking pertinent and penetrating questions, and creating new knowledge (Hansen, 1986) by the end of their undergraduate studies, courses throughout their studies introduce and develop each of these skills.

For example, in an industrial organisation and public policy course a research project that focuses on displaying command of existing knowledge might require students to conduct a review of literature in a single industry to determine its structure, firm conduct and economic performance in support of a determination of the position of that industry on the competitive-monopoly market continuum. More detailed discussion of skills development is found in the following section.