The Handbook for Economics Lecturers

Although it may seem elementary to define the research process, it is a logical first step in developing an undergraduate research experience. By defining the process through a series of key components, one can begin to identify important issues that must be resolved when developing projects. For the purpose of discussion, let us assume the definition of the research process in Figure 1.

Figure 1: Steps in the Research Process
  1. Identify economic issues
  2. Develop a research question
  3. Undertake a literature search
  4. Summarise relevant literature
  5. Identify an area of potential contribution
  1. Develop evidence
  2. Draw conclusions
  3. Compare conclusions to identified literature
  4. Apply analysis to current policy issue
  5. Present research to peers and/or faculty

Before proceeding to a discussion of some of the key aspects associated with each of these steps in the research process, a number of general issues should be kept in mind throughout this discussion. Although papers are typically structured in a traditional format starting with an introduction, this does not imply that the process of developing a final product should be conducted in that same order. In fact, students who try to write their introduction before their analysis is conducted often find themselves having to completely rewrite it because the focus of the paper has shifted from their initial conceptualisation. Students also need to understand that there are two audiences for their work. The final product should be of interest to their peers while also demonstrating students’ competencies to the instructor. Traditional instructor feedback might be complemented with peer review. Furthermore, feedback provided throughout the process should be both formative and summative so that students learn how to interpret critical comments and incorporate changes. Using the revise and resubmission method that is standard in our own process of research allows instructors to nurture students’ understanding and skills. These issues of evaluation are developed in greater detail in the ‘Evaluating Student Projects’ section of this chapter.

3.1.1 Identifying economic issues

Students typically bring many interests to the research process. In fact, getting them to narrow their interests to a single economic issue can be quite a challenge. It is important to have students distinguish between choosing a topic and identifying an economic issue. For example, a student might be interested in labour economics as a topic but their research issue must be much more narrow. For example, they might analyse the degree to which gender affects compensating wage differentials.

3.1.2 Developing an effective economic research question

Once students have settled on a viable topic and issue, they must narrow their focus by developing a research question. Students typically have the capacity to generate a lot of interesting questions with respect to a topic, but developing an effective question is a much more difficult task which will determine the success of the research. Effective economic questions have distinct characteristics that can be directly linked to the research process. Greenlaw (2006: 14–18 ) suggests that effective economic questions are problem-oriented, analytical, interesting and significant, amenable to economic analysis and feasible. Figure 2 provides examples of first attempts made by students at developing effective economic questions. Along with each question is an evaluation of the degree to which the criteria for an effective economic question are satisfied.

Figure 2: Developing Effective Economic Questions

Criteria:

  • Problem-oriented: to focus the question in a meaningful way the question is framed in the form of a problem to be addressed rather than a question that can be answered with a simple yes or no.
  • Analytical: the question should drive the researcher to explain rather than describe a problem.
  • Interesting and significant: in order to encourage the researcher to produce exemplary work and to engage the audience.
  • Amenable to economic analysis: what insights can be gained from applying economic analysis to the question at hand?
  • Feasible: can the proposed question be answered in the given time allotment?
Topic and Questions Effective? Evaluation:
criteria not satisfied
Topic: Consolidation in the Public Accounting market, specifically in the area of external audit
  1. Is the public accounting market no longer competitive due to consolidation?
  2. Does the public accounting market need to be competitive?
  1. no
  2. no
  1. Not problem-oriented.
  2. Not analytical, this question is opinion based.
Topic: Economic Impact of Hurricane Katrina
  1. Will the US economy continue to scuffle in the wake of Katrina or will reconstructive efforts provide the projected boost?
  2. How much will the reconstruction effort cost and will that be larger than the benefits given the characteristics of New Orleans?
  1. no
  2. no
  1. Not problem-oriented, analytical, or feasible.
  2. Not problem-oriented or feasible.
Topic: An analysis of the US Film Industry
  1. Do certain critics’ reviews have an influence on the demand for a film in the box office?
  2. An analysis of the American Film Industry during the 1990s.
  1. potentially
  2. no
  1. More detail needs to be provided to define what is meant by ‘certain critics’ and ‘influence’.
  2. Not problem-oriented, analytical, amenable to economic analysis or feasible.
Topic: The economic and environmental implications of agricultural subsidies
  1. What are the economic and environmental implications of California's Central Valley Project?
  2. Do the economic and environmental costs of direct payment agricultural subsidy programmes outweigh the benefits?
  1. no
  2. potentially
  1. Descriptive rather than analytical.
  2. The issue of concern is feasibility; namely, can these costs and benefits be measured? Also, this would need to be rephrased to make it more problem-oriented.
Topic: Tort reform and its effects on California compared to various other states
  1. Will tort reform considerably change the malpractice environment?
  2. Why has tort reform in California changed the malpractice environment?
  1. no
  2. potentially

The question is not problem-oriented and it is not clear what analysis would be used.

It is not clear what is meant by the malpractice environment and thus hard to judge whether it is problem-oriented or analytical.

Topic: The spa industry in relation to the small business world.
  1. Why has the recent boom in spa attendance started to decline?
  2. What are the most significant economic factors in creating a successful spa?
  1. yes
  2. no
  1. Fits all criteria, but care must be taken to make sure this is analytical and not descriptive.
  2. Descriptive rather than analytical.
Topic: AIDS in South Africa
  1. How does the incidence of HIV/AIDS in South Africa affect its GDP?
  2. How does the AIDS epidemic in South Africa affect its human capital and labour supply?
  1. yes
  2. yes
  1. Fits all criteria.
  2. Fits all criteria.
Topic: Whether or not increasing the amount of domestically produced oil, particularly in the Alaskan National Wildlife Preserve, would economically benefit the United States
  1. How has the price of oil affected indicators of economic well being such as the unemployment rate and the GDP in the United States over the past decade?
  2. How would drilling in the Alaskan National Wildlife Preserve (ANWR) affect the United States’ reliance on foreign oil?
  1. potentially
  2. yes
  1. The problem is whether this question is feasible and to what extent a micro change (price) can be used to explain macro conditions.
  2. Fits all criteria.
Topic: How are baseball players valued based on the MRP they bring to their respective teams
  1. How effective is the MLB free-agency market in efficiently valuing players?
  2. Why are some MLB small market teams continually competitive (i.e. Oakland Athletics) while others are not (i.e. Pittsburgh Pirates)?
  1. yes
  2. yes
  1. Fits all criteria.
  2. Fits all criteria, but care must be taken to make sure this is analytical and not descriptive.
Topic: The possibility of an organ market and the effect it would have on both the number of donors and the recipients waiting for organs.
  1. Would an open market for organs be an effective alternative to the present allocation techniques?
  2. What impact would restitution for donation of organs have on the overall supply of organs?
  1. no
  2. yes
  1. Not problem-oriented, analytical, or feasible.
  2. Fits all criteria

The examples provided in Figure 2 underscore the general lack of understanding that students have regarding the development of an effective research question and thus practise with existing questions can help. Requiring students to read papers that model acceptable formats for completed research (such as traditional quantitative analyses, historical perspectives, analysis of contemporary economic issues, comparing alternative viewpoints in economics, and community-based projects) and identify the key question is a valuable first step. Alternatively, students can be provided with a list of economic questions (such as first and final drafts of questions from student projects in previous semesters) and be asked to evaluate each question to demonstrate their comprehension of the characteristics. After a bit of practise in evaluating questions and developing a richer understanding of effective economic research questions, students can be asked to develop and justify their own questions.

3.1.3 Undertaking a literature search, summarising the literature and identifying an area of potential contribution

The completion of a well-developed literature review requires that a student demonstrates skill in locating and summarising publications that are key to their project. Unless guided to do otherwise, students are likely to rely on general internet searches to locate literature. Using academic search engines such as EconLit and Google Scholar, accessing working paper series that are available online such as the National Bureau of Economic Research (www.nber.org) and NetEc (http://netec.wustl.edu/NetEc.html), and searching more general resource sites such as Resources for Economists (http://rfe.org) will greatly enhance the probability of finding appropriate sources. Students readily pick up the skills associated with search techniques specific to the economics literature.

The literature review for an undergraduate research project is, arguably, different from those of professional researchers because it will be addressing a different audience – instructors/tutors and peers:

The literature review will be used by tutors to assess students’ understanding of the arguments and evidence presented in the literature. A literature review in a published article might be expected to be more truncated, demonstrating appreciation of the breadth of the field and locating research questions in the context of previous work. Published authors are generally not expected to demonstrate their understanding of arguments they refer to, only to avoid misrepresentation. Thus, a guide suggesting what details should be included for each article reviewed may include the article’s main question, evidence provided, caveats, conclusions and relevant policy arguments.

Tutors may also wish to assess students’ ability to relate their research findings to public debate and to communicate economic analysis to a range of audiences and the student peer group provides a practicable opportunity to assess this capability. For these reasons it might be helpful for students to refer to some non-academic sources (such as newspapers) in their literature review. Allowing the review to include popular press sources and interdisciplinary perspectives will help develop peer interest in the more narrowly focused analytical section of the paper. Inclusion of these alternative sources suggests, however, that students also develop skills in evaluating the reliability of such sources.

For more advanced undergraduate research projects that expect students to contribute to the knowledge base in economics, the review presents an additional challenge. Despite their skill at synthesising literature, students are more likely to struggle with the second goal of a literature review: providing a launching point for their own analysis and contribution. Students are much less practised at critiquing the process used by professionals, providing alternative interpretations of results generated and finding gaps in the literature that are ripe for exploration. Building off the modelling process in developing an effective economic question described above, students can use the same model articles to practise these critiques in cooperative settings.

3.1.4 Develop evidence

Research projects that require students to perform original work have an added set of hurdles to overcome. As those who practise the art of research on a regular basis, we often take for granted the process of analysis that is used in a research paper. Students, on the other hand, are likely to be facing this process for the first time and thus it is natural for them continually to fall back on what others have done rather than presenting original evidence and arguments. Convincing the audience of the validity of claims depends on the reasoning process employed and the evidence presented. Although most of the research that undergraduates attempt is applied empirical research and thus inductive in nature (using specific cases to reason out a general rule), both deductive (moving from general to specific) and warrant-based arguments (using underlying assumptions to make links between evidence and claims) can be employed as alternative methods of analytical reasoning (Greenlaw, 2006: 57–59 ).

Students also need to develop an understanding of these methods of reasoning and appreciate that audiences must be convinced by the evidence that is presented. Modelling this process using papers that are exemplars of acceptable formats will provide opportunities for instructors to address each of these issues. Students can be asked to break down the analytical section of a paper to identify the key components of the process and describe the evidence used. They can argue why they think the evidence presented is, or is not, sufficient and the degree to which evidence is open to alterative interpretations. The nurturing of students through this process provides them with the necessary understanding to attempt their own analysis. Figure 3 demonstrates a set of model questions that can be used to facilitate this process. Students are asked to read a model article and answer a series of questions that require them to identify the process of answering an effective economic question. A majority of the class discussion that is used as a follow-up activity to this assignment focuses on the types of evidence and the answer to the economic question. Students are placed into groups and asked to compile a comprehensive list of the evidence they have identified in the article. The final, and probably hardest, component is to have students begin to evaluate the evidence in terms of the degree to which it sufficiently answers the question posed and to identify potential caveats to the authors’ argument.

Figure 3: Developing the Evidence

The assignment:

Read KimMarie McGoldrick and Lisa Ford Voeks (2005) " 'We Got Game!' An Analysis of Win/Loss Probability and Efficiency Differences between the NBA and WNBA", Journal of Sports Economics, Vol. 6(1), pp. 5-23.
http://jse.sagepub.com/cgi/content/abstract/6/1/5

Goal of this assignment:

Look at Greenlaw's Table 2.1 (page 13) that describes "The Research Process in Economics" and the corresponding text. The WNBA is an example of an empirical paper and was developed in exactly this fashion. By discussing a research paper so early in the semester you will be more familiar with some of the components that you will need to complete in your own work. Thus, we will use this work to show an overview of the research process. I will also be discussing some of the hurdles that we had to overcome in conducting this research.

Questions:

  1. Provide a general description (3-4 sentences) of the purpose of this paper.
  2. What is the economic question that the authors are attempting to answer? Can you locate this specifically in the text? Where? What sub-questions are used to answer this main question? List these.
  3. What contribution to the literature does this paper make?
  4. What type of evidence does this paper provide in answering the economic question? Is this form of evidence consistent with that found in cited literature?
  5. Is the evidence convincing? Here you want to consider how much evidence is presented, the extent to which proxies are used, whether the data is complete or restricted in any way, etc. What additional evidence would add to the story told by the authors?
  6. What is the answer to the economic question? Can you locate this specifically in the text? Where?
  7. What was most confusing about the paper? Be specific and, if possible, provide page numbers.
  8. What was most clear about the paper? Be specific and, if possible, provide page numbers.
  9. Construct a well-detailed outline of this paper on a separate sheet of paper (typed). Bring two copies to class. Please note that a good outline does not have complete sentences and is detailed enough that you could actually write the paper from it if necessary.

We must allow for the possibility that a student may demonstrate good understanding and practice in the research process without managing to gather and analyse data that support their original hypothesis. Such ‘failures’ may turn out to be excellent learning experiences and students can be encouraged to use these as opportunities to develop a critical analysis of their own work in their conclusions.

3.1.5 Draw conclusions and compare conclusion to identified literature

Many students tend to use the final section of the research project to summarise key points already made in the body of the paper. A conclusion section should challenge students to place their work in the context of the larger body of knowledge. This requires that they first conduct critical reflection regarding the evidence that they have presented. Are proxies used in their analysis satisfactory? Does the evidence support their hypothesis? How does it compare to other conclusions in the literature on this topic? Is there an alternative interpretation? In light of these caveats, students should also be required to identify questions that remain unanswered and to indicate what they regard as priorities for further research. These observations might be linked specifically to what they would do if they had more time.

In order to facilitate this process, a few exercises can be implemented throughout the semester in order to model the critical thinking we wish to see in students’ conclusions. For example, when they are assembling their evidence students can be asked to develop a three-column table. The first column contains a description of each piece of evidence that when aggregated would be sufficient to support their claims. The second column would identify what ideal variables would measure, describe or quantify. The final column would list the actual data they have for each piece of evidence. By comparing the second and third columns of this table, students can begin to develop a sense of the extent to which they must state caveats to their research and suggest alternative explanations that cannot be ruled out given the limitations of their evidence.

3.1.6 Apply analysis to current policy issue

The value of doing research can be demonstrated to students through its connection with policy recommendations. In order to encourage students to begin making policy connections early in their research, one might implement a modelling exercise directly after the literature review section of their paper is complete. Students can be asked to identify all the relevant policy arguments included in the research cited in their review. A summary of these policies and the extent to which they might also apply in their own work can be a follow-up exercise that is used later when developing the conclusion.

3.1.7 Present research to peers and/or faculty

Effectively communicating key components of conducted research in front of peers and faculty may not be a comfortable task for students. Students often fall into the trap of trying to communicate everything they have done in a step-by-step description of their work rather than conveying the meaning of their findings. It is usually helpful to provide guides to the processes of summarising research and developing complementary visual aids. One logical starting point is for students to make a detailed outline of their paper and to use this as a guide for what to include in the presentation. As with professional presentations, however, one cannot hope to include all the work that has been completed for the project. Recommending that each sub-section of the paper be limited to one or two visual representations (such as an overhead or PowerPoint slide) with no more than four points on each is one step toward avoiding the problem of overwhelming presentations. Students should also be reminded of the need to keep their audience focused on key aspects of their work: the economic question, the way in which their question is related to previous work, the analysis of their evidence and their conclusions.