4. Types of classroom experiments

There are three ways of running classroom experiments: hand run, computerised and homework. For instance, we ran the above guessing game experiment by hand.

A. Hand run experiments

Hand run experiments can be as basic as asking for a simple raise of hands (or electronic polling). One can ask who would co-operate and who would defect in a symmetric prisoners’ dilemma. There are also simple experiments such as the previously discussed guessing game or the auctioning of a £1 coin. Two-by-two games can be played by having slips of paper in two different colours, one for each type of player.

Hand run games can become more sophisticated and require more interaction between the players. One of the first experiments run in a classroom was a hand run experiment by Chamberlin (1948), called the pit market experiment where students act as buyers and sellers (see case 2 below). The pit market experiment can be run with little effort using playing cards. (The pit market is like an old-style commodity exchange, where each commodity is traded around a pit.) The prisoners’ dilemma and public good games can also be run in the classroom using playing cards to cheaply and effectively distribute the pay-offs. More complex trading games can be used in order to illustrate the impact of asymmetric information on market efficiency. For more details see the following links:

A more involved hand run experiment is the International Trade Game (see handbook chapter on Simulations, Games and Role-play); although not based upon a research experiment, it is very useful at conveying a wide area of fundamental economic concepts.

Advantages

Hand run experiments have several advantages. Some are suitable for large lectures; others can take just a few minutes to run. Hand run experiments are often an excellent way to engage students, since the interaction is face to face (as in the trade game or the pit market) and some can involve physical activity (as with flower pot/tennis ball http://www.bized.co.uk/educators/16-19/economics/firms/lesson/dimreturns.htm or http://www.economicsnetwork.ac.uk/showcase/hedges_tennis).

Disadvantages

There may be reasons why you may want to use forms of experiments other than hand run experiments. One difficulty is that certain hand run experiments may require careful preparation, including room structure. They also may require assistants, volunteers or another lecturer. This requires careful coordination beforehand. From our own experience it is quite dangerous to try to ‘wing-it’. During one experiment in a class of 300, we were not organised about how to tabulate the results. Both of us had intended only to sample the data, but had not agreed how to do so. We were later told that in the middle of the class students were taking bets on which one of us would slug the other first.

Hand run experiments may require several practices before the lecturer gets the procedure down to a fine art. This may cause a variation in the student experience. There are also a limited number of rounds for which one can run within a lecture or session. Data collection and entry into an Excel spreadsheet can take time and effort. It is quite easy for the data to get lost. When the experiment involves large groups of students, feedback may be delayed for instance until the next lecture. It is also quite easy for students to avoid participating.

Helpful hints for hand-run experiments

  • At the beginning of the semester/term, cut simple strips of paper (you may want to use two colours).
  • Bring several plastic bags from home to collect answers.
  • Give verbal instructions/display question.
  • It is often sufficient to sample and evaluate only a few answers. Or one can randomly select some students to participate in one round of the experiment.
  • One can display last year’s results rather than wait until the data are evaluated.
  • Many experiments can be run using a deck of playing cards (see Holt, 2007).
  • Know what you are doing beforehand, particularly when working with assistants.

Case 2: Pit market (hand run)

This case study was written by Bradley Ruffle. A pit market is a suitable experiment for any level of student – sixth form up to postgraduate. It is particularly suitable for microeconomics, industrial organisation and public economics.

The primary benefit is to teach students the relevance and robustness of the competitive-equilibrium solution. Extensions allow for the demonstration of price floors and ceilings and the tax-liability-side equivalence theorem. The pit market is designed to be run by hand. For a computerised experiment that demonstrates the competitive solution, a double-auction market is the nearest equivalent.

Procedures

Prior to the experiment, prepare two sets of cards; one from which buyers' valuations are drawn and the other from which sellers' costs are drawn. You can use playing cards (see Holt, 1996) or prepare your own with any numbers you like on them. Make sure you choose the cards ahead of time so that the resultant supply and demand curves overlap where all or almost all of the units may be traded at a profit at the competitive price.  

When the students arrive, divide them into two groups of buyers and sellers. Ensure that there are at least four sellers and four buyers for convergence to the competitive outcome. The groups of buyers and sellers need not be of equal size. Give each student a record sheet (included along with instructions for participants in the downloadable file (link below)) to track their progress. Distribute randomly one or more cards to each of the buyers and sellers from their respective deck of cards. After everyone has received one or more cards, allow students to enter the pit (a large open space in the classroom) where they freely negotiate with one another. When a buyer and a seller agree upon a price, they report their negotiated price to one of the experimenters and turn in their cards face down. To speed up convergence to the competitive equilibrium, recruit a helper from among the students to write the negotiated price on the board for all to see. Have a timekeeper announce the time remaining at regular intervals. At the end of the round, collect all unused cards, shuffle and redistribute randomly for the next round.

Discussion of results

In an introductory microeconomics course, the pit-market experiment can be conducted prior to teaching supply and demand and the competitive equilibrium, to motivate the relevance of these topics. I prefer to conduct it immediately following the lecture on these topics. Begin by showing students the results from their experiment in a transactions graph (software downloadable from the link below).

Figure 3

Also, show them the distributions of buyers' valuations and sellers' costs and ask them to explain why prices converged to the particular observed levels (27 to 28 in the example above). Surprisingly, in a principles course, you will rarely, if ever, hear the correct answer. Instead, students will claim that the observed prices ‘are the average of all of the cards,’ or ‘at these prices buyers and sellers earn the same’. Use asymmetric supply and demand curves (like those in the figure below upon which the transaction prices above are based) in order to reject these explanations and focus on the profit maximisation motive and the forces of supply and demand.

Figure 4

Review the textbook assumptions underlying the competitive-equilibrium model and discuss why some of these assumptions are unnecessary for convergence (e.g. full information and the inability to collude or form cartels) and others are imprecise (e.g. ‘large’ numbers of buyers and sellers). Market efficiency, alternative market institutions and the role of displaying transaction prices on the board (or information more generally) are additional topics for class discussion.

Extensions

Expect prices and quantities to converge to the competitive equilibrium within three or four rounds. If you have additional time, you might want to shift either the demand or supply (this requires a careful change in playing cards used), then it is fun to ask the students to guess what you did (based upon the changes of price and quantity). Also you might try imposing a price floor above the competitive price or a price ceiling below it. More interestingly, announce an n-unit tax on the buyers imposed on each unit traded and listen to them groan. The following period replace the tax on the buyers with an (equivalent) n-unit tax on the sellers. Afterwards, you can display to students the outcomes from these two tax periods; namely, that the net prices paid and received and the quantity traded are equivalent in the two tax treatments and the incidence of the tax depends solely on the relative elasticities of supply and demand. See Ruffle (2005) for a further discussion of experimental tests of tax incidence equivalence and the analogous theorem for subsidies.

Further reading

There are two textbook chapters that describe how to run pit markets: Bergstrom and Miller (2000) and Holt (2007). In addition there are two articles describing the procedures Holt (1996) and Ruffle (2003).

Downloads

 

B. Computerised experiments

When an experiment requires many rounds and complicated matching schemes it is easiest done on a computer network. A typical example is the Bertrand duopoly game where it is best to use two different types of matching (fixed/random) and/or two different group sizes.

A large selection of computerised experiments is available via Charlie Holt’s Veconlab, Econport and our own FEELE website. Typing any of these keywords into Google will lead you directly to the relevant site. Veconlab offers the most information and help for a beginner. Once this system is familiar, it is easy to switch to our site which is deliberately of a similar design. Econport offers the best market experiment software, in particular for some experiments on financial markets. It is well-documented and easy to understand. How well it works in your computer lab depends on the ingenuity of your university IT group. The more complex and convoluted their firewall system, the less likely it is that the JAVA applets Econport uses will work and communicate without problems. This system has to be tested in every room where you intend to use it, preferably by your computing officers and IT services.

Advantages

Many of the most popular classroom experiments are offered via the Internet for free. Apart from this, the big advantage of computerised experiments is their availability and the ease at which beginners can get started. The student experience tends to be uniform. The results are available immediately and can easily be distributed to students for evaluation. In many cases, there are tools provided for simple analysis of the results. A large number of rounds can be run as well as several treatments. An experienced instructor or teaching assistant can handle a computer classroom alone, although it is easier, particularly if there are more than ten students, if one instructor concentrates on the software and another on the students.

Disadvantages

One main disadvantage is that the experiments are standardised. While there are some parameters a lecturer can change, there is not the broad opportunity for drastic innovation that a hand run experiment offers.

A second problem is the requirement to have a special computer room (and it helps to have a projection screen for the instructor’s computer). There is the usual hassle of reserving the room and organising the students to meet there. On a practical level, there is a limit to how many students can fit in the room. When there is a large class, you may have to split it up into several groups. This creates an additional teaching burden. The alternative is to use tutorials, which may have a higher opportunity cost.

Another problem is that in many cases the experiment tends to run as fast as the slowest student. You constantly have to control the monitor program to see how the experiment is progressing and may also have to check the screens of individual students. Students often check their email or Facebook accounts and therefore the experiment becomes stuck. If the experiment runs too slowly, boredom may set in, creating a free-rider problem for students paying attention. Once things are started and running smoothly, there is also the danger that the instructors may check their email as well and not realise that there is a delay. Luckily there is now a KIOSK program that keeps the computer locked in the experiment (see the hints).

Finally, there is a risk of technical problems, such as software bugs, network failure and IT-related problems. This is particularly true for experiments using technically more advanced software. However, such problems have only rarely occurred with our software and that of Veconlab, since they require only a standard web browser.

Helpful hints for computerised experiments

  • Place two students per computer to make decisions jointly. The students will discuss their options and this will typically lead to better decisions; they will ‘catch on’ quicker and have a deeper learning experience. It also helps foreign students who have difficulties with the oral or written instructions. Moreover, the amount of web surfing, etc. will be reduced. It also eliminates the danger of holding up the whole class with a single toilet break.
  • Give instructions by email or handouts beforehand. Again, this helps non-English speakers and students with reading difficulties, in particular dyslexic students. Getting students to read and understand the instructions takes substantial time out of the experiment.
  • Let students in different sessions play the same treatments in a similar order. Otherwise, they do not have the same learning experience.
  • Try setting up and playing the experiment beforehand. One can do this by setting up a smaller number of players in different web browsers on a single computer’s desktop (sometimes one needs different browsers rather than different tabs within a browser).This would help you not only decide if you like the experiment, but help decide what treatments and parameters to run. It is also important to have you see what the students see, for discussion and questions.
  • Try configuring the experiment beforehand. Doing so saves time and reduces the number of errors or restarts. There is, however, a limit to what is possible, since many experiments need the full number of participants (or terminals) to be logged in before the experiment can be started. Luckily, both Veconlab and our website are currently enhancing the possibilities to change the number of participants on the spot.
  • Prevent email checks by students. The terminals can be run in KIOSK mode which prevents students from using the computer for anything but the experiment. Via our FEELE website you can start the KIOSK program to either run our experiments or those on Veconlab (simply google ‘feele kiosk’; this requires Internet Explorer).
  • Distribute handouts explaining how to log in; reduces 'finger trouble' and saves time.
  • Number the handouts beforehand and distribute one per computer; avoids headcount errors when configuring subject numbers.

Case 3: The hold-up problem (computerised)

The hold-up problem is central to the theory of incomplete contracts. It shows how the difficulty in writing complete contracts and the resulting need to renegotiate can lead to underinvestment. We describe here the design of a simple teaching experiment that illustrates the hold-up problem. The model used is a simple perfect information game. The experiment can hence also be used to illustrate the concept of subgame perfect equilibrium and the problem of making binding commitments. In contrast to other perfect information games like the ultimatum or the trust game, the backward induction solution predicts well in our experiment. It is hence a good experiment to conduct in order to illustrate game theory before models where fairness considerations are discussed.

The hold-up problem (see Hart, 1995) results from situations where it is difficult to write complete contracts. When one party has made a prior commitment to a relationship with another party, the latter can ‘hold up’ the former for the value of that commitment. It is argued that the possibility of hold-up can lead to underinvestment in relationship-specific investments and hence to inefficiency. An often quoted (but also sharply disputed) historic example concerned the US car industry. Fisher Body had an exclusive contract to supply body parts for the cars of General Motors. They were the only ones who could deliver the parts according to the specifications needed by GM. In the 1920s there was a sharp increase in demand that exceeded all expectations that were held at the time when the contract was written. It is claimed that Fisher Body used this unforeseen development to hold up General Motors, amongst others, by increasing the price for the additional parts produced.

In our highly stylised game there are two players, the Buyer (aka GM) and the Supplier (aka Fisher Body). In a first stage, the Buyer makes a relationship-specific investment (i.e. decides to set up their line of production such that it depends on specific car body parts delivered by Fisher Body). Then (due to the unforeseen increased demand), the Supplier has the opportunity to raise the price (for the additional demand). In case the price is raised, the Buyer can, at their loss, change the Supplier.

We run two treatments of this game which differ only by one parameter. We sketch here the computerised version available via our FEELE website. More details, and a hand run version, are discussed in (Balkenborg, Kaplan and Miller, 2009a, b). In both treatments it is optimal for the Supplier to hold up the Buyer and for the Buyer to accept the hold-up. In the first treatment it is optimal to invest even if there is a hold-up while in the second treatment it is better not to invest due to the hold-up. We choose this set-up because it allows students first to learn that there will be a hold-up and then to experience the economic consequence of underinvestment caused by the hold-up problem. We tend to run 8–10 rounds of each treatment with a different random pairing for each round. An even number of players is needed.

The first game is given in game tree form in the following graph.

Part 1 Payoffs: (Buyer, Supplier)

If no investment is made, both players get zero. The investment costs 500 and the gross value produced is 1500. In the initial contract all surplus goes to the Buyer and they get 1000 while the Supplier makes zero profit. The Supplier can hold up the Buyer by raising their price by 750 and leaving the Buyer with 250. The Buyer could change the Supplier, but this hurts everybody. The Buyer loses their investment and the Supplier loses all their business with the Buyer.

Once the number of players is determined, we can complete the set-up of the experiment and give the students the access code to log in to the experiment via our website. They are then assigned the roles of Buyers and Suppliers and can work through the computerised instructions. In each period the program randomly matches Buyers and Suppliers. Sequentially the game is then played, with first the Buyer deciding whether to invest, followed, if applicable, by the Supplier’s decision whether to raise the price and the Buyer’s decision whether to change the Supplier

In the following screenshot the Supplier is asked to keep or raise their price. The design of the screen is very simple to keep the emphasis on the basic decision.

Figure 6

Typically subjects learn quickly to play the backward induction equilibrium. This means that the Buyer learns that their threat to change the Supplier is ineffective because it is too costly, and therefore the Buyer is held up, i.e. the price is raised. It still pays for the Buyer to make the investment.

This changes in the second treatment. The only number we alter is the cost of the investment which is raised to 1000. As a consequence, the Buyer loses from the investment if they are held up. The payoffs are now illustrated in the following game tree.

Part 2 Payoffs: (Buyer, Supplier)

Figure 7

In part 2 of the game the Suppliers are typically held up when possible, and the investment is made much less often.

The next figure shows how often each possible outcome arose in the experiment.

Figure 8

Notice that there is a minority of Buyers who switch Supplier after the price has been raised. (This did not happen in all the sessions we ran.) The rationality of these Buyers is an important point for class discussion: what were they trying to achieve?

The second figure shows the development from period to period.

Figure 9

Case 4: Price competition (computerised)

One of our favorite computerised experiments is on Bertrand competition (available on both Veconlab and FEELE). We have had success running this experiment with students from sixth form level up to corporate executives. Students act as firms in a market. Each period in time, they choose prices. The customers (played by the computer) go to the firm with the lowest price (in the event of a tie, the demand is split equally). Each firm has constant marginal cost and, given the demand of the consumers, the Nash equilibrium is for firms to charge a price equal to their marginal cost, leading to zero profits (see Kaplan and Wettstein, 2000).

You can see the results of the experiment in the figure below. These results are typical. With two firms in a repeated situation, the prediction of perfect competition fails. Even without explicit communication, firms can collude. To quote an anonymous student:

‘I learnt that collusion can take place in a competitive market even without any actual meeting taking place between the two parties.’

This changes quite drastically for a larger number of firms and random matching. Here, the competition is fierce and the profits are driven out of the market. To quote another student:

‘Some people are undercutting bastards!!! Seriously though, it was interesting to see how the theory is shown in practice.’
Figure 10

It is especially important in this experiment to display the selling price in addition to the average price chosen since that indicates the profits in the market. Only by seeing the selling price can one clearly see the strength of the equilibrium prediction.

Bertrand competition with complements

A lecture on industrial organisation will discuss the advantages and disadvantages of different market structures. A counter-intuitive concept is that more competition is not always better. Duopoly may be worse than monopoly. This is the case when a monopoly sells two complementary goods and is then split into two firms to sell each good separately. The theoretical analysis shows that consumers pay a higher price for a pair of commodities after the split. In a crude analogy, being robbed twice is worse than being robbed once for the consumer. The analysis is clearly relevant for competition policy: for instance, the decision on whether to split Microsoft up into two separate companies, one that sells the Windows operating system and one that sells Microsoft Office (Excel, Word, etc.) Krugman (2000) argues just this in his column entitled ‘Microsoft: What Next?’ In agreement with the economic analysis, the US government agencies decided against such a split.

To convey this concept, FEELE provides a computerised experiment based upon a similar hand-run experiment by Beckman (2003).

Looking at the following graph of results, we started students in a monopoly situation facing a demand of 15p and a constant marginal cost of 3p. The profit maximising price is 9p. Students found this price fairly rapidly. When we broke up the company into two separate companies producing complements and competing in a duopoly, there was a clear increase in the price to over 10p (the equilibrium price is 11p).

It is of particular teaching and learning value that the model is just a seemingly minor variation of the standard model of price competition which we use in microeconomics. (The standard model uses perfect substitutes instead of perfect complements.) For the standard model one observes sharp cut-throat competition which erodes profit possibilities: a completely opposite result.

Figure 11

One can run a combination of both Bertrand games against former subjects with the following:

http://www.projects.ex.ac.uk/feele/feele_experiments/subject_access.php?quick=bertrand

Case 5: Bank runs (computerised)

Once relegated to cinema or history lectures, bank runs have become a modern phenomenon that captures the interest of students. Now a simple classroom experiment based upon the Diamond-Dybvig Model (1983) can demonstrate how a bank run, a seemingly irrational event, can occur rationally. The computerised version of this experiment is available from our FEELE website.

This model captures elements of what a bank does. We will focus on the conversion of long-term loans (mortgages) into short-term deposits. It is this conversion that leads to the fundamental problem of bank runs.

In the model there are depositors and a bank. There are three time periods: yesterday, today and tomorrow. Depositors placed money (say £1000) in a bank (yesterday) before learning when they need the money. Depositors either need their money today (impatient) or tomorrow (patient). There is a 50% chance of being either type. The depositors that need money today get relatively little utility for the money tomorrow. The depositors that need their money tomorrow can always take the money today and hold onto it.

The bank has both a short-term and a long-term investment opportunity for the money. The short-term investment (reserves) is locking the money in the vault. This investment returns the exact amount invested. The long-term investment returns an amount R tomorrow. It is illiquid and returns only L<1 today. The depositors that invested £1000 yesterday have a contract with the bank. They can withdraw their money today and receive £1000 or wait until tomorrow and receive R*£1000. The bank meets these potential demands by taking half as reserves and half in the long-term investment.

If all the depositors withdraw the money according to their types, then the bank will meet all the demands. In this case, each depositor has an incentive to indeed withdraw according to their type. Hence, all impatient depositors withdrawing today and all patient depositors withdrawing tomorrow is a Nash equilibrium.

While the contract is fulfilled in this Nash equilibrium, in other cases the bank cannot always remain solvent. If too many depositors try to withdraw today, it will not be able to meet the contract tomorrow. It is then optimal for all depositors to withdraw today. This other equilibrium is a bank-run equilibrium.

The experiment is then to see under which conditions a particular equilibrium arises.

Results from a computerised session

The following figure shows the results of a classroom experiment run in Exeter on a single group of 18 students.[note 2] Investor types (roles) were randomly re-allocated at the start of every round, with 9 students being type A (impatient) investors and 9 students type B (patient) investors. The experiment lasted 23 rounds and there were 3 treatments. In the first treatment, lasting 8 rounds, conditions were set for R=2 and L=.5 (we call this ‘normal conditions’). Toward the last few rounds of this treatment, the students settled into the normal equilibrium. Type As withdrew today and type Bs withdrew tomorrow. In the second treatment, lasting 10 rounds, we had R=1.1 and L=.11. We might refer to this as a ‘credit crunch’. Tight conditions for the bank: not much leeway if depositors try to withdraw early. In this treatment, there was a run on the bank. In the third treatment, lasting 5 rounds, we also had R=1.1 and L=.11, but payments were halted after 9 depositors withdrew from the bank early. This suspension stopped the run on the bank. There was an instant effect that steadily improved.

Figure 12

There are many topics for lively discussion. Obviously, it is worthwhile to connect the experiment to current events. Another topic is to discuss various ways to help avoid a bank run (suspension, deposit insurance, the government stepping in). While not in the experiment or model, this leads to discussion about moral hazard.

Note

[2] There were 18 computer terminals and the students were paired up with an average of 2 to a computer.

C. Homework experiments

Homework experiments are simply classroom experiments that are meant to be played at home instead of during class hours. The most basic is a simple one question and answer format with feedback and summary of the results discussed in class. There is an elegant website by Ariel Rubinstein that is designed especially for this purpose. A slightly more complicated homework experiment is to run a more advanced individual choice experiment with some immediate feedback (for instance we have a computerised Monty Hall problem that is played several times). Finally, it is possible to have students play against a fictitious player such as a robot playing a particular strategy or against prior human players. The first example we know of using this option is Charlie Holt with his traveller’s dilemma experiment available on Veconlab. We now offer for most of our experiments ‘quick log-in’ versions where you play against a past group of participants.

Another innovation by Charlie Holt is running the standard multi-player experiments by having students log on from home at a specific time in the evening. There are also experiments (such as prediction markets) that can be run over several weeks. In fact, such homework experiments (such as the Iowa political stock market) predate the web.

Advantages

The main advantage of a homework experiment is that it can save lecture and tutorial time. There is very little hassle and one does not have to worry about time limits. They provide great flexibility to both students and lecturers.

Disadvantages

Overall, the lecturer has little control with homework experiments. There is no guarantee the student is the one playing the game. If the experiment requires interaction among subjects, there is no means to stop collusion. If it is an individual choice experiment, one student can advise another. Without additional incentives the overall participation rate can be low. Though for some experiments we have had the opposite problem of some students playing the game several times in order to beat the previous performance. Currently there is still a limited variety of home-run experiments which every student can do by him or herself. One can invest the extra co-ordination of running group experiments at a specific time. Even with these one needs to keep the group size small so one player does not hold the rest up (toilet breaks are problematic here).

Helpful hints for homework experiments

  • Use some sort of incentive for participation.

 

Case 6: Price discrimination (homework)

In essence students are given here the repeated opportunity to select the best price schedule when various forms of price discrimination are possible. The student is the seller who can sell up to two identical items to each of two different buyers. Each item costs £5 to produce. The computer takes the role of the two buyers who have the following valuations for the item:

 
1 item
2 items
Buyer A
£20
£20
Buyer B
£30
£40

As illustrated in the table, the second item adds no value for buyer A, but a value of 10 for buyer B. Twenty rounds are played. In the first five rounds the same, uniform price has to be set for each unit sold to any buyer (uniform price, no price discrimination). In the next five rounds different prices may be charged to different buyers, but the same price must be taken for each unit (third-degree price discrimination). In rounds 11–15 the prices have to be the same for both buyers, but different prices can be charged for different units (second-degree price discrimination). Finally, in the last five rounds different prices can be taken per unit and per consumer (first-degree price discrimination).

It is best to let students do the experiment before price discrimination is discussed in the lecture. One can then discuss each scenario in a classification of price discrimination. The lecturer can ask the students how much money it was possible to make in each scenario and why. It will become transparent why the detailed form of price discrimination matters.

In analysing results in second year microeconomics, 90 students participated in our experiment. Only two managed not to get the right answer ever in the first five rounds. The next five rounds are more difficult and about 25% have difficulties in finding the correct answer. Rounds 11–15 are the hardest and only 50% get it right most of the time (i.e. at least two times out of five). There is only a slight improvement for the last five rounds where about 40% of the students never get a profit above 40 and hence do not see how to get a higher profit out of buyer B by discouraging them to buy a second unit. Admittedly, we did not give incentives for good performance and so we see that there is a substantial fraction of non-serious answers (about 20%). Still, it is revealing to see where some of the students have serious difficulties to which one can respond in a class discussion