The Monash Critical Thinking Study
1. Web-based Argument Mapping
Description
One view about how critical thinking can best be taught is represented by the Quality Practice Hypothesis (van Gelder 2001). According to this theory, acquiring expertise in critical thinking, as in other areas, requires large amounts of deliberate practice. Deliberate practice must be:
1. Motivated: the student should be deliberately practicing in order to improve skills.
2. Guided: the student should have some way of knowing what to do next.
3. Scaffolded: in the early stages, there should be structures preventing mistakes
4. Graduated: tasks should gradually increase in complexity.
5. Feedback provided: the student should have some way of knowing whether they are doing the right thing or not.
The use of computer assisted exercises can help to achieve these goals, without expensive one-on-one tutoring. This is the fundamental idea behind the use of argument-mapping software such as Reason!able for improving critical thinking skills. Students are given many natural language arguments to analyse and must create an argument map to represent the structure of the argument. An argument map is a graphical representation of the logical structure of an argument - the way in which premises, intermediate steps and the final conclusion all fit together. Consider, for example, the following argument (from Fisher, 2001):
Radioactive elements disintegrate and eventually turn into lead. If matter has always existed there should be no radioactive elements left. But there is still plenty of uranium and other radioactive elements around. This is scientific proof that matter has not always existed.
This argument can be represented as an argument map as follows:
|
|
|
An example argument map |
Computer software can be used to help students create and manipulate these argument map diagrams; something which is not so easy using pencil and paper methods. Text can be typed into boxes and edited, supporting premises can be added, deleted or moved around and so on. In some systems, evaluations of the argument (assessments of the truth of premises and strength of inferential connections) can also be incorporated into the argument map.
The software itself supports the creation of argument maps in a way that is both guided and scaffolded. Gradually increasing difficulty and complexity is arranged for by the creator of the exercises.
A significant problem remains however - that of providing appropriate feedback to the student. Typically, tutors will provide feedback to students on whether their argument maps are correct or not. With large classes of course, this can be difficult; there may not be enough time to give every student the feedback they need. One solution is to provide model answers, so that students can assess themselves. However, students might not be able to work out why their answer is wrong and the model answer correct. Furthermore, they may not be able to tell when a difference between their map and the model answer is an important difference.
With this in mind, we investigated computer assisted argument mapping exercises where the computer is able to automatically provide instant feedback to the student as they construct a map of a given argument.
Figure 1 shows a simple example, using the software we designed for this course. The window in the top left hand corner contains the text of a simple argument. The student's task is to construct a map of the argument, using the mouse to select the appropriate segment of text and then clicking on the buttons below. The argument map gradually appears in the larger pane to the right.
|
|
| Figure 1. An argument ready for mapping |
In figure 2, the student has selected the segment of text that they take to represent the conclusion of the argument. The student then clicks on the button labelled 'Conclusion' to indicate their choice. In this case, the student's identification of the conclusion was incorrect, so a small red cross appears in the boxes underneath the buttons. The student knows they must think again. After re-reading the argument, the student selects the correct conclusion in figure 3. This time a green tick replaces the cross, so that the student knows they have correctly identified the conclusion. A box representing the conclusion now appears in the right hand pane.
|
|
| Figure 2. Conclusion misidentified |
|
|
| Figure 3. Conclusion correctly identified |
The next step is to identify the premises supporting this conclusion. In figure 4, the student has correctly identified the word 'since' as a premise indicator. Again, a green tick has appeared in the box to the right of the first tick, to indicate that this identification is correct (the word is then also underlined). The identification of the premise indicator provides a clue that that the text immediately following is a premise. In figure 4 the student has selected all of the text following the word 'since' and clicked on the 'Add premise' button. This is not correct however - there are actually two separate premises in this example. A red cross appears to indicate that the student has not correctly identified a premise of the argument.
|
|
| Figure 4. Premise misidentified |
In figure 5, the student has correctly identified one of the premises, by selecting the appropriate text and clicking on the 'Add premise' button (This adds a supporting premise underneath the currently selected box in the right hand argument map pane). A tick appears to indicate that this is correct and a box representing that premise is added to the argument map pane.
|
|
| Figure 5. One premise correctly identified |
The fact that only one box has yet to be ticked tells the student that they have only more component of this argument to identify. In addition, some of the buttons have now been disabled, so the student knows that the remaining item is either a co-premise or a supporting premise. This is the main way in which the software provides guidance and scaffolding.
In figure 6, the student has selected the appropriate segment of text and clicked on the 'Add premise' button. The function of this button is to the selected text as a supporting premise below the currently selected box in the argument map pane (in this case, the premise 'Wealth is the basis of political power'). This choice is incorrect however; the selected text does not support that premise, but rather acts as a co-premise supporting the main conclusion. So a red cross appears in the final box, to indicate that the student has made a mistake.
|
|
| Figure 6. Final premise misidentified as a supporting premise |
Finally, in figure 7, the student has correctly identified the selected text as a co-premise. The co-premise is added to the argument map and a green tick appears in the final box, informing the student that they have completed this exercise and can go on to the next one.
|
|
| Figure 7. The argument fully mapped. |
The exercises gradually become more complex as the student progresses through the course. An argument with a more complex structure is shown in figure 8.
An additional feature is the ability to incorporate unstated premises ('assumptions') into the argument maps. This is done by clicking on the 'Assumption' button and selecting the assumption from a list (see figure 9). Some exercises also incorporated a multiple-choice question, which asked for an evaluation of the argument.
|
|
| Figure 8. A more complex argument map |
|
|
| Figure 9. Adding an unstated premise (assumption) |
Click here to try out some sample exercises for yourself (Java plug-in required).
Procedure
Ten sets of exercises, consisting of 5-10 arguments for analysis were provided. These were made available on the WebCT site for the course. Students worked on these exercises in their scheduled tutorials, which took place in computer labs. Students worked at their own pace. On average, approximately 30-40 minutes each week were spent working on these exercises. The tutor was present to offer help if required. The exercises were made available on the WebCT website for the course. This allowed students to complete the exercises at home if they did not finish them in class. Several students took advantage of this opportunity, although the exercises were not graded.
Students completed the CCTST during the first half of the scheduled 2-hour tutorials for the course. The pre-test was completed in the first tutorial (week 1) and the post-test in the final tutorial (week 13). The tests were completed under examination conditions, as outlined in the test manual. Students were not informed of their test scores until after the end of the course. All students were given form A of the CCTST for the pre-test and form B for the post-test.
Results
Students showed a statistically significant improvement in critical thinking scores on the CCTST.
Average improvement: 14%. Effect size: 0.45 standard deviations. (n = 43)
GSA data for this semester is not available.
Sample characteristics
| Semester 1, 2004 sample | |
| Sample size | 43 |
| Sex | Female: 27 Male: 16 |
| Age (years) | Range : 17 (1) - 55(1) Median : 20 Mode : 18 |
| Year level | N/A |
| Degree | Arts: 18 (41.9%) Arts + (Education, Engineering, IT , Law, Social work): 5 (11.6%) Exchange: 8 (18.6%) Engineering: 3 (7%) Medical science: 2 (4.7%) Science: 2 (4.7%) Commerce/Economics: 2 (4.7%) Performing arts: 2 (4.7%) Communication: 1 (2.3%) |
Gains on critical thinking tests
|
CCTST (Max. score = 34) | |||
| Mean |
95% confidence interval | Standard deviation | |
| Pre-test | 18.209 (53.5%) | [16.73, 19.69] |
4.8 |
| Post-test | 20.233 (59.5%) |
[18.6, 21.87] |
5.32 |
|
Gain |
2.02 | [0.74, 3.29] | 4.1432 |
| Effect size | 0.45 | [0.17, 0.74] | |
| Proportional gain | 13.9% | [6.4, 21.3] |
24.25 |
Effect sizes calculated using pre-test standard deviation estimates of 4.45 CCTST points.
Proportional gain is the gain score score expressed as a percentage of how many points a student could have gained (or lost) relative to the maximum test score (or pre-test score if gain was negative).
Comparison with other studies
 |
| Gains for all studies measured using the CCTST. |
| * Two semester course. |