eWriting: ESL Writing Success
FIPSE Workshop on Pre and
Post Test Writing
Friday, November 7, 2003
Opening remarks and welcome note:
Michaela Tomova welcomed the participants and stated the objectives of the workshop – review of purposes of pre and post-tests, test writing strategies, discussion of samples of pre and post-tests, evaluation data collected from level 3 pre and post tests from LOs piloted in the classroom by faculty.
Dr. Marcia Cassidy defined the pre-test as “An instrument to measure starting points or pre-existing knowledge” and the post-test as “an instrument to measure learning that has taken place as a result of instruction”. She pointed out how the pre and post-tests are implemented for the purposes of the eWriting grant, i.e. to evaluate objectives attained, learning achieved and to gather evaluation data. Dr. Cassidy cautioned the participants for factors not controlled for, e.g. age, demographics, reading proficiency, etc. Further on she pointed out the characteristics of pre and post-tests for the eWriting lab– shared format, comparable level of difficulty, acceptable level of validity and similar length.
Dr. Greenberg provided hand-outs and raised the following issues for participants in the workshop to contemplate on:
· To see how much is being learned
· To judge the quality of an LO
· It’s a vital tool to make LOs better
· Provides feedback on the quality of an LO
Prof. Steven Donahue wanted to know whether there should be a correspondence between the number of items in the pre and post-tests and the number of items in the exercises within the LO. Dr. Greenberg clarified that the pre and post-tests should have enough testing items to cover the topic and content within the LO.
The idea of converting a post-test into the pre-test for the next LO is unacceptable for the eWriting lab since each LO is being designed as a portable, independent unit of an hour of on-line instruction.
Dr. Greenberg observed that both the pre and post-tests are vital tools in providing feedback on the quality of the LO. The evaluation data collected in the level 3 piloting classes so far does not show a significant increase in learning from pre to post test. The majority of level 3 LOs show an increase range of approximately 10-15% and the desirable one would be – a 30% increase in performance from pre to post-test. There are only individual LOs in level 3 that exhibit this increase in range from pre to post test performance of students.
Dr. Greenberg introduced questions to ponder: What kind of analyses/reporting may be done with the pre and post-test data? What conclusions may be derived from the analyses of pre and post-test data and what tools are available for item improvement.
He reminded the participants of what a good testing item is – students will answer incorrectly based on good distracters. A bad testing item is one that students who do not know the material answer correctly. He also raised the question of content validity and item analysis.
Discussion of samples of pre and post-tests from all six levels: All participants reviewed the samples of pre and post-tests in the hand-outs and recommended the following guidelines in pre and pos-testing writing strategies for the eWriting lab:
· Write approximately ten questions in the pre and post-test.
· Scramble the distracters in the pre and post-test.
· Cover the content of all teaching points and test whatever learning takes place within the LO.
· Choose the following types of activities for the pre and post-tests – binary questions, multiple-choice questions and matching activities.
Piloting strategies in the classroom were also discussed and flexible ways suggested as to how to proceed when piloting an LO in the classroom. An hour in the classroom is obviously insufficient to cover all activities, pre and post-test included, in an LO. Piloting an LO could be spread into two classes, the pre-test could be submitted in the class preceding the actual piloting of the content, since the latter is designed for at least an hour of on-line instruction. The post-test could be assigned for homework or completed in the next class.
Judith Garcia pointed out that we need sufficient feedback to show weaknesses of the LO and to work on areas that are questionable so that the online version will be a quality product.
Elizabeth Granados shared her experience with her piloting classes and told participants that there are time limitations in the classroom which allows her to employ flexibility in teaching the material, e.g. assigning pre, post-tests, activities and surveys for homework.
Outcomes and final decisions: