First Evaluation Report

FIPSE eWriting: ESL Writing Success

A project of

Miami-Dade Community College

Kendall Campus

ESL/Foreign Language Department

Developed under a grant from the Fund for the Improvement of Postsecondary Education (FIPSE)

Report prepared by:

Barry Greenberg, Ph.D.

Feedback Technologies, Inc.

June 26, 2003

I. Introduction:

In September 2002, Miami-Dade Community College (M-DCC) received a three-year FIPSE grant for the purpose of creating a unique online writing program for ESL students in need of improving their writing skills. The program is designed to develop a six-level, modularized, competency-based, interactive online writing lab and combine it with the anytime, anywhere access provided by the Internet.

The stated goals of the project are: 1) To develop 6 online English as a Second Language writing courses, ranging from beginner to advanced, by the end of the second year. Each course will contain 90 hours of instruction to fulfill the double contact hour requirement of the lab program.; 2) To prepare students in the pilot courses to achieve scores on their exit exams that are 10% higher than those of the control group in classroom-based courses; 3) To increase student retention in the online ESL writing lab classes by 10%, as compared to traditional ESL writing lab courses at the same level; 4) To increase students’ writing competence, preparing them a) to pass the writing portion of the CPT (College Placement Test) and b) to write successfully in college-level classes.

II. Purpose of the evaluation:

The evaluator was retained in May, 2003 to design and implement evaluation strategies suitable for each year of the program in keeping with FIPSE requirements and to prepare a year one report to be completed in July, 2003. This document serves as the year one evaluation report.

In the time since he has been retained, the evaluator has met with the Program Director and Grant Manager once, and with the Grant Manager on several occasions, has attended a regularly scheduled FIPSE Focus Group workshop on Learning Objects discussing progress and plans for grant materials, and the evaluator has received and reviewed extensive materials relating to the purposes and progress of the Program.

The purposes of the phase one evaluation are: 1) Document the extent to which intended strategies have been implemented; 2) Describe any difficulties experienced in the implementation of the strategies; 3) Describe factors which facilitate or inhibit program progress; 4) Indicate likelihood of completion of this phase of the program within stated time frames; and 5) provide plans for the evaluations to be conducted for years 2 and 3 of the program.

III. Evaluation Findings:

1. Document the extent to which intended strategies have been implemented

The Grant Manager provided documentation on all activities completed thus far including:

a. Training attendance records of project content developers (there were a total of 5 meetings between January and June, 2003 focusing on design, development and piloting of course materials including Learning Objects, pre and post testing, and various content issues)

b. Descriptions of training objectives and sample materials

c. Responses of participants to online survey of training effectiveness

d. Descriptions of (Writing Lab, levels 3,4,5 and 6) EAP grammar competencies.

e. List of suggested content-related themes and topics of online courses

f. Description of Focus Group on tools for creating online instruction with sample learning objectives developed by design teams; also provided with survey of participant reactions to class

g. Agendas and sign-in sheets January – June, 2003 project meetings

h. Suggested first-year program evaluation plans provided by FIPSE consultant

i. Learning Objects Content Elements Checklists for Levels 3, 4 and 5

j. Sample completed Learning Objects(from pre-assessment through post-assessment) being prepared for pilot testing; and

k. Summary tables for each year one project goal specifying activities undertaken with respect to each task established for each goal along with expected outcomes and an indication of whether the task has been completed.

l. Project web-site addresses, reflecting progress of content development

It is clear from an examination of the materials presented as well as discussions with the Grant Manager that the intended strategies have been implemented as originally specified. Most importantly, the Learning Objects, which represent the foundation of the online materials to be developed, are in process. Altogether, a total of 45 Learning Objects are to be developed for each of the six levels of the writing sequence. Piloting of the Learning objects started on June 2^nd, and so far from this total, 11 Learning Objects have gone through piloting having been reviewed by students and other faculty. Two of the 11 have been revised already and the other nine are being worked on using the data collected. The completed Learning Objects are then reviewed by technical support personnel to add to and enhance the already placed computerized portions. This development/piloting process will continue until all Learning Objects are ready for use by students.

The summary tables for each year one project goal provide specific information about how each of the Grant’s significant activities has been implemented. Careful examination of the tables reveal that all activities have been completed on schedule; participants have been very satisfied with outcomes of training; and that the project is on-track for completion of all activities.

2) Describe any difficulties experienced in the implementation of the strategies

The evaluator has not been made aware of any difficulties experienced in this regard. All strategies have been implemented as planned with no indication of any problems encountered which could not be readily addressed.

3) Describe factors which facilitate or inhibit program progress

The principals involved with the Grant appear totally committed and extremely knowledgeable about all processes. Steps have been taken to document all major activities with appropriate timelines indicated. The principals have kept all key people informed of relevant ongoing events. Project faculty whom the evaluator have met appear very familiar with their roles in accomplishing the goals of the project, and eager to contribute to the tasks at hand. Perhaps the greatest factors facilitating program progress is the commitment of the institution itself. As was noted in the Grant proposal,

“The eWriting project was not initiated by a grant opportunity, nor will it disband at the end of one. Two years ago, Kendall ESL faculty began to create online materials such as those described in this proposal, and they will continue after the project period has ended…MDCC is committed to the further development of the eWriting program. The on-line courses created through the grant will become permanent course options at the College where it will ultimately expand to serve over 12,000 ESL students college-wide.”

The activities undertaken to-date suggest that this commitment is real and that it serves to stimulate all players in the developmental process.

4) Indicate likelihood of completion of this phase of the program within stated time frames

Under the leadership of the Project Director and Grant Manager a great deal of work has been accomplished in a very short period of time. Currently, teams are in place to finish development of the Learning Objects in preparation for initial pilot testing to be followed by revisions, additions of computer applications and roll-out. A great deal of effort will be required to complete the effort but there is little if any doubt that will occur.

5) Plans for evaluation to occur in years two and three of the project.

a. Effects on students:

1. Students will be pre-and post-tested online for each of the target courses. Comparison students will be pre-and post-tested in three traditional lab classes on the same level. Analysis of Covariance will be used to determine whether significant differences exist between the groups with respect to dependent variables of interest, including retention in class, CPT scores, exit exams, and measures of writing competence controlling for entering levels of knowledge and other measures of writing competence which may be available.

2. Students will take attitudinal measures to indicate levels of satisfaction with various phases of the target courses and this will be an additional dependent variable in the study.

3. Initial and final writing samples of students in the target courses will be compared with writing samples from students in traditional lab classes on the same level.

4. Student levels of participation will be monitored and documented relating to levels of participation, hours of participation, assignments completed and extent of involvement with other phases of the system including emails, chat room, frequency of visits to content module pages and frequency of posts to discussions. Attempts will be made to relate levels of participation to effects on post-tests and writing samples.

b. Effects on faculty using the materials

1. Chairperson and project director observations of

the teachers’ use of materials will be conducted to verify usage of appropriate classroom strategies.

2. Teachers will be surveyed to determine their perceptions of various issues surrounding use of materials including changes in pupil responses.

c. Effects on faculty who participated in training sessions

1. Faculty perceptions of changes in training requirements as project continues through years 2 and 3 will be determined.

2. Faculty perceptions of changes in their instructional effectiveness based on new skills will be determined.

3. Faculty perceptions of training strategies appropriate in new settings as the project is rolled-out for use at other sites.

d. Other issues to be pursued

1. Were there unintended consequences of program implementation upon students, teachers, faculty developers or administrative staff?

2. What are the recommendations relating to use of program materials at other sites?

3. How adaptable is the model and how likely is institutionalization?

4. What information exists relating to the project’s cost-effectiveness?