| 
Results:
Statistics, Scores, Participation, and Discussion Postings
An immense amount
of data was collected throughout the course of the assignment, as well
as from the surveys. While future publications regarding this work will
eventually disclose a more thorough presentation of the data collected,
a discussion of the key highlights is afforded here.
Table
1 outlines the number of students that participated in the WebCT
Assignment and its surveys at various times in the semester. 669 students
were involved in the assignment at one point; 3 of which joined the class
after it began but before the first week ended. A high number of WebCT-delivered
Pre-Assignment surveys (585, 87.4%) and Post-Assignment surveys (609,
91.0%) were collected from the students which reflected the fact that
credit was assigned to the completion of the surveys. However, optional
paper surveys given at the midterm 3 weeks after the end of the WebCT
Assignment (302, 45.1%) and at the end of the semester (265, 39.6%) reflected
normal lecture attendance, but still provided enough data to analyze
student satisfaction long after the assignment had concluded.
Table
2 shows the scores that students earned for the assignment, which
was worth 10 points in the course (out of 1000). Student scores of 6-10
out of 10 are earned by participating in their subgroup’s efforts
toward answering the question. Students who earned 0/10 did so by meeting
one of the three superceding criteria in the rubric (vide
supra). Students found guilty of superceding criteria were given an option
to consult the facilitator after the assignment about their violation
and earned individual points were returned, though group points were
not restored. Those under this category had scores ranging from 1-5 out
of 10 points. Counting all students who started the assignment, the average
was 8.5/10. However, when discounting the scores of those who dropped
out the course before the end of the assignment, the average jumps to
almost 8.9/10. Furthermore, when considering only those students who
completed the assignment and participated (discounting drops and those
who didn’t help their respective subgroups at all), the average
jumps to 9.3/10. When taking into account those who finished the assignment
but dropped the course later in the semester, the averages didn’t
change much whether they participated or not, demonstrating that WebCT
Assignment score had absolutely no correlation with dropping the course
at a later date.
Though the score distribution
data is not shown here in its entirety, 62.0% of the students starting
the assignment earned perfect 10/10 scores and 88.9% of the students earned
scores of 6 or higher out of 10. When disregarding those who dropped before
finishing or those who did not help their subgroup, the percentage of students
earning 10/10 jumps to 67.9% and the percentage of those earning 6 or higher
raises to 97.4%.
Out of 669 students who began
the assignment but earned no credit for it, 26 students (3.9%) dropped
the course before the end of the assignment. An additional 14 students
who finished the assignment dropped before the end of the semester. 32
students (4.5%) did not contribute toward their subgroup’s efforts.
34 students (5.1%) did not post their 8 Nouns activity. 3 students (0.4%)
committed a Netiquette Violation, while another 6 students (0.9%) received
private e-mail warnings that their language approached the standard for
a Netiquette Violation. Netiquette Violations also led to revoking the
student’s discussion board privileges in the course for the remainder
of the semester.
Progress was also tracked at
the Subgroup level. One week into the two-week assignment, each subgroup
was evaluated for progress toward completing the assignment and given either
a “Progress” or “Lack of Activity” post with encouragement
to press ahead to the finish. The basis for which message was given was
discussed earlier (see Facilitation of the Assignment). 22 subgroups (19.6%)
had made progress with all members having introduced themselves via 8 Nouns.
37 subgroups (33.0%) had made progress in spite of not having all subgroup
members introduced or participating at the halfway point. In total, 52.7%
of the subgroups made progress toward completing the assignment according
to the timeline suggested in the WebCT Assignment handout.
53 subgroups (47.3%) had not established an agreed upon timeline for completion
of the assignment. However, 110 out of the 112 subgroups (98.2%) had submitted
their final answers by the deadline, with 2 posting their answers during
the 24 hour grace period after the deadline.
Though the assignment was designed
to make the subgroups as even as possible, students who dropped the class
or chose not to participate in their subgroup’s efforts inevitably
reduced the effective number of members of their respective subgroups. Table
3 displays the number of subgroups that had diminishing numbers of
members after the start of the assignment. Overall, the results do show
a promising number of subgroups that had optimum participation.
At the onset, 109 subgroups had
6 members while 3 subgroups started with 5 members. Fortunately, the 5-membered
subgroups did not suffer any reductions in members due to drops or non-participation
throughout the course of the assignment. However, the 6-membered subgroups
did not fare nearly as well. 24 of the 6-membered subgroups started with
at least 1 student who dropped the course during the course of the assignment.
22 of the 6-membered subgroups began with at least 1 student who stayed
in the course but did not participate in their subgroup’s efforts
toward answering the assigned question.
Indeed, when both sets of absent members are factored in, some subgroups
suffered significant losses to their team rosters. One subgroup lost 4
members while two subgroups lost 3 members. The effective 2-membered subgroup
and one of the 3-membered subgroups had “subgroup leaders” appointed
by the facilitator (see Facilitation) as a way to let the participating
members know they had the attention of the instructor and could proceed
on their own. Overall, 72 subgroups (64.3%) operated with full participation
of all its original members. This is significant as it shows that a minority
of subgroups will likely lose members from a large class design and that
only a handful of subgroups are likely to lose significant numbers of members
to the point of requiring special attention from the facilitator.
Finally, the number of discussion
board postings by everyone involved in the assignment was simply astonishing,
given that all of these posts occurred over a span of just two weeks. Table
4 displays the total number, the average, high, low, and standard deviation
of posts in each type of forum. These include postings from students, the
facilitator, and occasional comments from teaching assistants. 5672 posts
total were made as part of the assignment. Only 499 of those (8.8%) were
initial messages from the facilitator on usage of the forums and completing
the assignment, including: Purpose of the Forum, Tyson’s 8 Nouns,
mid-assignment subgroup progress post, How to Post Your Answer, announcements,
and the compiled answers in the WebCT Assignment Answers forum. As a comparison,
in the 13 weeks that followed, only 1937 posts were generated on post-WebCT
Assignment Instructor’s Office and Expresso Bar forums during the
remainder of the semester.
The Main and Notes forums were
used only for minor announcements over the course of the assignment. The
Instructor’s Office forum averaged 51 posts per group and contained
as roughly as many questions about the course in general as about the WebCT
Assignment. The Expresso Bar ranged from extremely active (202 posts) to
essentially non-active (39 posts). A fair amount of these posts were simple
greetings (i.e., Hello.) from students who had little experience. Most
of the activity was led by a relative few students who appeared to be discussion
board savvy. The 8 Nouns activity in the Orientation forum led to friendly
conversation in some subgroups, while a fair majority of others had little
to no response to their colleagues’ 8 Nouns. Reasons for this are
speculated on based on survey data presented later. The Group forums possessed
only the “Purpose of the Forum” post, “How to Post Your
Answer” post, and the answers from the individual subgroups, except
for two subgroups who posted additional modifications to their original
answers.
As expected, the Subgroup forums,
where the subgroups posted their discussions with one another, are where
the bulk of discussion board activity lied in the assignment. 3125 posts
(55.1% of all posts, 3349 minus 224 “Purpose” and “progress” posts)
were generated toward the effort of advancing subgroup answers over the
course of 2 weeks. The subgroups averaged 30 posts per subgroup. Most of
the subgroups had one or two members who provided a thorough answer with
subsequent critique and modification by the other members. Roughly a quarter
of the subgroups had each member provide his/her own answer and then discussed
the similarities and differences. Though most subgroups used an agreed
timeline for asynchronous postings, two subgroups brilliantly devised a
plan to begin posting to their subgroup board asynchronously at a specific
time, essentially transforming an asynchronous discussion board into a
modified synchronous chat room. Against expectations, no subgroups were
found to have any adversarial discussions (nor were any reported) or problems
with conflict resolution. Despite the large and diverse population of students
in this course possessing a variable range of computer skills and group
work experience, perhaps the lack of any serious (or even minor) subgroup
conflicts was one of the most pleasant outcomes from this project.
Results
Student Surveys
Can large classes be trained
to use WebCT in online discussion board work efficiently and successfully?
Can student confidence in discussion board use increase as a function
of online group work?
The answer to the first question is an unequivocal yes. The evidence shows
that the overall student participation rate in the assignment was very high
(88.9% of all students), despite the fact that the assignment was worth only
1% of the overall course grade. 98.2% of the subgroups submitted their answers
by the deadline and the remaining subgroups posted within 24 hours of the deadline.
Efficiency and success was reflected in student opinions as well. 58.6% of
all students felt that expectations for the assignment were clear; another
35.8% felt expectations were clear on some points but not so clear on others.
The total amount of time students took to complete the assignment (0 to 7 hours,
94.1%) matched what they expected to spend on the assignment (90.1%). Of those
students, a majority of them spent between 2 to 4 hours on the assignment (42.1%
expected, 46.0% actual). The students felt overall that the question difficulty
was either very appropriate, somewhat simple, or too easy for a team of 5-6
students to solve (78.0%, 47.3% - very appropriate). Only 11.7% felt the questions were
somewhat difficult or too difficult for a team of 5-6 students.
Survey results also showed that
student confidence in discussion board use did increase as a function of
the WebCT Assignment. Before the assignment, only 45.1% of students stated
they had some experience with discussion boards prior to taking the course.
However, after the assignment concluded, 50.9% of students stated their
confidence in communicating with discussion boards increased as a function
of the WebCT Assignment. 13.8% stated that their confidence has increased
slightly as a function of the assignment, but still have some problems.
17.1% claimed their confidence did not increase from the assignment because
they already knew how to use discussion boards. Only 7.2% stated their
confidence did not go up at all and 11.0% had no opinion one way or the
other. Going from 45.1% of students who had any experience whatsoever to
81.8% that could use discussion boards with at least some degree of confidence
is a very significant result.
Furthermore, 61.3% of students
felt the assignment was a somewhat positive, positive, or very positive
experience. 11.8% had no feelings one way or the other. On a continuous
scale from 0 to 10 (0 = not beneficial at all; 10 = very beneficial), the
overall class average, mean, median, and mode were all 5.0/10 with S.D.
= 2.7. The median (6.0/10) and mode (8.0/10) were slightly higher for Section
B1-Distance, but the section average was approximately the same as the
class average for all sections. Out of a random-ordered list of 10 ways
this assignment could be beneficial or not (check as many as applied),
a majority of students expressed they benefited most by learning how to
use discussion boards (50.2%), learning how to use one of the features
of WebCT (53.2%), and it gave them the opportunity to experience online
group work (54.4%). Table 5 presents
this data with respect to the class as well as by section. Coincidently,
the fourth-largest voted benefit was that they found the 8 Nouns activity
to be a nice icebreaker in meeting new people (24.1%), which is consistent
with the approximate number of subgroups witnessed to have significant
social discussions in the Orientation forum about their 8 Nouns. Students
also found the Instructor’s Office forum useful and helpful (51.6%);
most of the remaining students either were indifferent about the Instructor’s
Office forum’s utility and helpfulness (30.0%) or did not know whether
it was helpful or not (14.6%).
Finally, the WebCT Assignment
didn’t appear to change students’ feelings about group work
overall. Before the assignment, 71.6% of the students either liked group
work (25.1%) or believed it was OK and saw the need for it in the real
world (46.5%), while 14.2% of the students either didn’t like group
work because they felt that they carried the load for others (13.5%) or
saw no value in group work whatsoever (0.7%). 9.6% were completely indifferent.
When asked if their feelings about group work/collaborative work changed
after the WebCT Assignment, 41.0% had no opinion about it one way or the
other. 15.4% felt the assignment positively impacted their feelings about
group work, while 17.1% always liked group work irrespective of the WebCT
Assignment. Meanwhile, 11.5% felt the assignment negatively impacted their
feelings about group work in general and 14.8% always disliked group work
and the assignment did nothing to change that opinion. While a slightly
higher percentage of students either gained or maintained perceptions about
group work than those who possessed negative feelings, the overall change
in perception in the course is minimal.
Can a single primary instructor
effectively facilitate online discussion board group work for a large
class?
The answer here lies in how much time one can dedicate to designing, constructing,
monitoring, facilitating, and grading discussion boards around other duties.
In the case of this course, only one instructor participated in the WebCT Assignment.
The total number of hours of instructor labor associated with this one assignment
(just the hours mentioned in this article) is 214 hours, not counting the time
taken to accumulate and analyze score and survey data. However, this was an
assignment that had 669 students at the beginning. This same course offered
in the alternate semester usually has enrollments in the 300-350 range. This
project repeated again under this new scenario could prove to be much less
demanding. The overall scores and percent completion rates at the individual
and subgroup level are promising indicators of facilitator influence.
Indeed, the students noticed
the facilitator’s efforts. When asked the question “In your
opinion, how would you describe the facilitator’s (Mr. Miller) online
presence during this assignment? (Please check all that apply)”,
students felt the facilitator was friendly (99.2%), helpful (97.2%), and
available for help (99.1%), as opposed to unfriendly, not helpful, and
unavailable for help, respectively. Students also gauged the facilitator’s
overall online presence during the assignment. 70.8% of students described
the facilitator as “everpresent (I could sense Mr. Miller was always
there watching)”, while 24.9% described the facilitator as “sometimes
there (I could sense that Mr. Miller was there for us sometimes, but not
others)”. Despite the facilitator postings, constant monitoring,
and announcements that only one instructor was facilitating the assignment,
4.3% described the facilitator as “never there (I didn’t sense
any evidence that Mr. Miller was involved in this assignment)”.
Similarly, the teaching assistants’ (graduate
and undergraduate) optional role in facilitating the assignment was also
highlighted in student responses. When asked the question “How would
you describe the TA [graduate teaching assistant] or SI [Supplemental Instruction – undergraduate
teaching assistants] assistant facilitators presence during this assignment
(Please check all that apply)” and having the same options listed
for the facilitator question, they received quality marks for being friendly
(100%), helpful (75.4%), and available for help (83.2%). However, 73.2%
of the students described the assistants as “never there”,
22.7% described them as “sometimes there”, with only 4.1% describing
them as “everpresent”. These numbers reflect the fact that
the assistant’s primary role was to watch for student arguments,
unprofessional behavior, and violations and that it was not mandatory to
facilitate by posting.
To summarize, a single person
can successfully facilitate large classes of discussion board-based group
work. However, the time costs associated with proper monitoring and facilitation
can be very large. When this project is repeated in a semester with a lesser
enrollment, more can be said about the student/time ratio and whether this
type of work on this scale is practical.
Will distance education students
believe they have benefited from online group work experience as much
or more than their face-to-face student peers?
The surveys indicate that a higher percentage of distance students may have
benefited from the assignment as compared to the class overall.
It was reported previously that
the Section B1-Dist. students had a slightly higher median (6.0/10) and
mode (8.0/10) with similar class average (5.0/10) on the 0 to 10 scale
describing how beneficial the assignment was. So, the activity overall
was not found to be any more beneficial to the video section students than
to the class overall. However, Table
5 shows a significantly higher percentage of Section B1 students reported
benefit in all of the major categories mentioned previously by at least
8% or more over the class or the face-to-face sections: learning to use
discussion boards, using a feature of WebCT, and opportunity to experience
online group work. In addition, fewer distance students saw no benefit
to the assignment at all immediately after the assignment ended. Student
support for the assignment would wane over time though, as will be discussed
later.
Furthermore, a higher percentage
of distance students reported an increased confidence in using the discussion
boards as a result of the WebCT Assignment as opposed to the class overall
or the sections of face-to-face students. Table
6 shows that 61.4% of the distance students felt the assignment increased
their confidence in using discussion boards, as opposed to 50.9% of the
whole class, 52.1% of Section A, and 46.7% of Section B. Another interesting
statistic is that only 10.2% of the distance students had confidence in
using the discussion boards such that the assignment did not affect it,
as opposed to 17.1% for the class, 17.5% for Section A, and 18.8% for Section
B. While there are more students enrolled in Section A and B which can
affect the class average, the numbers show that far fewer students with
significant discussion board experience were in the distance section and
needed the experience gained from the assignment.
It is possible that the distance
students may have had a perceived need to gain more out of the assignment
(given that the assignment was designed to benefit the distance students
the most) and therefore reflected this sentiment in the surveys. Actual
benefit was not measured in other ways (more detailed pre-assignment surveys
about the above topics might gain more insight on the matter).
What will be the overall level
of student satisfaction with the exercise?
It has been shown previously that an intermittent use of technology in
the classroom without requiring its use in further work is perceived
negatively
by students over the course of time.[9] Since there were no other group work
assignments in the course and since the use of discussion boards were not required
after the WebCT Assignment, it was expected that student support and perceived
usefulness for the assignment would diminish over time. The question was whether
the assignment’s impact on student confidence in using discussion boards
would translate to moderate support for its implementation as a standard tool
for large classes. Another question was whether the distance students would
support the assignment more overall than the rest of the class.
Table
7 displays the survey data associated with student support for the
WebCT Assignment just as the assignment ended, at the midterm of the
course 3 weeks later, and at the end of the semester. The numbers clearly
show a sharp decline in perceived value of the assignment over time.
Just after the assignment ended, the “beneficial” value of
the assignment was 5.0 out of a 0 to 10 scale, including a slight increase
in median and mode for the distance Section B1 students. Three weeks
later, student support dropped as over half the class perceived the assignment
as either “mildly useful” or a “waste of time.”
However, examining the midterm
survey numbers carefully affords an interesting trend. Both sections of
students that did not have the WebCT Assignment facilitator as their face-to-face
classroom instructor gave a higher percentage of high-ratings for the assignment
than those students who had the facilitator as their primary lecture instructor.
Section B lecture was taught face-to-face by the WebCT Assignment facilitator
and had the highest enrollment of all sections. Only 27.0% of the students
in Section B gave the WebCT Assignment a “Useful” or higher
rating on the midterm survey. 37.8% of the distance students in Section
B1 (which had the WebCT Assignment facilitator as their video instructor)
and 40% of the students in Section A (who had an instructor different from
the WebCT Assignment facilitator) gave a “Useful” or higher
rating on the midterm survey. The numbers may reflect that students who
have an instructor different from the person running the online assignment
gain at least an equivalent perception of assignment usefulness as distance
education students. Furthermore, the numbers may reflect that students
who have increased face-to-face contact with the person running an online
group assignment tend to perceive discussion board-based interactions as
less necessary to classroom success.
The end of the semester survey
results reflect an overall further erosion of student support with time
and similar trends in support within sections as seen in the midterm survey.
Fewer students reflected on the WebCT Assignment as a “Waste of Time” in
both Sections A and B1, with B1 as the lowest. Interestingly, the percentage
of students who rated the assignment “Useful” and “Mildly
Useful” bears further analysis. In the case of Section A students
who had a different instructor from the facilitator, more students rated
the assignment as “Useful” than “Mildly Useful”,
unlike for Sections B and B1. The distance students of B1 had the highest
percentage of “Mildly Useful” ratings than any section in the
class, but it was higher than the percentage of students that rated “Waste
of Time” which did not happen for any other section. After the 2nd
midterm exam roughly 70% of the way into the course, the distance students
voted to have face-to-face instruction from the Section A instructor (with
occasional TA guest lecturers) over a consistent single video lecturer.
It is felt that the Section B1 student support for the assignment at the
end of the semester may have waned further than what was observed for Section
A as a function of having face-to-face instruction for the first time during
the semester.
In conclusion, had discussion
board-based group work been designed into the course beyond a one-time
assignment, student support for it might have been sustained throughout
the course. As is, it was shown that discussion board training for the
purposes of increased instructor access for distance students in large
classes alone was not sufficient to maintain student support for the assignment.
Background
Objectives of the Work
The WebCT Assignment
Experimental Design
Facilitation of the Assignment
Grading the Assignment
Results - Student Surveys
Acknowledgements and References
 Printer
Friendly Version
|
