\documentclass[]{article}
\usepackage{url}
\usepackage{graphicx}
\usepackage{verbatim, code, hyperlatex}

\textheight 8.6in \textwidth 6in \topmargin -0.25in \oddsidemargin
0.25in \evensidemargin 0.25in

\newcommand{\proposalTitle}{Computer-Supported Collaborative Learning %
in Support of SMET Undergraduate Retention:\\ %
A Practice-Oriented CSCL Research Agenda }
\newcommand{\mycomment}[1]{\textbf{\emph{#1}}}

\begin{document}

% Definition of title page:
\title{
    \proposalTitle
}
\author{
    PI: Mark Guzdial \\    % insert author(s) here
}
\date{May 2003}    % optional

\maketitle

\thispagestyle{empty}

\vspace*{.5in}

\noindent {\Large\bf Project Summary}

\vspace{.5em}

\noindent Collaborative learning in general and Computer-Supported
Collaborative Learning (CSCL) in particular can positively impact
student learning. Several studies have shown significant impacts
in SMET and other courses \cite{hoadley-linn,miyake,
coweb-english}. How it impacts student learning is still an open
matter. We know that dialog has an impact on student learning
\cite{roschelle,jeong}, but we know that that's not the only
mechanism engendered by collaboration which impacts learning, and
it may not even be a necessary one \cite{nodialog}.

One way that collaboration seems to impact student learning is by
sustaining student interest and motivation (to remain active
within a class) which can result in improved retention. We have
examples in the literature of collaborative learning situations
dramatically impacting retention in a SMET course with
historically low retention (e.g., \cite{pair-programming,
mediacomp}). However, we know that, in very similar courses,
collaboration can fail to have any impact at all
\cite{collab-no-work}.

\emph{Intellectual Merit}

We propose in this project to explore the ways in which CSCL can
impact student interest and motivation as factors in retention in
SMET courses. Our goal is to be able to make predictions about
student retention based on classroom culture and CSCL practices.
We then propose to directly apply the results by selecting courses
with low retention and testing our predictions by attempting to
improve retention by direct intervention in classroom practices
and through implementation of CSCL-based activities.

\emph{Broader Impact}

The broader impact is a deeper understanding of retention in SMET
and developing a research agenda around directly applying learning
science methods and technologies toward classroom practice
problems.

This work is in ROLE Quadrant Three, in that we are seeking to
improve SMET learning by improving retention via a CSCL-based
approach.  The third year of the project, where we seek to change
particular classes through our approach, will requirement
development of a systemic approach to identifying and addressing
retention problems, which starts moving the research into Quadrant
Four.

\newpage

\thispagestyle{empty}

\begin{center}
    {\Large \proposalTitle
}
\end{center}

\tableofcontents

\newpage

\begin{center}
    {\Large \proposalTitle
}
\end{center}

\pagenumbering{arabic}

\section{Introduction: The potential of CSCL to impact SMET learning}

Collaboration can be a significant factor in student learning.
Through collaboration students can reach understandings that none
of the collaborators held previously \cite{roschelle}.  Through
collaboration, students can learn how to collaborate better---a
significant result in fields where professionals must collaborate,
e.g., Engineering. Accreditation for Engineering curricula
requires collaboration \cite{abet} both because of the belief that
it aids in learning and because of the desire for students to
learn about collaboration itself \cite{augustine}.

Computer-supported collaborative learning (CSCL) is promising for
providing the benefits of collaboration in a form that can be more
effectively instituted in a wide range of classrooms.
Collaboration in its face-to-face form can be challenging to
institute in classes \cite{cohen}. Forming and maintaining groups,
arranging for meeting times, or possibly using class time can be
impediments to using collaborative learning opportunities. With
CSCL, collaboration can be just-in-time or asynchronous, and much
larger groups can participate without diminishing the learning
opportunities \cite{hoadley-linn}.

What we are starting to learn is that collaboration can also
influence retention.  Besides the opportunities that it provides
for learning, collaboration can influence factors that themselves
are important for learning.  In a couple of recent studies in
computer science (one of which used CSCL), collaborative learning
situations dramatically improved retention-students stuck with the
class longer than students had in similar classes without
retention, and (presumably) learned more than they would have had
they dropped out.

Retention is a significant issue in science, math, engineering,
and technology (SMET) education \cite{seymour, astin, hilton}.
Among minority populations and women, high attrition rates
exacerbate the problem of underrepresentation in many scientific
and technical fields. \cite{nsf-women, nsf-indicators}. It's clear
that we lose many students in the SMET educational pipeline. If we
can influence retention positively, we can improve SMET education.

\textbf{The goal of the proposed project is to understand how CSCL
can positively influence retention in SMET education, test our
understanding, and then use this understanding to change classroom
practice.}  We believe that CSCL can be effective at improving
retention and can be (relatively) easily instituted into classes.
Our three year study will develop a theory for how CSCL influences
retention in SMET classes, will test the theory by making
predictions about classes and matching them to observed classes,
and then going into SMET classes with low retention and attempt to
change them through use of CSCL-based activities in order to
improve retention.

This project is using a model for education and learning science
research that's related to design experiments \cite{brown-design},
but with different purposes.  Much learning science research draws
from psychology as its model-our goal is understanding of how
learning takes place at the cognitive level.  Our model in this
research is more of an \emph{engineering} model than a science
model, and we think that's an important distinction. We have a
desired end state, of a larger percentage of students remaining in
classes while learning. The point of our research is to study and
attempt to directly influence the factors that influence our end
state (e.g., motivation and student interests).  We expect to make
contributions to understanding about the factors that we identify
influencing retention, while we are attempting to directly change
education practice and outcomes.

\subsection{Where and How CSCL can Influence Retention}

The literature on collaboration does provide strong examples of
where collaboration has facilitated learning, e.g., Roschelle's
study of learning physics collaboratively \cite{roschelle}, or
Jeong and Chi's study of students learning anatomy in a
peer-learning setting \cite{jeong}, or the work on peer-tutoring
\cite{reciprocal}. Our definition of collaboration is where more
than one student is interacting with other students with a goal of
one or more of the students' learning.

Our most foundational work on the social aspects of learning
suggests that \emph{all} learning is social---that our internal
mental dialog is developed by copying the external dialogs we have
\cite{vygotsky}.  Much of our post-secondary education can be seen
as a form of joining a community of practice and learning the
social norms of that community \cite{lpp}.  We know that most
undergraduate practice does not recognize the value of the social
aspects of learning \cite{coweb-fie}.  Fortunately, we don't need
to change \emph{all} of educational practice for our
goal---rather, we can change culture subtly and introduce
collaboration with little cost using technology support (CSCL),
and thus gain some of the benefits of the social perspective on
learning.

CSCL can be effective in facilitating learning in a collaborative
setting . The earliest examples of CSCL showed marked improvement
on standardized test scores \cite{higher-agency, csile-world3}.
Later studies have shown that students have been facilitated in
their learning by CSCL in physics \cite{hoadley-linn}, cognitive
science \cite{miyake}, and, in our own work, English Composition
\cite{coweb-english}. CSCL-using students in our study performed
better on a post-test than students not collaborating.

Our theory of how collaborative learning takes place says that
much of the learning arises from the dialog between students
\cite{roschelle, jeong}. Through their discussions, students make
hypotheses, test their hypotheses (often simply through the
discussion), and reformulate their hypotheses \cite{roschelle}.
However, CSCL practice suggests that other mechanisms are in play,
too \cite{nodialog}.  In actual practice, much less dialog is
observed than our theory says is needed for learning to occur, yet
we still do measure learning in CSCL classrooms.

It seems that CSCL plays more of a role in facilitating learning
than simply providing an opportunity for dialog.  In CSCL, the
dialog is visible and persistent.  Students can come back to a
discussion later, look at what others have discussed,
contextualize the dialog by connecting it to an artifact
\cite{guzdialturns}, search among discussions, and even continue
discussions that seemed to have ended.  In our interviews with
students, we find that CSCL can play several roles in learning.
For some students, it's a catalyst for individual learning, e.g.,
``I saw what was posted, and decided to go explore before posting
anything.''  For others, it's a catalyst for impromptu
face-to-face collaborations, e.g., ``We discussed what we saw in
the [collaborative Web space].''

As a way of facilitating more student participation (particularly
in traditionally lecture-based courses), CSCL can provide distinct
advantages over face-to-face collaboration not only for logistical
reasons, but also because it encourages new patterns of
interaction \cite{hoadley-linn}. We know that student-instructor
and student-student interaction changes considerably when it moves
from the classroom to the chatroom. Discussion shifts from
primarily teacher-initiated to student-initiated
\cite{ircfrancais}.

 The use of CSCL in Architecture is a
useful case in point. Architecture is traditionally taught in
small studios \cite{schon}. What goes on in the studio is the
creation of a community of practice, artifact-centered dialog, and
``collaboration-in-the-air'' \cite{kafai} where simply hearing
others' dialog and seeing each others' work influences the
learning. At Georgia Institute of Technology (Georgia Tech), one
of the larger Engineering schools and technical institutes in the
country, the introductory courses have over 150 students---far too
large for a traditional studio. Through use of CSCL, much of the
interaction, sharing, and artifact-focused collaboration was able
to be replicated successfully in a large class setting
\cite{icls-2cool}.

In recent studies, we've also found that collaboration can be a
factor in improving retention.  Retention is not a cognitive
construct, but a measurable outcome of other cognitive constructs
such as student interest (if you're more interested in a topic,
you'll more likely stay in class) and motivation (your desire to
succeed in the class, sometimes despite interest \cite{pintrich}).
Collaboration, even in a computer-supported version, has been
shown to play a role in improving retention.

\subsection{Improving Retention in SMET Classes with
Collaboration}

Two recent studies in computer science education research have
highlighted the potential for collaboration to improve retention.
The first used face-to-face collaboration, while the second used
CSCL.

Pair programming is a relatively new practice developed in
industry where pairs of programmers work together
\cite{williams2}. Pair programmers are strictly forbidden to use
any program code that was written by either individual programmer.
Instead, all code is written with one person typing and the other
person observing-sometimes critiquing, sometimes making
suggestions, sometimes just thinking it through with the
programmer at the keyboard.  Typically, the pairs switch roles
regularly.

Researchers like Laurie Williams of North Carolina State
University have been exploring the use of pair programming in
introductory computer science.  Introductory CS has historically
low success rates---50\% of the class withdrawing or failing is
common \cite{retention-sigcse,failurerates}. Williams and her
colleagues have found that pair programming has not had
significant influence on performance in introductory classes
(e.g., the resultant code is not significantly better than
individuals' code) though it has in other classes
\cite{williams-phd}, nor has pair programming significantly
improved learning as measured by post-test performance
\cite{pair-programming}. However, there has been a significant
improvement in retention. Students in pair programming stick with
the class more than students who do not (see Table~\ref{tbl:pair}
from \cite{pair-programming}) .

\begin{table}[hb]
\begin{tabular}{||l||l|l||}\hline
 & Paired-condition & Solo programming \\ \hline\hline
 Non-CS Majors & 66.4\% & 55.9 \% \\ \hline
 CS Majors & 83.0\% & 84.0\% \\ \hline
 \end{tabular}
\caption{Success rates (received A, B, or C grade) under a
paired-programming condition\label{tbl:pair}}
\end{table}

The disadvantage to pair programming is that it's difficult to
manage. What happens when one of the pair doesn't show for a
session? Do you set up triplets or have the sole programmer work
alone (potentially at a disadvantage)? How do you form pairs? How
do you handle dysfunctional pairs? If CSCL in large groups could
have similar effects, we could have the advantages of
collaboration in improving retention without the cost of managing
the groups.

We recently developed and evaluated a pilot introductory computer
science course aimed particularly at non-majors \cite{mediacomp},
funded in part by NSF CCLI grant {\#}0231176. The course
\textit{Introduction to Media Computation} emphasized the
communications aspects of computation, by having students
manipulate pictures, sounds, text, and movies as the context for
their programming and learning. Collaboration played a significant
role in the course. 121 students enrolled in the course, 2/3
women.

\begin{itemize}
\item Students were encouraged to collaborate on a subset of the
homework assignments. These were not explicitly organized by the
instructor and mostly occurred face-to-face. There were also
pre-quiz activities that students worked on explicitly and
collaboration was encouraged.
\item The course also had its own \textit{CoWeb} (Collaborative Webspace) \cite{coweb-jls}
 which was used extensively in the course. Students could ask
 questions and make comments about the course (sometimes
 anonymously). Students were encouraged to post the media
 resulting from their work to share in \textit{Galleries}. Exam
 review questions were posted on the CoWeb with space created for
 questions, comments, and proposed solutions. The instructor would
 participate in the exam review and other activities to provide feedback on answers and
 help respond to questions. A \textit{Soapbox} that appeared at
 the top of every page in the CoWeb website was student editable
 and thus served as a way to raise awareness in the student
 community. Students were never required to use the CoWeb, and
 there were always several activities going on at once so that
 students could choose among activities.
\end{itemize}
The course had an 89{\%} success rate---only 10.7{\%} of the class
withdrew (3 of 121 students, 2.5\%) or received an F or D (10 of
121 students, 8.2\%). In contrast, non-majors in a pair
programming CS1 class had a 66{\%} success rate
\cite{pair-programming}---which was still significantly better
than non-majors in a traditional CS1 class.

On a final survey, students in the \textit{Media Computation}
class were asked what was the most important thing \textit{not} to
change about the class. Nearly 20{\%} of respondents named the
collaboration tool CoWeb and over 20{\%} mentioned collaboration
in general. Certainly, the media context influenced students'
interest in the material and motivation to say in the class.  But
the context in which the media is manipulated is important to
student affect, too \cite{kozmamultimedia}.  Wedo not know the
relative influence of the course context vs. collaboration (and
collaboration vs. CSCL) in the improved retention in this course,
but student responses on the final survey suggest that
collaboration may have played a significant role.

\subsubsection{The Possible Roles of CSCL in Improving Retention}
\label{subsubsec:mylabel2}

Why should collaboration play a role in improving retention, if it
does? That's one of the significant research questions of the
proposed research. We do have some early hypotheses.

We believe that the most significant influence of CSCL is by
enabling collaborative learning activities which impacts on
student interest and motivation, factors in retention
\cite{donald}. These factors are particularly important in
student's project work. SMET education is often project-based
\cite{abet,augustine}. Students work on external projects in the
course, often requiring multiple sessions to complete. Sustaining
motivation is a critical factor in project-based learning
\cite{pbl} and technology in general can play a role in sustaining
that motivation \cite{krajcik}. If students are interested in the
project, they tend to be more successful at it \cite{djoseph}, and
technology can play a role in maintaining interest
\cite{djoseph2}.

One of the factors that influences a students' motivation to
remain in the class is a sense of anonymity \cite{donald}. If a
student will be missed if she doesn't attend class, there is more
motivation to attend class. This is one of the explanations for
lower retention in large lecture-based classes. In a class that
uses face-to-face collaboration heavily (as in pair programming),
you will always be missed if you do not attend. Your collaborators
rely on you.

CSCL (at least, when the technology is asynchronous and
text-based) does not influence the same factor in the same way.
While students may develop a sense a community in the on-line
space \cite{amy-cscw}, thus reducing the sense of anonymity, there
is not the same pressure to physically be present in class.
Nonetheless, CSCL can inculcate some similar motivation to remain
a part of the class.

Our analysis of interviews with students in the \textit{Media
Computation} class suggests that CSCL influences both motivation
to continue in the course and student interest, and perhaps helps
with a sense of community . The particular role that CSCL plays is
not unlike the role of a case library in supporting project-based
learning \cite{lbdsw, cbr}. CSCL can serve as a \textit{source for
models} for both process and product and a \textit{source for
lessons-learned}, as a \textit{source for feedback} well as a
place to get questions answered.

\begin{itemize}
\item As a \textit{source for models}, we found that students used
the CoWeb to find out what others have done (e.g., in the
\textit{Galleries}) to be used as examples. They would also use
students' questions in the CoWeb to guide their own process.

\textbf{Q. What do you think about the homework galleries on the
coweb?}

\textbf{Student 2:} ``Oh like where you can put your picture up.
It's nice to see other people, like what they did with it. Cuz,
I'm not the most creative person and even if I was I wouldn't know
how to be creative in this! And there is no better feeling than
getting something done and knowing that you've done it right, like
the Soapbox sometimes is just like `I did it!' and posting to the
CoWeb just adds onto that.''

\item As a \textit{source for lessons-learned}, we found that
students would go to the CoWeb when first starting a project to
see the challenges that others were facing and sometimes to
jump-start their own process.

\textbf{Q: Do you think the CoWeb is beneficial? Why?}

\textbf{Student 1: }``Very beneficial, it keeps you on track,
keeps you organized. You get to see what people are having
problems with and maybe see... I always start off looking at what
people have had problems with.''

\textbf{Q. How do you plan to study for the final exam?}

\textbf{Student 2: }``Hard. I have actually read the book. I read
it before the first test. I'll go through all the lecture slides
and look at old tests. And the online review [on the CoWeb] is
awesome and I'm very grateful he did that. I like that you can
make comments, I can see stuff where other people had questions
cuz they ask stuff that I might not have thought of. And I'll
study with a group.''

\item As a \textit{source for feedback}, the CoWeb served as a place
to ask questions, inform others where you were in your process
(e.g., the Soapbox comment above), and to get feedback on the
clarity of your process. Anonymity, which is not possible easily
in face-to-face collaboration, played a role in putting students
at ease in asking questions---an important factor in retention
\cite{comfort-cs1}.

\textbf{Q. Have you ever posted to the CoWeb?}

``I think I've posted to everything. Sometimes I'll just make
random comments. Sometimes I ask a specific question and he [the
professor] asks for clarification.''

\textbf{Do you think the CoWeb is beneficial? }

``Yes. And there's no reason to feel uncomfortable because if you
feel dumb, just don't put your name at the end! I did that a few
times. ``
\end{itemize}

We can see how CoWeb's use in the Media Composition class meets
the requirements of the motivation literature in trying to improve
student motivation. The main factors that the teacher can control
in improving motivation are classroom engagement and teacher
expectations \cite{pintrich}. Offering students choices in the
classroom and having more than one way to accomplish a learning
goal are examples of using classroom engagement.  The use of CoWeb
does offer choices of activities in pursuit of a variety of
learning goals.  Teacher expectations are controlled with more
feedback--when students are rewarded for effort as it occurs, as
opposed to simply being told ``This is going to be hard'' at the
beginning with no other feedback until the end.  The involvement
of the teacher in the CoWeb, with frequent feedback, provided that
kind of control of teacher expectations.

\subsection{When CSCL Fails to Improve SMET Learning and Retention}
\label{subsec:mylabel2} At the same time that we know that
collaboration, and CSCL specifically, \textit{can} play a role in
learning and retention, we also know of specific SMET learning
situations where it does \textit{not}. Part of our work on our NSF
REPP grant ({\#}REC-9814770) involved coming to understand where
collaboration is \textit{actively resisted} in SMET classes.

The initial focus of the project was to encourage integrative
learning among Engineering students by encouraging collaboration
around a common theme across disciplines. The theme we selected
was \textit{computer modeling}. We teach computer programming
(including MATLAB) in Computer Science (and now, Engineering)
freshmen courses, differential equations in Mathematics, and
implementation of differential equations in programs to model
Engineering problems in upper-division courses. The problem is
that these classes are not at all integrated: Terms for the same
things change between courses, problems used in earlier courses
are not at all like those that students see in later courses, and
the faculty don't communicate about these courses--faculty
communication across disciplinary boundaries is quite uncommon
\cite{cuban}.

Our plan was to provide a collaboration space, CoWeb (or Swiki),
which was being used successfully in several other courses
\cite{coweb-jls}. The idea was to provide a place where students
could go for help on any aspect of computer modelling, but since
the space encouraged collaboration, the help might come from
students in other disciplines taking other classes whose overlap
might inform one another. For the Senior struggling to remember
differential equations, the Sophomore taking it just then is the
best help. For the Sophomore, the Senior's problems tell her what
she has to look forward to.

The actual execution was much more disappointing. Students
actively avoided our efforts.

\begin{itemize}
\item To encourage collaboration in CoWeb, we created a
mandatory assignment that required collaboration between a
Chemical Engineering and a Mathematics course. The students in
Chemical Engineering created simulations that generated data for
the Mathematics students to analyze, and then provide the results
back to the Chemical Engineers. 40{\%} of the Mathematics students
accepted a zero on the assignment rather than collaborate with the
Chemical Engineers.

\item One semester, we started using CoWeb in a Freshman
Architecture course ($n$=171) at the same time that we started in
a Senior Chemical Engineering course ($n$=24). After ten weeks
into the semester, the Architecture students had generated over
1500 pages, with some discussion pages having over 30 authors. In
the Chemical Engineering course, not a single student had made a
single posting yet. In another semester, in a Computer Science
course of 340 students, only 22 students participated.

\item We had a hypothesis that part of the inhibition to
participate in the Engineering and Mathematics class was a
technical one. The content of many of these courses involves
equations, and equations are difficult to post on the Web. If
students couldn't ``talk'' in the modalities that were the most
comfortable for them, it would make sense that they would avoid
our tool. So, we created an applet-like tool that allowed users to
create equations by simply dragging and dropping components from
palettes, and then drop the equations into a bucket for rendering
to a GIF format and for easy posting. We installed it in a CoWeb
for a Mathematics class and for a Chemical Engineering class.
Faculty used it and praised it. Not a single student even
\textit{tried} it in either class.
\end{itemize}

After two years and over a dozen relatively-unsuccessful attempts
to get collaboration activities working in Engineering, some CS,
and Mathematics courses, we shifted our focus from pushing the
collaboration agenda to understanding why there was such
resistance to it. We found three main reasons for the lack of
collaboration \cite{coweb-fie,collab-no-work}:

\begin{itemize}
\item \textit{Rational response to competitive conditions}: If
students perceived the course to be highly competitive, they often
believed it to be graded on a curve--even if there was evidence to
the contrary. When the course is graded on a curve, it's only
rational to avoid collaboration.

\item \textit{Learned helplessness}: In several courses, students
expected to do badly or expected to get little help, so
\textit{requesting} such help (in a collaborative, public setting)
was to label oneself and would probably not result in any help.

\item \textit{A lack of models}: Students don't see faculty
collaborating across disciplinary boundaries. They don't know what
to do, nor that it's valued.
\end{itemize}

The conclusion is that CSCL is no panacea. Students will not
necessarily participate, and without participation, we can have no
impact on learning nor retention. Thus, any theory that attempts
to explain how CSCL might influence retention must take into
account classroom culture and inculcating student interest even in
the collaboration itself.

\subsection{Articulating a Theory of CSCL and Classroom Culture Influence on Retention and Student Performance}

We can now articulate our early theory of how CSCL and changes in
classroom culture.  Figure~\ref{fig:influence} describes how we
hypothesize that introducing CSCL with requisite changes in
classroom culture can influence student withdrawl rate and student
performance.


\begin{figure}[htb]
\begin{center}
\includegraphics[width=4in]{influence-diagram.eps}
\end{center}
 \caption{Diagram of hypothesized influences
\label{fig:influence}}
\end{figure}

Our main inputs into the system are (a) a CSCL environment that
easily enables collaborative learning activities and (b)
encouraging faculty members to provide support for collaboration
through change in classroom culture. Both are necessary to make
collaborative activities successful: The technology makes it
low-cost and easier to adopt \cite{coweb-english} and the policy
changes are critical to change culture and encourage participation
\cite{coweb-fie}. Collaborative activities encompass student
experiences from \emph{explicitly} collaborative activities
created by and/or sanctioned by the course instructor through
collaborations that the course instructor allows and
\emph{implicitly} encourages by a liberal collaboration policy and
a grading policy that does not penalize students for working
together. We have learned that the implicit encouragement is
perhaps more critical for successful participation in
collaborative activities than the explicit encouragement
\cite{coweb-fie,collab-no-work}, but the explicit encouragement is
noted by the students (see quotes above) and does have an impact.

With the opportunity to collaborate enabled by collaborative
learning activities and encouraged by a supporting culture, we
believe that student participation is enabled based on our prior
work \cite{coweb-fie, collab-no-work}.  We can then explore the
specific hypotheses of this project.

\begin{itemize}
\item We believe that this participation in collaborative
activities influences students' interest and motivation to succeed
in the course.  It may also influence activities like
question-answering and developing a sense of community, which can
influence retention.
\item We believe that the strongest influence of these influences
(from the collaborative activities) is on the withdrawl
rate---convincing students \emph{not} to give up on the class.
This is the effect on retention that we're most concerned with and
that we see in the earlier cited studies \cite{pair-programming,
mediacomp}.
\item We believe that there is an additional influence on student
performance, seen in terms of grades.
\end{itemize}

Our project plan is about (a) investigating the research questions
implied by these hypotheses (e.g., are these hypotheses correct?
What other factors need to be considered in our model of
influencing retention?), (b) testing our answers to these
questions, and (c) applying our answers to improve the phenomena
in question.

\section{Project Plan: A Practice-Driven Process for Research, a Research-Driven
Process for Impacting Practice} \label{sec:mylabel2} The process
that we are proposing in this project means to coordinate research
with practice in such a way that we learn more about practice and
directly impact the practice. The research starts from a real
problem in practice, and then proceeds to develop a process for
applying the results of the research in practice to address the
original problem. Our proposed project seeks directly to improve
retention in SMET undergraduate classes using interventions based
on CSCL. We use the projects described
\cite{mediacomp,pair-programming} as evidence that collaboration
\textit{can} have an impact on retention. Our proposed work seeks
to understand when collaboration may be successful at improving
retention, test our understanding, and then apply that
understanding to change classes with historically low retention
rates.

This is a different model than most research in the learning
sciences, and we think it's an important one. While it's in the
vein of a design experiment \cite{brown-design}, our goals are
somewhat different. Like a design experiment, we begin in real
classrooms with all of their complexity, and our goal is to
understand better what's going on in the situation. But while a
design experiment seeks to understand individual student
psychology, our work seeks to understand the classroom dynamics
and not focus on the individual student. In some sense, our
approach is more of an engineering approach than a scientific
approach. While scientists seek to understand all of a phenomenon,
an engineer studies enough to make it work \cite{kay-foreword}. In
so doing, the engineer often gathers the data that drives the
scientific enterprize. We plan to play that kind of role. We plan
to learn how to directly affect retention rates through
manipulation of factors related to collaborative learning, and
then use that knowledge to impact courses with historically low
retention rates.

A brief sketch of our plan follows:

\begin{itemize}
\item Year One: We develop a theory about how classroom culture and
collaboration practices relate to retention, by studying some of
the 40 courses per term that use our CoWeb collaborative software
and by studying comparable courses that do not use any form of
collaboration. We essentially want to develop a qualitative
regression equation that describes how factors such as perception
of competition and the availability of models relate to
intermediate factors such as student interest and motivation which
drive our factor of interest, retention.

\item Year Two: We then test our theory and develop an
intervention strategy. Each term during Year Two, we will select a
sample of courses, some using CoWeb and some not using
collaborative learning activities at all, and develop a prediction
based on our theory of anticipated retention. We use CSCL because
we expect that if will be the easiest for teachers to adopt in our
Year Three efforts. We will study those courses (e.g., use surveys
to establish student interest and motivation in the course) and
refine our theory.

\item Year Three: We apply our theory. Given our theory, we will
develop a system to identify courses with historically low
retention rates which might be amenable to our interventions,
e.g., changing grading practices to reduce competition, and
provide models and discussion spaces via CoWeb. We will work with
faculty in those courses to bring about change, and then measure
our factors to look for impact of our intervention and match to
our theory.
\end{itemize}

\subsection{Year 1: Creating a Theory about Retention and CSCL Influence}
\label{subsec:mylabel3} During the first year of the study, we
plan to develop a theory about how collaboration impacts
retention. We are not under any illusions that collaboration is
the \textit{sole} factor impacting retention in any post-secondary
course, however we do believe that collaboration can be a
\textit{significant} factor, and it is a factor that we have some
control over.

Our strategy for developing a theory about collaboration is to
measure these variables in studying four kinds of classes.  We
will choose a sample of each of these classes to study during each
of the three semesters in Year 1, for a total of some 20 courses
over the course of the year.

\begin{itemize}
\item SMET classes with high retention. Our analysis of courses at
Georgia Tech suggest that high retention may be a relative
concept---rarely in first and second year courses is the WDF rate
(percent of enrolled students who Withdraw, or who earn a grade of
D or F) in single digits.

\item SMET classes with historically low retention.
Table~\ref{tbl:wdfrates} summarizes the WDF (Withdraw, or earned
grade of D or F) rates of several potential target courses at
Georgia Tech.

\item Similar SMET classes using collaborative activities. Not all
sections of these courses are taught in the same way---the
statistics hide the activities of particular instructors and the
use of collaborative activities within those classes. We do not
have evidence right now that there \textit{are} sections of
similar classes that are using collaborative activities without
technological support, but we plan as part of our survey to seek
out these kinds of activities.

\item As a significant subset of the second kind, we are interested
in similar SMET classes that are using CSCL for collaborative
activities of some kind. We know that simply using CSCL tools does
not imply participation nor successful learning \cite{coweb-fie,
collab-no-work}. However, use of a CSCL tool indicates the
potential for incorporating successful collaborative learning
activities, and use of a tool is easier to incorporate into
courses that have no collaborative component.

Moreover, CSCL-using classes with low retention help us to
understand better what CSCL can impact and what it cannot.  We
have hypotheses about what CSCL can impact and under what
conditions.  We don't believe that CSCL can influence \emph{all}
the factors that lead to better retention.  However, we believe
that we can reduce retention rates because there's \emph{enough}
leverage through CSCL.
\end{itemize}

At Georgia Tech, there are a reasonable number of courses in that
last group of SMET classes. Last semester, some 45 classes used
our CoWeb collaborative technology. Some 20 of these were in
Computer Science, 5 in Mathematics, and another 10 were in
Chemical Engineering. Currently, \textit{all} undergraduate
Chemical Engineering classes are automatically provided with a
CoWeb at the start of each semester.  There are other
collaborative technologies in use at Georgia Tech as well: WebCT
and newsgroups, for example.  We will include those tools in our
study, but based on our comparisons between different tools, we
feel that the CoWeb is the most flexible and thus will be the most
applicable to the widest range of kinds of classes
\cite{coweb-jls, guzdialturns, infoecol}.

\begin{table}[htb]
\begin{center}
\begin{tabular}{||l|p{1 in}|l|l|l|l|l||}
\hline\hline Course& Title& Term& $N$ & W& D/F&
DFW \\
\hline \hline CS1050 & Understanding and constructing proofs& Fall
2001& 234& 10.68{\%}& 14.10{\%}&
24.79{\%} \\
\cline{3-7}

 & & Fall 2002& 213& 4.23{\%}& 17.37\% & 21.6\%
 \\ \hline

CS1321 & Introduction to Computing & Fall 2001 & 1168 & 7.96\% &
21.06 \% & 29.02\% \\ \cline{3-7}

 & & Spring 2002 & 935 & 8.88\% & 25.99\% & 34.87\% \\ \cline{3-7}

 & & Fall 2002 & 1316 & 7.07\% & 18.77 \% & 25.84\% \\ \cline{3-7}

 & & Spring 2003 & 879 & 10.69\% & 27.08\% & 26.02\% \\ \hline

CHE2100 & Chemical Process Principles & Fall 2001 & 87 & 6.9\% &
10.34\% & 17.24\% \\ \cline{3-7}

 & & Spring 2002 & 31 & 0\% & 25.81\% & 25.81\% \\ \cline{3-7}

 & & Fall 2002 & 90 & 8.89\% & 16.67\% & 25.56\% \\ \hline

ECE2025 & Introduction to Signal Processing & Fall 2001 & 304 &
8.88\% & 9.54\% & 18.42\% \\ \cline{3-7}
 & & Fall 2002 & 272 & 6.25\% & 9.19\% & 15.44\% \\ \hline

Chem1311 & Inorganic Chemistry I & Fall 2002 & 136 & 10.29\% &
25.74\% & 36.03\% \\ \hline

Math1501 & Calculus I & Fall 2001 & 1421 & 3.31\% & 17.8\% &
21.11\% \\ \cline{3-7}
 & & Spring 2002 & 225 & 7.11\% & 42.22\% & 49.33\% \\ \hline

PHYS2211 & Introductory Physics I & Fall 2001 & 634 & 9.31\% &
29.02\% & 38.33\% \\ \hline\hline
\end{tabular}
\caption{A selection of Withdraw, D, or F rates for lower-division
undergraduate courses at Georgia Tech \label{tbl:wdfrates}}
\end{center}
\end{table}

We believe that collaboration impacts retention through at least
two direct variables that CSCL has the opportunity to impact.

\begin{itemize}
\item \textbf{Interest}: Collaboration can encourage and inculcate
student interest. We see (e.g., in earlier quotes) that students
are inspired by the work that they see other students doing. We
know that student discussions do inspire students to explore
issues more deeply \cite{nodialog}.

\item \textbf{Motivation to succeed in the course: }Collaboration
can help convince students that they can succeed (as well as
potentially help them learn so that they can succeed).
Collaboration can help in answering questions, in receiving
support in the course, and in reducing students' sense of
anonymity (unless they want it!). In general, collaboration can
make the class seem more \textit{doable}---that it's something
that students can be successful at.
\end{itemize}

We know from the literature that there are many other factors that
influence retention \cite{pintrich}.  Some of the more significant
ones that we plan on studying:
\begin{itemize}
\item Number of students in a class or section. Obviously, the
number of students doesn't \emph{directly} lead to students'
failure to succeed at a course. However, more students in a class
leads to a greater sense of anonymity which can influence
students' decision to withdraw or skip class \cite{donald}.
Greater numbers may also inhibit's students' comfort with asking
questions, which has been highly correlated with success in
introductory computer science courses \cite{comfort-cs1}.

\item The instructor plays a very significant role in retention in
a course.  We've found that instructors' attitudes and modelling
of behavior influences students perceptions about the course
\cite{coweb-fie, collab-no-work}.  If the teacher believes in
frequent feedback at smaller granularity, that can improve
motivation \cite{pintrich}.  If the teacher both motivates success
and lowers the cost of failure (e.g., with more opportunities to
perform and succeed), student motivation is increased
\cite{covington}.

\item Student self-perceptions about efficacy in the class
\cite{code-warriors}.  If students enter a class convinced that
they won't be successful, they probably won't be \cite{eccles}.
Student perception that the situation is manageable improves
motivation to succeed \cite{paris-motivation}.  To what students
attribute their efficacy is also correlated with success in the
course.  Students who do not succeed in introductory computer
science courses tend to attribute their performance to external
factors such as ``luck'' as opposed to internal factors such as
``hard work'' \cite{comfort-cs1}.  Controlling teacher
expectations is very important in reducing withdrawl rates,
especially among students with low expectations of themselves
\cite{bandura}.
\end{itemize}

Our plan is to conduct surveys and interviews among teachers and
students (both those who complete the course and those who do
not). Our study plan is described in Table~\ref{tbl:studyplans}.
The idea is to study the courses at Georgia Tech that can inform
us about retention and about the role that CSCL is and might play
with a focus on the variables that are identified in the
literature and in our own work that are significantly related to
retention.  Where possible, we plan to use and adapt existing
measures for these factors, such as the MSLQ for measuring
motivation that has been used successfully with undergraduate
students \cite{mslq}.

\begin{table}[htb]
\begin{center}
\begin{tabular}{||p{1.0in}|p{2 in}|p{0.75 in}|p{1 in}||}\hline\hline
Class of participants & Factors to be studied & Frequency & Form
of measurement \\ \hline\hline

Faculty & Attitudes toward collaboration, attitudes toward student
efficacy and achievement, policies toward collaboration,
collaborative activities in the class, expectations of class and
how those expectations are communicated, feedback policies &
Before class starts &
Surveys and interviews in particularly low-retention classes \\
\hline

All students & Interest in the course and topic, expected outcome
in the course, motivation to succeed in course, comfort asking
questions,
perceptions about course & First week of course & Surveys \\
\hline

Students who drop the course & Interest in the course, motivation
to succeed in course, reasons for dropping the course, perceptions
about course & After drop day & Interviews (including phone and
email) \\ \hline

Students who complete the course & Interest in the course,
motivation to succeed in course, expected grade in course, cause
of grade, perceptions about course & End of class &
Surveys and interviews with a sample of volunteer students \\
\hline\hline
\end{tabular}
\end{center}
\caption{Study plans in first year in targeted
courses\label{tbl:studyplans}}
\end{table}

\subsection{Year 2: Testing the Theory and Defining the Intervention}
\label{subsec:mylabel4}

Based on our study in Year 1, we will develop a theory that will
allow us to predict retention rates.  During Year Two, we will
apply that theory to develop \emph{a priori} predictions about
retention in the courses.  We will then replicate the study of
Year 1 to gather data to refine the theory.  At the end of each
term, we will establish the reliability of our theory.  By the end
of Year 2, we expect to have reasonable accuracy at predicting
retention rates.

Even from the start of Year 2, though, we will have a good idea of
what factors are most significantly related to retention in our
target classes, and which variables might be influenced by CSCL.
During Year 2, we will develop our intervention
plans---determining what factors we might influence via CSCL, and
then developing materials and activities for the target classes
which will focus on those factors.

Obviously, not all factors, not even all significant factors, will
be amenable to intervention using CSCL.  However, we do believe
that CSCL will be able to play a role in \emph{reducing} failure
rates in targeted courses.  We will begin with our hypotheses
about the roles that CSCL can play in influencing retention: a
\textit{source for models} for both process and product, a
\textit{source for lessons-learned}, and a \textit{source for
feedback} well as a place to get questions answered.

To prepare for intervention, we will create:
\begin{itemize}
\item Activity materials for teachers, like our successful \emph{CoWeb
Catalog} \cite{coweb-catalog} that describes activities that
teachers have invented with the CoWeb \cite{coweb-jls}.  The new
materials will focus on activities that will directly impact the
factors that we decide are most amenable for change.  The idea is
to show teachers ways that they can incorporate the CoWeb to
improve retention in their courses.
\item Course suggestion materials which will describe how to
change grading policies, provide productive and frequent feedback
via the CoWeb, and influence course perception in ways to improve
retention.
\item In some instances, modifications to our CoWeb software to
make it easier to implement these activities and suggestions.
\end{itemize}

\subsection{Year 3: Applying the Theory and Interventions}
\label{subsec:mylabel5}

Year 3 is about directly applying the theory and materials that
were developed and verified in Years 1 and 2.  There are two parts
to our goals for Year 3:
\begin{itemize}
\item We need to identify target courses for improving retention.
\item We need to implement the interventions.
\end{itemize}

\subsubsection{Identifying Target Classes}
\label{subsubsec:mylabel4}

We plan to develop a process for selecting courses for
intervention.  The goal is to find courses that are most amenable
to intervention and improvement, and to systematize the process so
that such targeting and intervention can become a regular practice
in the institution.  The process we are currently planning
includes:
\begin{enumerate}
\item Identifying potential target courses based on historically
low retention, e.g., WDF over 20\%.
\item Survey of teachers for those targeted courses in Year 3, to
determine which ones would have attitudes and policies most
amenable to intervention.
\item Approaching these teachers with our materials and asking
them if they would be willing to change their activities and
policies.
\end{enumerate}

We have the support of Robert McMath, Georgia Tech's Vice-Provost
for Undergraduate Education, that we can use in providing
legitimacy to our effort. We will also be working with Donna C.
Llewellyn, the Director of Georgia Tech's Center for Enhancement
of Teaching and Learning. Dr. Llewllyn has a great deal of
experience in working with faculty and encouraging changes in
classrooms.  We will be seeking her advice on how to handle Step 3
in the most effective manner.

\subsubsection{Implementing the Intervention}
\label{subsubsec:mylabel5}

Once we have identified a willing teacher, we plan to work with
the teacher through regular meetings to help him or her implement
the activities and changes that our theory will recommend.  We do
\emph{not} plan to be cautious and objective observers.  Our hope
is to cause a change in retention rates.  We hope to see an
institutional change such that the intervention becomes permanent,
but that's beyond the scope of this project.  Our goal here is to
cause a change through targeted intervention.  While we recognize
that we will be susceptible to a Hawthorne effect, we agree with
Ann Brown that that's a desirable outcome as part of a process
leading to deeper and persistent change \cite{brown-design}.

We will record the amount of effort used in the interventions, as
we have in prior studies \cite{coweb-english}.  Our belief is that
CSCL will allow for a relatively easy to implement change that
will impact retention.  We plan to measure how easy that is.  Our
current plan is to probe the teachers on a regular basis to get
estimates of how much time the intervention is requiring.  Our
evidence from our work in English Composition classes is that the
cost in teacher time to adopt CSCL activities is negligible
\cite{coweb-english}, but we recognize that SMET faculty may not
recognize their situation as being similar to an English
composition classroom.  By tracking that time-cost data, we can
report it to targeted faculty in subsequent semesters to show that
the cost is minimal.  We hope that the low cost, with the evidence
for improved retention across the first two years of the study,
will help to convince faculty to try our approach.

During the intervention, we will continue to gather data as
described in Table~\ref{tbl:studyplans} so that we can match
progress to our theory.  In this way, we hope that we can learn
about both the process of change of retention rates in the
classes.

\subsection{Project Roles and Involvement of Students}
\label{subsec:mylabel6}

Mark Guzdial will be the PI of the project.  He will be
responsible for overall management of the project and the
achievement of project and dissemination goals.

Allison Elliot Tew will be a Research Scientist on the project.
For four years, Allison has been the Director of Student Services.
She's familiar with the systems for gathering data on classes, and
with the task of encouraging and cajoling faculty into behavior
for the students' best interests.  She also has prior experience
in project management and software engineering.  She'll handle the
day-to-day management of the project, and will be the main person
developing the measurement instruments used in the project,
reviewing the data and results, developing the theory, evaluating
the theory, developing the interventions, and working with faculty
to implement the interventions.

We will hire two Graduate Student Research Assistants (GSRAs) on
the project.  Their responsibilities will include:
\begin{itemize}
\item Data collection and analysis, including interviews,
\item Developing and maintaining software associated with the
project,
\item Developing materials for the interventions,
\item Providing support to the teachers as they begin the
interventions.
\end{itemize}

In addition, we anticipate working with individual students who
will join us looking for research opportunities.  We have been
very fortunate in finding talented students who are interested in
the goals of our projects.  For example, on the \emph{Media
Computation} course project, all the interviews were undertaken by
a talented undergraduate student, Lauren Rich, who wanted to study
gender issues in the course.  In this way, we hope to be able to
explore particular angles and offshoots of the project as the
arise, while involving additional students.

\subsection{Dissemination Plans}
\label{subsec:mylabel7}

Primarily, we hope to disseminate our results to two audiences:
\begin{itemize}
\item Learning Scientists (e.g., \emph{International Conference of
the Learning Sciences}, \emph{Computer-Supported Collaborative
Learning Conference}, and \emph{Journal of the Learning Sciences})
who will be interested in our theory of retention related to CSCL,
our intervention plans, and our results, as well as our process
for targeting and intervening in low-retention classes.
\item SMET teachers and administrators who will be interested in
mechanisms for improving the poor retention rates in these
classes.  We anticipate reaching Engineering and Computer Science
educators through the IEEE/ASEE \emph{Frontiers in Education
(FIE)}, ACM \emph{SIGCSE} (SIG Computer Science Education), and
\emph{Innovation and Technology in CS Education} (ITiCSE)
conferences.  We also hope to publish in \emph{IEEE Transactions
on Education} and \emph{Journal of Engineering Education} as high
quality and visibility forums for this research.
\end{itemize}

As we target courses in other disciplines (e.g., Mathematics and
Physics), we plan to reach out to those audiences as we have with
our previous CoWeb work (e.g., \cite{coweb-math, lissa-cccc,
cce}).

We anticipate to have two conference papers per year, and at least
two journal papers over the course of the three year project.  In
addition, we will make all of our intervention materials and our
software available on our websites.

Our websites have been an effective mechanism of intervention for
us. Our CoWeb/Swiki software and associated materials are posted
there, indexed by Google.  A search today for ``Swiki'' on
\url{http://www.google.com} returns 126,000 hits. A review of the
first couple hundred of these hits shows that the vast majority of
hits are sites using our software (available at
\url{http://minnow.cc.gatech.edu/swiki}). American schools using
CoWebs to support teaching and research including University of
Colorado at Boulder, Allegheny College, Lakota University,
University of Kansas, and the National Center for Atmospheric
Research.  Uses of the CoWeb outside of the United States can be
seen in domains .sk, .be, ch., .jp, .se, .fr, and .nz and include
significant U.S. research collaborators such as ETH.

\section{Conclusion}
\label{sec:mylabel3}

The overall focus of our project is to conduct research targeted
at a significant and stubborn problem in SMET courses.  Our
approach is to study the problem and our potential solution to the
problem, to develop a theory and test our theory about the
problem, then to apply an intervention based on our theory.  Our
potential solution is collaborative learning activities,
implemented through CSCL.

We don't expect that we will be able to make 90\% of students
succeed in every SMET class at Georgia Tech. But if we can lower
some of the 20--30\% WDF rate classes into the 10--20\% range, we
will have accomplished something significant, since we will have
carefully engineered and documented our process of intervention.
In addition, our process of choosing a research question in terms
of practice and then addressing that question will serve as a
model for future joint practice-research agendas.

\newpage


\bibliographystyle{plain}
\bibliography{proposal,finalreport}


\newpage

\section{Budget Justification} \label{subsec:mylabel8}

\emph{Personal Services}. We request 1 month of summer salary for
Mark Guzdial per year, and 12 months of Allison Tew per year, for
all three years of the grant.   In addition, we request funding
and tuition for the two GSRA's on the project. We also request
funding for 1/2 of a shared post-doc for maintenance of laboratory
equipment in the GVU Center.  We are requesting fringe and
computing charges for these personnel.

\emph{Equipment}: None is requested.

\emph{Travel}: We request \$4,000/year for travel expenses for
attending conferences where we are presenting results from this
work (predicting two per year).

\emph{Materials and Supplies}: We ask for \$2,000/year for
miscellaneous expenses, like toner cartridges and software
upgrades.

\newpage
\section{Facilities}

The College of Computing maintains a variety of computer systems
in support of academic and research activities.  These include
more than 50 Sun, Silicon Graphics, and Intel systems used as file
and compute servers, many of which are quad-processor machines. In
addition, there are more than 1,000 workstation class machines
from Sun, Silicon Graphics, Intel, and Apple especially for
student use.  A number of specialized facilities augment these
general-purpose computing capabilities.  The hardware that will be
purchased for this project will be of similar quality to what the
students use, for testing purposes, but will be set up to
facilitate development.

The Graphics, Visualization, and Usability (GVU) Center houses a
variety of graphics and multimedia equipment, including
high-performance systems from Silicon Graphics, Sun, Intel, and
Apple.  The affiliated Multimedia, Computer Animation, Audio/Video
Production, Usability/Human Computer Interface, Virtual
Reality/Environments, Electronic Learning Communities,
Computational Perception, Software Visualization, Biomedical
Imaging, Collaborative Software, and Future Computing Environments
labs provide shared facilities targeting specific research areas.
These laboratories' equipments will be of use in developing our
multimedia projects.

PI Guzdial is the Director of the Collaborative Software Lab,
affiliated with GVU.  The Collaborative Software Lab has a bank of
ten servers supporting our experimental software for studying
computer-supported collaborative learning.  In addition, we have
three Linux workstations, two NT workstations, and two Apple
workstations used for development.  The focus of the Collaborative
Software Lab is on facilitating multimedia collaboration, so
multimedia facilities available include a high-end Alesis
keyboard, projection facilities, a Canon digital video camera, and
a Nikon digital camera.

All of the College's facilities are linked via local area networks
which provide a choice of communications capabilities from 10 to
1000 Mbps.  The College's network employs a high-performance OC12C
(622 Mbps) ATM and GigabitEthernet (1000 Mbps) backbone, with
connectivity to the campus ATM network provided via OC12C.  The
primary campus Internet connection is provided by a direct 100
Mbps link to the service provider's Atlanta switching center,
augmented by OC3C ATM and OC12C connections, respectively, to the
NSF vBNS (very high performance Backbone Network Service) and
Abilene research networks.  Georgia Tech is also leading southern
regional gigabit network efforts (SoX.net, the Southern
Crossroads) as part of Internet2.

Additional computing facilities are provided to the Georgia Tech
campus by the Institute's Office of Information Technology (OIT),
including five public-access clusters of Sun, Apple, and Dell
workstations, a collection of Sun multi-processors which are
treated as a single computational resource via login load sharing,
and various mainframes.

\end{document}
