\documentclass[12pt]{article}
%\usepackage[]{smalltalk}
\usepackage{url}
\usepackage{graphicx}
%\usepackage{doublespace}
\usepackage{verbatim, code, hyperlatex}
\usepackage{mysects}


\textheight 8.6in \textwidth 6.5in \topmargin -0.25in
\oddsidemargin 0in \evensidemargin 0in


\newcommand{\proposalTitle}{Developing Methods to Assess Innovative Curricula for Women and Non-Majors in Computer Science }
\newcommand{\mycomment}[1]{\textbf{\emph{#1}}}


\begin{document}

% Definition of title page:

\title{\Large \proposalTitle}
\author{
    PI: Mark Guzdial %\\ Co-PI: Blair MacIntyre
}
% \date{October 2001}    % optional

\maketitle

\thispagestyle{empty}

%\vspace*{.5in}

\noindent


{\Large\bf Project Summary}

\vspace{.5em}

\noindent

Computer science educators are aware that they have a problem in
motivating and engaging students, especially non-CS-majors and
women, who are withdrawing or failing introductory computing
courses at an alarming rate, or simply avoiding computing
entirely.

There are several innovative efforts to address this problem. For
example, CMU has made great progress on increasing the percentage
of women in their undergraduate program.  We have had a successful
trial offering of a course in \emph{Introduction to Media
Computation} aimed at non-CS majors. The course was quite
successful at its goals. 121 students enrolled---2/3 women. 89\%
of the class earned an A, B, or C in the course.  60\% of the
respondents on a final survey indicated that they would like to
take a second course on the topic.

The problem that this proposal addresses is how to go beyond
retention and WFD rates.  How do we know that students are really
\emph{learning} in these new approaches?  How do we know that we
are changing attitudes towards computer science?  How do we know
that we are having an impact long-term?

We propose to develop measures of CS1 concept learning, attitudes
toward computer science, and longitudinal impact of an
introductory computing aimed at women and non-CS majors.  We plan
to disseminate these measures through papers and also faculty
workshops at CS Education conferences such as ACM SIGSE and
ASEE/IEEE FIE.  We plan to study faculty use of these measures
through email surveys of faculty who attend our workshops or use
our course materials

\emph{Intellectual Merit}:  This proposal seeks to create methods
to understand the value of innovations to attract women and non-CS
majors to computing

\emph{Broader Impacts}: The potential broader impacts is to help
develop better methods to address the lack of women in computing
and create a broader base of computing-literate citizens, by
helping to provide an infrastructure for comparing methods.


\newpage

%Table of Contents Page
    \thispagestyle{empty}

\begin{center}
    {\Large \proposalTitle
}
\end{center}


\tableofcontents


\newpage

%Body of Proposal

\begin{center}{\Large \proposalTitle}
    \end{center}

\pagenumbering{arabic}


\section{Goal: Assessing New Routes to Computer Science}

% CS1 is not currently successful

Computer science departments are not currently successful at
reaching a wide range of students who are taking introductory
computer science.  The evidence for this statement includes
international studies of programming performance
\cite{iticse2001}, declining retention rates
\cite{retention-sigcse}, and failure rates sometimes as high as
50\% \cite{failurerates}.  This comes at a time when the need for
Information Technology (IT) professionals is growing
\cite{freeman}.

At Georgia Institute of Technology (``Georgia Tech''), all
students are required to take an introductory course in computing,
including programming skills. Our traditional CS1 course, the only
one that met the requirement before the Spring 2003 semester, is
undoubtedly one of the most unpopular courses on campus,
especially among those \emph{not} in explicitly computing-related
fields.  While this is certainly a problem for the College of
Computing at Georgia Tech (where the course has its academic
home), it points towards a larger problem for the field.

Alan Perlis in April 1961 made the first argument that programming
should be part of a liberal education for \emph{all} students.  If
Calculus is the study of \emph{rates}, and that's important enough
to be part of the liberal education, then so should computer
science.  Perlis argued that computer science is the study of
\emph{process}, which is certainly relevant to even more fields
than those concerned with rates.  The argument has been echoed and
strengthened over the intervening years---by Seymour Papert
arguing for a programming as a way of learning about learning
\cite{papert-memo}\cite{mindstorms}, to Andrea diSessa's arguments
for ``computational literacy'' as a critical component of many
fields \cite{changingminds}.  As long as non-CS majors have such a
dislike for computing, the hope is diminished for computer science
as an accepted part of a liberal education and for computing
generally to meet its potential for intellectual impact across the
range of disciplines, not just in computational science and
engineering.

% Most of all with women

Introductory computer science classes are also not meeting the
needs of women.  The rates at which women take computer science
classes are falling. While the factors causing women to avoid
computer science (and IT careers in general) are multi-faceted,
the curriculum does play a significant role \cite{margolis}.  A
report in 2000 by the American Association of University
Women~\cite{tech-savvy} suggested that part of the problem, at
least for women, is that computer science courses are, frankly,
too boring.  Specifically, they claim that computer science
courses are ``overly technical'' with little room for
``tinkering.''  At a session on increasing enrollment of women in
computer science at the latest ACM SIGCSE conference, speakers
reported that women who pursued computer science degrees were
surprised at how much ``creativity'' there was in later computer
science courses---introductory courses did not highlight that
aspect of CS~\cite{womensigcse2001}.  Additionally, women
undergraduate CS students tend to be interested in real
applications of computing, as opposed to simply computing for its
own sake \cite{margolis}\cite{tech-savvy}. If we can address the
reasons why women are avoiding computer science, we may be able to
interest more males, too---all those currently expressing
disinterest in computing. ``Women in engineering programs are kind
of like `canaries in the coal mine''', said Stephen W. Director,
Chair of the Engineering Dean's council. ``If women do well in a
program, most likely everyone else will also do well in the same
type of program.''~\footnote{Testimony by Stephen W. Director,
Chair, Engineering Dean's Council, American Society of Engineering
Education, to the Commission on the Advancement of Women and
Minorities in Science, Engineering, and Technology Development,
Washington, DC. July 20, 1999}

% Goals: Improved retention, better learning, better attitudes toward computer science
There are new, innovative efforts to address these issues.  Most
well-known among these is the work at Carnegie-Mellon University
(CMU) which was able to raise dramatically the percentage of women
in their undergraduate CS program \cite{margolis}.  New courses
are being developed, such as an Alice-based pre-CS1 course
\cite{alicecs1}, whose goal is to meet some of the problems
identified as preventing women and non-CS majors from succeeding
at Computing.  The Alice-based course is succeeding at having
higher retention rates than other introductory computing courses
\cite{alicecs1}.  New approaches within existing classes, like
pair-programming methods, are leading to improved retention in the
class and within the CS major
\cite{pair-retention,pair-programming}.

We also have developed a course, \emph{Introduction to Media
Computation}, to meet the needs of women and non-CS majors. The
course was offered as a pilot in Spring 2003 with 121 students,
2/3 women, with \emph{none} of the students majoring in CS or
Engineering. These students were from the College of Architecture,
Ivan Allen College of the Liberal Arts, Dupree College of
Management, and the School of Biology. The hypotheses of this
effort were that this course would demonstrate: (1) Improved
student retention, (2) better student attitudes toward computer
science, and (3) better student learning of computer science.

The course was quite successful at the goal of improving student
retention, but it's not clear if it met the goal of impacting
student attitudes or learning. 121 students enrolled -- 2/3 women.
89\% of the class earned an A, B, or C in the course, i.e., 11\%
Withdrawal-Failure-or-D (WFD) rate. 60\% of the respondents on a
final survey indicated that they would like to take a second
course on the topic--but at the same time, only 9\% claimed that
they would ever take another course in computer science! They did
learn enough programming to be successful at it. Students wrote
eight programs (six collaboratively and two individually) creating
or manipulating pictures, sounds, HTML pages, and movies, with
some of their programs reaching 100 lines of code. When we
attempted to use isomorphic exam problems (as in \cite{stable}) to
compare students between the classes, we ran into difficulties of
course sequencing and language differences.

% Why this is a general path
We do not believe that media computation is the \emph{only}
context in which students not traditionally attracted to computer
science might succeed.  For example, we have described another
potential CS1 course organized around information management in
which students might build Web harvesting programs and data
visualizations \cite{sigcse-editorial}. The general approach we
are exploring is to use an application domain of information
technology that is rich in CS concepts and relevant to the
students, then explore introducing computing concepts in terms of
that domain.  The Media Computation project is a trial and model
of that approach.

Rather, we anticipate that there will be \emph{many} innovative
approaches developed over the next five to ten years, seeking to
improve the motivation and engagement with computer science among
students in groups not currently succeeding at computer science.
We predict that they will all reach the same stage that we and
others are meeting.  It's easy to show that one's innovation
changes WFD rates or retention rates.  It's another thing to show
that the innovation is actually making a difference:
\begin{itemize}
\item How do we know that students are really \emph{learning}
after the changes?  One way of lowering WDF rates is to simply
make the class easier.

\item Are the innovative classes changing attitudes about computer
science? Or are the other classes facing the same muddle of
attitudes that we have---that students like our class, but still
don't like computer science?

\item How do the innovations impact the students beyond the
course?  We would like to see the students go on to take more
computer science classes, and perhaps some percentage become
computer science majors.  But we would also like to see an impact
within the students' major.  Introductory computing courses for
non-majors are growing (e.g., \cite{cs0brandeis} \cite{vrcs1}
\cite{graphicscs1}), but we have little idea of how those courses
is actually changing how students relate to technology.  For
example, we would like to know if students who take introductory
computing courses report that they perceive that they feel that
they can learn and use new technology, perhaps more easily (in
their estimation) than others without computing education.
\end{itemize}

% Scope of proposal: Description of Proof-of-concept course,
% assessment results, where we want to go next

In this proposal, we propose to create three measures to go beyond
issues of retention and WFD rates to show an impact of educational
innovations targeting women and non-CS majors.  Our measures will
focus on attitudes toward computer science and technology,
learning of concepts related to CS1, and on longer term impacts of
the computing course on the students' later academic career. We
begin by describing our proof-of-concept trial offering of the
course (funded by NSF CCLI grant {\#}0231176) and its evaluation
results, because it was the results from that trial that has led
to our motivation to develop these methods.  We end by describing
our dissemination plans, which include faculty workshops at ACM
SIGCSE and ASEE/IEEE FIE conferences to teach broadly (e.g.,
including our methods, but also including measures such as
retention and WDF rates) how to assess the impact of educational
innovation targeting under-represented groups in computer science.

\section{Results from Proof-of-Concept Trial}

\textit{Introduction to Media Computation} (CS1315 at Georgia
Tech) is our new introduction to computation and programming
contextualized around creation and manipulation of media. The
explicit goal is to interest groups typically disinterested in
such courses (especially women and non-CS majors). The course was
offered as a pilot in Spring 2003.

The course was a success on several measures.

\begin{itemize}
\item 89{\%} of the students who enrolled earned an A, B, or C. Our
  ``best-practice'' point of comparison in the literature had a
  66.4{\%} success rate for non-majors in a specialized CS1
  \cite{pair-programming}.
  \item The course met its goal of getting female students to find CS
  interesting, relevant, and creative where they hadn't previously.

\item We attempted direct comparisons on problems on exams between
our traditional CS1, a prototype engineering CS1 course, and the
Media Computation class. We found that such comparisons are
fraught with difficulties.  Nonetheless, student performance in a
problem described in the literature on CS education was relatively
comparable between the Media Computation and traditional CS1
students.

\item Students found the course relevant, and the majority of
  respondents to a final survey indicated that they would like to take
  a second course if it were offered.

\end{itemize}

Our assessment of the course, and the comparison with the other
courses, went beyond measures of success to explore how and why
the class was successful, and who was successful. For example,
Collaboration may have played as much a role in the success of
CS1315 as the context. When asked the open question on the final
survey, ``What absolutely must \textit{not} change about the
course?'', nearly 20{\%} of the Media Computation student
respondents named the collaboration tool CoWeb \cite{cowebiticse}
\cite{cowebfie} and over 20{\%} mentioned collaboration in
general.  In general, students from Georgia Tech's \emph{Ivan
Allen College for Liberal Arts} were the most positive about the
course (e.g., had the largest percentage of students who would
recommend it to others, who planned to use programming in the
future, and who planned to take further CS courses).

\subsection{Implementation of the Course}
The choice of multimedia as the course context in this project
allows us to address several of the issues identified as critical
for increasing the number of women in computing.  Several of the
academic units at Georgia Tech emphasize that multimedia literacy
is a critical component of educated professionals in the future
\cite{medialiteracy}. Multimedia is a real applications
domain---especially true at Georgia Tech, with Turner
Broadcasting, CNN, and the Cartoon Network based within a few
miles of campus. Multimedia is clearly not computing for its own
sake.  Further, the multimedia approach allows us to give students
assignments that are open-ended and creative.  The emphasis on
multimedia allows the class to be more concrete, more relevant,
and more creative than traditional CS1 approaches.

The premises and core concepts of the Media Computation course
are:
\begin{itemize}
\item All media are being published today in a digital format.
Digital formats are amenable to manipulation, creation, analysis,
and transformation by computer.  Text can be interpreted, numbers
can be transformed into graphs, video images can be merged, and
sounds can be created.  We call these activities \emph{media
computation}.

\item Software is the tool for manipulating digital media.  Knowing
  how to program thus becomes a communications skill.  If someone
  wants to say something that her tools do not support, knowing how to
  program affords the creation of the desired statement.

\item Core computer science concepts can be introduced through
media computation. For example, programs can get large and
cumbersome. Abstraction is our tool for managing program
complexity and allowing programs to become even larger yet more
flexible. However, computing has limitations.  There are some
programs that cannot complete in our lifetime, and knowing that
these limitations exist is important for technological
professionals.
\end{itemize}

%Language of the course

We used the programming language \emph{Jython} for the course.
Jython\footnote{\url{http://www.jython.org}} is a variation of the
Python\footnote{\url{http://www.python.org}} programming language
which has been designed to be easily used by novices and
non-technical users.  Jython is a variation of Python written in
Java. Jython can instantiate and subclass Java classes, and Jython
can be used for virtually anything for which Java can be used. We
considered other languages (e.g., Java and Scheme), but selected
Jython based on survey responses from students and teachers
\cite{mediacomp}. Jython's design is based on the research
findings on studying novice programmers (e.g., \cite{looping},
\cite{noiterate}, \cite{pane01}).

%The course was offered as a three credit hour course with five
%weeklong lab activities, six homeworks, three in-class
%examinations, and two take-home examinations (programming
%assignments) with optional recitations.


%\begin{itemize}
%\item Week 1: Introduction to the course and the argument for why
%  media computation. Introduction to variables and functions, in the
%  context of playing sounds and showing pictures.
%\item Weeks 2--3: Pictures as a media type, including psychophysics
%  (why don't we see 1024x768 dots on the screen?), looping to change
%  colors with a simplified \textbf{for} loop
%  (Figure~\ref{fig:greyscale}), conditionals to replace specific
%  colors, then indexing by index numbers to implement mirroring, rotating,
%  cropping, and scaling.
%  \begin{figure*}[htb]
%\begin{example}
%def greyScale(picture):
%  for p in getPixels(picture):
%    intensity = (getRed(p)+getGreen(p)+getBlue(p))/3
%    setColor(p,makeColor(intensity,intensity,intensity))
%\end{example}
%\caption{An example Jython program using our API to convert a
%picture to greyscale\label{fig:greyscale}}
%\end{figure*}
%
%\item Weeks 4--6: Sound as a media type, including psychophysics (how
%  human hearing limitations make MP3 compression possible), looping to
%  manipulate volume,
%   %(Figure~\ref{fig:normalize}),
%   then indexing by
%  index numbers to do splicing and reversing of sounds.
%  %(Figure~\ref{fig:backwards}).
%  Include discussion of how to debug and
%  how to design a program, as those issues arise. One lecture on
%  additive and FM sound synthesis.
%
%%\begin{figure*}[htb]
%%\begin{example}
%%def normalize(sound):
%%    largest = 0
%%    for s in getSamples(sound):
%%        largest = max(largest,getSample(s) )
%%    multiplier = 32767.0 / largest
%%
%%    print "Largest sample value in original sound was",  largest
%%    print "Multiplier is", multiplier
%%
%%    for s in getSamples(sound):
%%        louder =  multiplier * getSample(s)
%%        setSample(s,louder)
%%\end{example}
%%\caption{An example Jython program using our API to normalize sounds to a maximum volume\label{fig:normalize}}
%%\end{figure*}
%%
%
%
%%\begin{figure*}[htb]
%%\begin{example}
%%def backwards(filename):
%%  source = makeSound(filename)
%%  target = makeSound(filename)
%%
%%  sourceIndex = getLength(source)
%%  for targetIndex in range(1,getLength(target)+1):
%%    sourceValue = getSampleValueAt(source,sourceIndex)
%%    setSampleValueAt(target,targetIndex,sourceValue)
%%    sourceIndex = sourceIndex - 1
%%
%%  return target
%%\end{example}
%%\caption{Return the sound in the file backwards\label{fig:backwards}}
%%\end{figure*}
%
%\item Week 7: Text as a media type: Searching for text, composing
%  text, reading text from a file and writing it to a file. An example
%  program parses out the temperature from a downloaded weather page.
%\item Week 8: Networks,
%  including making the temperature-finding program work from the
%  ``live'' Web page. Introduction to HTML.
%\item Week 9: Discuss media transitions.  Moving from sound to text
%  and back to sound again.  Using Excel to manipulate media after
%  converting it to text.
%\item Week 10: Introduction to databases: Storing media in databases,
%  using databases in generating HTML.
%\item Week 11: Movies: How persistence of vision makes animations and
%  movies possible, generating frames using the various techniques
%  described earlier in the semester, manipulating whole directories of files.
%\item Week 12: ``Can't we do this any faster? Why is Photoshop faster
%  than Python?'' Introduction to how a computer works (e.g., machine
%  language), and the difference between an interpreter and a
%  compiler. Algorithmic complexity and the limits of computation.
%\item Week 13: ``Can we do this any easier?'' Decomposing functions,
%  modularity, and functional programming (map, reduce, filter, and
%  simple recursion).
%\item Week 14: ``Can't we do this any easier?'' Introduction to
%  objects and classes.
%\item Week 15: ``What do other programming languages look like?''
%  Brief overview of JavaScript and Squeak.
%\end{itemize}

The general structure of the course was 11 weeks of media-relevant
material (e.g., sound, pictures, movies, text manipulation,
network, and database), with four weeks of computer science that
answers questions that arose during the course of the 11 weeks.
Student homework assignments started with simple Photoshop-style
filters (average score 92\% with a standard deviation of 22.1).
Some of the assignments invited creativity, such as the third
homework
%(Figure~\ref{fig:hw3})
which required creation of a collage, but allowing students to
select whatever pictures they wanted in the collage (average score
86.3\% standard deviation 26.0). These collage programs became
large--the average number of lines of code (of students who gave
us consent to look at their homework results) was 64 with a
standard deviation of 32.4. The largest was 166 lines of code. The
final homework assignment involved programmed Web access and
creation of animations (average score 84.6\% standard deviation
29.2). Final exam questions spanned a wide range of computing
concepts, including issues of networking, databases, and computing
theory.

%{\small
%\begin{figure}
%\begin{tabular}{|p{6in}|}\hline
%Write a program named \textbf{hw1} to accept a picture as input,
%and change its pixels as follows:
%
%\begin{itemize}
%\item Set the green component to 125{\%} of its current value
%\item Decrease the blue by 25{\%}
%\item Decrease the red by 75{\%}
%\end{itemize} \\ \hline
%\end{tabular}
%\caption{First homework assignment by \emph{Media Computation}
%  students\label{fig:hw1}}
%\end{figure}
%
%%\begin{figure}
%%\begin{tabular}{|p{6in}|}\hline
%%At the Feb. 7 class, we recorded this sound
%%\emph{thisisatest2.wav} (``This is a test.''). Using MediaTools,
%%we found the end points for each of the words in the sound: (Table
%%omitted)
%%
%%Write a function \textbf{backSpliced} that splices the last word
%%(``test'') into the front of the word, but \textit{backwards}, so
%%that the result is:``Tset is a test.'' \\ \hline
%%\end{tabular}
%%\caption{First take-home exam assignment by \emph{Media
%%Computation}
%%  students\label{fig:te1}}
%%\end{figure}
%
%%\begin{figure}
%%\begin{tabular}{|p{6in}|}\hline
%%Create a collage of the same image at least three times fit onto the 7x9.5in
%%blank JPEG. (You are welcome to add additional images, too.) You can do any
%%of
%%
%%\begin{itemize}
%%\item scaling the image,
%%\item cropping the image,
%%\item creating a negative of the image,
%%\item shifting or altering colors on the image,
%%\item and making it darker or lighter.
%%\end{itemize} \\ \hline
%%\end{tabular}
%%\caption{Third homework assignment by \emph{Media Computation}
%%  students\label{fig:hw3}}
%%\end{figure}
%
%%\begin{figure}
%%\begin{center}
%%\includegraphics[width=3in]{StudentCollages2.eps}
%%\end{center}
%%\caption{Four student collages shared from the third homework
%%  assignment\label{fig:collages}}
%%\end{figure}
%
%\begin{figure}
%\begin{tabular}{|p{6in}|}\hline
%\emph{http://www.cnn.com} is a popular news site. You are to write
%a function (named ``hw6'') that will input a directory as a string
%then:
%
%\begin{itemize}
%\item Visit \emph{http://www.cnn.com} and pick out the
%  \textit{top three} news stories headlines in the \textbf{More News}
%  section.
%\item Create a ticker tape movie on the 640x480 canvas of all three
%  news stories. Have one come across at y=100, another at y=200, and
%  the third at y=300. Generate 100 frames, and don't have the ticker
%  tapes move more than 5 pixels per frame.  Store the
%  frames to files in the input directory.
%\end{itemize}
% \\ \hline
%\end{tabular}
%\caption{Final homework assignment by \emph{Media Computation}
%  students\label{fig:hw6}}
%\end{figure}
%
%%\begin{figure}
%%\begin{center}
%%\begin{tabular}{|p{6 in}|}\hline
%%
%%\begin{itemize}
%%\item Your father calls you. ``My tech support people are saying that the
%%company website is down because the database program is broken. What does
%%the \textit{database} have to do with our company \textit{website}?'' You explain to him how databases can be integral to running large websites.
%%Explain both (a) how the website comes to be authored through the database
%%and (b) how the HTML is actually created.
%%
%%\item  What are three reasons why someone should use multiple functions in
%%their programs, rather than one big function?
%%
%%\item What is a protocol and where is it specified in a URL?
%%
%%\item Is it possible to write a program to solve the \textit{Traveling Salesman} \textit{Problem}? Is it possible to
%%write a program to solve the \textit{Halting Problem}?
%%\end{itemize} \\ \hline
%%\end{tabular}
%%\end{center}
%%\caption{Some final exam essay questions from the \emph{Media Computation}
%%  class\label{fig:final}}
%%\end{figure}
%}


We developed \emph{JES} (Jython Environment for Students),
modelling it after the popular DrScheme environment \cite{htdp}.
Development of
  JES also included development of a simplified API for accessing the
  Java media library.
%  We were pleased to receive \emph{no}
%  reports of crashes or failures of JES over the course of the
%  semester.

We used our existing collaboration technology, CoWeb
  \cite{cowebjls, cowebiticse}, to support student question asking and
  sharing of media artifacts.  The \emph{constructionist} theory of
  learning suggests that the creation of public artifacts creates a
  strong potential for learning \cite{papertconstructionism}.  We have
  had good success in the past using our collaboration tools to
  support sharing of multimedia artifacts to encourage critique and
  motivate learning \cite{colleen}\cite{icls-2cool}\cite{coolstudio}.
The CoWeb became the central place for question-asking and
discussion, which is a critical role in CS1, since comfort in
asking questions has been highly correlated with attrition in CS1
\cite{comfort-cs1}.  From interviews, we know that the CoWeb
increased that comfort.

\begin{quote}
\textbf{Q. Have you consistently felt comfortable asking
questions? } Student D: Not at the beginning. One on one, yes. In
lecture, not at the beginning because I felt that I was so far
behind other people and the ones who were putting things on the
web were the ones who really know stuff. But now I have no
problem.

\textbf{Q. Do you think the CoWeb is beneficial? } Student D: Yes.
And there's no reason to feel uncomfortable because if you feel
dumb, just don't put your name at the end! I did that a few times.
\end{quote}


\subsection{Evaluation Methods and Results}

Our evaluation effort had three hypotheses: (1) the prototype
course will improve student retention, especially among
non-CS-majors and women; (2) the prototype course will improve
student motivation toward computer science; and (3) the prototype
course will lead to good student learning, perhaps better than in
a comparable course.

Our evaluation effort was conducted under the review of the
Georgia Tech Human Subjects Review Board, to whom this proposal
and all our instruments were submitted for review as they were
developed.  Students were informed about the research project, the
instruments, how they might be impacted, and potential risks.
Assessment was organized such that the PI and the assessment
team\footnote{Graduate research assistants
  Andrea Forte and Rachel Fithian, and undergraduate research
  assistant Lauren Rich} designed the assessment, but the assessment
team gathered and analyzed all data apart from the PI, since he
was also the teacher of the course.

We gathered and analyzed several sources of data.
\begin{itemize}
\item We had demographic data which we used to address the
  retention question.  We had course work from consenting students.
\item We took surveys during the first week of the term, midway
  through the term, and in the last week of the term.  We used the
  same surveys in two other classes: A section of our traditional CS1
  course ($n=127$), and a section of the prototype Engineering-oriented CS1
  ($n=75$).
\item We attempted to develop and distribute polymorphic problems
  (same problem, but changed in language or other minor details for
  the course context, a method we used successfully in \cite{stable}) that were offered in all three CS1's on
  exams--for example, on computing an average test grade and
  determining an appropriate letter grade, and on the infamous Rainfall
  Problem \cite{soloway-yale-studies}.
\item We interviewed a sample of female students in the \emph{Media
    Computation} class on their attitudes toward computer science and
    the course. We had seven interviews in total--three early in the
    semester, and all three were interviewed again at the end of the
    term along with another student at the end.
%  \item We also observed volunteer students working on their homework
%    so that we could have a sense of strategies and problems,
%    especially with the technology.  There were 11 observation
%    sessions, including 6 with just one (male) student to track
%    development.  The other observation sessions were all with female
%    students.
\end{itemize}


\subsubsection{Evaluation of Retention}

Retention rate is defined operationally in this study in two ways.
The \emph{completion retention rate} is defined as the
  ratio of the number of students completing the course to the number
  of students enrolled in the course. The \emph{course success retention rate} is defined as the
  ratio of the number of students completing the course with a C or
  better to the number of students who enrolled in the course.
  (Several academic units at Georgia Tech require students to re-take
  courses if they received a D.)

The literature on both rates is rather negative.  Completion rates
of less than 30\% is not uncommon in CS1 \cite{failurerates}. In
an email survey on the ACM SIGCSE members list, course success
retention rates of less than 50\% were reported at several schools
\cite{retention-sigcse}. The ``best-practice'' comparison that we
have for non-majors, in a pair-programming CS1 class, had a 66.4\%
course success retention rate \cite{pair-programming} (in that
study, compared to a non-majors success rate of 55.9\% in a
traditional CS1). Our traditional CS class typically does
remarkably well in comparison with the literature, with a
completion retention rate of 80-90\% and a course success
retention rate better than 70\%, so it offers a high standard to
meet.

Our course completion retention rate for Spring 2003 was 98\%. Our
course success retention rate was 89\%.  During that semester, our
comparison section of our traditional CS1 had a 57.1\% course
success rate.

\subsubsection{Evaluation of Motivation}

The stated goal for the prototype course was to address the
disinterest in computer science, especially among non-CS-majors
and women.  We were concerned with three kinds of motivational
factors:
\begin{itemize}
\item Do students find the current course engaging?  An engaging
  course is more likely to lead to learning \cite{intrinsic}
  \cite{pintrich} \cite{pbl}.
\item Do students find computer science engaging?
\item Would students consider taking future computer science courses?
  While the goal of the prototype course is to simply address issues
  of disinterest, positive response to this question would suggest
  that the media computation (``data-first'') approach has potential
  for attracting potential CS majors.
\end{itemize}

When asked what they like about the class in the midterm survey,
the students affirm that we're succeeding at creating a course
that students recognize for its relevance, particularly for non-CS
majors:

\begin{itemize}
%\item ``I like the feeling when I finally get something to work.''
\item ``Very applicable to everyday life.'' \item ``I dreaded CS,
but ALL of the topics thus far have been applicable to my future
career (and personal) plans.''
%\item ``When I finally get a program to work like I want it to.''
\item ``The professor answers questions in class and online and is
concerned about our success in the class. He also seems to
understand that most of us are not engineers and most likely won't
do straight programming in the future---just the way of thinking
is important.''
\item ``I think that we're doing things that I
could actually use as an architecture major--I like dealing with
pictures and sounds.
\end{itemize}

When we asked students ``What is something interesting,
surprising, or useful that you learned?'' we found that students
appreciated the relevance of the course and even found the
computer science interesting (again, all female respondents):

\begin{itemize}
\item ``The most useful things I have learned are the basics about computers and pictures/sound...
interesting and useful to real life applications.''
\item ``Just general concepts about programming. It's pretty logical, sort of like in math, so it's understandable.''
\item ``Programming is fun and ANYONE can do it!''
\end{itemize}

Students who have positive attitudes about CS and who enjoy
computing are likely to continue in computer science. While it is
too early to tell what CS courses Media Computation students will
take in the future, we asked on the final survey whether or not
they would be interested in taking a more advanced media
computation course. 60{\%} of the female respondents answered that
they would take a second course (overall course average of
63{\%}).

Surprisingly, when asked \textit{on the same survey} whether or
not they plan to take more CS courses, only 6{\%} of those same
female respondents responded affirmatively (9.3\% overall). Why
would 60{\%} be willing to take media computation, while only
6{\%} planned to take more CS courses? One possible explanation is
that there is currently no advanced media computation course, and,
given the current selection of CS courses, Media Computation
students don't see a compelling reason to take more. Another might
be interpretation of the question: Students may interpret ``CS
courses'' as ``traditional CS courses'' as opposed to specialized
courses like Media Computation.

Students indicated that the course changed their attitudes about
programming in general, into something that they could imagine
themselves doing in the future.  On the final survey, 30\% of the
students indicated that they believed that they will program again
in the future.   In interviews, students talk about programming in
the future.

\begin{quote}
\textbf{Interviewer: What do you think about the homework
galleries on the CoWeb?}

Student A: I don't ever look at it until after I'm done, I have a
thing about not wanting to copy someone else's ideas. I just wish
I had more time to play around with that and make neat effects.
But JES will be on my computer forever, so{\ldots} that's the nice
thing about this class is that you could go as deep into the
homework as you wanted. So, I'd turn it in and then me and my
roommate would do more after to see what we could do with it.
%
%\textbf{Interviewer: Have you ever written code outside of
%assignments?}
%
%Student B: Sometimes I would write other stuff on my way to an
%assignment but not just like I sat down and wrote something. I
%don't have time to play with it. Like, I'm not gonna delete JES
%off my computer, and I may play with it when I get some free time
%later on, but not yet.
\end{quote}

%{\small
%\begin{table}[htb]
%\begin{center}
%\begin{tabular}{|p{72pt}|p{36pt}|p{52pt}|p{52pt}|p{52pt}|p{52pt}|}
%\hline & $N$ respondents.& Will use programming again in the
%future.& Feel they learned to program.& Plan to take more CS. &
%Would recommend 1315 to friends who didn't have to take it. \\
%
%\hline Architecture& 8& 12.5{\%}& 50.0{\%}& 0.0{\%}&
%37.5{\%} \\
%\hline Ivan Allen (liberal arts)& 24& 41.7{\%}& 91.7{\%}&
%17.4{\%}&
%82.6{\%} \\
%\hline Management& 19& 21.1{\%}& 68.4{\%}& 5.3{\%}&
%68.4{\%} \\
%\hline Science& 3& 33.0{\%}& 100{\%}& 0.0{\%}&
%50.0{\%} \\
%\hline
%\end{tabular}
%\caption{\emph{Media Computation} student responses on final
%survey by
%  College of major\label{tbl:college}}
%\end{center}
%\end{table}
%}

%A striking example of change in attitudes toward CS can be found
%in the following statement made by a female Media Computation
%student when asked whether or not the course had changed her
%perception of computer science:
%
%\begin{spacing}{1.5}
%\begin{quote}
%  ``YES, I'm not intimidated by it anymore. My mom was SO surprised
%  when I told her that I want to be a TA\footnote{All of our CS1
%    courses make extensive use of undergraduate teaching assistants.}
%  she almost fell on the floor, cuz she's heard me complain for 3
%  years about taking this class and now I want to go do it to myself
%  again!''
%\end{quote}
%\end{spacing}
%
%Many students were clearly excited about the potential for using
%media in ways they had not previously encountered. Two students
%reported on the midterm survey that they had written programs to
%reverse popular songs, in order to find out if there were hidden
%messages. One student reported using Python to create an online
%scrapbook. Students often turned in homework assignments that
%included far more complex code than was required.

\subsubsection{Evaluation of Learning}

We developed several problems that we attempted to put on all
three courses' exams and quizzes. Unfortunately, a number of
issues such as different rates of development and sequencing of
concepts in each course, different times and numbers of exams and
quizzes, and problems equating concepts in different languages
prevented us from distributing the problems as uniformly as we
would have liked. In addition, those questions that did make it
onto multiple exams were modified by the individual course
instructors and teaching assistant to such an extent that it is
difficult to compare them.

In general, we found that the different programming languages
used, the differences in directions given on exams, and the kinds
of examples provided on the exams created unique conditions for
each course and rendered the results fundamentally incomparable.
For example, the most common problem among the Engineering
students' was mishandling of \code{ELSE} clauses. Media
Computation students never used an \code{ELSE} clause.  It's
difficult to compare these results in terms of learning.

A more reliable indicator of the Media Computation students'
programming achievement might be found in students' attempt to
solve a classically difficult problem, such as \textit{The
Rainfall Problem.} The rainfall problem is: ``Write a program that
repeatedly reads in positive integers, until it reads the integer
99999. After seeing 99999, it should print out the average.'' At
Yale in the mid-80's, Elliot Soloway gave this problem to several
groups of students \cite{soloway-yale-studies}. Of first semester
CS1 students, only 14\% of the students got it correct (completely
correct other than syntactic errors).  Only 36\% of CS2 students
got it correct, and only 69\% of students in a junior-senior level
systems programming course.  This was one of the first empirical
demonstrations that learning about programming was less than what
we might have expected. (This finding was replicated in
\cite{iticse2001}.)

Our students were not directly comparable to the Yale students.
For example, we hadn't covered the same things (e.g., at the time
of the exam, they had only seen \code{WHILE} once in class), and
we had to change the wording of the problem accordingly.
Nonetheless, 14 people out of 113 (12{\%}) who took the test ``got
it''--by Soloway's standards that means they had a correct
solution (aside from syntactic errors). With partial credit, the
average on the problem was 45{\%}.

The traditional CS1 also included a variation of the rainfall
problem on \emph{two} of their tests, but we were unable to get
the raw problems to code ourselves and compare. On the first test
(which stipulated the use of tail recursion), the result was an
average score of 53{\%}. On the second test (which was open to any
kind of iteration), the result was an average score of 60{\%}.

Overall, the rainfall problem continues to be a hard problem for
CS1 students. These results don't show that the Media Computation
students are learning programming remarkably well. But they do
show that these students aren't out of the ballpark either.  The
bottom line is that it's still an open question how much the Media
Computation students learned about programming or CS1-relevant
concepts, and how they compare to students in other classes.

\subsection{Dissemination and Self-Sustained Distribution}

During the time of this project, we have been very active in
disseminating the results and materials, helping others to adopt
the materials, and taking steps toward self-sustained
distribution.
\begin{itemize}
\item At SIGCSE2003, Guzdial hosted a tutorial on \emph{Media
    Construction Projects in Computer Science Courses} which covered
  the basics of doing media computation in Python, Java, and Squeak.
  The 11 participants all stated on the final evaluation that they
  would recommend this workshop to others.
\item A paper on the planning and development
  of the Media Computation course was presented at ITiCSE 2003
  \cite{mediacomp}, which will be presented in July 2003.
\item Guzdial spoke on the course at a meeting of
  representatives from all the CS departments of the University
  System of Georgia.  Several schools expressed interest in adopting
  the course.  Charles Fowler of Gainesville College, a two-year
  college, is offering the course as a trial this
  summer, with is currently offering two sections of the course.  Guzdial was also invited to speak at
  Kennesaw College and
  Augusta State College, each four year colleges.  Guzdial is actively
  working with Kennesaw on their plans to trial the course.
\item Prentice-Hall has signed a contract with Guzdial to
  produce a book \emph{Introduction to Media Computation} with no
  completion date at this time.
\end{itemize}

\section{Project Plan}

While our trial results are strong, it's also clear that our
support for our hypotheses about changing attitudes and learning
is incomplete.  At the same time, we see that similar efforts to
target innovation at women and non-CS majors (e.g.,
\cite{margolis} \cite{pair-retention} \cite{cs0brandeis}
\cite{alicecs1}) have similar problems and challenges.  For
example, the CMU effort used interviews to track female students'
attitudes, but did not develop any measures to check attitudes
across their female population as they implemented their
curricular changes.

Over the next few years, we can anticipate more efforts to
innovate in CS education explicitly targeting women and non-CS
majors.  We expect that these efforts will probably reach the same
place that the rest of us are at now---we can easily assess
retention and WDF rates, but we are challenged to assess learning,
attitudinal change, and long term impact.

We propose to develop methods to address learning, attitudinal
change, and long term impact on attitudes and use of technology.
We plan to develop these in the context of our course, but then to
disseminate them and evaluate how they are used in new contexts.
Our course is being offered to 300 students in Fall 2003 semester,
with plans to offer 400 seats in the Spring 2004 semester, with
similar levels of offerings in future years.

The summary of our project plan is:
\begin{itemize}
\item To develop a survey to measure attitudes toward computer
science, which will be validated internally and with interviews.

\item To develop an inventory of CS1-related concepts (knowledge
units as in \cite{cc2001}) in a language-independent context so
that we can get a sense of how well students in innovative
curricula are actually learning computer science.  We will
validate the inventory using expert review by ABET/CSAB
accreditation auditors. We will also use repeated offerings with a
sample of students to establish reliability.

\item To develop a survey to measure impact of introductory
computing on attitudes and use of technology beyond the scope of
an individual course.  We plan to establish validity with
interviews and use repeated offerings with a sample of students to
establish reliability.

\item We plan to offer workshops at ACM SIGCSE (Special Interest
Group in CS Education) and ASEE/IEEE Frontiers in Education (FIE)
conferences to help faculty learn how to assess curricular
innovations targeting women and non-CS majors.  Our plan is to
discuss issues like measurement of retention and WDF rates, as
well as the measures that we are using.  Through our publisher, we
also plan to directly seek out faculty interested in our
curriculum to invite them to use our measures as well.  We plan to
follow up with email surveys to learn how effective the measures
are at the other institutions.
\end{itemize}

All of our development and assessment will be overseen by the
Georgia Institute of Technology's Human Subjects Review Board. All
the researchers working on the project will be certified by the
Board as capable of doing human subjects research.  All our
instruments will be reviewed by the board to respect the rights
and safety of the subjects.

\subsection{Developing a Survey to Assess Attitudes towards Computer Science}

Many students attitudes toward computer science today, especially
women and non-CS majors, are pretty abysmal as described by
several studies \cite{margolis} \cite{tech-savvy}. An important
result of any introductory course should be an appreciation for
what's interesting and useful about a topic.  Students should
emerge from introductory computing courses with attitudes that are
positive about the usefulness of computer science and about its
relevance.

Our results so-far in the \emph{Media Computation} course have
been confounding on this point.  Our students emerge feeling that
programming to manipulate samples in arrays and pixels in matrices
is relevant and interesting, and worthy of additional
courses---but they are still overwhelmingly disinterested in
``computer science.''

We plan to develop a survey to measure attitudes that asks for
fine-grained distinctions about what interests the students about
their current course and its content and about what they perceive
as computer science.  We are interested both to understand the
seeming contradiction in our own students' attitudes and to
develop a measure that we can use to assess changes in the
attitudes as we change the course.  We want to understand issues
such as:
\begin{itemize}
\item What is it that students find interesting about technology,
the innovative course that they are taking, and its content?

\item What concepts, skills, and practices are they learning that
they find relevant for themselves now and for their future career?

\item What are their perceptions of what computer science is?

\item How relevant is computer science as they perceive it to
their current and future careers?

\item How has their attitude toward computer science changed do to
the innovation that they are engaged in?
\end{itemize}

We plan to ask the same question in multiple ways to establish
internal validity \cite{validation}.  For example, we may ask
students to state their agreement with the statements ``Computer
science will be useful to me in my career'' and ``I do not expect
to find computer science relevant to my future career.''  If we
find radical differences in these kinds of statements, we will
need to re-word and re-work the questions, using our interview
process as input.

Our plan to develop and validate the attitude survey is through an
iterative process of interviews and survey development using
methods described in \cite{assessincontext}
\cite{grading2understanding} \cite{classroomassessment}.  We plan
to use interviews on these topics to develop the survey questions.
We will then use these surveys, and interview a sample of students
who answer the surveys to ask them:
\begin{itemize}
\item Did we ask the right questions?  Are these issues important,
or are they the wrong ones?

\item Are there issues that you wanted to express or that are
relevant that are not addressed by these questions?

\item Did the questions make sense?
\end{itemize}

During the second and third years, we plan to use repeated
offerings with a sample of students, to establish the reliability
of the survey.

\subsection{Developing a CS1 Concepts Inventory}

We have found it difficult to establish what students are learning
in an innovative CS1 course.  We can anticipate that many new
educational developments will differ from traditional CS1 courses
on features such as order of topics, kinds of topics covered,
context for topics, and programming language used---as ours does.
Because of these differences, it's difficult to compare learning
on midterm exams or other measures, because of different ordering
of topics.  Even when we simply focus on end of term measurements,
simple programming activities are difficult to compare when there
are significant language differences (e.g., as in our case, Python
vs. Scheme vs. Matlab).

We propose to construct an instrument to measure concepts that one
would expect to be covered in a CS1 course, as defined by the
\emph{ACM/IEEE Computing Curriculum 2001} \cite{cc2001}, in a
manner as language independent as possible.  For example:
\begin{itemize}
\item We can ask what concepts like \emph{iteration},
\emph{conditionals}, and \emph{assignment} mean, without
presenting their syntax, but perhaps with asking the students to
present the syntax of their language (making the instrument
language-independent, but the evaluation of the instrument
langauge-dependent).

\item We can ask about data structures such as \emph{arrays},
\emph{matrices}, and \emph{trees} without requiring language
explanations.

\item We can ask about algorithmic complexity and similar
theoretical concepts.
\end{itemize}

Our plan for content validation is to use expert review by ABET
accreditation for CS programs auditors.  We plan to start at a
formative level \cite{formative} with our colleagues who are ABET
accreditation auditors (e.g., Dr. Rich LeBlanc\footnote{Dr.
LeBlanc was also part of the writing team for \cite{cc2001}} and
Mr. Mike McCracken). During our second year, we plan to seek out
ABET accreditation auditors outside our institution and ask them
to review the questions, to determine content validity by a panel
of experts \cite{validation}.

We will establish reliability by repeated use in a sample of our
students in the \emph{Media Computation} class.  We will ask for a
re-take of the instrument within a week with a sample size of
approximately $n=50$.

\subsection{Developing a Longitudinal Evaluation of Impact on Technology Attitudes and Use}

An additional area of evaluation that we plan to introduce is
longitudinal tracking of Media Computation students as they
continue their careers. We want to know:
\begin{itemize}
\item Do these students take more computer science courses?

\item Do these students become computer science majors or minors?


\item How does the course impact the students' use and learning of
computation within their own majors?  For example, do Liberal Arts
majors who use Photoshop learn it more easily or use it better (by
their own estimation) than students who do not take the course? Do
Architecture majors learn AutoCAD more easily or use it better?
\end{itemize}

Our plan is to use a combination of email surveys and in-person or
phone interviews.
\begin{itemize}
\item At the end of each year, 100\% of the students who have
taken the Media Computation course will be surveyed by email to
ask about the CS classes they've taken, their majors and minors,
and their use of computational tools.

\item At the end of each year, some small sample of students who
have taken the course (perhaps 3--5 students) will be interviewed
(by phone or in person) to understand how the course has effected
their later careers and interaction with computation.  These data
will also be used to inform further development of the course
materials and to validate our instrument, using a method similar
to what we plan to do for our attitude survey, e.g., asking if the
questions seem appropriate, what additional issues or impacts
would they consider relevant, etc.
\end{itemize}

\subsection{Dissemination Plans and Formative Evaluation with Faculty}

We have a three part plan for disseminating our measurement
instruments and methods.  The first part is fairly traditional: We
plan to write papers for ACM SIGCSE, the international ITiCSE
(Innovation and Technology in CS Education), and ASEE/IEEE FIE
conferences using our instruments to evaluate our own innovations,
as a model for others and to set a benchmark for others to compare
to (using our methods, or improvements upon our methods).  We have
been successful at using this approach in other, related research.
In our research on computer-supported collaborative learning
(CSCL) (sponsored, in-part, by the PI's NSF Career award, grant
\#REC-955048), we developed a set of measures for how well a CSCL
environment was being used by students \cite{infoecol}.  The
measures were used by others in assessing their own environments
(e.g., \cite{csilethreads}), and formed the basis of other
research to develop theory explaining the kinds of findings that
having these methods of comparison enabled \cite{bystander}. Thus,
we believe that using our methods in our own research will set a
model that can be an effective force for dissemination.

The second part is to offer faculty workshops at ACM SIGCSE and
ASEE/IEEE FIE\footnote{Workshops are sparsely attended when
offered at ITiCSE.} on methods for measuring impact of education
innovations targeted at women and non-CS majors.  We plan to
introduce our instruments, as well as others that are well-known
(such as retention and WDF rates) and methods that are developed
by others over the next few years.

The third part is to use the book for the \emph{Media Computation}
course as a source of potential faculty targets.  Working with our
publisher, we plan to get email addresses of faculty who seek an
evaluation copy of the book, and invite them (whether they use our
book or not) to use our methods to assess whatever innovations
they attempt to attract women and non-CS majors in CS courses. If
the faculty are interested in a course such as this, it may be
that they are planning to attempt some educational innovation to
attempt to attract a non-traditional audience, even if they
eventually choose a different path than our course.  We plan to
develop a manual as well as a CD with copies of our instruments to
send to interested faculty. We will continue to make all our
materials (for the course and for assessment) available on our
development
website\footnote{\url{http://coweb.cc.gatech.edu/mediaComp-plan}},
and we plan to create a project website (with a more easily
remembered URL) for wider dissemination.

Our plan is to follow up with faculty who attend our workshop
using email surveys on a regular (e.g., every six month basis) to
learn how their use of the instruments is going, to learn about
their results, and to solicit feedback on how we might improve our
manual or the instruments.


\section{Conclusion and Timeline}

Our goal is to help ourselves and others to assess the value of
the innovations that are being developed to target women and
non-CS majors to improve their odds of success in introductory
computer science.  It's easy to show that more women stay in a
course or program, or that more non-CS majors succeed where they
may have withdrawn or failed in a comparison traditional CS1
course.  It's much harder to show that there is learning
equivalent to a traditional CS1 going on, to show that there are
changes in attitudes, and to show that there is a significant
longer term impact, beyond the end of the course.  We plan to use
our own course and students as a testbed, but we will be active in
engaging others to use our instruments and give us feedback, so
that we can create measures and methods that truly benefit the
community.  We don't see that our measures will \emph{only} be
applicable to settings where the targets for innovation are women
or non-CS majors.  That's just where we'll be starting from. We
expect that our methods will be useful in assessing a broad range
of innovations where the objectives include CS1-level learning,
impacting attitudes toward CS, and generating impact at a
longer-term scale than a single class.

We believe that the potential broader impact of this work is to
create an infrastructure for comparing innovations to attract
under-represented groups (such as women) to computing and to
create a broader base of computing-literate citizens by helping
non-CS majors to be more successful at computing.  By making it
possible to compare innovative methods, on the basis of learning,
attitudes, and longer-term impact, we believe that we create the
opportunity to create better methods and to improve existing
methods.  Thus, creating assessment methods to go beyond retention
and WFD rate will help us to create better innovations to target
under-represented groups.

We are proposing a three year effort.  Our timeline follows:
\begin{itemize}
\item Year One: We will develop initial versions of all three of
our instruments: Attitude survey, CS1 concepts inventory, and
longitudinal impact survey.  We will iterate on all three in the
three semesters of the year.  We will also be engaging in
interviews to validate the two surveys, and we will ask colleague
ABET accreditation auditors to help us with content validation of
the inventory in a formative stage.  We plan to continue
publishing papers on our findings in the course.

\item Year Two: We will continue to iterate on the instruments,
but we will more actively seek to validate them and test them for
reliability.  We will begin proposing workshops at ACM SIGCSE and
ASEE/IEEE FIE conferences during year two.  Also during year two,
the book for the \emph{Media Computation} class will be published,
so we will begin targeting faculty interested in the book to use
our methods.

\item Year Three: We will focus on reaching out to faculty using
our methods (through the workshop or book) to invite their
feedback and to use it in updating the instruments.  As we update
the measures, we plan to re-validate and re-test for reliability
as needed.  Again, we plan to publish our findings, with a
particular emphasis on use of our methods as a way of establishing
a model for others.
\end{itemize}

%End Matter
    \newpage

\bibliographystyle{sigcse}
\bibliography{proposal2}


\newpage

\section{Budget Justification}

% Just asking for funding to support evaluation, iteration, and dissemination

To fund this effort, this proposal includes a budget request for a
three year project.
\begin{itemize}
\item One summer month of the PI's time in all three years, to
build the instruments, to plan and conduct the workshops, and to
work on dissemination.

\item Two graduate student research assistants.  One will focus on
development of the instruments and the manual/CD for faculty, as
well as the formative evaluation of the instruments. The other
will focus on use of the instrument and analysis of the data that
we gather from it, including testing for reliability and validity.
\item Our costs for these personnel include fringe, computing
charges (for support within the College of Computing), and
graduate student tuition,

\item Travel support for attending conferences to disseminate
materials and results and conduct workshops.

\item Materials and supplies, to support the development and
  evaluation effort.
\end{itemize}


\newpage
\section{Facilities}
The College of Computing maintains a variety of computer systems
in support of academic and research activities.  These include
more than 50 Sun, Silicon Graphics, and Intel systems used as file
and compute servers, many of which are quad-processor machines. In
addition, there are more than 1,000 workstation class machines
from Sun, Silicon Graphics, Intel, and Apple especially for
student use.  A number of specialized facilities augment these
general-purpose computing capabilities.  The hardware that will be
purchased for this project will be of similar quality to what the
students use, for testing purposes, but will be set up to
facilitate development.

The Graphics, Visualization, and Usability (GVU) Center houses a
variety of graphics and multimedia equipment, including
high-performance systems from Silicon Graphics, Sun, Intel, and
Apple.  The affiliated Multimedia, Computer Animation, Audio/Video
Production, Usability/Human Computer Interface, Virtual
Reality/Environments, Electronic Learning Communities,
Computational Perception, Software Visualization, Biomedical
Imaging, Collaborative Software, and Future Computing Environments
labs provide shared facilities targeting specific research areas.
These laboratories' equipments will be of use in developing our
multimedia projects.

PI Guzdial is the Director of the Collaborative Software Lab,
affiliated with GVU.  The Collaborative Software Lab has a bank of
ten servers supporting our experimental software for studying
computer-supported collaborative learning.  In addition, we have
three Linux workstations, two NT workstations, and two Apple
workstations used for development.  The focus of the Collaborative
Software Lab is on facilitating multimedia collaboration, so
multimedia facilities available include a high-end Alesis
keyboard, projection facilities, a Canon digital video camera, and
a Nikon digital camera.

All of the College's facilities are linked via local area networks
which provide a choice of communications capabilities from 10 to
1000 Mbps.  The College's network employs a high-performance OC12C
(622 Mbps) ATM and GigabitEthernet (1000 Mbps) backbone, with
connectivity to the campus ATM network provided via OC12C.  The
primary campus Internet connection is provided by a direct 100
Mbps link to the service provider's Atlanta switching center,
augmented by OC3C ATM and OC12C connections, respectively, to the
NSF vBNS (very high performance Backbone Network Service) and
Abilene research networks.  Georgia Tech is also leading southern
regional gigabit network efforts (SoX.net, the Southern
Crossroads) as part of Internet2.

Additional computing facilities are provided to the Georgia Tech
campus by the Institute's Office of Information Technology (OIT),
including five public-access clusters of Sun, Apple, and Dell
workstations, a collection of Sun multi-processors which are
treated as a single computational resource via login load sharing,
and various mainframes.

\end{document}
