Passport to Knowledge gives explicit objectives for student outcomes to be derived from participation in these modules. In both the Live from Mars and the Live from Antarctica 2 Teacher's Guides, the student outcome objectives are:

• obtaining content information more current than that found in textbooks;
• acquiring general knowledge about the exploration of space/Antarctica;
• developing positive scientific attitudes towards science and technology;
• attaining a better understanding of the scientific method and research process;
• practice in applying new technology tools and research skills.

While several teachers have reported significant gains in student achievement test scores in science which they attribute to PtK, it would be difficult to factor out PtK’s specific contributions since these modules were always designed to work in conce rt with the existing science curriculum. Impact is obviously related to which components a teacher uses and the previous sections provide evidence that this varies considerably among participants. Teachers are encouraged to adapt and adopt the PtK compo nents in each module in a wide variety of ways to suit the needs of their students and the logistical realities of their environment.

Certainly the stories depicted in the case studies gathered from the ongoing tracking study provide evidence of both content mastery and student skill acquisition in both science processes and technology applications. These can be augmented with post ings by teachers to the various project mail lists that describe their student achievements obtained through use of the modules.

There are three areas where Passport's impact on students can be assessed. First, we have asked teachers to evaluate their students' attitudes and enthusiasm towards science and scientific careers as a result of their utilization of PtK components. T hese areas continue to be assessed in our surveys and we will continue to do so in Year Three of the evaluation.

Second, we can look at student work products submitted through collaborative activities such as the PET project in LFM this year or by student work offered up for posting in the web site gallery or on school/classroom home pages.

Third, teachers can look at student behavior in their classrooms -- in their work products, in their questions and in their classroom discussions -- for evidence of all five of the points listed above.

While the first method is done in many studies of project-based science and we have done it here, the second and third techniques are rarely done in a consistent way with teachers sharing a common set of standards from which to make judgements. The wo rk on national, state and local standards provides just such a yardstick to help look at student work, be it written, oral, models or drawings, or electronic.

In this year's evaluations, as in those of the previous year, this first type of student outcome was addressed by asking teachers, as part of the surveys reported on in the previous sections, to consider the impact on students of a specific PtK module or a specific component within a module. Questions focused on eliciting teachers' judgements on how valuable various aspects of a PtK module were on student learning and attitudes (see Appendix 2 for the questions).

Teachers using Live From Mars, for instance, stated that the LFM module

1. Significantly increased their students excitement about studying Mars (87.5% of the respondents).
2. Enhanced their curiosity in careers in scientific research (79.4%)
3. Directly improved the motivation of students to learn about space-related topics (90.9%)

Live From Antarctica 2 produced similar evaluations from teachers:

1. 93% of the responding teachers felt that LFA2 significantly increased students' excitement about Antarctic research.
2. 84.1% stated that LFA2 increased their students' curiosity about careers in scientific research.
3. 93.3% stated that LFA2 increased students' motivation to learn more about Antarctica.

Some teachers also did end-of-project evaluations with their students. Here is an example from Tim McCollum, an experienced PtK teacher:

To: discuss-lfm@quest.arc.nasa.gov
From: cxtdm@eiu.edu (Tim McCollum)
Subject: PTK Open House & 96/97 Wrap-up
jwee@mail.arc.nasa.gov

Hi Fellow PTK'ers,
        Two more weeks and it's summer vacation......yea!  

        Hope your 96/97 PTK involvement has been an enjoyable, rewarding
and positive experience for both you and your students.  I'm looking very
forward to the continuation of LFM in the fall and focusing on the tropical rain forest in the spring. As classroom teachers collaborating with the PTK initiative we certainly aim to model the goal of being lifelong learners.

        On Monday evening, May 12th, we held a Passport to Knowledge Open
House.  During the event, students and parents drove our Lego Dacta rover - M.A.R.I.O ( Manually Activated Rover for Investigation and Observation.....named by the kids! ), shared their own contributions to the LFM and LFAII web sites, and were treated to
 a wonderful slide presentation on Antarctica by a husband/wife team of retired professors from our local university.  The couple had been to the Palmer Station area two years ago and their program related very well to our LFAII experience. 
      Now that the PTK initiatives for this school year are winding down, I'd like to offer some words of insight ( and humor ). In attempting to gather some narrative feedback from my students ( 190 - 7th & 8th graders ), I asked three questions. Per
haps some of their responses could be of use in planning your next PTK involvement.

1. Both LFM and LFAII involved following the work of real scientists.  What new insights and understandings have you gained about their actual work, their tools, and how they communicate their discoveries to others?

* I learned that scientists work hard, not only with their hands but with
their minds.
* Scientists don't just sit in the lab all day.
* When they plan to put a spacecraft on another planet, they must think of
everything!
* These projects made me realize that scientists are more than the Far Side
"white lab coats            and beakers" stereotype.
* They have to learn to work together as a team.
* I learned that sometimes it is hard to get information to other people.
* It has opened a whole new door of science that I might like to pursue.
* I learned how they use their tools and knowledge to overcome their problems.
* I have a greater respect for those who give up portions of their lives to
live in desolate places to do research.
* I thought it was neat how they shared their work with us.
* They have to organize a lot of data so it can be accurately used for
reference.
* I learned that they don't always get the recognition they deserve.
* Oil isn't as easy to clean up as I thought.
* A scientists's work is never done.


2. What lessons, activities, and/or topics did you find most interesting and enjoyable?

* The study of penguins because I think animals of remote places are interesting
* Listening to our PTK Open House speakers
* The live broadcasts because they were really happening as we were watching
* We got to ask questions about what we wanted to know and got answers
* By participating we got to find out what was really happening instead of
being left out
* Driving the rover, shrimp & krill labs, the oil spill lab, the PET
project, the CFC lab
* The chat sessions because we got to ask questions to REAL scientists (
and they got me out of math )
* All the interaction helped give a better idea of what we were learning
* Journal writing for the web site
* By getting to experience some of the same things as real-life scientists
so you can see if you want to go into that kind of profession
* The blubber glove, it was fun, cold and wet!
* The web site because you could go at a pace you liked and you learned
more that way
* Red Rover, being able to drive something hundreds of miles away
* The challenge questions, they made us think
* The MOLA project and converting our paper models into 3-D on the computers


3. What suggestions can you make about student participation in future PTK projects?

* More "hands-on" activities
* More chat sessions
* More MOLA type projects to show how things work
* Stopping the video tape at times for discussion
* More time to browse the web sites

        Hope these few bits of insight offer some food for thought in
planning for your future PTK involvement. Happy summer.....on to Mars!

Tim
......................................................................... 
Tim McCollum                                     217-345-2193 (school)
Charleston Jr. High School                       217-345-8121 (fax)
920 Smith Dr.                                    cxtdm@eiu.edu
Charleston, IL  61920
http://www.ux1.eiu.edu/~cxtdm/macsci.html        
http://www.ncsa.uiuc.edu/edu/nie/summer/tmccollum/
                                                                           

Another example comes from an excerpt from an end of the year message from an

elementary teacher Marilyn Kennedy Wall:

To: discuss-lfm@quest.arc.nasa.gov
From: mkennedy@head.globalcom.net (Marilyn Kennedy)
Subject: You were there...

You Were There....
                at our "Mission to Mars" Night

This past week students, parents, and guests of three school communities
joined together in first ever  "Mission to Mars" Night. This special event not only celebrated  our involvement with PTK "Live from Mars" project, but it also highlighted the efforts of what can happen when three county schools are drawn into collaboratio
n through the tools of technology. After being part of the  "Live from Mars" virtual conference in DC this past July and seeing the excitement that is generated when "strangers" from 50 states come together focused on a project and maintain that focus thr
ough online collaboration, I knew I had a mission.  The best way to spread the word was  to directly involve "interested" teachers, and invite them to join my students and me on this "out of world" venture.

….

Tuesday was our culminating Big Night out. It seemed so long ago way back
in October that we planned this event and marked it on our calendars. May
seemed so far away! This "Mars to Mission Night" was the  celebration of
the partnership and collaboration of the three schools. Teaching in a large county like ours, students and parents tend to stay in their own local county areas, so this event was  unusual in and of itself, the intermingling of the three different district
s of the county. There is more mingling of students on the middle and high school levels, but our elementary schools are more parochial in their school events.

The students worked together to set up the centers for the invited  parents and guests. Students selected their "favorite" Mars activities to
demonstrate,  using  our PTK teachers' guide, activities suggested by you, and activities I found at NSTA. It was fun watching parents "be students" and students "be teachers" at each activity center. Students instructed their guests on making craters, he
lping them discover shield volcanoes and lava layering. The students as teachers modeled the Solar System with their solar system snack just as we had done with them. They had their parents conduct investigations into the possibilities of "life" on Mars. 
 My own class had spent this last five months creating 3-D futuristic International Space Stations complete with descriptions and explanations with their essays on importance if "Space Exploration" (an idea I borrowed from Chris Rowan). Students explained
 various Mars Internet sites and helped their guests use pieces of space astronomy software. And the finale of our "Mission to Mars" was the students showing off their "Mars Rover Center", with their town "Sojourner"Rover and they instructed their parents
 on the programming of the rover over their Martian terrain.

What an awesome night! I could not have been more proud of these 4th and
5th grade students as they worked together as hosts of this special
celebration, enlightening the audience with their knowledge about this
year's "Missions to Mars"…. 

Anyway, I just wanted to share the kind of energy that is created when
traveling along with  PTK on the   "Mission to Mars" !!!

Thank you all for such a fabulous year!!

Marilyn K. Wall
John Wayland Elementary
here in the Shenandoah Valley in
Bridgewater, VA

The second area for examining student outcomes were the posted examples of student work. Both modules' web sites contain a gallery of student work. These clearly show mastery of electronic design and publishing tools by students. They also depict th e impact of some of the LFM and LFA2 activities on students.

While the opportunity to foster student work sharing will continue to be a part of Passport web sites and an activity encouraged on the list, we launched an effort this year to conduct a more systematic analysis of student work produced by PtK.

Our objective is to use an amalgam of age, process, and content items adapted from state standards documents, which are in turn based on the National Science Education Standards and the Benchmarks for Science Literacy, to form a customized checklist th at covers the content areas covered by this year's PtK modules. We then used this checklist to look at the student submissions.

The objective for this work is:

1. Can independent evaluators apply a checklist of these outcomes with high inter-rater reliability?
2. Can the evidence be directly attributable to PtK?

The results of this analysis are discussed below. The next step is to extend this work so that teachers can use the checklist or a modified version that includes their own relevant local or state standards to assess additional work that is not publicl y submitted, or is in some other form than electronic. This would include classroom reports, homework, presentations, models and classroom discussions.

This methodology is derived in part from work Project 2061 of the American Association for the Advancement of Science has done around building an evaluation tool for teachers to use in identifying curriculum strengths and weaknesses. The AAAS w ork derived and tested a process for training teachers to apply a valid and reliable procedure to reviewing large scale, comprehensive curriculum materials in science. The premises underlying this process involved making defensible judgements about how w ell the materials are likely to contribute to the attainment of specific learning goals while focusing on both the content and the instructional properties.

While the process developed by these authors is rigorous and promising in terms of helping educators screen materials for adoption, it produces judgements of the likelihood of effectiveness and not any empirical analysis of student learning produced by the use of the materials. Furthermore, the process was designed with full, comprehensive curriculum packages in mind.

We are adapting this process, albeit on a less rigorous scale, to address the objectives listed on the previous page. In this second year of the evaluation, we concentrated on a direct examination of student work produced and submitted on-line. We w ill refine and expand the process to include teachers in Year Three of the evaluation.

Due to the flexible nature of PtK and its actual utilization, we decided to pick two exemplary state science frameworks to use as the basis for our exploration (see the Methodology section in the Introduction for more details and Appendix Four for the actual list of outcomes we used). These embodied the NSES and the Benchmarks, but are more detailed at the content and process level and therefore are easier to use as a framework for student work analysis.

Our process had three staff reviewers independently look at the collected work samples or on-line web sites submitted. Each assigned the work as many codes from our list as she or he thought relevant. We did this for 30 separate pieces of work and co mpared notes. Our inter-rater reliability was above 90%. The remaining 104 items (N=134) were coded individually.

The activity that proved to be the most fruitful in providing evidence that was suited to this type of analysis was the Planet Explorer Toolkit collaborative activity. This acti vity generated a significant amount of student produced e-mail. The several levels of sequential work were oriented to student problem solving, analysis, and peer review so provided a great deal of evidence for the types of content and process outcomes t hat are stressed on the list of outcomes in Appendix Four. These type of multi-stage on-line collaboration project has become a staple of PtK modules so it was a good place to look for clear evidence of student learning.

The PET activity has student messages occurring over a four month span that show detailed evidence of all four broad categories. Classes that participated in the complete cycle all had messages that showed multiple outcomes in the Design, Process, Int erpretation and Sharing specific categories. The specific items that occurred the most often in the written student messages were:

Sample student items:

The following items were research papers typical of the type described in the case studies and suggested in several of the Teacher guides' activities. These are not direct experiments or investigations. However, they show clear evidence of communicat ing and technology usage as defined in the broad statements B-2 (student uses precise and complete descriptions and the presentation is supported by evidence. Descriptions show careful observations, organization of data, and translation of findings into clear language) ands B-3 (student uses appropriate tools, equipment to access information and share ideas or communicate results). These papers were part of a class submission from a fifth grade.

The following print outs of student postings to the PET activity list are examples of

2)

We have also included several additional messages from some of the classrooms described in the case study section to give the reader a sense of how the PET project evolved.

These student samples and the entire pool of 134 items of student work we analyzed represent a small fraction of the student work produced in PtK classrooms this year. This technique, the analysis of student work submissions according to science s tandards, is a viable means to explore the impact of PtK on student learning. In year three, we will take steps to increase the amount of work available electronically to be analyzed and will provide teachers with the tools to use it in their own assessm ent of student learning.

Teacher testimonials have been both ample and highly supportive of PtK's worth and will continue to be valuable sources for judging its effects. The structure of this outcome list (or a locally adapted version to incorporate local frameworks) provides the rigor to go beyond the anecdote. This will provide valid evidence of what PtK provides in terms of concrete instructional opportunities in a way that teachers can use to justify its use in their classrooms. Furthermore, building a common language a mong teachers and a shared sense of what constitutes appropriate expectations when encouraging and assessing student work within the context of PtK activities will help ensure that PtK's potential as a science learning tool is maximized during its impleme ntation.