Passport to Knowledge's Impact on Students

Passport to Knowledge gives explicit objectives for student outcomes to be derived from participation in these modules. In both the Live from Mars and the Live from Antarctica 2 Teacher's Guides, the student outcome objectives are:

While several teachers have reported significant gains in student achievement test scores in science which they attribute to PtK, it would be difficult to factor out PtK’s specific contributions since these modules were always designed to work in concert with the existing science curriculum. Impact is obviously related to which components a teacher uses and the previous sections provide evidence that this varies considerably among participants. Teachers are encouraged to adapt and adopt the PtK components in each module in a wide variety of ways to suit the needs of their students and the logistical realities of their environment.

Certainly the stories depicted in the case studies gathered from the ongoing tracking study provide evidence of both content mastery and student skill acquisition in both science processes and technology applications. These can be augmented with postings by teachers to the various project mail lists that describe their student achievements obtained through use of the modules.

There are three areas where Passport's impact on students can be assessed. First, we have asked teachers to evaluate their students' attitudes and enthusiasm towards science and scientific careers as a result of their utilization of PtK components. These areas continue to be assessed in our surveys and we will continue to do so in Year Three of the evaluation.

Second, we can look at student work products submitted through collaborative activities such as the PET project in LFM this year or by student work offered up for posting in the web site gallery or on school/classroom home pages.

Third, teachers can look at student behavior in their classrooms -- in their work products, in their questions and in their classroom discussions -- for evidence of all five of the points listed above.

While the first method is done in many studies of project-based science and we have done it here, the second and third techniques are rarely done in a consistent way with teachers sharing a common set of standards from which to make judgements. The work on national, state and local standards provides just such a yardstick to help look at student work, be it written, oral, models or drawings, or electronic.

Changes in Students' Attitudes towards Science and Scientific Careers

In this year's evaluations, as in those of the previous year, this first type of student outcome was addressed by asking teachers, as part of the surveys reported on in the previous sections, to consider the impact on students of a specific PtK module or a specific component within a module. Questions focused on eliciting teachers' judgements on how valuable various aspects of a PtK module were on student learning and attitudes (see Appendix 2 for the questions).

Teachers using Live From Mars, for instance, stated that the LFM module

  1. Significantly increased their students excitement about studying Mars (87.5% of the respondents).
  2. Enhanced their curiosity in careers in scientific research (79.4%)
  3. Directly improved the motivation of students to learn about space-related topics (90.9%)

Live From Antarctica 2 produced similar evaluations from teachers:

  1. 93% of the responding teachers felt that LFA2 significantly increased students' excitement about Antarctic research.
  2. 84.1% stated that LFA2 increased their students' curiosity about careers in scientific research.
  3. 93.3% stated that LFA2 increased students' motivation to learn more about Antarctica.

Some teachers also did end-of-project evaluations with their students. Here is an example from Tim McCollum, an experienced PtK teacher:

From: (Tim McCollum)
Subject: PTK Open House & 96/97 Wrap-up

Hi Fellow PTK'ers,
        Two more weeks and it's summer vacation......yea!  

     Hope your 96/97 PTK involvement has been an enjoyable, rewarding and positive experience 
for both you and your students.  I'm looking very forward to the continuation of LFM in the fall 
and focusing on the tropical rain forest in the spring. As classroom teachers collaborating with 
the PTK initiative we certainly aim to model the goal of being lifelong learners.

     On Monday evening, May 12th, we held a Passport to Knowledge Open House.  During the event, 
students and parents drove our Lego Dacta rover - M.A.R.I.O ( Manually Activated Rover for 
Investigation and Observation.....named by the kids! ), shared their own contributions to the LFM 
and LFAII web sites, and were treated to a wonderful slide presentation on Antarctica by a 
husband/wife team of retired professors from our local university.  The couple had been to the 
Palmer Station area two years ago and their program related very well to our LFAII experience. 

     Now that the PTK initiatives for this school year are winding down, I'd like to offer some 
words of insight ( and humor ). In attempting to gather some narrative feedback from my students 
( 190 - 7th & 8th graders ), I asked three questions. Perhaps some of their responses could be 
of use in planning your next PTK involvement.

1. Both LFM and LFAII involved following the work of real scientists.  What new insights and 
understandings have you gained about their actual work, their tools, and how they communicate 
their discoveries to others?

* I learned that scientists work hard, not only with their hands but with their minds.
* Scientists don't just sit in the lab all day.
* When they plan to put a spacecraft on another planet, they must think of everything!
* These projects made me realize that scientists are more than the Far Side "white lab coats 
  and beakers" stereotype.
* They have to learn to work together as a team.
* I learned that sometimes it is hard to get information to other people.
* It has opened a whole new door of science that I might like to pursue.
* I learned how they use their tools and knowledge to overcome their problems.
* I have a greater respect for those who give up portions of their lives to live in desolate 
  places to do research.
* I thought it was neat how they shared their work with us.
* They have to organize a lot of data so it can be accurately used for reference.
* I learned that they don't always get the recognition they deserve.
* Oil isn't as easy to clean up as I thought.
* A scientists's work is never done.

2. What lessons, activities, and/or topics did you find most interesting and enjoyable?

* The study of penguins because I think animals of remote places are interesting
* Listening to our PTK Open House speakers
* The live broadcasts because they were really happening as we were watching
* We got to ask questions about what we wanted to know and got answers
* By participating we got to find out what was really happening instead of being left out
* Driving the rover, shrimp & krill labs, the oil spill lab, the PET project, the CFC lab
* The chat sessions because we got to ask questions to REAL scientists ( and they got me out of 
  math )
* All the interaction helped give a better idea of what we were learning
* Journal writing for the web site
* By getting to experience some of the same things as real-life scientists so you can see if you 
  want to go into that kind of profession
* The blubber glove, it was fun, cold and wet!
* The web site because you could go at a pace you liked and you learned more that way
* Red Rover, being able to drive something hundreds of miles away
* The challenge questions, they made us think
* The MOLA project and converting our paper models into 3-D on the computers

3. What suggestions can you make about student participation in future PTK projects?

* More "hands-on" activities
* More chat sessions
* More MOLA type projects to show how things work
* Stopping the video tape at times for discussion
* More time to browse the web sites

     Hope these few bits of insight offer some food for thought in planning for your future PTK 
involvement. Happy summer.....on to Mars!

Tim McCollum                                     217-345-2193 (school)
Charleston Jr. High School                       217-345-8121 (fax)
920 Smith Dr.                          
Charleston, IL  61920

Another example comes from an excerpt from an end of the year message from an elementary teacher Marilyn Kennedy Wall:

From: (Marilyn Kennedy)
Subject: You were there...

You Were There...
                at our "Mission to Mars" Night

This past week students, parents, and guests of three school communities joined together in first 
ever  "Mission to Mars" Night. This special event not only celebrated  our involvement with PTK 
"Live from Mars" project, but it also highlighted the efforts of what can happen when three county 
schools are drawn into collaboration through the tools of technology. After being part of the  
"Live from Mars" virtual conference in DC this past July and seeing the excitement that is 
generated when "strangers" from 50 states come together focused on a project and maintain that 
focus through online collaboration, I knew I had a mission.  The best way to spread the word was 
to directly involve "interested" teachers, and invite them to join my students and me on this "out 
of world" venture...

Tuesday was our culminating Big Night out. It seemed so long ago way back in October that we 
planned this event and marked it on our calendars. May seemed so far away! This "Mars to Mission 
Night" was the  celebration of the partnership and collaboration of the three schools. Teaching in 
a large county like ours, students and parents tend to stay in their own local county areas, so 
this event was  unusual in and of itself, the intermingling of the three different districts of 
the county. There is more mingling of students on the middle and high school levels, but our 
elementary schools are more parochial in their school events.

The students worked together to set up the centers for the invited  parents and guests. Students 
selected their "favorite" Mars activities to demonstrate,  using  our PTK teachers' guide, 
activities suggested by you, and activities I found at NSTA. It was fun watching parents "be 
students" and students "be teachers" at each activity center. Students instructed their guests on 
making craters, helping them discover shield volcanoes and lava layering. The students as teachers 
modeled the Solar System with their solar system snack just as we had done with them. They had 
their parents conduct investigations into the possibilities of "life" on Mars. My own class had 
spent this last five months creating 3-D futuristic International Space Stations complete with 
descriptions and explanations with their essays on importance if "Space Exploration" (an idea I 
borrowed from Chris Rowan). Students explained various Mars Internet sites and helped their guests 
use pieces of space astronomy software. And the finale of our "Mission to Mars" was the students 
showing off their "Mars Rover Center", with their town "Sojourner"Rover and they instructed their 
parents on the programming of the rover over their Martian terrain.

What an awesome night! I could not have been more proud of these 4th and 5th grade students as 
they worked together as hosts of this special celebration, enlightening the audience with their 
knowledge about this year's "Missions to Mars"... 

Anyway, I just wanted to share the kind of energy that is created when traveling along with  PTK 
on the   "Mission to Mars" !!!

Thank you all for such a fabulous year!!

Marilyn K. Wall
John Wayland Elementary
here in the Shenandoah Valley in
Bridgewater, VA

Posted Student Work

The second area for examining student outcomes were the posted examples of student work. Both modules' web sites contain a gallery of student work. These clearly show mastery of electronic design and publishing tools by students. They also depict the impact of some of the LFM and LFA2 activities on students.

While the opportunity to foster student work sharing will continue to be a part of Passport web sites and an activity encouraged on the list, we launched an effort this year to conduct a more systematic analysis of student work produced by PtK.

Our objective is to use an amalgam of age, process, and content items adapted from state standards documents, which are in turn based on the National Science Education Standards and the Benchmarks for Science Literacy, to form a customized checklist that covers the content areas covered by this year's PtK modules. We then used this checklist to look at the student submissions.

The objective for this work is:

Does student work that is produced from direct participation in a PtK activity show evidence of grade-appropriate science learning as outlined in the science literacy benchmarks, science standards, or state frameworks?

  1. Can independent evaluators apply a checklist of these outcomes with high inter-rater reliability?
  2. Can the evidence be directly attributable to PtK?

The results of this analysis are discussed below. The next step is to extend this work so that teachers can use the checklist or a modified version that includes their own relevant local or state standards to assess additional work that is not publicly submitted, or is in some other form than electronic. This would include classroom reports, homework, presentations, models and classroom discussions.

This methodology is derived in part from work Project 2061 of the American Association for the Advancement of Science has done around building an evaluation tool for teachers to use in identifying curriculum strengths and weaknesses. The AAAS work derived and tested a process for training teachers to apply a valid and reliable procedure to reviewing large scale, comprehensive curriculum materials in science. The premises underlying this process involved making defensible judgements about how well the materials are likely to contribute to the attainment of specific learning goals while focusing on both the content and the instructional properties.

While the process developed by these authors is rigorous and promising in terms of helping educators screen materials for adoption, it produces judgements of the likelihood of effectiveness and not any empirical analysis of student learning produced by the use of the materials. Furthermore, the process was designed with full, comprehensive curriculum packages in mind.

We are adapting this process, albeit on a less rigorous scale, to address the objectives listed on the previous page. In this second year of the evaluation, we concentrated on a direct examination of student work produced and submitted on-line. We will refine and expand the process to include teachers in Year Three of the evaluation.

Due to the flexible nature of PtK and its actual utilization, we decided to pick two exemplary state science frameworks to use as the basis for our exploration (see the Methodology section in the Introduction for more details and Appendix Four for the actual list of outcomes we used). These embodied the NSES and the Benchmarks, but are more detailed at the content and process level and therefore are easier to use as a framework for student work analysis.

Our process had three staff reviewers independently look at the collected work samples or on-line web sites submitted. Each assigned the work as many codes from our list as she or he thought relevant. We did this for 30 separate pieces of work and compared notes. Our inter-rater reliability was above 90%. The remaining 104 items (N=134) were coded individually.

The activity that proved to be the most fruitful in providing evidence that was suited to this type of analysis was the Planet Explorer Toolkit collaborative activity. This activity generated a significant amount of student produced e-mail. The several levels of sequential work were oriented to student problem solving, analysis, and peer review so provided a great deal of evidence for the types of content and process outcomes that are stressed on the list of outcomes in Appendix Four. These type of multi-stage on-line collaboration project has become a staple of PtK modules so it was a good place to look for clear evidence of student learning.

The PET activity has student messages occurring over a four month span that show detailed evidence of all four broad categories. Classes that participated in the complete cycle all had messages that showed multiple outcomes in the Design, Process, Interpretation and Sharing specific categories. The specific items that occurred the most often in the written student messages were:

Table Nineteen Outcomes Demonstrated in Student Work N= 134 items

Percent of items showing indicators:

   32% D-1 Develops questions on scientific topics

   48% D-2 Chooses the steps necessary to answer a question.

   71% D-5 Proposes a design to solve a problem based on given criteria.

   76% P-1 Demonstrates accurate recording and reporting of observations.

   76% P-3 Collects data for investigation using measuring instruments.

   56% P-4 Collects data using consistent measuring and recording techniques.

   36% I-1 Describes an observed event

   54% I-2 Records and arranges data into logical patterns and describes the patterns.

   28% I-3 Compares individual and group observations and results.

   76% I-4 Participates in and understands the importance of peer reviews in improving the scientific process.

   64% S-1 Describes individual and group investigations clearly and accurately in oral or written reports.

   36% S-2 Constructs charts and graphs to display data and uses these to produce reasonable explanations.

   54% S-3 Reports the process and results of a scientific investigation in oral and written presentations.

   62% S-4 Makes, presents and defends conclusions drawn from investigation to a classroom audience in written or oral form.

   34% ES-3 Analyze and explain naturally occurring earth and space events.

   24% ES-4 Describe and explain interactions of earth components and solar system components.

   48% ES-5 Compare and explain short and long-term planetary and celestial variations (e.g., latitudinal effects on weather and climate, relative positions of the planets and stars).

Sample student items:

The following items were research papers typical of the type described in the case studies and suggested in several of the Teacher guides' activities. These are not direct experiments or investigations. However, they show clear evidence of communicating and technology usage as defined in the broad statements B-2 (student uses precise and complete descriptions and the presentation is supported by evidence. Descriptions show careful observations, organization of data, and translation of findings into clear language) and B-3 (student uses appropriate tools, equipment to access information and share ideas or communicate results). These papers were part of a class submission from a fifth grade.

The following print outs of student postings to the PET activity list are examples of

    • I-2 Records and arranges data into logical patterns and describes the patterns
    • D-1Develops questions on scientific topics
    • D-2 Chooses the steps necessary to answer a question.
    • P-1 Demonstrates accurate recording and reporting of observations.
    • I-4 Participates in and understands the importance of peer reviews in improving the scientific process.

We have also included several additional messages from some of the classrooms described in the case study section to give the reader a sense of how the PET project evolved.

These student samples and the entire pool of 134 items of student work we analyzed represent a small fraction of the student work produced in PtK classrooms this year. This technique, the analysis of student work submissions according to science standards, is a viable means to explore the impact of PtK on student learning. In year three, we will take steps to increase the amount of work available electronically to be analyzed and will provide teachers with the tools to use it in their own assessment of student learning.

Teacher testimonials have been both ample and highly supportive of PtK's worth and will continue to be valuable sources for judging its effects. The structure of this outcome list (or a locally adapted version to incorporate local frameworks) provides the rigor to go beyond the anecdote. This will provide valid evidence of what PtK provides in terms of concrete instructional opportunities in a way that teachers can use to justify its use in their classrooms. Furthermore, building a common language among teachers and a shared sense of what constitutes appropriate expectations when encouraging and assessing student work within the context of PtK activities will help ensure that PtK's potential as a science learning tool is maximized during its implementation.

Foreward Year 2 Report Introducation Who Is Using P2K? How Is P2K Being Used Student Outcomes