|A S S E S S M E N T I N F O R M A T I O N
Passport to Knowledge's Impact on Students
Passport to Knowledge gives explicit objectives for student outcomes to be derived from participation in these modules. In both the Live from Mars and the Live from Antarctica 2 Teacher's Guides, the student outcome objectives are:
While several teachers have reported significant gains in student achievement test scores in science which they attribute to PtK, it would be difficult to factor out PtK’s specific contributions since these modules were always designed to work in concert with the existing science curriculum. Impact is obviously related to which components a teacher uses and the previous sections provide evidence that this varies considerably among participants. Teachers are encouraged to adapt and adopt the PtK components in each module in a wide variety of ways to suit the needs of their students and the logistical realities of their environment.
Certainly the stories depicted in the case studies gathered from the ongoing tracking study provide evidence of both content mastery and student skill acquisition in both science processes and technology applications. These can be augmented with postings by teachers to the various project mail lists that describe their student achievements obtained through use of the modules.
There are three areas where Passport's impact on students can be assessed. First, we have asked teachers to evaluate their students' attitudes and enthusiasm towards science and scientific careers as a result of their utilization of PtK components. These areas continue to be assessed in our surveys and we will continue to do so in Year Three of the evaluation.
Second, we can look at student work products submitted through collaborative activities such as the PET project in LFM this year or by student work offered up for posting in the web site gallery or on school/classroom home pages.
Third, teachers can look at student behavior in their classrooms -- in their work products, in their questions and in their classroom discussions -- for evidence of all five of the points listed above.
While the first method is done in many studies of project-based science and we have done it here, the second and third techniques are rarely done in a consistent way with teachers sharing a common set of standards from which to make judgements. The work on national, state and local standards provides just such a yardstick to help look at student work, be it written, oral, models or drawings, or electronic.
Changes in Students' Attitudes towards Science and Scientific Careers
In this year's evaluations, as in those of the previous year, this first type of student outcome was addressed by asking teachers, as part of the surveys reported on in the previous sections, to consider the impact on students of a specific PtK module or a specific component within a module. Questions focused on eliciting teachers' judgements on how valuable various aspects of a PtK module were on student learning and attitudes (see Appendix 2 for the questions).
Teachers using Live From Mars, for instance, stated that the LFM module
Live From Antarctica 2 produced similar evaluations from teachers:
Some teachers also did end-of-project evaluations with their students. Here is an example from Tim McCollum, an experienced PtK teacher:
Hi Fellow PTK'ers,
Hope your 96/97 PTK involvement has been an enjoyable, rewarding and positive experience for both you and your students. I'm looking very forward to the continuation of LFM in the fall and focusing on the tropical rain forest in the spring. As classroom teachers collaborating with the PTK initiative we certainly aim to model the goal of being lifelong learners.
On Monday evening, May 12th, we held a Passport to Knowledge Open House. During the event, students and parents drove our Lego Dacta rover - M.A.R.I.O ( Manually Activated Rover for Investigation and Observation.....named by the kids! ), shared their own contributions to the LFM and LFAII web sites, and were treated to a wonderful slide presentation on Antarctica by a husband/wife team of retired professors from our local university. The couple had been to the Palmer Station area two years ago and their program related very well to our LFAII experience.
Now that the PTK initiatives for this school year are winding down, I'd like to offer some words of insight ( and humor ). In attempting to gather some narrative feedback from my students ( 190 - 7th & 8th graders ), I asked three questions. Perhaps some of their responses could be of use in planning your next PTK involvement.
1. Both LFM and LFAII involved following the work of real scientists. What new insights and understandings have you gained about their actual work, their tools, and how they communicate their discoveries to others?
* I learned that scientists work hard, not only with their hands but with their minds.
2. What lessons, activities, and/or topics did you find most interesting and enjoyable?
* The study of penguins because I think animals of remote places are interesting
3. What suggestions can you make about student participation in future PTK projects?
* More "hands-on" activities
Hope these few bits of insight offer some food for thought in planning for your future PTK involvement. Happy summer.....on to Mars!
Another example comes from an excerpt from an end of the year message from an
elementary teacher Marilyn Kennedy Wall:
You Were There....
This past week students, parents, and guests of three school communities joined together in first ever "Mission to Mars" Night. This special event not only celebrated our involvement with PTK "Live from Mars" project, but it also highlighted the efforts of what can happen when three county schools are drawn into collaboration through the tools of technology. After being part of the "Live from Mars" virtual conference in DC this past July and seeing the excitement that is generated when "strangers" from 50 states come together focused on a project and maintain that focus through online collaboration, I knew I had a mission. The best way to spread the word was to directly involve "interested" teachers, and invite them to join my students and me on this "out of world" venture.
Tuesday was our culminating Big Night out. It seemed so long ago way back in October that we planned this event and marked it on our calendars. May seemed so far away! This "Mars to Mission Night" was the celebration of the partnership and collaboration of the three schools. Teaching in a large county like ours, students and parents tend to stay in their own local county areas, so this event was unusual in and of itself, the intermingling of the three different districts of the county. There is more mingling of students on the middle and high school levels, but our elementary schools are more parochial in their school events.
The students worked together to set up the centers for the invited parents and guests. Students selected their "favorite" Mars activities to demonstrate, using our PTK teachers' guide, activities suggested by you, and activities I found at NSTA. It was fun watching parents "be students" and students "be teachers" at each activity center. Students instructed their guests on making craters, helping them discover shield volcanoes and lava layering. The students as teachers modeled the Solar System with their solar system snack just as we had done with them. They had their parents conduct investigations into the possibilities of "life" on Mars.
My own class had spent this last five months creating 3-D futuristic International Space Stations complete with descriptions and explanations with their essays on importance if "Space Exploration" (an idea I borrowed from Chris Rowan). Students explained various Mars Internet sites and helped their guests use pieces of space astronomy software. And the finale of our "Mission to Mars" was the students showing off their "Mars Rover Center", with their town "Sojourner"Rover and they instructed their parents on the programming of the rover over their Martian terrain.
What an awesome night! I could not have been more proud of these 4th and 5th grade students as they worked together as hosts of this special celebration, enlightening the audience with their knowledge about this year's "Missions to Mars"...
Anyway, I just wanted to share the kind of energy that is created when traveling along with PTK on the "Mission to Mars" !!!
Thank you all for such a fabulous year!!
Marilyn K. Wall
Posted Student Work
The second area for examining student outcomes were the posted examples of student work. Both modules' web sites contain a gallery of student work. These clearly show mastery of electronic design and publishing tools by students. They also depict the impact of some of the LFM and LFA2 activities on students.
While the opportunity to foster student work sharing will continue to be a part of Passport web sites and an activity encouraged on the list, we launched an effort this year to conduct a more systematic analysis of student work produced by PtK.
Our objective is to use an amalgam of age, process, and content items adapted from state standards documents, which are in turn based on the National Science Education Standards and the Benchmarks for Science Literacy, to form a customized checklist that covers the content areas covered by this year's PtK modules. We then used this checklist to look at the student submissions.
The objective for this work is:
Does student work that is produced from direct participation in a PtK activity show evidence of grade-appropriate science learning as outlined in the science literacy benchmarks, science standards, or state frameworks?
The results of this analysis are discussed below. The next step is to extend this work so that teachers can use the checklist or a modified version that includes their own relevant local or state standards to assess additional work that is not publicly submitted, or is in some other form than electronic. This would include classroom reports, homework, presentations, models and classroom discussions.
This methodology is derived in part from work Project 2061 of the American Association for the Advancement of Science has done around building an evaluation tool for teachers to use in identifying curriculum strengths and weaknesses. The AAAS work derived and tested a process for training teachers to apply a valid and reliable procedure to reviewing large scale, comprehensive curriculum materials in science. The premises underlying this process involved making defensible judgements about how well the materials are likely to contribute to the attainment of specific learning goals while focusing on both the content and the instructional properties.
While the process developed by these authors is rigorous and promising in terms of helping educators screen materials for adoption, it produces judgements of the likelihood of effectiveness and not any empirical analysis of student learning produced by the use of the materials. Furthermore, the process was designed with full, comprehensive curriculum packages in mind.
We are adapting this process, albeit on a less rigorous scale, to address the objectives listed on the previous page. In this second year of the evaluation, we concentrated on a direct examination of student work produced and submitted on-line. We will refine and expand the process to include teachers in Year Three of the evaluation.
Due to the flexible nature of PtK and its actual utilization, we decided to pick two exemplary state science frameworks to use as the basis for our exploration (see the Methodology section in the Introduction for more details and Appendix Four for the actual list of outcomes we used). These embodied the NSES and the Benchmarks, but are more detailed at the content and process level and therefore are easier to use as a framework for student work analysis.
Our process had three staff reviewers independently look at the collected work samples or on-line web sites submitted. Each assigned the work as many codes from our list as she or he thought relevant. We did this for 30 separate pieces of work and compared notes. Our inter-rater reliability was above 90%. The remaining 104 items (N=134) were coded individually.
The activity that proved to be the most fruitful in providing evidence that was suited to this type of analysis was the Planet Explorer Toolkit collaborative activity. This activity generated a significant amount of student produced e-mail. The several levels of sequential work were oriented to student problem solving, analysis, and peer review so provided a great deal of evidence for the types of content and process outcomes that are stressed on the list of outcomes in Appendix Four. These type of multi-stage on-line collaboration project has become a staple of PtK modules so it was a good place to look for clear evidence of student learning.
The PET activity has student messages occurring over a four month span that show detailed evidence of all four broad categories. Classes that participated in the complete cycle all had messages that showed multiple outcomes in the Design, Process, Interpretation and Sharing specific categories. The specific items that occurred the most often in the written student messages were:
Table Nineteen Outcomes Demonstrated in Student Work N= 134 items
Percent of items showing indicators:
32% D-1 Develops questions on scientific topics
48% D-2 Chooses the steps necessary to answer a question.
71% D-5 Proposes a design to solve a problem based on given criteria.
76% P-1 Demonstrates accurate recording and reporting of observations.
76% P-3 Collects data for investigation using measuring instruments.
56% P-4 Collects data using consistent measuring and recording techniques.
36% I-1 Describes an observed event
54% I-2 Records and arranges data into logical patterns and describes the patterns.
28% I-3 Compares individual and group observations and results.
76% I-4 Participates in and understands the importance of peer reviews in improving the scientific process.
64% S-1 Describes individual and group investigations clearly and accurately in oral or written reports.
36% S-2 Constructs charts and graphs to display data and uses these to produce reasonable explanations.
54% S-3 Reports the process and results of a scientific investigation in oral and written presentations.
62% S-4 Makes, presents and defends conclusions drawn from investigation to a classroom audience in written or oral form.
34% ES-3 Analyze and explain naturally occurring earth and space events.
24% ES-4 Describe and explain interactions of earth components and solar system components.
48% ES-5 Compare and explain short and long-term planetary and celestial variations (e.g., latitudinal effects on weather and climate, relative positions of the planets and stars).
Sample student items:
The following items were research papers typical of the type described in the case studies and suggested in several of the Teacher guides' activities. These are not direct experiments or investigations. However, they show clear evidence of communicating and technology usage as defined in the broad statements.
B-2 (student uses precise and complete descriptions and the presentation is supported by evidence. Descriptions show careful observations, organization of data, and translation of findings into clear language)
B-3 (student uses appropriate tools, equipment to access information and share ideas or communicate results).
These papers were part of a class submission from a fifth grade.
The following print outs of student postings to the PET activity list are examples of
We have also included several additional messages from some of the classrooms described in the case study section to give the reader a sense of how the PET project evolved.
These student samples and the entire pool of 134 items of student work we analyzed represent a small fraction of the student work produced in PtK classrooms this year. This technique, the analysis of student work submissions according to science standards, is a viable means to explore the impact of PtK on student learning. In year three, we will take steps to increase the amount of work available electronically to be analyzed and will provide teachers with the tools to use it in their own assessment of student learning.
Teacher testimonials have been both ample and highly supportive of PtK's worth and will continue to be valuable sources for judging its effects. The structure of this outcome list (or a locally adapted version to incorporate local frameworks) provides the rigor to go beyond the anecdote. This will provide valid evidence of what PtK provides in terms of concrete instructional opportunities in a way that teachers can use to justify its use in their classrooms. Furthermore, building a common language among teachers and a shared sense of what constitutes appropriate expectations when encouraging and assessing student work within the context of PtK activities will help ensure that PtK's potential as a science learning tool is maximized during its implementation.