Evaluation of NSF IMD Grant

This section of the Report begins by summarizing the most significant findings from the entire 3 years of the project, and then presents some of EDC's quantitative and qualitative findings, organized by project Goals as listed in the previous section.

Key Findings
Secondary school teachers benefited from interactions with working researchers
Teachers appreciated learning science concepts and facts directly from scientists and researchers. For most K-12 educators, regardless of the number of years they had been teaching, this was a unique experience. It made them feel more professional and responsible. They expressed, in the surveys and in their spontaneous messages to the on-line mail lists, a sense-often for the first time-of being active participants in the scientific community.

P2K/LIVE FROM was a model for, and showed the value of, inquiry in science education
The majority of teachers indicated that working with P2K helped them gain experience with using project-based science in their classrooms. They felt P2K was successful, that students gained more from it than from their usual instructional methods, and that they now felt comfortable moving beyond the textbook. As can be seen in this comment from middle school science teacher, Mike Reynolds, from Michigan, the P2K materials encouraged exactly the kind of inquiry-based learning targeted by NSF:

 

I only taught space science once before-using a text, very boring. But this (P2K) material is very exciting. Before kids were able to predict the results without even doing the experiments. This time it was really different, they were doing real science. They were coming up with hypotheses, and testing them out. The materials promoted these types of activities. With the typical curriculum I had been using, kids were never asked to create the experiments so this was a great experience. I had some kids who were C students in the regular curriculum. Now they are A students in LIVE FROM MARS, they were very involved and put in extra time for the Mars activities. This reaches kids that otherwise we would miss.

 

New information and communications technologies, ICTs, can have a measurable, positive impact on classroom instruction
For most of the participating teachers, P2K was a first experience with using on-line materials directly with students (NSF's support began in 1995.) For many, P2K became the model in their school and/or district that was used to justify the nation's growing investment in the technology. Teachers often reported moving from being a total novice in using telecommunications technology to being the exemplar held up by their administration because P2K helped them use the technology effectively.

The Internet made possible a genuine and useful "virtual community" of science teachers
Over 80% of surveyed teachers credit P2K for building their interest in, and providing opportunities for, collaborating with other teachers, something whose importance was frequently emphasized in their on-line postings. P2K provided an opportunity for teachers to develop an active role as facilitators, mentors and "buddies" for other teachers. In choosing to contribute to fellow educators in this way, many found professional recognition and rewards both within their district and beyond. (See the discussion of the P2K Advocates and Awards following in the "Project Training/Development" section of this Final Report.)

Veteran teachers were excited to become "learners" alongside their students: P2K re-energized their repertoire of instructional strategies
Throughout the case studies documented by EDC, teachers reported that they were learning content and process, along with their students. This was a novel experience for many of them, but they reported that P2K made them feel comfortable with it. They were encouraged to work with the new content because of the quality and ease of use of the materials, the supportive community of fellow teachers, and the "real world, real science" context in which the subject matter was placed. The direct access to leading scientists and researchers provided educators with a "virtual faculty" through whom to acquire content and for professional development. P2K also provided a great deal of flexibility in how it could be adapted to local interests, needs and constraints. This was seen as a particularly important aspect of the program.

P2K events generated an excitement which left students open to exploring core content
The connection to real scientists and to the continuing research they are engaged in, and the use of that research as the real-world context for many of the basic standards appropriate to grade level, was consistently singled out as a key strength of the project.

 

The opportunity to supply real input on the use of an instrument such as the Hubble Space Telescope was absolutely incredible. These are kids who've had chances to talk to astronauts on orbit ...so we've done a lot of interesting things, but this was, by far, from my perspective at least, the "hottest" activity that they've ever gotten involved in. It was really very exciting to be a part of it.
Rob Theriaque, Physics and Aerospace Studies, Nashua High School, NH

 

While effective in school, P2K's materials and experiences also had an impact in homes, with parents and caregivers, and in the wider community P2K materials engaged parents and the community: students went home excited about what they had done in school: parents came to conferences curious to know what new things were happening, strengthening the "home-school connection." (For more on this topic, see "Outreach," following.) Science centers that participated found P2K materials also facilitated powerful informal science education experiences.

What follows are data and participant comments documenting these key findings, organized by the Goals stated in the original proposal.

GOAL: Reach students and teachers in schools across America in a high profile project demonstrating to students, parents and policy-makers the effectiveness of new tools to redesign, support and sustain science education in all our schools

Analysis of mail and on-line surveys by EDC showed that:

 •  

P2K reached more middle school students (6-8th grade) than any other age range, thus reaching its target audience, yet also attracted significant numbers of upper elementary and high school students.

 •  

P2K reached all regions of the nation, with participation at or above national norms in rural, suburban and small cities, but slightly below in larger cities.

Percent of Participating Schools by Community Population Density
 
National (1995 Digest of Educational Statistics)
LFA2 Registrations N=389
LFM Registrations N=923
LFRF Registrations N=168

 Rural

31.5%
26.8%
25.96%
32.6%

 Suburban

23.7%
26.8%
26.91%
23.9%

 Small City

16.9%
30.9%
25.24%
28.3%

 Med- Large City
 (over 1,000,000)

25.6%
15.5%
16.63%
15.2%

GOAL: Suggest hands-on inquiry based activities for the classroom, and contribute current content, tied to NSES and AAAS/Project 2061 "Benchmarks," and state guidelines, to the existing science curriculum

EDC's evaluation found that:

 • 

For all Modules, teachers judged the quality of the three genres of Multiple Media components highly, on average rating them at above 3.5 on a 4.0 scale

 • 

In all grades, teachers judged P2K materials relevant to the what they were teaching, but especially so in the middle school grades

Relevance of LFM program to content you are teaching? (N= 579)
Relevance/relatedness, by grade, of LFM videos and Activities to curriculum
Closely related/Related
Partially related
Unrelated

 4-6 (N=76)

73.08%
26.92%
0%

 Middle (6,7,8, N=79)

89.66%
10.34%
0%

 9-12 (N=63)

61.54%
23.08%
15.38%

In Pittsburgh, C.J. Rodkey (Gifted and Talented, High School freshmen and sophomore) commented on the impact of P2K on how students regarded science (EDC, Year 3):

"What do you think are the primary goals of this project in terms of what students are exposed to and/or learn about? C.J. thinks that P2K is real science as opposed to contrived. Students see that the science is contemporary; it is happening as we speak. C.J. also sees P2K as constructivism in action-it presents opportunities for problem-based learning. Technical literacy is another goal directly addressed by the project."

"LIVE FROM THE HUBBLE" brought real-time data from outer space into my classroom. We're a very small school district; we graduate maybe 90 students a year. They were able to work real time, with real scientists… They were able to bring in the data that we wouldn't have been able to see otherwise. They were able to talk electronically, through cyberspace, to professionals in the field, people who are extraordinarily gifted, at that. This is very much like being with the "All-Star" team of the basketball league. The people that you deal with might be the Michael Jordan's, or the Mario Lemieux's of the aerospace world. It's an extraordinary experience.

Terri Garvin (a Presidential Teacher from South Carolina) found P2K's model preferable to other projects with web components she had tried, and powerfully effective in engaging students in inquiry-based learning. As EDC reported in one of its verbatim interviews of case study teachers:

How important was interaction with real scientists in your decision to participate in P2K? The interaction with real scientists was very important in Terri's decision to participate. She said that the scientists give validity to the kids' work. This is something she says that she doesn't get with the GLOBE project. "We are collecting all the data, but we don't get any real interaction."

What do you think are the primary goals of this project in terms of what students are to be exposed to and/or learn about? First, she thinks that her students have learned that science is not something you have the answer to in 45 minutes. It is ongoing and changing. We are learning all the time, and not in isolation. Second, she thinks that her students "have learned really good, basic science concepts." She is very pleased to see that photosynthesis will be a topic in LIVE FROM THE RAINFOREST. When (the EDC interviewer) asked her to give (me) examples of the "good, basic science concepts" her students did learn, she cited their understand of food webs and food chains ...She remembers their excitement about the krill study and research.

GOAL: Encourage teachers and students to navigate in a rich and rewarding tele-computing environment

As noted above, for many teachers P2K was their first experience with integrating on-line materials into regular instruction.

Teachers who had Used Other On-line Projects
 
LFA2 (1996) (N=389)
LFM (1996-97) (N=923)
LFRF (1998) (N=168)

 None

66.4%
52.75%
28.3%

 One

5.0%
8.85%
13.8%

 A few

19.5%
23.68%
41.3%

 Many

9.1%
9.81%
16.7%

(In the preceding table, please note the expected increase in familiarity with on-line projects over time, paralleling the increased accessibility of the Internet in schools and at home.) EDC's surveys found that teachers rated their virtual and real-time interactions with working scientists very highly: P2K functioned as a kind of de facto professional development.

Benefits of participation in P2K to Teachers
Importance
Very
Moderate
Less
Not

 Learning about content from scientists

75.0%
21.4%
3.6%
0%

 Experience with project-based science

71.4%
25.0%
3.6%
0%

 Gaining experience with on-line resources

66.7%
18.5%
14.8%
0%

 Collaborating with other teachers

46.2%
38.5%
11.5%
3.8%

In addition, the use by teachers of the on-line discussion groups-in which they (67% of messages), not project staff (26%)-posted most of the messages, showed teachers' willingness to use this medium of communications to share resources (36% of all messages) and respond to questions from fellow teachers (32%.)

The interactive and "democratic" character of P2K's Internet components, as opposed to the top-down and static character of textbooks or most video projects, was midwife to an authentic on-line community from which teachers benefited. "It brings together experienced and highly committed teachers who are willing to share their time and skills with teachers who are either new to integrating technologies into their classroom practice or are new to science education that goes significantly beyond textbook-based instruction. These teachers are joined by scientists and researchers along with their support staffs. PASSPORT also brings in science curriculum specialists. It is this community that shares in both the development of the modules and in the active implementation that distinguishes PASSPORT from most instructional materials--participants are also partners and not just recipients. (Emphasis added.) The bridging of these normally separate professional communities is made possible by PASSPORT's use of the multiple communications technologies. It is also this commitment and active involvement of the participants which differentiates PASSPORT modules from science documentaries on public television or cable and from most electronic field trips now offered to schools." (EDC report, Year 3.)

As participating science teacher Ginny Dexter commented:

The Internet's made me connected with all the other teachers who want to be involved in this project. We develop our lessons together, we find out from each other how it's working. So we 'Internet' together. And with the kids, it gives them a chance, when they Internet, to say, 'Gee, I get to know this person, and I talked with a scientist who lives here,' and they start looking at maps and figuring out where people are from. It's two-tiered: it helps the teacher and it helps the student.

It is important to emphasize, however, that even at the height of the Internet "bubble", P2K never regarded the Internet or any other single education technology as some kind of magic bullet. EDC, drawing on the research of Pea, reminded us that: "We now know that it is the particularities of the individual teaching and learning contexts-the orientations and activities of the teachers, students, schools, and families-that make the difference in desirable impacts from technology. Whatever else is effective, it is not the educational technologies per se. The social contexts in which they are utilized are all important. It is the blend of the technology, the content and the teaching strategies."1

Terri Garvin reported how the process of sending and receiving back answers through "Researcher Q&A" had powerful results:

"Terri would always screen the questions that students wanted to send to the scientists. Those that they could find the answers to on their own were not sent. Again, Terri appreciated the scientists' feedback because it gave the kids validity. She also appreciated others answering questions because it 'gets me [Terri] off the hook.' For example, there was one time that Terri had taken her students to the science center to do an activity on decomposers (unrelated to P2K). Back in the classroom and returning to their work on Antarctica, one of her students asked how things decompose in Antarctica. 'A dead penguin must freeze before he even hits the ice!' They couldn't find the answer, and as it turns out, the first scientist (to respond) didn't know the answer either. It made both Terri and her students proud to have asked such an interesting and difficult question. (There are, in fact, micro-organisms which have adapted to the cold and act as decomposers in Antarctica. There are also many predators who help rid the landscape of things like dead penguins.) Terri said that the girl who asked this question gained 'invaluable knowledge.' Knowing that she asked an excellent question and getting it answered may have set her onto something that she will pursue the rest of her life."

GOAL: Show that careers in science are possible goals to male and female students of all ethnic backgrounds
From 85-90% of teachers reported that P2K increased student excitement about scientific content, and in careers in science and technology, and gave a clear or some understanding about the topics covered

Teacher Dave Grott (who also lectures at the college level on teacher education) saw how the real-world aspects of P2K-from the mundane to the sublime, as cited in the following comment-engaged a wide range of students, across pre-existing gender barriers:

He remembers the time in LIVE FROM ANTARCTICA 2 when scientists had poop on their hands. His "kids were roaring on floor, but they took it so seriously. These same students asked to see the LIVE FROM ANTARCTICA 2 videos again because there wasn't any information in the library." Students really had to rely on the videos for content, and they asked about them again and again. Dave said that his kids gained "self-confidence, which cut across gender lines." Dave saw his kids all "working together and realizing that they all had learned about science …they all had a hand in the process. They no longer thought that 'girls couldn't learn science.'" He feels their interest in science is so important. They are eager to do because they weren't using the textbook. When they come back in September, all of his students had watched the July 4th broadcast of the Pathfinder landing. Dave believes that the project improved both his students' technology utilization and communication skills. Dave says the differences in terms of student learning/engagement between P2K and other activities are "night and day." (EDC, Year 3 report.)

This year we participated in LIVE FROM THE HUBBLE, and the thing that I really liked best about it was that girls actually got interested in science. The girls who were following Heidi Hammel and participating on-line found things that I hadn't even heard about, and they were saying, "When we grow up, we want to be astronauts." Lynn Davis Hammond, Sacred Heart Academy, Hawaii

GOAL: Add visual context to computer network interaction

EDC's evaluation found that teachers regarded the videos as an important component, and one that distinguished P2K from other projects, and also documented statistically significant consequences of viewing the videos.

 •  

76% of teachers used the Video components

 •  

54%-79% used them on tape while still rating the originally live interactions as very important

 •  

90% planned to re-use the LIVE FROM MARS videos on tape, consistent with earlier findings

 •  

student interest showed a 20-30% increase in student interest in and awareness of careers in science between those who had not, and those who had, viewed the videos

EDC's Year 3 report found a specific, positive "video effect."

 •  

74% of the students felt that participating in LIVE FROM MARS made them more interested in learning more about science and just under 50% of the students stated that their involvement made them more interested in finding more about science careers. It is interesting to note that when (EDC) performed a crosstab analysis on this last question with an analysis of whether they had viewed the videos, the number of students interested in finding out more about science careers jumps to just under 80%.

 •  

62% of the students responded that they learned that there are more kinds of scientists than they knew about before participating in LIVE FROM MARS. Again when we control for video viewing, this percentage grows to 83%.

The integrated P2K model meant that even teachers previously skeptical of the educational benefits of using video in class were prepared to reconsider:

I have been hesitant about using the video programs in my classroom, because I felt students viewed videos as a way to get out of doing work. They have often been used as babysitting type tools and not as a way to enhance learning. What has surprised me about the P2K videos is that the kids really pay attention and have gotten a lot out of them. (Teacher interview, EDC Year 3 report)

For my students living in a rural area, these programs offer my students another perspective about what they might do in their future. Science becomes real, alive and interesting and my students perhaps for the first time think about a possible career in science. (Teacher interview, EDC Year 3 report)

GOAL: Evaluate changes in student knowledge of, and interest in, specific content and concepts

EDC developed measures of student learning outcomes during the 3 years of evaluation which they (and NSF's External Auditor) believed were state-of-the-art, and appropriate for and adequate to the multidimensional character of P2K. As EDC wrote: "PASSPORT TO KNOWLEDGE, because of its innovative design, standards-based content, and high level of teacher participation in many aspects of the program's development, is useful as a laboratory to explore key questions around the role information and communications technologies can play in improving the practice of science education in our country." The AAAS's Project 2061 project developed techniques to analyze how instructional materials embody national science standards. During the "Planet Explorer Toolkit" on-line collaborative activity EDC adapted AAAS's methodology to track how well student work conforms to the standards. EDC's techniques proved able to analyze student postings with high reliability during "Weather Worlds" (a follow-on on-line collaborative activity to the "Planet Explorer Toolkit" activity-LIVE FROM MARS) for evidence of work which embodied not just the NSES, but also the more detailed state frameworks of West Virginia and Illinois.

To the central question of "Does student work that is produced from direct participation in a P2K activity show evidence of grade-appropriate science learning as outlined in the science literacy benchmarks, science standards, or state frameworks?" it was possible to answer "Yes", and quantify the improvement. EDC methodology (content analysis with tests for inter-coder reliability) showed a 22% increase in students' ability to design an experimental procedure to test a selected hypothesis, and was able to link the improvement specifically to one component of the on-line activity: the "peer review" process in which students submitted and then received comments back from other classes." Adding this quantitative data to the positive survey findings moves P2K's results to a level substantially more robust than that of most projects.

However, P2K also learned that there were many challenges to implementing the LIVE FROM model still more extensively, no matter how positive the responses from teachers and students who participated. As NSF suggests, these challenges and problems are also reported as "findings."

Challenges experienced during P2K / LIVE FROM... 1995-1998
The use of live television broadcasts meant teachers had to rely on local PBS stations for carriage, or needed satellite access to NASA-TV, or a direct downlink. The 13:00-14:00 Eastern time slot chosen to reach the maximum number of teachers and students in schools all across America sometimes conflicted with PBS's popular "Ready to Learn" programming. Given the data about teachers' willingness and intent to re-use the programs on tape, cited above, P2K concluded that "live" video programs in themselves are neither necessary nor sufficient to achieve all positive aspects of such projects, and has responded to this Finding in the PASSPORT TO Modules described below and in the "Products" / "Other Products" section of this Report.

Sometimes live questions from students suffered from technical or performance glitches: teachers reported that when other students hesitated on camera, their own students were dismissive. Also when younger students asked questions, older students felt the program less for them.

Lastly while P2K was gratified by the depth and sincerity of overwhelmingly positive teacher responses, it was not clear that "mainstream teachers" were readily adopting the rich suite of learning materials, replacing textbooks or other instructional materials. P2K Advocates were winning awards, but sometimes they were in fact resented as "alpha" teachers and "early adopters" by others in their school for whom the extra time and effort were not feasible, nor readily offered.

Responses to Lessons Learned
P2K has already responded to the first two points: the video and other content secured during the initial NSF-supported grant period has been completely re-edited to form the basis of 3 "evergreen" Modules, distributed on tape. The 3 Modules, PASSPORT TO THE RAINFOREST, PASSPORT TO ANTARCTICA and PASSPORT TO THE SOLAR SYSTEM, each comprise of 8 x 15:00 classroom videos, and 2 x 30:00 Teacher Resource programs. The Modules use the original Teacher's Guides (all bearing the appropriate acknowledgment of NSF support) and a new Implementation Guide providing a complete road map to the package of video, hands-on and on-line resources. These materials have been licensed by public television stations and statewide educational networks across the USA, reaching a potential K-12 population in excess of 10 million students. NSF's original grant lives on in the new format, thereby amortizing and leveraging the initial investment. Self-distributed by P2K and supported by web and on-line mail lists, this is effectively equivalent to publication and national dissemination.

PBS stations such as WNET, New York, make PASSPORT Modules a feature of their teacher training institutes. Teacher reviewers in Wisconsin, Virginia, Iowa, California, Florida and many other states rate the P2K Modules and their multiple media materials highly: without such educator support, PBS stations would not carry the P2K Modules.

Further, the NSF grant and the positive results documented above have led P2K to the articulation of a model which has been implemented in follow-on projects not supported by NSF funds.

LIVE FROM THE SUN (supported by NASA) included 3 hour-long programs. LIVE FROM THE STORM (supported by NOAA and NASA) had 2 programs. LIVE FROM A BLACK HOLE and LIVE FROM THE EDGE OF SPACE AND TIME debuted in the 2000-2001 school year. LIVE FROM THE STORM has already been re-versioned into PASSPORT TO WEATHER AND CLIMATE in the "evergreen" format described above. The pilot program for this series was rated #1 out of 163 submissions at FirstView 2000, the annual public television screening of new instructional television programming, and debuts in Fall 2001.

The positive and negative findings cited above have also helped shaped a new proposal to ESIE/IMD, "Science Concepts in Context", SCiC. SCiC applies and extends the lessons learned from more than 6 years of R&D, 3 of which were supported by the NSF grant: by name and title it is designed to appeal to mainstream teachers for whom EFTs and "the rainforest" and "Antarctica" might seem discretionary in the ever-more pressured school environment. But, as described above, SCiC incorporates some of the most important positive lessons of P2K: the use of video to excite learners of multiple talents and intelligences; the use of hands-on activities to make key science principles real and memorable, and on-line interactions to connect students and teachers with researchers at work in real locations. The SCiC pilot program, 3.3 FORCE AND MOTION, was again rated #1 at the 2001 FirstView screening, recently completed.

P2K's original "bumper sticker" slogan read: "Real Science, Real Scientists, Real Locations, Real Time." SCiC has now substituted "Real Learning" for the latter phrase, but the findings from previous NSF support lead P2K to believe that connecting classrooms and core science curriculum to real world research and researchers remains a powerful, effective and worthwhile mechanism through which to enrich, enliven and empower middle school science education.

NSF's requested "External Audit"
In addition to the evaluation undertaken by EDC, NSF's program officer (Gerhard Salinger) requested what he called an "External Audit" in all 3 years of the project. (It should be emphasized that this was to focus on non-fiscal matters, contrary to the usual implication of the word "audit." This would be someone otherwise not connected to the project and owing no loyalty to it, but who would have free rein to request access to any aspect of the project, and "kick the tires" (in Dr. Salinger's words), reporting directly to NSF about pluses and minuses. NSF suggested Dr. Carl Pennypacker, PI of HANDS-ON UNIVERSE, who proved willing to undertake this role. Pennypacker enlisted Jodi Asbell-Clarke, then associated with TERC, and reported each year to NSF and P2K. What follows are excerpts from all 3 years of External Audits, edited to provide running commentary on the goals and other findings reported above. (We assume the original reports are all accessible in full at NSF.) Except where noted (comments in parentheses by the reporting PI, or Editor) the following comments are taken verbatim from the External Audits, with truncations noted by (...)

External Audit, Year 1
The materials produced by P2K are of excellent quality. They must be viewed as a complete package to see the well-roundedness of this project. Each component (the live or video broadcast, the WWW site, the teacher activities and materials) provides an essential element to the overall project...

...The broadcasts are professionally produced and give the effect of an action-packed TV show while retaining the authenticity of live student/scientist interaction. They are visually exciting and filled with current factual information. The use of high school students as moderators and guides during the broadcast and the ample time spent on student questions is a clear demonstration of P2K's commitment to make this a student-centered experience. There were also a notable number of roles played by women and visible minorities. In fact 3 out of 4 of the primary scientists on the LFHST team were women. Most importantly, this selection of role models is not overstated but rather dealt with as a natural element.

...The highly visual, multi-media, and real-time nature of the P2K projects enables more than creation of excellent curriculum activities. The electronic field trip not only excites students about science that is being done by others, but involves the students, themselves, in the science. Choosing the planets to be observed by HST (Hubble Space Telescope), visiting the HST control room for the space telescope, and watching the maneuvers the satellite makes in order to complete the student's own observations are powerful tools for young people… this event is certain to have a very important impact on the students involved. They experienced for themselves the scientific process of choosing observations and following through with the analysis. They became connected with leading scientists and scientific equipment. A significant contribution to science that this project makes is educating a group of future voters, politicians, and global citizens to recognize the value of the process and tools for scientific research.

...There is no doubt that P2K is highly successful in the hundreds of schools that are currently using the project materials in their fullest potential. There are countless other schools that may watch the broadcast or visit the homepage intermittently, having access to a kid and teacher-friendly method of obtaining Hubble images and reading scientist journals. It is unclear what impact P2K will have on classes that do not have WWW access or cable access in their classroom. A NASA broadcast may reach many schools in a relatively inexpensive fashion, but the live broadcast may not be available at the same time that the students are scheduled with the teachers that are using P2K in their class.

Evaluation
...P2K has evaluation forms posted in the back of each teacher activity book and requests for the completion of these forms go out during the broadcasts and through notices on the WWW site. There is an evaluation plan in place with EDC to analyze the data from these forms and follow up with teachers. EDC's plan to access information and cover a broad base of experiences appears solid and doable. The shortfall of the current evaluation plan lies in the scope of the questions themselves. The questions deal primarily with the amount of material covered, successes and failures of implementation, and the attitudes of teachers and students towards the program itself. There appears to be very little in the way of assessment techniques developed to measure the content of what students are learning... (See above for how EDC and P2K responded to this critique: Editor)

Long Term Impact and Future Direction
The time-critical nature of the real-time interactivity of P2K projects does not lend itself to long-term use of each project. It is doubtful that using the videos from previous projects along with the teacher activities will have the same effect as being involved with the email and WWW communications along with the scientists and other classes. It is clear that the few classes that are selected as participants of the live broadcast have the ultimate interactivity and thus presumably the most dramatic impact from the P2K experience. It is unlikely that the students from Lincoln High School in East LA will soon forget their time doing image processing live with Heidi Hammel via TV broadcast.

...P2K has many projects planned for the future that will carry on the newly set "tradition" of electronic field trips. It is likely the best use of staff time to make the real-time interaction of the new field trips as exciting and educational as the previous missions, rather than spreading their efforts too thinly by keeping up with old field trips. On the other hand, this makes each one year mission disposable and a very short-lived project for the vast resources put into it. This is a conundrum that the team will have to examine carefully. Perhaps a few key teachers that become very involved with an individual project can be recruited as future moderators and resource agents for the long-term implementation of that project. (This is exactly what has been implemented with the PASSPORT TO Modules described elsewhere in this Report, and in the proposed SCiC project: Editor.)

Conclusion
P2K has grown rapidly and has a very impressive past and future, particularly given the youth of this program. The overall result of this external audit is that P2K is well on its way to becoming a preeminent science educational program. The staff is very responsive to lessons to be learned from previous experience and the project is evolving rapidly as they learn how to implement such an innovative educational technique over a broad audience. Other projects using multi-media and real-time interactivity for science education can look to P2K for valuable lessons and most certainly there will be a growing list of teachers and students around the world waiting to see what P2K will be coming up with next.

External Audit, Year 2
...The format and structure of P2K have not changed dramatically since the 1996 external audit, however some of the concerns and suggestions brought up in our first review appear to have been addressed. The curriculum is more focused for the middle school grade level, rather than trying to suffice for all age groups. The investigations chosen for each of the topics are student-appropriate, they are interesting to the right age group and are rich in content that is outlined in national math and science education standards for those grade levels.

...The interaction with the scientists involved in each project is brought to life on the WWW site. Each scientist keeps an electronic journal, recording daily events or progress with an experiment. As students and teachers read these, over time they begin to "know" the scientists. This is evident when they shout out their names in recognition when they "meet" them on the videos. The relationship is further enhanced as students develop questions. Some of the questions are answered during the broadcasts, with a few classes participating in live CUSeeMe type sessions, others are asked over email and either answered on-line or via email later.

Why Do Teachers Love P2K?
As necessary as the three components of P2K... are, it seems an intangible, yet key, fourth component is the relationship among the teachers, staff and the scientists. It is that relationship that motivates students (and teachers) to involve themselves in the curriculum activities, to log on and read what the scientist's journal entry for the day has to say, to feel comfortable in asking a question, and to shout out in jubilance when they see the scientist at work in the video. It is the personal relationship that makes the video broadcast differ so greatly from an ordinary TV documentary. These videos were made for the P2K audience, an audience that existed before the show was created and helped shape the broadcast. This is most evident in the segment of the show when the scientists field questions from the live on- line audience, but also in the manner that the narrator and scientists refer to the activities and themes that the students have been studying all along.

...As EDC obtains more results on how P2K is changing teaching and learning, it is critical that this information is disseminated objectively. P2K will undergo many successes and failures as an envelope-pushing project. It is imperative that this type of project continue to grow as both an incentive to educators to become more involved in innovative science and technology programs, but also as a canary in the coalmine finding the pitfalls and how to avoid them, and finally as a prototype of a successful program for others to emulate. (Responding to this comment, EDC's Year 2 report was published in full on-line, and remains accessible. Editor)

External Audit, Year 3
This review is part of an on-going external audit of PASSPORT TO KNOWLEDGE, as requested by the National Science Foundation. Previous reviews have been undertaken by the present auditor and Jodi Asbell Clarke. Because of time commitments, Ms. Clarke could not be part of the audit...

(P2K) has developed a rich set of materials to yield productive activities far beyond the broadcast time. The evaluation of P2K by EDC is excellent, and has done a remarkable job at finding exactly where P2K is helping teachers change to help support the standards. Portions of the EDC collected and evaluated data yielding some of the better proofs that this type of work can change students, too.

In general, P2K is broadly successful, measured by a number of metrics...

...P2K now is carried on 287 Public TV stations. Its on-line list-serve is large, and it is inconceivable that portions of its materials are not being used in thousands of classrooms. There are large numbers of hits on the web pages (where curriculum materials, chat rooms, bbs's are located), and the tapes are viewed afterwards by large numbers of students. If judged by numbers alone, P2K would be termed a success.

However, what is encouraging now-based on the EDC study-is that P2K has been shaped so that it is almost certainly changing some reasonable fraction of the users to science education that supports the standards. The evaluation studied those teachers who returned postcards on filled-in on-line forms. In the following paragraphs, I (Carl Pennypacker) list some of the ways P2K is succeeding.

IV. P2K Supporting Science Standards and Evaluation
There is a growing sense amongst science educators that "doing science" is the best way to learn science. Indeed, the science standards, particularly the NRC (pg. 143) delineate "As a result of activities in Grade 5-8, all students should develop: i) abilities necessary to do scientific inquiry; ii) understandings about scientific inquiry. P2K is taking significant steps towards developing that kind of inquiry, I am convinced. The way the experiments are designed, discussed, reviewed by other students, and the published is very well thought through.

In particular, the notes on the "Student Impacts" section of the evaluation (cited above: Editor) was some of the most convincing evidence I have ever seen that P2K is improving students' understanding of inquiry-based science. EDC should be acclaimed for their methodology, their insight, and their persistence at discovering this gem. That is, EDC found that students improved by 22% their ability to "design procedures to test selected hypothesis" in pre/post-tests...

X. Thoughts for Future Sustainability
A. Early Users and Reaching the Average Teacher
One of the principle questions for P2K is can "average" teachers engage as much as the self-starters/advocates. The answer depends on the continual progress P2K makes in making the system function and succeed in ordinary classrooms. This is an ambitious goal, but one that has to be always at the forefront of P2K's thinking. P2K should not cater solely to the advocates-the advocates can put up and succeed with anything, they are so enthusiastic. But the system should not base its growth on their feedback alone, now.

B. Cost Recovery/Foundations
I wish I had some great wisdom to share here that P2K could use to generate a stable, self-sustaining system. However, I think they are still one grant away from being self-sustaining. They have made tremendous strides in quality of materials. The argument for more grants is simple to me in some ways... Once P2K can make it through the next grant cycle, it should then start to cost recover a little bit more... In some sense, "fighting for funding" is just ridiculous for good programs like P2K. It shows some gross imbalance in our society. That is, here is a program that is being used for months and months in thousands of classrooms in America, yet has to justify its existence every year or two... (Carl Pennypacker, LBL, PI, Hands-On Universe)

Activities Findings Outreach Contributions