Passport to Knowledge is an innovative educational program that focuses on middle school science learning by combining video broadcast events with extensive print and on-line resources as well as interactive applications of the Internet. The project is funded by the National Science Foundation (NSF) and the National Aeronautics and Space Administration (NASA). Each year, the project offers teachers at least two different multiple media "field trips" that range in length from several weeks to some that span the entire school year and beyond. Annually thousands of teachers use at least some of Passport to Knowledge’s components.

As part of a three year National Science Foundation grant under the Instructional Materials Development program, Passport to Knowledge has contracted with the Center for Children and Technology of the Education Development Center to evaluate the project. This report presents the findings gathered during the second year of the three years.

The purpose of the evaluation is both formative and summative: it describes who is using the program, how they are using it, and what is the project’s impact on teaching and learning. The evaluation involves developing and implementing a comprehensive array of documentation and evaluation strategies to provide critical feedback to the Passport to Knowledge staff as their curriculum development work evolves. At the same time, we are providing information and analysis to help the staff, the project funders, and the participants assess the project’s impact on student learning and teacher professional development.

This past year Passport to Knowledge produced Live from Antarctica 2 and Live from Mars. The research with teachers this past year reveals the following key points:

• Passport to Knowledge reaches an extremely broad and diverse audience. Teacher participants are spread across all grades, but middle school teachers are the most active users, (29% for LFA2, 53% for LFM). LFA2 was more popular with elementary teachers. Middle and high school teachers using the materials are typically science teachers and use the materials with four classes, on average. Module participants consistently include a significant number of adults who are not classroom teachers, and who may not even be working with young people (as much as 60% of the respondents to some surveys).

• Educators’ responses to Passport to Knowledge (PtK) were uniformly positive. The Passport to Knowledge materials were viewed by participants as being made "by teachers for teachers" -- that they were informative and well-designed to support real classroom practice. Although teachers found all components of each module useful, they indicated that the online resources were the most valuable to them and the ones they would most like to see expanded upon in the future. Teachers particularly valued the depth of background information provided on relevant topics through module elements such as the "background information" area on the LFM web site, and the scientist biographies for LFA2.

• The on-line resources continued to grow in importance. This reflects the fact that more teachers are finding it easier to get on-line then ever before. The web site has become a major avenue for attracting teachers into full project participation. It is the place where teachers get information about the video broadcast schedules and where many (the majority) now obtain the teachers guide. Teachers particularly valued the depth of background information provided on relevant topics through module elements such as the "background information" area on the LFM web site and the scientist biographies for LFA2.

• Teachers in widely varying classrooms are using Passport to Knowledge materials in hundreds of different ways. No two classrooms using this material look exactly alike. Passport to Knowledge is successfully supporting activities ranging from grade school students tracking the course of the Pathfinder via the LFM web site, to homeschoolers participating in the PET project with students from across the country, to adult learners in literacy programs communicating with scientists. Within the context of particular educators’ participation, the materials also prove to be flexible. Teachers report that they plan to re-use their Mars or Antarctica material, and many teachers report using the materials with a range of grade levels and across disciplines.

• While Passport to Knowledge is not meant to be a comprehensive curriculum for middle school science, teachers are able to successfully integrate it with their existing science curriculum. The vast majority of teachers using either module report that it aligns well with their existing curriculum and their teaching objectives.

• Over half (53%) of the of participants in LFM, and 66% of LFA2 participants, had never used telecommunications before. This clearly indicates that a major audience for Passport to Knowledge is teachers who are looking for a meaningful online activity to introduce them and their students to using multiple electronic media as an integral part of their classroom. Passport to Knowledge is serving as a comprehensive, successful technology-integration experience for teachers in schools that are just beginning to get connected to the Internet.

• Recruiting and investing in a core group of teacher-advocates who were at the forefront of a range of Passport to Knowledge activities proved to be a highly successful decision. These teacher-advocates have contributed significantly to Passport to Knowledge, most importantly by supporting other participating teachers with day-to-day advice, suggestions, and collaborative partnerships via a highly active listserv. The advocate experience was beneficial for the program and for other participating teachers, but also for the advocates themselves: Advocates reported that their experience with Passport to Knowledge improved the quality of their own teaching; their mastery of the scientific content; and the quality of their relationships with other teachers and with their own administrators. The Advocates have become a close-knit group, and have established themselves as teacher-leaders in the use of educational technology in their local districts.

• For some teachers, participation in Passport to Knowledge focused less on student learning and more on their own professional development. For example, 38% of classroom teachers participating in LFM reported that they were not using LFM with students, but instead were focusing on building their own content knowledge, learning new teaching techniques, interacting with other teachers, and learning to use the online components of the program. Similarly, in relation to LFA2, many teachers reported that they were more likely to be using each component of the module for professional development than for student activities. In both cases, teachers were doing this as a dry run in preparation for using with students in the next academic year. Another group, usually building or district level curriculum leaders or technology coordinators were using the module to evaluate Passport to Knowledge's usefulness for widespread adoption and to prepare themselves to serve as a local resource when adoption occurred.

• Passport to Knowledge modules continue to attract and hold onto experienced teachers and those who have received acknowledgement as some of this country's best science teachers. In addition to attracting those new to telecommunications and educational technology integration in science instruction, some of Passport to Knowledge's most ardent and vocal supporters are leading science educators or state or national award winners who have voluntarily self-selected these materials. These are educators who are at the forefront of science reform, who have actively participated in creating science frameworks and benchmarks, and who are collectively among the most knowledgeable users of science curricula and materials.

• Teachers reported that the Passport to Knowledge materials had a positive impact on affective variables for students. Teachers uniformly reported that the Passport to Knowledge videos increased their students’ excitement about the module topics, their motivation for learning about science and their curiosity about science careers. They also reported that the videos, websites and print materials were valuable instructional resources that were appropriately geared to their students, and were interesting and relevant to them.

• Passport to Knowledge materials effectively support teachers in moving toward standards-based science teaching. Key Passport to Knowledge activities such as the Planet Exploration Toolkit activity, the topographic and geographic activities, and the rocket and payload design activity, were uniformly perceived as succeeding in placing students in the role of investigators, contributing to authentic scientific practice and increasing student knowledge in the relevant content domains. Work produced by students exhibits the learning outcomes associated with science standards and benchmarks.

• The key barriers that prevented some teachers from fully participating in Passport to Knowledge are the quality of in-school access to the Internet and broadcast television and the rigidity of pre-existing and/or required science curriculum. However, the flexibility of the Passport to Knowledge materials maximized the number of teachers who could pursue their own and their students interest in these modules despite being constrained by these variables.

• Teachers clearly indicate that the modules have shelf-life beyond the duration of the live events. Teachers report using the videos during the live broadcast and on tape over a period of may weeks following the original broadcast. Over 75% of them indicate that they are saving the tapes for repeat use in future years. Teachers also report using the overview video included in the full kits as a means to introduce the module to their students.

• Those classrooms that actively participate in Passport to Knowledge activities have students producing work that clearly demonstrates appropriate science benchmarks. This work is produced in classroom discourse, queries to experts, writing assignments, and projects as well as diverse formats ranging from poetry to drawings and models to multimedia presentations.

This evaluation report covers the second year of Passport to Knowledge’s three year grant from the National Science Foundation for Instructional Material Development. As part of this grant, the Center for Children and Technology at the Education Development Center is conducting an assessment of the challenges faced by Passport to Knowledge and its achievements in meeting those challenges. The work will culminate in a final summative report due in Year Three. However, as shown in this interim progress report, there is already clear evidence of the project’s impact on teachers and learners in a wide variety of situations.

Before presenting the details of the evaluation, it is important to set Passport to Knowledge’s work in a larger context. As this report will document, Passport to Knowledge is attempting to transform significant aspects of science education for middle school students. It is doing this by developing innovative instructional materials and experiences for students and placing them within a multiple medium communications infrastructure that leverages the respective power of television, video, and the Internet as educational tools.

At the same time, it is using this infrastructure to foster an engaged professional community that centers on teachers but also includes scientists, engineers, parents and other community members. This ambitious undertaking puts Passport to Knowledge squarely in the middle of the intersection of two major forces that are working to reshape American education. One force is the exponential increase in the purchase of technology in general, and computer-based telecommunications technology in particular, for use in education. The second force affecting education today is the widespread public interest in school improvement, the pressure for accountability and results, and the active debate in our society on how best to achieve substantial reform of our schools. These two forces need to be examined in more detail.

The United States, and in fact most countries around the world, are in the midst of a major investment to increase the use of technology in K-12 education. Whether one measures it in terms of financial commitment, or the re-alignment of human resources and energies within schools, or by the heightened public expectations around what technology will do for our children, the scale of this investment is enormous. During the 1996-1997 school year, the period covered by this report, there was significant acceleration in the number of teachers and students gaining access to the Internet.

When Passport to Knowledge began working on its NSF IMD grant in 1995, the National Center for Educational Statistics surveyed schools for the first time to find that 35% of public schools (mostly secondary) had some form of Internet access. In the short time from that survey to their latest data from the fall of 1996, access has grown to 64% of public schools reporting Internet access. The percentage of elementary and middle schools in comparison to secondary schools with access has increased, as has the level of access within classrooms within those schools (from only 3% to over 14% of classrooms). The gains have continued unabated since that survey and perhaps even accelerated in the past year. And this change in schools only mirrors the changes in connectivity taking place in the larger society in which schools are situated: business, community agencies, and homes are also connecting at a rapid rate.

Recent reports demonstrate this penetration. QED’s 1997 report on technology penetration in U.S. public schools for 1995-96 (using sampling techniques that are somewhat inadequate given the rapid rate of investment in many states), estimates that 98% of schools have access to computers (with an average student to computer ratio of 10 to 1), 97% have VCRs and 76% have cable TV access. Over 85% of the schools have access to computers with multimedia capabilities and 54% have CD-ROM drives. Satellite penetration has grown to just under 20% of our schools.

Spending on educational technology infrastructure increased at the local, state and federal level. This year marked the beginning of the President’s Technology Literacy Challenge program, which provides states with significant funds to direct to local education authorities for planning and implementing technology. This builds upon other sources of systemic funding being used to apply technology within education, such as the Goals 2000 block grants and Title One funds, as well as the significant funding allocations made by many state legislatures this year that explicitly target technology expenditures for schools.

1996 also saw the creation of the Universal Service Program as part of the Telecommunications Reform Act. This program, through its "E-rate" discounts for schools, is already having a profound effect in spurring planning and investments, even though its funds will not be available until 1998 at the earliest.

While there are growing concerns that all this rapid investment and desire for quick fixes might precede, or even hinder, the necessary planning required to ensure its optimal impact, all indications seem to point out that we are only near the bottom of a parabolic curve. The trends toward growing technology investment and the demands for a return on these investments that tail immediately behind them will continue to accelerate.

Coupled with this push for technology infusion is the national discourse on school improvement. This manifests itself in the work done on curriculum standards at local and state levels which in turn has benefited from the national work done by professional organizations such as the American Association for the Advancement of Science’s Project 2061 Benchmarks for Science Literacy and the National Research Council’s National Science Education Standards. This work has evolved from building a consensus on what students should know and be able to do at various levels to specific state and local descriptors that can be used for as the basis for curriculum and teaching decisions and for student assessment of progress. Forty-nine out of the fifty states have some sort of standards in place or under development.

Technology is often cast as a potential means, sometimes as even a necessary component, of successful educational reform. As stated in a recent monograph reporting TERC’s work on the Model Schools Partnership:

Even materials that focus exclusively on connecting schools to local and wide area networks give credence to the role this technology can play in school change. The potential of networking for schools and districts has been depicted in many places. For instance, in NCSA’s report on their NSF NIE work, the authors state:

There are some voices of caution among the general noise and enthusiasm surrounding this technological magic bullet. Many observers note that technology, by itself, will do little to change or impact education. It is only when the technology is applied within the framework of specific teaching and learning at the district, school, and classroom levels that we begin to see educational transformations. Where it works, technology use is supported within schools by broader change strategies with ample teacher professional development that includes development within the classroom and during ongoing instruction. This often means focusing on the students as part of the technology integration process rather than just as ultimate beneficiaries of it at some point further down the road.

We now know that it is the particularities of the individual teaching and learning contexts – the orientations and activities of the teachers, students, schools, and families – that make the difference in desirable impacts from technology. Whatever else is effective, it is not the educational technologies per se. The social contexts in which they are utilized are all important. It is the blend of the technology, the content and the teaching strategies.

In light of these developments, projects like Passport to Knowledge can contribute more than just the specific instructional materials they have been funded to develop. They need to be examined for what we can learn about large-scale, implementation models for these types of resources and activities. Few projects in existence today address content, process and pedagogy in an integrated fashion. The scale of Passport to Knowledge, in terms of widespread availability and utilization in numerous contexts, is more characteristic of a national testbed or a design experiment implemented on a national scale. It is a concerted effort to address educational reform at a critical point in the teaching and learning process: the teacher and students interactions with stimulating real-world scientific endeavors. It uses a coordinated mixture of technologies to accomplish this but it also encourages the active participation of teachers and students as more than recipients of an instructional delivery system. It is this combination using various delivery formats while developing a human infrastructure to support its teaching and learning that makes PtK an interesting endeavor.

At the same time, it is important to acknowledge that Passport to Knowledge modules are not intended to be comprehensive whole year or multi-year curricula in science nor are they meant to entirely replace existing instruction. However they are providing concrete, direct experiences for both teachers and students in the meaningful utilization of education technology that fundamentally changes science teaching and learning in participating classrooms. The effectiveness of this project can and should be evaluated based on the learning environments it assists teachers in creating and the capabilities it provides to do things one could not do otherwise, rather than considering effectiveness as a function of the technology itself.

Passport to Knowledge’s significance lies in part in its widespread availability for immediate use in classrooms today. It does not require that the infrastructure be fully realized before teachers and students can learn how to use it. As enumerated herein, teacher teams at the building level, individual teachers, and even parents at home are choosing to participate in Passport to Knowledge. They are utilizing its resources to aid in their own professional development and they are improving the instructional opportunities for the learners for whom they are responsible. The numbers that detail PtK’s audience are impressive. Beneath those statistics lies the equally compelling story of PtK’s impact on individual classrooms and teachers. One of the most articulate and widely disseminated such story is that of Rhonda Toon, a public school teacher from Lamar County, Georgia as published earlier this year in Business Week:

Ms. Toon’s expression is notable but it is not unique, as a search on the growing archive of messages posted to the various discussion lists used within PtK modules will quickly reveal.

It is within this broader context that our evaluation takes on added interest: our task is not just to document the reach and the impact of Passport to Knowledge modules, such as this year’s two offerings, Live from Mars and Live from Antarctica 2, although that certainly is part of it. In the report that follows, we are presenting the initial work to draw out the larger lessons that can be gleaned from Passport to Knowledge’s position. Even at this interim stage in the three year evaluation effort, it is clear that Passport to Knowledge is doing something that has implications for all who anxious to see that the opportunities arising from this historic convergence of public investment, high expectations, and desire for serious improvement in education are not squandered.

This evaluation is a work in progress. Evaluating the impact of any technology on learning is challenging and evaluating it within a project like Passport to Knowledge, which is be used by many different types of teachers, in a wide range of grade levels, who are appropriating it in so many diverse learning situations, is all the more so.

This report, along with its final successor produced in year three of the evaluation, add to the work in overcoming these challenges to evaluation by addressing two critical areas as identified in the OERI January 1997 report "The effectiveness of using technology in K-12 Education: A preliminary framework and review." One area is distinguishing between the "hype", assertions, hopes and expectations, and data-based research findings that tell us what really works and what does not. Both in the formative feedback provided to the PtK staff throughout the year, and in the analyses contained here, our focus has been on addressing this issue. Secondly, it is often noted that technology’s impact on learning is difficult because there are few settings in which it is used optimally.

The accessibility of the Passport modules and their voluntary use by teachers in so many different settings offers a research window that goes beyond the interesting but limited-in-scale projects that characterize many of the current innovative technology applications to education. The need for this research has been underscored in recent studies and policy papers. The ETS study and the newly released Presidential report conducted exhaustive reviews of research on the impact of educational technology and have found only a small number of studies that have tackled the more pedagogically complex uses of technology. Both reports contain calls for new and ongoing evaluations of cutting edge technology projects while acknowledging the methodological and practical challenges facing such work. It is our intention that the evaluation of Passport to Knowledge will make a contribution to addressing these needs both through its findings and through the techniques we are evolving.