Introduction

Passport to Knowledge (PtK) develops integrated multi-media projects that blend live television broadcast and videotape, print materials, and on-line resources to give students and teachers first-hand insights into areas where science is an exciting, a ctive process. Each module focuses on a real-world event or activity that is happening now. In doing this, it helps teachers use a spectrum of instructional methodologies and communication technologies. Passport to Knowledge is one of a small number of education projects that are structured around engaging working scientists, engineers and technical support staffs with students and teachers who are themselves engaged in related hands-on investigations of scientific topics appropriate to a middle school science curriculum.

This report examines both the scope of this work and its impact with students and teachers as the project completes its second year of funding from the National Science Foundation (NSF). This period, from August 1, 1996 through July 31, 1997, was also supported by continuing key contributions from several groups within NASA, by PBS, Mississippi State University, the American Museum of Natural History, and many others. It is an interim report and one that emphasizes both what is working for Passport t o Knowledge and what might still be enhanced as the project continues to evolve.

As part of an ongoing three year evaluation that will culminate in a comprehensive examination of Passport to Knowledge’s impact, the Center for Children and Technology at the Education Development Center (CCT/EDC) worked with Passport to Knowledge sta ff and the educators and learners that use the materials. Our objectives in this middle year of the research were to continue helping guide the project’s development through regular feedback of a formative nature on major project events and initiatives, to examine both the ongoing challenges and any new ones facing the project, and to document its achievements.

At the same time, we are also looking at Passport to Knowledge as a comprehensive research opportunity to gather knowledge about how the power of educational technologies can be harnessed to create meaningful learning experiences for students and teach ers. As such, we are concentrating on how PtK is being used as way to explore key questions around improving the practice of science education in our country through the effective use of new educational technologies.

The topics selected by Passport to Knowledge this year - the ongoing Mars missions and the wide range of polar research undertaken by scientists worldwide - are at the forefront of positive public interest in science. PtK leverages the reports in newp apers, on television and in popular magazines. This heightened interest, combined with the significant public accessibility offered by repeated broadcasts of the Passport to Knowledge materials on public television and through cable systems and an open, highly visible web site, creates an unique opportunity for sustained, systemic impact rarely found in new curriculum development projects. Few curriculum projects in science or any other discipline area are voluntarily picked up and adapted both above an d below the middle school target. In fact, PtK modules are used outside school settings entirely -- they consistently attract interested adults. This popularity range makes PtK rare, and worthy of study for that reason alone. However, it is Passport's impact on teaching and learning within middle school classrooms that is our central focus herein.

Our work on the Passport to Knowledge project began in March of 1996 and during the first year we produced two reports. CCT/EDC’s first report covered our evaluation work on the Passport to Knowledge module Live from the Hubble Space Telescope a nd it also provided some analysis of the earlier module Live from the Stratosphere and the original Live from Antarctica.

The second report focused exclusively on survey data from participants involved in the summer of 1996 Live from Mars Teacher Workshop. This evaluation analyzed the experience of on-site participants as well as those that participated in the wor kshop remotely through one or more forms of electronic media.

This is the third report in the series. It is aimed at helping Passport to Knowledge staff, funders, and users - the students and teachers - reflect on the progress and impact of the project and to inform its ongoing development and refinement. Throu ghout the report, we have integrated summaries of data with qualitative analysis to achieve this dual purpose. As with earlier reports, we summarize these recommendations, in this case for Year Three, in the final section of this report.

CCT/EDC’s evaluation work on Passport to Knowledge during the current year focused on the main programmatic features of PtK that were offered or developed in the past year. A full description of the modules offered in year two is presented in Appendix Five. Briefly, these include:

The new, multi-school year module Live from Mars (LFM) that started in the fall of 1996 and will run through December of 1997. Previous modules were shorter in duration and all occurred within a semester. LFM consists of five broadcast program s, including two at the time of Pathfinder’s landing on Mars in July. The materials include a comprehensive teacher’s guide, an optional classroom kit with posters, background materials, videotape overviews, and a CD-ROM, as well as an elaborate web site , e-mail lists, and interactive cyber-events such as a regular schedule of web chats with NASA scientists and engineers. Also this module features an on-line collaborative activity, the Planet Exploration Toolkit, that built on the interactive on-line pr ojects of past modules but was for more elaborate and ambitious that what had previously been offered.

A three-program Live from Antarctica 2 module that followed the Antarctic summer scientific investigations of researchers based in Antarctica. It was more like previous modules in that the live aspects of the program spanned three months in the middle of the school year. LFA2 also had a comprehensive teachers guide, an optional classroom kit, and on-line features that included e-mail lists and a web site. The web site for this module was used to demonstrate and test some design implications of our earlier findings about the range of participants using PtK modules and the increasing importance of the web within PtK.

This was also the year that some of Passport’s most experienced teachers along with some new but energetic teachers were selected to form a teacher outreach corps, called PtK advocates, and these teachers offered local presentations, workshops, product development critiques, and on-line assistance.

The focus of our work was on gathering information and analysis to help answer the broad framing questions for PtK’s evaluation, rooted in the original grant proposal to NSF, as part of the three year cumulative study of Passport to Knowledge. Those q uestions were generated at the start of our work in 1996 and still provide the overarching framework for the evaluation. See Appendix One for a list of the questions.

During this year, we significantly increased the number of participants surveyed and interviewed while at the same time we refined and improved our instruments piloted in the first year studies. We also added assessment of student work as a means to b etter understand the impact of Passport to Knowledge participation on student outcomes. In our previous reports and those covering the evaluation of pre-NSF pilots of Passport to Knowledge, student evaluation questionnaires were used to gather student ra tings at the end of their school year. While somewhat useful in gauging students’ reactions to module components, these instruments were unable to provide an indepth look at what students were really learning from their involvement with Passport to Knowl edge experiences. Therefore, this year we moved to using actual student work products as a basis for looking at learning outcomes. While this work will be most apparent in our overall project assessment in Year Three, these techniques are already bearin g fruit as demonstrated in this report.

Our work this year utilized several tools devised and piloted in the first year. Based on the feedback from the questionnaire used in earlier evaluations, we undertook a substantial revision to simplify the format while delving deeper into issues related to the framing questions. We had piloted sending out and collecting surveys through e-mail in the first two reports. We made extensive use of that technique this year and augmented it with web-based surveys and registration forms for both module s. As a result, we have surveyed a good cross-section of Passport to Knowledge users and can make some statements about their habits and opinions with confidence. See Appendix Two for copies of the various survey instruments we have used this year.

The surveys targeted users of the videos, the print guides, the listservs and the web site in order to access usage patterns, types of users, and how the materials were perceived by both teachers and students. Since the basis for these surveys was e-m ail and the web site, we augmented this bias by sending out print versions of relevant surveys to postal addresses of users we were able to identify. Our main technique to reach those who do not use the electronic components of Passport to Knowledge was through prepaid quick response postcards for teachers and the inclusion of it in all teachers guides sent out for both Live from Mars and Live from Antarctica 2. Appendix Three contains the text for these postcards, which is also the same a s that used for the web-based registration forms.

We are supplementing the survey questionnaires with case studies based on our work from last year. However, we have expanded the pool of participants to better match the various categories of Passport to Knowledge classroom users. We selected case st udies to reflect the range of experience among participants with respect to prior use of telecommunications projects and previous participation in Passport to Knowledge.

We also continued coding messages on the project's e-mail message lists and have now added a web-based search and retrieval tool so that evaluation staff and Passport to Knowledge staff can search for messages based on this coding or on any salient wor d in the message's title or body. The coding categorizes each message on two dimensions. Author-type refers to who posted the message originally – student, teacher, Passport to Knowledge staff, scientist, and so on. Subject-type codes each message into one or more functional areas that describe the purpose of the message such as request for help or student work sharing. A complete listing of the codes and an explanation of the general break out is in the next section. The database of coded messages i s at http://www.edc.org/ptkarch/mailsearch.html.

Finally we looked at science standards from a number of states in an effort to devise effective assessment criteria and rubrics with which to examine student work. While Passport to Knowledge has always referenced the National Science Education Standa rds from the National Research Council, we wanted to find more specific embodiments of these general principles to use as models for tangible performance statements at various grade levels in various areas of science education.

We ended up focusing on two states’ frameworks - Illinois’ Academic Standards for Science and the Instructional Goals and Objectives for West Virginia Schools. These two states have a strong commitment to standards-based educational reform and both ha ve been leaders in using educational technologies, specifically the Internet, as part of their NSF-supported State Systemic Initiative, in the case of Illinois, and the Rural Systemic Initiative, for West Virginia. Both states have put significant effort into working with teachers and other key stakeholders in writing their objectives in language that is understandable to diverse audiences and that can serve as the basis for assessing outcomes. We selected those content and performance objectives that w ere especially relevant to the two Passport to Knowledge modules used this year and we focused on the objectives related to the middle school grades. Appendix Four provides the list of objectives we gleaned and modified from these two sources.

This year we assessed the adequacy of these selected objectives for use with student work samples. The samples were gathered the Live from Mars Planet Explorer Toolkit on-line activity (see Appendix Five for an overview of this online collabora tive activity that spanned four months) and some of the summary projects created in many classrooms.

One basic task this year was to continue to provide an analysis of who used Live from Mars and Live from Antarctica2 as well as a look at how these modules were used. While we used a variety of methods to gather the data for this par t of the evaluation, a common factor uniting all of them is that the respondents were teachers. Therefore, this portion of our research is ultimately concerned with the impact of PtK on teachers, although we sometimes include data from other interested a dults.

Building on the positive response rates found in the evaluation’s first year, we expanded our use of electronic survey instruments. We sent the surveys out via e-mail or mounted them as forms in appropriate locations on the web sites. Responses were automatically processed thus removing a costly and time-consuming step that plagues large scale survey research. As a result, we able to send out surveys immediately following a live broadcast and ask questions specific to the program’s content and its r elationship to the other PtK resources and to classroom practices. We were able to send separate surveys to gather information and reactions to the web and print resources. This is a significant improvement over a once a year or one per module questionn aire where all of the questions are chunked into one instrument and it is administered long after some of the events have transpired.

The nature of these surveys also allowed us to begin to target specific users vs. the entire set of participants. Thus the PtK Advocates, a group comprised of some of the most experienced PtK users and often exemplary teachers in their own right, were broken out for questions specific to their important role within the project.

In addition to the surveys that were sent out following broadcasts or after the web site had been used, we added a voluntary registration process for all new users who wished to identify themselves as teachers. It was voluntary in part due to a desire by the PtK staff and the NASA staff not to force a mandatory registration on visitors to the web sites. Despite the option to opt out of the registration survey, we got good sample sizes and now can make some generalizations about the user base for the PtK modules.

To augment this electronic registration, we also included pre-paid postage postcards in all of the print teachers guides and kits that were sent out. These print forms had the same questions as the on-line forms in effort to reach a potential group of PtK participants who might be active with the videos and the print but were not on-line. While we got a good response rate to these postcards, only a few were from teachers without e-mail or web access. Most returned the postcards because they received it first before going on-line but would have been equally likely to complete the on-line version when they accessed one of the sites. There were no significant differences between the responses of on-line submissions and those choosing to mail in the po stcard.

This leads us to hypothesize that the vast majority of PtK teachers are using at least some of the on-line components of the project. The group that used the television broadcasts and the print materials but lacked e-mail access in the very early days of PtK (two years ago) has now gotten access of some kind. We feel fairly confident in stating that there is no longer a significant number of participants unable to use the on-line components.

Note that this does not mean that there aren’t any teachers out there who lack on-line connectivity – even with the dramatic increases in connectivity over the last two years, there are still plenty that face barriers in getting the basic level of acce ss required to use e-mail professionally. Rather, we have no evidence that there are significant numbers of teachers actively participating in this project who are either unable to use on-line tools or choose not to do so.

This has some implications for future resource allocations which we will discuss in more detail in the recommendations section. While PtK has always felt it is important to address equity issues and provide ways to accommodate some level of participat ion by the largest number of potential teachers, the project has been successful in getting existing participants to use on-line. New participants are coming to the project, in part at least, because it contains on-line aspects. The PtK WWW sites are an increasingly important channel for recruiting new participants. This is a fairly dramatic shift from the work on PtK’s pilot program where the on-air aspects served as a strong mechanism for publicizing and enticing teachers on-line. As demonstrated in the next section, the on-line services are increasingly the means for teachers to find out about the on-air components.