Participate in discussions relevant to informal science research, learning and evaluation.

In the coming month we will be releasing a forum section on the new InformalScience.org site.
Please give us your thoughts about what you would like to see this new discussion section evolve into and support.

Tags: New InformalScience.org

Join a discussion with Randi Korn about a broader role for evaluation in the activities of museums, the opportunities and challenges of the new NSF ISE Framework for Measuring Impacts of Informal Learning Experiences, and supporting the professional development of evaluators.

Tags: Evaluation, Impacts, Evaluation Framework, Professional Development

Join this month's conversation with Paul Martin, the director of exhibits at the Science Museum of Minnesota in an online discussion forum. Post your remarks about the interview, or raise your own topic of interest.

Tags: exhibit design, science musuem,

Join us for an open discussion about AERA 2008.
Not all of us were able to attend or see all the sessions, so let the community know about interesting talks and thought-provoking discussions you heard, especially as they relate to research about learning in informal environments.

Tags: AERA2008, informal learning

Join this month's conversation with Angela Wenger in an online discussion forum. Post your remarks about the interview, or raise your own topic of interest.

Tags: community programs, diversity, aquariums

About the new RFP, I noticed logistical, structural and content changes.

The logistical changes will mainly impact the way the public and program officers interact, probably to make the process more consistent across all the solicitations in DRL. Changes include:

1. The limitations on numbers of proposals per PI and institution are lifted (Happily for many).
2. A letter of intent rather than a preliminary proposal (and panel) is required.
3. In both in the solicitation and on the website central numbers and emails are provided rather than a list of program officers. The message at the phone number refers to the DRL Life Long Learning Cluster, evidently the cluster ISE belongs to.

About the new central phone and email message system, my guess is that a person with an inquiry would be directed to the right and available program officer so there isn't as much phone tag and email-lag. It makes sense internally to provide the best response time to the public, and might allow the group to better incorporate new program officers in the ISE effort but the drawback is it makes the group faceless (or rather nameless!)

4. The structural changes in the Division of Research on Learning in Formal and Informal Settings (DRL) are introduced in a section devoted to how ISE (and other programs) fit within the goals of the new division structure. ISE seems to fit neatly into DRL with out a lot of change in its focus. ISE's emphases within DRL’s cycle of innovation and learning are bolded here: Synthesize, Study, Design, Implement, Evaluate. I'm not sure why evaluate is not a priority as the new framework for evaluating ISE impacts has been released, but the narrative does indicate that all program care about all five elements of the model.

5. One new ISE deliverable that goes beyond just logistical is the new learning technology deliverable defined in this solicitation. This deliverable needs to meaningfully combine interactive technologies, new approaches to learning experiences, STEM content, and research-based understanding of and strategies for reaching the targeted audience.

6. The rfp notes an additional resource to proposal planners. ISE's resource center CAISE, Center for the Advancement of Informal Science Education (a partnership between ASTC, Oregan State University, UPCLOSE, and VSA) has been working on effort to initiate:
-a community web site and monthly newsletter with highlights of NSF-funded projects and relevant research
-conferences and online workshops for principal investigators of NSF-funded projects
-inquiry groups to study and report on critical and emerging issues fellowships and mentoring programs for those new to the field

This new website- www.insci.org is listed in the rfp and links to resources for potential PIs around the new project evaluation system, a portfolio review, the ISE rfp, and standard NSF proposal preparation materials.

informalscience.org continues to house these type of materials as well and PIs are still required to submit the Summative Evaluation of the project for posting to the web site www.informalscience.org

7. And finally, there is also a line in review process section of the rfp highlighting the ability of PIs to recommend reviewers. This is worth considering, as you know your field the best and the more qualified reviewer names you can send to program officers (or the ISE program in general through the new centralized email! the richer the review process overall. The best way to do it is send the potential reviewer's resume and contact information.

Join this month's conversation with Mary Marcussen in an online discussion forum. Post your remarks about the interview, or raise your own topic of interest.

Tags: Innovation, NSF

Join this month's conversation with John Falk in an online discussion forum. Post your remarks about the interview, or raise your own topic of interest.

Assessing Learning in Informal Settings: Recent Pressures for Accountability

Roundtable Discussion Table 4
Internal Vs. External Evaluators
Summary prepared by Minda Borun



What are the implications for the relationships between external and internal evaluators in museums/zoos/aquariums/gardens? Is a separate external evaluator for summative evaluation more objective than one who has conducted front-end and formative evaluation for the same project?

Introduction
In recent years NSF/ISE has required an external evaluator for summative evaluation of their grant-funded projects. This must be a different person from the internal evaluator who may have conducted front-end and formative evaluation. The implication is that the evaluator who has been involved in project development is insufficiently detached to objectively assess the final impact of the completed project.

As an evaluator who works both inside and outside of museums, I have encountered some difficulties with this arrangement:

1. Many ISE projects are programs. In contrast to exhibits, programs occur over time and come to an end. It is difficult for an external evaluator to assess a completed program.

2. Evaluation studies often compare the state of a situation at the outset of a project and compare it to that at the end in order to see how things have changed. It’s useful for an evaluator to be involved in all phases of a project.

3. If evaluation studies have a scientific basis, shouldn’t they be as objective as any other science investigation? Would federal agencies send an outside microbiologist into a laboratory to conduct the final experiments?

4. As the field of Visitor Studies has grown, more and more museums have established an internal evaluator position. At present there are more than 25 full time internal evaluators in VSA. If the new assessment model for federal programs requires external evaluation, what will happen to these internal evaluators and to the field of Visitor Studies?


The round table discussion saw some pros and cons of using external evaluators for summative evaluations and arrived at some suggested solutions:

Pros:

1. It can be helpful for the internal person to work with someone who has a new perspective on a project or a new set of skills.

2. The external evaluator is there for quality assurance/quality control (QA/QC).


Cons:
1. The internal person has been through brainstorming and prototyping and knows the objectives and key points of the exhibit. The external person should be involved from day one. It’s often difficult because the external person isn’t onsite long enough to really get into the process.

2. Sometimes external evaluators just use the internal evaluator’s instruments, data collectors, and experimental design and put their name on the report.

3. Some projects have research issues that are embedded in exhibit or program development. It’s frustrating not to be able to do the summative to see how it turns out.

4. We don’t want all the internal evaluators to become consultants. We would lose the front-end and formative, which is arguably more important than the summative. If you produce a bad program and then do a summative evaluation and say it’s bad, who cares? But, the funders care. They are switching their emphasis to summative.

5. An internal evaluator can build a comparative database of results across exhibits for tracking, timing, sweeps, and exit interviews. Then you can see standards emerge. For example, in my museum, holding power ranges from 11 minutes to 27 minutes, so we know that 20 minutes or more spent in a 5000 square foot exhibition is quite good. The exit interviews ask a fairly standard set of questions. If I stop doing the summative I can’t continue to contribute to this table. If an external person uses this model, it could contribute to the database, but most external evaluators have their own approach.

6. What is frustrating is to hear a program officer say external evaluators are more objective. Nobody is objective. The point is to put your biases on the table and make sure they are integrated into the report.



Solutions:

1. We need more external evaluators. Those available don’t have sufficient time to spend on any one project. We need to grow our profession. We have a responsibility to the field to take in students and new people.

2. We need clear guidelines and contracts for external evaluators so that the internal and external evaluator have a plan for working together and summative evaluation is a collaboration between them. If we establish criteria for quality assurance/ quality control then we can have internal evaluators doing summatives.

3. We also need a logic model for the field or at least some standardization in instruments and approaches across sites so we can build a database. The idea of standardization might feel like a constraint, but some basic measures are “necessary but not sufficient”-- have some standardized measures and then add on additional questions. When you go to the doctor, they measure your height, weight, blood pressure, and temperature; but the diagnosis and treatment go in different directions depending on your symptoms.

4. The issue is to make sure that the push for standardization does not expand beyond some fundamental measures that we all agree are critical to any exhibit or program. What’s wrong with the federal model is that it purports to be the whole thing.



Tags: Internal evaluation, external evaluation,

Can we talk? A few months ago several of us asked ourselves that question. As people who do research and evaluation in informal settings, we had been feeling some pressure. I had certainly noticed some things affecting my own work—requirements to submit to institutional review boards, expectations for pre/post test designs, and questions about competencies need to do visitor studies from several associations. We decided to offer a roundtable session at the Visitor Studies Association Conference in Ottawa to start a discussion. Ellen Giusti and Minda Borun are chairing the session. Sue Allen will present a summary of recent events impacting these issues. Everyone, including Christine Reich and I will chair roundtable discussions. If you are attending, come to the session at 9:00 a.m. on Saturday morning. But, we want a wider discussion and consideration of these issues. They are important. Some require some appropriate response from various stakeholders. If you produce or use visitor studies such as those in the informalscience.org data base—YOU are a stakeholder. Following the conference, we are going to summarize the discussion for the Visitor Studies Association e-newsletter. We will also post brief summaries on this blog and ask you to extend the discussion. Questions to be addressed include:
* What problems do we anticipate from the increasing demands for quantitative accountability? Is the push for quantitative accountability likely to dissipate or increase with time? What can we learn from other fields on this topic?
* How can we ensure that qualitative and naturalistic methods are valued by funders? What are some ways to incorporate qualitative and quantitative approaches? Is testing with randomized control trials and quasi-experimental designs the only way to demonstrate accountability?
* Should we develop a field-wide logic model, and if so, how should we go about it? Should outcomes shape formative evaluation, or only summative? Should methods or instruments be standardized to some degree to support aggregation?
* What are the implications for relationships between external and internal evaluators in museums/zoos/aquariums/gardens? Is a separate external evaluator for summative evaluation more objective than one who has conducted front-end and formative evaluation for the same project?
* What is the impact of Institutional Review Board requirements on the design and implementation of studies in informal settings? How can the field support appropriate and consistent interpretations of human subjects regulations, particularly when this involves children?
• How can VSA be an active participant in determining accountability standards and methods? What kinds of infrastructure for further conversations and work shall we create?

This is how we framed the issues. We need to hear from you. Which of these issues is impacting your work? Which is highest priority for you? Why? Are there related issues we need to get out on the table? I look forward to hearing from you. Let’s talk!

Carey Tisdal
Tisdal Consulting
St. Louis


Tags: VSA Conference Ottawa, Roundtable Discussion, Research and Evaluation Issues

I am writing a dissertation on family learning in museum. What we were trying to do was to increase interaction between family members and subsequently increase learning related behaviour. I am trying to find information about labels and information pannels for kids, but cannot find anything. If somebody has come accross literature or has some experience and tips, that would be great...

Tags: family learning, museum's learning