|
Who Was in the Sample?
We began by attempting to develop a mailing list that
included as many of the informal science institutions
(ISIs) that currently exist in the United States as
we could possibly locate. Because no overall database
of informal science institutions exists, we relied on
the following sources to develop our mailing list:
- American Association of Botanical
Gardens and Arboreta (AABGA)
- American Zoo and Aquarium Association
(AZA)
- Association of Children’s Museums
(ACM)
- Association of Science-Technology
Centers (ASTC)
- Institutions classified as “science”
in the 2004 Official Museum Directory, published by
the American Association of Museums (AAM)
- ISIs known to the research team through
personal communications
Our final mailing list contained 2,597 institutions that
were located throughout the United States. As the majority
of these institutions are affiliated in some fashion with
AAM or another professional ISI-based organization, the
mailing list was probably more likely to include larger
institutions and less likely to include smaller institutions
than the ISI field as a whole. However, our final data
set is comparable to that used in other surveys of the
museum and ISI field (IRA 1996; ASTC 2002; AAM 2003).
We mailed the 2,597 institutions a survey that covered
the following topics:
- Background information about the institutions
such as ISI type, budget size and staff size.
- Information about the institution’s
relationships with schools and districts, including
the number of schools/districts served, the characteristics
of served schools/districts (e.g., urban vs. rural,
grade level, number of underserved students) and the
funding/support behind the programs.
- Information about the programs themselves,
including who the programs serve, the level of demand
for the programs, and methods of evaluation for the
programs.
- Institutions could respond to the
survey through the mail or through a secure Web site.
The vast majority (72 percent) chose to respond through
the mail.
The survey included a table describing thirteen different
types of programs that ISIs could offer. The last third
of the survey asked specific follow-up questions about
one particular program from this table. Thirteen versions
of the survey were developed, each with a different program
in the first position of the table and the remaining twelve
programs in a different random order. Respondents were
asked to answer the final questions in terms of the first
program from the table that was offered by their institution.
Each institution was randomly assigned to one of the thirteen
versions at the time of the first mailing, and all subsequent
mailings and communication reflected that assignment.
(See the complete survey (PDF
file) instrument.)
In addition to the first mailing, a preliminary letter
was mailed out that introduced the survey and told respondents
to expect it, and two follow-up mailings were sent to
remind people of the survey and increase the response
rate.
We received completed surveys from 514 institutions, for
an overall response rate of 20 percent. After the removal
of 39 institutions whose responses indicated they were
not informal science institutions (see Characteristics
of Non-ISI Institutions), we were left with a final
sample of 475 ISIs. These 475 responding ISIs included
aquaria, arboreta, botanical gardens, children’s
museums, natural history museums, nature centers, planetaria,
science centers and zoos, as well as institutions that
classified themselves as “other” rather than
one of those nine categories. (The percentages of each
type of institution in our final sample are shown in Figure
6.)
There were 106 institutions in our sample who identified
themselves as “other” rather than one of the
nine main categories of ISI (aquarium, arboretum, etc.)
in our survey, which is evidence that we were successful
in reaching a diverse set of institutions within the informal
science education field. The most common descriptions
these institutions gave of themselves were anthropology
museum, aviation museum, general museum, historical site/history
museum, and national park; some institutions also identified
themselves as having very specialized disciplines (for
example, railroad museum, health museum, oil museum).
Is the Sample Representative of the Population?
One important question that always arises for every large,
voluntary survey is whether those who respond to the survey
are typical examples of those who were sent the survey.
In survey terminology, we want to know if the sample is
representative of the population, or if there was a self-selection
bias where institutions that responded were somehow different
from institutions that did not respond.
Because there is no existing definitive list of all institutions
of informal science learning, it is difficult to answer
this question with certainty. However, a recently conducted
survey by the American Association of Museums (AAM 2003)
provides some useful estimates for comparison. They estimated
the number of institutions for specific disciplines based
on two methods: 1) by “assuming that the percentage
of museums of this type responding to the survey is their
actual percentage in the United States, and multiplying
this by our estimate of 16,000 museums to arrive at a
total for this discipline” (AAM 2003, page 24);
and 2) through an estimate by the discipline’s associated
professional organization, where such an organization
exists.
Table 1 shows the two estimates
by AAM for each of the types of institutions targeted
in the current survey, as well as a new estimate using
the same percentage method with the current survey and
assuming a baseline estimate of 2600 ISIs in the United
States. (Because we have no way of estimating how many
ISIs were not part of our mailing population, we chose
a number close to our mailing population of 2597 for the
illustrative purposes of this table.) Note that AAM did
not include planetaria or science-oriented “other”
institution types as specific categories in their survey
estimates, and that they had no corresponding “professional
association” estimate for nature centers.
The table also shows the number of surveys returned for
each type of institution, and the range of the estimated
response rates: how many were returned for each type divided
by the lowest and highest estimates by all three methods
for the number of institutions of that type.
As can be seen in the table, estimates based on CILS data
of the prevalence of different types of ISIs in the nation
are relatively consistent with estimates based on AAM’s
data. |
