|
FAQs about
Flashlight Program l
FAQs about Flashlight Online 1.0 l
FAQs
about Flashlight Online 2.0
Frequently Asked Questions (FAQs)
about the Flashlight Program
1.
What is Flashlight? Is it
for-profit?
2. What does Flashlight do?
3.
Is Flashlight finished?
4.
Who are the people?
5.
Where is Flashlight?
6.
What are your
findings?
7. Are the items in your tool kit validated? Can I see them?
8.
What
are the "seven principles"? Where can I learn about the research
behind them?
9. Do you offer workshops?
10.
What does "activity centered evaluation" mean?
11.What is Flashlight Online? Do I need to
install software on our Web server in order to use it?
12.
Are there any limits or extra charges on large
numbers of authoring accounts, or surveys, or respondents for Flashlight
Online?
13.
If we leave the Network, do we lose
consulting time that we have not yet used? How much time can we carry
forward if we resubscribe?
14.
Why is it called
"Flashlight?"
15.
The TLT Group site license limits use
of its materials to the institution that subscribes. What is an
"institution?" If there is a main site and several branches or outreach
sites, does each site need its own license?
16.
Flashlight Online: if the survey asks respondents for a username, or
username/password combination, can they respond more than once?
17.
I have
a question about using Flashlight Online 1.0.
18.
How secure is
Flashlight Online?
19. Are there any limits on the amounts
of text users can enter in text boxes? Can I, as an author, set any
limits?
20.
Is
Flashlight Online 1.0 ADA-compliant? Can a blind person use a screen
reader to respond to a survey?
21. I know you can't change a Flashlight
Online survey once it has been "started." (made available to
respondents). But can you add respondents?
22. If an author sets a survey so that an ID
is required from respondents before they can answer, does that ID remain
associated with their responses?
23. When I used a Flashlight Online
demonstration account, it looked as though all other Flashlight Online
users could see my survey and data. Is that true?
24.
How would you summarize the
unique advantages of Flashlight Online in just one sentence? I want to
persuade others that we should subscribe.
25.
I'm using four
groups of a particular type of custom question (e.g., 5-way
block format questions) and I'd like to add more groups of
those questions. The system doesn't seem to let me do that.
Is there a way around the problem?
26. Can you delete a Flashlight Online 1.0 survey? How?
27.
When is Flashlight Online 1.0 being shut
down? What if I realize later I need a copy of one of my old
surveys or some of my old data that's still in the system?
Flashlight Online 2.0 Questions
1)
What is the
Flashlight Program in a nutshell? Is it a research project? A for-profit company?
Since 1992, the Flashlight Program has been helping educators and their institutions study and improve educational
uses of technology. More recently, our methods, tools, training and
consulting have been used for a wider variety of needs in the
scholarship of teaching and learning, student course evaluation, program
evaluation, accreditation support, and other needs. Since 1998,
Flashlight has been a program of the non-profit TLT Group.
Click
here for an interview that describes the development and work of the
Flashlight Program.
2)
What does
the Flashlight Program do?
The Program's goal is to help faculty members,
administrators and students ask the questions and gather the data they
need to guide and improve their own educational uses of technology. (This
article explains why the Program is pursuing this goal.)
Flashlight:
-
Creates tool
kits (to help you design your own studies) and standard tools, such as
Flashlight Online, accompanied by
designs, cases, related articles, and an annotated list of resources.
-
Helps technology
users become evaluators by providing attractive events and training workshops;
-
Provides active assistance (e.g., external evaluations; creating evaluation procedures for institutions;
aid in devising assessment strategies for
accreditation self-studies) and leadership opportunities
-
Works to help develop a culture of assessment, within institution
and internationally. That includes helping institutions share
information about their inquiries and work together on join studies,
helping
evaluators and researchers at your institution become consultants to 'novice institutions'
in your region; inviting Flashlight users at your institution to contribute to new
Flashlight tool kits, casebooks, etc.
3) Is
Flashlight finished?
New tool kits
and tools are continually being developed and released. Our most recent release:
The Flashlight Cost Analysis Handbook.
Coming soonest: new study packages on distance education in nursing and
faculty use of presentation software such as PowerPoint; a tool kit for
gathering data from faculty members; a study package for evaluating
support services for learners off-campus. We are also about to
introduce some new services, including active step-by-step consulting help
for you as you do a study and a multi-dimensional evaluation of a Web site
(issues of educational potential, diversity, and accessibility for the
disabled).
RETURN TO
TOP
4)
Who are the
principle people?
The Program's director is
Stephen C. Ehrmann, Ph.D., Vice President of the TLT Group. Associate
Director. Hannah Weldon of the TLT Group is the Project Manager. Gary Brown of Washington State
University was a pioneering user of Flashlight and has led the team
developed and operates Flashlight Online. Cheryl Bielema, Phyllis
Dawkins, Patricia Derbyshire, Tom
Henderson, Barbara Krauth, Tom Laughner and Frank Parker are among our
more frequently used consultants.
5)
Where is Flashlight?
It's headquartered at the TLT Group in Takoma Park, MD
in the United States but Flashlight is an international project and
development occurs around the world. Hannah Weldon works from her
home office in Austin, TX and handles the Flashlight Online 2.0 project.
6)
What does
the national Flashlight study show?
There is no national
Flashlight study although Flashlight methods and items could be used for such a study.
We provide tools, tool kits, training and consulting to help users do
the studies they need. User data is private; not even we
know what they (or you) may find. We do try to help users publish their findings.
In near future, we will begin to offer study packages; these will
include standardized, online surveys that will enable interested users
to share data; we will also develop national reports aggregating data
from interested users. The first study packages will deal with uses of
the Web in nursing education and uses of Web Course Management Systems
(e.g., WebCT, Blackboard).
RETURN TO
TOP
7)
Are the
questions normed and validated? Can I see them before our institution buys a site license?
Our focus has been on
research and content validity. The issues selected for study are those raised by research
as the most promising potential benefits and the most worrisome potential problems. Most
items in the Flashlight Current Student Inventory
(the items tapped by Flashlight Online)
have been subjected to trial use and
then to follow-up focus groups to make sure that the wording is clear to many types of
respondents.
The CSI is an item bank. As such it can't have true validity and
reliability since both depend so much on the order of the questions in a
static instrument. However, the items in the CSI have content validity
because a) many of them are based on the seven principles of good
practice in higher education and b) different versions were reviewed by
experts from five pilot institutions.
We then tested items for face validity by having more than 40
different surveys created from the item bank at these 5 pilot
institutions. Approximately 2,000 respondents completed these surveys.
We then organized focus groups with the respondents, and with the
faculty and administrators responsible for interpreting the results. All
of the teaching and learning items were able to be tested in this way
(not all of the demographic items were used, but most of these are of a
standard format that have been well-tested elsewhere).
Since the CSI was released, we have created a number of standard
templates that are based on the item bank. One of these is the
Evaluating Educational Uses of the Web in Nursing (EEUWIN benchmarking
survey). It was pilot tested at 3 Nursing programs. This survey has been
tested for validity and for internal reliability. Over three years it
has demonstrated a consistent Cronbach's alpha of .85-.90. We are
continuing to work on other templates that test other areas of
subscriber concern and will make those reliabilities available when they
are generated.
One way to "see before buy" for institutions that are seriously
interested is to request a two week guest account for Flashlight Online
(which includes access to database of CSI items as well as to survey
templates on different topics). Send e-mail to
flashlight@tltgroup.org to
request an account for your institution. Only one account per
institution, please.
8)
What
are the 'seven principles'? Where can I learn about the research behind
them?
In
1987 Arthur Chickering and Zelda Gamson answered the question,
"According to educational research, what practices
tend to produce better educational outcomes?" by listing
"seven principles of good practice in undergraduate
education." They pointed to the following characteristics of
teaching-learning as being especially valuable for improving learning
outcomes (i.e., if a course or institution increases what it does in
these areas, learning outcomes of many sorts are likely to improve):
-
Encourages
student-faculty contact.
-
Encourages
cooperation among students.
-
Encourages
active learning.
-
Gives
prompt feedback.
-
Emphasizes
time on task.
-
Communicates
high expectations.
-
Respects
diverse talents and ways of learning.
The
Web usually has a number of sites that provide more detail on each of
the seven principles but the URLs seems change yearly, sometimes
monthly. So instead of recording some of those current URLs here and
frustrating you when the links break, we suggest using a search engine
and doing a search with terms such as "seven principles"
"Gamson" and "Chickering." For a book on the
education research behind the seven principles, Zelda Gamson recommends Applying
The Seven Principles For Good Practice In Undergraduate Education: New
Directions in Teaching and Learning, #47, published in 1991 by
Jossey-Bass; it's now out of print but it may be in your library's
collection.
The chapter by Mary Deane Sorcinelli reviews the research literature.
Many
of the items in the Flashlight Current Student Inventory (and thus also
in Flashlight Online) ask students about how often these practices
occur, how often technology is used to carry them out, and how
appropriate available technology is for carrying them out.
For much more on the seven principles and uses of technology to
advance them, see http://www.tltgroup.org/seven/home.htm
RETURN TO
TOP
9)
What
happens at the workshops? How often and where are they offered? How much do they cost?
The
workshops are tailored to your needs. A talk (not a workshop) might run an hour or so and
provide a briefing on Flashlight's capability. A half day workshop usually focuses on the
issues and difficulties of planning evaluation studies, including a brief introduction to
Flashlight. Advanced workshops (a day or more) provide intensive training in Flashlight
use, and are especially useful to help teams devise shared evaluation strategies; advanced
workshops are available face-to-face or on-line. Please contact
Flashlight@tltgroup.org
for more information.
10) I've heard
people call Flashlight an activity-centered
approach to evaluation. What does that
mean?
The
purpose of evaluation is to help people make decisions. Knowing that technology was
used and that outcomes improved (or stayed the same, or got worse) doesn't help anyone
decide anything. To make improvements you need to know how the technology was used (the
activity that used it). Focusing your attention not just on technology and on outcomes but
also on activities yields some surprising dividends. For example, suppose your
institution invested in e-mail in part so that students would learn better. How is e-mail
supposed to help? Let's suppose that an important activity is students doing homework
together. The focus on the activity raises some important questions for your study.
For example, do students have access to e-mail from the places they do homework? Do
faculty encourage, or discourage, students working together on homework. Do students
believe collaboration of this sort is helpful? a waste of time? cheating? The
answers to these kinds of questions help determine how helpful e-mail will be for such
collaboration and thus whether the e-mail (indirectly) helps students learn the material.
11) What is
Flashlight Online
and how can I get it?
Flashlight
Online is a Web-based system that lets you combine your own survey questions with ours; it
provides access to a database of our items. In addition you can also administer your
surveys and collect data using the Web. The system operates from servers at Washington
State University and (soon) at mirror sites so you do not need to buy equipment, install
software, or provide technical administration. Flashlight Online is normally
available only through institutional site license and only to institutions with site
licenses for Flashlight tools. If your institution has paid for Flashlight Online, you can
get a free account. Institutions investigating Flashlight Online can receive a free
demonstration account for 2-3 weeks.
12) Are
there any limits or extra charges on large numbers of authoring accounts,
or surveys, or respondents for Flashlight Online?
No. An institution with a subscription to the
Comprehensive Collection or to the TLT/Flashlight Network can give (if it
wants) authoring accounts to all its faculty, staff and students. Each
author can create an unlimited number of surveys, and each survey can have
an unlimited number of respondents. The only limitations are that
authoring accounts can only be given to members of the institution (its
own faculty, staff and students) and the surveys must be for institutional
purposes (e.g., faculty or staff can't use the institution's site license
to do surveys for private consulting unconnected with the institution)
without a separate license from The TLT Group.
RETURN TO
TOP
13)
If our institution joins the Network, we do not use all our consulting
time, and a year later we do not resubscribe, can we still use the
remaining consulting time?
If
your institution does not resubscribe, it loses any consulting time it did
not use while a member. If the institution renews its subscription
to the Network, it can carry forward a maximum of two days of unused
consulting time.
14)
Why is the program called "Flashlight?"
Three reasons:
a) Someone says to you, "Let's evaluate
this educational program." Where do you pay attention? There
are an almost infinite number of possible studies that could be done of
even one assignment. A successful study almost always focuses on
one particular question (while remaining open to the
unexpected).
b) Why "Flashlight" and not
"Floodlight" or "Spotlight?" These studies,
especially those that can be done by faculty, staff, departments or
institutions about their own operations, are not methodologically
powerful. So focus and humility are required: a pocket flashlight is a
more metaphor than a gigantic spotlight.
c) Imagine yourself trying to teach, or run an
innovative program, in the dark. Someone hands you a flashlight. How
would you use it? You'd think first about the direction you'd like to
go, and shine it in that direction a little way ahead to see what you're
getting into. The assessment equivalent are studies of need, of
similar practices at other institutions, and some focused environmental
scanning. If you're smart you'll also then pivot and point the
flashlight immediately behind you: how quickly have you been moving
forward? what problems have you encountered? how might you respond to
them the next time? That's the image we have in mind when we say,
"Flashlight."
RETURN TO
TOP
15) The TLT Group site license limits use of Flashlight Online and
TLT/Flashlight materials to the institution that subscribes. What is an
"institution?" If there is a main site and several branches or outreach
sites, does each site need its own license?
The US Department of Education has a data bank about
higher education called IPEDS, and The TLT Group uses that reference for
US institutions. If two or sites share the same UNITID in the IPEDS
system, we consider them to be different sites of the same institution and
only one subscription is required. If, on the other hand, if they each
have a different UNITID then we consider them different institutions. We
are happy to negotiate a single discounted fee to cover all these related
institutions, however. For information about all this, and for non-US
institutions, please send your inquiry to
online@tltgroup.org.
16. Flashlight Online: if the survey asks respondents for a username, or
username/password combination, can each person respond more than once?
No, the system permits only one response
for each username, or username/password combination.
17. I have a
question about using Flashlight Online
Click here to see our guide to using the system.
It contains answers to dozens of commonly asked questions. We suggest that
users print it and have it by their PCs when using Flashlight Online.
18. How secure is Flashlight
Online?
No one from another
institution can see your surveys unless you give them authorization to be
in a "member group" that is under your local administrative group.
Only the author of a
survey (the person who started the survey), or someone else that the
author explicitly designates, can download the raw data at any time.
Recently Charles Ansorge of the University
of Nebraska wrote, "I submitted an IRB (Institutional Review Board) proposal
using a Flashlight Survey as the tool and they want to know whether the data
will be sent to a secure server, whether the data will be encrypted while
in transit, and will it collect IP addresses? I am unclear how much of this
applies to how Flashlight works. For my study, the survey is anonymous - so
no id number is given to the participants."
We replied:
-
Secure server?— Yes,
we take every measure to protect the security of the data on our server.
It is behind a secure firewall and all security updates are applied to
the server in a timely fashion. It is also physically secured in the
server room which includes video surveillance.
-
Encrypted in transit?
— No, we do not encrypt data passed in transit.
-
Collect IP numbers? —
While no IP numbers are stored with the survey responses we do collect
IP numbers in the server log. The server log is replaced on a rotation
schedule and there is no link that can definitely be made between any
respondent and the IP number they took the survey from. If a survey has
respondent IDs this is also the case. Those IDs are in no way tied to
the server log of IP numbers.
19. Are there any
limits on the amount of text that users can enter in text boxes? Are
there any limits that I, as an author, can set?
Flashlight Online does not set any limits, nor does it
offer authors the ability to set limits on amount of text entered. However
the World Wide Web consortium has a limit of 100K for files of this type.
These are text files (ascii), not word processing, so the chances that your
respondents answers will total more than 100K are slim. We've not heard of
any such cases. If you were reaching such a point, you might have trouble
with html customization.
20. Is Flashlight Online
1.0 ADA-compliant? Can a blind person use a screen reader to respond to a
survey?
Flashlight Online 1.0 was not designed with the Americans
with Disabilities Act (ADA) in mind. Nonetheless, we're told, a screen
reader called JAWS usually works well, assuming that html customization
doesn't get in the way.
21. I know
you can't change a Flashlight Online survey once it has been "started."
(made available to respondents). But can you add respondents?
If a Flashlight Online survey has already been set up to
require a respondent ID, or an ID plus a password, then people can be added
to this list even after the survey has been "started."
22. If an author sets a
survey so that an ID is required from respondents before they can answer,
does that ID remain associated with their responses?
Yes, when
IDs are used, the author can download the data and see how each
respondent answers. So, if the IDs are used to import other data about
the respondent (e.g., gender, test scores, grades, place of residence),
the survey data can be analyzed with those other data (how does place of
residence correlate with student responses regarding time on task). This
ordinarily means that the responses are not anonymous, unless the author
uses some extra steps to assure personal anonymity (e.g., using IDs
created especially for the study and held by a third party, who is the
only one who knows which ID belongs to which respondent).
23. When I used a
demonstration account, it looked as though all other Flashlight Online
users could see my survey and data. Is that true?
No. Surveys can be seen (if you wish) by other authors
who have membership in the same 'member group' (workspace) where you
created and stored your survey. Lots of people have been given
demonstration accounts (which usually last three weeks); if you chose to
make your survey visible to others in that member group, then they can
see your survey and the summary analysis of any data you collected. Only
the author, however, has the power to download the data. And any author
has the option, at any time, to make his or her survey and data
invisible even to other members of the same member group.
24. How
would you summarize the unique advantages of Flashlight Online in just
one sentence? I want to persuade others that we should subscribe.
Flashlight Online is a simple survey system that helps
authors by giving them item banks, model surveys, and the ability to
adapt surveys written by authors at 100 other institutions of higher
education.
25.
I'm using four groups of a particular type of custom question (e.g.,
6-way block format questions) and I'd like to add more groups of those
questions. The system doesn't seem to let me do that. Is there a way
around the problem? Suppose you're using groups of
block format questions, each with 6 answer columns. Use the four groups
of 10 questions that Flashlight Online allows. For additional questions,
use seven-way scale items; leave the right hand column blank. Once
you're completely finished adding questions, click on customize html.
[Customizing html allows you to do many things, from adding your logo,
to more easily reordering questions, to adding video prompts for your
questions. We use a free
web editing program called Nvu to do our own customizing. To see the
guide on how to customize the html of your survey,
click here.] Delete the seventh column. And you're ready to
go.
26.
Can you delete a Flashlight Online survey? How?
Surveys that have not been 'started' can be
deleted. The way you do it is counterintuitive for
someone accustomed to deleting other kinds of
computer files. In other kinds of files, the
document must be closed in order to be deleted. In
Flashlight Online 1, the survey must be open in
order to be deleted, because the delete button is
inside the survey, It's the fourth black button
from the left, between "copy" and "survey
properties." Once deleted, surveys cannot be
undeleted. Any survey in a member group that is
not currently 'started' (open to response) can be
archived by any member of that group. On the "Pick a
Survey" page, click the archive box to the left of
the survey(s), go to the bottom of your list of
surveys, and click the "Click to archive" button.
To get a survey back from archives, click the
"View Archives" button at the top of the "Pick a
Survey" page. You'll see all the archived
surveys in that member group. 27. When is
Flashlight Online 1.0 being shut down? What if I realize
later I need a copy of one of my old surveys or some of
my old data that's still in the system?
The Flashlight Online 1.0 server
will be shut down and the data destroyed after March
2010. Once the server is shut down it will be
impossible to retrieve any of the old surveys
or data. Users must make copies of any surveys they want
to save, and download any data they might need in the
future before March 31, 2010. If an author needs
help, he or she should contact their local
administrator. if local administrators need help they
should contact the TLT Group. Please do not wait until
March. We're a tiny non-profit and our ability to
provide help is quite limited. To see a guide
for leaving Flashlight Online 1.0 behind, and moving
into 2.0,
click here.
RETURN TO
TOP
What do you recommend for users who do want to continue to use their
current surveys under the new version?
Surveys will need to be entered 'by hand'
(copy and paste the question, and then copy and paste the block of answer
options) into
Flashlight Online 2.0. You cannot upload old Flashlight Online 1.0
data into Flashlight Online 2.0.
Will the current database of questions continue to be available?
(obviously that really means the same questions in a database accessible
to 2.0)
We've begun importing the Flashlight Current
Student Inventory into Flashlight Online 2.0, and updating
it as we go; the Flashlight Faculty Inventory will be next. If you or your
colleagues use the Flashlight Current Student Inventory, and want it to
be implemented in Flashlight Online 2.0 quickly, please let us know. It
will help us set development priorities and schedules.
RETURN TO
TOP
What features will
Flashlight Online 2.0 have?
Click here
for a list of features and a
recently recorded demo
(posted August 20, 2007) Click here to see a
comparison with other systems.
RETURN TO
TOP
I've heard Flashlight Online called a collaborative tool. Can it be used
to help survey authors work together? even if they're at different
institutions?
Flashlight Online runs on
computers at Washington State University. Each subscriber institution can create
accounts for authors; each author gets a unique username and password.
Authoring is done in "member groups" (shared workspaces). Each author
can be a member of any number of such member groups, including member
groups created by other Flashlight Online institutions. No password is
required to enter a group once the author has been given membership; the
group's name shows up on the "Select Groups" screen and the author just
has to click its name to enter the group. Once in the group the author
can see any survey created by any other group author (if the author has
given permission for the group to see it). It's also relatively easy to
share access to data with other group members, to copy other group
members' surveys and rewrite them, to collaborate in writing a survey
with other group members, and so on.
Each person needs to have only
one account on it and only one password (even if they're doing multiple
kinds of surveys and even if some of their work is done in a member group
created by a different Flashlight Online institution)
Flashlight Online 2.0's
features should make it an even more powerful collaborative tool (e.g.,
creating and managing item banks within and across institutions).
Will Flashlight Online 2.0 be available as
open source?
Yes, that's our plan.
The code will not be available until we're confident the system runs
well as Flashlight Online 2.0. We have not decided whether or how much
to charge for the code, or for support services. We would like to make a
subscription service available for open source users so they can have
some access to the growing library of surveys and item banks created by
Flashlight Online 2.0 users and, perhaps, contribute as well. These are
hopes at the moment - we do not (as of Sept. 2006) have operational
plans.
Will Flashlight Online 2.0 'plug and play'
with course management systems? Will it be standards-compliant?
Flashlight Online 2.0
will follow IMS standards. We are quite interested in working with
vendors to make it as easy as possible for Flashlight Online 2.0 to work
smoothly with enterprise systems (course management systems, registrars
systems). No plans are in place yet. If you are a subscriber and
would like to help us develop such linkages with vendors you use, please
let us know. Contact Steve Ehrmann (ehrmann@tltgroup.org)
or Hannah Weldon (weldon@tltgroup.org).
What are survey meta-data?
Authors and
administrators will be able to 'tag' information with descriptors that
can be used later to search for, and find, that information. In
Flashlight 1.0 many items in the Flashlight Current Student Inventory
have such tags, which is how a user can call up dozens of items that
relate to 'collaborative learning' even though none of those items
include the word 'collaborative.'
Flashlight Online 2.0
will allow local administrators and users to tag items, surveys, and
item banks. Meta-data are essential for administering complex course
evaluation systems - information about particular authors, courses,
surveys, items, item banks, and the like.
Will Flashlight Online 2.0 offer rank ordering
questions? Yes. A drop down menu
assures that respondents can't pick a value that is out of the range set
by the author. As with other Flashlight Online 2.0 questions, there is no
limit on the number of options the author can offer.
Is the Current
Student Inventory (CSI) available in Flashlight Online
2.0? Yes. In
fact CSI 2.0 is gradually appearing in Flashlight Online
2.0. When we considered simply copying CSI 1.0
into Flashlight Online 2.0, we realized we should update
the CSI to a) take advantage of matrix survey
capabilities, and b) updat the content. For more,
click here.
When I set times
for Flashlight Online 2.0 surveys to turn on or off,
what time/calendar is used?
Flashlight Online 2.0 runs on Pacific
Time in the US. We have users around the world To
see what day and time it is currently (for Flashlight
Online), look at the lower left hand corner of your
browser.
RETURN TO
TOP
If you've read
this and have other questions, please send them to
Flashlight@tltgroup.org.
|
