|
Summary
As a federally funded AmeriCorps program with more than 270
tutors across the state (350 for the 2007-08 program year)
and just 16 staff members available to manage the program,
America Reads – Mississippi (ARM) is always seeking improved
and more efficient ways to:
·
Get tutors
the ongoing training and guidance they need to address their
individual goals and challenges with each student.
·
Facilitate
the development and ongoing growth of a learning community
among current and past tutors in the program, regardless of
their existing relationships with one another.
·
Build and
access institutional memory across program years of which
tutoring techniques and strategies work at each of its 59
tutoring sites (80 sites for Program Year 2007-08).
·
Know
whether its program is on target with tutor training and
programmatic goals at any moment so that if it is not,
regional and central office staff can make immediate changes
to get back on target before it's too late.
·
Decrease
the time and money staff spends on tutor monitoring, data
collection and reporting, and use that freed up time and
money to directly enhance tutor competencies and other
aspects of the program.
·
Create well
written reports that carry significant weight with reliable
and valid data for funding sustainability.
·
Spend only
a minimal amount of time training tutors and staff members
to use any adopted process.
This case
study documents the reasons ARM decided to begin using the
America Learns National Performance Measurement & Support
Network in August 2004 to accomplish the above goals and to
ultimately provide more valuable services to Mississippi
children. The document then tracks the returns of ARM’s
investment in the Network over the past three years. The
study is organized into the following sections:
-
ARM
Program Overview & Goals Leading to Network Adoption
-
Strategy, Adoption &
Implementation
-
The Research Driving
the Network
- Managing and Measuring the Network
- Lessons Learned
- - - - - - - - - - - - -
- - - - - - - - - - - - - - -
Section 1: ARM Program Overview &
Goals Leading to Network Adoption
About
America
Reads - Mississippi
Since 1998, ARM has been working to simultaneously address
the massive inequities in Mississippi
public schools, the major teacher shortage in its state, and
the enormous group of Mississippi adults seeking gainful
employment[i].
By hiring and training low income (or zero income)
individuals to teach the state’s most struggling students to
read, ARM not only gives Mississippi children an improved
chance at academic and life success, but also reduces the
demands on public welfare assistance by employing adults for
up to two years and motivating the best performers to
further their own education and become the state’s next
cohort of top-notch teachers. Since the program’s
inception, 75% percent of ARM’s 1,910 AmeriCorps members
(its tutors) have reported that they intend to become
certified teachers. AmeriCorps, the federal program funding
ARM, provides tutors with $4,725 for each year of service
that can be used to fund higher education and pay back
student loans. AmeriCorps also awards members a small
monthly stipend that they use to support themselves during
their term of service.
Mississippi’s children not only benefit from ARM’s work in
the short term by receiving personalized tutoring and
mentoring services to help them hurdle the historical,
cultural, and social challenges they face, but also gain by
having better trained teachers in the classroom and
communities with more consistently employed adults (leading
to a number of positive outcomes, including more stable
households and neighborhoods).
Programmatic Challenges
A number of challenges work against ARM’s efforts to ensure
that its tutors have the competencies to deliver high
quality services and that its managers can track, evaluate
and respond to tutors and students' needs over the short and
long term. A number of factors further complicate these
challenges:
·
Tutors’ education and professional backgrounds:
Most ARM tutors are not child development experts and have
not had professional experience as reading tutors. Thus,
the program must provide a significant amount of initial and
ongoing training to its tutors throughout the year that
promotes a steep (almost instantaneous) learning curve.
Given everything the tutors must learn – content and
pedagogical knowledge – providing a cornucopia of general
trainings that aren’t aligned with any one individual tutor
or student’s needs is insufficient. While ARM’s regular,
in-person monthly trainings address issues that all tutors
benefit from greatly, finding a way to provide
individualized training and support between those trainings
is just as essential.
·
Geography:
ARM tutors are spread across the state. Many of them live
and work 50 miles or more from the closest full-time staff
person. ARM administrators believed that in order to make
the impact the organization was aiming for, the program
needed to find a way to overcome this barrier and to begin
tracking and quickly responding to each individual tutor's
(and therefore, each student’s) goals and challenges with
rapid, targeted support. The program also wanted a way to
facilitate ongoing sharing and learning among the tutors
despite the distances between many of them.
·
Staff workloads and stress: ARM’s 16 staff members have huge workloads consisting of
human resources administration (including payroll),
accounting, tutor training, data collection, reporting,
newsletter writing, other public relations work, and more.
Any new process or solution the program implements must
never tax its team even further.
·
Technology access and familiarity:
Many ARM tutors do not have much experience with computers
or the Internet. If the program decided to pursue any
solution that relied on Internet use, ARM needed to ensure
that its tutors could learn to use it with minimal training
and almost no support from overburdened staff members.
ARM’s Business Process Goals
ARM’s central office administrators continually look to
bring efficiencies to the program.
|
"The
Network is giving our AmeriCorps members the
immediate, personalized technical assistance
they need while providing regional and central
office staff with instant feedback on how well
the program is operating at each region,
information about how members are meeting
program goals, and what staff need to do to
improve our in-person training efforts."
Randee Williams
State Director |
Specifically, the program always looks for ways to:
·
Get tutors
the ongoing training and guidance they need to address their
individual goals and challenges with each student.
·
Facilitate
the development and ongoing growth of a learning community
among current and past tutors in the program, regardless of
their existing relationships with one another.
·
Build and
access institutional memory across program years of which
tutoring techniques and strategies work at each of its 59
tutoring sites (80 sites for Program Year 2007-08).
·
Know
whether its program is on target with tutor training and
programmatic goals at any moment so that if it is not,
regional and central office staff can make immediate changes
to get back on target before it's too late.
·
Decrease
the time and money staff spends on tutor monitoring, data
collection and reporting, and use that freed up time and
money to directly enhance tutor competencies and other
aspects of the program.
·
Create well
written reports that carry significant weight with reliable
and valid data for funding sustainability.
·
Spend only
a minimal amount of time training tutors and staff members
to use any adopted process.
While
ARM adopted a number of effective policies and procedures
over the years to help address these goals, the challenges
detailed above still led to undesirable amounts of time and
money being spent to realize the program’s mission.
- - - - - - - - - - - - -
- - - - - - - - - - - - - - -
Section 2:
Strategy, Adoption & Implementation
Strategy for Meeting its Goals
While ARM had the goals listed above for some time, the
program did not believe it was possible to address all of
them in a simple way until a colleague passed along an
e-mail she received in July 2004 about a new service called
the America Learns National Performance Measurement &
Support Network ("the Network"). ARM decided to review the
Network two times via a Web-based tour led by an America
Learns team member.
Examining the Network
As ARM’s central office administrators examined the Network,
they made sure it would be cost effective, could be
customized to fit with the many facets of its program[ii],
would work quickly over old computers and dial-up modems,
would not disrupt tutors and staff members’ personal and
professional daily routines, would be easy to use by people
unfamiliar with computers, and would be accurate in getting
tutors and managers the training, guidance, support and data
they needed right away.
ARM also confirmed that its staff would have quick access to
technical assistance when needed – both support built into
the application and via phone and e-mail.
Administrators further ensured that its staff would not
spend a lot of time generating and uploading new tutor
training content to the Network. Because the America Learns
Network came with high quality content designed just for
reading tutors, and because America Learns team members
regularly analyze the challenges faced by its clients’
tutors and then create new strategies to overcome those
challenges, ARM staff did not have to spend time populating
a database of training content from scratch. When ARM’s
central office staff saw the Network’s potential to meet its
goals and had colleagues review it, ARM’s strategy quickly
became one of amending its budget to find funding to pilot
it. ARM made this happen in about three weeks[iii].
- - - - - - - - - - - - -
- - - - - - - - - - - - - - -
Section 3: The Research Driving the
Network
The R&D Process
The America Learns Network was built upon hundreds of
interviews with tutoring and mentoring program staff members
and their tutors and mentors. Much of that research
involved asking these individuals whether they even wanted
to receive ongoing training and support, and if so, what
types of training and support they would make time to take
advantage of.
Tutors and mentors' responses focused more around content
delivery systems rather than content itself, and most of the
responses were not "I want to use X" expressions such as, “I
want to use a social networking and bulletin board
website.” Most responses were of the type, “I’ll use
anything except X.” People ruled out generic resource
websites (imagine something called “tutorsupport.org” that
contains a tutoring strategies database), e-mail listservs,
discussion boards, guidebooks, online videos, videotapes,
and additional in-person workshops. The main reasons for
those media not being attractive was that they are simply
too time consuming to use, and that when tutors and mentors
do use them, they usually do not find the information they
need.
Interviews with program staff members nationwide revealed
that they not only wanted to provide support to their tutors
and mentors, but also wanted a way to get a better picture
of what exactly their tutors and mentors were doing with
students on a day-to-day or week-to-week basis so that they
could design training opportunities that were aligned with
those actual needs. When staff provided that support, they
didn’t want to deliver it weeks or months after their tutors
and mentors needed it.
Staff members also cried out for ways to reduce the amount
of time spent on data collection, reporting and evaluation.
In recent years, grant makers have significantly increased
the amount and types of performance measurement and
evaluation data that grantees must
report to retain and renew funding. This culture
shift has led people hired to manage programs to become
“accidental researchers.” According to Randee Williams,
ARM’s State Director and long-time resident of the
AmeriCorps community, “These individuals’ limited knowledge,
skills, and experience in this area jeopardize future
eligibility for grant awards.”
|
In short, through
the process of collecting the data ARM needs,
ARM is also able to provide individualized
support to its tutors. |
The Basics
of How it Works
The Web-based, patent pending Network allows ARM to fulfill
its needs of collecting important data and best tutoring
strategies coming out of its program, sharing those
practices when tutors need them, and measuring tutor impact
in a single step – the completion of a survey form, journal
or reflection log. Unique aspects of the Network also allow
ARM tutors to share their strategies with and access the
best strategies of other educators using the Network
worldwide. ARM is able to accomplish these goals without
using media its tutors usually lack time to use on a regular
basis – e-mail lists, web-based forums, bulletin boards,
formal courseware, blogs, wikis or guidebooks.
In short, through the process of collecting the reporting
and measurement data ARM needs, ARM is also able to provide
individualized support to its tutors. The Network’s impact
on strengthening and increasing tutor competencies extends
into the physical world as well, as ARM uses the data it
collects to design in-person training workshops, one-on-one
and small group check-ins, and paper-based newsletters to
make those forms of training and support more relevant to
tutors’ daily needs. As discussed more Since the tutors
are able to use the Network to give its staff the tracking,
monitoring and evaluation data ARM needs, staff members are
able to provide the tutors with the timely training,
guidance and support they need.
Launching the Network
Year 1 Training (Pilot Year)
During its first year using the Network, ARM conducted
live demonstrations of it for all of its tutors with an
emphasis on the reasons ARM invested in the Network for
them. The main benefits ARM discussed included:
1.
Providing
tutors with timely, specific, step-by-step strategies to use
that are related to their own goals and challenges;
2.
Giving
tutors an opportunity to share their own successes with
others in their program and with other educators using the
Network across the nation;
3.
Helping
tutors to stay focused and being more accountable to
themselves and to their students; and
4.
Allowing
their program managers to be more responsive to them.
ARM
required its Team Leaders to participate in the pilot and
also recruited additional volunteer participants (for a
total of 100 users)[iv].
Because the Network is so easy to use and because all of its
tutors are familiar with the process of completing surveys,
ARM found that its training needed to last one hour.
America Learns reduced ARM’s training preparation workloads
by providing materials to use and hand out during the
trainings.
Years 2 and 3 Training
ARM provided similar training and support during its second
and third years using the Network when it increased the
number of tutors using it to 150.
Year 1 Ongoing Culture Integration
ARM called the tutors piloting the America Learns Network
"America Learns-ARM Members" and gave them special "happies"
(paper weights in the shape of apples, memo note holders)
throughout the year to thank them for their participation.
(ARM also provided awards to these members during its
end-of-year celebration.) Further, ARM regularly shared the
data generated by the Network with its tutors to demonstrate
how ARM was using it to improve programming and to report to
the Corporation for National Community Service (the federal
agency that funds the program). Given the numerous demands
and limited financial awards its tutors receive, ARM
administrators believe that their tutors must know that
their time and effort is valuable and is benefiting not only
themselves but the program as a whole.
Years 2 and 3 Culture Integration
In addition to the activities above, ARM continued to market
the Network internally during its second and third years by
celebrating an author of each region's "tutoring strategy of
the month" (the best strategy submitted by a tutor via the
America Learns Network), giving awards to those tutors at
regional meetings. (Each ARM region selected a winner.)
From the regional winners, the central office administrators
selected the winning state tutoring strategy of the month
and highlighted this person in its monthly newsletter and
gave the person an engraved plaque.
- - - - - - - - - - - - -
- - - - - - - - - - - - - - -
Section 4:
Managing & Measuring the Network
How ARM
Manages the Network
Weekly,
Personalized Survey Completion Reminders
The Network sends reminder e-mails to all of its tutors each
week, asking those who have not yet completed a survey to do
so. The e-mails act as "automated managers" for ARM.
|
Since
implementing its plan, ARM's weekly
survey completion rate average has skyrocketed
from 64% to over 90%, a completion rate rarely
achieved in the social services sector. |
Monitoring Survey Completion Rates & Following Up
ARM regional staff members monitor survey completion rates
within their regions and central office administrators
monitor the state as a whole. Staff take advantage of the
Network's easy to use features designed explicitly for
following up with those tutors who do not complete surveys
to learn the reasons they did not do so, and to remind
tutors that completing the surveys is not only a privilege,
but a requirement for all America Learns-ARM Members. ARM
worked closely with America Learns staff to develop an
internal Network management plan so that ARM could build the
process of staying on top of this information into its
culture. Since implementing its plan, ARM’s weekly survey
completion rate average skyrocketed from 64% (2004-05) to
77% (2005-06) to over 90 percent (2006-07), a completion
rate rarely achieved in the social services sector.
How ARM Measures the Impact the Network is Making
Use of Recommended Strategies by Tutors
It's essential that tutors learn from the Network and apply
that learning in their tutoring situations. Since ARM
doesn’t have the manpower to observe the tutors in action
more than a few days a year, ARM relies upon tutors’ reports
via the Network to learn whether tutors are using the
strategies. During the 2005-06 survey, out of 337 surveys
completed over a number of periods, one hundred percent of
members who reported that the Network had ever recommended a
strategy to them following the completion of the survey also
reported that they had found at least one recommendation to
be helpful. When asked to explain the reasons they use the
strategies, tutors wrote responses such as, "They were very
simple to follow and easy to understand", "Because I ran out
of ideas and needed more new ideas," "The strategies help in
every possible way with the students," and "The strategies
helped because [they] gave me a new way to teach something,
therefore making it more interesting to me and to the
students."
|
"The strategies
helped because [they] gave me a new way to teach
something, therefore making it more interesting
to me and to the students."
ARM
AmeriCorps member |
Based on
this and other data, ARM has learned that it needs to add a
number of resources around non-tutoring issues that its
tutors struggle with as AmeriCorps members. All AmeriCorps
members, for example, must recruit volunteers in their
community, and many must participate in homeland security
and emergency preparedness activities. (For program year
2007-08, selected members will also perform hurricane
recovery service and others will mentor children of
incarcerated parents.) Because America Learns' current
content expertise does not cover these issues, ARM will need
to invest time to create its own content and make specific
requests of its tutors to submit their successes in these
areas.
ARM also
surveys tutors around how supported they feel and what else
they need from staff to accomplish their mission.
Use of Data by Regional and Central Office Staff, and
Resulting Financial Returns
Aside from using survey data to improve training and overall
programming, ARM also uses it for reports and evaluation.
The Network has reduced the amount of time ARM
administrators and staff members spend on annual reporting
by more than 660 hours (four weeks’ worth of time) –
translating into an annual dollar savings of more than
$8,000 once its financial investment in the Network is taken
into consideration. (Note that this figure only
includes savings the Network brings to ARM around data
collection, performance measurement, reporting and
evaluation, not tutor training and support.) ARM has been
able to reallocate the inordinate amount of time its staff
members used to spend collecting and reporting data to
using data to provide enhanced in-person training and
support to its tutors. While the accompanying staff stress
level reduction is tough to measure, administrators report
that “It's priceless.”
ARM also uses the data to shape significant portions of its
in-person training workshops, making those trainings more
meaningful to its tutors. The program sometimes uses
strategies from America Learns to shape its workshops. As
an example, following is a portion of an e-mail sent to
America Learns from an ARM administrator in response to ARM
staff finding that its tutors were running into a number of
classroom management challenges:
“…we
are planning to include classroom management to our training
curriculum and offer alternatives/strategies that our
members can use in certain situations. We will definitely
take advantage of the America Learns strategies.”
On a
four point scale (Poor, Fair, Good, Excellent), more than
90% of ARM tutors using the America Learns Network rate
in-person trainings as “Good” or “Excellent”.
|
ARM has been
able to reallocate the inordinate amount of time
its staff members used to spend collecting and
reporting data to using data to provide
enhanced in-person training and support to its
tutors. While the accompanying staff stress
level reduction is tough to measure,
administrators report that “It's priceless.” |
Randee
Williams notes, “The Network is giving our AmeriCorps
members the immediate, personalized technical assistance
they need while providing regional and central office staff
with instant feedback on how well the program is operating
at each region, information about how members are meeting
program goals, and what staff need to do to improve our
in-person training efforts. I can pull up this information
from any Internet connection 24/7.”
Further, ARM’s use of the Network is making its quarterly
reporting to government funders a breeze. While regional
staff used to have to compile data and send it separately to
the central office, central office administrators now pull
up and copy the data from the Network into their reports and
into WBRS, the web-based reporting system managed by the
Corporation for National & Community Service.
- - - - - - - - - - - - -
- - - - - - - - - - - - - - -
Section 5: Lessons Learned
Following
is a list of some of the most important lessons ARM has learned since beginning
to use the Network:
|
"The worldwide
collaboration the America Learns Network
facilitates is what the national service and
education worlds have been waiting for. The
fact that we can easily couple these benefits
with reporting, performance measurement and
evaluation makes the Network a must have for
us."
Randee Williams
State Director |
1.
Technology is available to help ARM accomplish its goals
listed above.
The Network is allowing ARM to improve the delivery of
services to Mississippi children.
2.
Develop and implement
a formal management plan to implement technology such as the Network.
During its first year using the Network, ARM did not make enough time to really
build it into its culture. Since then, ARM has worked with America Learns to
think through each action that needs to happen to effectively capitalize on the
Network's intended and unintended benefits. (America Learns now uses an
adaptation of the management plan ARM developed with all of the other
organizations it serves to ensure that those organizations effectively build the
Network deeply into their cultures.
3.
It’s important that
the tutors feel that their opinions matter and that they are an integral part of
the program’s evaluation process and improvement plans. The tutors are “in the trenches,” and ARM believes
that that reality must be acknowledged and respected. The Network offers the
necessary vehicle for ARM’s program staff to constantly learn about and then
respond quickly to tutors’ experiences and opinions.
4.
When ARM provides its
tutors with ongoing training resources that respect tutors’ time constraints,
tutors are far more likely to use those resources. By incorporating reporting, evaluation, strategy sharing
and ongoing tutor support into one step – the completion of a survey – ARM has
found an answer to getting much of what it needs from tutors and providing what
it needs to provide to them throughout the year.
5.
It’s essential to
train tutors around what a well-written strategy looks like.
As mentioned earlier, ARM uses the Network to constantly collect and make use of
the best strategies submitted by its tutors. ARM found that it has to set clear
expectations around what these strategies look like so that when tutors submit
them, they’re sufficiently detailed for others. While ARM still has a ways to
go in this area (much of its success hinders on its tutors’ writing abilities),
ARM has made substantial progress in getting its tutors to submit higher quality
strategies. As a result, America Learns highlighted two of its members’
strategies as National Strategies of the Month during the 2006-07 school year,
an honor bestowed to only twelve tutors each year.
6.
Capture and use the
amazing strategies and activities its tutors are creating and sharing via the
Network. Until ARM
began using the Network, its tutors' original, innovative strategies were lost.
ARM collected some success stories on paper, but the program did not have the
ability or capacity to organize and then distribute those resources to tutors
when they needed them. Now, when tutors complete their weekly surveys, they
instantly receive tips and resources from their colleagues statewide -- even
from colleagues who are no longer with the program. And since the program is
connected via the Network to other programs America Learns serves nationwide,
the strategies its tutors are creating are helping other students across the
country in programs ARM does not formally interact with.
Randee Williams notes, “With
the Network, it’s now possible to create a culture that values and facilitates
ongoing sharing and learning among tutors, regardless of their physical location
and relationship with other tutors in a community – in this case, the ARM and
America Learns Network communities. The whole notion of ‘Think globally, act
locally,’ has been rearranged through this process. ARM and all other
organizations using the America Learns Network are thinking locally, acting
locally, and making a global impact simultaneously. The worldwide collaboration
the America Learns Network facilitates is what the national service and
education worlds have been waiting for. The fact that we can easily couple
these benefits with reporting, performance measurement and evaluation makes the
Network a must have for us.”
Endnotes
[i
(back)]
Mississippi’s unemployment rate is the highest in the
nation at 6.9% (SOURCE:
bls.gov;
April 20, 2007)
[iii
(back)]
Rather than budgeting the Network as a separate line item, ARM found it
easier to justify its use of the Network to federal grant officers when the
program budgeted it under line items dealing with AmeriCorps member
development and program evaluation.
[iv
(back)]
Team Leaders are AmeriCorps members who receive specialized leadership training so that they can effectively
assume additional responsibilities that help to successfully implement the ARM’s programming at each school.
These members do not receive additional financial compensation.
|
