Making a Difference in the Peace Corps
Mammography Pioneer Sees Eye to Eye with Patients
Innovative Lectures Engage Students' Interest
Challenging Conventional Wisdom
KEEP US INFORMED:
Please send us your comments on this issue, ideas for future issues, and news about your professional interests and accomplishments.
Al Dorof, Editor
adorof@brynmawr.edu
info@brynmawr.edu
©2007
Challenging Conventional Wisdom
 |
Julia Littell |
In 2005, Julia Littell — then an associate professor in Bryn Mawr's Graduate School of Social Work and Social Research (GSSWSR) — published a series of papers that raised questions about a widely used treatment for young people in the juvenile justice, mental health and child welfare systems. One article (J.H. Littell, "Lessons from a systematic review of effects of multisystemic therapy," Children and Youth Services Review, 27:445-63, 2005) touched off a heated debate in the field.
The paper grew out of Littell's longtime research interest in services for abused and neglected children. "For 15 or 20 years, there had been a trend to develop in-home services as an alternative to foster-care or out-of-home placement," Littell explains. The cost amounted to "literally billions of dollars," but the results were disappointing, she says. In the 1990s, she participated in a series of large, randomized experiments that found comprehensive in-home services did not prevent foster-care placement or a recurrence of child mistreatment.
Many practitioners and policymakers began to embrace multisystemic therapy (MST) — a short-term home- and community-based intervention — as an alternative. "I became intrigued," Littell recalls. She embarked on a systematic review of the MST literature to assess the effectiveness of the treatment.
Research Synthesis
With GSSWSR doctoral students Melania Popa Mabe and Burnee Forsythe, Littell set out to review and synthesize all available evidence of MST's effectiveness, including unpublished and published studies. Her study involved meta-analysis, a statistical synthesis of results that takes into consideration factors such as a study's methodology, size and precision. There were strict rules for inclusion or exclusion of studies, so as to avoid selection based on results or other factors. While it's human nature to have preconceived notions, Littell explains, "We want to make sure it's not a self-fulfilling prophecy."
Traditional literature reviews are subject to "publication bias," Littell says. "The literature is biased toward positive results — a study that finds that a treatment is effective is more likely to be published than an equally worthy study that finds the treatment is not effective."
Finding unpublished studies is not easy. "It's very labor-intensive and, as a result, costly," Littell says.
Since 2000, Littell has been involved with two international, interdisciplinary collaborations of scholars, practitioners and policymakers that have established principles for minimizing bias in the synthesis of research on treatment effects. The Cochrane Collaboration synthesizes research on health care; the Campbell Collaboration synthesizes research in the social sciences. "There are 50,000 people involved worldwide, and it's almost entirely voluntary," Littell says. Her systematic review of research on effects of multisystemic therapy is jointly registered under the Cochrane and Campbell collaborations and is published in their libraries.
Results and Fallout
The results of Littell's study differed sharply from those of previous reviews of the MST literature. Her findings "indicate that MST has few if any significant effects on measured outcomes, compared with usual services or alternative treatments," she wrote. Littell says she was not surprised that there would be a strong reaction to this conclusion. But, she adds, "I didn't expect it to be this personal."
Littell was in contact with the developers of MST "right from the beginning," she says. She sent them the first draft of the paper and asked for additional information on several studies. "Initially, the response was patient," she recalls. "Then they were frustrated, and then it was quite clear that we were going to get an angry response."
The researchers' rebuttal to Littell's paper was published in 2006 (S.W. Henggler et al., "Methodological critique and meta-analysis as Trojan horse," Children and Youth Services Review, 28:447-57, 2006). Their response notes that her conclusion "contrasts with that reached by leading federal entities and reviewers."
"Their response is largely ad hominem, focused on my motives and qualifications," Littell says. "But that's not what this is about. The problem is that the evidence isn't as good as we would like it to be." Littell responded to the rebuttal in the same issue (Children and Youth Services Review, 28:458-72, 2006).
Since the controversy erupted, "I'm more determined to work on these issues, and to get accurate information out there," Littell says. "Most of my work has shifted toward scholarly processes — how we know what works in social services and how evidence is disseminated, interpreted and used in policy-making." She has presented her work internationally. "I've had some constructive debates," she notes, "especially in the Nordic countries, where there is a concern about accurate interpretation of research findings, and promoting high-quality services for kids."
Littell plans to update her MST review. "There are more data to be added," she says. "That will be very interesting, and it's an important thing to do." She also plans to review another program said to be very effective for children and families.
Littell received the Pro Humanitate Literary Award, North America's premier literary award for the field of child welfare, honoring "literary works which best exemplify the intellectual integrity and moral courage required to transcend political and social barriers to promote best practice in the field of child welfare."
"This was a tough process, but well worth it," Littell notes.
Barbara Spector writes on science and technology as well as business topics. She is the editor-in-chief of Family Business magazine and former editor of The Scientist.