Comments on: Ask Reid Lyon

By: children disabilities

children disabilities — Sun, 08 Jun 2008 07:43:08 +0000

children disabilities…

What I do say, is that I don’ t want to be turned into something I’ m not, and I don’ t wish that upon anyone, including this child. The aim of support or treatment for this child, or me, or anyone, should not be to“ normalize” them. When in …

By: Reid Lyon

Reid Lyon — Wed, 14 May 2008 17:43:33 +0000

I think Mr. DeRosa gives a little too much credit to the depth of knowledge that many congressional members and staffers bring with them when working on education legislation. Believe me, SBRR was as foreign a concept to many in congress as it was to states, districts, and schools. That is not a criticism. They are inundated with a tremendous amount of other legislative information on a myriad of issues. Also, much of the exposure to the characteristics of scientifically based research and its promise in guiding policy that members would receive at congressional hearings would typically be presented by researchers who would overwhelm members by using technical language and would frequently qualify every point made. As I mentioned in an earlier post, Bill Goodling and his staff understood in depth the need to guide education policy and practices based on the best evidence possible, as did a few other members on both sides of the aisle.
But people much more experienced than I working in the bowels of the congressional process can explain the impact of historical ideas about the federal role in education, the very strong influence of outside special interest groups and lobbyists, and the political infighting that is conducted just to make others look like they have failed on what finally emerges from conference as law. To me it’s a real ugly process. I heard more times than I can count that I just did not understand how it is done, and I probably still do not get it. For example, recall that the majority of education formula funding grants was considered by recipients to be, in reality, entitlement awards. A recipient basically had to account for process, not product, and let the feds know where the money ended up. Outcome was a concept that did not resonate with many in the education world. The prescriptive nature of Reading First was foreign to many legislators, and many fought hard to loosen the criteria every step of the way, not only on effectiveness criteria, but on other critical aspects of the law (accountability guidelines, data reporting requirements, etc.).
What I learned is that the best of intentions go into a legislative meat grinder and come out packaged in law that gives you some of what was intended and takes away some of what was intended. I learned that logic was not high on the list of essentials in the legislative process. Given this, my partner Bob Sweet would counsel me to keep in mind we were inching forward and it simply had no chance of being perfect. We were able to bring the concept of research-based education into the law of the land. That had never been done before. And that was precisely why many in the education community fought and are still fighting to remove the word “research-based” from the legislative lexicon. And, to be sure, they are receiving help from mistakes made in the implementation of Reading First and from wildly misconstrued findings reported in the interim impact study report. But it is the fact changes in the legislative language were not a result of some conspiracy to fill the coffers of particular publishers or vendors. They came about through a compromise process driven by many special interests particularly a strong opposition to create federal rules that states should follow if they were going to receive federal money. Let’s pile up that money on the stump and let the feeding frenzy begin.

By: KDeRosa

KDeRosa — Wed, 14 May 2008 11:54:30 +0000

So it sounds like those in Congress close to educatation would have known the precise scope of the language and which kinds of reading programs were intended to be funded and which were not based on the final language. They would have also known that DOE for all its careless procedures actually funded the right programs and there is no evidence that they funded any program improperly despite the potential conflicts. So why the political theater?

By: Reid Lyon

Reid Lyon — Tue, 13 May 2008 22:39:15 +0000

The OIG has a record of the original language drafted by Bob Sweet and myself on the Reading First effectiveness criteria. The same language was sent to the OIG when they requested recommendations for re-authorization. I had incorrectly assumed that the Department had modified the language during their drafting of the Reading First guidance which would not have involved either Sweet or me. My incorrect assumption led to my incorrect answer to one of Salvato’s questions. Had I asked Bob, he would have told me what he has reported – the language went through several iterations in the negotiations process with precise wording being debated every step of the way. Neither Sweet nor I would have any final say in the structure of the language passed into law. I would contact Bob directly as HE is the record having done the heavy lifting on turning research principles into legislative language and guiding the legislation through the conference negotiations. He can certainly provide a very accurate summary of the inner workings of that process. What I do know is that the opposition to the language as submitted was bipartisan. The republicans, in general, felt it was too prescriptive. Many democrats had program favorites that would not have met the effectiveness criteria at that time. There are clearly overlaps between these two groups. Members on both sides were also lobbied heavily by publishers and vendors.
The problem is that we will not know from the impact study, at least as it was recently summarized, whether particular educational materials showed any particular effect on reading proficiency. This question was not included in the current evaluation design f as required in the law (see Section 1205) that we drafted. The evaluation was narrowed to the four questions indicated in the interim report despite having substantial funding to carry out the most comprehensive of evaluations. The null effect reported in the interim study is difficult to interpret because of sampling and implementation issues, even before you get to program specific effectiveness. Questions regarding effectiveness have to be addressed within the context of fidelity of implementation factors, the qualifications of those imparting a particular program, the additional reading activities taking place in school and elsewhere, the time allotted for reading instruction, the issue of combining a core program with a supplemental intervention, and several other potential confounds. Even if a program has been found to be effective in trials elsewhere, that program might not achieve the same effectiveness in a different set of contextual variables. That is why the original language also included a statement of the conditions under which a program had been found to be effective. These are not trivial issues when considering program specific impacts. There is substantial complexity involved from a design and methodology perspective and obviously from an implementation perspective. It can be done but it is expensive. That is why $25 million dollars a year for five years was allocated to the Reading First evaluation.

By: KDeRosa

KDeRosa — Tue, 13 May 2008 21:37:34 +0000

Typo alert : those last two paragraphs should have been deleted.

I’d also point out that the inclusion of all these untested programs undoubtedly affected the impact study results and is even more of a reason why the programs funded under RF should be evaluated separately in the final report — to avoid a loosening of the statutory language in the future if we expect better results out of SBR legislation going forward.

By: KDeRosa

KDeRosa — Tue, 13 May 2008 21:28:56 +0000

I didn’t mean to imply that it as your or Bob Sweet’s decision to alter the statutory language. At this point there does not appear to be any independent source that confirms how this change came about.

There is your 1/6/06 interview with Nancy Salvato in which you state “The Department of Education made the decision to make the criteria more general. ” Then there is your 3/27/2007 correction of that interview in which you state essentially what you’ve stated above. Then there are the the many statements post IG reports by you and Bob Sweet maintaining this position.

So, given that there was some question as to where the funding line was being enforced by DoE in the IG report, the GAO report, Congressman Miller and Senator Kennedy, I think it would be helpful to nail down how this chane ecame into being and by whom as a defense to these accusations that are relying on the fact that no independent source is backing you and Sweet up and has admitted responsibility for this change to date.

Any thoughts?

, and I am aware of all your statements you both have made.

There does seem to be some confusion as to how that language came to be changedand at whose insistence.

By: reid lyon

reid lyon — Tue, 13 May 2008 20:59:46 +0000

I should have also pointed out in my response to Mr. DeRosa that the Reading First Impact Study as presented in the interim report DID NOT address the requirement in the law that the evaluation would analyze the degree to which different instructional materials improved reading proficiency – See section 1205 – in the Reading First legislation. I am sure there will be more written on the considerable difference between the impact evaluation that was carried out by IES and the comprehensive evaluation that was required in the legislation. I discussed this issue in addition to the dilution of the Reading First effectiveness language in a recent interview published in EdNews.org

By: reid lyon

reid lyon — Tue, 13 May 2008 20:45:04 +0000

I certainly do regret the decision to permit the vast majority of Reading First funding to be allocated for untested programs. However, it was neither I nor Bob Sweet – the two individual who drafted the legislation – that made this decision. In the drafts we submitted for congressional review, funding was contingent on a program’s demonstration of effectiveness using the appropriate methods and designs required to determine causal effects. We also coupled this language with a sentence requiring that claims of effectiveness had to be combined with information indicating g the conditions under which effectiveness was established.
This language was diluted at the congressional level by members and staff who felt that restricting funding to only those programs that had established effectiveness would make implementation very difficult. Neither Sweet nor I were involved in that decision. As I have written many times, I did not agree then and I do not agree now with the scrapping of our language and replacing it with the much broader “based on” language. I am on the record many times stating that this congressional decision allowed for a significant opportunity for many publishers to hawk their wares under the guise of being “based on” SBRR. This went against the entire thrust and spirit of Reading First. What you might want to do is research this issue more deeply and look at the number of publishers and vendors that lobbied congress with vigor to change our original language.

By: KDeRosa

KDeRosa — Tue, 13 May 2008 19:41:33 +0000

I’m wondering Dr. Lyon regrets the decision to permit the vast majority of Reading First funding to go to untested and scientifically unvalidated reading programs instead of to the reading programs that had actual scientific support. That decision was a gross misuse of scientific research. It was a logical fallacy to conclude that since all the scientifically validated reading programs contain that five essential components of reading instruction that any program containing these components would also be effective. Now it seems that the chickens are coming home to roost with the interim study, flawed though it may be, showing that many of these untested reading programs showed little increased reading achievement. I would not be surprised if the fallout of this bad decision sets scientifically based reading instruction back by years.

By: Greg Forster

Greg Forster — Tue, 13 May 2008 13:04:59 +0000

Dr. Lyon describes the alternative to science as throwing mud against a wall and seeing what sticks. I think even that is giving too much credit to the anti-science crowd. “Seeing what sticks” is the job of science. The alternative to science is to keep throwing mud against the wall year after year without bothering to find out what results.

Thanks for this informative post – having spent my career so far on the governance side of the education field, I didn’t know most of this history on the pedagogical side. It really is sad that we had all this knoweldge about reading for all that time and few even knew about it! Not surprising, of course, but sad.

Your post, especially your acknowledgement of Jeanne Chall and others who did this work before anyone outside the field was paying attention, reminds me of something J.S. Mill wrote in his defense of free speech. He said there were twenty Martin Luthers before the Reformation. What he meant was that Luther’s ideas weren’t new; lots of people had said the same things before. What changed was the social conditions that made it possible for him to survive attempts to silence him and to reach a wide audience with those ideas. Obviously this isn’t quite the same thing, but the dynamic feels similar.