Did Harvard Really Declare the Enciclomedia Program Useless?

The image “https://i2.wp.com/i113.photobucket.com/albums/n216/cbrayton/Stuff/harvardstudy.png” cannot be displayed, because it contains errors.
“Harvard Study Recognizes: Educational Programs Are Meeting Their Goals.” Click to zoom.

What [the Harvard team] found and presented in November (2006) was a mixed bag: some of the programs and mandates were working, but others needed major changes, more money, and a greater commitment from everyone involved.

The essence of the scientific method involves observations that can be repeated and verified by others. Hence, psychologists do not make up data or modify their results to support a hypothesis .. Errors of omission are also prohibited. Psychologists do not omit troublesome observations from their reports so as to present a more convincing story. Publication Manual of the American Psychological Association, 6.05 “Ethics of Scientific Publication, Principle 6.21 “Reporting of results”

If you cannot bring good news then don’t bring any.
–Bob Dylan, “Wicked Messenger”

I keep trying to double-check the claim reported by La Jornada recently about the Enciclomedia digital chalkboard program in Mexico. See

(I am exaggerating a bit for effect there. What was claimed is that the program did not have the anticipated effect on student testing performance.)

Did Bill Gates’ alma mater really say that?

What did Bill Gates’ alma mater really say?

Professor Ilona Holland was cited by the Guardian in June of this year as saying something quite different.

Enciclomedia is “amazing” and “phenomenal”!

The Harvard study was certainly impressed. “Enciclomedia brings together an amazing number of resources and integrates them meaningfully,” says Ilona Holland, the professor who headed the study. “All of a sudden you have everything together and it is phenomenal the difference that makes.”

The Guardian reporter goes on to endorse the views of her interview subject.

A phenomenal difference for Mexican teachers, too, who grew up in a tradition that sees education as learning facts. Most Mexicans still believe the best student is the one who gets 10-out-of-10 in tests based on regurgitating lessons learned, rather than one who has acquired what Holland terms “higher-order thinking skills”. So it is hardly surprising that teachers are complaining they have not had sufficient training and do not get enough time to prepare lessons. Concerns about teacher comfort are among the most frequently cited problems with the project.

Really? Cited by whom?

I am reading the “white book” published by the Mexican Education Secretary (SEP), dated October 2006, on the program, and I am reading the ASF audit.

I am reading the SEP press release (November 6, 2006) on the presentation of the Harvard study, headlined “Educational Programs Meet Their Goals (Harvard Study Says)”

I am still trying to find the report from the education committee in the lower house of the Mexican congress that substantiates the claim that the Harvard study found no improvement of test scores among students using the system.

(UPDATE: Here, at least, is an executive summary attributed to the chair of the committee:

Are they referring to a recent study, or are they referring to the raw data on which the “white book”was (partially) based? Those claims are described in

I am noting that the appendix in the SEP “white book” that I got a copy of — dated October 2006 — containing the responses to auditors has been classified. As in, “this information is classified. The public cannot know it.”

Oh, okay, I have located a breathless “narrative journalism” account of Holland’s delivery of the program audit to Fox, on the Harvard Education Web site, which I will produce below for annotation. What it reports:

What they found and presented in November (2006) was a mixed bag: some of the programs and mandates were working, but others needed major changes, more money, and a greater commitment from everyone involved.

Harvard recommends throwing more money into programs in which quite a lot of money has yet to be accounted for, according to the federal auditor’s review of the 2005 budget.

How embarrassing for Harvard.

The principal contractor on the program is a large corporation founded by a Harvard alum and, apparently, a generous donor.

In the meantime, I come across a report by Cisco from 2006, titled “Technology in Schools: What the Research Says.”

Citing Larry Cuban of the Harvard faculty as a critic of such programs. Oversold and Underused, the book he wrote. Professor Cuban was not invited along to Mexico for some reason. Is he the Enciclomedia program’s General Shinseki?

The image “https://i0.wp.com/www.hup.harvard.edu/images/jackets/CUBOVE.jpg” cannot be displayed, because it contains errors.

The Cisco-Metiri report’s general conclusion is that educational technology evangelists have made promises they could not keep.

The reality is that advocates have over-promised the ability of education to extract a learning return on technology investments in schools. The research studies now suggest that their error was not in citing the potential of technology to augment learning—for research now clearly indicates that the effective use of technology can result in higher levels of learning. This review of the past decade suggests four miscalculations on the part of educators:

Lack of effort and wishful thinking:

  1. First, in being overly confident that they could easily accomplish the depth of school change required to realize the potential technology holds for learning—not an easy task
  2. Second, in their lack of effort in documenting the effect on student learning, teacher practices, and system efficiencies
  3. Third, in overestimating the time it would take to reach a sufficiency point for technology access
  4. Fourth, in underestimating the rate of change in technology, and the impact of such rapid, continuous change on staff time,budgeting, professional development, software upgrades, and curricular and lesson redesign

As a result:

As a result, the real potential of technology for improving learning remains largely untapped in schools today.

From the Harvard School of Education news service — Winter 2007 — with reporting on actual findings buried deep under a cliche-ridden “narrative journalism” lede that builds a sense of anticipation and tension, while pontificating about the need to base the program on hard data about its practical results, but provides precisely none.

As the two vans pull up to the heavily guarded gates of Los Pinos, Mexico’s version of the White House, which is at the end of a long drive lined with palm trees and manicured gardens, a crowd of about 30 is already gathered. They are a mixed group — politicians, academics, and staff from Mexico’s Secretariat of Public Education — and there’s a sense of anticipation in the air. Not only are they waiting to be let into the building, but they are waiting for President Vicente Fox to arrive, and, more importantly, they are waiting for the day’s big event: It’s November and the Harvard team, in the vans, is back to present their findings from a nine-month study of four of Mexico’s national education programs.

My baby drove up in a brand new Cadillac
Yes she did!
My baby drove up in a brand new Cadillac
She said, hey, come here, daddy!
I ain’t never comin’ back!

In one of the vans, Ed School lecturer Ilona Holland, Ed.M.’85, Ed.D.’91, is in the back seat, still hovering over her speech. Although she has been working on it and practicing her Spanish greetings for days with Mexican native Eugenia Garduño, Ed.M.’06, she is nervous.

“Make it better, make it better,” she says, laughing as she continues to tweak the text.

Harvard professors apparently now have to reduce their findings on complex technical issues to four-syllable advertising slogans. Or what? Risk losing their grant money?

Holland, along with the rest of the team — Dean Kathleen McCartney, Professor Richard Murnane, Professor Hiro Yoshikawa, Professor Fernando Reimers, Ed.M.’84, Ed.D.’88, (principal investigator of the project), Garduño, and doctoral student Sergio Cárdenas, Ed.M.’04 — knows it’s important to get it right because, as President Fox would later say at the presentation ceremony, “Education is one of the most important and heartfelt issues facing Mexico.”

And Elba Esther Gordillo insists on being in charge of reforming it.

Which is why, last spring, the Mexican Secretariat of Public Education contacted Reimers to design an evaluation of four education projects that had been started under the Fox administration: preschool education reform (McCartney, Yoshikawa); national literacy (Reimers, Professor Catherine Snow); PEC, a program that decentralizes decision-making in schools (Murnane, Cardenas, Professor John Willett); and a technology program called Enciclomedia (Holland, Garduno, Professor Chris Dede, Senior Lecturer Jim Honan, Ed.M.’85, Ed.D.’89, Professor David Perkins).

In March, the group made their first visit to the country, one of the most densely populated in the world, to visit schools, conduct initial interviews, and collect data. What they found and presented in November was a mixed bag: some of the programs and mandates were working, but others needed major changes, more money, and a greater commitment from everyone involved.

The image “https://i2.wp.com/i113.photobucket.com/albums/n216/cbrayton/Stuff/enciclo.png” cannot be displayed, because it contains errors.
Schematic for the “interactive chalkboard” (50,000 pesos per unit).

Again: Harvard recommends throwing more money into programs in which quite a lot of money has yet to be accounted for, as we learn later.

The principal contractor on the program is a large corporation founded by a Harvard alum and generous donor.

Go figure. You ask me, this looks really, really bad. You can rebrand “conflicts of interests” as “innovation synergies” all you want to, but doublespeak is still doublespeak. Toxic sludge is not good for you. Studies show.

Back at Los Pinos, Fox stands at the podium and, despite the fact that everything he’s heard isn’t completely positive, he praises the process.

“In the past, the practice was that the results of education evaluations were best kept secret,” he says. “Today, with democracy and transparency, the results are with everybody, good and bad.”

Allegedly, the results of SEP’s “white book” program audit were suppressed.

Some folks are working on comparing the raw data, procured through a “freedom of information” request, with the official document produced, I think I read. On which more soon.

Education secretary Reyes Tamez, holding a copy of the paperback produced from the evaluation, Aprender Más Y Mejor, agrees.

“It’s not about right or wrong,” he says, “but setting goals and reaching them.”

The Enciclomedia program fell short of its goal for classrooms to be equipped by some 11,000, the federal auditor found.

And there is no solid accounting of where the equipment got to.

For the most part, the Ed School’s findings are positive. When it comes to the law that made preschool a requirement for three-, four-, and five-year-olds, for instance, McCartney and Yoshikawa found that attendance by four- and five-year-olds was nearly perfect: 98 percent for five-year-olds, 81 percent for four-year-olds. They were especially impressed that the country created the mandate in the first place.

“Mexico clearly understands the importance of getting children off on the right foot,” McCartney says to the crowd of about 500.

On the flipside, they also found that attendance by three-year-olds is disappointingly low — only 25 percent — and suggest that the policy be reconsidered so that families who want to send their three-year-olds would be supported, but not required.

Murnane and his team also found mixed results when they evaluated Mexico’s five-year-old Programa Escuela de Calidad, known as PEC. Schools that agreed to become “PEC schools” were given greater freedom to develop their own plans for improvement — a sharp contrast to earlier days when all schools were run in the same way under federal guidelines. The team found that after three years of participation, PEC schools had lower dropout rates, but not in the country’s poorest states.

“These are the states in the greatest need of improvement in their educational systems,” Murnane says at the presentation, in Spanish.

Another new initiative studied involved Enciclomedia, a software package being tested in schools that digitalizes traditional textbooks and adds innovative technology, such as videos and maps, and can be tailored to the country’s many indigenous cultures.

Overall, Holland found that the software had significantly improved learning.

“It’s like bringing libraries of information right into the classrooms,” she says (not sounding nervous). A few weeks later, back in Cambridge, she says that’s part of what got her interested in the project in the first place.

“Not sounding nervous”? What is this, the high school newspaper, with giggling in-jokes?

“Fernando asked me to become one of the leads on this project. When he explained that I would be evaluating a national computer-based curriculum that had been placed in thousands of classrooms in Mexico, I was very intrigued,” she says. “I was interested because I thought this was an opportunity to help millions of children. If the program was strong, empirical data was needed to support its scaling and sustainability. If it was weak, the opportunity would be there to identify ways to improve the program in order to enable it to better meet its goals. Either way, my work would make a difference.”

None of that hard data is presented here.

What her team found was that although the Mexican press, in articles written after the Los Pinos presentation, focused on the lack of electricity in many cash-poor areas of the country as a major problem, the real obstacle for Enciclomedia is poorly trained teachers who are not always held accountable for their performance. In addition, the program is offered only to fourth and fifth graders and needs to be expanded to other grades.

You might also have said, “the failure to provide adequate teacher training or performance measurement.”

The software is bring great improvement, but the fact that teachers do not know how to use it is an obstacle to its bringing great improvement. Quack.

I read a lot of the Mexican press coverage on questions about the program.

I would say it focused less on the electricity issue — though the notion that the equipment was delivered to locations without any electrical socket to plug it into is, of course, a striking talking point, if true (is it?) — and more on the oversight failures, lack of technical support and maintenance, lack of adequate training, and the poor quality of the curriculum available in the digital content repository.

Among other issues. The team appears to have read the press selectively, and to be suggesting that the program is a victim of bad press rather than poor planning and execution.
La Jornada, November 7, 2006: Harvard Study Finds Enciclomedia Works Better in Communities With Electricity.

At Los Pinos, Reimers, who also gives his presentation in Spanish, talks about literacy issues, which Mexico is trying to address with a National Classroom Literacy Program that supplements literacy instruction by providing classroom libraries and professional teacher development in all nine grades of compulsory education. He says that the program has increased the availability of varied reading materials, but the main challenge is developing eff ective approaches of professional development that can support teachers as they integrate using the classroom libraries with literacy instruction.

“Access to a wide variety of fiction and nonfiction texts is indispensable to produce the kind of sophisticated readers that an evolving democracy and economy such as Mexico’s need,” he says. “The next generation of reforms should focus on providing teachers opportunities to enhance their skills in using these resources at a high level of effectiveness.”

As the Ed School group gets back in the vans at the end of a nearly 12-hour day of talking about and explaining their work (most of the time wearing headphones for the translators), they feel that it went well and that have learned as much from the collaboration as the Mexicans have.

“I was inspired by the vision of the Secretariat of Public Education for the Mexican people,” says McCartney. “They understand the importance of education for their democracy, and they plan to invest in policies and practices that work.”

They have a inspiring vision and a plan to invest.

Techie slang for technical systems consisting principally of inspirations, visions and plans is “vaporware” or “brochureware.”

Is that it? They spent, what was it? $700 million? on implementation, and what we are left with is praise for the inspiring vision they started with?

I have inspiring visions all the time.

All that is required to produce them is the cost of the food (and an occasional bottle of Mendoza shiraz) required to keep my wetware working.


Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s