Why are DORE so bad at research?

When I first heard of DORE, I thought they sounded like a good idea. There is not a lot of research into dyspraxia in particular (which I have), so anything that produces good results would be very welcome.

The problem is that they keep publishing papers which are a bit rubbish really, with silly mistakes, and where the results don't quite say what their write-up implies. They charge a lot of money for the treatment, & it requires much time & effort from children & parents. I did lots of chucking beanbags around as a Duckling, I know how pants it is to be made to stand on one leg throwing beanbags into shoeboxes & being reminded how rubbish you are every day while the other children are out playing or learning things that are actually useful. It has never been useful for me in my adult life to try to throw a beanbag into a shoebox with the wrong hand standing on one leg. These things are not harmless, and should be properly researched before being sold for a lot of money to desperate parents & children who already have enough to deal with.

If DORE was the 'Miracle Cure' they claim in a book title, it would be fantastic. So why is DORE research so badly done & inadequate? They have money, they have people doing the programme to test, so if it works & they want to encourage its wider use, they should do a better job of proving it.

The latest research is here: http://www.dore.com.au/researchscience/DORE_MATCHED_DATA_STUDY.pdf

Quick summary for those of you with better things to do than read a point-by-point ramble through a 30-page paper. This research was not designed in such as way as to be able to show that DORE works. There is no control group, no attempt to look at whether their measures correlate with real-world success in school etc, no follow-up, and only three in five of their participants were actually diagnosed with Dyslexia. Even if you take the research on its own terms, it shows that for most people then DORE doesn't work - only the bottom few % showed any improvement, and this was mostly to do with stuff like bead-threading, not measures of reading and writing. Without a control group it is not possible to tell whether this would have happened anyway, for example as the children got older or if they were being given extra help in school too. It is not surprising if children get better at things over time, especially if they have parents who are willing to put a lot of effort & money into helping them learn.

The Holfordwatch blog has had a look at this paper too, in perhaps a slightly more amusing fashion than my usual undergrad plodding through (I have to read lots of papers on Dyslexia every week because I am studying it as part of an an undergraduate psychology course). They can be found here: http://holfordwatch.info/2007/11/23/dore-research-paper-shows-that-dore-is-not-useful-for-a-substantial-proportion-of-potential-clients/

I am feeling particularly geeky today, so I have read it all & put down a point-by-point criticism. I might have missed things or got them wrong, please add a comment if you think I have.

Authors:

Wenjuan Zhang: http://www2.warwick.ac.uk/fac/sci/statistics/staff/research/wenjuan_zhang
'Her involvements in recent consultancy projects include contracts with LSC, DDAT, Dft, Education Walsall, and MG Rover'.
I haven't heard about her before. Seems to have just done the stats. Now, Maths is not my strongest point, but can any readers please explain why they are using 'deciles'? The stats are put together in a way I'm not really familiar with, but that could be just because I am but a humble & not-very-clever undergrad who does not yet know enough about such things.

Professor David Reynolds: done lots of pro-DORE sresearch, caused mass resignations from the editorial board of the journal 'Dyslexia'. Has been previously heavily criticised for financial links to DORE. He's published similar-sounding stuff before:
http://www.dyslexiaaction.org.uk/Page.aspx?PageId=21#Dore. 

Dr Roy Rutherford. He is the 'Global Medical Director for Dore' http://www.dore.co.uk/KeyPeople.aspx (it is rather odd for an organisation which claims a miracle cure for dyslexia to have quite so many spelling and grammatical mistakes on that webpage, including confusing homophones, but there you go).
Now, I should admit that I have not been so keen on Dr Rutherford since he called one of my lecturers a 'very aggressive lady' in the Times newspaper, without actually having met her. http://www.timesonline.co.uk/tol/life_and_style/health/child_health/article1344439.ece

My first grumbles on a very quick skim-read through the paper:

The usual gripes about them using their own tests. Some of them are fairly standard assessment tools, some aren't.

'More than 60% of subjects in this cohort have been previously
assessed by a specialist and diagnosed with dyslexia prior to
attending Dore'. Hmmm - 40% haven't? Only 50% of people in my final-year project will have been diagnosed, but I would expect to get different results on a range of measures - some of them similar to what DORE use - for the 50% who do have a diagnosis & the 50% who don't.

'Currently between 70-80% of Dore clients complete the program and receive final DST testing'
Have to check, but I expect this is poorer follow-up than you'd get with a school-based programme (the usual for phonics research), & obviously the people who drop out will be more likely to be the ones showing no improvement.

'In fact in the literature on interventions for literacy the opposite is found to be the case i.e. that those with the more severe deficits in both cognitive and literacy performances make the smallest responses in terms of literacy improvement. Thus even when we look at relatively crude data assessing dyslexia based changes with Dore it is immediately obvious that the reverse is true.'
??? which literature and which interventions are they looking at ??? Their paper has no references at all. I have to put lots of references in my work, and only some poor underpaid postgrad will ever read most of it. I wonder what happened to their references?

p11: 'It can be seen that looking at the data in this way seems to suggest that there are highly significant improvements is a range of important cognitive skills related to dyslexia but not in performance
in literacy based tests (OMR, TMS, NWR and OMW).'
This is what you would expect if the intervention did not work. People get better on the things they are trained on, but this does not make them better at the things they aren't trained on but are actually important. (though later p12 onwards they claim actual improvements in useful endpoints for lowest few %).

p11: 'Using this overall analysis literacy scores hold their own over time which is not what is usually seen in practice where there is a tendency to decline down the scale with time and subjects tend to fall further and further behind their peer groups.'
This does not happen using an effective intervention, where catch-up can take place - showing DORE performs worse than phonics [I'm most familiar with phonics-based approaches so I will use them as a comparison - other ideas are available...]

p17: 'Even taking this into account we still see significant improvements in most areas with occasional exceptions in one minute reading and spelling (age group 12.5-16.5) and nonsense word passage (age groups 6.5-9.5, 9.5-1.5). One minute writing does not show much change throughout the age groups but the mean performances are high initially and well into the normal range.'
So some of the most ecologically valid stuff is what isn't changing?
'Ecological Validity' is not about making sure you print your reports on recycled paper. It is a term psychologists use to describe how well what you are measuring fits with the real world. So if you say that your results show that children have improved by 2 academic years in just 6 months, but the only thing you have used to test them on is a block design task and they are not getting better marks at school, the block design task would have poor ecological validity because it would not relate to anything which is important in the real world. It is quite easy to teach people to do better on the sorts of tests used in psychological research. A bit of practice often does the trick on its own. However, making improvements in real life is what's important, and how much you can write in one minute (how fast you can write) may be an important skill in real life - more so in most careers than bead threading.

p19: 'It has been recognised in these peer studies that tests of full reading skill (i.e requiring reading of word passages and comprehending written language) that subjects using Dore are shown to make considerable progress. We expand on this whole issue here as it has caused considerable debate amongst reading academics and has led to inappropriate criticism and ignoring the highly positive outcomes of the Dore research work so far.'
Suggest you read the whole paragraph. Not utterly implausible, but would like a reference or further work to back this up before accepting. Taking such a personal tone isn't usual in an academic paper - looks like someone has hurt their feelings?

p19: 'We can also announce that the majority of children making up this group have been previously formally diagnosed with dyslexia. This fact rather discounts prior criticism of the peer reviewed studies where not all children had a previous formal diagnosis.'
A 60% majority is still pants. Diagnosed by who? 'Peer reviewed studies' are all very well, but when your 'peer reviewed study' leads to 5 resignations from the board of a prestigous journal and 9 published rebuttals, maybe quite a lot of 'peers' disagree with you.

p20: 'Postural stability forms part of the battery of assessments as many studies show that balance and posture can be impaired in dyslexia. In fact Stoodley showed precision balance performance and reading performance are linked across the spectrum. Balance is of course a fundamental area of cerebellar control.'
Co-morbidity between dyslexia & dyspraxia (often finding both in the same person) does not mean that there is one underlying dysfunction. This is a 3rd variable problem. For example, both dyslexia and dyspraxia could come from a genetic or developmental problem with brain development, but different areas of the brain could be affected in both. Assuming that they are directly linked is like saying that meningitis and head injury are the same thing because they share the symptom of headache. Having a motor control problem does not automatically imply 'cerebellar dysfunction'. 

p20: 'This test used here is a rather crude screening tool useful for more significant postural deficits. In fact many studies have shown that it is with precision balance testing and often under dual tasking
conditions where postural control is found to be deficient in dyslexics and ADHD children.'
There's not so much wrong with making up your own 'ultra-precise' tests, psychologists usually have to give tasks which are more difficult than you would do in real life to find out how far your mind can go. But beware of tests with no real-world implications. It is a bit like a cosmetics company promising 'microscopically smooth skin' - unless you usually look at people's skin with a microscope, it may make no difference.

p20: 'Some argue that verbal working memory is deficient due to poor underlying phonological skills. However we are aware of few studies which suggest that phonological training enhances working memory skills.'
There's a fair bit out there on working memory in dyslexia. 5-second search of Web of Science for 'phonolog* AND working AND memory AND dyslexi*' kicks up 154 hits, including stuff like:

Savage R, Lavers N, Pillay V., Working memory and reading
difficulties: What we know and what we don't know about the
relationship
EDUCATIONAL PSYCHOLOGY REVIEW 19 (2): 185-221 JUN 2007

Smith-Spark JH, Fisk JE., Working memory functioning in developmental
dyslexia MEMORY 15 (1): 34-56 JAN 2007

Conti-Ramsden G, Durkin K., Phonological short-term memory, language
and literacy: developmental relationships in early adolescence in
young people with SLI JOURNAL OF CHILD PSYCHOLOGY AND PSYCHIATRY 48
(2): 147-156 FEB 2007

Brambati SM, Termine C, Ruffino M, et al. Neuropsychological deficits
and neural dysfunction in familial dyslexia  BRAIN RESEARCH 1113:
174-185 OCT 3 2006

McCallum RS, Bell SM, Wood MS, et al. What is the role of working
memory in reading relative to the big three processing variables
(orthography, phonology, and rapid naming)? JOURNAL OF
PSYCHOEDUCATIONAL ASSESSMENT 24 (3): 243-259 SEP 2006

Savage RS, Frederickson N
Beyond phonology: What else is needed to describe the problems of
below-average readers and spellers?
JOURNAL OF LEARNING DISABILITIES 39 (5): 399-413 SEP-OCT 2006

Berninger VW, Abbott RD, Thomson J, et al.
Modeling phonological core deficits within a working memory
architecture in children and adults with developmental dyslexia
SCIENTIFIC STUDIES OF READING 10 (2): 165-198 2006

Thomson JM, Richardson U, Goswami U
Phonological similarity neighborhoods and children's short-term
memory: Typical development and dyslexia
MEMORY & COGNITION 33 (7): 1210-1219 OCT 2005

Savage R, Frederickson N, Goodwin R, et al.
Evaluating current deficit theories of poor reading: Role of
phonological processing, naming speed, balance automaticity, rapid
verbal perception and working memory
PERCEPTUAL AND MOTOR SKILLS 101 (2): 345-361 OCT 2005

bored now, but there's more...
My current textbook is book 'Alloway, T. P. & Gathercole, S. E.
(2006). Working memory in neurodevelopmental conditions. Psychology Press.', which would not be a bad place to start should someone want an intro to what's out there. *More* research would be nice (want to fund my PhD?) but there's already a fair bit.
Phonological training may or may not enhance working memory skills (actually, that's very close to my lecture topic next week - maybe I should go & do a bit of the reading to check), but there is a lot of research on working memory in dyslexia.

p27: 'This is a very large study of consecutively completing clients from Dore centres who are essentially receiving no 'special attention' (as is the case with many controlled studies) but are experiencing the typical Dore product.'
This be Stoopid. Of course DORE is 'special attention', & it is hardly beyond the realms of possibility that more effort is being put into reading & writing skills too when people are doing DORE. There is something called the 'Hawthorne Effect', where the act of measuring something & paying extra attention to it in itself causes a change (it's a bit like psychology's version of the Heisenberg Uncertainty Principle). A psychologist called Hawthorne was studying the effects of factory lighting on productivity. When he turned the lights up, people worked more. When he turned the lights down, people worked more. When he took all his clothes off & did the can-can, people worked more (ok, I made that last one up). But whatever he changed, people worked more. Being measured & having changes going on changed the behaviour that Hawthorne was trying to measure. It mucked his study up, but he did get a cool effect named after him - fair swap I think.
What do the authors mean by 'controlled study'? There *is no* control group for comparison, everyone is getting DORE.

p27: 'which later transfer solidly to responses to literacy support and practice.'
Haven't actually shown this in any ecologically valid way. They've shown that for a particular subgroup (lowest %) there's some improvement on some tests, not that this translates into stuff like
doing better in class.

'As Dore involves no specific literacy or cognitive based training of any sort then the improvements are theorised as being directly linked to the observed neurological improvements in cerebellar function.'
They haven't controlled for attention, maturation, placebo, Hawthorne, cohort effects, even proper diagnosis, and a whole bunch of stuff. I'd expect a good GCSE student to have a better understanding of the need for a 'fair test'.

p27: 'However they differ in as much as we have been able to reduce the watering down effect of those subjects with initial normal or superior performances in some tests.'
Why were you treating people who performed above average in your tests?

The whole conclusion is semi-detached from the report's actual
findings, & reads as a sales pitch.

p28: 'The previously published research studies'
You mean the ones that caused 5 Dyslexia editors to resign, have been subject of multiple rebuttals, etc etc? Oh, *those* studies.
Riiight...

p28: 'anecdotally from Dore clients over much longer time spans.'
Why can't they FOLLOW UP. It's a bit trickier, but not actually impossible!

p28: 'One of the original criticisms of the published research studies was that not every child who participated was diagnosed as  dyslexic. In this study we know that at least 60% of subjects were diagnosed with dyslexia prior to attending Dore. It was also found that the dyslexic group showed initial literacy test performances which were slightly weaker than the non-diagnosed group. However the outcomes in both groups were equivalent after Dore. This tends to dismiss initial criticisms and additionally suggests that an initial diagnosis of dyslexia is not essential to benefit from the Dore program.'
60% diagnosed is not adequate to draw conclusions. Still, two in five children who DORE has cost lots of time, money, commitment, practice, parental attention, opportunity to learn useful things or do what normal children do, have NOT been diagnosed with dyslexia. DORE is not a miracle cure. Even its supporters agree that it is hard work & expensive. To put two in five children through that when their difficulties weren't significant enough for a formal diagnosis is IMO unethical. This paragraph scares me. Is *everyone* going to be able to benefit from DORE? Is the plan to make *every* parent feel guilty for not paying lots of money to DORE & making their child throw beanbags around, even if they don't have a diagnosis?

p28: 'What is exciting about these findings is that Dore appears, by stimulating and improving cerebellar function, to impact on core cognitive skills associated with dyslexia. Correcting these learning related skills rather than focussing on training literacy skills directly leads to transfer to literacy acquisition without any
specific intensive training in literacy.'
They have NOT demonstrated this. There has been no demonstration of a 'transfer to literacy acquisition', just some proxy endpoints (~surrogate measures) which may or may not have anything to do with a child's ability to read & learn in a classroom. Is it included specially to be quotable? This is a conclusion detached from the paper's actual findings. 

p28: 'The sad part is that rather than embrace this intervention the reading industry led by the phonological theorists have chosen to severely criticise and ridicule it through manipulation of information and hiding behind authoritative academic positioning.'
FIGHT! FIGHT! FIGHT!
My lecturer may or may not be 'aggressive', but I'm afraid I rather like challenging people's assertions with data or methodology. There's nothing wrong with criticising work that just isn't adequate to show what it claims to show. That's how science works.

I'd love DORE to be proved right, because if it worked it would make life easier for me & a lot of my family. Doing poor research isn't a good way to prove that your treatment works, and by not doing a decent job on the research then they are wasting moneythat could be used for good research, they are making it harder to have good research accepted, and they are wasting time when good research could get an effective intervention to more people sooner.
What does the 'reading industry' mean? DORE costs a lot of money. Most of the stuff done by 'the phonological theorists' is provided through schools for free. Come & look round the department car park, those who don't have bicycles aren't exactly driving BMWs.
'Academic positioning'? I can't hide behind authorative academic positioning. I'm but a humble undergrad, the lowest of the low, not worthy so much as to clean out the cages of the lab rats. But even I know that to show an intervention is effective you need a comparison group.

Whether you think DORE works or not, this latest bit of research won't shed much light on the matter. It should be a disappointment to everyone on any side of the argument. Doing things properly, with a control / comparison group, proper diagnosis, follow-up, and a real-world measure of how the children did in school, isn't that difficult to do. One really good study is worth more than dozens which are so badly designed & run that they can't show anything, however good the treatment is. Whether you are a 'supporter' or a 'sceptic', you should be angry about this waste in an area that urgently needs research.