Accountability in decentralized systems: rethinking how we evaluate schools

This week, Dr. Melanie Ehren expands upon her most recent research on accountability and decentralization in Scotland’s schools. Here she provides background on a project she touched on in last week’s post, in which she explored the connection between testing and teaching in UK schools. This piece also follows a recent post IEN published on Chris Chapman’s work with school improvement efforts and learning partnerships as well as earlier posts on “centralized decentralization” in Singapore. Dr. Ehren has also written about her work in a project on school inspections in Europe and in an earlier IEN post comparing school inspections in England, the Netherlands, Sweden, Austria, Ireland and the Czech Republic.

 

Many countries have given schools more autonomy over the last decade, according to the OECD. Examples are the Netherlands where schools have autonomy in allocating resources as well as in making decisions about curricula and assessments and England where publicly funded independent academies and free schools have freedom in setting pay and conditions for their staff, decide on how to deliver the curriculum, have the ability to change the length of terms, and set their own school hours. The assumption behind greater school-level decision-making is to give power to make decisions to those who have first-hand knowledge of the challenges they face and what they need to solve those challenges. Greater autonomy often goes hand in hand with increased accountability as national governments want to have a safety net in case schools misuse their freedom. The combination of decentralization of decision-making with centralized accountability creates an interesting paradox of freedom with greater control, which can, according to the OECD, lead to improved student outcomes if combined intelligently.

Decentralized decision-making, centralized accountability

In a recent comparative EU-funded study we have started to look at what such intelligent combination might look like, analyzing countries that have recently implemented more locally embedded inspection models, often alongside existing centralized inspections of schools. Newer models are inspired by ‘theory-driven evaluation,’ which take the purposes of the evaluation and how these purposes are expected to be achieved, as a starting point. The foundation for theory-driven evaluation were laid by Peter Rossi, along with Carol Weiss and Huey-Tsych Chen who explained how programme theories and logic models can be constructed to guide an evaluation. Mayne’s ‘contribution analysis,’ also focuses on the causes of outcomes by gathering various perspectives on the degree of impact on observed results. Finally, Michael Quinn Patton’s developmental evaluation provides relevant approaches that help people to learn to think and act as evaluators with a goal of ensuring that evaluations have a lasting impact. This approach encourages those involved in an initiative to constantly assess their work, to reflect on whether it has the intended outcomes, and to make adjustments.

In our EU-funded study we are mapping examples of countries that have incorporated such models in their accountability systems. One of the interesting examples comes from Scotland where the Inspectorate of Education (Education Scotland) is collaborating with other stakeholders in Scotland’s ‘School Improvement Partnership Programme’ (SIPP); an approach that clearly features ‘developmental evaluation.’ In March 2013, the Scottish Cabinet Secretary for Education and Lifelong Learning announced the SSIP as a solution focused approach to tackle the steadfast link between socio-economic deprivation and low educational attainment. The aim of the programme is to make explicit links to strategic improvement planning in schools and local authorities. The School Improvement Partnerships is described by Education Scotland as an approach based on action research and a process of collaborative inquiry. Schools have been asked to take the lead in developing projects around a number of key themes (e.g. differences in achievement by gender, improving transition, differences between small and large schools); projects are also expected to operate across local authority boundaries (cross-sectoral, multi-disciplinary, partnerships with independent sector) and involve a partnership with local authorities, Education Scotland and other agencies. Each partnership is expected to share and try out effective approaches and to indicate what success will look like, with a strong focus on impact in making a difference to young people’s achievement and ultimately life chances. Each of the partnerships develops a structured, collaborative inquiry approach along three phases of preparation (analyzing the context and agreeing on inquiry questions and purposes), exploring the evidence and testing change.

In this approach, evaluation and accountability are key drivers for change and improvement, but they are framed by local questions and local issues instead of centralized frameworks. The programme also offers support to build evaluation and improvement capacity by a trio of named individuals from Education Scotland, local authorities and university researchers who each undertake their own inquiry to explore how the partnership project contributes to the overarching programme inquiry. Education Scotland’s role is to coordinate the development and implementation of the programme, deciding on which partnership gets funding, brokering national partnering and making links across authorities and university researchers and visiting partnerships to identify key challenges in and monitor developments of the programme. Its role is also to provide assistance in collating statistical information about the schools and partnerships to inform their decision-making; supporting a database and communication system to facilitate the sharing of knowledge and resources within and between partnerships, and bringing partnerships together to share experiences at appropriate points within the programme. According to Education Scotland, “This ambitious programme seeks to harness the professional creativity and innovation that exists within the Scottish education community. The programme provides exciting opportunities to rethink roles and relationships within the system and generate and share new practice.”

Expanding the menu of school accountability to include such decentralized approaches allows for greater flexibility in implementing evaluations that fit different purposes and inform system-wide improvement in an ever changing education landscape. The next step in our project is to understand the impact such decentralized accountability models have in improving student outcomes in schools and education networks, but initial results are promising.

For more information:

www.schoolinspections.eu

http://www.springer.com/gp/book/9783319310015

 

 

Life after levels: Is the new Year 6 Maths test changing the way teachers teach?

This week we share a blog post written by Melanie Ehren and Nick Wollaston. Originally published on the IOE London Blog, of University College London, this blog is part of a Nuffield foundation funded research project Dr. Ehren coordinates. The research looks at the Key Stage 2 test in mathematics in England and how the test affects teaching of primary mathematics. The test is administered in year 6 (end of Primary school) and is considered to be high stakes as schools performing below the floor standard are monitored by Ofsted (the Inspectorate of Education), face potential forced academization, and test outcomes are used in (teachers’ and head teacher’s) performance management reviews. The test has undergone changes this year to reflect the new national curriculum, and the researchers have asked teachers (after the administration of the new test) how they are changing their teaching in response to the changes in the test. More info on the project (and a broader introduction) is on the website: www.highstakestesting.co.uk

Here we share the blog post in full. To read the post on the IOE London Blog, click here.

 

Life after levels: is the new Year 6 maths test changing the way teachers teach?

Earlier this month (5 July), the Department for Education published the results of the Key Stage 2 test for 10 and 11-year-olds. The publication was awaited with more anxiety than usual as this year’s test was the first one on the new national curriculum. One of the major changes in the test is the removal of the ‘old’ national curriculum levels 3, 4 and 5, where children were expected to reach at least a level 4. The level 6 paper for the most able children has also gone and results are now reported as ‘scaled scores[1]’. Each pupil now has to achieve at least a score of 100 to reach the expected standard. It seems like a minor change with little impact on how teachers teach mathematics and prepare children for the test, but recent findings from our Nuffield-funded study suggest otherwise.

We interviewed 30 Year 6 teachers in schools performing both below and above the floor standard in Mathematics. Interviews took place prior to the changes in the test in May/June 2015, and again after the changes in the test in May/June 2016. In the interviews in 2015, levels were one of the key topics teachers talked about when we asked them about notable features of the test that would inform their teaching. They explained how each of the two written Maths test papers would start with easy level 3 questions, have level 4 questions in the middle and finish with the difficult level 5 items at the end. This order of questions according to difficulty level would allow the lower attaining children to access the test, according to these teachers, and would build their confidence in answering the questions and their motivation to do well on the test. Teachers tell us in the second round of interviews, how all the questions are now ‘at level 5’ and how some of their lower attaining children stared at them in horror when opening their test booklet, asking them where the easy questions had gone.

Not only does the abolition of levels seem to have an impact on children’s motivation and confidence in test taking, it also appears to have a profound impact on how teachers come to understand and teach mathematics. Prior to the introduction of scaled scores, teachers would talk about gradually building up the level of difficulty when teaching specific mathematical content areas, such as ‘number sense and calculation’, ‘data handling’ or ‘shape and space’. Level 3, 4 and 5 test items on past Key Stage 2 test papers would help them understand the hierarchical nature of mathematics and how to introduce children to, for example, increasingly more difficult calculations (e.g. moving from one step to multistep problems, or from adding and subtracting whole numbers to adding and subtracting decimals). Resources such as Test Base would allow them to access available questions according to content area and difficulty level and they could simply download relevant questions when teaching a specific skill. Now that the levels have been removed, some of the teachers tell us that they just focus on getting all students to perform at level 5 in number and calculation as this is where most of the marks on the test are given and some hardly teach shape and space at all. These teachers also talk about moving towards a more ‘mastery style’ of teaching where they ensure that all students master the basics before they move on to teach more complex skills or other (more complex) content domains, such as algebra or geometry.

It is too early to know how widespread these changes are and the effect they will have on children’s understanding of mathematics. Our study, however, indicates that we need to keep a close eye on the breadth and depth of what our children are learning as some of these changes may be masked by an average single test score.

 

[1] A pupil’s scaled score is based on their raw score. The raw score is the total number of marks a pupil scores in a test, based on the number of questions they answered correctly. The Standard and Testing Agency develops tests each year to the same specification, but because the questions must be different, the difficulty of tests may vary slightly each year. This means that the raw scores pupils get in the tests need to be converted into a scaled score to be able make accurate comparisons of pupil performance over time. Every scaled score will represent the same level of attainment for a pupil each year, so a pupil who scores 103, for example, in 2016 will have demonstrated the same attainment as a pupil who scores 103 in 2017. A scaled score of 100 will always represent the expected standard on the test. Pupils scoring 100 or more will have met the expected standard on the test. In 2016, panels of teachers set the raw score required to meet the expected standard on each test.

Lead the Change interview with Karen Edge

Dr. Karen Edge

Dr. Karen Edge

Dr. Karen Edge is a Reader in Educational Leadership at UCL Institute of Education and Pro-Vice Provost (International) at UCL. Dr. Edge studied Biology and Environmental Science (Bsc) and Higher Education Policy (MA) prior to pursuing a PhD at OISE/ University of Toronto in Educational Administration. Before joining UCL, she worked for the Minister of Education (Ontario), the Centre for Educational Leadership at UC-Santa Barbara, and the World Bank (USA). Dr. Edge’s work continues to focus on bringing policy, practice, and research together to influence understanding and action to improve education for all students and adults in our education systems.

Dr. Edge consults domestically and internationally on a range of strategy, leadership, and research topics. Partner organizations have included DfID, British Council, PLAN (UK), Gates/Hewlett Foundation, and STIR Education. Dr. Edge is a member of the Economic and Social Research Council (UK), the Danish Strategic Research Review Panels, and the Executive Board at UKFIET. She is the past Editor-in-Chief of Educational Assessment, Evaluation and Accountability and has held recent visiting academic appointments in Canada, Chile, and Malaysia. She sits on the international advisory panel for the International School Principal Program in Ontario, Canada. Dr. Edge regularly delivers keynotes and workshops for academic and professional audiences related to leadership, knowledge management, talent spotting, retention, gender and leadership, and system-level reform.

In this interview, which is part of the Lead the Change Series of the American Educational Research Association Educational Change Special Interest Group, Edge shares findings from her Global City Leaders study and her thoughts on educational change:

Based on our observations, we believed quite strongly that this emerging generation of school leaders may be experiencing their careers differently due to their own generational positioning and characteristics. Generational influences on professional practice and careers remains under-researched in the public sector. As a result, these factors may not be fully considered by policy makers and scholars.

This Lead the Change interview appears as part of a series that features experts from around the globe, highlights promising research and practice, and offers expert insight on small- and large-scale educational change. Recently Lead the Change has also published interviews with Diane Ravitch, and the contributors to Leading Educational Change: Global Issues, Challenges, and Lessons on Whole-System Reform (Teachers College Press, 2013) edited by Helen Janc Malone, have participated in a series of blogs from Education Week.

Scan of Ed News: Exploring what the #Brexit vote means for #education in the UK

576d421757868Since the British voted to leave the European Union we have seen a variety of news reports focusing on how the move will affect the British educational system. In this short scan we share some of the conversations we have seen emerge in the past few weeks.

One strand of articles point out what the Brexit vote supposedly reveals about the overall quality of the British education system. According to the Evening Standard, the Chief Executive of Credit Suisse, Tidjane Thiam, said that the level of inequality he witnessed in Britain explained the outcome of the vote. Thiam argued that Britain should raise taxes to counteract the impact of globalization. According to an editorial in the Telegraph, the vote represents “an appetite among young people for a more internationalist approach to education.” As John Walmsley argues, three-quarters of 18-24 year olds voted to remain in the European Union, either out of a desire to live and travel throughout Europe, to help refugees, to effectively battle climate change, or access the European single market. Walmsley writes, “Even in an unstable modern world…young people simply do not have the same concerns with immigration, collaboration and pluralism that older generations have.” According to Peter Horrocks of The Open University, writing for The Times Higher Education (THE), the fact that the outcome of the vote correlated closely with percentage degree attainment shows a pressing need for a more inclusive and diverse higher education sector that offers the flexibility and support that students rightly demand, alongside specific policies to address their particular needs.”

Another strand of articles point to the implications of the Brexit vote. Russell Hobby, General Secretary of the National Association of Head Teachers, argues in Schools Week that the short term impact of the vote will be “distraction and delay,” resulting in the disruption of policies that need the attention and focus of policymakers. Long term, Hobby shares his concern that the vote might serve as impetus for change in both leadership and education policy (for example, will funding for early childhood education remain a priority?). Muhammad Zulfikar Rakhmat, writing for the Huffington Post UK, argues that the most likely immediate implication will be a reduction in the number of EU students studying in the UK, citing the approximately 125,000 EU students studying in UK institutions, and the nearly 3.7 billion pounds and 380,000 jobs they contribute to the economy. These European students might be more likely in the future to choose non-UK universities to study in, such as those in Germany or the Netherlands. Rakhmat speculates that UK universities might end up recruiting more students from developing countries, such as China, India and Indonesia. This shift might influence educational outcomes as well, as many EU students arrive with an advanced educational background. In another article in the Huffington Post UK, Steve Spriggs agrees that British schools might suffer the loss of some 5,000 students from EU countries who currently attend British boarding schools. As he argues, “An exodus of international students would mean a vast net outflow of money from the UK from associated industries: student accommodation, cultural tourist events to name but a few.” However, Spriggs also raises a question about teachers, citing speculation that up to 400,000 teachers might be forced to leave the UK at a time when there is already a shortage of qualified teachers. On July 5th, NHT teachers organized a one-day strike to protest what they see as a crisis point. As Lola Okolosie argues in The Guardian, in addition to the unknown implications of the Brexit vote, teachers are concerned about the loss of jobs, cuts to per pupil spending, and the national commitment to make all schools academies by 2020.

Mark Tucker,  president and Chief Executive Officer of the National Center on Education and the Economy, also provides one take on the implications of Brexit for education and the US election. As he puts it in a recent Washington Post article, “Just as in England, those with the least education are those who have been hurt most by globalization and free trade.  They are most likely, as we see now in the way they are reacting to the candidacies of Donald Trump and Hillary Clinton, to reject not just the leaders of their parties but also the experts they think have ignored them and their interests.”  He concludes, “The Brexit vote is a warning shot across our bow.  Will we hear it?”

-Deirdre Faughey

 

Exploring the rising stature of Estonia’s education system

This week, we shared an article (via Twitter) from our colleagues at The Hechinger Report about the rising stature of the Estonian education system. We also asked several colleagues from Estonia, including Margus Pedaste, Professor of Educational Technology and Vice Dean of the Faculty of Social Sciences and Education, University of Tartu, to comment on the article.

Estonia has not made big changes in the education system, Dr. Pedaste explained, but it has updated the national curriculum. Further, an increased focus on “general transferable competences (e.g. mathematical competence, digital competence, learning skills, communication competence),” has required some changes in teaching practice “even if this change is not often too big.” He also suggested that e-learning software and hardware are often used in Estonian schools today and may have influenced the results.

In responding to the comment that no one in Estonia “would say the school system is doing fine,” Pedaste concurred. “Yes, that’s what we hear very often in Estonia. We are very outcome-oriented and less process oriented and, even if teachers value student-centered approaches in learning, their lessons are still rather teacher-centered.” He added that Estonian teachers, on the whole are veterans, in fact their level of experience in teaching is among longest within OECD countries, at more than 20 years. As he put it “ This experience allows them to use the extensive practical knowledge they have and this might be one of the main reasons of good academic results.”

Pedaste also pointed out, that, as in Finland, most of Estonian teachers have a Master’s Degree, and “subject teachers” usually have a Master Degree in their subject plus a year of teacher training. He added that during the last 10-15 years new integrated Master Programs in teacher education have been developed and regularly updated based on international research-based and innovative ideas. Pre-service education for teachers is also always at least five years which likely contributes to the academic results. However, Pedaste continued, concerns in Estonia include that very often graduates do not want to become teachers, especially in subject areas like science and mathematics. From his perspective, this means that “something is wrong. And it is probably that not enough attention is given to developing soft skills, on collaboration and supporting each other. Too often Estonians are individualistic and competitive while real success and joy comes from collaborative effort.” While recent strategies have emphasized collaboration among teacher, Pedaste concluded “we still need more time to create a cultural change.”

 

Draining The Semantic Swamp of “Personalized Learning”–A View from Silicon Valley (Part 1)

Larry Cuban on School Reform and Classroom Practice

No surprise that a catch-phrase like “personalized learning,” using technology to upend traditional whole group  lessons, has birthed a gaggle of different meanings. Is it  updated “competency-based learning?” Or “differentiated learning” in new clothes or “individualized learning” redecorated?  (see here, here and here). Such proliferation of school reforms into slogans is as familiar as photos of sunsets. “Blended learning,” “project-based teaching,” and “21st Century skills” are a few recent bumper stickers–how about “flipped classrooms?”– that have generated many meanings as they get converted by policymakers, marketeers, researchers, wannabe reformers, and, yes, teachers into daily lessons.

For decades, I have seen such phrases become semantic swamps where educational progressives and conservatives argue for their version of the “true” meaning of the words. As a researcher trained in history, since the early 1980s, I have tracked policies as they get put into practice in schools and classrooms.  After all, the…

View original post 1,328 more words

An Event of Latin American Education: Discussing the Education in the Americas Conference with Daniel Friedrich

Is there such a thing as a distinctly Latin American education or Latin American curriculum? What might such a curriculum look like and who might be able to participate in such an education? With the recent Education in the Americas: Knowledges and Perspectives conference at Teachers College, a number of scholars were able to pursue these and other questions.

This year, the Latin American Studies Association (LASA) conference, where scholars across disciplines gather to share research on and from Latin America, took place in New York City. Having attended the conference in the past, Dr. Daniel Friedrich, Associate Professor of Curriculum and Teaching at Teachers College, Columbia University, had observed that the conference’s education section was limited in both scope and size. With the conference’s 50th anniversary taking place in New York, Friedrich considered the setting a unique opportunity to develop an event in relation to the conference that could offer a more robust examination of the topics of education and Latin America. With an organizing committee that included Professors Regina Cortina, Maria Paula Ghiso, Hank Levin, Nicholas Limerick, and Jackie Simmons. and doctoral student Natalie Flores, he began building the general structure of a conference to be held at Teachers College.

A basic concept organizing the conference would be to offer a venue in which people might gain access to frameworks and perspectives to which they might not typically be exposed. Friedrich suggests that the organizing committee did not simply want to hear people working in Latin America sharing how they were using the same tools with which faculty and students at Teachers College would likely be familiar. Instead, the conference aimed to present ideas about education that might appear unfamiliar to many in attendance. For example, keynote speaker Elsie Rockwell explored the shifting logics of schooling, both for rural and popular education. Her examination of the very question of what school is and how “schooling for the people” and “schooling of the people” converges and conflicts presented something that was both distinctly produced through empirical research based in Mexico but explored around the world, and something that could relate to those in education with little experience in Latin America. At the same time, these objectives meant that the conference had to make several concessions. Nearly half of the panels were held in Spanish, something that Friedrich recognizes excluded some who may have otherwise attended panels save the language barrier. Yet, he contends that as a way to offer alternative voices, this concession was necessary and acceptable.

To build panels and speakers that would satisfy the ambition of presenting new and alternative voices (at least to many attendees) the organizing committee did not open a call for papers. Instead, they invited individuals and groups to present. A kind of dialogic construction of the conference ensued. Organizing members invited those whose work they admired and also observed what disciplines or places might not yet be represented at either LASA or in their own conference. Hank Levin, for example, organized a panel around the experiences of the African diaspora in Latin America.

With a two-day conference program established, participants from many geographies and disciplines came to Teachers College to participate on May 31st and June 1st. Panels ranged from ethnographic studies of migration and education to a documentary on Cuba’s national literacy program. The conferenced helped to solidify networks of scholars who, according to Friedrich, “now know each other and share with each other…which can maybe lead to creating some projects with each other.” Though the organizers had established a number of objectives and many compelling debates occurred, Friedrich believes that much of the success of the conference is the surprising, emergent nature of what came out of the conference and the unresolved or unresolvable nature of these debates. Participants in a revolutionary pedagogies panel debated what specific characteristics about their research were distinctly Latin American. Similarly, a panel on education and violence in Central America explored the possibility of scholars evading their own thinking when thinking of other ways of knowing. In other words, the conversation asked if it is possible to present alternative voices without imposing existing scholarly frameworks onto what is shared. Yet, these debates ultimately provoked further thinking and conversation rather than definitively offering or imposing solutions.