Jeanne Marie Iorio & Susan Matoba Adler’s (2013) commentary “Take a Number, Stand in Line, Better Yet, Be a Number Get Tracked: The Assault of Longitudinal Data Systems on Teaching and Learning” takes aim at a number of issues, including the growth in educational data in state databases and the unique numbers to identify and collect the records for students and teachers saying:
Statewide longitudinal databases are becoming sources for decision-making by policymakers, administrators, and teachers. These databases are tracking children and teachers, reducing the performance of children and the work of teachers to numbers. We call for an end to the obsession with the quantitative and hope for a rethinking of assessment and teaching practices that trust children and teachers as capable and critical to learning, teaching, and assessment.
The authors also question an organization called The Data Quality Campaign (DQC) linking the DQC to a litany of concerns about how collecting and using data is changing education. I agree with Iorio and Adler this is an important topic. In a new book titled “Assessing the Educational Data Movement” published by Teachers College Press (Piety, 2013) I explore this time of change in American education and look specifically at relationships between different organizations involved in promoting the use of educational data. I studied the DQC in some depth and highlighted its mission that, in its own words, “supports state policymakers and other key leaders to promote the development and effective use of statewide longitudinal data systems.” I featured the DQC for two reasons. First, it is historically significant that an organization focused on the quality of educational data would emerge during this time. The DQC began around 2005, which makes it a relative newcomer in a field where some communities have existed for over a century. Second, the kind of information that the DQC provides is unique and important for understanding how data is and can be used. Much of educational research has historically been regional. Much of it, rightly so, has focused on districts, schools, teaching, and students. This has left a gap in understanding what is occurring in and across the different states and the DQC’s work and publications provides important insights a growing national educational information infrastructure. Understanding the role of organizations playing a role in the formation of national policy should be fair game. Being fair, as scholars, about their work and intentions is equally so.
With over 100 partners—including the Council of Chief State School Officers, The American Association of School Administrators, Consortium for School Networking, and the American Federation of Teachers—the DQC is now a fixture in the conversation about data in American education. It has received support from large philanthropies, including the Bill and Melinda Gates Foundation, as well as strong support from the current and past Administrations and members of congress. It is a non-commercial entity with little connection to vendors. The DQC frequently hosts forums that bring together different points of view around data questions. I have attended over a dozen such events on topics that include data privacy and governance, using data to understand college and career-ready educational processes, and teacher effectiveness. I have never heard discussions about reducing any part of education to numbers. Rather there has been a deepening appreciation for the enormous challenges associated with making education a more information-driven field. A common phrase the DQC uses in its advocacy as been “data should be used as a flashlight not a hammer.” Recently the director of the DQC, Aimee Guidera (2012), wrote a commentary for Education Week titled “Moving Beyond the Single Data Point” that argued against releasing teachers names with value-added scores as had been done in New York and Los Angeles.
To be clear, I believe that asking questions about educational data is critical. The decisions made today in terms of infrastructures impact our children’s future and deserve a healthy debate. While there may not be technical limitations to including broad kinds of evidence in these systems, as Iorio and Adler suggest, those areas that are developed first will be well positioned later. The use of value-added models for math and literacy is directly related to those subjects being tested in response to NCLB. Whether one supports these measures or not, the linkage between the collection of certain kinds of data and its subsequent uses seems clear. National conversations about what kinds of data to include should include many voices, the DQC and others with similar views. Before scholars or advocates attempt to place their views on a moral high ground, they might also consider the validity of other perspectives. Some believe the collection of certain kinds of data distorts education and increases inequity. Others believe the lack of data obscures educational practice where inequity is embedded. Education is a vast, complex endeavor and both of these interpretations are supported by at least some evidence. How we frame these differences and treat other perspectives have implications for the kinds of relationships we have later on. If we want to make real progress in our field, it is important to engage in a thoughtful and inclusive debate. I respect, but do not always agree with, Iorio and Adler’s point of view when they say:
Reduced to unique identifiers, children and teachers are positioned as people with no voice and no valid contribution to how education might be imposed upon each of them…[T]he teacher is explained through her teacher education program, her professional development experiences, and the performance of her students on a quantitative test… How can we reduce the complexity of the child and teacher to numbers? Why are we ignoring the richness of relationships between teacher and student to quantify performance of children and credentials of teachers? How can policy and administrative decisions be based on this small and limited quantitative facet of the people a school system serves? Have the ease of numbers and belief that quantitative work is not biased become an easy way to forgo thoughtful decision-making?/
Strong feelings about educational data are understandable because they can be so consequential for students and teachers. With each advance of technology comes concern about the changes they bring. It is hard to find a technology—from the printing press to steam engines to television—that has not brought apprehensions about what is slipping away. Students of Plato may recall his dialog “Phaedrus” written in 370BC where King Thamus debates with the Egyptian God Theuth who had given man the new technology of writing. Plato (Hackforth, 1972) posited that the youth would lose their powers of memory saying:
[F]or this discovery of yours will create forgetfulness in the learners’ souls, because they will not use their memories; they will trust to the external written characters and not remember of themselves. The specific which you have discovered is an aid not to memory, but to reminiscence, and you give your disciples not truth, but only the semblance of truth; they will be hearers of many things and will have learned nothing; they will appear to be omniscient and will generally know nothing; they will be tiresome company, having the show of wisdom without the reality.
While a different time and technology, the tone is similar. While many appreciate the aesthetic of old cars, period movies, and the pastoral charm of simpler days, few people reading this would consider living without refrigeration, personal computers, or modern vehicles with Bluetooth capabilities. Progress and change happen. My book argues that the use of data is just one way in which technology is moving education forward, but that the ways it is improving education are not always clear. I argue that much is missing—specifically an understanding of educational practice in classrooms and schools—that can inform the currently oversimplified views of educational data.
In the modern era, the development of digital records has given rise to fears about people being reduced to numbers and overreach of “the state.” Each year, most adults have various forms of communication that include a social security number (SSN). When the Social Security Act was new, many people were similarly alarmed by the idea of ‘reducing people to numbers.’ Now, the SSN is prosaic and these numbers are usually assigned shortly after birth. If today the US Government entertained doing away with these common identifiers, many people would be alarmed and wonder instead how matters of public safety and finance could be managed without good record keeping?
In the 1960s, before optical character reading and electronic commerce, many checks and bills began to be printed on forms that could be read by the computers of that era. Many came with a statement that read “DO NOT FOLD, SPINDLE, OR MUTILATE.” (Spindling was a process of piercing paper documents so they could be strung together for filing). The machines used to read those documents would jam when the paper wasn’t just right. In those early days some protesters would deliberately fold, spindle, mutilate (and even staple) those documents to upset the machine of the modern state. Successive waves of newer technology rendered those concerns moot. While that phrase about folding, spindling, and mutilating is now part of our cultural history (Lubar, 1992), we can find at least two messages in it today that relate to what Ioro and Adler wrote. First, some of the immediate fears about new technology fade quickly with time and technological advances. The concerns about student and teacher identifiers may then also fade, although some of the issues that are implicated in data collection such as school accountability and evaluating teachers may not. The second has to do with how we represent others in our work. In advocating for our own point of view do we unwittingly fold, spindle, and mutilate the message and well intentioned work of others? I confronted this question in my book when discussing a number of organizations and projects that have attracted some polarizing views. Using principles of qualitative inquiry, I consulted with the organizations I wrote about, including the DQC, to give them an opportunity to help represent their work in what I wrote. Some may not agree with every part of this book, but will hopefully believe the approach is fair, encourages thoughtful discussion, and for these reasons has value.
The world of educational data presents challenges for study. As Coburn and Turner (2011) said “In many ways, the practice of data use is out ahead of research. Policy and interventions to promote data use far outstrip research studying the process, context, and consequences of these efforts.” The US Department of Education (2010) after a large study of this area stated, “The data systems themselves, the assessment data that populate district data systems, and school and district practices around data use are all changing rapidly.” Educational researchers have some catching up to do. Researchers from other fields may find educational data a challenging area for them as well. Education is unlike other fields, not only because it is a social practice, but it is social on many levels. These realities show up in the data; in its quality and consistency and in how difficult it is to use for comparisons. There are indeed many good and important questions about what kinds of artifacts are best to collect and how to integrate them into practices. As my book details, teachers and other practitioners have been far less represented in the design of educational data infrastructures than communities representing institutional interests. There is an opportunity for Iorio and Adler and those who agree with them.
One thing that Plato missed almost 2400 years ago and others are missing today is how new technologies work beyond the level of individuals. Historically, sociotechnical revolutions from the printing press to television to the Internet have changed professions and organizational structures. Tome and again, these kinds of revolutions can occur on many levels simultaneously, including individuals, teams, markets, and systems (Brynjolfsson, 1993; Cibbora, 1993). The history of writing (Harris, 1986) shows it had great impact on administrative structures and commerce. Not surprisingly, much of the emphasis around data in education has been on how individual practices can be assessed and changed, including with students, teachers, and schools. The organizational and professional changes are less considered. Blended learning school designs—new ways of structuring schools—are also emerging during this data movement. The emergence of the DQC is then not so surprising after all.
Organizations like the DQC, while they may not have all the answers, exist today as important parts of the national conversation whose work has represented the goals and interests of their stakeholders. Unfortunately, at the time of this commentary, the list of DQC partners does not include any colleges of education. As Gummer and Mandinach (2013) recently discuss, even though data matters for teachers in their work, there has been little attention to developing even basic literacy skills by colleges of education. There are fresh signs of progress, however as a number of leading universities are developing programs in learning analytics and data sciences and new associations, including an International Educational Data Mining Society and Society for Learning Analytics Research, have emerged with conferences and journals.
While it is popular to refer to other countries (ex: Finland, Singapore, etc.) in terms of educational models, in terms of data our situation is very American. Constitutionally, education is largely a state matter and the history of local control gives us a highly decentralized management of information when compared to other countries. At the upcoming American Educational Research Association (AERA) meeting in San Francisco I have organized a symposium titled “Big Data American Style.” Our discussant is the incoming president of AERA and the authors of papers include faculty from Stanford University and Teachers College, A Vice President for Pearson’s Center for Digital Data, Analytics & Adaptive Learning, The first Chief Privacy Officer at the United States Department of Education, and the DQC. The DQC’s paper is titled “The 4 Ts of State Data Systems: Turf, Trust, Technology, and Time: Policy Perspective on Empowering Education Stakeholders with Data.” For many in educational research, the DQC will bring a new and important perspective. Will it be completely representative of all of the interests in educational data? Probably not. None will. We come together in the American spirit of encouraging diverse points of view to help collectively inform our understanding. Practitioner oriented researchers are welcome to come and ask thoughtful questions and help campaign for high quality discussions about educational data.
Brynjolfsson, E. (1993). The productivity paradox of information technology. Communications of the ACM, 36(12) 66-77 .
Ciborra, C. (1993). Teams, markets and systems: Business innovation and information technology. Cambridge: Cambridge University Press.
Coburn, C. E., & Turner, E. O. (2011). Research on data use: A framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9(4), 173-206.
Guidera, A (2012). “What’s in a Name? More Than a Single Data Point” Commentary in Ed Week August 6. 2012. http://www.edweek.org/ew/articles/2012/08/08/37guidera.h31.html.
Hackforth, R. (Ed.). (1972). Plato: Phaedrus. Cambridge University Press.
Harris, Roy. 1986. The Origin of Writing. London, England: Duckworth.
Iorio and Adler (2013). Take a Number, Stand in Line, Better Yet, Be a Number Get Tracked: The Assault of Longitudinal Data Systems on Teaching and Learning. Teachers College Record March 8, 2013.
Lubar, S. (1992). Do Not Fold, Spindle or Mutilate: A Cultural History of the Punch Card. Journal of American Culture, 15(4), 43-55.
Mandinach, E. B., & Gummer, E. S. (2013). A Systemic View of Implementing Data Literacy in Educator Preparation. Educational Researcher, 42(1), 30-37.
Piety, P. (2013). Assessing the Educational Data Movement. Teachers College Press. New York, NY.
Postman, N. (1993). Technopoly: The surrender of culture to technology. Vintage.
U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (2010), Use of Education Data at the Local Level From Accountability to Instructional Improvement, Washington, DC
Dr. Piety is a national expert in educational data. He is on the faculty of the Robert H. Smith School of Business at the University of Maryland where he teaches information and database technologies, cloud computing, and social media. He is the author of Assessing the Educational Data Movement, a book on using data to improve school success.
Leave a Reply