Introduction

In a recent essay titled Innovating Education with MOOC/FLIP, Princeton University professor Mung Chiang thoughtfully discusses Massive Online Open Courses (MOOCs) and some related innovations that are presenting educators with dilemmas and bringing to the surface pressures for possible change in traditional higher educational structures. Chiang sketches some of the recent history of these rapidly developing phenomena and shares some of his experiences developing and teaching a MOOC around computer and social networks.  The core of Chiang’s essay—the ten questions focusing on opportunity and challenges—deals with some of the most important tensions surrounding MOOCs today.  These tensions include huge initial course enrollments and low completion rates (at or under 5%), developing high-quality assessments that are practical at a large scale, emergence of “new” education economics with a potential to increase academy access through free education while also needing sustainable business models, achieving both large scale and personalization, and using technologies in principal roles that are themselves often still developing and incomplete.  He concludes with speculations on MOOC productivity with an equation that describes the enrollment, completion ratio, and graduation size.

My reflection draws upon sociotechnical theory to help conceptualize some interrelated innovations including MOOCs and flipped classrooms.  Is uses a frame that deconstructs these movements and places their possible impacts into a historical perspective.  Like Chiang’s piece, this reflection also focuses on both opportunities and challenges.  It presents a way of viewing innovations like MOOCs and suggests some implications for a research and development agenda to support these kinds of innovations.  The “flipping of classrooms,” something I discuss in more detail later, is less emphasized in this paper than MOOCs are. While Chiang’s MOOC experience involves the rearrangement of the classroom (ex: flipping lecture and individual support), there are some important differences between MOOCs and flipping.  Both have emerged around the same time and are topics of similar reform rhetoric.  However, MOOCs can occur without flipping and vice versa.  They are both interrelated sociotechnically and so their co-occurrence in Chiang’s essay is not accidental.

As a field, we find ourselves often asking some of the same kinds of questions about MOOCs, flipping classrooms, and other related topics that include learning analytics and digital personalization.  These questions, like those asked about other educational technologies include: “Will they live up to their hype or will there be another fad and fashion next year to replace them?; Are they really that different from what we have today?”  “How much of a real impact can these new approaches make on substantive educational problems many have been working on for a long time?” The sociotechnical frame helps with these kinds of questions as it draws upon other kinds of educational movements and looks broadly at not only a specific set of innovations, but periods of change that can involve complexes of technologies and related practice areas. The sociotechnical frame is similar to another that is common today—disruptive innovations—popularized by Harvard professor Clayton Christensen (Christensen, Baumann, Ruggles, & Sadtler, 2006, Christensen, Horn, & Johnson, 2008).  It differs, however, in not only considering how specific technology can be a change agent, but also how a technology is situated within social and technical contexts that can enable and make possible its integration into social practice.
The sociotechnical approach for many issues differs from many approaches common to a particular education sub-discipline.  Because many technologies can impact several parts of an educational system at the same time, the discussion of some topics can draw on several different literatures.  Rather than looking for the kinds of measures common to traditional communities, the sociotechnical lens will often focus on different kinds of effects that reflect the evolution and social impact of the technologies. The result is often a better picture of the shape of the movement of an innovation through society rather than its momentary measurable effects.

Sociotechnical Framing of the MOOC Phenomena

I agree with Chiang in his framing of the issues by using the term disruptive forces. This, I believe, is more appropriate than calling MOOCs disruptive innovations because MOOCs have yet to actually disrupt the business of traditional higher education.  Despite their dramatic increase in popularity, their ultimate impact is today unproven.  MOOCs are, however, putting pressure on traditional structures can be framed in the current context as forces rather than actual disrupters.  However, MOOCs do fit the classical model of diffusing innovations (Rogers, 1995). Diffusing innovations are ones that move through societies and organizations, often rapidly, according to fairly well established patterns of early adoption followed by broader acceptance.  Certainly MOOCs fit this kind of innovation.  However, not all innovations that diffuse are lasting.  Some fade quickly and some have a longer life.  In fact, as Chiang points out in several ways, the path for MOOCs to reshape higher education is not an easy one and his questions focus us on the extent to which we should expect that MOOCs will reshape different parts of education and how quickly.

As we consider MOOCs today, the sociotechnical frame will allow us to see them within a continuum that includes other forms of online learning and blended models (Bonk, Graham, Cross, & Moore, 2006; Horn and Staker, 2011; Means, et at., 2010). Open educational resources (OERs) that include the kinds of videos combined with simple assessments used by Khan Academy; big data and learning analytics (Brynjolfsson, 2012; LaValle, Lesser, Shockley, & Kruschwitz, 2010; Lohr, 2012; Pea, Childress, and Yowell, 2012); and next generation models of learning are related innovations with disruptive potential.   Furthermore, media and data infrastructures (NSF, 2007) that support the high-bandwidth exchange of information are also importantly related to many of these innovations.  While one might have imagined something like MOOCs decades ago, the medium to transmit high-quality video to local devices and the devices to play them took some years to develop so infrastructures are critically related to MOOCs and their sibling innovations.

One issue Chiang raises at the end of his paper that I agree is important and little discussed is the variation in different kinds of subject areas such as the sciences and humanities.  It is easy to talk about education as a unitary field.  From a distance it can seem so.  However, in reality, education is made up of many different overlapping fields. All of these overlapping fields are sensitive to context, but the contexts frequently differ.  If we just consider the profession of teaching and how difficult it would be to move teachers from one job in a particular grade or even kind of school to another, the variation in education becomes clear.  This variation across subjects, grades, and social settings make diffusion and disruption complicated.  Many of the technical challenges that Chiang discusses, including assessments and defining standards for learning, vary considerably across the curricular landscape so that sociotechnical effects will be complicated and uneven. As we consider these topics from the sociotechnical perspective, we see how care should be taken when responding to some of the expectations for radical change or even extrapolate from a handful of existence proofs of disruption in one corner of education to other parts of this vast educational enterprise.  The experience of other innovations in education that had transformative potential is too sobering. All too often, the result has been like the personal computer: less fundamental change in practice from technologies than a longer process of assimilation and adaption into the special circumstances of education (Cuban, 2009).

Four Key Sociotechnical Issues for MOOCs

Below are four key issues that are related to MOOCs.  Educational researchers often look immediately for some direct measure of improvement, whether better learning or greater access.  The sociotechnical frame does not focus so much on productivity as the overall context that innovations are situated in; influenced by and influencing.  Each of these issues then help to highlight how MOOCs can be seen in terms of some of the changes they can bring and some of the important factors that may affect their success.

1. MOOCs are hybrids of social activism, innovation and entrepreneurship and commerce

MOOCs are more than a technology to deliver content.  They are more than new ways of assessing students to be able to scale a course and lower its cost. MOOCs transcend technical innovation.  For the current moment at least, they represent a conflux of interests that includes social activism, innovation and entrepreneurship, and commerce. To understand MOOCs as a sociotechnical movement that has gained so much momentum in such a short amount of time we need to bring together several viewpoints that are not always considered together in contemporary educational research.

For many, MOOCs represent a social movement to democratize education; break down barriers that have excluded underprivileged students from the best educations possible.  It is no coincidence that MOOCs are promoted by the same funders and interests that seek to reform traditional education so that it better serves needy populations than people think it does.  While there may be legitimate questions about how the ways these interests seek to accomplishing that mission, it is clear that for many the goal of MOOCs is to provide access to high-quality education as Stanford Professor Andrew Ng (founder of for-profit MOOC platform Coursera) expressed in April, 2013 by saying “Education should not just be for the elite. Education should be a fundamental human right…Students shouldn’t have to choose between tuition and groceries.”  The idea of opening up education to the masses has been one of the signature hallmarks of the MOOC movement.  It is likely one of the reasons the hype has gotten so far ahead of any reliable research.

At the same time, the MOOC space is about more than lowering barriers.  Those promoting MOOCs also believe in the power of innovation and entrepreneurship to help drive educational improvements.  Along with the MOOC platforms (Coursera, EdX, udemy, etc.), there are now countless innovators developing related products to work with MOOCs.  Many are working with venture funders, developing products, and hoping to capitalize on new opportunities.  This movement now has a vibrant development sector is bringing a stream of new components and products to be used to support distance learning.  Many of these new innovations have broader use so that they help create new opportunities in many educational areas in addition to MOOCs.  Whether or not the MOOC wave continues to have the force it has today, many of the new products developed in response to the current interest in them will be available for other uses.  This innovation sector also aligns with some views of equity because many of the MOOC backers believe that the education enterprise fundamentally needs new ideas and innovation.  This view is also not supported with research, perhaps because there have historically been very few studies of these dimensions.

Clearly the commercial dimensions of MOOCs remain one of the most curious areas to consider. On the surface the MOOC seems to give away what universities have traditionally charged a lot of money for. They come along at a time of scarcity rather than abundance so the emergence of free education may create pressures in the higher educational landscape that has long been stable with universities traditionally providing the labor of a professoriate and their assistants to deliver the instruction. By is using technology to deliver a comparable product to many more people and by using technology to help automate the assessment of students, MOOCs can be universities using tools to cannibalize from their own foundations. However, there are other ways MOOCs can create revenue. Some MOOC providers are looking at charging for the data they collect about students.  In these cases the providers can sell potential employers lists of students with profiles of their class performance so employers can assess not only content knowledge, but competencies such as teamwork and collaboration (Young, 2012).  Some universities are dealing with one of the essential problems with the MOOC platform, lack of a credential, by selling a completion certificate.  I also suspect that before long we will see MOOCs being used not so much as another way to deliver traditional university classes, but as a complementary service.  Perhaps universities will use a MOOC to help provide a survey of a field for those considering further study, an onramp so to speak, that allows students a low cost trial at what might be a larger and more traditional commitment later on.  This model would be similar to the “freemium” software marketing where a basic version of some product is available for free, but the more advanced features require a fee.

2. Flipping Classes and Changes in Professional Roles/Organizational Structures

We see in Chiang’s experience several realignments of traditional roles and organizational structures.  For example, his virtual office hours allow common topics to be presented through pre-packaged video and then the few exceptional questions to be handled by specific messages.  The course Wiki can be used to help develop future materials, which then frees up instructor time for other activities.  These uses of technologies to, as Chiang says, “redraw the boundaries of education” are actually not specific to MOOCs or even to higher education.  They are examples of how technology helps make malleable ways of working(Brynjolfsson & Hitt, 2000; Ciborra, 1993). As sociotechnical changes occur, one of the common side effects is changes in professional roles and organizational structures. For example, personal computers and word processing did more than increase individual productivity.  They impacted the careers of secretaries as executives and others began to write their own correspondence rather than rely on staff for that. More recently, the Internet has now through the introduction a range of communications tools made telecommuting and off-shore services a viable and healthy path in a number of fields.  One of the most salient of these areas today is the switching of class and individual time so that lecture through recorded video becomes an individual activity and class time used for tutoring and extra help.  This is what has been called “Flipping” the classroom by Sal Khan (2012) and others following him. Chiang’s experience combines a MOOC with flipping, but flipping the classroom is also not specific to MOOCs, but initially attempted in the K-12 space and one of a number of blended learning models that innovators are experimenting with involving younger children (Horn & Staker, 2011).

One lesson here is that as MOOCs become established, they can be seen not just in terms of ways to teach more students with the great teachers within a traditional course structure, but as opportunities to redesign educational processes where the roles of teaching assistants and professors and others such as on-call tutoring experts, can be brought to bear as needed. As the complexes of technologies used in MOOCs mature and become more integrated they will likely be able to support even more kinds of work structures. The organizational fluidity that these technologies have brought may very well lead to reconsiderations of how certain organizational elements are designed. While Chiang and most others working with MOOCs have used them in the context of higher education, they might also be adaptable to certain K-12 and community college settings and other professional  learning settings as well.  If a high-school has a need to provide additional coursework in statistics, but no teacher available, a MOOC could be used as another instructional option that would be more self-directed and provide transition opportunities to secondary students. The ultimate implications of this flexibility in terms of institutional design is not yet clear as most educational organizations, including those distance universities that operate fully within a digital infrastructure, are designed according to traditional instructor-course-student constraints.

Another possible scenario is outsourcing of MOOC functions. Based on other history with technologies (Brynjolfsson & Hitt, 2000; Cibborra, 1993; Piety, 2013a), it would be likely that commercial firms will play active roles in providing MOOC elements, including media, assessments, and infrastructure.  The outsourcing of educational support and tutoring will fit well within the MOOC model because of the use of distance technologies.

3. The A/B Test and New Approaches to Research and Knowledge Creation

These disruptive forces are helping to open up new research and evidence possibilities.  Technology can ease some of the traditional constraints that have been long-standing in educational research while also making more evidence collection processes possible. For example, when there are thousands of students in a class and those students are interacting with digital tools, it can be cost effective and easy to collect data in sufficient numbers to support quantitative evidence about the learning process.  This data can be used to generate knowledge that informs the design of the program as well as for data-driven decision making within the course itself (what to reteach and to whom). For many years, educational knowledge creation—research—has been largely dominated by external measures and studies that attempt to understand what educational systems are producing or how things are working within them.  Even qualitative research that includes situating the researcher within the setting is external in terms of the knowledge creation.  In an ethnographic study the researcher will collect various forms of evidence and then afterwards will analyze, compile, and publish.  The kinds of measures used in quantitative research—assessments, surveys, outcome data—are largely collected external to the event that they represent and are for the most part external to learning.

An important factor in the determination of what settings were aligned with qualitative versus quantitative research has been the size of the participants. Studies of learning in classroom and other structured settings tended to attract more qualitative research where the small numbers of students make quantitative studies problematic, while those external to learning environments tend to be able to use larger number of students and quantitative methods, but with less detail about learning. One of the most widely discussed examples of new knowledge creation is the A/B test.  In A/B testing, different approaches to some digital experience are randomly assigned to individuals and data collected about the different conditions without interrupting the natural activity of the user.  The experience tested can be a web page sequence, design layout options, or topics in a course.

Many of the early MOOC successes have been in subjects involving science, technology, engineering, and math (STEM) where decomposition of topics into cognitive sequences with dependencies is possible.  A computer science course, for example, may need to teach students about different kinds of data structures and also different methods for using them. A common instructional question is which order is better.  Understanding the structures entails understanding how to use them.  Understanding the tools for using the structures requires some knowledge of the structures put the tools into context.  When instruction occurs in small discrete elements, this kind of testing can be fast and extremely cost effective.  However, it is not clear which parts of the curriculum this kind of testing can be applied to.  As the scale of the cohesive curricular unit expands, for example with problem-based learning and with many areas of literacy and the humanities, it is not clear how well A/B testing can be used.

At the same time that MOOCs have been ascending in popularity, so has another topic: learning analytics. Learning analytics is an emergent field that draws on many different disciplines to help understand both micro processes of learning that occur within a specific setting and also the systemic processes as students move through different educational engagements (Piety, Behrens, & Pea, 2013).  These kinds of analytics, however, raise new questions about evidence, inferences and rigor as their technology-rich environments bring many new kinds of artifacts for analysis.  There are clickstreams and logs of student activity, peer grading and discussion threads, as well as projects and other forms of student work that can be analyzed both within and external to classes.  Furthermore, many of the more familiar kinds of educational evidentiary artifacts such as tests and test items can be confused with similar tasks created by less formal methods. These tests and items may look externally to be of the same kind as those developed professionally for accountability purposes, but those looks may be deceiving.  When MOOCs and similar offerings are assembled using Open Educational Resources (OERs), these “new” kinds of  assessments will be used and provide useful information, but with very different design affordances in terms of how the data from them can be compared to other evidence forms and what kinds of inferences they support (Behrens, Mislevy, Piety, and DiCerbo, 2013).

4. Importance of Information Architecture

Throughout Chiang’s discussion, including the ten opportunities and challenges he poses are issues that can be described as information architecture. Information architecture is a broad and fairly recent term that can mean both the graphical displays of data, often related to visualization technologies, and the behind-the-scenes ways in which data systems (including assessment systems) are constructed (Piety and Palincsar, 2006). Information architecture can include the coding and classification schemes (Bowker and Starr, 2000) that are seen in the organization of learning standards (called by Chiang “ontologies of knowledge”) and the various groupings and structures of instructional units and materials.  Even the “MOOC supports” that Chiang describes in his experience—the online office hours space and the Wiki, that provide information channels for his students—can be seen as information architectural components. As MOOCs increase the scale of individual offerings from the tens or few hundred students common in traditional higher-education settings to thousands and in some cases tens of thousands of students, these information architectures will play an increasing role in the structuring of the activity system. While a traditional course has a design that is loosely reflected in a professor’s syllabus, that design can be flexible; the professor can alter instruction according to his or her perceptions and the dialog that emerges with the students. In a MOOC-sized event, much of that communication will be facilitated digitally and so the digital artifacts and their architecture will be important for ensuring there are comparable feedback systems and the ability for students to personalize their experience in the way that students can when their access to the instructor is more direct.  It is not clear whether better information architectures could improve the low completion rates in many MOOCs (3-5%), but I suggest this is an area to consider for research.

Beyond Learning: Opportunities and Challenges for NSF, IES, and Education Schools

The wave of innovation that includes MOOCs is moving fast and presenting new, often unproven, ideas.  In a few cases these innovations are here today and merit some response. There are both opportunities and challenges of MOOCs and other recent innovations for NSF, the Institute of Education Sciences, and education schools that receive much of their direction and funding from these agencies.  There will be new research questions.  Some of these will and should focus on the processes of learning in these settings; on productivity differences between MOOCs and more traditional structures and on how different kinds of students participate and thrive in these different circumstances. These studies will likely align with prior research into learning in general and learning using distance education. It seems that at least some of these questions could be taken up by communities in the future academic landscape. What those communities will look like and how these innovations will relate to more traditional lines of inquiry is unclear.

The sociotechnical perspective augments, rather than replaces, other lines of inquiry bout MOOCs.  It is still important to look at this new medium for its impact on learning and teaching.  Different domains of education, for example STEM and humanities educators, will look at MOOCs for what they mean for specific learning goals.  These investigations, however, are likely to not focus on some of the larger and historical dimensions of MOOCs and to explore the directions they may take in the future.  The sociotechnical perspective helps with these questions.  While MOOCs themselves are new, MOOCs are not the first big thing to come along with great promise for education and many questions about impact.  Back in 1980, there was computer programming in the book Mindstorms by Papert.  There were similar promises from the Internet, social networking, and handheld devices.  None of these innovations reshaped education fundamentally.  All, however, had some impact in the complex field of education.  In a few cases they have added new capabilities that have not existed before.  From a sociotechnical perspective, each had an impact on practice and its use in education has been shaped by practice in ways consistent with what has occurred in other sociotechnical revolutions (Bikjer, 1993; Carayannis, Gonzales, and Wetter, 2003).  We can see in MOOCs similarities in terms of the early promise as well as some reasons that promise may be difficult to realize, many of which Chiang discusses.  Will MOOCs impact learning processes?  They probably will in some ways. Will they impact the knowledge base about learning?  They probably will help in that area as well as in our understanding of how some technologies mediate in the learning process (Pea, 1993).  However, if history is a guide, these advances will likely be modest compared to the totality of the educational enterprise.  I believe the terrain of public education is too vast and complex for any single innovation to reshape it and that waves of related sociotechnical developments, of which MOOCs are a recent example, will be dispersed by the complexity of this educational world in the same ways that waves on a shore find new trajectories with rocks and docks and turf. The sociotechnical lens helps us to see how these waves are related to other waves and how their impact is often related and cumulative.  In an period of heightened expectation, this lens can help to provide a balanced perspective, not only for reflection, but for action.  By looking not only at the single hallmark innovation, but the interrelated sociotechnical progress, this view will help funders and educators to be able to anticipate some future directions this process can take so that they can do more than study, they can innovate and take advantage of the real opportunities that this movement provides.

Another consideration that needs to be considered is the nature of evidence that can support scientific claims. Their rapid growth and change makes them important and also difficult to study as a program or intervention that should be associated with specific learning or educational outcomes.  While, these outcomes measures are important, in many cases the innovations are changing as they are being studied and the organizations are also changing as they adapt to these new innovations so that the a clear assessment of impact may take some time to be realized. In responding to these innovations, academics and researchers may need to consider not only their impact on productivity and completion rates as Chiang suggests, but also the broader systemic effects (Piety, 2013b).  These systemic effects can include adoption rates, relationship between MOOCs and student future matriculation, cost effectiveness of MOOC approaches, equity issues, and other changes that the MOOCs are implicated in.

There is much about MOOCs to be learned and much about learning that could influence the design of different MOOC approaches.  The sociotechnical lens provides a way to conceptualize the innovation broadly to see how its evolution is related to other important developments in technologies that are impacting practices that involve learning. This understanding will be helpful in shaping responses in higher education, human capital pipeline development, and K-12 learning.

References

 

Bijker, W., E. (1995). Of bicycles, bakelites, and bulbs: toward a theory of sociotechnical change. Cambridge, MA: MIT Press.

Bonk, C. J., Graham, C. R., Cross, J., & Moore, M. G. (2006). The handbook of blended learning: Global perspectives, local designs. Pfeiffer San Francisco, CA.

Brynjolfsson, E. (2012)  Erik Brynjolfsson on Big Data: A revolution in decision-making improves productivity – MIT Sloan Experts. Retrieved November 26, 2012, from http://mitsloanexperts.mit.edu/erik-brynjolfsson-on-big-data-a-revolution-in-decision-making-improves-productivity/

Brynjolfsson, E., & Hitt, L. (2000). Beyond Computation: Information Technology, Organizational Transformation and Business Performance. The Journal of Economic Perspectives, 14(4), 23-48.

Carayannis, E. G., Gonzalez, E., & Wetter, J. (2003). The nature and dynamics of discontinuous and disruptive innovations from a learning and knowledge management perspective. In L. V. Shavinina (Ed.), International handbook on innovation (pp.116–138). Elsevier Press.

Chiang, M. (2013) Innovating Education with MOOC/FLIP. Working paper. Princeton University EDGE Lab. http://scenic.princeton.edu/files/MOOC_FLIP_thoughts.pdf.

Christensen, C. M., Baumann, H., Ruggles, R., & Sadtler, T. M. (2006). Disruptive innovation for social change. Harvard business review, 84(12), 94

Christensen, C., Horn, M., & Johnson, C. (2008). Disrupting class: How disruptive innovation will change the way the world learns. New York: McGraw-Hill.

Ciborra, C. (1993). Teams, Markets, and Systems: Business Innovation and Information Technology. Cambridge: Cambridge University Press.

Coursera (2013) Coursera Announces Professional Development Courses to Facilitate Lifelong Learning for Teachers. Retrieved May 21, 2013, from http://blog.coursera.org/post/49331574337/coursera-announces-professional-development-courses-to

Cuban, L., & Cuban, L. (2009). Oversold and underused: Computers in the classroom. Harvard University Press.

DiCerbo, K. E., & Behrens, J. T. (2012). Implications of the Digital Ocean on Current and Future Assessment. In R. Lissitz & H. Jiao (Eds.) Computers and Their Impact on State Assessment: Recent History and Predictions for the Future. (pp. 273-306). Charlotte, NC: Information Age Publishing.

Horn, M. and Staker, H. (2011). The Rise of K-12 Blended learning.   Innosight Institute. Retrieved on September, 7, 2011 (http://www.innosightinstitute.org/innosight/wp-content/uploads/2011/05/The-Rise-of-K-12-Blended-Learning.pdf).

Khan, S. (2012). The One World Schoolhouse: Education Reimagined. Twelve.

LaValle, S., Lesser, E., Shockley, R., M, H., &  Kruschwitz, N. (2010, December 21). Big Data, Analytics and the Path From Insights to Value. MIT Sloan Management Review.

Lohr, S. (2012, February 11). Big Data’s Impact in the World. The New York Times. Retrieved from http://www.nytimes.com/2012/02/12/sunday-review/big-datas-impact-in-the-world.html

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Evaluation of Washington, D.C.

National Science Foundation. (2007). NSF’s cyberinfrastructure vision for 21st century discovery. Retrieved from http://www.nsf.gov/pubs/2007/nsf0728/nsf0728_6.pdf

Pea, R.; Childress, S.; and Yowell, C. (2012).  The Demand for Education Data Scientists working paper by H-Star Institute. Stanford, CA. Stanford University.

Pea, R. D. (1993). Practices of distributed intelligence and designs for education. Distributed cognitions: Psychological and educational considerations, 47-87.

Piety, P. Behrens, J. and Pea, R. (2013). Educational Data Sciences  and Interpretive Skills. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

Piety, P.J. (2013a). Assessing the Educational Data Movement. Teachers College Press, New York.

Piety, P.J. (2013b). Beyond Productivity: Considering Systemic Effects alongside Outcome Measures.  Comments for IES/NSF Common Standards for Research and Development Proposals.

Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York: Free Press.

Young, J. (2012) Providers of Free MOOC’s Now Charge Employers for Access to Student Data. Chronicle of Higher Education (December 4, 2012).