A new way of seeing collaboration in the process of designing interactive digital educational artifacts
The case of the REMASCO project to redesign and reinvent the digital school textbook
Les groupes de conception doivent être créatifs et inventifs. La performance d'un groupe de conception inventif est essentiellement mesurée en fonction de trois facteurs : le nombre total de contributions (idées), l'égalité du taux de contribution et l'exploration de nouvelles idées. Les technologies numériques remettent en question les utilisations, la forme, la conception et les interactions des artefacts d'apprentissage numériques. Nous avons identifié que dans l'écosystème éducatif, il y a un manque d'inventivité lors de la conception des versions numériques des artefacts d'apprentissage. Nous proposons un nouveau modèle d'interactions collaboratives qui favoriserait le niveau d'inventivité dans le processus de conception des artefacts d'apprentissage numériques. Pour tester notre hypothèse, nous avons choisi le manuel scolaire numérique comme un artefact d'apprentissage numérique, car il peut être décrit comme une simple numérisation de manuels imprimés, avec un manque considérable d'inventivité. Nous avons conçu une étude expérimentale, formant six groupes de conception hétérogènes composés de tous les acteurs de l'écosystème éducatif. Ainsi, nous avons tracé et quantifié toutes les interactions en ligne entre les participants de chaque groupe de conception. Nous avons constaté que 1) les interactions collaboratives dans le processus de conception des artefacts d'apprentissage améliorent significativement la qualité de la circulation des idées par rapport aux groupes de conception coopératives qui ont divisé le travail et 2) une corrélation positive entre le coefficient de collaboration de chaque groupe de conception et l’inventivité de son prototype conçu. Ces résultats suggèrent que le modèle d'interactions collaboratives peut créer un équilibre entre l'espace individuel d'une personne et l'espace collectif du groupe de conception, facilitant les interactions collaboratives au sein du groupe et favorisant ainsi l'inventivité.
Design groups need to be creative and inventive. The performance of an inventive design group is essentially measured based on 3 factors : the total number of contributions (ideas), the equality of contribution rate, and the exploration of new ideas. Digital technologies challenge the uses, form, design and interactions of digital learning artifacts. We identified that in the educational ecosystem, there is a lack of inventiveness when designing digital versions of learning artifacts. We propose a new Collaborative Interactions Model that would foster invention in the design process of digital learning artifacts. To test our hypothesis, we have chosen the digital school textbook as a digital learning artifact, because it can be described as a simple digitization of printed textbooks, with a considerable lack of inventiveness. We designed an experimental study, forming six heterogeneous design groups composed of all the actors in the educational ecosystem. We traced and quantified all the interactions between participants in each design group, finding 1) that collaborative interactions in the design process for learning artifacts significantly improve the quality of the flow of ideas compared to cooperative design groups that divided the work, and 2) a positive correlation between the collaboration coefficient of each design group and its invention output. These results suggest that the Collaborative Interactions Model can create an equilibrium between a person's individual space and the collective space of the design group, facilitating collaboration interactions inside the group and thus promoting inventiveness.
1. Introduction
From Sidney L. Pressey’s teaching machine in the 1920's (Benjamin, 1988) to today's digital learning artifacts, the educational ecosystem has been trying to improve education with new technological artifacts for decades. But these efforts have fallen short when trying to transform the learning experience (Tricot, 2017 ; Cuban, 2008 ; 2001). Research context aiming to measure performance and inventiveness in networks of people ranges from brainstorming sessions with 2 to 5 participants per team (Woolley et al., 2010 ; Gersick et Hackman, 1990) to interactions between people in entire companies (Waber et al., 2010). In this context, we can identify two ways of working interactions : 1) cooperation, where partners divide the work, solve the subtasks individually or in subgroups, and then gather the partial results to obtain the final result, and 2) collaboration, where partners do the entirety of the work together without dividing it (Helle et al., 2006 ; Dillenbourg, 1999).
In the best case, companies and organizations that design and develop digital learning artifacts are divided into teams or sub-groups to create the different parts of the learning artifact, so that they go through a cooperation working process inherited from the industrial era (Durkheim, 2014). Cooperation has been shown to be very efficient throughout history to accomplish repetitive tasks faster and then later to assemble the parts at the end (Durkheim, 2014). But this way of working penalizes the equality of participation amongst partners in the whole group. Penalizing the equality of participation can impact the inventive performance of the group, as equality of participation in a group of individuals is a huge predictor of creative or inventive collective performance (Pentland, 2015 ; Engel et al., 2014 ; Onnela et al. 2014).
1.1. Overview
In this paper, we investigate whether a collaboration system approach to the design process of digital learning artifacts – as opposed to a cooperative approach – can improve the invention process of a group made up of the different actors within the educational ecosystem. To address this question, we developed a new Collaborative Interactions Model that tries to create an equilibrium between the individual space of each actor within the group and the collective space of the whole group.
1.2. Background and related work
Much of the research around cooperation and collaboration working interactions was made by focusing on the medium that supports these kinds of interactions (Holliman et Scanlon, 2006 ; Mcalister et al., 2004 ; Jones et al., 2000), but does not address the impact of these interactions on the working process. Some research has been conducted on the distinction between collaboration and cooperation interactions in group work (Dillenbourg et al., 1995 ; Roschelle et Teasley, 1995). From this research, Dillenbourg et al. (1995, 190), based on the work of Roschelle and Teasley (1995), make the same distinction that we do between cooperation and collaboration, but note that in the written research there is no agreement over the distinction of the two terms.
Regarding the interactions (indistinctly cooperative and collaborative), Pentland (2015) proposed a new theory that he calls Social Physics, inspired by Auguste Comte, the father of modern sociology, who coined the phrase back in the 19th century and aspired to explain social reality by developing a set of universal laws. The theory mathematically describes connections between the flow of ideas and information among individuals in a group, and the individuals’ behavior (inventiveness, creativity, social learning, norms, etc.). There are two important parts in the theory : 1) Idea Flow within social groups is composed of two elements : exploration (the process of finding new ideas), and engagement (getting every person in the social group to coordinate and equally participate inside the group) ; and 2) Social Learning, which describes the process of adopting ideas, and how this adoption can be accelerated and shaped by social pressure.
One of the problems occurs when groups create "echo chambers", where the same ideas often circle around to a specific part of the group again and again (Pentland, 2015), decreasing the collective inventive process (Paulus et al., 2015).
Pentland (2015) sees the flow of interaction and ideas in Social Physics as a social process and collective rationality instead of an individual process and individual rationality (Woolley et al., 2015 ; Kahneman, 2011). Here, the dynamics of the exchange are more important than the knowledge and experience of the individual participants (Pentland, 2015).
2. The collaborative interactions and flow of ideas model
Based on Pentland’s (2015) Social Physics Theory, we propose a Collaborative Interactions Model to promote effective collaboration in a design group regardless of its size. There are three important elements illustrated below (see Figure 1) :
Figure 1. The Collaborative Interactions Model presents the three main elements for promoting an efficient collaboration process : energy, engagement, and exploration.
The first element is Energy. Energy is defined by the frequency of communication between two individuals in a group. In the Social Physics Theory, the key to high performance lies not in the content of a group’s discussions, but in the manner in which it was communicated and the frequency of this communication (Pentland, 2015 ; Onnela et al., 2014).
The second element is Engagement. Engagement is defined by the coordination and the distribution of the flow of ideas within the group. Engagement in the Social Physics Theory seeks to understand how the distribution of the flow of ideas and information within a group of individuals translates into behavioral changes (Pentland, 2015, 5) like promoting the invention process of the group (Woolley et al., 2010).
The third element of the model is Exploration. As each individual of the group comes from their own world, they see the artifact to design as a “boundary object” (Star et Griesemer, 1989), bringing new information from their own world into the group. In a performant exploration process, each individual has to bring different information to the group (Montjoye, et al., 2014). As illustrated in Figure 1, the danger is when individuals enter an “echo chamber”, where there is no exploration process and the same information revolves again and again in a loop inside and outside the group.
To follow the flow of ideas of the group we also propose a Flow of Ideas Model that will complement the Collaborative Interactions Model described above, based on Cerisier (2014) Instrumental Genesis and Social Interactions Model (Cerisier, 2011, 114). This model is illustrated in Figure 2.
Figure 2. The Flow of Ideas Model used to understand how ideas flow inside a group.
The model is represented by two big zones, with the upper circle representing the individual, and the lower circle representing the group. Each circle has two main zones. To explain each zone, we can follow the flow of an idea, represented by a cube, within a group of individuals : 1) an idea starts in the exploration zone, which we also refer to as the “idea invention zone” ; the idea is new both for the individual and for the whole group. 2) The idea can move to the next zone, the collective space of the group, where the idea is available to all the individuals of the group. 3) Once the idea is in the collective space of the group, it has the potential to be adopted by an individual, in what we call the “adoption potential zone” or the “novelty zone”, where the idea is a novelty for an individual. 4) If the individual adopts the idea, the idea moves to the individual space zone, in this stage the idea is both in the individual space and in the collective space of the group at the same time. We used this model to trace the flow of ideas of a design group during its design process.
3. Experimental Research design
We designed an experiment to investigate a set of hypotheses regarding the impact of collaborative and cooperative interactions on the flow of ideas within a design group and its collective invention process. The following subsections describe our hypothesis in greater detail, and how the experiment was planned and executed.
3.1. Hypothesis
The main hypothesis is that a collaborative interaction process – as opposed to a cooperative one – in the design process of digital learning artifacts can improve the collective invention process of a diverse design group that is made up of different actors within the educational ecosystem. More precisely, we hypothesize :
-
H1. Collaborative design groups will have a higher flow of ideas than cooperative design groups, measured by a Collaboration Coefficient defined as the product of the Energy and the Engagement elements from our Collaborative Interactions Model. In this context, we hypothesize that a cooperative way of working impacts negatively both the Energy and the Engagement elements.
-
H2. A higher Collaborative Coefficient is correlated with a higher Invention Coefficient, which is computed based on the calculation of new patented inventions and is defined by the product of Novelty and the Perception of Use of each design solution.
3.2. Participants
We assembled six heterogeneous design groups out of the whole educational ecosystem. We recruited nine different profiles per design group. Participants were volunteers and were recruited through partners of the REMASCO project (different partners of the French National Education ecosystem). They were randomly assigned to the six groups, and counter-balanced across conditions (see below) with respect to their gender (female 50 % ; male 50 %).
3.3. Conditions
We formed three design groups to work with the experimental (COLLABORATION) condition and three design groups to work with the control (COOPERATION) condition. Each design group was constituted with the same nine profiles from the educational ecosystem. A collaborative online platform was carefully predefined to be appropriate to each condition. In the case of the experimental condition (COLLABORATION), the three groups worked with the whole Collaborative Interactions Model. In the case of the control condition (COOPERATION), the other three groups worked partially with the Collaborative Interactions Model ; they were divided into two sub-groups at the beginning of each design stage (see below) to reunify their ideas at the end of each design stage, simulating the cooperation working process that we defined above. As the conditions were randomly assigned to the six groups, we maintained the following configuration in the results below :
-
experimental condition (COLLABORATION) : groups 1, 3 and 4 ;
-
control condition (COOPERATION) : groups 2, 5 and 6.
3.4. Protocol
The experimental protocol procedure lasted three months and was divided in three design stages : 1) Discovering, which was aimed at discovering the problem, 2) Appropriating, which encouraged participants to appropriate the problem and generate a new solution, and 3) Making, which aimed to build a prototype of the solution. The experiment was part of a real industrial project called REMASCO (to reinvent the digital school textbook), which aimed to design 6 prototypes of digital school textbooks with contribution from all actors of the educational ecosystem ; participants were not informed about the experimentation, and thought they were working for the REMASCO project. Participants did not know each other, and their challenge was to work with their group partners completely online (to avoid the impact of hierarchies and to facilitate tracing the flow of ideas within each group) for three months. As we said before, participants were not aware of the experimentation protocol (they only knew that they were working for the REMASCO project) and only knew that they should follow the instructions to respect the 3 design stages cited above. We facilitated this design process through instructional videos with design methods that were introduced at the beginning of each design stage.
After the three month-long design process, we tested the six prototype solutions that each group designed in order to obtain an Invention Coefficient per prototype. These tests were made by 66 real potential users (high school students and teachers) and individuals working in the design of digital learning artifacts. In this context, 396 individual evaluations of each prototype were made using a framework based on the calculation of new patents, evaluating the level of Novelty and the level of the Perception of Use of each prototype solution.
4. Results
During the three months in which the individuals of the six groups interacted completely online to design a prototype solution of a digital school textbook, we traced all interactions and analyzed them using Network Theory. This allowed us to obtain a Collaboration Coefficient from each team by tracing the Energy element (quantity of interactions) and the Engagement element (the distribution of the information in each group).
As we explained above, we also obtained an Invention Coefficient for each prototype ; this was calculated by evaluating the level of Novelty and the level of Perception of Use in each prototype, using a framework based on the calculation of new patented inventions (Sarkar et Chakrabarti, 2011).
4.1. The quality of the flow of ideas
We calculated the quality of the flow of ideas for each team based on the Social Physics Theory (Pentland, 2015) and from our Collaboration Interactions Model, obtaining what we called a Collaboration Coefficient calculated as follows :
Collaboration Coefficient (CC) = Energy (E) x Engagement (EN)
We calculated the Collaboration Coefficient using three factors : 1) the total number of interactions, 2) when individuals build on the ideas of others, and 3) when individuals were inspired to build on other individuals’ ideas. As we said before, the following design groups were collaborative : 1 – 3 – 4, and the following design groups were cooperative : 2 – 5 – 6.
Figure 3. a) Flow of ideas of the total number of interactions with its respective Collaboration Coefficient per design group ; we can see the difference regarding the Engagement element between the three collaborative design groups (1, 3, 4) and the three cooperative ones (2, 5, 6). b) The quantity of interaction per individual. c) The difference regarding the Collaboration Coefficient between collaborative design groups (in blue) and cooperative design groups (in red).
- Note de bas de page 1 :
-
A Modularity algorithm measures the strength of division of a network into modules
As we can see above, the Collaboration Coefficient of the collaborative groups (1, 3, 4) is significantly larger than the coefficient of the cooperative groups (2, 5, 6). We can explain these findings in two parts : 1) The Engagement element (calculated using a modularity algorithm1 from Network Theory) of cooperative groups was negatively impacted, as they were divided into two sub-groups during each stage of the design process, creating small “echo chambers” within the whole group. 2) It is very interesting to note in the above results the difference in the Energy element (the quantity of the exchange of ideas among individuals) between collaborative and cooperative groups : collaborative groups have a higher number of ideas flowing between individuals and in the collective space of the group.
Considering the significant difference of the total number of ideas flowing in collaborative and cooperative groups, we wanted to see what happened specifically during the process of building on the ideas of other individuals. The results are shown in Figure 4.
Figure 4. a) Flow of ideas when individuals build on the ideas of others. b) Quantity of interactions when building on the ideas of others. We can see a change in two collaborative design groups, group number 3 (decremented the idea generation) and group number 4 (incremented the idea generation). (c) The difference regarding the Collaboration Coefficient between collaborative design groups and cooperative design groups didn’t change.
For individuals in the six groups, it was much more difficult to build on the ideas of other individuals ; even though this is the case, however, we can still see a big difference between collaborative and cooperative groups in the flow of ideas when individuals have built on the ideas of others. Here, the only change that we can observe is between two experimental (COLLABORATIVE) groups (number 3 and number 4).
Lastly, if we add another element to the process of building on the ideas of other individuals, namely the process of inspiring others to build new ideas, we can continue to see the same difference between the three collaborative groups and the three cooperative ones. This whole process supports hypothesis H1.
Figure 5. a) Flow of ideas : to the process of building on the ideas of others we added the inspiration interactions (when ideas inspired others). b) Regarding the quantity of interactions, we don’t see a considerable change with respect to Figure 1 and Figure 2. c) The only change regarding the Collaboration Coefficient is design group number 3’s increase in the coefficient.
4.2. The impact of the collaboration coefficient over the invention output of each design group
In total, 396 individual evaluations were made to test two factors that helped us to define and quantify inventiveness. To carry out these 396 evaluations, 66 individuals used the framework to measure inventiveness as described above. In this way, the Invention Coefficient was defined as follows :
Invention Coefficient (IC) = Novelty (N) x Perception of Use (P)
The variation of the evaluations of each prototype by the public educational sector, teachers, and high school students was very high, but shows a clear tendency between collaborative (1, 3, 4) and cooperative (2, 5, 6) groups, as we can see in Figure 6. In this way, we found a positive relationship between the Collaboration Coefficient and the Invention Coefficient, r(396) = .51, p < .000, which supports our second hypothesis H2.
Figure 6. Variability of the individual evaluations to obtain an Invention Coefficient for each prototype designed by the collaborative (1, 3, 4) and cooperative (2, 5, 6) design groups.
5. Discussion
We defined collaboration as the balance between two elements : 1) Energy and 2) Engagement, as defined in the Social Physics Theory (Pentland, 2015). As a consequence of a cooperative working process on the Engagement element, the impact of the division of work created “echo chambers” within the whole group. One surprising result is the huge impact that a cooperative way of working had on the Energy element, with cooperative groups possessing a considerably lower level of this element than collaborative ones. Concerning inventiveness, it was surprising to observe the big difference between the three collaborative design groups and the three cooperative ones. When we observe the changes in the Collaboration Coefficient of design groups number 3 and number 4 (collaborative), namely by looking both at the total number of interactions as well as at the number of interactions when building on the ideas of other individuals, we can compare these with their respective Invention Coefficient in order to identify the importance of building on the ideas of other individuals in the design process.
Concerning the limitations of the study, this is a study in a very specific context, with different actors from the educational ecosystem working and interacting completely online to design a digital school textbook. It will be interesting to replicate the same study in different educational problem-solving contexts ; we are working to allow other researchers to replicate the study in the same and in other educational contexts to see if we can obtain the same results.
6. Conclusions and future work
In this paper, we investigated the important effects of collaborative and cooperative interactions when designing digital learning artifacts on the inventiveness of a design group. Our results show that the elements of Energy and Engagement are negatively impacted by cooperative interactions, and that if these elements are in balance, they are the keys to promoting inventiveness in heterogeneous design groups from the educational ecosystem.
As we have shown, another important element is Exploration. In this paper we did not focus our attention on this element (as the groups were purposely heterogeneous), but we are studying the individual impact of collaborative and cooperative interactions when looking at the artifact being designed as a “Boundary Object” (Star et Griesemer, 1989), where every individual sees the artifact to design from their own world. In future work, we intend to design a new Collaborative Interactions Model that will include a feedback process to help individuals balance the elements of Energy, Engagement, and Exploration.