CSCW – Module 3 – Page 1 P. Dillenbourg & N. Nova Module 2 : Measuring effects.

Présentation au sujet: "CSCW – Module 3 – Page 1 P. Dillenbourg & N. Nova Module 2 : Measuring effects."— Transcription de la présentation:

1 CSCW – Module 3 – Page 1 P. Dillenbourg & N. Nova Module 2 : Measuring effects

2 CSCW – Module 3 – Page 2 P. Dillenbourg & N. Nova Scientific approach Experiment Results Hypothesis Experiment CSCW Course Project 1 D1: Qualitative analysis of task distribution D2: Quantitative comparison of task performance D3: Qualitative and quantitative dialogue analysis Experimental research covers a variety of data analysis methods Log Files Results Project 1 Report

3 CSCW – Module 3 – Page 3 P. Dillenbourg & N. Nova Scientific approach Experiment Hypothesis Experiment CSCW Course Project 1 D1: Qualitative analysis of task distribution Log Files Results 30.11.2004

4 CSCW – Module 3 – Page 4 P. Dillenbourg & N. Nova Deliverable 1  30.11.04 Condition S Condition F Pair 1 Pair 2 Pair 3 Pair 4 Pair 5 Pair 6 Explanation & Results Do pairs in condition A organize their work differently than pairs in condition B ?

5 CSCW – Module 3 – Page 5 P. Dillenbourg & N. Nova Scientific approach Experiment Hypothesis Experiment CSCW Course Project 1 D2: Quantitative comparison of task performance Log Files Results Inferential Statistics Are pairs in condition A more effective than pairs in condition B ?

6 CSCW – Module 3 – Page 6 P. Dillenbourg & N. Nova Performance Time cars need to cross the city ? Group 1; N=20 Condition “functional roles” Group 2; N=20 Condition “structural roles” Is this difference in group means due to sampling or is it the effects of condition changes? If group 2 had been in condition “functional” and group 1 in condition “structural”, group 2 would still get higher performance ?

7 CSCW – Module 3 – Page 7 P. Dillenbourg & N. Nova Population to whom one want to generalize results 6000 epfl students Sample with whom one obtain results 2 X 20 epfl students Inferential statistics

8 CSCW – Module 3 – Page 8 P. Dillenbourg & N. Nova Performance Number of 20-subjects groups GroupeSizePerformance 12067 22056 32022 42078 52010 62054 72033 82090 92056 102062 112068 122071 132063 142061 152059 162071 172049 182049 192055 20 75 212062 222065 232048 242070

9 CSCW – Module 3 – Page 9 P. Dillenbourg & N. Nova Performance Number of 20-subjects groups Number of 30-subjects groups

10 CSCW – Module 3 – Page 10 P. Dillenbourg & N. Nova Performance Time cars need to cross the city Sampling effect ? Group 1; N=20 Condition “functional roles” Group 2; N=20 Condition “structural roles”

11 CSCW – Module 3 – Page 11 P. Dillenbourg & N. Nova Condition SCondition F Performance Time cars need to cross the city ? 1 pair

12 CSCW – Module 3 – Page 12 P. Dillenbourg & N. Nova Cond S Cond F Performance Time cars need to cross the city Cond S Cond F Time cars need to cross the city 

13 CSCW – Module 3 – Page 13 P. Dillenbourg & N. Nova Cond S Cond F Performance Time cars need to cross the city ? Cond S Cond F Time cars need to cross the city ? Is there a real effect ? Is the means difference due to the difference of conditions or to random variations inherent to (semi-)random sampling? The answer does not simply depend of the size of the means difference. This difference must be compared to the group heterogeneity (variance, standard deviation, error, noise) Inter-Group Differences Intra-Group Differences

14 CSCW – Module 3 – Page 14 P. Dillenbourg & N. Nova Cond S Cond F Performance Time cars need to cross the city Cond S Cond F Time cars need to cross the city  Problems with small samples :  fewer chances to obtain significant differences  do not respect the applicabilty criteria of statistical tests But large samples are expensive !

15 CSCW – Module 3 – Page 15 P. Dillenbourg & N. Nova Strijbos, J.W, Martens R., Jochems, W & Broers N. THE EFFECT OF FUNCTIONAL ROLES ON GROUP EFFICIENCY: Using Multilevel Modeling And Content Analysis To Investigate Computer-supported, SMALL GROUP RESEARCH, Vol. 35 No. 2, April 2004 195-229 http://www.ou.nl/info-alg-english-r_d/OTEC_research/people/jan-willem_strijbos.htm Perceived Group Efficiency Roles: 1, Project Planner 2. Communicator 3. Reporter 4. Data Collector Non-role groups F = 2.86; p >.10

16 CSCW – Module 3 – Page 16 P. Dillenbourg & N. Nova Experimental research terminology QuestionDo role incentives change group performance HypothesisExpected results performance (cond F) > performance (cond S) Independent variablewhat do you vary? (or Factors)FonctionalRoles versus StructuraleRoles Dependent variableshow do you measure effects ? Time cars need to cross the city Time group needs to succeed the task Controlled variablesthings you try to keep constant ? Pairs previous degree of mutual knowledge Experimental settings (instructions, …) Intermediate variablesRelate IndependentVarianles to DependentVariables Roles adherence; Roles distribution adequacy Quality and intensity of interactions “Significant” differenceIf we conclude that the DV effects are due to IV, we some probability to make a mistake:.01.05.10 GreatOK TrendNo

17 CSCW – Module 3 – Page 17 P. Dillenbourg & N. Nova Experimental research terminology Plan2 X 2 factorial plan Factorsplan dimensions Modalityrow and column headings Conditionsplan cells Factor 1 FonctionalRoles Structural Roles Factor 2 High Mutual Knowledge Medium Mutual Knowledge Low Mutual Knowledge Condition HFCondition HS Condition MFCondition MS Condition LFCondition LS

18 CSCW – Module 3 – Page 18 P. Dillenbourg & N. Nova Experimental research terminology Plan2 X 2 factorial plan Factorsplan dimensions Modalityrow and column headings Conditionsplan cells Effectsmain effects interaction effects Factor 1 FonctionalRoles Structural Roles Factor 2 High Mutual Knowledge Medium Mutual Knowledge Low Mutual Knowledge 6035 6550 5567

19 CSCW – Module 3 – Page 19 P. Dillenbourg & N. Nova Example Study: group learning & animated pictures Apprend-on mieux ou moins bien à partir d’animations par rapport aux images statiques ? Peut-on améliorer l’efficacité des animations en compensant l’aspect fugace de ces dernières ? Apprend-on mieux ou moins bien à 2, à partir d’animations Sangin, Rebetez, Betrancourt, Dillenbourg

20 CSCW – Module 3 – Page 20 P. Dillenbourg & N. Nova Hypothèses et méthodologie Hypothèses théoriques : 1.L’animation est plus riche en informations qu’une image statique. Elle devrait donc induire un meilleur apprentissage. 2.La charge cognitive due à l’interaction commulée à celle due à la fugacité des animations porterait préjudice à l’apprentissage à 2, à partir des animations. 2.La présence d’un historique rendant permanentes des étapes antérieures d’une animation diminuerait la charge cognitive due à la fugacité des animations, et permettrait aux apprenants d’avoir plus d’éléments pour le grounding. La permanence améliorerait ainsi l’apprentissage.

21 CSCW – Module 3 – Page 21 P. Dillenbourg & N. Nova Méthode PrétestMatérielNasa-tlxPost-test Astronomie Intro PrétestMatérielNasa-tlxPost-test Géologie Intro FinTest corsi + paper-folding soloduo Accueil + contrat de consentement

22 CSCW – Module 3 – Page 22 P. Dillenbourg & N. Nova Méthode : facteurs et observations Variable indépendantes: –Mode de présentation (statique vs dynamique) –Permanence de l’information (Avec vs Sans) –Mode d’apprentissage (Solo vs Duo) Variables dépendantes : –Score de rétention –Score d’inférence Variables intermédiaires –Charge cognitive perçue (cinq échelles tirées du nasa-tlx) –Capacités de rotation mentale (paper-folding test) Plan 2 X 2 X 2 [20 paires¨]

23 CSCW – Module 3 – Page 23 P. Dillenbourg & N. Nova Animation*Collaboration L’animation améliore les performances –de rétention (F (1 ;152) =9.178 ; p<.01) –et d’inférence (F (1 ;152) =6.246 ; p<.05) En inférence, seuls les duo semblent vraiment bénéficier de l’animation –(F (1 ;76) =15.1 ;p<.01)

24 CSCW – Module 3 – Page 24 P. Dillenbourg & N. Nova Permanence*Collaboration La permanence ainsi que la collaboration n’ont pas d’effet simple significatif L’interaction sur le score d’inférence est significative (F (1 ;152) =6.630 ; p<.05) Les différences d’inférence entre solo et duo sans permanence sont significatives –(F (1 ;74) =5.96 ;p<.05) L’effet de la permanence sur les solos est marginalement significative –(F (1 ;79) =3.91 ; p=.052)

25 CSCW – Module 3 – Page 25 P. Dillenbourg & N. Nova Capacités de rotation mentale Les capacités de rotation mentale sont corrélées à la réussite aux questionnaires (r=.67 et.68; p<.01) En créant des groupes selon le niveau de paper folding, un effet simple important apparaît –En rétention (F (1 ;72) =36.13 ;p<.01) –Comme en inférence (F (1 ;72) =37.02 ; p<.01 )

26 CSCW – Module 3 – Page 26 P. Dillenbourg & N. Nova Does an experiment prove anything? A first experiment shows an effect G1>G2. The next experiment shows G2 < G1… Why ? Because it’s impossible to control all contextual factors. So what ? Compare multiple experiments  meta-analysis Replace experimental approach by a deeper qualitative approach  namely ethnological methods.

27 CSCW – Module 3 – Page 27 P. Dillenbourg & N. Nova If 20 studies give contradictory results, how to conclude on roles effect? Positive (study 1) Positive (study 5) Positive (study 8) Positive (study 9) Positive (study 13) Positive (study 15) Positive (study 16) Positive (study 17) Positive (study 18) Negative (study 2) Negative (study 4) Negative (study 6) Negative (study 7) Negative (study 11) Negative (study 12) No effect (study 3) No effect (study 7) No effect (study 10) No effect (study 14) No effect (study 19) No effect (study 20)

28 CSCW – Module 3 – Page 28 P. Dillenbourg & N. Nova Weight studies with effect size + 2 (study 1) + 5 (study 5) + 9 (study 8) + 5 (study 9) + 3 (study 13) + 4 (study 15) + 3 (study 16) + 4 (study 17) + 5 (study 18) -1 (study 2) -10 (study 4) -7 (study 6) -8(study 7) -5 (study 11) -9 (study 12) No effect (study 3) No effect (study 7) No effect (study 10) No effect (study 14) No effect (study 19) No effect (study 20)

29 CSCW – Module 3 – Page 29 P. Dillenbourg & N. Nova Weight studies with sample size and methodological robustness + 2 (study 1) N=120 + 5 (study 5) N=80 + 9 (study 8) N=320 + 5 (study 9) N=90 + 3 (study 13) N=140 + 4 (study 15) N=200 + 3 (study 16) N=330 + 4 (study 17) N=10 + 5 (study 18) N=100 -1 (study 2) N=20 -10 (study 4) N=100 -7 (study 6) N=60 -8(study 7) N=200 -5 (study 11) N=60 -9 (study 12) N=40 No effect (study 3) No effect (study 7) No effect (study 10) No effect (study 14) No effect (study 19) No effect (study 20)

30 CSCW – Module 3 – Page 30 P. Dillenbourg & N. Nova R=1 R= -0.33 R=0.17 R=0.85 Using correlation in experimental research

31 CSCW – Module 3 – Page 31 P. Dillenbourg & N. Nova Résultats Matrice de corrélations entre les variables principales et secondaires : Rétention Inférenc e word_fre q Gest_ freqP_freqT_freqC_freqM_freq Rétentionr 1.596**.167.053-.002.078.040.074 Sig..000.144.650.986.504.729.525 Inférencer.596**1.156.095.060.232*-.020.032 Sig..000.172.414.604.043.863.782 word_freqr.167.1561.484**.449**.275*-.015.333** Sig..144.172.000.016.900.003 Gest_freqr.053.095.484**1.836**.666**.287*.596** Sig..650.414.000.012.000 P_freqr -.002.060.449**.836**1.599**-.098.163 Sig..986.604.000.402.160 T_freqr.078.232*.275*.666**.599**1.001.153 Sig..504.043.016.000.990.188 C_freqr.040-.020-.015.287*-.098.0011.277* Sig..729.863.900.012.402.990.015 M_freqr.074.032.333**.596**.163.153.277*1 Sig..525.782.003.000.160.188.015

32 CSCW – Module 3 – Page 32 P. Dillenbourg & N. Nova MOO experiment, 20 pairs He lies! Example Study Space may narrow down the conversational context

33 CSCW – Module 3 – Page 33 P. Dillenbourg & N. Nova 20 pairs 2 X [5 worlds]

34 CSCW – Module 3 – Page 34 P. Dillenbourg & N. Nova Hypothesis 1: The proximity of the emitter to the referred object clarifies the referential context. Emitter.223** Receiver

35 CSCW – Module 3 – Page 35 P. Dillenbourg & N. Nova but not a causal link Correlation indicates the strength of a relation but not a causal link Trust X Similarity Brain Size X IQ (r = 0.44) http://www.lrainc.com/swtaboo/stalkers/jpr_gould_paid.html Size X Number-of-children Order Education X Vote  … Each state's voting percentage for Kerry (Y) versus percentage of population in that state who have obtained an advanced degree or more 0.77 http://www.issues2000.org/askme/Education_vs_voting.htm Is intelligence correlated with voting for Kerry?

36 CSCW – Module 3 – Page 36 P. Dillenbourg & N. Nova Scientific approach Experiment Hypothesis Experiment CSCW Course Project 1 D2: Quantitative comparison of task performance Log Files Results Inferential Statistics Are pairs in condition A more effective than pairs in condition B ?

37 CSCW – Module 3 – Page 37 P. Dillenbourg & N. Nova Your deliverable 2 The independent variable is define: functional versus structural roles Choose one dependent variable: VD –Task performance –Time to complete the task –Number of actions –Degree of division of labour –… Compute VD for 2 X 21 pairs Compute ANOVA with N.NOVA Write a 1-2 page summary: 1.hypothesis; 2.variables 3.descriptive statistics (mean + SD) 4.inferential tests (with SPSS, EXCEL, R, …) 5.discussion of results