Construire et g
Construire et g
Construire et g
Construire et gé
é
é
érer des
rer des
rer des
rer des é
é
évaluations
é
valuations
valuations
valuations
standardis
standardis
standardis
standardisé
é
é
ées de qualit
es de qualit
es de qualit
es de qualité
é
é
é. Comment ?
. Comment ?
. Comment ?
. Comment ?
Avec quelles ressources ?
Avec quelles ressources ?
Avec quelles ressources ?
Avec quelles ressources ?
Prof. Dr.
Jean-Luc GILLES
Université de Liège
Système Méthodologique d’Aide à la Réalisation de Tests (SMART)
Service de Didactique générale et intervention éducative
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 2
Service de
didactique générale
et intervention
éducative
Assess Group S.A.
Assessment Systems
and e-Solutions Group
SMART
Système
Méthodologique d’Aide
à la Réalisation de
Tests
Demandes de prestations liées à l’évaluation de la
part d’entreprises, de grandes administrations, de
Besoins
des acteurs
des
évalua-tions
Critères
de qualité
validité
fidélité
sensibilité
diagnosticité
praticabilité
équité
communicabilité
authenticité
Modèles
& procédures
d’évaluation
Ressources
en personnel
et en
maté-riel
Quatre facteurs clés pour une approche qualité en
docimologie :
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 4
Système
Méthodologique d’Aide
à la Réalisation de
Tests
(ULg)
Solutions matérielles :
Une chaîne complète de lecture optique de marques
Un système de boîtiers de
votes électroniques qui
permettent d’animer des
séances de cours interactives
et de fournir des feedbacks en
direct aux étudiants
Assess-Group
Système
Méthodologique d’Aide
à la Réalisation de
Tests
(ULg)
Assess-Group
S.A.
Solutions logicielles :
Testing Interactif Multimédias
via Internet (TIMI)
electronic Network for
Global Evaluation of
Learners’ Satisfaction
(e-NGELS)
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 6
Système
Méthodologique d’Aide
à la Réalisation de
Tests
(ULg)
Assess-Group
S.A.
R&D en ingénierie docimologique :
electronic Construction & Quality Control
in Standardized Testing
2. Design
Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP)3. Questions
Selection or writing ofthe questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of thequestions defined
Blank tests (paper or web based)
5. Testing
The standardizedtest is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final versionof the results
online access FB
6. Correction
Set up of the correction of the testQuality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people whointervened in the test
Advices for the next test
2. Design
Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection2. Design
Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection1. Analysis
Points to evaluate (PE), Priorities (P), Categoriesof Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP)3. Questions
Selection or writing ofthe questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of thequestions defined
Blank tests (paper or web based)
5. Testing
The standardizedtest is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final versionof the results
online access FB
6. Correction
Set up of the correction of the testQuality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the testQuality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people whointervened in the test
Advices for the next test
(Gilles, 2002 ; Gilles & al., 2005)
2
ème
bac. psychologie et sciences de l’éducation (229 étudiants)
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 8
2
ème
bac. psychologie et sciences de l’éducation (229 étudiants)
Cote finale
Travail de groupe
« présentation en octuor »
= 50%
Test standardisé
« papier crayon »
= 50%
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
1. Analyse
préalable de
l’épreuve
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8. Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8. Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
2. Design de
l’examen
standardisé
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 12
CP x MQ
3. Création des
questions
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version
of the results online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test Advices for the
next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version
of the results online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test Advices for the
next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
4. Information
des évalués
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version
of the results online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version
of the results online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
5. Questionnaire
de l’examen
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8. Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8. Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
6. Acquisition,
correction et
contrôle qualité
des résultats
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8. Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8. Regulation
analysis of the opi-nions of the people who
intervened in the test
Advices for the next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 22
7. Feedbacks
aux étudiants
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. AnalysisPoints to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which
are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test Advices for the
next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP)
Specification table (PE x CP)
1. Analysis
Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of the questions which
are linked to the trinomials selected
Item bank
4. Training
…on the modalities of the questions defined
Blank tests (paper or web based)
5. Testing
The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback
Personalized and tailored feedbacks are proposed on the basis of final version of the results
online access FB
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction
Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation
analysis of the opi-nions of the people who
intervened in the test Advices for the
next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
Nombre de consultations des feedbacks par
les étudiants en fonction de l'heure
0
100
200
300
400
500
600
700
o
m
b
re
d
e
c
o
n
s
u
lt
a
ti
o
n
s
d
e
s
f
e
e
d
b
a
c
k
s
v
ia
i
n
te
rn
e
t
Nombre de consultations des feedbacks par
les étudiants en fonction de l'heure
0
100
200
300
400
500
600
700
N
o
m
b
re
d
e
c
o
n
s
u
lt
a
ti
o
n
s
d
e
s
f
e
e
d
b
a
c
k
s
v
ia
i
n
te
rn
e
t
pics de
consultations
à l'ULg
pics de
consultations
hors l'ULg
ULg
hors ULg
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 24
8.
Macro-régulation
2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing ofthe questions which are linked to the trinomials selected
Item bank
4. Training …on the modalities of the
questions defined
Blank tests (paper or web based)
5. Testing The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback Personalized and tailored feedbacks are proposed on the basis of final version
of the results
online access FB
6. Correction Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation analysis of the opi-nions of the people who
intervened in the test
Advices for the next test 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 2. Design Modalities of the Questions (MQ), importance of each MQ (CP x MQ) (PE x CP x MQ) selection 1. Analysis Points to evaluate (PE), Priorities (P), Categories
of Performances (CP)
Specification table (PE x CP)
1. Analysis Points to evaluate (PE), Priorities (P), Categories of Performances (CP) Specification table (PE x CP) 3. Questions Selection or writing of
the questions which are linked to the trinomials selected
Item bank
4. Training …on the modalities of the
questions defined
Blank tests (paper or web based)
5. Testing The standardized test is proposed
Online exams, paper tests linked
with OMR
7. Feedback Personalized and tailored feedbacks are proposed on the basis of final version
of the results
online access FB
6. Correction Set up of the correction of the test
Quality check and discussion of the results
before the final version
6. Correction Set up of the correction of the test
Quality check and discussion of the results
before the final version
8.Regulation analysis of the opi-nions of the people who
intervened in the test
Advices for the next test
Un exemple d’utilisation : l’examen du cours de
psychologie éducationnelle (ULg)
Comparaison des moyennes des avis des étudiants à propos des examens dans le 1er cycle de la FAPSE-ULg (de 1997 à 2000)
1
2 3 4 5 6[1
.1
] L
e
m
od
e
d'
év
al
ua
tio
n
ét
ai
t a
dé
qu
at
.2
] L
'en
tra
în
em
en
t a
va
nt
l'
ex
am
en
é
ta
it
su
ff
is
an
t
[1
.3
] L
es
q
ue
st
io
ns
é
ta
ie
nt
c
la
ire
s
[1
.4
] Q
ue
st
io
ns
b
ie
n
ad
ap
té
es
à
la
m
at
iè
re
od
e
d'
év
al
ua
tio
n
pe
rm
et
a
u
pr
of
. d
'av
oi
r u
ne
b
on
ne
re
pr
és
en
ta
tio
n
de
s..
.
[1
.6
] L
'év
al
ua
tio
n
es
t é
qu
ita
bl
e
et
im
pa
rti
al
e
ge
nc
es
d
e
l'e
ns
ei
gn
an
t s
on
t c
la
ire
m
en
t p
ré
se
nt
ée
s
ge
nc
es
d
e
l'e
ns
ei
gn
an
t s
on
t
pr
és
en
té
es
e
n
te
m
ps
u
til
es
] L
'en
se
ig
na
nt
m
et
l'
ét
ud
ia
nt
à
l'a
is
e
à
l'e
xa
m
en
o
ra
l
[2
.4
] L
'h
or
ai
re
fi
xé
po
ur
l'
ex
am
en
e
st
re
sp
ec
té
po
ns
es
c
or
re
ct
es
so
nt
c
om
m
un
iq
ué
es
a
ux
é
tu
di
an
ts
a
pr
ès
l'
ex
am
en
ys
e
st
at
is
tiq
ue
d
e
la
q
ua
lit
é
de
s q
ue
st
io
ns
(r
.b
is
) e
st
c
om
m
un
iq
ué
e
l'
ex
am
en
, l
'ét
ud
ia
nt
p
eu
t o
bt
en
ir
de
s
ex
pl
ic
at
io
ns
su
r l
a
qu
al
ité
de
se
s r
ép
on
se
s a
u.
..
M o y e n n e s d e s a v is1ère candidature 1997-1998 (n=227)
2ème candidature 1997-1998 (n=163)
1ère candidature 1998-1999 (n=150)
1ère candidature 1999-2000 (n=250)
2ème candidature 1999-2000 (n=134)
Construire et gérer des évaluations standardisées de qualité. Comment ? Avec quelles ressources ? – Jean-Luc Gilles – USIP Cochabamba, Bolivie – 29/03/07 26 (Approche qualité en évaluation - Gilles, 2002)