IAEA 2008 Learning 2: Creating Assessments That Differentiate Between Experts & Novices

The key note address was by Prof. Robert J Mislevy (Prof of Statistics and Measurement - University of Maryland http://www.education.umd.edu/EDMS/mislevy/). He has done research on characteristics that seperate an expert from a novice and used the learning to design assessments that are able to better identify experts and novices. The PPT is available at (http://www.iaea2008.cambridgeassessment.org.uk/ca/digitalAssets/135859_Mislevy.ppt).

A quick snapshot ... Humans have two limitations while solving problems

1) Processing Limitations (Limited Attention & Working Memory).
2) Knowledge Limitations (not knowing what info is relevant, not being able to integrate info, not knowing what to expect, lack of production profiency)

An "EXPERT" in any field is one who can OVERCOME these limitations by enhanching his/her capabilties. (Capabilities are reasoning with patterns, improvement with practice, think about thinking, benefiting from process, methods, symbols etc).

The factor that differentiates Experts in any fields from novices are
- Experts organize knowledge more effectively
- Experts interaction with a situation through "percieve - understand - act" cycles. They dont jump to act.
- Experts use knowledge representations - like symbols, nomenclature, tech jargon etc

So if you are designing an assessment to seperate Experts from Beginners keep the following in mind

- Ask these questions while desinging the assessment - What needs to be measured? What behavior or performance reveals what needs to be measured? What tasks draw out those behaviors or performance.

- In actual experiments what has been found is that experts and novices both succeed in achieving an outcome in a given task. The difference is that an Expert spends more time planning and less time executing so an expert ends up doing less revision involving rework. Experts also assume implicit constraints that novices do not consider leading to sub-optimal solutions.

- Some definitions of Expert versus Novice in the context of dentists but equally extendable to generic use are given in the tables below.

Expert	Retrieves and uses appropriate, clear, sophisticated, accurate, and precise terminology. Uses spontaneous declaration and can retrieve without effort. Creates messages that are easily understood by the target audience.
Novice	Strained, unreliable, and effortful retrieval of terminology. Hesitant, delayed, and labored responses due to time required to process and lack of knowledge. Uses terminology unsystematically. Prone to canned speeches and explanations without awareness of individual patients’ needs.

Expert	Tries to use all sources of information all the time. Constructs a model of the patient, with each source of information an imperfect and incomplete window on some aspect of the total situation. Exhibits movement back and forth between sources (resources and personal knowledge/experience), trying to fit the partial clues together into a unified whole.
Novice	Uses single information sources in isolation. Compartmentalized use of information and failure to integrate information across sources.

Expert	Forms problems and generates hypotheses using efficient, focused, and targeted action. Uses forward and deductive reasoning in formulating problems. Thinks strategically and functions within the problem space. Efficiently generates and prunes search trees.
Novice	Uses forward reasoning but scope and depth of knowledge to support forward reasoning is limited. Generates a search tree but possesses limited tools for pruning the search tree.

Actual examples of tests where the Professor has implemented this are Architectural Registration Examination, DISC Simulator and CISCO NetPASS.