The MS Word document (9 pages) attached to this post discusses different Assessment Standard setting processes.
Click the link below to download this document.
Showing posts with label Assessment Workshop. Show all posts
Showing posts with label Assessment Workshop. Show all posts
Saturday, May 9, 2009
Standards (In General)
This post defines Standards and lists the uses of a Standard. This is a preamble to a detailed discussion on Assessment Standards.
What Is A Standard?
A Standard is a published document that contains a technical specification or other precise criteria designed to be used consistently as a rule, guideline, or definition. Standards help to make life simpler and to increase the reliability and the effectiveness of many goods and services we use. Standards are created by bringing together the experience and expertise of all interested parties such as the producers, sellers, buyers, users and regulators of a particular material, product, process or service.
Standards are designed for voluntary use and do not impose any regulations. Any Standard is a collective work. Committees of manufacturers, users, research organizations, government departments and consumers work together to draw up Standards that evolve to meet the demands of society and technology.
What Are The Benefits Of Standards?
Standards are a powerful tool for organizations of all sizes, supporting innovation and increasing productivity. Effective Standardization promotes forceful competition and enhances profitability, enabling a business to take a leading role in shaping the industry itself. Standards allow a company to:
• Attract and assure customers
• Demonstrate market leadership
• Create competitive advantage
• Develop and maintain best practice
Standards Are Powerful Marketing Tools
The ability to demonstrate compliance with widely recognised and respected Standards is an effective means of differentiation in a competitive marketplace. In addition, manufacturing products or supplying services to appropriate Standards maximises their compatibility with those manufactured or offered by others, thereby increasing potential sales and widespread acceptance.As consumers become increasingly informed about their choices, conformity to recognised Standards becomes pivotal. An example is the international Standard for environmental management (ISO 14001), increasingly used by businesses to demonstrate environmental responsibility.
Standards Are A Respected Badge of Quality
Customers look for the independent verification that technical Standards provide. Certification marks earned by businesses whose products and practices consistently stand up to rigorous examination are instantly recognizable and act as respected badges of quality, safety and performance.
Standards Within Business
In modern business, effective communication along the supply chain and with legislative bodies, clients and customers is imperative. Standardization can deliver measurable benefits when applied within the infrastructure of a company itself. Business costs and risks can be minimised, internal processes streamlined and communication improved. Standardization promotes interoperability, providing a competitive edge necessary for the effective worldwide trading of products and services.
How Are Standards Made?
The task of drafting formal full consensus Standards (BS, EN, ISO) is usually delegated by a technical committee or subcommittee to a drafting group or panel. Occasionally a committee may commission a consultant to complete the drafting.There are specific rules for drafting Standards that must be adhered to. These are designed to ensure that Standards meet their aim of providing, for common and repeated use, rules guidelines or characteristics for activities. They are founded on usability, verifiability and commonality.
How Are Standards Used?
The use of Standards is increasingly becoming a prerequisite to worldwide trade. A very large percentage of export is influenced by the European and international Standards business. For instance all European Union Standards are automatically adopted as British Standards.
Standards are also used as flexible alternative to regulation.
In fast-moving industries and sectors, Standards can also offer huge savings in Research and Development costs. Where common platforms for technological advances are established, tested and shared with all interested parties this can ensure commercial viability and consumer confidence.
Standards are essential to trade in increasingly competitive markets. They ensure any business offering products, services or processes is:
• cost-effective and time efficient• commercially viable• credible• safe.
They can also make significant impact on society at large. For example, as purchasers or users of products we would soon notice of they turned out to be of poor quality, do not fit, are incompatible with equipment we already have, are unreliable or dangerous. We are usually unaware of the role played by Standards in raising levels of quality, safety, reliability, efficiency and interchange ability - as well as in providing such benefits at an economical cost.
A Standard that illustrates this is:
• BS 8300:2001 Design of buildings and their approaches to meet the needs of disabled people - Code of practice
Above all, any business, large or small, can benefit from the conformity and integrity that Standards will bring. Through the development and adoption of best practice guidelines companies and organizations can make sure they are meeting consumer concerns and keeping up with best practice.
Assessment is a key part of the Standards reform movement. The first part is to set new, higher Standards to be expected of every student. Then the curriculum must be aligned to the new Standards. Finally, the student must be assessed if they meet these Standards of what every student “must know and be able to do”. A high school diploma which is given on passing a high school graduation examination or Certificate of Initial Mastery is awarded only when these Standards are achieved.
A criterion is set up for Standards of what every student or child is expected to know, and a score is set compared to these benchmarks rather than a ranking compared to a norm. It is fully expected that every child will become proficient in all areas of academic skills by the end of a period, typically 10 years but sometimes longer, after the passing of education reform bill by a state legislature.
What Is A Standard?
A Standard is a published document that contains a technical specification or other precise criteria designed to be used consistently as a rule, guideline, or definition. Standards help to make life simpler and to increase the reliability and the effectiveness of many goods and services we use. Standards are created by bringing together the experience and expertise of all interested parties such as the producers, sellers, buyers, users and regulators of a particular material, product, process or service.
Standards are designed for voluntary use and do not impose any regulations. Any Standard is a collective work. Committees of manufacturers, users, research organizations, government departments and consumers work together to draw up Standards that evolve to meet the demands of society and technology.
What Are The Benefits Of Standards?
Standards are a powerful tool for organizations of all sizes, supporting innovation and increasing productivity. Effective Standardization promotes forceful competition and enhances profitability, enabling a business to take a leading role in shaping the industry itself. Standards allow a company to:
• Attract and assure customers
• Demonstrate market leadership
• Create competitive advantage
• Develop and maintain best practice
Standards Are Powerful Marketing Tools
The ability to demonstrate compliance with widely recognised and respected Standards is an effective means of differentiation in a competitive marketplace. In addition, manufacturing products or supplying services to appropriate Standards maximises their compatibility with those manufactured or offered by others, thereby increasing potential sales and widespread acceptance.As consumers become increasingly informed about their choices, conformity to recognised Standards becomes pivotal. An example is the international Standard for environmental management (ISO 14001), increasingly used by businesses to demonstrate environmental responsibility.
Standards Are A Respected Badge of Quality
Customers look for the independent verification that technical Standards provide. Certification marks earned by businesses whose products and practices consistently stand up to rigorous examination are instantly recognizable and act as respected badges of quality, safety and performance.
Standards Within Business
In modern business, effective communication along the supply chain and with legislative bodies, clients and customers is imperative. Standardization can deliver measurable benefits when applied within the infrastructure of a company itself. Business costs and risks can be minimised, internal processes streamlined and communication improved. Standardization promotes interoperability, providing a competitive edge necessary for the effective worldwide trading of products and services.
How Are Standards Made?
The task of drafting formal full consensus Standards (BS, EN, ISO) is usually delegated by a technical committee or subcommittee to a drafting group or panel. Occasionally a committee may commission a consultant to complete the drafting.There are specific rules for drafting Standards that must be adhered to. These are designed to ensure that Standards meet their aim of providing, for common and repeated use, rules guidelines or characteristics for activities. They are founded on usability, verifiability and commonality.
How Are Standards Used?
The use of Standards is increasingly becoming a prerequisite to worldwide trade. A very large percentage of export is influenced by the European and international Standards business. For instance all European Union Standards are automatically adopted as British Standards.
Standards are also used as flexible alternative to regulation.
In fast-moving industries and sectors, Standards can also offer huge savings in Research and Development costs. Where common platforms for technological advances are established, tested and shared with all interested parties this can ensure commercial viability and consumer confidence.
Standards are essential to trade in increasingly competitive markets. They ensure any business offering products, services or processes is:
• cost-effective and time efficient• commercially viable• credible• safe.
They can also make significant impact on society at large. For example, as purchasers or users of products we would soon notice of they turned out to be of poor quality, do not fit, are incompatible with equipment we already have, are unreliable or dangerous. We are usually unaware of the role played by Standards in raising levels of quality, safety, reliability, efficiency and interchange ability - as well as in providing such benefits at an economical cost.
A Standard that illustrates this is:
• BS 8300:2001 Design of buildings and their approaches to meet the needs of disabled people - Code of practice
Above all, any business, large or small, can benefit from the conformity and integrity that Standards will bring. Through the development and adoption of best practice guidelines companies and organizations can make sure they are meeting consumer concerns and keeping up with best practice.
Assessment is a key part of the Standards reform movement. The first part is to set new, higher Standards to be expected of every student. Then the curriculum must be aligned to the new Standards. Finally, the student must be assessed if they meet these Standards of what every student “must know and be able to do”. A high school diploma which is given on passing a high school graduation examination or Certificate of Initial Mastery is awarded only when these Standards are achieved.
A criterion is set up for Standards of what every student or child is expected to know, and a score is set compared to these benchmarks rather than a ranking compared to a norm. It is fully expected that every child will become proficient in all areas of academic skills by the end of a period, typically 10 years but sometimes longer, after the passing of education reform bill by a state legislature.
Tuesday, February 3, 2009
Ebook: Basic Principles of Item Response Theory
This post is to announce the release of Dr. V. Natarajan's ebook on IRT titled "The Basic Principles of Item Response Theory".
Dr. V. Natarajan is one of the world's leading experts on Item Response Theory. He has worked extensively on theory and applications of IRT. He is passionate about IRT and believes it is one of the greatest inventions of the 20th century. Through this ebook, he wishes to make IRT accessible and interesting to young learners of assessment theory. He has made this ebook available for free downloads since he wishes it to reach as many students of assessment as possible.
Dr. V. Natarajan, is Professor Emeritus at MeritTrac Services. He is an engineer and a D.Litt in Educational Evaluation and Administration, guides the Test Development and Research teams in MeritTrac. The author of 68 books and over 60 international papers on assessment, he is a pioneer in the area of education assessments in India. He is a visiting faculty at ETS, Princeton, USA, and has been a member of the Association of Indian Universities for over three decades. He had the privilege of being taught the advanced principles of IRT from Dr. Fred Lord of ETS.
You can click the link below to download a copy of the ebook.
http://www.docstoc.com/docs/8937076/Basic-Principles-of-Item-Response-Theory
Thursday, July 12, 2007
Creating Parallel forms - An Introduction and Basic Guidelines
This Post is Authored by Sree Krishna (During migration the authors name could not be migrated)
These days, students usually log onto various freshers' portals to check out question papers which are used for entry-level recruitment by companies. It is obvious then, that by some means or the other, the questions/tests used in Campus Events find their way into these portals.
Needless to say, Security has been an issue for any assessment process. The primary target, it seems, is always the question papers used. Test users using assessments in their recruitment process is concerned about the security and would want assessments to be unique and different from others to ensure a fair recruitment process. This results in a great demand for creating different forms for a Test. So, to ensure security, Test creators need to build different forms of the tests. For example: if a Test creator has created a test on Electronics, having components A, B and C, he/she is expected to create different tests using the same components. This would mean that the test takers are not aware of the content of the tests before they take them.
In MeritTrac, the demand for unique sets exists because the Clients need unique sets for different events so that strict confidentiality is maintained, and questions do not get leaked. This is where Parallel Forms plays a major role.
Definition - Parallel forms:
Two or more forms of a test are considered parallel when they have been developed to be as similar to one another as possible in terms of the test specifications and statistical criteria.
Importance of Equivalence in Parallel forms:
If the parallel forms are not equal in terms of content or statistical criteria (p-value and discrimination index), it would impact the scores and hence any decision based on these scores will be inaccurate. Hence, we should take proper care before we release parallel forms to ensure we are fair to the candidates taking the test.
Let me explain this:
Suppose, we have a test of 5 items (questions) and we have created 2 forms (sets/papers) using 5 items each, for each paper. If the items are designed to check Prepositions, then, all these items should check only for Prepositions and nothing else. That is, the test taker does need to apply knowledge of any other “parts of speech” to answer these questions on Prepositions. All he/she needs to know is Prepositions. Each item will have to be created carefully so that it retains its validity measure. Otherwise, it will affect the test takers.
Similarly, if the DL (Difficulty Level) for a given question is not set after statistically verifying it, but set at random, based on the Item creator’s assumptions, the performance of the test takers will vary. For example, a typical assumption while setting a DL could be : Out of 100 test-takers who attempt this question, 40 would answer it correctly. But in reality, it could actually be 60. If a Statistical process were stringently followed to derive the DL, it would involve exposing this item to a similar group of test takers. That is, expose this item only to College Freshers or Developers with 2 years of experience, but never to a mixed audience, which comprises of both. However, while conducting such sampling exercise, the candidate should not be marked/scored on this item. That is answering this item should not affect his scores in any way. Considering our shortlist criteria to be fixed, we cannot have discrepancies in our items in terms of content and DLs, because it would impact the throughput and also the test takers.
Equivalence of test score means, variances, and errors are interpreted as evidence that tests are parallel
Considering the business demands, we might not be able to check the P and DI value before we use items in our tests. But we should create a process to monitor each item’s performance and tag the question accordingly in terms of statistical criteria. Creating a good Question Bank having all the necessary statistical criteria will enable us to create parallel forms.
Key points to remember while building Parallel forms:
These days, students usually log onto various freshers' portals to check out question papers which are used for entry-level recruitment by companies. It is obvious then, that by some means or the other, the questions/tests used in Campus Events find their way into these portals.
Needless to say, Security has been an issue for any assessment process. The primary target, it seems, is always the question papers used. Test users using assessments in their recruitment process is concerned about the security and would want assessments to be unique and different from others to ensure a fair recruitment process. This results in a great demand for creating different forms for a Test. So, to ensure security, Test creators need to build different forms of the tests. For example: if a Test creator has created a test on Electronics, having components A, B and C, he/she is expected to create different tests using the same components. This would mean that the test takers are not aware of the content of the tests before they take them.
In MeritTrac, the demand for unique sets exists because the Clients need unique sets for different events so that strict confidentiality is maintained, and questions do not get leaked. This is where Parallel Forms plays a major role.
Definition - Parallel forms:
Two or more forms of a test are considered parallel when they have been developed to be as similar to one another as possible in terms of the test specifications and statistical criteria.
Importance of Equivalence in Parallel forms:
If the parallel forms are not equal in terms of content or statistical criteria (p-value and discrimination index), it would impact the scores and hence any decision based on these scores will be inaccurate. Hence, we should take proper care before we release parallel forms to ensure we are fair to the candidates taking the test.
Let me explain this:
Suppose, we have a test of 5 items (questions) and we have created 2 forms (sets/papers) using 5 items each, for each paper. If the items are designed to check Prepositions, then, all these items should check only for Prepositions and nothing else. That is, the test taker does need to apply knowledge of any other “parts of speech” to answer these questions on Prepositions. All he/she needs to know is Prepositions. Each item will have to be created carefully so that it retains its validity measure. Otherwise, it will affect the test takers.
Similarly, if the DL (Difficulty Level) for a given question is not set after statistically verifying it, but set at random, based on the Item creator’s assumptions, the performance of the test takers will vary. For example, a typical assumption while setting a DL could be : Out of 100 test-takers who attempt this question, 40 would answer it correctly. But in reality, it could actually be 60. If a Statistical process were stringently followed to derive the DL, it would involve exposing this item to a similar group of test takers. That is, expose this item only to College Freshers or Developers with 2 years of experience, but never to a mixed audience, which comprises of both. However, while conducting such sampling exercise, the candidate should not be marked/scored on this item. That is answering this item should not affect his scores in any way. Considering our shortlist criteria to be fixed, we cannot have discrepancies in our items in terms of content and DLs, because it would impact the throughput and also the test takers.
Equivalence of test score means, variances, and errors are interpreted as evidence that tests are parallel
Considering the business demands, we might not be able to check the P and DI value before we use items in our tests. But we should create a process to monitor each item’s performance and tag the question accordingly in terms of statistical criteria. Creating a good Question Bank having all the necessary statistical criteria will enable us to create parallel forms.
Key points to remember while building Parallel forms:
- When items are created for a particular content area, multiple checks have to be conducted to ensure that the items belong to the field desired. That is, while building content for Pointers in C, we should not have questions related to Arrays.
- P Values (Facility values) & Discrimination Index value (The index of discrimination is a useful measure of item quality whenever the purpose of a test is to produce a spread of scores, reflecting differences in test takers’ achievement, so that distinctions may be made among the performances of test takers) for each item should be computed.
- Once we decide on the Blueprint, we need to select items, which satisfy content and cognitive specifications (problem solving, application, knowledge) & statistical criteria (P value & DI value).
Tuesday, June 26, 2007
"Test Development" by Patrick & Harley - A review
This Post is Authored by Sree Krishna (During migration the authors name could not be migrated)
There are different resources, which help us to understand the concepts of Testing or the Test Development process. I feel, that all these materials cannot be used in isolation to understand the concept. Different authors emphasize on different aspects and hence we should use more than one reading material to understand the concept. A single point reference is never advisable in these cases. As a matter of fact, some of them might not be suitable at all and might further mislead or confuse the audience.
I would suggest a few reference materials to facilitate a better understanding of the science/concept of Assessments, from beginners to advanced learners. Here are a few of them:
URLs
http://www.ukcle.ac.uk/resources/trns/mcqs/nine.html
http://www.msu.edu/dept/soweb/writitem.html
http://pareonline.net/getvn.asp?v=5&n=2
Books
Psychological Testing by Paul Kline
Psychological Testing by Anastasi
Item writing by Thomas M Haladyna
I did go through "Test Development" by Patrick and Harley and the first thought that struck my mind is "it does not cover all the aspects of Test Development in detail", "this cannot be a training document for a Test Development Team". It can be, at the best, a book to understand the basic elementary aspects of Test Development.
My apprehensions about recommending this book as a training document:
a) Does not cover the methodology of understanding/analyzing the requirements and mapping the requirements to assessments
b) Does not cover the types of assessments (so, the differences are not clearly understood)
c) Explanations are not elaborate – For example Bloom's taxonomy
d) Reasons for using Objective Assessments and their limitations - not mentioned at all
e) Guidelines for Item writing – does not emphasize the importance / significance of the line items mentioned, for example: "Research studies have proven that the negative words will have difficulty in understanding the meaning of negatively phrased items. A negatively-stated item does require a test taker to switch his or her mind set from that of looking for the best answer to that of locating the most definite non-answer" or "Use of specific determiners such as always, never, totally, absolutely, only, and completely in all the distracters or only in the right answers serve as flags to the test taker"
f) Examples provided for Item Writing Guidelines are not appropriate for MeritTrac or rather dated. The learning audience may not relate the course content back to the work place scenario.
g) Many questions are left unanswered. For example, why are Norms used, is not answered. Similarly Explanations are not satisfactory for z scores. The concepts need more clarity in terms of explanation and examples.
h) Explanations on Reliability and Validity are brief and sketchy.
I feel that Item Creators/Reviewers can use this material for reference purpose, after they have undergone a formal/extensive and interactive training on Assessment. And the textual reference will definitely enhance the acquired learning gained from the training sessions.
There are different resources, which help us to understand the concepts of Testing or the Test Development process. I feel, that all these materials cannot be used in isolation to understand the concept. Different authors emphasize on different aspects and hence we should use more than one reading material to understand the concept. A single point reference is never advisable in these cases. As a matter of fact, some of them might not be suitable at all and might further mislead or confuse the audience.
I would suggest a few reference materials to facilitate a better understanding of the science/concept of Assessments, from beginners to advanced learners. Here are a few of them:
URLs
http://www.ukcle.ac.uk/resources/trns/mcqs/nine.html
http://www.msu.edu/dept/soweb/writitem.html
http://pareonline.net/getvn.asp?v=5&n=2
Books
Psychological Testing by Paul Kline
Psychological Testing by Anastasi
Item writing by Thomas M Haladyna
I did go through "Test Development" by Patrick and Harley and the first thought that struck my mind is "it does not cover all the aspects of Test Development in detail", "this cannot be a training document for a Test Development Team". It can be, at the best, a book to understand the basic elementary aspects of Test Development.
My apprehensions about recommending this book as a training document:
a) Does not cover the methodology of understanding/analyzing the requirements and mapping the requirements to assessments
b) Does not cover the types of assessments (so, the differences are not clearly understood)
c) Explanations are not elaborate – For example Bloom's taxonomy
d) Reasons for using Objective Assessments and their limitations - not mentioned at all
e) Guidelines for Item writing – does not emphasize the importance / significance of the line items mentioned, for example: "Research studies have proven that the negative words will have difficulty in understanding the meaning of negatively phrased items. A negatively-stated item does require a test taker to switch his or her mind set from that of looking for the best answer to that of locating the most definite non-answer" or "Use of specific determiners such as always, never, totally, absolutely, only, and completely in all the distracters or only in the right answers serve as flags to the test taker"
f) Examples provided for Item Writing Guidelines are not appropriate for MeritTrac or rather dated. The learning audience may not relate the course content back to the work place scenario.
g) Many questions are left unanswered. For example, why are Norms used, is not answered. Similarly Explanations are not satisfactory for z scores. The concepts need more clarity in terms of explanation and examples.
h) Explanations on Reliability and Validity are brief and sketchy.
I feel that Item Creators/Reviewers can use this material for reference purpose, after they have undergone a formal/extensive and interactive training on Assessment. And the textual reference will definitely enhance the acquired learning gained from the training sessions.
Sunday, April 1, 2007
Creating Assessments - Module 2
This Post is Authored by Sree Krishna (During migration the authors name could not be migrated)
This is a PPT which is Module 2 of the MeritTrac Creating Assessments Workshop. It covers score interpretation, understanding norms and classifying norms. Click the link below to download the presentation.
http://www.docstoc.com/docs/8986758/Creating-Assessments-Part-2
The link below is an "Assessment on Assessments" which is adminstered after Module 2
http://www.docstoc.com/docs/8986689/Assessment-on-Assessments
This is a PPT which is Module 2 of the MeritTrac Creating Assessments Workshop. It covers score interpretation, understanding norms and classifying norms. Click the link below to download the presentation.
http://www.docstoc.com/docs/8986758/Creating-Assessments-Part-2
The link below is an "Assessment on Assessments" which is adminstered after Module 2
http://www.docstoc.com/docs/8986689/Assessment-on-Assessments
Creating Assessments - Module 1
This Post is Authored by Sree Krishna (During migration the authors name could not be migrated)
The attached PPT is Module 1 of the MeritTrac Creating Assessments Workshop. It covers an introduction to assessments and defines different types of assessments. Click the link below to download the PPT.
http://www.docstoc.com/docs/8986726/Creating-Assessments-Part-1
The attached PPT is Module 1 of the MeritTrac Creating Assessments Workshop. It covers an introduction to assessments and defines different types of assessments. Click the link below to download the PPT.
http://www.docstoc.com/docs/8986726/Creating-Assessments-Part-1
Sunday, February 18, 2007
Introduction To Assessments - For Recruiters & Recruitment Managers
Please click on the link below to download a Powerpoint Presentation that provides an introduction to Assessments specifically meant for Recruiters and Recruitment Managers. In case you have problems downloading you can right click the link and choose "Save Target As" on the menu.
Intro To Assessments(For Recruiters/Recruitment Managers)
Subscribe to:
Posts (Atom)
Hit Counter
