2006 School Certificate Standards Package
Introduction
Standards-referenced reporting
In 2006 the Board of Studies NSW revised the suite of statewide tests for the School Certificate credential. These tests assess individual achievement in the foundation knowledge and skills needed to successfully undertake subjects in Year 11 and 12 and to participate effectively in the wider community.
Students also are awarded a grade by their school for each of the courses they have studied in Years 9 and 10. The grades are based on a set of Course Performance Descriptors developed by the Board of Studies. They indicate a student’s full range of achievements in each course, providing a detailed report of the student’s overall performance.
Student performance in the School Certificate tests is reported in relation to standards (or levels of achievement). Six performance bands are used for each test, or, in the case of Australian History, Geography, Civics and Citizenship, each test component. Band 1 covers the range of marks from 0 to 49, band 2 from 50 to 59, band 3 from 60 to 69, band 4 from 70 to 79, band 5 from 80 to 89, and band 6 covers the range of marks from 90 to 100.
Except for band 1, the level of achievement represented by each performance band is described in a statement summarising the knowledge and skills typically demonstrated by students who have achieved that performance band. There is no statement corresponding to band 1, which is considered to be below the minimum standard expected.
Students are awarded a particular band (and mark) if they demonstrate they have reached the performance standard associated with that band. There is no predetermined proportion of students placed in each performance band. Each test is reported independently. Since the same standards are applied each year, it is possible to make comparisons between the performances of students who have sat for the test in different years.
How the standards were set following the 2006 School Certificate tests
Pre-Stage 1
The first step in the process was to have the judges individually study the materials in the most recent School Certificate Standards Package for the test – the band descriptions, the test paper and marking guidelines, and the sample student scripts. Drawing on their professional experience and understanding of their subject, each judge used the packages to develop an understanding of the knowledge and skills typically possessed by students on the borderline between two bands: they developed an ‘image’ of these particular students.
Stage 1
The judges individually and independently considered each test question in turn. For questions that were scored right or wrong (ie 'dichotomously') each judge recorded the probability that a borderline student would get the question right. For questions that involved several steps or essay responses (ie were scored 'polytomously') each judge recorded the mark that they believed a borderline student would receive. At the end of this process, each judge’s borderline marks for each question were then added to give that judge’s cut-off marks between the bands. The average of the borderline marks between bands 5 and 6 proposed by each judge was then calculated: this value was the initial cut-off mark between band 5 and band 6. A similar process was followed for the other borderlines.
Stage 2
The next step involved the judges meeting to compare and discuss the decisions they had made individually at Stage1. This process also involved them in reviewing statistical reports specially developed to support the standards-based approach. A random sample of students was selected and their marks for every question in the test were extracted. These data were then analysed and presented in a form that made it easy for the judges to see how students at various ability levels performed in the questions. The judges reviewed and discussed the statistical data. During these discussions they had the opportunity to modify any cut-off marks they had previously set for particular questions. If these differed from the values predicted by the statistical analysis, the judges did not necessarily have to change their values: that is, they could choose whether or not to change them. For each borderline, after all changes were made, the cut-off marks proposed by the judges for each question were averaged. The average marks for all questions were then added up to obtain a cut-off mark for that borderline; by the end of this step a revised set of cut-off marks had been calculated.
Stage 3
In the next step, the judges considered a sample of student answers that had been awarded marks close to the total cut-off marks that they had established in Stage 2. In reviewing and discussing these answers, the judges were asked to satisfy themselves that, holistically, such students were demonstrating levels of performance on the borderline between two bands. If the judges were not satisfied on this point, they then studied, for each question, answers that had been awarded a mark either side of the cut-off mark, to identify where the discrepancy had occurred. Given this task, the judges chose whether to make adjustments to their borderline decisions and, if so, a third set of cut-off marks was calculated.
Consultative Committee meeting
Finally, the judges recommended to the Consultative Committee that the Stage3 cut-off marks be applied to the mark distribution from the test.
(The Consultative Committee is a team of educational measurement experts appointed to oversee the School Certificate test setting, marking and standards-setting processes. Based on the information provided by each of these processes, the Consultative Committee decides either to accept the judges’ recommended cut-off marks or to make some adjustments.)
At the conclusion of this process, the final cut-off marks were applied to the mark distribution from the test, and bands were allocated to students accordingly. This was done by setting the cut-off marks to the band borderline marks of 50, 60, 70, 80 and 90 that are shown on the performance scale. In this way the raw test marks awarded to students were aligned to the performance scale. For example, a student who received a raw test mark that was between the borderline for band 3 / band 4 and the borderline for band 4 / band 5 would receive a mark between 70 and 80 through a simple linear mapping process. In most cases, the test mark reported will be different from the raw test mark.
This package
A sample of student responses that were awarded the final band cut-off marks for each question established by the judges has been chosen to be included in this Standards Package. Also provided are the statistics on how those students on the borderline of two bands performed in the multiple-choice and other objective items, a copy of the 2006 test paper, and a copy of the band descriptions. A copy of the syllabus has also been provided for reference purposes. This material will form an essential part of the operation in future years by exemplifying and clarifying the standards that are applied.
It is important to note that the samples of student answers at each band cut-off typify the standard of work produced by students on the borderline between performance bands. They should not be regarded as exemplary responses, nor should they be seen as typical of all students achieving a mark placing them in a particular band.
Exemplary Samples
In addition to the student answers that illustrate the performance standards, some that received full marks are included. For each examination question where at least one student received full marks, one exemplary answer is provided to demonstrate characteristics of answers awarded full marks. In some questions other students may have used different approaches to answering the question, yet still received full marks. Answers to different questions may vary in quality of expression, detail and/or argument. Exemplary answers may contain errors that are to be expected from responses produced within the time constraints and under the pressures of an examination.
The structure of the material
For each section of the test requiring an extended answer, the package includes up to three answers at a standard typical of students placed at the borderline between band 5 and band 6. Up to three typical student answers at the borderlines between band 4 and band 5, band 3 and band 4, band 2 and band 3, and band 1 and band 2 are also provided.
To access this information for the test paper:
Step 1: Select the required section in the test paper
Step 2: Select the required question
Step 3: Select the required borderline to view, in turn, the answers of each of the students at that borderline.
In the case of those sections of the paper containing multiple-choice questions, information is provided in tabular and graphical form showing how students at each borderline have responded to each question. Students included in this analysis are those who have achieved a score within one mark of the borderline for the whole paper.
How it can be used
Teachers will gain a very clear understanding of the standard of work typically produced by students at the borderlines by studying the samples provided in this package. Teachers’ understanding can be enhanced by re-reading the band descriptions used to report student achievement and matching these to the images they have formed of the students at each borderline.
Where sections of a test paper comprise multiple-choice questions, teachers and students can see from the tables and graphs for each question – for the groups of students whose marks are at the borderline between two bands – the percentage of each group that selected answers A, B, C and D. Reading the question and comparing with the student answers develops an understanding of how well students at each borderline answered each question and, importantly, the types of errors they tended to make. This analysis will help to build a picture of the level of knowledge and skills typical of students at each level of performance.
Where sections of a paper consist of free-response questions, some of which require a short answer (eg a number or diagram or a few sentences or paragraphs), others requiring an extended answer (eg a solution to a problem, or an essay), copies of the students’ answers are provided. Teachers can look at the questions, study the students’ actual answers, and then read the corresponding band descriptions to ascertain why these were typical of work by students placed at the borderline between the two bands.
