National Joint Council for Local Government Services

National Joint Council for Local Government Services

Employers' Secretary:

Charles Nolda

Layden House, 76-86 Turnmill Street

London, EC1M 5LG

Tel: 020 7296 6600 Fax: 020 7296 6739

www.lg-employers.gov.uk

Trade Union Secretaries

Jack Dromey, TGWU

Mick Graham, GMB

Heather Wakefield UNISON

Address for correspondence:

1 Mabledon Place

London, WC1H 9AJ

Tel: 0845 355 0845

Fax: 020 7387 7217

To: Chief Executives in England, Wales & N Ireland

(with additional copy for the Chief Personnel Officer)

Members of the National Joint Council

Provincial/Associated Joint Secretaries

Local Authority Associations' Secretaries

3 September 2003

Dear Chief Executive

ADVISORY NOTE ON QUALITY ASSURANCE TO AID EFFECTIVE IMPLEMENTATION WHEN USING THE NJC JOB EVALUATION SCHEME

The NJC's Job Evaluation Technical Working Group has drawn together some advice on broad principles for undertaking consistency checks on the outcomes of evaluations throughout the job evaluation process. The advice ~~will~~ [Author ID1: at Tue Aug 5 11:08:00 2003 ]is intended to [Author ID1: at Tue Aug 5 11:08:00 2003 ]help authorities achieve faster and smoother implementation of the scheme. The advice deals with the specifics of the NJC scheme, both paper based and computerised, and the ways in which both individual outcomes and the range and spread of results can be subjected to systematic checks. The advice~~will be of assistance to authorities~~ [Author ID1: at Tue Aug 5 11:08:00 2003 ] covers all stages [Author ID1: at Tue Aug 5 11:08:00 2003 ]from the pilot or benchmark exercise through to completion and provides the necessary information and assistance to quality assure on a systematic basis. The advice takes into account that the range of jobs covered will vary between authorities. We trust that the attached advice is of assistance to local authorities in the implementation of the NJC scheme.

Yours sincerely

Charles Nolda

Jack Dromey

Mick Graham

Heather Wakefield

Joint Secretaries[Author ID0: at ]

[Author ID0: at ]

[Author ID1: at Tue Sep 2 15:50:00 2003 ]

NJC JES: QUALITY ASSURANCE OF JOB EVALUATIONS:

SOME `SORETHUMBING'/ MODERATION CHECKS

1. Introduction

1.1 Sorethumbing' is the term often used for the consistency checks which need to be carried out for all job evaluation exercises (on any scheme and whether paper-based or computerised) to quality assure the exercise and to identify any factor assessments or jobs which appear `out of place'. For computerised job evaluation, the term `moderation' is often used, but the checks are very similar.

1.2 This does not mean that any jobs identified through `sorethumbing' or moderation are necessarily wrongly evaluated, but that further investigation is required to see why they have been evaluated in the way they have.

1.3 `Sorethumbing'/ moderation should be carried out:

After the benchmark or pilot exercise: at this stage it is also a check on how the local conventions are operating: and
At regular intervals during the remaining evaluations (depending on how high the level of consistency is: can be spaced out further once good consistency is established): and
When all evaluations have been completed and before publication of any results

1.4 `Sorethumbing'/ moderation can be done by a series of `spot checks', but is best carried out systematically. The following sections describe and explain a series of systematic checks:

Usage of factor levels
Distribution of evaluations
Specific factor checks
Overall range of total weighted scores
Specific checks of total weighted scores

1.5 These are summarised in checklist form at the end.

2. Usage of Factor Levels

2.1 The NJC JES factor levels were developed to represent discrete levels of demands over the range of local government jobs. Over a representative sample of jobs, therefore, all factor levels should be used, although not necessarily to the same degree. Where this is not the case, there should be an obvious explanation.

2.2 In particular, it is indicative of either a misunderstanding of the system by evaluators or a seriously unrepresentative job sample, if intermediate factor levels between the top and bottom have not been used.

2.3 Level 1 may be used less than other levels, but there should always be some jobs at this level in a representative sample of local authority jobs.

2.4 There may be explanations for the instances where the highest factor levels have not been used, usually depending on the range of jobs within the scope of the exercise, for example:

Knowledge: level 8; Responsibility factors: level 6: these levels were designed primarily for professional managerial jobs showing greater breadth of knowledge requirements and responsibilities than for practitioners. So, if the exercise excludes professional managerial jobs, then these levels may not be used, but note that some jobs not traditionally regarded as managerial may show similar features.

Initiative and Independence: level 8: in contrast to other NJC JES factors, the Initiative and Independence factor is relatively hierarchical, that is, one would normally expect a managerial job to be assessed at a higher level than the subordinate jobs. Use of the higher levels, therefore, depends on the range of jobs to be evaluated. Where senior managerial jobs are excluded from the NJC JES evaluation exercise, few if any jobs would be expected to score at level 8.

Physical Demands: level 5; Working Conditions: level 5: these levels were intended to apply to the most physically demanding jobs in local government, for example, refuse collectors, grave diggers, possibly sewerage operatives. So, if the exercise does not cover these or similar groups of ex-manual workers, then these levels may not be used.

Emotional Demands: level 5: this level was intended to apply to the most emotionally demanding jobs in local government, for example, Child and Family Social Workers, Approved Mental Health Social Workers. So, if the exercise does not cover these or similar jobs, then these levels may not be used, for instance, in a district council.

3. Distribution of Factor Evaluations

3.1 The expected distribution of factor evaluations over the relevant levels can be deduced from the nature of the factor in question and verified by experience in user local authorities. The distribution can be checked by sorting the scores from highest to lowest for each factor on a spreadsheet of results, or using the Gauge system reports system, then by visual inspection or by creating a histogram of the results.

3.2 The following observations on expected factor distributions using the NJC JES apply where all jobs evaluated are distinct from each other. If samples have been taken of several versions of a large population job to check whether they evaluate similarly (e.g. Home Carer), then the distributions may be skewed as a result. The range of jobs employed by an individual local authority may also affect distributions.

3.3 Statistical analysis of distributions also depends on having a relatively large data sample - in a job evaluation exercises, this is a large number of evaluated jobs. For a pilot or benchmark evaluation exercise, caution must be exercised because the distributions may be affected by the sample of jobs selected. In particular, the bulk of jobs, for example, those that are likely to evaluate towards the bottom of a pyramid distribution factor or the middle of a more normal distribution factor are likely to be under-represented in the pilot sample. So, the same pattern should emerge, but a less exaggerated form than with large numbers of jobs:

Knowledge, Initiative and Independence, for these factors a relatively normal

distribution is to be expected, that is with the largest numbers of jobs evaluated at the middle levels, tailing off towards the lowest and highest levels.

Responsibility for Supervision, Responsibility for Financial Resources, Physical Demands, Emotional Demands: for these factors, a `pyramid' distribution is anticipated, with the largest numbers of jobs evaluated at the lowest level, tapering gradually towards the highest levels. This is generally true of most Responsibility and Effort factors.

Skills factors, Responsibility for People, Responsibility for Physical Resources,

Mental Demands: a skewed distribution is to be expected, with the base of the `pyramid' and the bulk of jobs at levels 2 and/or 3 tailing off towards the higher levels, but with relatively few jobs at level 1.

4. Specific Factor Checks

4.1 The construction of any job evaluation scheme gives rise to specific factor checks. These are best made using the sorted level assessments for each factor, as in 2 above, so that the rank order of levels and list of job titles can be scanned for specific points to be checked against the scheme and agreed local conventions:

Knowledge: check that `standard' qualified practitioner jobs from all occupational groups are assessed at the same level, either level 5 or 6, depending on local conventions, and that there is justification for jobs scored above or below this level.

Mental Skills: some jobs exist for their ability to apply knowledge using analytical and/or developmental skills and these jobs should have been assessed towards the higher levels on this factor, even if they do not score particularly highly in some other areas. This category of jobs includes policy, research, statistical and some finance jobs. For other jobs the factor is more hierarchical and manager and supervisor jobs normally score at least one level higher than those they manage of supervise, on account of their greater problem solving, decision-making and probably planning skills.

Communication Skills: check that all jobs assessed above level 3 meet the local conventions for `developed' skills at level 4 and `highly developed' skills at level 5.

Physical Skills: level 5: this was included by the scheme designers for jobs requiring exceptional levels of physical dexterity and co-ordination and is rarely used, for example, in one authority, only for conservators of Roman pottery finds. Any job assessed at this level should be checked.

Initiative & Independence: this is a hierarchical factor, so supervisor and manager jobs should normally be assessed at least one level higher than the jobs of those they supervise or manage. Any jobs that do not fit this pattern should be checked.

Physical Demands: administrative and managerial jobs should normally have been assessed at level 1. Most of the jobs at level 3 and above are likely to be ex-manual jobs. Jobs involving sitting in a constrained position (e.g. data input, word processing) or standing (e.g. crossing patrols) should be at level 2.

Mental Demands: this factor covers 3 distinct aspects of mental demands - sensory attention; mental concentration; work pressures. It is useful to check the rank order of assessments for each of these aspects separately. This can be done, if evaluators have indicated the selected letter option on their rationale sheets.

Emotional Demands: jobs at level 3 and above are normally client-related jobs - not necessarily in social services - could be housing or similar. Jobs at level 2 should be more mixed and could include investigatory or inspecting jobs from any department or area. Any exceptions to these patterns should be investigated.

Responsibility for People: jobs assessed at level 3 and above will normally have direct client-related responsibilities (including policy and/or advisory responsibilities for the group in question). Other public facing and/or service providing jobs are likely to be assessed at level 2. Local authority office-based support jobs with little or no contact with members of the public would normally be at level 1 (because they will score on other factors). Answering telephone queries from members of the public should only take this latter group to level 2 if this is a significant feature of the job.

Responsibility for Supervision: this is the second inevitably hierarchical factor in the NJC JES. Manager and supervisor jobs should normally be assessed at a higher level than the jobs they manage or supervise.

Responsibility for Financial Resources: it is important that jobs which do have responsibilities for finance but do not obviously match the words of the level definitions have been correctly assessed on an equivalence basis (e.g. income generation jobs). It may be necessary to develop local help screens for Gauge to ensure that such jobs are correctly measured (this can be done retrospectively as a result of `sorethumbing' or moderation of Gauge results at the benchmarking stage.

Responsibility for Physical Resources: very many jobs should have been assessed at level 2, on account of the wide range of options available at this level. Few jobs are likely to have been assessed at level 1. Jobs assessed at level 3 and above should have clear and obvious resource responsibility, either as their primary function (e.g. maintenance and estates jobs, transport and roads jobs, information systems jobs); or as a distinct secondary function (e.g. finance job with distinct responsibilities for a financial information system; direct people job with distinct responsibilities for supplies or property).

Working Conditions: as with Physical Demands, administrative and managerial jobs should normally have been assessed at level 1 and many of the jobs at level 3 and above are likely to be ex-manual jobs. However, in this front line service providing jobs, where the work involves going out into the community, to homes or on site visits or inspections, should have been assessed above level 1. Jobs that are primarily outdoors jobs should be at level 3 or above.

5. Overall Range of Total Weighted Scores

5.1 As all jobs must be assessed at level 1 at least on each factor and no job is likely to score at the lowest level on all factors, the minimum possible total weighted score is just over 200 points. Taking into account that managerial and most professional jobs are likely to be assessed at level 1 or 2 on the Physical Skills, Physical Demands and Working Conditions factors, the maximum realistic score is not the theoretical 1000, but more like 800 - 850 points.

A realistic range for the full spread of jobs is from around 220-240 to 750-800, with a slightly lower maximum if more senior managerial jobs are excluded from the exercise. The points range should emerge from a representative benchmark or pilot sample as well as from evaluation of all jobs.

A range from 220 to 770 is very reasonable by comparison with the outcomes of local authorities that have completed their evaluation exercises and should allow for the development of a sensible grading structure.

If a narrower range of scores has emerged or there is excessive `bunching' of jobs in some parts of the overall range, then these should be investigated, taking into account the representativeness of the range of jobs evaluated.

6. Specific Checks of Total Weighted Scores

6.1 Sorting the overall rank order of total weighted scores by department or job family allows for additional checks, for example:

Manager Jobs: should normally score higher overall than the jobs they manage, even where the manager job is office based and the managed job is a front line service providing jobs. The manager job is likely to have scored at least one level higher than the job managed on - Knowledge (20 points); Mental Skills (13 points); Initiative & Independence (13 points); Responsibility for Supervision (13 or 26 points); at least one other Responsibility factor (e.g. level 5 compared with level 4 for Responsibility for People for a direct client job; at least one level higher on Responsibility for Finance for a finance job) (13 points). These greater scores (over 70 points) should outweigh the managed job's higher score(s) for Effort and Environment actors. Any exceptions should be investigated. If there are no problems over either information or evaluation, then this is indicative that the manager job is not what it appears from the job title. This is not an impossible situation as, with the development of jobs over the years, job titles and descriptions may not have kept up with changes, for example, there may have been greater devolvement of responsibilities than the title indicates.

Colleague Jobs: where a sample of jobs with the same or a similar job title have been evaluated, then these should evaluate similarly, although not necessarily identically, but the range should normally be less than 50 points for genuinely similar jobs. Greater differences than this should be investigated. If there are no problems over either information or evaluation, then this is indicative of a misleading job title and suggests that there is more than one job within the group.

Same Organisational Level Jobs: although less exact than the above two checks, it is also worth checking jobs which are considered to be at similar levels in the organisational hierarchy, but in different departments. Other things being equal, these should come out with similar overall total weighted scores, albeit by different routes in terms of which factors they score highly on. So, for example, third and fourth tier management jobs should each come out with similar total scores (say, within 50 points). Standard professional jobs should normally also have similar total weighted scores. Variations from these generalisations may be legitimate, but they should be explicable.

SUMMARY CHECKLIST

Have all factor levels been used?

Have all the intermediate factor levels been used? If not, why not?

Have all top levels been used for the Knowledge and Skills and Responsibility factors? If not, is this because some senior jobs have been excluded from the exercise? If not, have evaluations been unduly compressed?

Have all the top levels been used for the Physical Effort and Working Conditions factors? If not, is this because the range of jobs to be evaluated does not include the most demanding ex-manual jobs (e.g. Refuse Collectors, Sewerage Operatives, Gravediggers)? If not, have evaluations been unduly compressed?

Have the top levels been used for the Emotional Effort factor? If not, is this because the range of jobs to be evaluated does not including the most emotionally demanding social care jobs (e.g. Child and Family Social Workers, Approved Mental Health Social Workers)? If not, have evaluations on this factor been unduly compressed?

Have all bottom levels been used for at least some jobs? If not, have evaluators been unduly generous?

Are the distributions of factor level of assessments as would be expected with the NJC

JES?

Are the Knowledge and Initiative distributions normal, with the bulk of the jobs assessed at the middle levels and the numbers tapering towards both the bottom and top levels?

Are the distributions for the Responsibility for Supervision, Responsibility for Financial Resources, Physical Demands and Emotional Demands `pyramid' shaped, with the majority of jobs assessed at level 1 and decreasing numbers at each of the higher levels?

Are the distributions for the Responsibility for People, Responsibility for Physical Resources, Mental Demands `skewed', with some jobs at level 1, but the most jobs assessed at level 2 and decreasing numbers at each of the higher levels?

Have the factor rank orders been checked?

Knowledge: are all the standard qualified practitioner jobs at the same level, either 5 or 6, depending on local conventions? If not, is there clear justification for any variations?

Mental Skills: are policy, research, advisory and statistician jobs assessed at relatively high levels on this factor?

Communication Skills: do all the jobs assessed at level 4 meet the local convention for `developed skills'? Do all the jobs assessed at level 5 meet the local convention for `highly developed skills'?

Physical Skills: have any jobs been assessed at level 5 on this factor? If so, can that be justified in terms of the level of precision required for the job?

Initiative & Independence: have manager and supervisor jobs been assessed at a higher level than the jobs they manage or supervise?

Physical Demands: are most of the jobs assessed at level 3 or above genuinely physical jobs (probably ex-manual jobs)?

Mental Demands: have separate checks been made on jobs assessed for sensory attention, mental concentration and work pressures; and have each of these aspects been consistently assessed?

Emotional Demands: are the jobs assessed at level 3 and above client-related jobs?

Responsibility for People: do all jobs assessed at level 3 or above have direct client-related responsibilities (including policy and/or advisory responsibilities)?

Responsibility for Supervision: have manager and supervisor jobs been assessed at a higher level than the jobs they manage or supervise?

Responsibility for Financial Resources: have jobs with clear financial responsibilities been correctly assessed on this factor, even if their responsibilities do not immediately match the level definitions (e.g. income generation jobs)?

Responsibility for Physical Resources: do jobs assessed at level 3 or above have clear resource responsibilities as a primary or specified secondary feature?

Working Conditions: have primarily outdoor jobs been assessed at level 3 or above (depending on any additional unpleasant features)? Have jobs involving making regular home or site visits been assessed at level 2 at least?

Is the overall range of total weighted scores between 220-240 and 750-800?

If the highest total weighted score is less than 750, is this because managerial jobs have been excluded from the exercise? If not, have evaluators been too conservative in their assessments?

If the lowest total weighted score is more than 250, is this because the authority does not employ jobs likely to fall at the bottom end of the range? If not, have the evaluators been too generous?

Is there excessive bunching of scores in some parts of the overall range?

Have the total weighted scores been analysed by department or job family?

Do manager jobs score higher overall than the jobs they manage? If not, why not?

Do colleague jobs score similarly to each other, or at least within a range of 50 points? If not, is this because there have been some inconsistent evaluations? If not, is this indicative that the jobs are actually significantly different from each other?

6. Do jobs at the same level in the organisation score similarly or within a range of around 50 points? If not, is there a clear explanation for this?

END

Page 7 of 9