Introduction
Before assessors can assess, they need to know what criteria they're assessing and when they need to assess it.
Company admins are responsible for building the foundations of the Assessment module.
Glossary
Criteria: the performance criteria which assessors assess candidates against
Supplemental Criteria: items that are not directly assessed but underpin other criteria. These usually come in the form of guidance or technical skills.
Level Set: the grading system applied to criteria sets. These are configurable.
Schedule: a candidate's competency plan. This is made up of two aspects; a criteria set and a schedule type which dictates how many assessments the plan must include and when they must occur.
Period: the window of opportunity in which an assessment should occur.
Influx: a status which denotes that a record is in an editable state on ACMS.
Sealed: a status which denotes that a record is in use and therefore in an uneditable state on ACMS.
Deprecated: a status which denotes a record that is no longer in use going forward but current records can still be updated and amended.
Assessment Overview
Criteria Sets
Performance criteria within ACMS are managed using the Criteria Sets Admin.
Menu > Criteria Set
Adding a new criteria set
The Version and Status fields ensure that people are not assessed against out-of-date criteria. There are 4 possible statuses:
- Influx: criteria set can be edited
- Sealed: criteria set cannot be edited
- Deprecated: criteria set is no longer in use
- Deleted: criteria set has been deleted from the system
It is recommended that all Criteria Sets should be 'Sealed' once they are created, as this prevents sets from being changed accidentally. This could result in candidates being signed off on something they haven't been assessed against. If Criteria Sets do need to be edited, admins can right click on a set and select 'Edit' to then change the status of the set to 'Influx.'
Criteria Set Components:
- Criteria Unit: collection of criteria items.
- Criteria Items: individual elements that are assessed against on the schedule.
- Supplemental Criteria: are those that are not directly assessed but underpin other criteria. These usually come in the form of guidance or technical skills. These are added to the criteria set and then associated with the criteria items they relate to.
Editing an existing Criteria Set
Criteria sets that are assigned to schedules which are currently in use (assigned to candidates) cannot be edited. If changes are made to the content of existing criteria sets that have already been used in assessments, this may alter previous assessments retroactively. It is possible to edit influx criteria sets.
Under certain circumstances AssessTech can make minor changes to sealed criteria sets. In order to do so we require a Redmine ticket which explicitly explains the nature and reason for the changes.
In most cases, the best way to make changes to a sealed criteria set is to make a copy of it and make the changes to a new version of the criteria set. If you edit a criteria set in this way, new schedules will need to be created and candidates will need to be migrated over to them. That's why it's best to take time to validate criteria sets when they are first added to ACMS!
Schedule Types
Schedule Types need to be set up before being assigned to a criteria set to form a schedule that can be added to candidates. Schedule Types are used to specify the number of assessments that need to occur within a schedule and when they each need to occur.
Menu > Schedule > Type
On ACMS 'periods' are defined as the window of opportunity in which an assessment should be completed. Each period has the following information:
- Offset - the number of days from the start of the schedule that the assessment window begins
- Number of days - the length of the period in days
As period lengths are measured in days rather than months, it is best to follow the sequence of 30, 30, 31 days to standardise the length of the months in a year.
Flex
In addition to defining the periods, whether a schedule is FLEX or not can be set when creating/editing the Schedule Type.
Where FLEX is set, the period with the highest offset specified on the Schedule Type (i.e. number of days since the start of the schedule) will realign with the candidate schedule end date, should the candidate schedule end date be altered to extend the length of the schedule.
This becomes useful where an extension is required to a summary assessment, for example.
Schedule Type Properties
Schedule Type Properties can be used to enable further configuration. For CMS schedules the most important things to remember are:
- 'Dashboard' field should be set to YES - this adds the schedule to the list of schedules which can be added to a candidate through the candidate dashboard
- 'CMS' field should be set to YES - this makes sure that any assessments associated with this schedule are included in CMS Events
Setting a newly created schedule type property to 'DASHBOARD-NO' allows the new schedule to be created, ready to go, in advance of the go live date.
Schedule Templates
In order to match each schedule type with the relevant criteria set, Schedule Templates must be created.
Menu > Schedule > Schedule Item/Template
The schedule wizard guides you through the steps required to set up a new schedule template, this includes assigning different criteria units to each period. Only criteria sets in a 'Sealed' state can be used in Schedule Templates. It is not necessary to assign criteria units to the Summary period as the system will automatically pick up any un-assessed or review item criteria from throughout the schedule in this period.
The Tracked Criteria Unit doesn't need to be assigned to any of the schedule periods to perform a tracked assessment.
Make sure to tick the Events box if you require assessments associated with the schedule to display in CMS Events. Easy to forget but important to remember!
CDP Schedules
Assessable CDP
An assessable CDP is set up by adding many small schedules, which allows for the bespoke creation of CDP schedules for individuals.
This is done by creating Schedule Types and Templates that open windows for 1 month for each type of assessment that may take place. For example: practical, unobtrusive, simulator, professional discussion, etc.
These can then be added with start dates on an individual to build a CDP schedule unique to that individual. These periods can then be assessed as usual.
Non-Assessable CDP
This schedule does not define the individual interventions that will take place but is a marker that “this candidate is on a CDP”. These can then last anywhere from 1 day to any amount of years.
Non-CMS schedules are created by selecting 'YES' in the 'CMS Disabled' schedule type property.
A more detailed explanation of CDPs is included in the Competence Development Plans guide.
Schedule Deprecation
A schedule template can be deprecated by deprecating the schedule type or criteria set. This will not remove past assessments or schedules assigned to candidates but it prevents the template being used in the future. It is important to check whether each schedule type and criteria set is used in any other schedule templates before deprecating them.
Conditions
Conditions are added to assessments within waypoints to log additional information about the assessment and possible training needs. For example, noting the different weather conditions and daylight and darkness.
Menu > Reference Data > Condition
Condition Type examples:
- Certificates and licences
- Weather
- Assessment methods
Condition examples:
- SCWID carried, train driver licence carried
- Darkness, daylight, icy, misty
Conditions are added in the Conditions admin. The only information required is Name, Description and Condition Type.
Any edits made to conditions will be reflected in past assessments that the condition has been assigned to.
Level Sets
The levels which candidates are assessed against. The level description and colour must be defined, as well as whether each level requires a comment from the assessor.
Menu > Reference Data > Assessment > Level Set
AssessTech default level set:
Typically, 'Tracked' and 'Unassessed' levels remain hidden so that they cannot actually be used to grade criteria.
Overdue Authorisation Reasons
The pre-defined list of reasons which assessors can select from to request authorisation of overdue assessments. Reasons which are valid for a longer period of time (e.g. maternity leave) can be set up so that the assessments are excluded from CMS events.
Menu > Reference Data > Schedule > Overdue Authorisation Reason
Examples:
- Assessor not available (still to be done)
- Candidate not available (still to be done)
- Long Term Sick
- Maternity Leave
- Removed from safety critical work (medical reason)
Overdue authorisation reasons are added in the admin. The only information required is Name, Description and whether or not assessments with this overdue reason should be excluded by default from the CMS events list.
Useful Reports
- Criteria set reports - these reports are generated from the criteria set admin by clicking on the criteria set you would like to run a report for and then selecting the report admin button at the top of the page.
- Criteria set assessed items report
- Criteria set details report
- Criteria set items assessed levels report
- C2.03 Candidate Current Schedule - lists all candidates and their current schedule.
- C3 Candidate No CMS Schedules - report on candidates with no CMS schedule.
- S2 Schedules (Non-CMS) Report - lists all personnel on a Non-CMS schedule.
- S3 Schedules Overrun Report - lists all personnel with a schedule that has overrun.
Useful Widgets
Overdue schedules
Displays the number of overdue schedules on the system. This widget can be edited to show a count for the entire company, a region, a depot or a team.
Configurable Features
- Date Format → Sets the format of the start and end dates for CMS assessments on the candidate dashboard.
- CMS Tracker Criteria Item Code → Defines the item code that denotes whether an assessment is a tracked assessment. This ensures the tracked assessment goes into a completed state automatically.
- Assessment Report URI → Sets the report format for the assessment report.
- Email report on submit → When an assessment is completed the assessment report will automatically send to the assessor who sealed the assessment.
- Candidate comments required → Mandates candidate comments upon an individual sealing their assessment. By default, comments are required when candidates object to an assessment.
- Use Assessment Auto Save → Activates the auto save functionality.
- Expiry (milliseconds, minutes, seconds) → Dictates how often the auto save functionality is applied.
- Include schedule date → Dictates whether the schedule date of tracked assessments is displayed in the list of schedules on the candidate dashboard.
- Show Assessment notes column on dashboard → Dictates whether assessment notes are displayed by default in the assessment tab on the candidate dashboard.
- Show original start column → Dictates whether the original start date and time are displayed by default on the assessment tab of the candidate dashboard.
- Pick future period → Dictates whether or not a period which starts in the future can be assessed on the web.
- Candidate Schedule Period Event Search Type → Default filter applied on CMS events.
- Overdue Start → Number of days to go back to include events in CMS events e.g. Only the last 30 days showing as overdue.
- Percentage required → The required percentage coverage for criteria in order for an assessment period to be counted and removed from CMS events.
- Start tabs open (CMS Events) → Defines whether the rows in CMS events expand.
- aAssess hidden notes → Dictates whether notes display on the candidate schedule screen.
- aAssess not assessable → Dictates whether a non-assessable schedule, e.g. support plan, is visible within the app.
FAQs
One of my criteria sets isn't displayed as an option to assign to a schedule in the schedule wizard, why is this?
Only 'sealed' criteria sets will appear as options so it is likely that the criteria set is still 'influx'. You can change the status of criteria sets by right clicking and selecting 'Edit' in the Criteria Set admin.
Can I make changes to a criteria set which is already in use?
Depending on the nature of the change, AssessTech are able to make changes to criteria sets as long as the request is on a Redmine ticket. Amendments are reflected in any assessments completed against the criteria set therefore if the changes alter the fundamental meaning of the criteria we will not proceed with them.
The overall length of a schedule type is out by a couple of days, why is this?
Due to the fact that period lengths are calculated in days rather than calendar months, you may find that the length of schedules is slightly incorrect. Try using the sequence of 30, 30, 31 days to standardise the length of the months in a year. This is described in the 'Schedule Types' section above.
I've created a new schedule but it's not displaying in the list of available schedules to assign to a candidate, why is this?
Make sure that the 'Dashboard' schedule type property is set to YES otherwise the schedule will be not be available to assign to a candidate.