Transparency
Assessment Technical Manual
Last updated: September 2026 · Pre-release (n < 500)
This manual documents the psychometric properties of the TalentPath DigComp 3.0 assessment. Several sections require live data from enough completions to be statistically meaningful. Until at least 500 complete sessions are recorded, those sections show the planned analyses rather than results. Each section states whether it reports live data or a prospectus.
See also: Methodology
Purpose of this manual
The TalentPath DigComp 3.0 assessment issues verifiable credentials that attest to digital competence levels. Any credential that carries consequences — for hiring, compliance, or self-development — owes its audience evidence that the underlying test measures what it claims, does so consistently, and treats candidates fairly.
This manual is that evidence. It follows the reporting framework of the Standards for Educational and Psychological Testing(AERA, APA, & NCME, 2014) and the European Guidelines on Computer-Based and Online Assessment(ATP/E-ATP, 2023). Where the assessment is still below the sample size needed for a given analysis, the section describes the planned method instead.
Test blueprint
Construct definition
The construct is digital competence as defined by DigComp 3.0 (Cosgrove and Cachia, JRC, November 2025), the fifth edition of the EU's digital competence framework. Its five competence areas also underlie the EU's digital skills indicator and the Europass digital skills self-assessment.
Domain structure
Each of the five areas is assessed independently. A candidate may sit any subset. 12 items are drawn per area from the item bank, with stratified sampling so that every competence within the area is represented. Total test length for all five areas is 60 items.
| Area | Items drawn | Item types |
|---|---|---|
| Information and Data Literacy | 12 | Know, Judge |
| Communication and Collaboration | 12 | Know, Judge |
| Digital Content Creation | 12 | Know, Judge |
| Safety and Wellbeing | 12 | Know, Judge |
| Problem Solving | 12 | Know, Judge |
Item writing and review
Items are authored as typed TypeScript records and checked by a build-time linter that enforces the DigComp 3.0 taxonomy, correct answer coding, and test-wiseness rules (no absolute language confined to distractors, no “all of the above”, minimum and maximum stem length, and checks that the key is not given away by option length). Items that fail the linter cannot ship.
Each item is tagged with its DigComp area, competence, proficiency level band (Basic / Intermediate / Advanced), and question type (Know or Judge). Option order is randomised per candidate using a session-derived shuffle key, and the answer key is never sent to the browser.
Scoring model
Scoring uses conjunctive banding: a level is awarded only when the candidate meets both a global threshold and a per-area minimum. This prevents strength in one area from compensating for absence in another.
| Level | Overall minimum | Per-area minimum |
|---|---|---|
| Basic | 50% | 40% |
| Intermediate | 70% | 60% |
| Advanced | 85% | 75% |
Levels 7–8 of DigComp (“Highly advanced”) are never awarded: they describe creating practice that a professional field adopts, which an unsupervised multiple-choice test cannot observe. A 24-hour cooldown between attempts discourages guessing.
Item statistics
Trial items
New items enter the bank as trial items. Each area of a sitting shows up to two of them on top of its scored items. They look like any other question and never count towards a result. They become scored items only once their responses show a sensible difficulty, a positive discrimination and distractors that people actually choose.
Until then, the level an item is filed under is its author’s judgement, not a measurement.
Awaiting data (n < 500)
Once at least 500 complete sessions are recorded, this section will report classical item statistics for each item in the bank:
- Difficulty (p-value) — proportion of candidates answering correctly
- Discrimination (rpbis) — point-biserial correlation between the item score and total domain score
- Distractor analysis — response distribution across options, with flags for non-functioning distractors (selected by fewer than 5% of candidates)
Items with p < 0.20 or p > 0.95 will be reviewed for floor/ceiling effects. Items with rpbis < 0.15 will be reviewed for weak discrimination or possible miskeying.
Reliability
Awaiting data (n < 500)
Reliability evidence will be reported as:
- Internal consistency — Cronbach’s alpha (α) for each domain and for the full 60-item test. Target: α ≥ 0.80 per domain.
- Standard error of measurement (SEM) — expressed in percentage points, so candidates and employers can interpret score precision.
- Decision consistency — proportion of candidates who would be classified at the same level on a hypothetical retest, estimated via the Livingston–Lewis method.
Until n ≥ 500, the thresholds are set from published comparators and expert judgement. The methodology page states this openly.
Standard setting
Awaiting data (n < 500)
The current cut scores (50/40, 70/60, 85/75) are provisional, set by anchoring to DigComp proficiency level descriptors and published digital literacy assessment benchmarks. A formal standard-setting study is planned once item statistics stabilise:
- Method: Modified Angoff — panellists estimate the probability that a minimally competent candidate at each level would answer each item correctly.
- Panel: 6–10 subject-matter experts in digital skills education, drawn from at least 3 EU member states.
- Output: Defensible cut scores with confidence intervals, replacing the current provisional values.
Fairness and DIF
Awaiting data (n < 500)
Differential Item Functioning (DIF) analysis will flag items that perform differently across demographic groups (where candidates have consented to provide that information):
- Method: Mantel–Haenszel procedure for uniform DIF; logistic regression for non-uniform DIF.
- Groups: Language (native/non-native English), age bands, gender (where reported).
- Action: Items flagged with large DIF (C category) will be reviewed and retired or revised.
The assessment is delivered in English only. Items avoid culture-specific references (UK tax forms, US healthcare terminology) and use EU-neutral scenarios. Option order is randomised per candidate to eliminate position bias.
Limitations
The following limitations are inherent to the current assessment design and are stated openly:
- The test is unsupervised and cannot verify candidate identity. The credential attests to a performance, not to the person behind the keyboard.
- Multiple-choice items can only measure knowledge recognition and judgement application, not skill demonstration. Competence areas that require hands-on performance (e.g. programming, content creation) are measured indirectly.
- DigComp levels 7–8 (“Highly advanced”) cannot be assessed by this format and are excluded.
- Cut scores are provisional until a formal standard-setting study is conducted (see section 7).
- The item bank is periodically refreshed; exposure control limits repeated items across attempts, but is not a substitute for bank size. Current bank depth is documented in the methodology.
- The assessment is available in English only. Non-native English speakers may face construct-irrelevant difficulty.
References
- AERA, APA, & NCME (2014). Standards for Educational and Psychological Testing. American Educational Research Association.
- ATP/E-ATP (2023). Guidelines on Computer-Based and Online Assessment. Association of Test Publishers / European Association of Test Publishers.
- Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational Measurement (2nd ed.). American Council on Education.
- Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179–197.
- Vuorikari, R., Kluzer, S., & Punie, Y. (2022). DigComp 2.2: The Digital Competence Framework for Citizens. EUR 31006 EN. Publications Office of the European Union.
- Cosgrove, J., & Cachia, R. (2025). DigComp 3.0: European Digital Competence Framework, fifth edition. JRC144121, EUR 40491. Luxembourg: Publications Office of the European Union.
- Zieky, M. J., & Perie, M. (2006). A Primer on Setting Cut Scores on Tests of Educational Achievement. Educational Testing Service.