About


The Information and Communication Literacy Test (ICL Test) is a specially developed instrument to assess information and computer literacy of basic school graduates.

What is IC literacy?

We live in a modern civilization known as the Information Age. It is nearly impossible to participate effectively in the life today without the ability to work with information. A person’s Information and Communication literacy (IC literacy) is an indicator of this ability.  

In general, IC literacy refers to the ability to utilize specific computer programs and information technologies. In our case, IC literacy refers to an individual’s ability to use modern technologies to access information, then to define, manage, integrate, evaluate, and communicate the said information in a creative manner, while following ethical aspects and legal norms.

IC literacy refers to a range of skills, including but not limited to:

  • Comprehensive work with information (access information by searching; integrate, manage and analyze information; effectively communicate the information; use information according to the goal assigned).

  • The ability to solve practical tasks using information and communication technologies.

  • The ability to think and interact effectively in the modern digital society using the wide range of information flows.

An individual with a high level of IC literacy will have the skills necessary to live, work and be successful in today’s modern, knowledge-based Information Age.

Why do we need the ICL Test?

The purpose of the ICL Test is to provide a realistic and complex assessment of IC literacy by using practical, scenario-based tasks.

The ICL Test allows to assess the IC literacy level of an individual student, or a group of students. It also provides information about the basic school (lower secondary education) graduates’ preparedness to live in today’s information-based society.

The ICL Test is a vitally important instrument for school principals and education authorities at all levels. The test results enable educators to develop and improve an education policy specific to their school, region, or even their country. Furthermore, the test can provide evaluation of how implementation of the ICT helps the education system to develop as a whole.

The ICL Test target audience

Students graduating basic school (students of 9th Grade)

What is unique about the assessment approach?

Most ICT tests are aimed at defining the level of the technological skills and algorithmic thinking which are traditionally formed in Russian school at ICT classes. Apart from these, ICL Test makes focus on the assessment of not so much technological proficiencies (using different computer programs or technical capacities of a computer) but first and foremost of cognitive skills. Thus, ICL Test allows to assess a student’s ability to use computer and other modern information technologies in order to acquire new knowledge, to communicate, to conduct research. As a result this will help him/her to develop skills of lifelong  learning.

Therefore, the ICL Test ideology implies that digital technologies as such are not the goal   but a possibility to work with information effectively and, as a result, to become successful in the information society.

How is IC competence measured?

The ICL Test is based on 7 aspects of IC literacy (proficiencies):

 

 

Each aspect of IC literacy requires cognitive, ethical, social and technological skills The intention of the ICL test is to measure the cognitive and ethical aptitude  within the context of technological skills. ­­­

The Test structure

Each variant of the ICL Test includes 16 scenario-based tasks. Two hours are given for the test completion.

The ICL Test structure combines simple, medium and complex tasks, application of specific strategies is required to solve each of them.

The ICL Test developers try to motivate participants as much as possible by rotating academic and personal tasks, alternating between simple tasks and that of medium and complex level of difficulty. 

A test-taker uses a wide range of digital technologies, including e-mail, text editors, presentation tools, electronic tables, graphs, databases, multimedia services, social networks and other Internet services.

Each task is based on a real-life situation, contains a  problem to be solved, and is aimed to assess either one or several IC proficiencies. The ICL Test developers motivate participants by rotating challenging academic tasks with practical problem-solving tasks. Thus, a balance between academic and extra-curricular context is achieved.

ICL Test is based on the Evidence Centered Design

The core framework of the ICL Test is a method of collecting evidences (Evidence Centered Design).

Since the proficiencies are values that cannot be measured in the traditional sense, a number of observables (evidences) are used for measuring each of them.  Thus, a level of IC literacy is what we aim to measure while the observables (evidences) allow us to reveal and interpret what cannot be measured directly – level of IC proficiencies. The evidences we receive in the test results provide the information about the level for each of seven IC proficiencies.

Development of each task requires a number of certain procedures to be performed. Each of the procedures should provide an answer to two questions:

  • What exactly in the students’ performance will be an evidence of whether s/he has or not the proficiencies we would like to measure?

  • Which task scenario should be developed so that the observation could be performed and the evidence could be collected?

The idea of the method is that on each level of the task development, the test developers think about what conclusions could be made based on the test results and develop the test in a way that these conclusions could be valid.

Development of each task requires a number of certain procedures to be performed. Each of the procedures should provide an answer to two questions:

  • What exactly in the students’ performance will be an evidence of whether she/he has or not the proficiencies we would like to measure?

  • What task scenario should be developed so that the observation could be performed and the evidence could be collected?

The idea of the method is that on each level of the task development the test developers should think about what conclusions could be made based on the test results and develop the test in a way that these conclusions could be valid.

 

The test task example

The test task: «Discovery of America»

A test-taker has to create an article for the school’s online encyclopedia. This includes choosing a title and illustration, and creating hyperlinks for specific items within the text.

Assessment aspect:  proficiency «Create» (information)

Task specifications:

  • Duration: 10 minutes

  • Context: academic

  • Content: social sciences

The presentation of the task on the screen:

 

 

What should be done? A test-taker has to choose the most suitable title for the article from three suggested titles, select a relevant illustration, identify geographical names that require hyperlinks to the relevant encyclopedia articles, then create the links.

How are the test results scored?

IC literacy level of each test-taker is defined on the ICL Test completion depending  on the level of proficiency for each of seven aspects of IC literacy.  There are five levels of IC literacy:

  • Advanced

  • Above basic

  • Basic

  • Below basic

  • Developing

For better comprehension of results, students receive one of the following responses:

1.                 General assessment of IC literacy:

  • Congratulations! You have outstanding results!  

This response combines «advanced» and «above basic» levels.

  • Well done! Your results are above average for other students of your age.

This response corresponds to the «basic level».

  • Well done! Your results are around average for other students of your age.

This response corresponds to «below basic» and «developing» levels.

2.                 Individual recommendations on skills that require further development:

Examples of the screen on the  test completion:

 

What are the outputs?

The ICL Test results processing enables not only to assess the IC literacy level of a test-taker at the end of test-taking period, but to record and process information on his/her actions during the tasks completion. Reports on test results, both for an individual student and the entire group of students, are available on a website with limited access.

The results of students’ IC literacy tests are available in the following formats:

  • Average results of IC literacy in a group and a sub-group

  • Comparison of these average indicators within sub-groups and/or test completion time

  • Proportion of students with a particular level of IC literacy

  • Comparison of percentage indicators in different sub-groups and/or with account of completion time

  • Results in relation to tasks complexity

  • Results in relation to the IC literacy level.

School representatives or education authorities receive the test results, which enable them to identify general trends in students’ IC literacy development. For this purpose, it is possible to compare an average level of IC literacy in the present year with an average level of IC literacy in previous years or an average level of IC literacy of basic school graduates in a particular region with other regions.

Comparison of data obtained from test-takers’ background questionnaires (an integral part of the ICL Test) enables school representatives to indicate the relationship between the IC literacy of the test-taker and his/her parents’ education level, the type of school, social and economic aspects of his/her life, location and more.

Validity of test results

According to the Standards for Educational and Psychological Testing, “Validity refers to the degree to which evidence and theory support the interpretation of test scores entailed by proposed uses of tests” (AERA, APA, & NCME, 1999, p.9). During test development, a wide range of validity studies took place. Our aim was to gather evidence of construct validity of test results.

This type of evidence is complex and can come from many sources. It can be expressed through various sub-types of validity: convergent, discriminant, concurrent, content and others.

In order to measure convergent, discriminant and concurrent validity, we calculated correlation coefficients of ICL Test results with additional measurements included in our background questionnaire. All results were within expected ranges, which is a strong argument in favor of validity of ICL Test results.

Content validity was measured by expert evaluation and alignment studies. We also performed factor analysis to confirm internal consistency of ICL test items.

Test reliability was measured with Chronbach’s Alpha and corrected split-half reliability coefficients, which showed results ranging from 0.77 to 0.83.

In summation, the results of our analysis indicate construct validity of test results, which means that ICL Test scores may indeed be interpreted as representations of examinees’ IC literacy.

 The ICL Test advantages

1. Assessment of cognitive rather than technological skills

The ICL Test assesses a test-taker’s ability to work effectively with information and solve practical tasks using information and communication technologies, without assessing technical skills.

2. Analysis complexity

The ICL Test enables researchers to assess both IC literacy as a whole and its separate proficiencies.

3. Real-life connection

The test tasks are based on real life situations that participants consistently face. More than 92 percent of students who completed the ICL Test stated that the tasks were interesting to solve.

4. User-friendless and affordability

It is possible to take the ICL Test on any computer, regardless of installed operation systems or Internet access.  

5. Automatic data scoring

Bayes network is used for data scoring. This ensures the methodology transparency and enables test developers to automate analysis and evaluation of ICL Test results.

6. High validity and reliability of assessment

The ICL Test has been developed in the Evidence Centered Design (ECD) framework. The ECD framework is a system that constructs assessments based on evidentiary reasoning, which facilitates coordination between test developers and ensures purposeful use of test items. Developing tests in the ECD framework is considered to be a strong argument in favor of construct validity of test results. Construct validity and high reliability of ICL test results have been further confirmed by various psychometric analyses.

7. Practical character of the test results and possibility for their wide use

The obtained results can be used for formative assessment among 8th and 9th grade students, and final assessment for basic school graduates. This also includes development of the recommendations on how to optimize educational process, and improve the level of IC literacy among students.

8. Successful approbation

The ICL Test has been successfully approbated in the Russian Federation, Great Britain, the Republic of Belarus and the Republic of Armenia.

9. International experts approval

The ICL Test has been approved at an international level. Its development has been conducted under the auspices of International Bank for Reconstruction and Development. It has successfully passed audit in Manchester University and the Center for Advanced Study in Education in New York University.

10. Responsible approach to developing the test tasks

Test developers are aware of their role in development of socially responsible personalities. Compliance with legal and ethical standards of digital technology, instruments and communication tools is obligatory for ICL Test completion.