Inter-coder reliability refers to “the degree to which an independent coder evaluates the characteristics of a piece of information or artifacts and reaches the same conclusion” (Lombard, Snyder-Duch, & Bracken, 2004). The specific implementation process is to invite all coders to participate in the research to code the same data, and calculate the reliability test results between coders. If the coefficient result does not reach the required level, coding correction is required, and the reliability test process of the coding is repeated until the reliability between coders reaches a reasonably acceptable level. When the reliability test coefficient results reach an acceptable level, the coders can proceed to formal coding.

Step 1: Establish a Pretest Pool. Go to the [Coding Pool], and click on [Randomly sample to Pretest Pool]. You can also select a piece of data, and click [Pretest Pool] button on the right side of the data to import it one by one. Go to the [Pretest Pool] to see the imported data in the pool.

Step 2: Coding Test. All coders enter the [Reliability Test]-[Coding Test] section; read the text, and complete the coding test process.

Step 3: After all coders completed the coding test, select coders and reliability formula in [Reliability Calculation], and click [Run] to obtain the reliability test result.

Step 4: In the [Tracking] section of [Reliability Test], you can easily check the coding test situation of each individual coder, and their coding difference, which is convenient for coding correction and improvement.

Step 5: If the reliability between coders reaches a reasonably acceptable level, you can start the formal coding. If the reliability test coefficient result is not good, you need to repeat the reliability test procedures several times until the reliability coefficient reaches a credible level.

For Subject with low inter-coder reliability testing coefficient, coders need to be re-trained and instructed, and reliability coding should be re-tested.

To rebuild the Pretest Pool,  [Empty Pretest Pool] in the [Pretest Pool] section, and all the coding results in the pretest pool will be cleared accordingly. Rebuild a Pretest Pool in the [Coding Pool], and each coder re-codes the content independently in the new pretest pool.

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2004). A call for
standardization in content analysis reliability. Human Communication Research, 30(3), 434-437.