What is Disqualification Control and how to set it up?

Modified on Mon, 20 May, 2024 at 12:03 AM

The Disqualification Control allows you to control how strict or lenient you can be on the way participants answer the survey.

The quality control system is a built-in feature on the inca Platform, which automatically evaluates and may disqualify participants during surveys. A participant's quality score is determined based on demerit points that they may accumulate; and whether they are disqualified or not depends on the DISQUALIFICATION CONTROL you set for your project.

How to set up the DISQUALIFICATION CONTROL

On the Project Overview page, you can use the disqualification control to set whether participants may be disqualified based on their demerit points:

Strict — participants are disqualified at 5 (or more) demerit points

Normal — participants are disqualified at 10 (or more) demerit points

Disable — no participant will be disqualified

Note: Demerits Points are evaluated differently from how participants answer an open-ended questions as well as close-ended question. For close-ended questions, demerits points are only applicable for Popup Cards and Tradeoff questions.

Open Ended Demerit Points

Users may select any combination of the following checks through the Quality Control tab for each open-ended question:

Check Type	Description
Gibberish	Checks for nonsensical responses such us "lakshdjashdfk". Not recommended when answers may be short brand names.
Uninformative Answers	Checks for low effort responses such as "I don't know" or "nothing". Not recommended when low-information responses (e.g. “nothing”) may be considered a valid response.
Duplicate Answer	Checks for responses which match responses given in other open-ended questions which have this check enabled.
Demerit AI (Pertinence/Contextual Consistency)	Intelligently analyze semantics of respondent’s response to question so that it can flag responses which appear semantically irrelevant, such as copy-and-pasted spam or incoherent rambling or a good but off-topic response. E.g. people answer "I like going to Tesco for grocery shopping as it's close to me" to a question "What do you like about this ad?"

Questions with any of these checks enabled will have a maximum demerit point influence which you can control the Quality Control tab. You can select either Strong (10 Demerits Points) Normal (5 Demerit Points) or Weak (2 Demerits Points)

If any of the enabled checks are triggered, then the participant will be allocated demerit points. For open ended questions without probing, this will equal the maximum demerit point influence. When probing is enabled, the maximum demerit point influence is only allocated if all of the responses trigger an enabled check; otherwise, each response contributes a diminishing amount of points, with the prime response having the most influence and each subsequent probe response having less.

Overall these are the main factors that significantly influence the demerit points on open-ended questions:

Completeness of the Response: If the response is incomplete or consists of gibberish strings (e.g., “xwoiqiojiocjvi”), it is highly likely to result in a high demerit points.
Consistency and Appropriateness of Language: If the response is not written in the specified language or differs from the question's language, it is very likely to result in a higher demerit points. Additionally, inappropriate words will also generate a high demerit points.
Pertinence/Contextual Consistency: If the response is pertinent to the question, it will be treated as high quality, resulting in a low demerit points.)
Typographical errors and grammar issues: The demerit scores of flagged participants are also generally affected by typographical errors and grammar issues even though the response is not "bad or low quality"

Close Ended Demerit Points

Close-ended demerit points are included automatically (by default and you can't control the settings), and are currently only available for Popup and Tradeoff question types: straight-lining is detected for each of these, and when detected participant will be given as many demerit points as there are cards/concepts within the question (e.g. a 7-card popup question will give a participant 7 demerit points).

Quality Control Results

The following results are available in inca dashboard and/or data export:

Quality Score — available in data export file as well as a dashboard variable. Determined for each participant based on the descriptions below:

Good Quality — indicates that participant had perfect score (no demerit points).
Disqualified — indicates that participant was disqualified based on their demerit points.
Flagged — indicates that participant did not have perfect score, but was not disqualified. Demerit Points — only available in data export file, and broken down into following types:

Demerit Points — only available in data export file, and broken down into following types:

Total Demerit Points — number of demerit points accumulated over the whole survey.
Close Ended Demerit Points — number of demerit points associated with a specific popup or tradeoff question.
Dialogue Exchange Demerit Points — number of demerit points associated with a participant’s response each open-ended prime or probe question.
Dialogue Demerit Points — number of demerit points accumulated over throughout a single dialogue, i.e. associated with one open-ended prime question and all of its subsequent probe questions.