GenerateHITReviewPolicy {pyMTurkR}

Generate HIT and/or Assignment ReviewPolicies

Description

Generate a HIT ReviewPolicy and/or Assignment ReviewPolicy data structure for use in CreateHIT.

Usage

GenerateHITReviewPolicy(...)

Arguments

...

ReviewPolicy parameters passed as named arguments.

Details

Converts a list of ReviewPolicy parameters into a ReviewPolicy data structure.

A ReviewPolicy works by testing whether an assignment or a set of assignments satisfies a particular condition. If that condition is satisfied, then specified actions are taken. ReviewPolicies come in two “flavors”: Assignment-level ReviewPolicies take actions based on “known” answers to questions in the HIT and HIT-level ReviewPolicies take actions based on agreement among multiple assignments. It is possible to specify both Assignment-level and HIT-level ReviewPolicies for the same HIT.

Assignment-level ReviewPolicies involve checking whether that assignment includes particular (“correct”) answers. For example, an assignment might be tested to see whether a correct answer is given to one question by each worker as a quality control measure. The ReviewPolicy works by checking whether a specified percentage of known answers are correct. So, if a ReviewPolicy specifies two known answers for a HIT and the worker gets one of those known answers correct, the ReviewPolicy scores the assignment at 50 (i.e., 50 percent). The ReviewPolicy can then be customized to take three kinds of actions depending on that score: ApproveIfKnownAnswerScoreIsAtLeast (approve the assignment automatically), RejectIfKnownAnswerScoreIsLessThan (reject the assignment automatically), and ExtendIfKnownAnswerScoreIsLessThan (add additional assignments and/or time to the HIT automatically). The various actions can be combined to, e.g., both reject an assignment and add further assignments if a score is below the threshold, or reject below a threshold and approve above, etc.

HIT-level ReviewPolicies involve checking whether multiple assignments submitted for the same HIT “agree” with one another. Agreement here is very strict: answers must be exactly the same across assignments for them to be a matched. As such, it is probably only appropriate to use closed-ended (e.g., multiple choice) questions for HIT-level ReviewPolicies otherwise ReviewPolicy actions might be taken on irrelevant differences (e.g., word capitalization, spacing, etc.). The ReviewPolicy works by checking whether answers to multiple assignments are the same (or at least whether a specified percentage of answers to a given question are the same). For example, if the goal is to categorize an image into one of three categories, the ReviewPolicy will check whether two of three workers agree on the categorization (known as the “HIT Agreement Score”, which is a percentage of all workers who agree). Depending on the value of the HIT Agreement Score, actions can be taken. As of October 2014, only one action can be taken: ExtendIfHITAgreementScoreIsLessThan (extending the HIT in assignments by the number of assignments specified in ExtendMaximumAssignments or time as specified in ExtendMinimumTimeInSeconds).

Another agreement score (the “Worker Agreement Score”), measured the percentage of a worker's responses that agree with other workers' answers. Depending on the Worker Agreement Score, two actions can be taken: ApproveIfWorkerAgreementScoreIsAtLeast (to approve the assignment automatically) or RejectIfWorkerAgreementScoreIsLessThan (to reject the assignment automatically, with an optional reject reason supplied with RejectReason). A logical value (DisregardAssignmentIfRejected) specifies whether to exclude rejected assignments from the calculation of the HIT Agreement Score.

Note: An optional DisregardAssignmentIfKnownAnswerScoreIsLessThan excludes assignments if those assignments score below a specified “known” answers threshold as determined by a separate Assignment-level ReviewPolicy.

Value

A dictionary object HITReviewPolicy or AssignmentReviewPolicy.

Author(s)

Tyler Burleigh, Thomas J. Leeper

References

API Reference: QuestionForm

API Reference (ReviewPolicies)

APIReference (Data Structure)

Examples

## Not run: 


# Generate a HIT Review Policy with GenerateHITReviewPolicy

lista <- list(QuestionIds = c("Question1", "Question2"),
              QuestionAgreementThreshold = 75,
              ApproveIfWorkerAgreementScoreIsAtLeast = 75,
              RejectIfWorkerAgreementScoreIsLessThan = 25)
policya <- do.call(GenerateHITReviewPolicy, lista)


# Manually define a HIT Review Policy

policya <- dict(
		list(
			'PolicyName' = 'SimplePlurality/2011-09-01',
			'Parameters' = list(
			  dict(
				  'Key' = 'QuestionIds',
				  'Values' = list(
				    'Question1',
				    'Question2'
			  	)
			  ),
			  dict(
			  	'Key' = 'QuestionAgreementThreshold',
			  	'Values' = list(
			  	  '75'
			  	)
			  ),
			  dict(
			  	'Key' = 'ApproveIfWorkerAgreementScoreIsAtLeast',
			  	'Values' = list(
			  	  '75'
			  	)
			  ),
			  dict(
			  	'Key' = 'RejectIfWorkerAgreementScoreIsLessThan',
			  	'Values' = list(
			  	  '25'
			  	)
			  )
			)
		))



# Generate an Assignment Review Policy with GenerateAssignmentReviewPolicy

listb <- list(AnswerKey = list("QuestionId1" = "B", "QuestionId2" = "A"),
              ApproveIfKnownAnswerScoreIsAtLeast = 99)
policyb <- do.call(GenerateAssignmentReviewPolicy, listb)


# Manually define an Assignment Review Policy

policyb <- dict(
		list(
			'PolicyName' = 'ScoreMyKnownAnswers/2011-09-01',
			'Parameters' = list(
			  dict(
				  'Key' = 'AnswerKey',
				  'MapEntries' = list(
					dict(
						'Key' = 'QuestionId1',
						'Values' = list('B')
					),
					dict(
						'Key' = 'QuestionId2',
						'Values' = list('A')
					)
			  	)
			  ),
			  dict(
			  	'Key' = 'ApproveIfKnownAnswerScoreIsAtLeast',
			  	'Values' = list(
			  	  '99'
			  	)
			  )
			)
		))


## End(Not run)