data_detectors {modeldatatoo}

Predictions from GPT Detectors

Description

Data derived from the paper GPT detectors are biased against non-native English writers. The study authors carried out a series of experiments passing a number of essays to different GPT detection models. Juxtaposing detector predictions for papers written by native and non-native English writers, the authors argue that GPT detectors disproportionately classify real writing from non-native English writers as AI-generated.

Usage

data_detectors(...)

Arguments

...

Arguments passed to pins::pin_read().

Details

A data frame with 6,185 rows and 9 columns:

kind

Whether the essay was written by a "Human" or "AI".

.pred_AI

The class probability from the GPT detector that the inputted text was written by AI.

.pred_class

The uncalibrated class prediction, encoded as if_else(.pred_AI > .5, "AI", "Human")

detector

The name of the detector used to generate the predictions.

native

For essays written by humans, whether the essay was written by a native English writer or not. These categorizations are coarse; values of "Yes" may actually be written by people who do not write with English natively. NA indicates that the text was not written by a human.

name

A label for the experiment that the predictions were generated from.

model

For essays that were written by AI, the name of the model that generated the essay.

document_id

A unique identifier for the supplied essay. Some essays were supplied to multiple detectors. Note that some essays are AI-revised derivatives of others.

prompt

For essays that were written by AI, a descriptor for the form of "prompt engineering" passed to the model.

Value

tibble

tibble print

data_detectors()
#> # A tibble: 6,185 x 9
#>    kind  .pred_AI .pred_class detector     native name  model document_id prompt
#>    <fct>    <dbl> <fct>       <chr>        <chr>  <chr> <chr>       <dbl> <chr> 
#>  1 Human 1.00     AI          Sapling      No     Real~ Human         497 <NA>  
#>  2 Human 0.828    AI          Crossplag    No     Real~ Human         278 <NA>  
#>  3 Human 0.000214 Human       Crossplag    Yes    Real~ Human         294 <NA>  
#>  4 AI    0        Human       ZeroGPT      <NA>   Fake~ GPT3          671 Plain 
#>  5 AI    0.00178  Human       Originality~ <NA>   Fake~ GPT4          717 Eleva~
#>  6 Human 0.000178 Human       HFOpenAI     Yes    Real~ Human         855 <NA>  
#>  7 AI    0.992    AI          HFOpenAI     <NA>   Fake~ GPT3          533 Plain 
#>  8 AI    0.0226   Human       Crossplag    <NA>   Fake~ GPT4          484 Eleva~
#>  9 Human 0        Human       ZeroGPT      Yes    Real~ Human         781 <NA>  
#> 10 Human 1.00     AI          Sapling      No     Real~ Human         460 <NA>  
#> # i 6,175 more rows

glimpse()

tibble::glimpse(data_detectors())
#> Rows: 6,185
#> Columns: 9
#> $ kind        <fct> Human, Human, Human, AI, AI, Human, AI, AI, Human, Human, ~
#> $ .pred_AI    <dbl> 9.999942e-01, 8.281448e-01, 2.137465e-04, 0.000000e+00, 1.~
#> $ .pred_class <fct> AI, AI, Human, Human, Human, Human, AI, Human, Human, AI, ~
#> $ detector    <chr> "Sapling", "Crossplag", "Crossplag", "ZeroGPT", "Originali~
#> $ native      <chr> "No", "No", "Yes", NA, NA, "Yes", NA, NA, "Yes", "No", NA,~
#> $ name        <chr> "Real TOEFL", "Real TOEFL", "Real College Essays", "Fake C~
#> $ model       <chr> "Human", "Human", "Human", "GPT3", "GPT4", "Human", "GPT3"~
#> $ document_id <dbl> 497, 278, 294, 671, 717, 855, 533, 484, 781, 460, 591, 11,~
#> $ prompt      <chr> NA, NA, NA, "Plain", "Elevate using technical", NA, "Plain~

Source

https://simonpcouch.github.io/detectors/ doi:10.1016/j.patter.2023.100779

Examples

data_detectors()