gstudy {gt4ireval}

G-study (Generalizability)

Description

gstudy runs a G-study with the given data, assuming a fully crossed design (all systems evaluated on the same queries). It can be used to estimate variance components, which can further be used to run a D-study with dstudy.

Usage

gstudy(data, drop = 0)

Arguments

data

A data frame or matrix with the existing effectiveness scores. Systems are columns and queries are rows.

drop

The fraction of worst-performing systems to drop from the data before analysis. Defaults to 0 (include all systems).

Value

An object of class gstudy, with the following components:

n.s, n.q	Number of systems and number of queries of the existing data.
var.s, var.q, var.e	Variance of the system, query, and residual effects.
em.s, em.q, em.e	Mean squares of the system, query and residual components.
call	A list with the existing data and the percentage of systems to drop.

Author(s)

Julián Urbano

References

R.L. Brennan (2001). Generalizability Theory. Springer.

J. Urbano, M. Marrero and D. Martín (2013). On the Measurement of Test Collection Reliability. ACM SIGIR, pp. 393-402.

See Also

dstudy

Examples

g <- gstudy(adhoc3)

# same, but drop the 20% worst systems
g20 <- gstudy(adhoc3, drop = 0.2)