Browsing by Subject "Peer review"

Now showing 1 - 3 of 3

Open Access
Asymptotically optimal assignments in ordinal evaluations of proposals
(2009) Atmaca, Abdullah
In ordinal evaluations of proposals in peer review systems, a set of proposals is assigned to a fixed set of referees so as to maximize the number of pairwise comparisons of proposals under certain referee capacity and proposal subject constraints. The following two related problems are considered: (1) Assuming that each referee has a capacity to review k out of n proposals, 2 ≤ k ≤ n, determine the minimum number of referees needed to ensure that each pair of proposals is reviewed by at least one referee, (2) Find an assignment that meets the lower bound determined in (1). It is easy to see that one referee is both necessary and sufficient when k = n, and n(n-1)/2 referees are both necessary and sufficient when k = 2. It is shown that 6 referees are both necessary and sufficient when k = n/2. Furthermore it is shown that 11 referees are necessary and 12 are sufficient when k = n/3, and 18 referees are necessary and 20 referees are sufficient when k = n/4. A more general lower bound of n(n-1)/k(k-1) referees is also given for any k, 2 ≤ k ≤ n, and an assignment asymptotically matching this lower bound within a factor of 2 is presented. These results are not only theoretically interesting but they also provide practical methods for efficient assignments of proposals to referees.
Open Access
Ordinal covering using block designs
(IEEE, 2010) Atmaca, Abdullah; Oruc, A.Y.
A frequently encountered problem in peer review systems is to facilitate pairwise comparisons of a given set of documents by as few experts as possible. In [7], it was shown that, if each expert is assigned to review k documents then ⌈n(n-1)/k(k-1)⌉ experts are necessary and ⌈n(2n-k)/k 2⌉ experts are sufficient to cover all n(n-1)/2 pairs of n documents. In this paper, we show that, if √n ≤ k ≤ n/2 then the upper bound can be improved using a new assignnment method based on a particular family of balanced incomplete block designs. Specifically, the new method uses ⌈n(n+k)/k2⌉ experts where n/k is a prime power, n divides k2, and √n ≤ k ≤ n/2. When k = √n , this new method uses the minimum number of experts possible and for all other values of k, where √n < k ≤ n/2, the new upper bound is tighter than the general upper bound given in [7]. ©2010 IEEE.
Open Access
Ordinal evaluation and assignment problems
(IEEE, 2010) Atmaca, Abdullah; Oruç, A. Yavuz
In many assignment problems, a set of documents such as research proposals, promotion dossiers, resumes of job applicants is assigned to a set of experts for ordinal evaluation, ranking, and classification. A desirable condition for such assignments is that every pair of documents is compared and ordered by one or more experts. This condition was modeled as an optimization problem and the number of pairs of documents was maximized for a given incidence relation between a set of documents and a set of experts using a set covering integer programming method in the literature[5]. In this paper, we use a combinatorial approach to derive lower bounds on the number of experts needed to compare all pairs of documents and describe assignments that asymptotically match these bounds. These results are not only theoretically interesting but also have practical implications in obtaining optimal assignments without using complex optimization techniques. ©2010 IEEE.