Reward-rate maximization in sequential identification under a stochastic deadline
Date
2013Source Title
SIAM Journal on Control and Optimization
Print ISSN
0363-0129
Electronic ISSN
1095-7138
Volume
51
Issue
4
Pages
2922 - 2948
Language
English
Type
ArticleItem Usage Stats
143
views
views
121
downloads
downloads
Abstract
Any intelligent system performing evidence-based decision making under time pressure must negotiate a speed-accuracy trade-off. In computer science and engineering, this is typically modeled as minimizing a Bayes-risk functional that is a linear combination of expected decision delay and expected terminal decision loss. In neuroscience and psychology, however, it is often modeled as maximizing the long-term reward rate, or the ratio of expected terminal reward and expected decision delay. The two approaches have opposing advantages and disadvantages. While Bayes-risk minimization can be solved with powerful dynamic programming techniques unlike reward-rate maximization, it also requires the explicit specification of the relative costs of decision delay and error, which is obviated by reward-rate maximization. Here, we demonstrate that, for a large class of sequential multihypothesis identification problems under a stochastic deadline, the reward-rate maximization is equivalent to a special case of Bayes-risk minimization, in which the optimal policy that attains the minimal risk when the unit sampling cost is exactly the maximal reward rate is also the policy that attains maximal reward rate. We show that the maximum reward rate is the unique unit sampling cost for which the expected total observation cost and expected terminal reward break even under every Bayes-risk optimal decision rule. This interplay between reward-rate maximization and Bayesrisk minimization formulations allows us to show that maximum reward rate is always attained. We can compute the policy that maximizes reward rate by solving an inverse Bayes-risk minimization problem, whereby we know the Bayes risk of the optimal policy and need to find the associated unit sampling cost parameter. Leveraging this equivalence, we derive an iterative dynamic programming procedure for solving the reward-rate maximization problem exponentially fast, thus incorporating the advantages of both the reward-rate maximization and Bayes-risk minimization formulations. As an illustration, we will apply the procedure to a two-hypothesis identification example.
Keywords
Bayes-risk minimizationDynamic programming
Reward-rate maximization
Sequential multihypothesis testing
Speed-accuracy trade off
Bayes-risk minimization
Computer science and engineerings
Dynamic programming techniques
Identification problem
Iterative Dynamic Programming
Multi-hypothesis testing
Optimal decision-rule
Trade off
Costs
Dynamic programming
Economic and social effects
Equivalence classes
Intelligent systems
Inverse problems
Iterative methods
Optimization
Stochastic systems
Decision making