Motivated by a class of Partially Observable Markov Decision Processes with application in surveillance systems in which a set of imperfectly observed state processes is to be inferred from a subset of available observations through a Bayesian approach, we formulate and analyze a special family of multi-armed restless bandit problems. We consider the problem of finding an optimal policy for observing the processes that maximizes the total expected net rewards over an infinite time horizon subject to the resource availability. From the Lagrangian relaxation of the original problem, an index policy can be derived, as long as the existence of the Whittle index is ensured. We demonstrate that such a class of reinitializing bandits in which the projects' state deteriorates while active and resets to its initial state when passive until its completion possesses the structural property of indexability and we further show how to compute the index in closed form. In general, the Whittle index rule for restless bandit problems does not achieve optimality. However, we show that the proposed Whittle index rule is optimal for the problem under study in the case of stochastically heterogenous arms under the expected total criterion, and it is further recovered by a simple tractable rule referred to as the 1-limited Round Robin rule. Moreover, we illustrate the significant suboptimality of other widely used heuristic: the Myopic index rule, by computing in closed form its suboptimality gap. We present numerical studies which illustrate for the more general instances the performance advantages of the Whittle index rule over other simple heuristics.

This paper introduces a special collection of research contributions written to honour Professor Erol Gelenbe of his 70th birthday which was celebrated on September 20–25, 2015 at Imperial College in London where he holds the Dennis Gabor Professorship and is Head of the Intelligent Systems and Networks Group in the Department of Electrical and Electronic Engineering (see http://san.ee.ic.ac.uk/Gelenbe2015). All but three of the fourteen papers presented here were written by his students and descendants. These papers are centered on probability models related to computer systems and networks, an area where Erol plays a pioneering role in which he remains very active with many innovations and new directions that he has introduced in the last ten or twelve years. We first briefly review Erol's work in these areas, and then discuss each of the contributions that appear here together with their links to Erol's own published research.

Let (X1, …, Xn) be a multivariate normal random vector with any mean vector, variances equal to 1 and covariances equal and positive. Turner and Whitehead [9] established that the largest order statistic max{X1, …, Xn} is less than the standard normal random variable in the dispersive order. In this paper, we give a new and straightforward proof for this result. Several possible extensions of this result are also discussed.

For the weighted k-out-of-n system with statistically dependent component lifetimes, we have a discussion on the system performance through investigating the ordering properties of the total system weight with respect to component weight vector. Applications of the present ordering results to signature of coherent systems are presented as well.

We consider a system consisting of a single server serving a fixed number of stations. At each station, there is an infinite queue of customers that have to undergo a preparation phase before being served. This model is connected to layered queueing networks, to an extension of polling systems and surprisingly to random graphs. We are interested in the waiting time of the server. For the case where the server polls the stations cyclically, we give a sufficient condition for the existence of a limiting waiting-time distribution and we study the tail behavior of the stationary waiting time. Furthermore, assuming that preparation times are exponentially distributed, we describe in depth the resulting Markov chain. We also investigate a model variation where the server does not necessarily poll the stations in a cyclic order, but always serves the customer with the earliest completed preparation phase. We show that the mean waiting time under this dynamic allocation never exceeds that of the cyclic case, but that the waiting-time distributions corresponding to both cases are not necessarily stochastically ordered. Finally, we provide extensive numerical results investigating and comparing the effect of the system's parameters to the performance of the server for both models.

We consider a monotone binary system with ternary components. “Ternary” means that each component can be in one of three states: up, middle (mid) and down. Handling such systems is a hard task, even if a part of the components have no mid state. Nevertheless, the permutation Monte Carlo methods, that proved very useful for dealing with binary components, can be efficiently used also for ternary monotone systems. It turns out that for “ternary” system there also exists a combinatorial invariant by means of which it becomes possible to count the number C(r;x) of system failure sets which have a given number r and x of components in up and down states, respectively. This invariant is called ternary D-spectrum and it is an analogue of the D-spectrum (or signature) of a system with binary components. Its value is the knowledge of system failure or path set properties which do not depend on stochastic mechanism governing component failures. In case of independent and identical components, knowing D-spectrum makes it easy to calculate system UP or DOWN probability for a variety of UP/DOWN definitions suitable for systems of many types, like communication networks, flow and supply networks, etc.

We focus on Erol Gelenbe's scientific and technical contributions to probability models in the computer and information sciences, but limit our survey to the last fifteen years. We start with a brief overview of his work as a single author, as well as his work in collaboration with over 200 co-authors. We discuss some of his recent and innovative work regarding a new probability model that represents Intermittent Energy Sources for Computing and Communications, introducing Energy Packet Networks which are a probabilistic representation of the flow, storage and consumption of electrical energy at the microscopic level (in electronic chips), and at the macroscopic level (e.g. in buildings or data centers) and for its routing and dynamic usage by consuming units (such as computer elements, chips or machines). We next discuss his work on designing computer and communication systems that parsimoniously use energy in order to achieve a satisfactory level of quality of service (QoS). Trade-offs between system QoS and energy consumption are also considered. Then we turn to Prof. Gelenbe's pioneering work on Autonomic Communications and the design and implementation of CPN, the Cognitive Packet Network, and we also briefly review his spiking random neural network that was used in CPN. This is followed by a brief review of work that he conducted since 1999 on human evacuation from dangerous or catastrophic environments, and the design of technology driven Emergency Management Systems. His research since the late 2000s on Gene Regulatory Networks is then covered together with its application to the detecting possible disease from microarray data. Finally, we briefly discuss some novel analytical models that he developed in this period with publications appearing in journals of physics and applied mathematics.

We analyze a construction for optimal nested group-testing procedures, and show that, when individuals are independently positive with probability p, the expected number of tests per positive individual, F*(p), has, as p→0, the asymptotic behavior

$$F^{\ast}(p) = \log_2 {1\over p} + \log_2 \log 2 + 2 + f\left(\log_2 {1\over p} + \log_2 \log 2\right) + O(p),$$

$$f(z) = 4\times 2^{-2^{1-\{z\}}} - \{z\} - 1,$$

The purpose of this paper is to analyze the so-called back-off technique of the IEEE 802.11 protocol in broadcast mode with waiting queues. In contrast to existing models, packets arriving when a station (or node) is in back-off state are not discarded, but are stored in a buffer of infinite capacity. As in previous studies, the key point of our analysis hinges on the assumption that the time on the channel is viewed as a random succession of transmission slots (whose duration corresponds to the length of a packet) and mini-slots during which the back-off of the station is decremented. These events occur independently, with given probabilities. The state of a node is represented by a two-dimensional Markov chain in discrete-time, formed by the back-off counter and the number of packets at the station. Two models are proposed both of which are shown to cope reasonably well with the physical principles of the protocol. The stability (ergodicity) conditions are obtained and interpreted in terms of maximum throughput. Several approximations related to these models are also discussed.

This paper investigates continuous-time optimal portfolio and consumption problems under loss aversion in an infinite time horizon. The investor's goal is to choose the optimal portfolio and consumption policies to maximize total discounted S-shaped utility from consumption. The problems are solved under two different situations respectively for the reference level: exogenous or endogenous. For the case of exogenous reference level, which is independent of the consumption policy, the optimal consumption policy and wealth process are obtained through the martingale method and replicating technique. For the case of endogenous reference level, which is related to the past actual consumption, the optimization problem with stochastic reference level is first transformed into an equivalent optimization problem with zero reference point, the corresponding relationship between them is proved, and then the relevant optimal consumption policy and wealth process are also obtained. When the investment opportunity sets are constants, the closed-form solutions of the portfolio and consumption policies are derived under two different situations respectively.

The modified-offered-load approximation can be used to choose a staffing function (the time-varying number of servers) to stabilize delay probabilities at target levels in multi-server delay models with time-varying arrival rates, with or without customer abandonment. In contrast, as we confirm with simulations, it is not possible to stabilize blocking probabilities to the same extent in corresponding loss models, without extra waiting space, because these probabilities necessarily change dramatically after each staffing change. Nevertheless, blocking probabilities can be stabilized provided that we either randomize the times of staffing changes or average the blocking probabilities over a suitably small time interval. We develop systematic procedures and study how to choose the averaging parameters.

Consider a two-station, heterogeneous parallel queueing system in which each station operates as an independent M/M/1 queue with its own infinite-capacity buffer. The input to the system is a Poisson process that splits among the two stations according to a Bernoulli splitting mechanism. However, upon arrival, a strategic customer initially joins one of the queues selectively and decides at subsequent arrival and departure epochs whether to jockey (or switch queues) with the aim of reducing her own sojourn time. There is a holding cost per unit time, and jockeying incurs a fixed non-negative cost while placing the customer at the end of the other queue. We examine individually optimal joining and jockeying policies that minimize the strategic customer's total expected discounted (or undiscounted) costs over finite and infinite time horizons. The main results reveal that, if the strategic customer is in station 1 with ℓ customers in front of her, and q1 and q2 customers in stations 1 and 2, respectively (excluding herself), then the incentive to jockey increases as either ℓ increases or q2 decreases. Numerical examples reveal that it may not be optimal to join, and/or jockey to, the station with the shortest queue or the fastest server.

The objective of this note is to study the distribution of the running maximum of the level in a level-dependent quasi-birth-death process. By considering this running maximum at an exponentially distributed “killing epoch” T, we devise a technique to accomplish this, relying on elementary arguments only; importantly, it yields the distribution of the running maximum jointly with the level and phase at the killing epoch. We also point out how our procedure can be adapted to facilitate the computation of the distribution of the running maximum at a deterministic (rather than an exponential) epoch.

We study the optimal buffer capacity K for the M/M/1/K queue under some standard cost and reward structures by comparing various Markov reward processes. Using explicit expressions for the deviation matrix of the underlying Markov chains, we find the bias optimal value for K in the case of a tie between two consecutive optimal gain policies. We show that the bias optimal value depends both on whether the reward is granted upon arrival or departure of the customers, and on the initial queue size. Moreover, we demonstrate that in some specific cases the optimal policy is threshold-based with respect to the initial queue size.

The introduction of the class of queueing networks called G-networks by Gelenbe has been a breakthrough in the field of stochastic modeling since it has largely expanded the class of models which are analytically or numerically tractable. From a theoretical point of view, the introduction of the G-networks has lead to very important considerations: first, a product-form queueing network may have non-linear traffic equations; secondly, we can have a product-form equilibrium distribution even if the customer routing is defined in such a way that more than two queues can change their states at the same time epoch. In this work, we review some of the classes of product-forms introduced for the analysis of the G-networks with special attention to these two aspects. We propose a methodology that, coherently with the product-form result, allows for a modular analysis of the G-queues to derive the equilibrium distribution of the network.

“Guess Who?” is a popular two player game where players ask “Yes”/“No” questions to search for their opponent's secret identity from a pool of possible candidates. This is modeled as a simple stochastic game. Using this model, the optimal strategy is explicitly found. Contrary to popular belief, performing a binary search is not always optimal. Instead, the optimal strategy for the player who trails is to make certain bold plays in an attempt catch up. This is discovered by first analyzing a continuous version of the game where players play indefinitely and the winner is never decided after finitely many rounds.

A combined optimal dividend/reinsurance problem with two types of insurance claims, namely the expected premium principle and the variance premium principle, is discussed. Dividend payments are considered with both fixed and proportional transaction costs. The objective of an insurer is to determine an optimal dividend–reinsurance policy so as to maximize the expected total value of discounted dividend payments to shareholders up to ruin time. The problem is formulated as an optimal regular-impulse control problem. Closed-form solutions for the value function and optimal dividend–reinsurance strategy are obtained in some particular cases. Finally, some numerical analysis is given to illustrate the effects of safety loading on optimal reinsurance strategy.

We study G-networks with positive and negative customers and signals. We consider two types of signals: they can make a subnetwork of queues operational or down. As signals are sent by queues after a customer service completion, one can model the availability of a sub-network of queues controlled by another network of queues. We prove that under classical assumptions for G-networks and assumptions on the rerouting probabilities when a subnetwork is not operational, the steady-state distribution, if it exists, has a product form steady state distribution. Some examples are given.

Motivated by non-Poisson stochastic variability found in service system arrival data, we extend established service system staffing algorithms using the square-root staffing formula to allow for non-Poisson arrival processes. We develop a general model of the non-Poisson non-stationary arrival process that includes as a special case the non-stationary Cox process (a modification of a Poisson process in which the rate itself is a non-stationary stochastic process), which has been advocated in the literature. We characterize the impact of the non-Poisson stochastic variability upon the staffing through the heavy-traffic limit of the peakedness (ratio of the variance to the mean in an associated stationary infinite-server queueing model), which depends on the arrival process through its central limit theorem behavior. We provide simple formulas to quantify the performance impact of the non-Poisson arrivals upon the staffing decisions, in order to achieve the desired service level. We conduct simulation experiments with non-stationary Markov-modulated Poisson arrival processes with sinusoidal arrival rate functions to demonstrate that the staffing algorithm is effective in stabilizing the time-varying probability of delay at designated targets.

In this paper, we study birth/immigration-death processes under mild (binomial) catastrophes. We obtain explicit expressions for both the time-dependent (transient) and the limiting (equilibrium) factorial moments, which are then used to construct the transient and equilibrium distribution of the population size. We demonstrate that our approach is also applicable to multidimensional systems such as stochastic processes operating under a random environment and other variations of the model at hand. We also obtain various stochastic order results for the number of individuals with respect to the system parameters, as well as the relaxation time.