# Spring 2018, Math 171

#### Week 6

1. Exit Distributions

1. A person is terminally ill. On a day when the person is awake, there is an 0.2 chance they will die overnight, and they are equally likely to be awake or unconscious the next day. On a day when the person is unconscious, there is an 0.2 chance they will be awake the next day, and they are equally likely to stay unconscious or die.

1. Let $$X_n$$ be the person’s state on day $$n$$ (awake, unconscious, or dead). Show that $$(X_n)_{n\ge 0}$$ is a Markov chain. Find its transition matrix.

• (Answer) $\begin{matrix}& \mathbf A & \mathbf U & \mathbf D \cr \mathbf A & 0.4 & 0.4 & 0.2 \cr \mathbf U & 0.2 & 0.4 & 0.4 \cr \mathbf D & 0 & 0 & 1\end{matrix}$
2. Compute the probability that the person spends at least one day awake before dying given that they are initially unconscious.

• (Answer) $$\frac{0.2}{1-0.4} = \frac 1 3$$
3. Compute the expected number of days the person will spend awake before dying given that they are initially unconscious.

• (Answer) $$\frac 5 7$$
2. In the Markov chain corresponding to the following transition matrix $\begin{matrix} & \mathbf 1 & \mathbf 2 & \mathbf 3 & \mathbf 4 \cr \mathbf 1 & 0.1 & 0.5 & 0.2 & 0.2 \cr \mathbf 2 & 0.2 & 0.4 & 0.4 & 0 \cr \mathbf 3 & 0 & 0.5 & 0.3 & 0.2 \cr \mathbf 4 & 0.5 & 0.5 & 0 & 0 \end{matrix}$ compute the probability that the chain reaches state 1 before it reaches state 4 for each starting state.

• (Answer) Define $$h(x)=P_x(T_1 < T_4)$$. Then $$h(1)=1$$, $$h(4)=0$$, and $\begin{bmatrix}h(2) \cr h(3)\end{bmatrix} = \left(I - \begin{bmatrix}0.4 & 0.4 \cr 0.5 & 0.3\end{bmatrix}\right)^{-1}\begin{bmatrix}0.2 \cr 0\end{bmatrix}$
3. (Discussed) In the Markov chain corresponding to the following transition matrix $\begin{matrix} & \mathbf 1 & \mathbf 2 & \mathbf 3 & \mathbf 4 & \mathbf 5 \cr \mathbf 1 & 0.1 & 0.5 & 0.2 & 0.2 & 0 \cr \mathbf 2 & 0.4 & 0.2 & 0.3 & 0 & 0.1\cr \mathbf 3 & 0 & 0.5 & 0.3 & 0.2 & 0\cr \mathbf 4 & 0.2 & 0 & 0.5 & 0.1 & 0.2 \cr \mathbf 5 & 0.1 & 0.1 & 0.2 & 0.1 & 0.5 \end{matrix}$ Compute the probability that the chain reaches states 1 or 2 before it reaches state 5 for each starting state.

2. Exit Times

1. In the Markov chain from 1.2 compute the expected time taken to reach state 4 from each of the other states.

• (Answer) $$g(x)=\mathbb E_x[T_4]$$. Then $$g(4)=0$$ and $\begin{bmatrix}g(1) \cr g(2) \cr g(3)\end{bmatrix} = \left(I - \begin{bmatrix}0.1 & 0.5 & 0.2 \cr 0.2 & 0.4 & 0.4 \cr 0 & 0.5 & 0.3 \end{bmatrix}\right)^{-1} \begin{bmatrix}1 \cr 1 \cr 1\end{bmatrix}$
2. (Discussed) In the Markov chain from 1.3 compute the expected time taken to reach either of states 2 or 5 from each of the other states. That is, $$\mathbb E_x[T]$$ where $$T = \min \{n \ge 0 \mid X_n \in \{2, 5\}\}$$