Lecture 16: Markov Chains I

• Download this transcript - PDF (English - US)

The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To make a donation or view additional materials from hundreds of MIT courses, visit MIT OpenCourseWare at ocw.mit.edu.

PROFESSOR: So we're going to start now with a new chapter. We're going to talk about Markov processes. The good news is that this is a subject that is a lot more intuitive and simple in many ways than, let's say, the Poisson processes. So hopefully this will be enjoyable.

So Markov processes is, a general class of random processes. In some sense, it's more elaborate than the Bernoulli and Poisson processes, because now we're going to have dependencies between difference times, instead of having memoryless processes. So the basic idea is the following. In physics, for example, you write down equations for how a system evolves that has the general form. The new state of a system one second later is some function of old state.

So Newton's equations and all that in physics allow you to write equations of this kind. And so if that a particle is moving at a certain velocity and it's at some location, you can predict when it's going to be a little later. Markov processes have the same flavor, except that there's also some randomness thrown inside the equation. So that's what Markov process essentially is. It describes the evolution of the system, or some variables, but in the presence of some noise so that the motion itself is a bit random.

So this is a pretty general framework. So pretty much any useful or interesting random process that you can think about, you can always described it as a Markov process if you define properly the notion of the state. So what we're going to do is we're going to introduce the class of Markov processes by, example, by talking about the checkout counter in a supermarket. Then we're going to abstract from our example so that we get a more general definition. And then we're going to do a few things, such as how to predict what's going to happen n time steps later, if we start at the particular state. And then talk a little bit about some structural properties of Markov processes or Markov chains.

So here's our example. You go to the checkout counter at the supermarket, and you stand there and watch the customers who come. So customers come, they get in queue, and customers get served one at a time. So the discussion is going to be in terms of supermarket checkout counters, but the same story applies to any service system. You may have a server, jobs arrive to that server, they get put into the queue, and the server processes those jobs one at a time.

Now to make a probabilistic model, we need to make some assumption about the customer arrivals and the customer departures. And we want to keep things as simple as possible to get started. So let's assume that customers arrive according to a Bernoulli process with some parameter b. So essentially, that's the same as the assumption that the time between consecutive customer arrivals is a geometric random variable with parameter b.

Another way of thinking about the arrival process-- that's not how it happens, but it's helpful, mathematically, is to think of someone who's flipping a coin with bias equal to b. And whenever the coin lands heads, then a customer arrives. So it's as if there's a coin flip being done by nature that decides the arrivals of the customers. So we know that coin flipping to determine the customer arrivals is the same as having geometric inter-arrival times. We know that from our study of the Bernoulli process.

OK. And now how about the customer service times. We're going to assume that-- OK. If there is no customer in queue, no one being served, then of course, no one is going to depart from the queue. But if there a customer in queue, then that customer starts being served, and is going to be served for a random amount of time. And we make the assumption that the time it takes for the clerk to serve the customer has a geometric distribution with some known parameter q.

So the time it takes to serve a customer is random, because it's random how many items they got in their cart, and how many coupons they have to unload and so on. So it's random. In the real world, it has some probability distribution. Let's not care exactly about what it would be in the real world, but as a modeling approximation or just to get started, let's pretend that customer service time are well described by a geometric distribution, with a parameter q.

An equivalent way of thinking about the customer service, mathematically, would be, again, in terms of coin flipping. That is, the clerk has a coin with a bias, and at each time slot the clerk flips the coin. With probability q, service is over. With probability 1-q, you continue the service process.

An assumption that we're going to make is that the coin flips that happen here to determine the arrivals, they're all independent of each other. The coin flips that determine the end of service are also independent from each other. But also the coin flips involved here are independent from the coin flips that happened there. So how arrivals happen is independent with what happens at the service process.

OK. So suppose now you want to answer a question such as the following. The time is 7:00 PM. What's the probability that the customer will be departing at this particular time? Well, you say, it depends. If the queue is empty at that time, then you're certain that you're not going to have a customer departure. But if the queue is not empty, then there is probability q that a departure will happen at that time.

So the answer to a question like this has something to do with the state of the system at that time. It depends what the queue is. And if I ask you, will the queue be empty at 7:10? Well, the answer to that question depends on whether at 7 o'clock whether the queue was huge or not. So knowing something about the state of the queue right now gives me relevant information about what may happen in the future.

So what is the state of the system? Therefore we're brought to start using this term. So the state basically corresponds to anything that's relevant. Anything that's happening right now that's kind of relevant to what may happen in the future. Knowing the size of the queue right now, is useful information for me to make predictions about what may happen 2 minutes later from now. So in this particular example, a reasonable choice for the state is to just count how many customers we have in the queue. And let's assume that our supermarket building is not too big, so it can only hold 10 people.

So we're going to limit the states. Instead of going from 0 to infinity, we're going to truncate our model at ten. So we have 11 possible states, corresponding to 0 customers in queue, 1 customer in queue, 2 customers, and so on, all the way up to 10. So these are the different possible states of the system, assuming that the store cannot handle more than 10 customers. So this is the first step, to write down the set of possible states for our system. Then the next thing to do is to start describing the possible transitions between the states.

At any given time step, what are the things that can happen? We can have a customer arrival, which moves the state 1 higher. We can have a customer departure, which moves the state 1 lower. There's a possibility that nothing happens, in which case the state stays the same. And there's also the possibility of having simultaneously an arrival and a departure, in which case the state again stays the same.

So let's write some representative probabilities. If we have 2 customers, the probability that during this step we go down, this is the probability that we have a service completion, but to no customer arrival. So this is the probability associated with this transition. The other possibility is that there's a customer arrival, which happens with probability p, and we do not have a customer departure, and so the probability of that particular transition is this number.

And then finally, the probability that we stay in the same state, this can happen in 2 possible ways. One way is that we have an arrival and a departure simultaneously. And the other possibility is that we have no arrival and no departure, so that the state stays the same. So these transition probabilities would be the same starting from any other states, state 3, or state 9, and so on. Transition probabilities become a little different at the borders, at the boundaries of this diagram, because if you're in a state 0, then you cannot have any customer departures.

There's no one to be served, but there is a probability p that the customer arrives, in which case the number of customers in the system goes to 1. Then probability 1-p, nothing happens. Similarly with departures, if the system is full, there's no room for another arrival. But we may have a departure that happens with probability q, and nothing happens with probability 1-q. So this is the full transition diagram annotated with transition probabilities.

And this is a complete description of a discrete time, finite state Markov chain. So this is a complete probabilistic model. Once you have all of these pieces of information, you can start calculating things, and trying to predict what's going to happen in the future. Now let us abstract from this example and come up with a more general definition. So we have this concept of the state which describes the current situation in the system that we're looking at.

The current state is random, so we're going to think of it as a random variable Xn is the state, and transitions after the system started operating. So the system starts operating at some initial state X0, and after n transitions, it moves to state Xn. Now we have a set of possible states. State 1 state 2, state 3, and in general, state i and state j. To keep things simple, we assume that the set of possible states is a finite set.

As you can imagine, we can have systems in which the state space is going to be infinite. It could be discrete, or continuous. But all that is more difficult and more complicated. It makes sense to start from the simplest possible setting where we just deal with the finite state space.

And time is discrete, so we can think of this state in the beginning, after 1 transition, 2 transitions, and so on. So we're in discrete time and we have finite in many states. So the system starts somewhere, and at every time step, the state is, let's say, here. A whistle blows, and the state jumps to a random next state. So it may move here, or it may move there, or it may move here, or it might stay in the place. So one possible transition is the transition before you jump, and just land in the same place where you started from.

Now we want to describe the statistics of these transitions. If I am at that state, how likely is it to that, next time, I'm going to find myself at that state? Well, we describe the statistics of this transition by writing down a transition probability, the transition probability of going from state 3 to state 1. So this transition probability is to be thought of as a conditional probability. Given that right now I am at state i what is the probability that next time I find myself at state j? So given that right now I am at state 3, P31 is the probability that the next time I'm going to find myself at state 1.

Similarly here, we would have a probability P3i, which is the probability that given that right now I'm at state 3, next time I'm going to find myself at state i. Now one can write such conditional probabilities down in principle, but we need to make-- so you might think of this as a definition here, but we need to make one additional big assumption, and this is the assumption that to make a process to be a Markov process. This is the so-called Markov property, and here's what it says. Let me describe it first in words here.

Every time that I find myself at state 3, the probability that next time I'm going to find myself at state 1 is this particular number, no matter how I got there. That is, this transition probability is not affected by the past of the process. It doesn't care about what path I used to find myself at state 3.

Mathematically, it means the following. You have this transition probability that from state i jump to state j. Suppose that I gave you some additional information, that I told you everything else that happened in the past of the process, everything that happened, how did you get to state i? The assumption we're making is that this information about the past has no bearing in making predictions about the future, as long as you know where you are right now. So if I tell you, right now, you are at state i, and by the way, you got there by following a particular path, you can ignore the extra information of the particular path that you followed.

You only take into account where you are right now. So every time you find yourself at that state, no matter how you got there, you will find yourself next time at state 1 with probability P31. So the past has no bearing into the future, as long as you know where you are sitting right now. For this property to happen, you need to choose your state carefully in the right way. In that sense, the states needs to include any information that's relevant about the future of the system. Anything that's not in the state is not going to play a role, but the state needs to have all the information that's relevant in determining what kind of transitions are going to happen next.

So to take an example, before you go to Markov process, just from the deterministic world, if you have a ball that's flying up in the air, and you want to make predictions about the future. If I tell you that the state of the ball is the position of the ball at the particular time, is that enough for you to make predictions where the ball is going to go next? No. You need to know both the position and the velocity. If you know position and velocity, you can make predictions about the future.

So the state of a ball that's flying is position together with velocity. If you were to just take position, that would not be enough information, because if I tell you current position, and then I tell you past position, you could use the information from the past position to complete the trajectory and to make the prediction. So information from the past is useful if you don't know the velocity. But if both position and velocity, you don't care how you got there, or what time you started. From position and velocity, you can make predictions about the future.

So there's a certain art, or a certain element of thinking, a non-mechanical aspect into problems of this kind, to figure out which is the right state variable. When you define the state of your system, you need to define it in such a way that includes all information that has been accumulated that has some relevance for the future. So the general process for coming up with a Markov model is to first make this big decision of what your state variable is going to be. Then you write down if it may be a picture of the different states. Then you identify the possible transitions.

So sometimes the diagram that you're going to have will not include all the possible arcs. You would only show those arcs that correspond to transitions that are possible. For example, in the supermarket example, we did not have a transition from state 2 to state 5, because that cannot happen. You can only have 1 arrival at any time. So in the diagram, we only showed the possible transitions.

And for each of the possible transitions, then you work with the description of the model to figure out the correct transition probability. So you got the diagram by writing down transition probabilities.

OK, so suppose you got your Markov model. What will you do with it? Well, what do we need models for? We need models in order to make predictions, to make probabilistic predictions. So for example, I tell you that the process started in that state. You let it run for some time. Where do you think it's going to be 10 time steps from now? That's a question that you might want to answer.

Since the process is random, there's no way for you to tell me exactly where it's going to be. But maybe you can give me probabilities. You can tell me, with so much probability, the state would be there. With so much probability, the state would be there, and so on. So our first exercise is to calculate those probabilities about what may happen to the process a number of steps in the future. It's handy to have some notation in here.

So somebody tells us that this process starts at the particular state i. We let the process run for n transitions. It may land at some state j, but that state j at which it's going to land is going to be random. So we want to give probabilities. Tell me, with what probability the state, n times steps later, is going to be that particular state j?

The shorthand notation is to use this symbol here for the n-step transition probabilities that you find yourself at state j given that you started at state i. So the way these two indices are ordered, the way to think about them is that from i, you go to j. So the probability that from i you go to j if you have n steps in front of you. Some of these transition probabilities are, of course easy to write. For example, in 0 transitions, you're going to be exactly where you started. So this probability is going to be equal to 1 if i is equal to j, And 0 if i is different than j.

That's an easy one to write down. If you have only 1 transition, what's the probability that 1 step later you find yourself in state j given that you started at state i? What is this? These are just the ordinary 1-step transition probabilities that we are given in the description of the problem. So by definition, the 1-step transition probabilities are of this form. This equality is correct just because of the way that we defined those two quantities.

Now we want to say something about the n-step transition probabilities when n is a bigger number. OK. So here, we're going to use the total probability theorem. So we're going to condition in two different scenarios, and break up the calculation of this quantity, by considering the different ways that this event can happen. So what is the event of interest? The event of interest is the following. At time 0 we start i. We are interested in landing at time n at the particular state j.

Now this event can happen in several different ways, in lots of different ways. But let us group them into subgroups. One group, or one sort of scenario, is the following. During the first n-1 time steps, things happen, and somehow you end up at state 1. And then from state 1, in the next time step you make a transition to state j. This particular arc here actually corresponds to lots and lots of different possible scenarios, or different spots, or different transitions. In n-1 time steps, there's lots of possible ways by which you could end up at state 1. Different paths through the state space.

But all of them together collectively have a probability, which is the (n-1)-step transition probability, that from state i, you end up at state 1

And then there's other possible scenarios. Perhaps in the first n-1 time steps, you follow the trajectory that took you at state m. And then from state m, you did this transition, and you ended up at state j. So this diagram breaks up the set of all possible trajectories from i to j into different collections, where each collection has to do with which one happens to be the state just before the last time step, just before time n. And we're going to condition on the state at time n-1.

So the total probability of ending up at state j is the sum of the probabilities of the different scenarios -- the different ways that you can get to state j. If we look at that type of scenario, what's the probability of that scenario happening? With probability Ri1(n-1), I find myself at state 1 at time n-1. This is just by the definition of these multi-step transition probabilities. This is the number of transitions.

The probability that from state i, I end up at state 1. And then given that I found myself at state 1, with probability P1j, that's the transition probability, next time I'm going to find myself at state j. So the product of these two is the total probability of my getting from state i to state j through state 1 at the time before. Now where exactly did we use the Markov assumption here? No matter which particular path we used to get from i to state 1, the probability that next I'm going to make this transition is that same number, P1j.

So that number does not depend on the particular path that I followed in order to get there. If we didn't have the Markov assumption, we should have considered all possible individual trajectories here, and then we would need to use the transition probability that corresponds to that particular trajectory. But because of the Markov assumption, the only thing that matters is that right now we are at state 1. It does not matter how we got there.

So now once you see this scenario, then this scenario, and that scenario, and you add the probabilities of these different scenarios, you end up with this formula here, which is a recursion. It tells us that once you have computed the (n-1)-step transition probabilities, then you can compute also the n-step transition probabilities. This is a recursion that you execute or you run for all i's and j's simultaneously. That is fixed. And for a particular n, you calculate this quantity for all possible i's, j's, k's. You have all of those quantities, and then you use this equation to find those numbers again for all the possible i's and j's.

Now this is formula which is always true, and there's a big idea behind the formula. And now there's variations of this formula, depending on whether you're interested in something that's slightly different. So for example, if you were to have a random initial state, somebody gives you the probability distribution of the initial state, so you're told that with probability such and such, you're going to start at state 1. With that probability, you're going to start at state 2, and so on. And you want to find the probability at the time n you find yourself at state j.

Well again, total probability theorem, you condition on the initial state. With this probability you find yourself at that particular initial state, and given that this is your initial state, this is the probability that n time steps later you find yourself at state j. Now building again on the same idea, you can run every recursion of this kind by conditioning at different times. So here's a variation. You start at state i. After 1 time step, you find yourself at state 1, with probability pi1, and you find yourself at state m with probability Pim. And once that happens, then you're going to follow some trajectories. And there is a possibility that you're going to end up at state j after n-1 time steps.

This scenario can happen in many possible ways. There's lots of possible paths from state 1 to state j. There's many paths from state 1 to state j. What is the collective probability of all these transitions? This is the event that, starting from state 1, I end up at state j in n-1 time steps. So this one has here probability R1j of n-1. And similarly down here. And then by using the same way of thinking as before, we get the formula that Rij(n) is the sum over all k's of Pik, and then the Rkj(n-1).

So this formula looks almost the same as this one, but it's actually different. The indices and the way things work out are a bit different, but the basic idea is the same. Here we use the total probability theory by conditioning on the state just 1 step before the end of our time horizon. Here we use total probability theorem by conditioning on the state right after the first transition. So this generally idea has different variations. They're all valid, and depending on the context that you're dealing with, you might want to work with one of these or another.

So let's illustrate these calculations in terms of an example. So in this example, we just have 2 states, and somebody gives us transition probabilities to be those particular numbers. Let's write down the equations. So the probability that starting from state 1, I find myself at state 1 n time steps later. This can happen in 2 ways. At time n-1, I might find myself at state 2. And then from state 2, I make a transition back to state 1, which happens with probability-- why'd I put 2 there -- anyway, 0.2. And another way is that from state 1, I go to state 1 in n-1 steps, and then from state 1 I stay where I am, which happens with probability 0.5.

So this is for R11(n). Now R12(n), we can write a similar recursion for this one. On the other hand, seems these are probabilities. The state at time n is going to be either state 1 or state 2. So these 2 numbers need to add to 1, so we can just write this as 1 - R11(n). And this is an enough of a recursion to propagate R11 and R12 as time goes on. So after n-1 transitions, either I find myself in state 2, and then there's a point to transition that I go to 1, or I find myself in state 1, which with that probability, and from there, I have probability 0.5 of staying where I am.

Now let's start calculating. As we discussed before, if I start at state 1, after 0 transitions I'm certain to be at state , and I'm certain not to be at state 1. If I start from state 1, I'm certain to not to be at state at that time, and I'm certain that I am right now, it's state 1. After I make transition, starting from state 1, there's probability 0.5 that I stay at state 1. And there's probability 0.5 that I stay at state 2. If I were to start from state 2, the probability that I go to 1 in 1 time step is this transition that has probability 0.2, and the other 0.8.

OK. So the calculation now becomes more interesting, if we want to calculate the next term. How likely is that at time 2, I find myself at state 1? In order to be here at state 1, this can happen in 2 ways. Either the first transition left me there, and the second transition is the same. So these correspond to this 0.5, that the first transition took me there, and the next transition was also of the same kind. That's one possibility. But there's another scenario. In order to be at state 1 at time 2 -- this can also happen this way. So that's the event that, after 1 transition, I got there. And the next transition happened to be this one.

So this corresponds to 0.5 times 0.2. It corresponds to taking the 1-step transition probability of getting there, times the probability that from state 2 I move to state 1, which in this case, is 0.2. So basically we take this number, multiplied with 0.2, and then add those 2 numbers. And after you add them, you get 0.35. And similarly here, you're going to get 0.65.

And now to continue with the recursion, we keep doing the same thing. We take this number times 0.5 plus this number times 0.2. Add them up, you get the next entry. Keep doing that, keep doing that, and eventually you will notice that the numbers start settling into a limiting value at 2/7. And let's verify this. If this number is 2/7, what is the next number going to be? The next number is going to be 2/7 -- (not 2.7) -- it's going to be 2/7. That's the probability that I find myself at that state, times 0.5-- that's the next transition that takes me to state 1 -- plus 5/7-- that would be the remaining probability that I find myself in state 2 -- times 1/5. And so that gives me, again, 2/7.

So this calculation basically illustrates, if this number has become 2/7, then the next number is also going to be 2/7. And of course this number here is going to have to be 5/7. And this one would have to be again, the same, 5/7. So the probability that I find myself at state 1, after a long time has elapsed, settles into some steady state value. So that's an interesting phenomenon. We just make this observation.

Now we can also do the calculation about the probability, starting from state 2. And here, you do the calculations -- I'm not going to do them. But after you do them, you find this probability also settles to 2/7 and this one also settles to 5/7. So these numbers here are the same as those numbers. What's the difference between these?

This is the probability that I find myself at state 1 given that I started at 1. This is the probability that I find myself at state 1 given that I started at state 2. These probabilities are the same, no matter where I started from. So this numerical example sort of illustrates the idea that after the chain has run for a long time, what the state of the chain is, does not care about the initial state of the chain.

So if you start here, you know that you're going to stay here for some time, a few transitions, because this probability is kind of small. So the initial state does that's tell you something. But in the very long run, transitions of this kind are going to happen. Transitions of that kind are going to happen. There's a lot of randomness that comes in, and that randomness washes out any information that could come from the initial state of the system.

We describe this situation by saying that the Markov chain eventually enters a steady state. Where a steady state, what does it mean it? Does it mean the state itself becomes steady and stops at one place? No, the state of the chain keeps jumping forever. The state of the chain will keep making transitions, will keep going back and forth between 1 and 2. So the state itself, the Xn, does not become steady in any sense.

What becomes steady are the probabilities that describe Xn. That is, after a long time elapses, the probability that you find yourself at state 1 becomes a constant 2/7, and the probability that you find yourself in state 2 becomes a constant. So jumps will keep happening, but at any given time, if you ask what's the probability that right now I am at state 1, the answer is going to be 2/7.

Incidentally, do the numbers sort of makes sense? Why is this number bigger than that number? Well, this state is a little more sticky than that state. Once you enter here, it's kind of harder to get out. So when you enter here, you spend a lot of time here. This one is easier to get out, because the probability is 0.5, so when you enter there, you tend to get out faster. So you keep moving from one to the other, but you tend to spend more time on that state, and this is reflected in this probability being bigger than that one. So no matter where you start, there's 5/7 probability of being here, 2/7 probability being there.

So there were some really nice things that happened in this example. The question is, whether things are always as nice for general Markov chains. The two nice things that happened where the following-- as we keep doing this calculation, this number settles to something. The limit exists. The other thing that happens is that this number is the same as that number, which means that the initial state does not matter. Is this always the case? Is it always the case that as n goes to infinity, the transition probabilities converge to something?

And if they do converge to something, is it the case that the limit is not affected by the initial state i at which the chain started? So mathematically speaking, the question we are raising is whether Rij(n) converges to something. And whether that something to which it converges to has only to do with j. It's the probability that you find yourself at state j, and that probability doesn't care about the initial state. So it's the question of whether the initial state gets forgotten in the long run.

So the answer is that usually, or for nice chains, both of these things will be true. You get the limit which does not depend on the initial state. But if your chain has some peculiar or unique structure, this might not happen. So let's think first about the issue of convergence. So convergence, as n goes to infinity at a steady value, really means the following. If I tell you a lot of time has passed, then you tell me, OK, the state of the probabilities are equal to that value without having to consult your clock. If you don't have convergence, it means that Rij can keep going up and down, without settling to something. So in order for you to tell me the value of Rij, you need to consult your clock to check if, right now, it's up or is it down.

So there's some kind of periodic behavior that you might get when you do not get convergence, and this example here illustrates it. So what's happened in this example? Starting from state 2, next time you go here, or there, with probability half. And then next time, no matter where you are, you move back to state 2. So this chain has some randomness, but the randomness is kind of limited type. You go out, you come in. You go out, you come in. So there's a periodic pattern that gets repeated. It means that if you start at state 2 after an even number of steps, you are certain to be back at state 2. So this probability here is 1.

On the other hand, if the number of transitions is odd, there's no way that you can be at your initial state. If you start here, at even times you would be here, at odd times you would be there or there. So this probability is 0. As n goes to infinity, these probabilities, the n-step transition probability does not converge to anything. It keeps alternating between 0 and 1. So convergence fails.

This is the main mechanism by which convergence can fail if your chain has a periodic structure. And we're going to discuss next time that, if periodicity absent, then we don't have an issue with convergence. The second question if we have convergence, whether the initial state matters or not. In the previous chain, where you could keep going back and forth between states 1 and 2 numerically, one finds that the initial state does not matter. But you can think of situations where the initial state does matter. Look at this chain here.

If you start at state 1, you stay at state 1 forever. There's no way to escape. So this means that R11(n) is 1 for all n. If you start at state 3, you will be moving between stage 3 and 4, but there's no way to go in that direction, so there's no way that you go to state 1. And for that reason, R31 is 0 for all n.

OK So this is a case where the initial state matters. R11 goes to a limit, as n goes to infinity, because it's constant. It's always 1 so the limit is 1. R31 also has a limit. It's 0 for all times. So these are the long term probabilities of finding yourself at state 1. But those long-term probabilities are affected by where you started. If you start here, you're sure that's, in the long term, you'll be here. If you start here, you're sure that, in the long term, you will not be there. So the initial state does matter here.

And this is a situation where certain states are not accessible from certain other states, so it has something to do with the graph structure of our Markov chain. Finally let's answer this question here, at least for large n's. What do you think is going to happen in the long term if you start at state 2? If you start at state 2, you may stay at state 2 for a random amount of time, but eventually this transition will happen, or that transition would happen. Because of the symmetry, you are as likely to escape from state 2 in this direction, or in that direction, so there's probability 1/2 that, when the transition happens, the transition happens in that direction.

So for large N, you're certain that the transition does happen. And given that the transition has happened, it has probability 1/2 that it has gone that particular way. So clearly here, you see that the probability of finding yourself in a particular state is very much affected by where you started from. So what we want to do next is to abstract from these two examples and describe the general structural properties that have to do with periodicity, and that have to do with what happened here with certain states, not being accessible from the others.

We're going to leave periodicity for next time. But let's talk about the second kind of phenomenon that we have. So here, what we're going to do is to classify the states in a transition diagram into two types, recurrent and transient. So a state is said to be recurrent if the following is true. If you start from the state i, you can go to some places, but no matter where you go, there is a way of coming back. So what's an example for the recurrent state? This one. Starting from here, you can go elsewhere. You can go to state 7. You can go to state 6. That's all where you can go to. But no matter where you go, there is a path that can take you back there.

So no matter where you go, there is a chance, and there is a way for returning where you started. Those states we call recurrent. And by this, 8 is recurrent. All of these are recurrent. So this is recurrent, this is recurrent. And this state 5 is also recurrent. You cannot go anywhere from 5 except to 5 itself. Wherever you can go, you can go back to where you start. So this is recurrent.

If it is not the recurrent, we say that it is transient. So what does transient mean? You need to take this definition, and reverse it. Transient means that, starting from i, there is a place to which you could go, and from which you cannot return. If it's recurrent, anywhere you go, you can always come back. Transient means there are places where you can go from which you cannot come back.

So state 1 is recurrent - because starting from here, there's a possibility that you get there, and then there's no way back. State 4 is recurrent, starting from 4, there's somewhere you can go and-- sorry, transient, correct. State 4 is transient starting from here, there are places where you could go, and from which you cannot come back. And in this particular diagram, all these 4 states are transients.

Now if the state is transient, it means that there is a way to go somewhere where you're going to get stuck and not to be able to come. As long as your state keeps circulating around here, eventually one of these transitions is going to happen, and once that happens, then there's no way that you can come back. So that transient state will be visited only a finite number of times. You will not be able to return to it. And in the long run, you're certain that you're going to get out of the transient states, and get to some class of recurrent states, and get stuck forever.

So, let's see, in this diagram, if I start here, could I stay in this lump of states forever? Well as long as I'm staying in this type of states, I would keep visiting states 1 and 2 Each time that I visit state 2, there's going to be positive probability that I escape. So in the long run, if I were to stay here, I would visit state 2 an infinite number of times, and I would get infinite chances to escape. But if you have infinite chances to escape, eventually you will escape. So you are certain that with probability 1, starting from here, you're going to move either to those states, or to those states.

So starting from transient states, you only stay at the transient states for random but finite amount of time. And after that happens, you end up in a class of recurrent states. And when I say class, what they mean is that, in this picture, I divide the recurrent states into 2 classes, or categories. What's special about them? These states are recurrent. These states are recurrent. But there's no communication between the 2. If you start here, you're stuck here. If you start here, you are stuck there.

And this is a case where the initial state does matter, because if you start here, you get stuck here. You start here, you get stuck there. So depending on the initial state, that's going to affect the long term behavior of your chain. So the guess you can make at this point is that, for the initial state to not matter, we should not have multiple recurrent classes. We should have only 1. But we're going to get back to this point next time.

Markov Chains I (PDF)

Free Downloads

Video

• iTunes U (MP4 - 114MB)
• Internet Archive (MP4 - 114MB)

Subtitle

• English - US (SRT)