The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare continue to
offer high-quality educational resources for free.
To make a donation or view additional materials from
hundreds of MIT courses, visit MIT OpenCourseWare at
ocw.mit.edu.
SHAN-YUAN HO: OK.
So today's lecture is going to be on
Finite-state Markov Chains.
And we're going to use the matrix approach.
So in last lecture, we saw that the Markov chain, we
could represent it as a directed graph or as a matrix.
So the outline is we will look at this transition matrix and
its powers.
And then we'll want to know whether this p of n is going
to converge for very, very large n.
Then we will extend this to Ergodic Markov chains, Ergodic
unichains, and other finite-state Markov chains.
So remember in the Markovity, these Markov chains, the
effect of the past on the future is totally summarized
by its state.
So we want to analyze the probabilities of properties of
the sequence of these states.
So whatever the state you are in, all the past is totally
summarized in that state.
And that's the only thing that affects the future.
So an ergodic Markov chain is a Markov chain that has a
single recurrent class and is aperiodic.
So this chain doesn't contain any transient states.
And it doesn't contain any periodicity.
So an ergodic unichain is just ergodic Markov chain, but it
has some transient states in it.
So the state x sub n of this Markov chain at step n depends
only on the past through the previous step.
So for n steps, we want to be at state j.
And then we have this path.
x sub n minus 1 is i, and so forth, up to x0.
It's just the probability from i to j, from
state i to state j.
So this means that we can write the joint probability of
all these states that we're in, so x0, x1, all the way up
to xn, as a function of these transition probabilities.
So in this transition probability matrix, we can
represent these transition probabilities.
We see that here, in this example, this is a 6-state
Markov chain.
So if I want to go from, say, state 2 to state 1 in one
step, it would just be p of 2,1.
If I want to go from state 6 to itself--
this is last one, which is p of 6,6.
So this is a probably transition matrix.
So if we condition on the state at time 0 and then we
define this p of ijn is equal to the probability that we're
in state j at the n-th step, given that we start x0 is
equal to i, let's look at what happens when n is equal to 2.
So in a 2-step transition, we go from i to j.
It's just the probability that at step 2, x2 is equal to j,
x1 is equal to some k, and x0 is equal to i.
So remember, we started in state i.
But this has to be multiplied by probability that x1 is
equal to k, given that x0 is equal to i.
And we have to sum this over all the states k, in order to
get the total probability from--
Oh, stand back?
OK.
There.
OK.
So this is just probability of ij in two steps.
It's just the probability of i going to k times the
probability of k going to j, summed over all k states.
So we notice that this term right here, the sum over k or
ik, kj is just the ij term of the product of the transition
matrix P with itself.
So we represent this as P squared.
So we multiply the transition matrix by itself.
This gives us the 2-step transition matrix of this
Markov chain.
So if you want to go i to j, you just look at ij element in
this matrix.
And that gives you the probability in two steps,
going from state i to state j.
So for n, we just iterate on this for
successively larger n.
So for n state to get from state i to state j, we just
have this probability x sub n, given j, given x of n minus
the previous step is equal to k, x sub n minus 1 equals k,
given x0 is equal to i, summing over all k.
So this means that we broke this up for n-th step.
In the n minus one step, we visited state k.
And then we multiplied that one-step transition from k to
j because we want to arrive at j starting at i.
But again, we have to sum over all the k's in order to get
the probability from i to j in n steps.
So p of n right here, this representation is just the
transition matrix multiplied by itself n times.
And this gives you the n-th step transition probabilities
of this Markov chain.
So computationally, what you do is you take p, p squared, p
to the fourth.
If you wanted p to the 9th, you'd just take p eighth
multiplied by p, to to multiply by this.
So this gives us this thing called the Chapman-Kolmogorov
equations, which means that when we want to go from step i
to step j, we can go to an intermediate state and then
sum up all the states that would go into the
intermediate state.
So in this case, if the step is m plus n transition, we can
break it up into m and n.
So it's the probability that it goes from i to k in exactly
m steps and k to j in n steps, summing over all the k's that
it visits on its way from i to j.
So this is very useful a quantity that we can
manipulate our transition probabilities when we get
higher orders of n.
So the convergence of p to the n.
So a very important question we like to ask is as n goes to
infinity whether this goes to a limit or not.
In other words, does the initial state matter, all
initial sates matter in this Markov chain?
So the Markov chain is going to go on for a long, long,
long, long, long time.
And at the n-th state where n is very large, is it going to
depend on i?
Or is it going to depend on n, which is the number of steps?
If it goes to this quantity, some limit, then it won't
depend on this.
So let's assume that this limit exists.
If this limit does exist, we can take the sum of this limit
and then multiply it by p of jk, summing over all j.
So we do a sum of over j.
So we're going from j to k on both sides, and we
sum over all j.
So we take this limit right here.
We notice that this left side going from i to k to n plus 1
is just that this limit at state k exists.
Because we saw assumed up here that this exists for
all i and all j.
So therefore, if we take the n plus 1 step, we take this n
going to infinity of i to k, it has to go to pi of k.
So when we do this, we could simplify
this equation up here.
And if it doesn't exist, we have this pi sub k for all the
states in the Markov chain.
So this is just a vector.
So pi sub k is equal to pi sub j times the probability from k
to j, summed over all j.
So if you have an m state Markov chain, you have exactly
m of these equations.
And this one, we'll call it the vector pi, which consists
of each element of this equation, if the limit is
going to exist.
But we don't know whether it does or not, at
this point in time.
So if it does exist, what's going to happen?
So that means I'm going to multiply this probability
matrix, P times P,P P, P, P, P, P all the way.
And if the limit exists, then that means for each row, they
must be all identical.
Because we said the limit exists, then going from 1 to
j, 2 to j, 3 to j, 4 to j, they should be
all exactly the same.
This is also the equivalent of saying, when I look at this
large limit, as n is very, very large if the limit
exists, that all the elements in the column should be
exactly the same as well, or all the rows.
So the elements are equal to each other, or all the rows,
if I look at the row, which is going to be this pi vector.
They should be the same.
So we define this vector.
If this limit exists, the probability vector
is this vector pi.
Because we said it was an m state Markov chain.
Each pi sub i is non-negative, and they obviously have
to sum up to 1.
So this is what we call a probability vector, called the
steady-state vector, for this transition
matrix P, if it exists.
So what happens is this limit is easy to study.
In the future in the course, we will study these pi P, this
steady-state vector for various Markov chains.
And so you see, it is quite interesting, many things that
could come about it.
So we notice that this solution can
contain more than one.
It may not be unique.
So if it contains more than one, it's very possible that
it has more than one solution, more than one probability
vector solution.
But just because a solution exists to that, it doesn't
mean that this limit exists.
So we have prove that limit exists, first.
So for ergodic Markov chain, here we have another way to
express this that this matrix converges is that the matrix
of the rows--
the elements in the column are all the same for each i.
So we have this theorem.
And today's lecture is going to be completely this theorem.
This theorem says that if you have an ergodic finite-state
Markov chain-- so when we say "ergodic," remember it means
that there's only one class, every single state in this is
recurrent, you have no transient states, and you have
no periodicity.
So it's an aperiodic chain.
And then for each j, if you take the maximum path from i
to j in n steps, this is non-increasing in n.
So in other words, this right here, this is non-increasing.
So if I take the maximum path from state i to j, it gives
you exactly n steps.
So that means this is maximized over all
initial states i.
So it doesn't matter what state you start, and I take
the maximum path.
And if I increase n, and I take maximum of that again,
the maximum path, this is not increasing.
And the minimum is non-decreasing in n.
So as we take n, the path from i to j, this n getting larger
and larger, we have that the maximum of this path, which is
the most probable path, is non-increasing.
And then the minimum of this path, the least likely path,
is going to be non-decreasing.
So we're wondering whether this limit is going to
converge or not.
In this theorem it said that for an ergodic finite-state
Markov chain, this limit actually does converge.
So in other words, the lim sup is equal to lim if of this and
will equal pi sub j, which is the steady-state distribution.
And not only that, this convergence is going to be
exponential in n.
So this is the theorem that we will prove today.
So the key to this theorem is this pair statements, that the
most probable path from i to j, given n steps--
so this is the most probable path--
is non-increasing at n, and the minimum is
non-decreasing in n.
So the proof is almost trivial, but let's see what
happens in this.
So we have a probably transition matrix.
So this is the statement right here.
And the transition is just one here and one here, with
probability 1, 1.
In this case, we want to say, what is the maximum path that
we're in state 2, given n steps?
So we know that this probability alternates between
1 and 2, it's non-increasing, it's not decreasing, it's
always the same.
So those two bounds are met with equality.
So in this here.
So the second example is this.
We have a two-state chain again.
But this time, from 1 to 2, we have the transition of 3/4.
So that means that we have a chain here of 1/4.
See, the minute we put a self-loop in here, it
completely destroys the periodicity.
Any Markov chain, you put a self-loop in it, and the
periodicity is destroyed.
So here we have 3/4.
So this has to come back with 1/4.
All right.
So in this one, let's look at the n step going from 1 to 2.
So basically, we want to end up in state 2
in exactly n steps.
So when n is equal to 1, what is the maximum?
The maximum is if you start it in this state and then you
went to state 2.
The other alternative is you start at state 2, and you stay
in state 2.
Because we want to end at state 2 in exactly one step.
So the maximum is going to be 3/4, and the minimum
is going to be 1/4.
You get n is equal to 2.
Now we want to end up in state 2 in two steps.
So what is going to be the maximum?
The maximum is going to be if you visit
state 1 and then back.
So n is equal to 1.
Then P1 from 1 to 2 is equal to 3/4.
So the probability from 1 to 2 in two steps is equal to 3/8.
So it goes 1/4 plus 3/4, 3/4 plus 1/4.
It should be equal to 3/8, right?
Is that right?
OK.
And then it when P1,2 to 3, if there are three transitions
from 1 to 2, then it's equal to 9/16.
So if for 2, if I want to transition
from 2 to 2 n steps--
so P2,2 is equal to 1/4.
So it just stayed by itself.
So P2,2 in two steps, you don't have a choice.
You have to go from 3/4 to 3/4.
So that's 9/16.
But For thing is I can also stay here by 1/4 times 1/4.
So that gives me 5/8 and so forth.
So basically, the sequence going from 1 to 2 is going to
be oscillating between 3/4, 3/8, 9/16, and so forth.
And then going from 2,2, it's going to be oscillating too.
We can see that's 1/4, 5/8, 7/16.
So what happens is this oscillation is going to
converge-- it's going to approach, actually, 1/2.
So if we take the maximum of these two, so P1,2 and P2,2--
because that means that we're going to end at state 2.
And maximum over n steps, then we just look at these two
numbers, the 3/4 and 1/4, if we want the maximum, then it's
going to be 3/4.
For the 3/8 and 5/8, the maximum is going to be 5/8,
the 9/16 and 7/16, the 9/16 will be the maximum.
And similarly, we compare it, and we take the minimum.
And the minimum is 1/4, 3/8, and 7/16.
So we see that the maximum is going to be--
it starts high.
And then it's going to decrease toward 1/2.
And the minimum, what happens is it's going to start low,
and then it's going to increase to 1/2.
So this is exactly this one here.
So P's transition makes this an arbitrary finite-state
Markov chain.
Therefore, each j, this maximum path, the most problem
path from i to j in n steps is non-increasing n.
And the minimum is non-decreasing n.
So you take n plus 1 steps from i to j.
So we're going to use that Chapman-Kolmogorov equation.
So we take the first step to some state k.
And then we go from k to j in n steps.
But then we sum this over all k.
But this P n for state to j to k in n steps, I can just take
the maximum path.
So I take the most probable path, the state that gives me
the most probable path, and I substitute this in.
When I substitute this in, obviously every one of these
guys is going to be less than or equal to this.
Therefore, this outside term is going to be
less than or equal.
So now this is just going to be a constant.
So I sum over all k, and then this term remains.
So therefore, what we know is if I want to end up in state
j, and for n steps, if I increase the step more, to n
plus 1, we know that this probability is going to stay
the same or decrease.
It's not going to increase.
So you could do exactly the same thing for the minimum.
So if this is going to be true, then of course, if I
think the maximum of this, it's also going to
be less than that.
Because this limit's true for Markov chain.
It doesn't matter.
It just has to be a finite-state Markov chain.
So this is true for any finite-state Markov chain.
So if I take the maximum of this, it's less than or equal
to the maximum of the n-th step.
So n plus 1 steps, the path is going to be less probable when
I take the maximum path, the fact that I end up at
state j than n.
So before we complete the proof of this theorem, let's
look at this case where P is greater than zero.
So if we say Pis greater than zero, this means that every
entry in this matrix is greater than 0 for all i, j,
which means that this graph is fully connected.
So that means you could get from i to j in one step with
nonzero probability.
So if P is greater than 0--
and let this be the transition matrix.
So we'll prove this first, and then we'll extend it to the
arbitrary finite Markov chain.
So let alpha here is equal to the minimum.
So it's going to be the minimum element in this
transition matrix.
That means it's going to be the state that contains the
minimum transition.
So let's call alpha-- it's the minimum probability.
Excuse me.
So let all these states i and j.
And for n greater than or equal to 1, we have these
three expressions.
So this first expression says this, that if I have an n plus
1 walk from i to j, I take the most probable of
this walk over i.
So my choices, I can choose my initial starting state.
In n plus 1 steps, I want to end in state j.
So I pick the most probable path.
If I minus this, which is the least probable path--
but you get to minimize this over i, over the initial
starting a state.
So this is less than or equal to the n step.
It's exactly this term here, the n step
times 1 minus 2 alpha.
So alpha is the minimum transition probability in this
probability transition matrix.
So this one, it's not so obvious right now.
But we are going to prove that in the next slide.
So once we have this, we can iterative on n to get the
second term.
So for this term inside here, the most probable path to
state j in n steps, minus the least probable path to state j
in n steps, is equal to exactly the same thing in n
minus 1 steps times 1 minus 2 alpha.
So we just keep on iterating this over. n, and then we
should get this.
So to prove this to this, we prove it by induction.
We just have to prove the initial step, that the maximum
single transition from l to j, minus the minimum single
transition from l to j, is less than or equal
to 1 minus 2 alpha.
So this one is proved by induction.
So as n goes to infinity, notice that this term is going
to go to 0.
because alpha is going to be less than a 1/2.
Because if it's not, then we can choose 1 minus alpha to be
this minimum.
So if this is going to 0, this tells us the difference
between the most probable path minus the least probable path,
the fact that we end up in state j.
So if we take the limit as n goes to infinity of both of
these, they should equal.
Because the difference of this, we notice that it's
going down exponentially in n.
So this shows us that this limit indeed does
exist and is equal.
We want to prove this first statement over here.
So in order to prove this first statement, what we're
going to do is we're going to take this i, j transition in n
plus 1 transitions.
And then we're going to express it as a function of n
transitions.
So the idea is this.
We're going to use the Chapman-Kolmogorov equations
to have an intermediary step.
So in order to do this i to j in n plus 1 steps, the most
probable path, we're going to go to this intermediate step
and then on to the final step.
In this intermediate step, it's going to be
a function of n.
So we're going to take one step and then n more steps.
So what we're going to do is, the intuition is, we're going
to remove the least probable path.
So we remove that from the sum in this
Chapman-Kolmogorov equation.
And then we have the sum of everything else
except for that path.
And then the sum of everything else, we're going to bound it.
Once we bound it, then we have this expression.
The probability of i to j in n plus 1 steps is going be a
function of a max and a min over n steps
with a bunch of terms.
So that's the intuition of how we're going to do it.
So the probability of ij going from state i to state j in
exactly n plus 1 steps is equal to this.
So it's the probability of going from i to k, this
intermediate step.
We're going to take one step to a state k.
And then we're going from k to j in n steps,
summing over all k.
So this is exactly equal to this with Chapman-Kolmogorov.
So now what happens is we're going to take--
Before we get to this next step, let's define this l min
to be the state that minimizes p of ij, n over i.
So l min is going to be the state that's going to be such
that the choices I pick over i that in n steps I arrive at j
that's going to be the least probable.
So this is l min over here.
It's the l min that satisfies this.
Then I'm going to remove this.
So this is one state. l min is just one state that i is going
to go to in this first step.
So we're going to remove it from the sum.
So then, this is just here.
So that path goes from i to l min times l to j in n steps.
So remove that one path from here.
Now we have the sum over the rest of the cases because we
just removed that.
So we have ik, kj to n, where k is not
equal to that element.
So we removed that path, the one that goes to that state.
But p of kj, n, the path that goes from k to j in n steps,
we can just bound this term by the maximum over l
from l to j of n.
So then we're going to take the most probable path in n
steps such that we end up in state j in n.
So this term right here is bounded by this term.
Becomes is bounded by this, that's why we have this less
than or equal sign.
So we just do two things from this step, the first step, to
the second step.
So we took out the path that's going to minimize that right
at the j-th node in n steps.
And then we bounded the rest of this sum by this.
So when we sum this all up, this is just a constant here.
And ik here is just all the states that i is going to
visit except for this one state, l min.
Since it's just all of them except for that, it's just 1
minus the probability that it goes from state i to l min.
So this sum here is just equal to this sum here.
So this arrives here.
And this term is still here.
So going from here, what happens is we just to
rearrange the terms.
So nothing happens right here.
It's just rearranging.
Now we have this term here.
So we look at this term, P from i going to l min--
Remember, we chose alpha to be the minimum single transition
probability, single transition in that
probability transition matrix.
So i to l has to be greater than that.
But the minus of this has to be less than, the negative has
to be less than.
So this we can substitute here.
So now we have this.
So the maximum over i of this n plus 1 step actually shows
you the probability.
Because this I can write as an n plus 1 step
path from i to j.
So if this is less than this entire term, of course I can
write the maximum path from i to j.
It also has to be less of this because this is
satisfied for all i, j.
So therefore, we arrive at this expression here.
So now we're kind of in good business because we have the n
plus one step at transition, the maximum path from i to j
in n plus 1 steps as a function of n, which is what
we wanted, and a function of this alpha.
So we repeat that last statement.
And the last one is here, the last line.
So now we have the maximum.
So now we want to do is we want to get the minimum.
So we do exactly the same thing, with the same proof.
And with the minimum, what we're going to do is we're
going to look at the ij transition in n plus 1 steps.
And then what we're going to do is we're going to pull out
the maximum this time.
So we pull out the most probable path in n steps such
that it arrives in state j.
Then we play the same game.
Would bound everything--
above, this time-- by the minimum of the n step
transition probabilities to get to j.
So once we do that, we get this expression, very similar
to this one up here.
So now we have the maximum path, which is n plus 1 steps
to j, and the minimum of n plus 1 steps to j.
So we could take the difference between these two.
So if you subtract these equations here, so this first
equation minus the second equation, we have this on the
right-hand side here and then these terms over here on the
left-hand side.
So these terms over here on the left-hand exactly proves
the first line of the lemma.
So the first line of the lemma was here.
So now, to prove the second of the lemma, remember, we're
going to prove this by induction.
in order to prove this by induction, we need to be first
initial step.
So the initial step is this.
So if I take the minimum transition probability from l
to j, it has to be greater than here with the alpha.
Because we said that alpha was the absolute minimum of all
the single-step transition probabilities.
Then the maximum transition probability has to be greater
than or equal to 1 minus alpha.
It's just by definition of what we choose.
So therefore, if I take this term, the maximize minus the
minimum is just 1 minus 2 alpha.
So that's your first step in the induction process.
So we iterate on n.
When we iterate on n, one arrives at this
equation down here.
So this shows us from here that if we take the limit as n
goes to infinity of this term, this goes down
exponentially in n.
And both of these limits are going to converge, and they
exist, and they're going to be greater than 0.
So they'll be greater than 0 because of our initial state
that we chose this path with a positive probability.
Yeah, go ahead.
AUDIENCE: It seems to me that alpha is the minimum, the
smallest number in the transition matrix, right?
SHAN-YUAN HO: Alpha is the smallest number, correct.
AUDIENCE: Yeah.
How does it fall from that, like that?
So my is, the convergence rate is related to f?
SHAN-YUAN HO: Yes, it is, yeah.
In general, it doesn't really matter because it's still
going to go down exponentially in n.
But it does depend on that alpha, yes.
Any other questions?
Yes.
AUDIENCE: Is the strength that bound it proportional to the
size of that matrix, right?
SHAN-YUAN HO: Excuse me?
AUDIENCE: The strength of that bound is
proportional to the size?
I mean, for a very large finite-state Markov chain, the
strength of the bound is going to be somewhat weak because
alpha is going to be--
SHAN-YUAN HO: Alpha has to be less than 1/2.
AUDIENCE: OK, yes.
But the strength of the bound, though, it's not a very tight
bound on max minus min.
Because in a large--
SHAN-YUAN HO: Yes.
This is just a bound.
And the bound is what when we took it that
minimum-probability path, the l min, remember?
The bound was actually in here.
So we took the minimum-probability path in n
steps, this l min that minimizes this over i.
And then this is where this less than or equal to here is
just a substitution.
Any other questions?
So what we know is that what happens is that these
limited-state probabilities exist.
So we have a finite ergodic chain.
So if the probability of the elements in this transition
matrix are all greater than 0, we know
that this limit exists.
But we know that in general, that may not be the case.
We're going to have some 0's in our transition matrix.
So let's go back to the arbitrary finite-state ergodic
chain with probability transition matrix P. So in the
last slide, we showed that this transition matrix P of h
is positive for h is equal to M minus 1 squared plus 1.
So what we do is, we can apply lemma 2 to P of h with this
alpha equals to minimum going from i to j
in exactly h steps.
So why is this M minus 1 squared plus 1?
So in the last lecture--
so what it means is this.
So what is says is here.
This was an example given in the last lecture.
It was a 6-state Markov chain.
So what it says is that if n is greater than or equal to M
minus 1 squared plus 1-- in this case, it's going to be 6.
So if n is greater than or equal to 26, then I take P to
the 26th power, it means it's greater than zero.
That meas if I take P to the 26th power, every single
element in this transition matrix is going to be
non-zero, which means that you can go from any state to any
state with nonzero probability, as long as n is
bigger than that.
So basically, in this Markov chain, if you go long enough,
long enough.
Then I say, OK, I want to go from state i to state j in
exactly how many steps, there is a positive probability that
this is going to happen.
So how did this bound come across?
Well, for instance, in this chain, if we look at P1,1 so
we have here?
So I'm going to look at the transition
starting at state 1.
And I want to come back to 1.
So you definitely could come back at 6, because these are
all positive probability 1.
So 6 is possible.
So n is equal to 6 is possible.
So what's the next one that's possible? n is
equal to 11, right?
Then the next one is what?
16 is possible, right?
So 0 to 5 is impossible, is 0.
So if I pick n between 0 and 5, and 7 and 10, you're toast.
You can't get back to 1.
And so forth.
So 18 is possible.
21--
let's see, is 17 possible?
Yeah, 17 is also possible.
AUDIENCE: Why is 16 possible?
SHAN-YUAN HO: So I go around here twice, and
then the last one.
Is that right?
So if I go from here to here to here to here, if I go
twice, and then one more in the final loop.
AUDIENCE: That's 12.
SHAN-YUAN HO: Oh, it's 12?
No.
I'm going to go this inner loop right here.
So if I go from 1 to 2 to 3 to 4 to 5 to 6, down to 2.
Then I go 3, 4, 5, 6, 1.
That's 11, isn't it?
So 16 is I'm going to go around the inner loop twice.
OK.
Go ahead.
Question?
AUDIENCE: So everything 20 and under is possible, right?
SHAN-YUAN HO: No.
Is 25 possible?
Tell me how you're going to go 25 on this.
You just do the 5 loop 5 times.
SHAN-YUAN HO: Yeah, but I want to go from 1 to 1.
You're starting in state 1.
AUDIENCE: Oh, oh, sorry.
OK.
SHAN-YUAN HO: 1 to 1, right?
AUDIENCE: OK, cool.
OK, I see.
SHAN-YUAN HO: So you know for this one that this bound is
actually tight.
So 25 is impossible.
So P1,1 of 25 is equal to 0.
There's no way you can do that.
But for 26 on, then you can.
So what you're noticing is that you need this loop of 6
here and that any combination of 5 or 6 is possible.
So basically, in this particular example, if n is
equal to 6k plus 5j, where k is greater than or equal to
1-- because I need that final loop to get back--
or j is greater than or equal to 0--
So any combination of this one, then I can express n.
I can go around it to give me a positive probability of
going from state 1 to state 1.
So I'm going to prove this using extremal property.
So we're going to take the absolute worst case.
So the absolute worst case is that for M state finite Markov
chain is if have a loop of m and you have a
loop of m minus 1.
You can't just have a loop of m.
The problem is now this becomes periodic.
So we have to get rid of the periodicity.
If you add a single group here, that doesn't help you.
Then after 6, then it I get 7, 8, 9, 10, 11, 12.
That didn't have this.
So the absolute worst case for an M state chain is going to
be something that looks like this.
1 that goes to 2-- you're forced to go to 2--
so forth, until state M. And then this M is going to go
back to 2 or is going to go back to 1.
So in other words, the worst case is if you have--
n has to be some combination of Mk plus M minus 1 j.
So this will be the worst possible case for M state
Markov chain.
So it'll be Mk plus M minus 1 j.
So k has to be greater than or equal to 1.
And then j has to be greater than or equal to 0, because
you need to come back.
So I'm just looking at the case probability that I start
in state 1 and I come back in state 1.
So all right.
So how do we get this bound?
Well, there is an identity that says this.
If a and b are relatively prime, then the largest n such
that it cannot be written-- so we want to find the largest n
such that ak plus bj--
but this is k and j greater than or equal to 0--
that it cannot be written in this form.
The largest integer that it cannot be written is ab
minus a minus b.
This takes a little bit to prove, but it's not too hard.
If you want to know this proof, come see me offline
after class.
This is the largest integer.
If n is equal to this, it cannot be
written in this form.
But if n is greater than this, then it can.
So all we do is substitute M for a and M minus 1 for b
because M and M minus 1 are relatively prime.
But remember, we have a k here that has to be greater than or
equal to 1.
We need at least one k.
But this so identity is for k and j greater than 0.
So therefore, we have to subtract out that k.
So therefore, we have M times M minus 1, minus M
minus M minus 1.
But the thing is we have to add the extra M, because this
k is greater than or equal to 1.
So we have to add up one of the M's because of this.
So this is just equal to M minus 1, squared.
So this number, if n is equal to this, it's the largest
number that it cannot be written like that.
So therefore, we have to add 1.
So that's why the bound is equal to 1.
So the upper bound that n can be written is going to be M
minus 1, squared plus 1.
AUDIENCE: Why did you add the 1 at the end?
SHAN-YUAN HO: This one?
AUDIENCE: No, we've got to do the 1 at the end.
AUDIENCE: We already have that in there.
SHAN-YUAN HO: Oh, where is it?
No, it's in here, right?
AUDIENCE: No, it's not here.
SHAN-YUAN HO: Did I--
What are you talking about?
Where's the 1?
AUDIENCE: At the end, the last equation.
SHAN-YUAN HO: This one?
AUDIENCE: Yes.
SHAN-YUAN HO: OK.
This is the "cannot," largest n which you cannot write.
You cannot write this.
So this bound is tight.
It means that this is the one that you can.
So if n is greater than or equal to
this, then it's possible.
This is the largest one it cannot.
Based on this, it cannot.
So we have to add the 1.
So therefore, in here, you could do 26.
So starting from 26, 27, 28, you can do that.
Any questions?
AUDIENCE: Relatively prime, what do you mean by
"relatively"?
SHAN-YUAN HO: There is a greatest common divisor of 1.
So if we take h here, h is going to be positive.
So if h is equal to M minus 1, squared plus 1, then now all
the elements are positive.
Because we just proved that we can write this--
every state can be visited by any other state, with positive
probability.
So we say, looking at P, we know that P of h is positive
for h greater than or equal to this bound.
So what we do is we applied this lemma 2 probability to
this transition matrix P of h, where we have picked alpha--
remember, alpha is the single-step transition
probability.
So instead of the single transition, we have lumped
this P into P to the h power.
So it's h steps.
Because we proved the result before for positive P. So this
P to the h is positive, so we take alpha as the minimum from
i to j of P to the h in this matrix.
So it doesn't really matter what the value of alpha is,
only that it's going to be positive.
And it has to be positive because it's a probability.
So what happens is, if we follow the proof of what we
just showed in the lemma, then we show that the maximum path
from l to j--
h times M. So M is going to be an integer, so in
multiples of h--
this upper limit is going to be equal to the lower limit.
So the most probable path is equal to the
least probable path.
So this is multiple of h's.
So if we take this as M goes to infinity, this has
got to equal to--
Oops, this should be going to pi sub j, excuse me.
This little temple here.
And this is going to be greater than 0.
So the problem is now we've shown it for multiples of h's,
what about the h's in between?
But the fact is that lemma 1, we showed that this maximum
path from l to j in n is not increasing in n.
So all those states, all those paths, the transition
probability for the paths in between these multiples of
h's, in between them it's going to be not
increasing in n.
So even if we're taking these multiples of each of h and n
here, here, here, and we know that this limit is increasing,
we know that all the ones in between them are also going to
be increasing to the same limit because of lemma 1.
To remember, the maximum is going to be not increasing,
and the minimum is going to be
non-decreasing in any one path.
So this must have the same limit as
this multiple of this.
So the same limit applies.
So any questions on this?
So this is how we prove it for the arbitrary finite-state
ergodic chain when we have some 0 probability transition
elements in the matrix P. So the proof is the same.
So now for ergodic unichain.
So we see that this limit as n approaches infinity from i to
j of n is going to just end up in the steady-state transition
pi of j for all i.
So it doesn't matter what your initial state is.
As n goes to infinity of this path, as this Markov chain
goes on and on, you will end up in state j with probability
pi sub j, where pi is this probability vector.
So now we have this steady-state vector, and then
we can solve for the steady-state vector solution.
So this pi P is equal to pi.
Yeah?
Go ahead.
AUDIENCE: Where did you prove that the sum of all the pi j's
equal to one?
Because you say that we proved that this is
the probability vector.
But did prove only that it is non-negative?
SHAN-YUAN HO: It's non-negative.
But the thing is because as n goes to infinity, you have to
land up someone, right?
This is a finite-state Markov chain.
You have to be somewhere.
And the fact that you have to be somewhere, your whole state
space has to add up to 1.
Because it's a constant, remember?
For every j, as n goes to infinity, it goes to pi sub j.
So you have that for every single state.
And then you have to end up somewhere.
So if you have to end up somewhere, the space has to
add up to one.
Yeah, good question.
So why are we interested in this pi sub j?
The question is that because in this recurrent class, it
tells us that as this goes to infinity, we see this sequence
of states going back and forth, back and forth.
And we know that as n goes to infinity, we have some
probability, pi sub j, of landing in state j, pi sub i
of landing in state i, and so forth.
So it says that in the n step, as n goes to infinity, that
this is the fraction of time that, actually, that state is
going to be visited.
Because at each step, you have to make a transition.
So it's kind of the expected number of times per unit time.
So it's divide by n.
It's going to be that fraction of time that you're going to
visit that state.
It's the fraction of time that you're going
to be in that state.
It's this limiting state as n gets very, very large.
So we will see that in the next few chapters when we do
renewal theory that this will come into useful play.
And we give a slightly different viewpoint of it.
So it's very easy to extend this result to a more general
class of ergodic unichains.
So remember the ergodic unichains, now we have
increased these transient states.
So before, we proved this.
We just proved it for it contains exactly one class.
It's aperiodic, so we have no cycles, no periodicity in this
Markov chain.
And so we know that the steady-state transition
probabilities have a limit.
And the upper limit and the lower limit of these paths as
they go to infinity--
in fact, they end up in a particular state--
has a limit.
And we have this steady-state probability vector that
describes this.
So now we have these transient states.
So these transient states of this Markov chain, what
happens is there exists a path that this transient state is
going to go to a recurrent state.
So once it leaves this transient state, it goes to
recurrent state.
It's never going to come back.
So there is some probability, alpha, of leaving the
class at each step.
So there's some transition probability in this transient
state that's going to be alpha.
And the probability of remaining in this transient
state is just 1 minus alpha to the n.
And this goes down exponentially.
So what this says is that eventually, as n gets very
large, it's very, very hard to stay in that transient state.
So it's going to go out of the transient state.
And then it will go into the recurrent class.
So when one does the analysis for this, what happens in the
probability in this steady-state vector is those
transient states, this pi, will be equal to 0.
So this distribution is only going to be non-zero for
recurrent states in this Markov chains.
And the transient states will have probability equal to 0.
In the notes, they just extend the argument.
But you need a little bit more care to show this.
And it divides the transient states into a block and then
the recurrent classes into another block and then shows
that these transient states' limiting probability is going
to go to 0.
So let's see.
So this says just what I said, that these transient states
decay exponentially, and one of the paths will be taken,
eventually, out of it.
So for ergodic unichains, the ergodic class is eventually
entered, and then steady state in that class is reached.
So every state j, we have exactly this.
The maximum path from i to j in n steps--
and the minimum path.
We look at the minimum path in n steps and the maximum path
in n steps.
And for each n, we take the limit as n goes to infinity.
These guys, these limits are exactly equal, and it equals
to this pi sub j, which is equal to the j state.
So your initial states, how you went the paths that you
have gone is completely wiped out.
And all that matters is this final state,
as n gets very large.
So the difference here is that pi sub j equals 0 for each
transient state, and it's greater than 0 for the
recurrent state.
So other finite Markov chains.
So we can consider a Markov chain with
several ergodic classes.
Because we just considered it with one ergodic class.
So if the classes don't communicate, then you just
consider it separately.
So you figure out the steady-state transition
probabilities for each of the classes separately.
But if you have to insist on analyzing the entire chain P,
then this P will have m independent steady-state
vectors and one non-zero in each class.
So this P sub n is still going to converge, but the rows are
not going to be the same.
So basically, you're going to have blocks.
So if you have one class, say 1 through k is going to be in
one class, and then k through l is going to be another
class, and then l through z is going to another class, you
have a block.
So this steady-state vector is going to be in blocks.
So you can see the recurring classes only communicate
within themselves.
Because these don't
communicate, so they're separate.
So you could have a lot of 0's in limiting state, if you look
at this, P sub n goes to infinity.
So there m set of rows, one for each class.
And a row for each class k will be non-zero for the
elements of that class.
So then finally, if we have periodicity.
So now if we have a periodic recurrent chain with period d.
We had the two where it's just a period of 2.
So with periodicity, what you do is you're going to divide
these classes into d different states.
So you have to go to one state--
So if there's d states, this is a period of d, you separate
or you partition the states into d of them, d subclasses,
with a cycle rotation between them.
So basically, each time unit, you have to go from one class
to the next class.
And then we do that, then for each class, you could have the
limiting-state probability.
So in other words, you are looking at this transition
matrix, pi d.
Because when it cycles, it totally depends on which one
you start out at.
But if you look at the d intervals, then that becomes
the ergodic class by itself.
And there are exactly d of them.
So the limit as n approaches infinity of P of nd, this
thing also exists, but exists in the subclass sense of there
is d subclasses if it has a period of d.
So that means a steady state is reached within each
subclass, but the chain rotates from
one subclass to another.
Yeah, go ahead.
AUDIENCE: In this case, if we do a simple check with 1 and
2, with 1 and 1, it doesn't converge.
SHAN-YUAN HO: No, it does.
It is 1, converges to 1.
So it's 1, and then it's going to be 1.
AUDIENCE: It's 1, 1, 1, 1, 1, 1, 1, 1.
So you go here?
Like, it's reached--?
SHAN-YUAN HO: No, no.
It converges for here.
But this d is equal to 2, in that case.
So you have to do nd, so you've got
to look at P squared.
So if I look at P squared, I'm always a 1--
1, 1, 1, 1, 1, 1, 1, 1.
That's converging.
The other one is 2, 2, 2, 2, 2, 2.
That's also converging.
OK.
So is there any other questions about this?
OK, that's it.
Thank you.