Markov Analysis Accurately Models Dynamic Behaviors

Combinatorial models such as reliability block diagrams (RBDs) and fault trees are used to predict the reliability of complex systems. However, they cannot accurately model such dynamic system behaviors as:

• Repairs.
• Common-cause and dependent failures.
• Shocks (shared loads and induced failures).
• Sequence/state-dependent failure rates (standby components).
• Variable configurations.
• Complex error handling and recovery mechanisms (common pool of repair technicians).
• Phased mission requirements.

Because of their flexibility, generalized stochastic processes are widely used to assess system reliability and related characteristics in mission critical systems.

Stochastic Processes

Stochastic processes have a number of states that describe the behavior of a set of random variables. The behavior of the stochastic process varies with respect to an index. In reliability engineering, this index is generally system time. This means that the stochastic process is used to describe the dynamics of a system with respect to time.

State space is the set of all possible states of a process, and index space is a set of all possible index values. At a particular time (index value), a system will be in one of its possible states. In each state, a set of events can occur. The occurrence distribution of each state depends on the history of the system (all previous events and state transition times).

In reliability engineering, the state space is generally discrete. For example, a system might have two states: good and failed. There are, however, applications in which state space can be continuous. Examples include the water level in a tank (where tank failure characteristics depend on the water level), the load on a shaft, the waiting time for repair, etc. If the state space is discrete, then the process is called a chain.

Similarly, the state index can be discrete or continuous. In most reliability engineering applications, the state index (time scale) is continuous, which means that component failure and repair times are random variables. However, cases exist where the state index is discrete. Examples include time-slotted (synchronous) communication protocol, shifts in equipment operation, etc.

Markov Processes

Markov processes are a special class of stochastic processes that uniquely determine the future behavior of the process by its present state. This means that the distributions of events (rates of occurrences) are independent of the history of the system. Furthermore, the transition rates are independent of the time at which the system arrived at the present state. Thus, the basic assumption of a Markov process is that the behavior of the system in each state is memoryless. The transition from the current state of the system is determined only by the present state and not by the previous state or the time at which it reached the present state. Before a transition occurs, the time spent in each state follows an exponential distribution.

In reliability engineering, these conditions are satisfied if all events (failures, repairs, switch-overs, etc.) in each state occur with constant occurrence rates (failure rate, repair rate, switch-over rate, etc.). Because the basic behavior of the process is time-independent, these processes are also called Time Homogeneous Markov processes or simply Homogeneous Markov processes. However, failure and repair rates of a component can depend upon the current state. Because of constant transition rate restriction, the Homogenous Markov process should not be used to model the behavior of systems that are subjected to component wear-out characteristics. General stochastic processes should be used instead.

In most cases, special classes of the stochastic processes that are generalizations to the Homogenous Markov processes are used. The corresponding models include:

• Semi-Markov models. Although very similar to Homogeneous Markov models, the transition times and the probabilities (distributions) depend on the time at which the system reached the present state. This means that the transition rates in a particular state depend on the time already spent in that state, but that they do not depend on the path by which the present state was reached. Thus, transition distributions can be non-exponential.
• Non-homogeneous models. Although very similar to Homogeneous Markov models, the transition times depend on the global system time rather than on the time at which the system reached the current state.

A non-exponential distribution (such as normal or Weibull) can be approximated as a set of exponential distributions. In this case, even the distributions are non-exponential, and homogeneous Markov models can be used. However, the results are approximate.

As noted earlier, Markov processes are classified based on state space and index space characteristics. The following table lists the characteristics of the four types of Markov processes and their corresponding model names.

State Space	Index Space	Common Model Name
Discrete	Discrete	Discrete Time Markov Chains
Discrete	Continuous	Continuous Time Markov Chains
Continuous	Discrete	Continuous State, Discrete Time Markov Processes
Continuous	Continuous	Continuous State, Continuous Time Markov Processes

Markov Model Types

In most reliability engineering applications, the state space is discrete and the index space (time scale) is continuous. Thus, Discrete State Space, Continuous Index Space Homogenous Markov processes are the most commonly implemented. Because the term Markov chain is generally used whenever state space is discrete, the above table refers to these models as Continuous Time Markov Chains. In many textbooks, these models are simply called Continuous Markov Models.

In addition to being an important concept in reliability analysis, Markov models find wide applications in other areas, including:

• Artificial music.
• Spread of epidemics.
• Traffic on highways.
• Occurrence of accidents.
• Growth and decay of living organisms.
• Emission of particles from radioactive sources.
• Number of people waiting in a line (queue).
• Arrival of telephone calls at a particular telephone exchange.

Markov models are the only accurate method for modeling complex situations. The complex proofs related to these models can be found in many reliability engineering handbooks and related publications.

Limitations of Homogeneous Markov Models

Homogeneous Markov models are limited by two major assumptions:

• The transitions (probabilities) of changing from one state to another are assumed to remain constant. Thus, a Markov model is used only when a constant failure rate and repair rate assumption is justified.
• The transition probabilities are determined only by the present state and not by the system's history. This means future states of the system are assumed to be independent of all but the current state of the system.

Part 2 of this article will discuss the creation of state transition diagrams for Markov analyses.

If you would like additional information about Markov Analysis and how it is implemented in the Relex Reliability Software Suite, please email info@relex.com.