Probability 1
The ASTA team
Introduction to probability
Events
Consider an experiment.
The state space \(S\) is the set of all possible outcomes.
Example: We roll a die. The possible outcomes are \(S=\{1,2,3,4,5,6\}\).
Example: We measure wind speed (in m/s). The state space is \([0,\infty)\).
An event is a subset \(A\subseteq S\) of the sample space.
Example: Rolling a die and getting an even number is the event \(A=\{2,4,6\}\).
Example: Measuring a wind speed of at least 5m/s is the event \([5,\infty)\).
Combining events
Consider two events \(A\) and \(B\).
The union \(A\cup B\) of is the event that either \(A\) or \(B\) occurs.
The intersection \(A\cap B\) of is the event that both \(A\) and \(B\) occurs.
- The complement \(A^c\) of \(A\) of is the event that \(A\) does not occur.
Example: We roll a die and consider the events \(A=\{2,4,6\}\) that we get an even number and \(B=\{4,5,6\}\) that we get at least 4. Then
Probability of event
The probability of an event is the proportion of times the event \(A\) would occur when the experiment is repeated many times.
The probability of the event \(A\) is denoted \(P(A)\).
Example: We throw a coin and consider the outcome \(A=\{Head\}\). We expect to see the outcome Head half of the time, so \(P(Head)=\tfrac{1}{2}\).
Example: We throw a coin and consider the outcome \(A=\{4\}\). Then \(P(4)=\tfrac{1}{6}\).
Properties:
\(P(S)=1\)
\(P(\emptyset)=0\)
\(0\leq P(A) \leq 1\) for all events \(A\)
Probability of mutually exclusive events
Consider two events \(A\) and \(B\).
If \(A\) and \(B\) are mutually exclusive (never occur at the same time, i.e. \(A\cap B=\emptyset\)), then
\[ P(A\cup B) = P(A) + P(B). \] 
- Example: We roll a die and consider the events \(A=\{1\}\) and \(B=\{2\}\). Then
\[P(A\cup B) = P(A) + P(B) = \tfrac{1}{6} + \tfrac{1}{6} = \tfrac{1}{3}. \]
Probability of union
- For general events \(A\) an \(B\),
\[ P(A\cup B) = P(A) + P(B) - P(A\cap B).\]
- Example: We roll a die and consider the events \(A=\{1,2\}\) and \(B=\{2,3\}\). Then \(A \cap B =\{2\}\), so
\[P(A\cup B) = P(A) + P(B) -P(A\cap B)= \tfrac{1}{3} + \tfrac{1}{3} - \tfrac{1}{6} = \tfrac{1}{2}.\]
Probability of complement
- Since \(A\) and \(A^c\) are mutually exclusive with \(A\cup A^c = S\), we get \[ 1= P(S) = P(A\cup A^c) = P(A) + P(A^c), \] so \[ P(A^c) = 1 - P(A).\]
Conditional probability
Example: We toss a coin two times. The possible outcomes are \(S=\{HH,HT,TH,TT\}\). Each outcome has probability \(\tfrac{1}{4}\). What is the probability of at least one head if we know there was at least one tail?
- Let \(A=\{\text{at least one H}\}\) and \(B=\{\text{at least one T}\}\). Then \[ P(A|B) = \frac{P(A\cap B)}{P(B)} = \frac{2/4}{3/4} = \frac{2}{3}.\]
Independent events
Two events \(A\) and \(B\) are said to be independent if \[ P(A|B) = P(A).\]
Example: Consider again a coin tossed two times with possible outcomes \(HH,HT,TH,TT\).
Let \(A=\{\text{at least one H}\}\) and \(B=\{\text{at least one T}\}\).
We found that \(P(A|B) = \tfrac{2}{3}\) while \(P(A) = \tfrac{3}{4}\), so \(A\) and \(B\) are not independent.
Independent events - equivalent definition
Two events \(A\) and \(B\) are said to be independent if and only if \[ P(A\cap B) = P(A)P(B).\]
Proof: \(A\) and \(B\) are independent if and only if \[ P(A)=P(A| B) = \frac{P(A\cap B)}{P(B)}. \] Multiplying by \(P(B)\) we get \(P(A)P(B)=P(A\cap B)\).
Example: Roll a die and let \(A=\{2,4,6\}\) be the event that we get an even number and \(B=\{1,2\}\) the event that we get at most 2. Then,
- \(P(A\cap B) = P(2) =\tfrac{1}{6}\)
- \(P(A)P(B)= \tfrac{1}{2}\cdot \tfrac{1}{3} =\tfrac{1}{6}\).
- So \(A\) and \(B\) are independent.
Stochastic variables
Definition of stochastic variables
A stochastic variable is a function that assigns a real number to every element of the state space.
Example: Throw a coin three times. The possible outcomes are \[S=\{HHH,HHT,HTH,HTT,THH,THT,TTH,TTT\}.\]
- The random variable \(X\) assigns to each outcome the number of heads, e.g. \[X(HHH)=3,\quad X(HTT)=1.\]
Example: Consider the question whether a certain machine is defect. Define
- \(X =0\) if the machine is not defect,
- \(X=1\) if the machine is defect.
Example: \(X\) is the temperature in the lecture room.
Discrete or continuous stochastic variables
A stochastic variable \(X\) may be
Discrete: \(X\) can take a finite or infinite list of values.
Examples:
Number of heads in 3 coin tosses (can take values \(0,1,2,3\))
Number of machines that break down over a year (can take values \(0,1,2,3,\ldots\))
Continuous: \(X\) takes values on a continuous scale.
Examples:
- Temperature, speed, mass,…
Discrete random variables
Discrete random variables
Let \(X\) be a discrete stochastic variable which can take the values \(x_1,x_2,\ldots\)
The distribution of \(X\) is given by the probability function, which is given by \[f(x_i)=P(X=x_i), \quad i=1,2,\ldots\]
Example: We throw a coin three times and let \(X\) be the number of heads. The possible outcomes are \[S=\{HHH,HHT,HTH,HTT,THH,THT,TTH,TTT\}.\] The probability function is
- \(f(0) = P(X=0) =\tfrac{1}{8}\)
- \(f(1) = P(X=1) =\tfrac{3}{8}\)
- \(f(2) = P(X=2) =\tfrac{3}{8}\)
- \(f(3) = P(X=3) =\tfrac{1}{8}\)
The distribution function
Let \(X\) be a discrete random variable with probability function \(f\). The distribution function of \(X\) is given by \[F(x)=P(X\leq x) = \sum_{y\leq x} f(y), \quad x\in \mathbb{R}.\]
Example: For the three coin tosses, we have
- \(F(0) = P(X\leq 0) =\tfrac{1}{8}\)
- \(F(1) = P(X\leq 1) = P(X=0)+ P(X=1) = \tfrac{1}{2}\)
- \(F(2) = P(X\leq 2) = P(X= 0 ) + P(X=1) + P(X=2) =\tfrac{7}{8}\)
- \(F(3) = P(X\leq 3) = 1\)
- For a discrete variable, the result is an increasing step function.
A few examples
The binomial distribution: An experiment with two possible outcomes (success/failure) is repeated \(n\) times. Let \(X\) be the number of successes. Then \(X\) can take the values \(0,1,\ldots,n\).
Example: Flip a coin \(n\) times. In each flip, the probability of head is \(p=\tfrac{1}{2}\). Let \(X\) be the number of heads.
Example: We buy \(n\) items of the same type. Each has probability \(p\) of being defect. Let \(X\) be the number of defect items.
\[P(X=x) = \binom{n}{x}p^x(1-p)^{n-x}, \quad x=0,1,\ldots,n\]
The Poisson distribution is the natural distribution for counting variables.
Example: Number of cars passing on a road within one hour. Number of radioactive decays from a radioactive material within a fixed time period.
\[P(X=x ) = \exp(-\lambda x )\frac{\lambda^x}{x!}, \quad x=0,1,2,\ldots\]
Mean of a discrete variable
- The mean or expected value of a discrete random variable \(X\) with values \(x_1,x_2,\ldots\) and probability function \(f(x_i)\) is \[\mu = E(X) = \sum_{i} x_iP(X=x_i) = \sum_{i} x_if(x_i).\]
Interpretation: A weighted average of the possible values of \(X\), where each value is weighted by its probability. A sort of “center” value for the distribution.
Example: Toss a coin 3 times. What are the expected number of heads? \[E(X) = 0 \cdot P(X=0) + 1\cdot P(X=1) + 2\cdot P(X=2) + 3\cdot P(X=3) \\
= 0 \cdot \tfrac{1}{8} + 1\cdot \tfrac{3}{8} + 2\cdot \tfrac{3}{8} + 3\cdot \tfrac{1}{8}= 1.5.\]
Variance of a discrete variable
- The variance is the mean squared distance between the values of the variable and the mean value. More precisely, \[\sigma^2 = \sum_{i} (x_i-\mu)^2P(X=x_i) = \sum_{i} (x_i-\mu)^2f(x_i).\]
A high variance indicates that the values of \(X\) have a high probability of being far from the mean values.
- The standard deviation is the square root of the variance \[\sigma = \sqrt{\sigma^2}.\]
The advantage of the standard deviation over the variance is that it is measured in the same units as \(X\).
Example Let \(X\) be the number of heads in 3 coin tosses. What is the variance and standard deviation?
- Solution: The mean was found to be \(1.5\). Thus, \[\sigma^2 = (0-1.5)^2 \cdot f(0) + (1-0.5)^2\cdot f(1) + (2-1.5)^2\cdot f(2) + (3-1.5)^2\cdot f(3) \\
= (0-1.5)^2 \cdot \tfrac{1}{8} + (1-0.5)^2\cdot \tfrac{3}{8} + (2-1.5)^2\cdot \tfrac{3}{8} + (3-1.5)^2\cdot \tfrac{1}{8}= 0.75.\] The standard deviation is \(\sigma = \sqrt{0.75} \approx 0.866.\)
Continuous random variables
Distribution of continuous random variables
- The distribution of a continuous random variable \(X\) is given by a probability density function \(f\), which is a function satisfying
\(f(x)\) is defined for all \(x\) in \(\mathbb{R}\),
\(f(x)\geq 0\) for all \(x\) in \(\mathbb{R}\),
\(\int_{-\infty}^{\infty} f(x)dx = 1\).
- The probability that \(X\) lies between the values \(a\) and \(b\) is given by
\[P(a<X<b) = \int_a^b f(x) dx.\]
Notes:
Condition 3. ensures that \(P(-\infty < X < \infty) = 1\).
The probability of \(X\) assuming a specific value \(a\) is zero, i.e. \(P(X=a)=0\).
Density shapes
Distribution function of continuous variable
Let \(X\) be a continuous random variable with probability density \(f\). The distribution function of \(X\) is given by \[F(x)=P(X\leq x) = \int_{-\infty}^{x} f(y) dy, \quad x\in \mathbb{R}.\]
Example: For the uniform distribution on \([0,1]\), the density was
\[
f(x)=
\begin{cases}
1, & 0 \leq x \leq 1, \\
0, & \text{otherwise.}
\end{cases}
\] Hence, \[F(x)=P(X\leq x)=\int_{-\infty}^x f(y) dy = \int_0^x 1 dy = x, \quad x\in [0,1].\]
Mean and variance of a continuous variable
- The mean or expected value of a continuous random variable \(X\) is \[\mu = E(X) = \int_{-\infty}^{\infty}xf(x) dx.\]
- The variance is given by \[\sigma^2 = \int_{-\infty}^\infty (x-\mu)^2f(x)dx.\]
Consider again the uniform distribution on the interval \((0,1)\) with density \[
f(x)=
\begin{cases}
1 & 0 \leq x \leq 1 \\
0 & \text{otherwise}
\end{cases}
\] Find the mean and variance.
Solution: The mean is \[\mu = E(X) =\int_{-\infty}^\infty xf(x) dx = \int_{0}^1 x \cdot 1 dx = \left[\tfrac{1}{2}x^2\right]_0^1 = \tfrac{1}{2},\] and the variance is \[\sigma^2 = \int_{-\infty}^\infty (x-\mu)^2f(x) dx = \int_{0}^1 (x-\tfrac{1}{2})^2dx \\
= \left[\tfrac{1}{3}(x-\tfrac{1}{2})^3\right]_0^1 = \tfrac{1}{12}.\]
Rules for computing mean and variance
- Let \(X\) be a random variable and \(a,b\) be constants. Then,
- \(E(aX + b) =aE(X) + b\) .
- \(\text{Var}(aX+b) = a^2\text{Var}(X)\).
